►
From YouTube: 2021 09 08 Sharding Group AMA
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
A
A
And
we
have
two
questions
in
the
doc.
What
I'm
going
to
to
do
is
at
least
vocalize
the
first
one
then
we'll
see.
If
there
are
more
questions
there
is
a
slide
deck
that
actually
provides
an
overview
of
our
current
favored
strategy
for
scaling
our
database,
which
is
decomposition
and
we'll
see
how
it
goes.
If
folks
have
specific
questions
we'll
do
that,
otherwise
I
may
go
through
some
slides
and
provide
more
context.
A
Okay.
So
the
first
question
is
from
kenny
and
he
asks
where
could
we
read
more
about
the
plan
for
decomposition?
What
work
is
required
when
it's
expected
to
be
completed?
How
will
it
be
rolled
out
and
there
is
a
single
epic
6168-
I
linked
it
here.
I
also
put
it
through
the
links.
That
is
the
entry
point
for
all
of
the
work
that
is
being
done
by
the
sharding
group
right
now.
It
also
has
linked
work
items.
A
A
So
it's
a
it's
a
delicate
method
and
as
an
example,
there
is
also
there
are
work,
ethics
for
specific
sections,
so,
for
example,
dev
or
ops,
where
the
work
is
being
tracked
for
issues
that
are
being
found
out
so
go
check
that
out.
It's
all
in
the
epic
tree
and
then
second
question
from
from
nathan.
Are
you
here
to
vocalize
your
question.
B
A
B
A
A
Yeah
we'll
follow
up.
I
can
give
you
a
high
level
understanding
that
I
have
and
I'll
follow
up
with
andrew
thomas
who's,
the
infrastructure
product
manager.
A
B
C
You
I
have
the
next
question.
This
is
for
the
lay
person
potentially
is
sharding
or
decomposition
a
project,
or
is
this
a
new
ongoing
approach?
It's
the
first
time
I've
seen
the
word
in
print
when
I
got
invited
to
the
ama.
D
So
my
answer,
my
answer
to
that
is
like
we
identify
this
like
us
and
like
the
best
way
to
move
forward
to
buy
immediate
headroom
like
4
to
10
x.
This
is
what
we
are
kind
of
estimating
on
the
decomposition
and
really
depending
on
the
outcome
of
these
projects.
D
We
may
decompose
another
aspect
or
we
may
consider
starting
this
aspect,
because
what
we
are
right
now
doing
is
really
like
the
first
step
on
the
approach
to
improve.
Like
the
scalability
of
the
github,
I
mean
overall,
like
we
are
heading
into
more
horizontal
scalability
of
the
github,
but
in
order
to
get
to
that
point,
we
need
to
have
like
some
contained
complexity
of
existing
components,
to
figure
out
what
is
the
best
way
to
approach
it.
D
So
there
is
like
some
ongoing
work
on
solving
blueprint
that
tries
to
figure
out
and
describe
like
where
the
composition
and
sharding
is
on
the
spectrum
of
the
database.
Scalability
work
that
may
answer
some
of
your
questions.
D
C
Partially,
so
let
me
ask
it
a
different
way:
can
the
word
sharding?
Can
it
become
a
past
tense?
Like
great
news,
we
went
through
this
and
we
sharded
git
lab.
It
was
a
tremendous
project
and
a
great
success
and
it
was
a
significant
effort
to
get
us
ready
and
now
we
don't
have
to
do
it
again
or
is
it
from
now
on
in
perpetuity?
We
will
either
choose
one
of
these
and
it'll
just
be
the
way
we
design
and
develop
the
product.
So
if
I
could
just
rephrase
my
own
question,
does
that
help.
A
That's
a
great
question,
so
I
think
you
you
touch
on
something
really
important
here.
Ultimately,
what
we
are
concerned
with
the
scaling
gitlab
based
on
our
growth,
and
so
we
are
evaluating
several
strategies
on
how
we
can
scale
get
lab
and
sharding
or
decomposition
our
strategies
to
address
the
scalability
of
our
database
so
with
what
we're
doing
right
now,
which
is
essentially
we're
focusing
on
a
set
of
tables
for
our
continuous
integration,
we're
going
to
move
them
somewhere
else,
there's
an
end
date
to
that.
We
can
speak
of
that
in
the
past
tense.
A
At
some
point
and
say:
we've
done
this
right.
We've
accomplished
our
our
goal
here
and
at
that
point
we
can
evaluate
again
or
do
it
in
parallel
and
say
like
okay,
do
we
want
to
take
another
part
of
our
product
and
do
the
same
thing
over
again
or
do
we
need
to
think
about
sharding,
which
is
essentially
saying
we
select
a
namespace,
for
example,
so
something
that
you
can
loosely
corresponds
to
a
customer
and
we
start
horizontally
fanning
that
out
and
that
will
also
have
an
end
date.
A
If
you
add
new
customers,
for
example,
you
can
do
that
for
a
very
long
time
and
and
scales
that
may
get
us
to
100x,
whereas
if
you
decompose
ci
is
a
very
significant
part
of
our
rights
to
the
database,
so
once
we
move
that
somewhere
else,
that's
kind
of
done.
We
can't
we
can't
do
that
again
for
the
next
ci,
because
we
only
have
so
many
so
many
functions.
A
E
Yeah,
so
I
was
asking
how
the
kind
of
the
two
interrelate
right,
how
will
decomposition
impact
you
know?
I
know
we've
been
talking
about
charting
for
a
long
time.
E
Probably
ever
since
I've
been
here,
which
is
four
years,
and
I
knew
of
the
you
know-
issue
with
the
ci
table
growth-
I
this
is
the
first
I
heard
of
decomposition
so
kind
of,
like
tim,
not
the
first,
I
heard
of
sharding,
but
the
first
I
heard
a
decomposition.
So
I
was
wondering
how
the
two
things
will
impact
each
other
in
the
future
and
if
we
expect
like
our
self-serve
customers,
self-managed
customers
to
use
either
or
both
decomposition
or
sharding
in
the
future,.
D
So
we
don't
know
about
the
sharding
architecture
and
like
what
it's
gonna
be
looking
like
in
the
future.
For
example,
in
the
presentation
of
fabian,
you
could
write
that
we
may
use
application
level
sharding
like
a
horizontal,
or
we
may
want
to
use
some
database
technology
like
evitas
or
maybe
cytos.
D
It's
usually
pretty
heavy
stuck.
We
like
to
run
this
started
horizontally,
scarlet
databases.
So
I
cannot
answer
what
is
the
future?
I
can
answer
how
we
see
the
the
composition.
The
composition
uses
exactly
everything
that
your
omnibus
uses
today.
So
it's
from
the
installation
perspective.
What
we
want
this
to
look
like,
we
already
have
possible
server.
D
D
So
for
vanilla,
installs,
the
simple
installs
like
we
will
just
provision
two
logical
databases
on
the
same
postgres
server.
D
That
already
allows
you
to
scale
out
very
easy
to
have
significantly
more
headroom
into
the
future,
but
from
our
perspective,
it
makes
it
light
with
how
you
run
it
on
gitlab.com,
but
also
like
by
design.
It
will
be
more
performant
because
of
the
concerns
separation
and
the
complexity,
separation
and
dataset
separation.
So
that's
really
like
our
idea
behind
running
on
premise:
we
run
on
premise:
we
ask
goes
to
github.com.
D
We
don't
use
any
custom
postgres
today,
so
we
don't
really
have
this
kind
of
like
problem
of
running
the
composi,
the
compose
ci
on
on
premise,
installation,
it's
more
like
the
a
way
to
figure
out
like
how
we
want
to
migrate
existing
installations
over
some
period
of
time
to
make
it
actually
be
executed.
On
those.
D
E
Question
it
does
it.
I
I
much
like
tim
also
have
a
follow-up
though,
and
I
was
typing
it
as
you
were
finishing
up
camille.
Thank
you.
Is
there
any
advantage
to
the
two
logical
databases
beyond,
like
the
scale
issue
of
number
of
records
that
we're
seeing
in
ci,
like
would
a
self-managed
customer
have
a
reason
to
separate
those
logical
databases
into
servers?
Do
you
see
unless
they
had?
You
know
what
we
have,
which
is
hundreds
of
thousands
of
users,
and
you
know
millions
of
millions
of
job
minutes
a
day.
D
That's
that's
very
interesting
question
today:
ci
versus
the
rest
is
like
one
third
of
the
data
store
about
half
of
the
rights,
so
as
by
splitting
like
this
main
versus
ci,
we
are
kind
of
mostly
splitting
our
database
in
half.
D
So,
as
you
can
see,
as
for
gitrap.com,
we
can
take
very
long
way
till
we
need
to
run
to
different
databases.
To
be
honest,
if
our
customers
will
need
that,
I
don't
know
really.
It
really
depends
like
about
their
their
workflow.
If
they
trigger,
let's
say
10
million
beers
a
day.
D
The
answer
is
not
conclusive,
but
it's
there.
A
B
Okay,
great
thanks
brendan,
you
mentioned
slack
on
slide
10,
and
you
said
it
took
three
plus
years.
I
just
started
reading
the
blog.
Is
this
the
expectation
for
git
lab
and
have
we
learned
anything
from
what
slack
did
to
help.
A
That's
a
great
question,
so
slack
did
actually
something
that
helps
you
know,
which
is
using
a
different
database
technology
called
tess
which
uses
mysql
and
then
move
to
a
system
where
slack
is
essentially
sharding
its
database
by
by
customer.
So
it's
a
horizontal
approach,
it's
not
decomposition,
but
that
may
be
in
our
future.
So
we
are
actually
looking
at
the
tests,
for
example,
as
a
potential
future
strategy
to
get
us.
You
know
to
100x.
A
I
think
the
main
learning
here
is
that
this
is
not
an
easy
feat
and
we
have
to
be
very
mindful
on
like
going
down
down
that
path,
and
maybe
there
are
simpler
things
that
we
can
that
we
can
do
to
avoid
going
on
a
multi-year
year
journey.
So
for
like
one
thing,
for
example-
and
that's
that's
important
to
note-
is
that
vtes
exists
and
uses
mysql,
we
don't
we
use
postgres
and
v-test
does
not
support
postgres,
so
that
immediately
leads
to
questions
like
okay.
A
So,
given
that
no
such
system
like
we
test
exists,
should
we
maybe
think
about
adding
postgres
support
to
the
test?
That's
a
really
big
thing
and
those
considerations.
So
it's
mainly,
I
think,
for
me
at
least
a
reminder
that
some
of
these
things
take
a
long
time
and
it's
worth
evaluating
the
the
options
when
you
want
to
go
forward
right
now.
I
think
we
end
a
spot
where
we
have
to
make
smaller
iterations,
because
we
need
to
move
forward.
You
know
if
we
take
three
years
to
scale
our
database.
A
B
Thanks,
I
took
mark's
question
a
little
bit
more
literally
about
three
years,
so
the
for
this
project,
for
what
we
originally
focused,
this
ama
on,
was
decomposition
and
we
expect
that
to
take
six
months,
you
know
give
or
take,
and
this
especially
on
the
give
or
take
part
so
that'll
buy
us.
We
think
you
know
four
to
10x
headroom
on
the
database
and
buy
us
some
more
times
that
we
can
look
at
sharding
for
the
100x
future
scale.
As
fabian
mentioned.
D
I'm,
like
maybe
adding
another
question
like
doing
any
sharding
technology
like
we
were
trying
that
like
for
some
time,
I
think,
since,
like
two
years
or
even
more
trying
if
there
is
like
some
database,
sharding
technology
that
we
could
use
and
in
the
monolithic,
highly
tangled
set
of
the
data,
that
is
proven
to
be
super
hard
thing
really
like
to
get
right
like
with
the
sharding
and
like
the
horizontal
scaling
of
let's
say
charging
by
customers
or,
like
some
other
one.
D
Like
some
other
factors
like,
we
didn't
discover
that
like
we
would
have
to
clone
a
lot
of
the
current
database.
It's
like
with
all
the
like
figuring
how
to
look
like
all
of
these
queries
like
how
they
would
behave
across
crossing
like
different
physical
servers
and,
like
this
alone,
like
kind
of
sparked
us.
That's
like
it's
like
a
multi-year
project
to
get
it
right
and
it's
more
like
almost
all
or
nothing
to
some
extent.
So
we
were
trying
to
find
something
that
is
significantly
simpler.
D
So
this
even
like
by
thinking
that
about
the
relation
between
these
tables
is
making
this
problem
much
more
easier,
even
like
to
map
in
your
head.
So
the
composition
is
really
like
a
way
to
break
the
complexities
and
prepare
the
application
to
something
more
sophisticated
in
the
future,
and
we
are
kind
of
finding.
What
is
this
so
sophisticated?
D
D
While
they
are
doing
that,
if
you
would
have
to
do
it
like
like
starting
the
whole
gitlab.
I
I
I
I
would
not.
D
I
think
three
years
can
be
accurate
or
optimistic
if
you,
if
you
consider
us
like
as
a
whole
product
project,
like
a
moving
target,
that
you
have
constantly
teams,
adding
new
features
on
which
you
need
to
kind
of
like
like
fix.
While
you
are
doing
this
project.
So
these
three
hours
is
basically
due
to
the
complexity
of
the
product
of
they
were
trying
to
execute
and
doing
that
completely
transparently
to
the
customers,
even
though
they
were
doing
something
completely
opposite.
D
B
A
Super
I
have
a
few
finishing
words.
I
think
something
that
is
important
to
to
note
for
folks
here
in
the
room.
Is
that
scaling
our
database
is
a
great
problem
to
have.
It
is
a
consequence
of
our
growth,
so
it's
hard,
but
also
exciting,
but
it
is
also
only
a
part
of
what
we
need
to
do
as
a
as
a
company.
So
this
is
a
strategy
to
buy
us
time
and
headspace,
but
there
are
many
other
things
that
also
need
to
happen.
A
So
the
working
group
for
database
scalability
is
also
working
on
defining
blueprints
and
best
practices
for
our
developers
so
that
they
can
utilize
functionalities
that
allow
us
to
scale
better.
So
a
good
example
of
this
is
time
decay.
So
if
we
are
accumulating,
for
example,
log
data
that
we
only
need
to
keep
for
three
months,
then
we
can
drop
it
after
that
time
and
that
allows
us
to
scale
a
lot
better
than
storing
that
data
for
years
and
bloating
tables.
A
So
I
think
that's
an
important
thing
to
to
consider
that
decomposition
and
later
on.
Sharding
is,
you
know,
a
really
important
part
of
scalability,
but
there
are
many
other
things
that
we
also
need
to
do
and
those
are
parallel
tracks
and
that's
also
exciting,
because
there's
still,
I
think,
a
lot
of
room
in
the
system
for
for
those
kinds
of
improvements,
I'm
very
positive
about
that
and
then
lastly,
oh
yeah
from
camille.
He
has
a
here's.
A
comment
on
this
that
I
think
puts
this
into
perspective.
D
D
A
Thank
you.
I
think.
Lastly,
I
really
appreciate
all
of
your
time.
We
are
available
in
the
slack
channel
that
I
posted
in
here.
If
you
ever
have
any
questions
or
you
don't
get
something
or
you
are
concerned,
or
you
don't
know,
your
team
is
impacted
or
how
your
customers
are
impacted.
Please
reach
out.
We
are
available
and
happy
to
help
and
I
believe
we
are
also
pretty
much
at
time
at
least
gitlab
time
I'll
look
to
official
timekeeper
craig.
Is
that
correct?