►
From YouTube: 2020 06 22 Sharding Working Group
Description
This video culminates in the conclusion of the Sharding Working group. The timelines below are approximate:
0:00 Intro, read agenda
4:05 Gerir Feedback on sharding approach
5:30 Clarification of Sharding Terminology
7:10 Discussed time and ability to explore other scalability options
8:02 Questioned Audit Events priority
8:36 Requested Lexicon
9:12 Doing Partitioning not doing Sharding - other low hanging fruits
10:15 Garbage in garbage out
13:40 Start talking about alternate solutions
16:10 Talking about Big instances of GitLab (EU, Federal, etc)
18:04 Propose closing Sharding Group and focus on Scalability
20:00 Decision to move forward with Scaling Working Group
21:43 Charter for Scaling Working Group
A
Thanks
everybody
for
joining
the
sharding
working
group
start
running
down
the
agenda
topics
here
so
today,
immediately
after
this
meeting,
we
have
the
tenant
sharding
proposal
overview.
Looking
for
some
outcomes
here,
we've
been
talking
about
namespace
vs.
tenant
charting
for
a
couple
weeks,
so
hopefully
we'll
get
some
clarity
on
which
direction
we
should
take
and
when
you
start
getting
some
ideas
on
the
next
steps
there.
A
If
it's
not
clear
what
the
outcome
and
what
the
direction
should
be
from
that
meeting,
then
perhaps
we
can
set
a
date
for
a
decision
and
start
moving
forward
in
one
direction
or
another
or
I'm
sure
it's
entirely
possible.
A
different
direction
could
come
out
of
those
conversations
as
far
as
timelines
for
partitioning
events,
you
can
read
it
all
there
so
in
this
milestone,
we're
looking
to
finish
up
the
table
and
migration
implementation.
A
Migration
process
should
start
and
possibly
finish
during
this
milestone,
but
more
likely
we're
gonna
test
them
as
your
performance
in
production
and
we've
got
an
issue
out
there
to
start
defining.
What
UI
changes
would
need
to
take
place
to
make
sure
that
we
are
steering
users
into
the
optimal
path
for
the
time
partitioning
key,
so
any
questions
on
timeline.
A
Night
I'll
take
that
as
a
no
following
the
standard
format
here.
So
what's
been
done
last
week,
josè
an
infrastructure
team
and
alvaro
added
some
more
comments
about
concerns
about
namespace
and
porn
data
wrapper
a
pro
namespace
sharding
with
foreign
data,
wrapper
approach
again
to
reiterate:
I,
don't
think
anybody's
in
support
right
now
of
namespace
sharding
using
for
data
wrappers
and
the
reasons
are
explained
within
that
doc.
So
if
anybody
has
any
comments
or
questions,
please
feel
free
to
add
it
to
the
doc
in
related
work.
B
Four
we'll
say
that
in
in
in
general,
I'm,
not
I'm,
not
a
hundred
percent.
Sure
if
namespace
is
the
right
key
for
partitioning,
because
that's
something
I
haven't
had
time
to
look
at
myself
but
I'm
in
general,
in
favor
of
the
option
of
sharding
using
foreign,
it
wrappers,
which
is
mostly
sharding
in
general
class,
for
a
particular
subset
of
the
queries
using
foreign,
the
wrappers
as
a
pattern
for
it
I
think
it's
a
good
solution
and
may
work
here
and
that's
why
I
left
those
comments
in
the
document
there
I.
B
Yeah,
so
the
idea
is
application.
Assisters
argument,
it's
not
really
query.
Execution
is
just
query
up.
You
know
that
the
application
is
aware
of
whether
a
given
query
requires
to
touch
more
one
or
one
shard
or
more
than
one,
and
if
it's
only
one
directly
directly
to
the
short
and
you
avoid
going
through
any
four
in
the
wrapper
and
avoiding
any
performance
hit
by
that,
and
only
if
it
needs
to
start
several
shards
you
throw
it
to
the
coordinator,
which
is
a
foreigner
wrapper.
A
Was
hoping
jerry
would
actually
be
here
for
this
conversation
too,
as
you
can
see
he's
proposing,
we
adopt
some
scalability
best
practices
and
made
a
comment.
He
is
not
in
favor
of
sharding.
At
this
stage,
indicating
namespace
sharding
is
not
workable.
In
tenon
is
too
invasive.
He
did
indigo,
there's
Jerry
yeah
Jerry,
one
verbalise
sure.
C
These
are
folks
that
have
done
significant
amount
of
scalability
for
the
respective
companies
were
part
of
a
technical
group,
and
they
ask
that
I
don't
use
their
names
but
I'm
happy
to
share
who
they
were
off
the
record.
So
I
I
wasn't
trying
to
say
this
is
the
way
we're
gonna
do
Sharlene,
but
sort
of
what
their
experience
has
yielded
and
instead
of
looking
for
best
practices
to
evaluate
different
solutions.
C
A
Okay,
so
yeah
encourage
folks
to
read
it.
I
read
through
it
once
this
morning.
I
need
to
go
through
it
again
and
kind
of
absorb
it
a
little
better.
But
my
question
you
put
in
there
Gary
that
you
are
strongly
not
in
favor
of
sharding
at
the
stage
and
charting
his
bin
and
probably
will
be
an
overloaded
term
for
a
while.
So
can
you
clarify
what
you
mean
by
sharding.
C
C
C
F
C
In
is
there
data
that
you
can
take
out
of
the
data
Basin.
We
know
we
do
and
we've
been
doing
some
of
that.
The
dis
migration
is
one
good
example
of
lots
of
data
that
we're
extracting
from
the
database
and
putting
somewhere
else,
so
that
will
proudly
extend
that
runway
and
I'm
just
making
the
assumption
that
it's
probably
gonna
be
18
months,
because
there's
going
to
continue
to
refine
the
the
saturation
analysis
from
the
database
he's
not
adopting
Andrews
methodology
to
sort
of
homogenized
how
we
look
at
services.
C
So
that
means
that
we
have
something
like
six
quarters
before
you
actually
face
this
and
I
guess
my
point:
is
we
don't
have
to
rush
to
shower
right
now?
If
we
want
to
set
a
clock
of
it
would
take
us
I,
don't
know
nine
months
to
get
some
Charlene
out
the
door.
We
still
have
a
lot
of
time
to
explore
options
and
to
see
if
this
sort
of
best
practices
make
any
sense
and
see
if
they
help
us,
for
instance,
try
to
determine
whether
a
different
approach
to
sharding
makes
sense
or
not.
C
So
that's
that's
sort
of
what
I
was
going
with
that,
yes,
we
will
need
charting
or
some
form
of
scalability.
The
other
point
to
that
is
that
we're
not
gonna
do
this
and
then
stop
for
another
two
years.
Scalability
is
gonna,
become
something
that
is
just
going
to
be
with
us
forever,
because
as
time
progresses,
we'll
find
the
stresses
in
the
system
continue
to
move
and
get
laptop
conferral
in
terms
of
purposes.
Is
it's
an
entity
that's
alive
and
it's
gonna
grow
in
ways
that
we
can't
predict.
C
E
A
E
C
D
Would
also
agree
from
from
a
database
development
perspective.
I
think
there
is
a
lot
of
potential
in
improving
the
database
side
outside
of
sharding
and
I
think
we
should
sort
of
harvest
those
those
low
hanging
fruits
before
we
consider
the
more
complex
solutions
and
partitioning
that
is,
that
is
in
the
tool
belt.
I
think
this
is
a
this
is
a
common
practice
for
that
size
of
a
database,
and
there
are
other
tools
on
the
belt
as
well
that
we
should
leverage,
sometimes
as
easy
as
thinking
about
like
data
retention
strategies.
D
For
how
long
do
we
keep
data
to
to
limit
growth
and-
and
perhaps
even
you
know,
fixing
and
anti
patterns
that
we
already
know
of
where
I
think
database
sharding
would
wouldn't
even
help
addressing
those.
So
we
can't
do
that.
We
can't
continue
doing
those
anyways,
so
yeah
there's
a
little
potential
there.
C
Yeah
one
of
the
expressions
that
one
of
these
these
people
I
spoke
with
was
garbage
in
garbage
out.
So
if
you're
feeding,
you
know
a
lot
of
extra
weight
into
this,
this
sort
of
scalability
approaches-
that's
gonna,
have
a
ripple
effect
right.
So,
if
you're
trying
to
ferret
this
monster
tables
that
actually
have
a
lot
of
extra
craft
in
them,
then
we're
paying
that
price
right.
So
I
totally
agree
that
I
think
the
short
term
focus
I
mean
I
started.
Putting
an
epic
together
that
talks
about
this,
and
it's
seen
that
section
is
paying.
C
Are
they
data
technical
debt
right?
Let's
do
an
effort
to
to
really
streamline
the
database
because
that's
charlena
or
scaling
I'm
just
going
to
use
a
generic
term
scaling
using
this,
these
other
of
techniques
like
sharding
or
or
creating
components.
It
really
is
it's
it's
easier.
If
we,
if
the
data
is
lean.
D
C
B
C
B
But
if
I
would
have
unlimited
capabilities
of
what
I
would
do
in
my
case,
I
would
also
go
for
exploring
or
continue
exploring
charting
alternatives
in
parallel
to
that,
if
that's
possible
at
all,
because
if
we
have
now
a
wall
in
twelve
months,
let's
say
it's
18
and
let's
say
we
even
buy
more
time
with
this
scalable
it
be
measures
and
it's
24
months.
Any
starting
solution
or
starting
equivalent
solution
is
gonna,
take,
let's
say
12
months,
18
months
to
be
developed
and
ready
for
production
usage.
So
that's
pretty
much.
B
If
we
start
now
we'll
be
on
time,
we
would
start
later,
maybe
not
that
much
and
at
some
point,
I
believe
that
charting
is
gonna
be
required,
especially
if
there's
10x
or
superior
aims
for
scalability.
For
the
database
in
general
plus
also
the
fact
that
in
even
though
totally
scale
scaling
doesn't
totally
solve
it,
but
still
a
single
cluster
is,
logically
speaking,
a
single
point
of
failure.
If
the
cluster
is
down,
the
whole
application
is
down.
B
G
C
So
there
are
other
aspects
of
our
stack
that
will
have
the
same
effect
so
either
we
take
a
sort
of
a
what's
more
a
comprehensive
approach
to,
and
this
is
one
of
the
items
that
we
talked
about
in
the
best
practices
in
terms
of
creating
swim
lanes.
So
we
can
do
all
the
shouting
in
the
one
on
the
database
and
then
Redis
hits
a
wall
and
oh
well.
So
that's.
F
C
I
understand
I'm,
just
saying
that
we
need
to
contextualize
these
conversations.
So,
yes,
we
should
start
the
database.
Yes,
we
should
do
isolation,
but
we
should
also
you
know:
we
don't
have
infinite
amounts
of
people
and
time
and
resources
and
cache,
and
all
that
so
picking
our
targets
carefully.
I
think
it's
important
I'm
not
saying
stop
all
work
on
sharding
right
now,
I'm
just
saying
we
have
a
lot
of
time.
We
know
how
much
time
we
have.
We
now
have
data
that
backs
that
that
time
estimate.
So
we
shouldn't
rush.
C
D
Now
this
is
also
an
opportunity
to
recognize
that
the
what
we
previously
discussed
database
sharding
this
is
very
complex
compared
to
other
solutions
and
with
it
more
the
more
Headroom
that
we
have
I.
Think
it's
good
to
reevaluate
like
like
juries,
are
looking
at
but
big
picture,
and
not
only
looking
at
the
database,
because
I
think
we
can't
just
assume
that
we
got
we're
going
to
find
a
solution
in
the
database.
But
you
know
fixes
all
our
scalability
problems.
E
E
Maybe
we
need
to
focus
on
having
multiple
big
instances
of
get
labs
so
not
necessarily
sharni
get
Lancome
but
get
lab
D.
You
get
lots
en
get
like
a
federal
and
making
sure.
All
of
that
is
automated,
so
that
we
don't
need
five
new
teams
of
people
so
making
sure
everything
runs
on
kubernetes
everything
works
through
the
operator
pattern
and
making
sure
that
things
are
resilient
through
that
and
we
so
there's
there's
lots
of
things
and
we
have
a
limited
amount
of
people.
We
can
pick
only
a
few
so.
C
This
is
probably
why,
in
in
the
best
practices
for
scalability,
there
was
a
small
section
that
I
need
to
expand
on
called
swimlanes.
So
you
know
there
are
all
these
swimlanes,
where
we
need
to
sort
of
map
the
obligation
and
figure
out.
Where
is
the
right
place,
where
we're
going
to
do
some
of
things,
for
instance,
that
you're
talking
about
and
that's
actually
something
that
I
do
mention
in
there
that
maybe
customers
in
Germany
we
must
store
their
data
in
Germany.
So
it's
not
an
all
or
nothing.
C
It's
just
time
is
a
time
is
definitely
a
variable,
and
so
picking
where
we're
going
to
to
do
these
things
is
important.
So
that's
part
of
what
I'm
saying
yes,
we're
focused
on
database
sharding,
that's
what
we're
talking
about
in
this
meeting,
but
it's
really
I
think
we
need
to
do
this
sort
of
swimmingly,
an
approach
to
essentially
decide
which
aspects
are
we
going
to
separate
in
which
were
not.
A
F
A
So
I
was
actually
gonna
propose
this
next
week,
but
that's
alright
accelerated
timeline.
So
what
I
was
thinking
was
based
on
the
conversation
the
day
of
a
tenant
sharding,
but
now
that
jerry
has
brought
up
his
suggestion
and
proposal.
I,
don't
think
that
what
we're
working
on
now
really
falls
under
the
working
group
umbrella
and
please
feel
free
to
correct
me
if
I'm
wrong
with
the
wall,
the
proverbial
wall
we've
discovered
that
we're
at
least
twelve
months
out,
if
not
more
so,
the
we
have
time
to
research
the
best
path
forward.
E
C
E
G
A
E
You
already
did
thanks
for
that
yep
and
that's
as
for
the
Charter,
for
the
scaling
workgroup
I
mean
there's
two
two
things
that
are
super
important.
First
of
all,
make
sure
that
we
have
visibility
in
every
single
component
like
make
sure
that
we
have.
We
know
12
months
out
of
the
databases
come
scale.
E
Super
important
make
sure
we
increase
the
resiliency,
so
regional
failover
for
the
cloud
provider
would
be
an
example
and
I'm
in
third
kind
of
reduce
the
operational
burden
of
running
a
forge
if
we
can
make
it
more
affordable
to
run
different
incidents
of
get
labs.
That
would
really
help,
but
get
like
China
get
left.
Europe
get
like
federal
for
people's
context
like
China
is
gonna,
be
the
world's
biggest
market
soon.
E
So
we,
the
help
if
get
my
sauce
was
big,
dared,
kill,
not
Europe
you're
becoming
its
own
data
space
people,
one
hosting
guarantee
they
want
it
completely
with
on
European
soil,
including
like
the
entrance
to
home
road
to
it,
maybe
in
the
future,
even
administrators
I
think
it.
My
federal
would
be
a
giant
opportunity
to
please
to
use
it
in
the
federal
government
of
the
u.s..
These
are
big
revenue,
opportunities
and
right
now,
they're
constrained,
because
we
know
how
expensive
it
is
to
run
get
the
bomb.com.