►
From YouTube: GitLab 14.8 Kickoff - Enablement:Database
Description
Kickoff for the Database Group for the GitLab 14.8 release
Planning issue: https://gitlab.com/gitlab-org/database-team/team-tasks/-/issues/228
Database Group Past Kickoff Videos: https://youtube.com/playlist?list=PL05JrBw4t0KqP3MYrcoQHrqPUqn_jJZSN
Presentation by: Yannis Roussos, Sr. Product Manager, Memory and Database Groups
A
Hi,
I'm
janja
russos
the
product
manager
for
the
database
group
and
I'd
like
to
take
you
through
what
we're
planning
to
achieve
in
gitlab
4.8,
which
is
scheduled
to
be
released
in
on
february
22nd
of
2032.
we're
working
on
a
lot
of
interesting
topics.
So
let
me
get
started.
Our
top
priority
for
14.8
remains
updating,
gitlab
to
work
with
multiple
databases,
we're
supporting
the
sharing
group
in
the
composing
gitlab
database
to
improve
scalability.
A
This
is
one
of
the
most
important
availability
reliability
initiatives
in
gitlab,
as
it
allows
very
large
instances
like
gitlab.com
to
keep
on
scaling
by
by
allowing
for
multiple
primary
database
servers
to
be
used.
The
primary
database
server
is
the
bottleneck
for
any
database
architecture
as,
however,
however
many
replicas
we
may
have
all
database
rights
have
to
go
through
the
primary
database
and
there
is
always
a
limit
on
how
much
we
can
scale
a
database
server
vertically
a
limit
that
we
are
going
close
to
towards
gitlab.com.
A
A
Our
next
priority
are
the
batch
background.
Migrations.
Our
plan
is
to
make
the
bus
background
migrations
framework,
the
standard
for
all
data
related
operations
and
deprecate
the
existing
way
of
performing
them.
We
anticipate
that
this
will
be
a
major
step
forward
for
the
stability
and
availability
of
gitlab
instances
of
any
size,
while
at
the
same
time
making
most
background
data
operations
complete
considerably
faster.
In
most
cases,
that
means
that
once
we
have
this
released,
we
reach
general
availability.
A
We
expect
most
of
data
operations
to
be
more
stable,
less
problems
and
also
faster,
so,
for
example,
sometimes
before
upgrading
gitlab
gitlab
instance,
we
may
have
to
wait
for
all
background
migrations
to
finish
after
batch
background
materials
are
released.
We
expect
those
to
take
way
less
time,
so
we
have
a
lot
more
details
about
what
background
migrations
are,
what
the
current
problem
is
and
motivation,
etc.
Please
check
the
related
epic.
If
you
want
more
details,
this
will
be
our
ongoing
work
for
a
few
milestones
going
forward.
A
Our
third
top
priority
is
a
throating
at
the
throaty
mechanism
for
large
data
changes.
This
is
in
general.
It's
not
only
about
background
migrations,
as
we
discussed
before,
so
we
have
had
a
few
database
related
incidents
that
were
directly
impacted
by
the
rate
of
updates
on
specific
tables
or
in
which
the
rate
of
updates
amplified
the
problem
and
caused
more
incidents.
A
We
may
decide
to
stop
the
operation,
for
example,
because
we
say
no,
there
is
a
problem
or
we
may
decide
to
do
maintenance
operations
like,
for
example,
if
we
see
that
a
lot
of
cases
in
postgres
and
other
systems
that
use
mvcc
or
other
similar
types
of
consistency,
you
may
need
to
auto
vacuum
to
vacuum.
So
you
may
say:
okay
in
order
not
to
affect
the
the
system
anymore.
A
Let
me
vacuum
the
daily
and
then
I
will
continue
or
let
me
pause.
For
example,
an
auto
vacuum
is
running
on
a
very
large
table
and
we're
affecting
it.
So
let's
pause
and
wait.
So
it's
like
with
bad
background
migrations,
we'll
be
able
to
to
pause
and
wait
when
the
resources
are
depleted
or
the
server
is
overloaded.
Here
we
take
into
account
even
more
things.
A
So
we
have
completed
our
coverage
of
most
database
operations,
migration
spots,
migrations
and
everything.
The
only
big
thing
that
we
are
missing
are
background
migrations.
Those
are
the
big
jobs
that
run
in
the
background
and
we
and
update
lots
of
data.
This
is
a
non-trivial
extension
of
the
automated
database
testing
framework
because
background
migrations
are
meant
to
run
for
a
long
period
even
for
days
in
gitlab.com
scale
in
gitlab.com
scale.
A
So,
for
example,
there
are
cases
where
you
run
a
big
data
operation
over
a
data
update
over
a
large
table
depending
on
the
table
and
also
the
way
you
the
queries
that
you
have
there.
You
may
have
performances
at
the
start
of
the
table
or
at
the
middle
of
the
table
or
at
the
end
of
the
day.
There
is
you
cannot
know
beforehand,
so
we
have
to
be
smart
on
testing
jobs
all
over
the
range
of
ids
and
the
physical
space
of
the
table
and
the
final
to
final
top
priority
is
adding
the
database.
A
A
A
Finally,
database
schema
validation,
so
in
general
scheme
inconsistencies
are
rather
encountered
when
a
database
schema
is
different
than
what
we
expect
to
be
in
gitlab's
structure.the
square.
For
example.
A
date
field
is,
does
not
have
a
time
zone.
Why
we
expect
to
have
a
time
zone
or
there
is
a
missing
index
or
an
index
is
defined
slightly
differently
than
what
we
expected
to
be
defined.
A
Those
inconsistencies
are
the
major
reason
for
unexpected
arrows,
while
performing
updates
both
on
gitlab.com
and
then
add
on
self-managed
synthesis
directly
it
and
they
directly
impact
our
ability
to
deploy
new
updates
to
gitlab.com
or
upgrading
self
management,
and
I
know
with
the
number
of
github
instances
in
existence
out
there.
There
is
a
very
large
number,
and
especially
with
some
details
like
a
lot
of
instances
that
were
upgraded
from
mysql
to
postgres
and
may
have
some
additional
schema
issues,
some
some
of
them
not
all
of
them.
A
It
is
very
difficult
to
address
all
possible
scheme
and
consistencies,
but
we
want
to
at
least
address
all
known:
inconsistencies,
gracefully
recover
from
errors
related
to
inconsistent,
schema
errors
and,
finally,
figure
out
ways
to
prevent
similar
inconsistencies
in
the
future
and
also
make
the
whole
upgrade
upgrade.
The
experience
for
gitlab
way
smoother
like
a
fridge,
not
thinking
about
it.
That
much!
That's
it.
Thank
you
for
watching
and
talk
to
you
next
month.