►
From YouTube: 2021-02-03 GitLab.com disaster recovery working group
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
All
right,
I
think
this
is
pretty
much
what
we'll
see
from
attendance,
which
is
good,
because
we
have
enough
people
around
scarborough.
Do
you
want
to
take
the
henry
henry's
item.
B
Yeah
we
did
a
test
on
the
second
yesterday.
There
is
a
slight
issue
that
flared
up
again
regarding
the
secrets
that
prevented
us
from
being
able
to
authenticate
to
the
database
upon
promotion.
B
As
henry
noted,
it
appears
to
be
a
problem
when
he
was
attempting
to
resolve
the
initial
issue.
The
second
part
of
this
is
that
the
geopostgres
service
ends
up
being
disabled.
So
when
we
come
back
around
to
demoting
the
node,
our
documentation
is
simply
incorrect
about
this,
and
he's
got
an
issue
to
address
this
other
than
that
the
promotion
went
just
as
fine
as
it
has
been
in
the
past.
So
we
still
have
a
few
things
that
we
need
to
work
on
at
this
point,
but
hopefully
nothing
major
for
a
single
node
instance.
B
Perfect,
so
this
is
mostly
a
notification
that
there's
going
to
be
a
little
bit
more
strain
on
the
staging
environment.
The
delivery
team
has
an
okay.
Are
we
going
to
be
doing
some
testing
of
the
ability
to
perform
rollbacks
and
potentially
some
automation
around
that
so
because
of
this
staging
is
going
to
be
probably
blocked
at
certain
points
of
time,
so
they
can
perform
some
testing.
B
So
if
we
need
to
perform
some
testing,
we're
probably
going
to
want
to
coordinate
with
them
at
certain
periods
of
time
just
to
make
sure
that
we're
not
testing
on
top
of
each
other.
C
And
I
think
also
before
the
ace,
the
geo
team
is
also
testing
the
maintenance
mode.
So
this
will
be
a
little
bit
more.
You
know
conflicts
here.
D
So
do
you
will
probably
not
do
another
staging
test?
I
think
we
are
probably
just
done
with
it.
B
You
have
the
discussion
point,
I'm
gonna
stop
putting
my
name
next
to
things.
Brent
has
been
trying
to
get
a
conversation
started
about
this,
so
I
created
an
issue
to
help
jump
start
the
conversation,
because
we
infrastructure
is
going
to
need
guidance
on
how
we
need
to
build
out
the
secondary
site.
Multi-Node
configuration
we
all
know
we
have
a
single
node.
We
all
know
that
that's
not
sufficient.
B
B
There
are
pluses
and
minuses
to
both
that
I
briefly
highlighted,
but
I
probably
could
have
done
a
very
job
when
I
wrote
the
issue,
and
both
of
these
will
involve
a
lot
of
infrastructure,
time
and
effort
to
deploy
no
matter
which
decision
we
go
with.
So
it
would
be
wise
if
we
could
figure
out
what
we
need
to
make
this
decision
as
soon
as
possible,
so
that
we
are
not
constantly
blocked
in
trying
to
progress
forward.
The
purpose
of
this
working
group,
I
would
appreciate
guidance,
either
from
marin
or
brent,
to
help
guide.
A
Okay,
so
first
thing
we
have
10
people
in
this
meeting.
Can
people
take
notes
a
bit
while
we
are
talking
because
it's
it's
hard
to
actually
manage
the
meeting
plastic
notes.
Bus
answer
questions,
so
I
appreciate
help
scarborough.
This
is
a
really
good
question
and
I
can
make
a
decision
on
the
spot.
That's
not
the
problem,
it's
just
that
no
one
is
going
to
like
that
decision,
so
I
think
we
need
to
talk
a
tiny
bit
more
about
it.
A
I
do
have
a
couple
of
questions
for
you
that
we
might
be
able
to
answer
here
in
this
call.
First
of
all
has
jio
used
get
at
all
so
far,.
E
F
F
F
A
That
makes
sense,
and
what
kind
of
resources
do
you
need
like?
Let's
say,
hypothetical,
that
we
asked
the
geo
team
to
actually
help
out
with
the
setup
of
get
right
like
to
build
out
a
secondary
site.
What
kind
of
access
would
you
need
like?
You
would
probably
need
access
to
the
staging
gcp
account,
I'm
guessing
to
boot
up
the
nodes,
but
what
else.
F
A
So
nick
is
this:
I
mean
the
other
nick
engineering
manager.
Gionee.
A
A
So
if
we
provide
the
service
account
and
the
the
base
things
that
you
need
to
build
out
the
secondary
site,
and
then
we
take
over
the
part
of
actually
configuring
right
like
the
primary
and
the
secondary
sites
and
like
work
with
you
to
figure
out
exactly
what
is
necessary.
That
would
help
out
a
lot
move
this
forward
and
it
wouldn't
it
wouldn't,
drag
us
in
the
direction
of
needing
to
figure
out
whether
we
can
even
use
get
within
the
environment.
We
have
would
that
be
even
an
option.
G
Yeah
I
I
yeah,
I
definitely
think
it
would.
I
I
I
guess
I
want
to
ask
nick
w
here
as
well
what
he
thinks,
since
I
think
we'd
also
be
leaning
pretty
heavily
on
on
his
knowledge
and
work
with
with
get
to
to
spin
all
that
up.
F
A
All
right,
I
think,
the
interesting
part
there
will
be
to
figure
out
also
like
whether
we
can
fit
this
in
our
infrastructure,
as
it
is
now
starbuck
right,
like
that's,
going
to
be
like
the
larger
challenge.
But
if
two
sides
are
working
on
this
right,
like
you
know,
gets
nick
and
someone
from
infra
knows
how
to
tie
this
in
then
it
becomes
significantly
significantly
easier
decision
to
make
on
just
let's
use
what
we
ship
to
people.
E
But
I
think
we
shouldn't
underestimate
that
part
as
well
like
this
is
the
like.
I
personally
from
the
product
perspective,
would
very
much
like
us
to
use
something
that
we
also
recommend
to
our
customers,
because
then,
even
if
like
and
if
we
have
a
rotation,
we
do
this
regularly.
You
know,
I
think
it's
just
you
know
so
good
on
many
different
levels
plus,
and
I
think
this
is
maybe
a
long
term
item.
E
G
And
I
think
this
also
provides
an
a
good,
a
good
opportunity
to
to
maybe
use
get
in
in
somewhat
maybe
more
of
a
real
world
situation,
because
you
know
maybe
there
will
be
a
lot
of
customers
who
already
have
a
multi-node
geo
setup
that
are
a
multi-node
gitlab
setup.
That's
not
geode,
where
they
want
to
add
geo
to
it.
So
so
we
can
maybe
make
some
improvements
to
get
to
to
accommodate
that
scenario.
A
Yeah
that
that
that
sounds
exactly
probably
like
the
most
common
case,
actually
right,
like
they
already
have
something
that
they
configured
within
whatever
they
use
scarborough.
How
do
you
feel
about
this?
B
B
Met
with
expectations
within
various
team
members,
so
that's
the
only
reason
why
I
shy
away
from
get
otherwise.
I
welcome
this
and
I
would
highly
encourage
it
because
it
would
be
interesting
to
see
what
we
could
do
here
and
maybe
transitioning
from
our
current
existing
infrastructure
methods
to
using
this
full
ansible
setup
entirely.
C
Yeah,
does
it
make
sense
to
invite
grant
young
to
be
in
this
group,
because
here
is
the
main
or
dri
of
that
tool
and
he
basically
implemented
all
of
the
gadget.
C
So
not
all
I
mean
there
are
other
qe
members,
but
grant
is
the
dri
of
this
tool
and
this
to
be
to
be
honest,
this
will
be
the
moving
forward.
We
want
to
make
this
a
standard
automation
for
deployment
for
oklahoma,
yeah.
A
Now
that
that
also
makes
sense,
I
think
nick
nick
currently
can
be
that
representative.
While
we
make
the
decision
right
like
you,
can
advise
us
nick
and
then
we
can
pull
grant
when
necessary,
but
that's
a
good
idea.
A
Scarbec
on
to
the
the
infrastructure
side,
I'm
not
calling
the
shots
here,
unfortunately,
or
fortunately,
whatever
I
wanna
say
it,
but
this
is
one
of
the
opportunities
where
infra
has
a
chance
to
start
at
ground
zero
because
get
is
not
out
yet
really
it's
beta,
which
means
if
we
get
involved
right
now,
everyone
will
learn
over
time,
and
that
would
also
mean
that
will
improve
it
for
everyone
else.
A
So
that's
going
to
be
an
easy
thing
to
solve,
I
think,
is
the
the
hard
part
is
going
to
be
like
for
me
at
least
it
seems
like
how
do
we
actually
get
something
that
is
not
that
is
made
to
manage
the
whole
life
cycle,
not
manage
the
whole
life
cycle,
because
we
have
other
things
that
manage
the
life
cycle.
G
G
B
In
this
issue,
first,
because
there's
a
lot
of
stuff
that
our
chef
infrastructure
currently
does
that
I
don't
know
if
get
handles
appropriately.
You
know
we
do
a
lot
of
configurations,
there's
our
own
tooling,
that
we
put
in
place
on
all
the
servers
and
there's
all
of
our
ssh
keys
and
etc.
Like
there's
a
lot
of
stuff
that
we
have
already
built
inside
of
chef-
and
we
would
have
to
supplement,
get
to
ensure
that
infrastructure.
A
A
We'll
continue
talking
in
the
issue,
but
I
would
really
really
like
us
to
take
that
issue
and
make
it
into
what
is
necessary
for
us
to
run,
get
and
then
get
a
gap,
analysis
and
then
ship
that
to
whoever
owns,
get
and
tell
them.
This
is
why
we
can't
use
it
and
in
the
interest
of
time
for
the
geo.
Oh
sorry,
for
the
disaster
working
group,
we
will
move
on
with
whatever
we
move
on
with,
unless
you
help
us
out
get
this
in
now
right,
that's
an
approach.
C
E
I
also,
I
think,
it's
important,
though,
to
to
separate
these
two
things,
even
though
they're
very
related
right.
I
think
the
difficulty
here
is
to
decide
on
the
tooling
right
to
actually
create
this
environment
right.
That
runs
geo
in
this
instance,
but
you
know
get
can
do
non-geo
configurations
as
well.
That's
sort
of
the
the
thing
that
we
need
to
determine
and
then
on
top
of
that
right,
this
is
where
geo
functionality
actually
kicks,
in
is
to
say,
hey.
We
have
a
3k
reference
architecture.
E
Now
we
actually
need
to
do
the
failover
right.
How
do
we
do
that
right?
And
I
think
this
is,
I
think,
for
geo
the
or
for
this
working
group,
probably
the
the
more
directly
aligned
with
what
we're
trying
to
prove
right.
But
the
other
thing
is,
I
think,
a
great
thing
to
discuss
now,
because
it
also
highlights
how
these
things
are
interconnected
right,
it's
like
for
us
for
our
customers.
Difficulty
may
not
even
be
executing
the
geo
commands.
It
may
be
setting
all
of
this
up
right.
A
Yeah
yeah
exactly
and
the
case
fabian
will
be
made
easier
than
to
say
that
the
primary
site
also
needs
to
be
rebuilt
in
get
in
order
for
us
to
get
access
to
those
commands
that
will
allow
us
to
manage
the
life
cycle
for
geo
installation
right.
So
that's
also
another
transition,
and
I
know
it's
not
strictly
related
to
jio
right
now,
but
it
it
is,
it
kind
of
is
yeah
kind
of
half
of
the
work.
Half
of
the
work
is
actually
setting
up
to
you,
configuring.
It.
A
Items
from
here-
let's
maybe
like
summarize
some
of
this
discussion
in
the
issue
and
I'll
get
an
item
I'll
get
an
item
with
brands
tomorrow
to
discuss
exactly
what
we
need
to
do
here
and
how
and
it
would
be
good
nick
and
to
involve
you
in
that
as
well.
Async,
obviously,
to
just
like
see
what
kind
of
time
commitment
we
are
talking
about
here
from
from
geosite,
and
we
can
hopefully
find
a
common
ground.
B
A
B
A
A
Maybe
we
can
start
with
item
number
two
I
see
davis
is
in
here,
so
I
just
wanted
to
kind
of
maybe
cover
some
things
if
necessary,
if
not
davis
feel
free
to
call
out
and
say
no
everything
is
written
down
or
you
can
also
tell
us
go
or
no
go
with
3k
architecture
in
staging.
That
would
be
the
most
useful
thing.
H
Yeah,
so
I
just
joined
because
me
and
andrew
were
talking
about
this,
and
I
guess
I'll
link
issue
34
in
here,
or
did
you
already
link
that
in
here?
Okay?
So
I'm
not
issue
going
over
all
the
different
I'm
trying
to
go
over
all
the
different
components
of
disaster
recovery,
because
we're
going
to
need
to
put
this
in
the
plan
for
next
year,
because
it's
definitely
going
to
be
over
and
above
our
forecast,
so
we're
basically
trying
to
figure
out
how
much
each
component
is
going
to
cost.
H
H
If
we
can't
separate
out
paid
users-
and
things
like
that
is
there
like?
Are
there
other
things?
We
can
do
to
reduce
that
cost?
So,
basically,
just
trying
to
lay
out
like
what
the
biggest
cost
services
are,
that
we'll
probably
have
to
bring
down
and
then
for
the
ones
like
web
and
api.
That
probably
aren't
as
big
of
a
deal
just
like
figuring
out.
How
much
how
many
warnings
you
actually
need
to
get
a
better
idea
of
the
cost.
H
A
A
I
guess
like
what
I
wanted
to
find
out
in
this
call,
and
we
we
are
running
at
time
almost
is:
would
a
3k
architecture
in
staging
cause,
a
problem
for
infra
right
now,
so
we
can,
like
3k
architecture,
has
a
list
of
nodes
exactly
like
what
kind
of
type
you
need
and
so
on.
So
if
we
can
get
them
blocked
on
that,
then
we
can
talk
in
parallel
about
the
larger
production
story
or.
H
A
A
H
Yeah,
if
I
remember,
I
think
that
one
should
be
fine
we'll
just
I
can
talk
to
joey
about
putting
it
in
just
from
today
on
yeah
we're
trying
to
figure
out
the
timeline
because,
for
instance,
if
we
needed
to
make
major
changes
in
gili,
I
guess
my
question
was
like
when
it
doesn't
even
make
sense
to
start
testing
now
we're
not
gonna,
be
able
to
roll
anything
out
because
it'll
get
blocked
by
other
things
in
production.
That's
a
good
point.
It's
not
as.
E
Much
I'm
strongly
in
favor
of
having
all
of
these
operational
and
testing
things
figured
out
well
ahead
of
time,
even
if
the
product
development,
if
that
is
the
case,
lags
behind.
I
think
I
think
we
learned
so
much
about
many
other
things
that
I
think
it
is
well
worth
it.
But
that's
my
my
two
sense.
It
is
very
likely,
though,
and
I
need
to
talk
to
mark
wood,
the
pm
for
gitly-
that
we
need
significant
feature
development
to
reduce
the
cost
of
italy.
E
I
think
overall,
which
then
will
enable
the
dr
site
as
well,
and
if
that
is
not
possible,
we
may
want
to
resort
to
selective
thinking
and
some
configuration
bits,
but
I'm
cautioning
here
that
I
don't
know
yet
what
the
best
solution
for
this
is.
I
know
what
the
problem
is,
and
I
think
that's
well
understood,
but
I
will
need
to
do
more
more
research,
also
in
collaboration
with
those
teams,
because
that
you
know
that
may
be
something
that,
for
example,
geo
shouldn't
actually
handle
right.
It's
better
kept
in
italy
or
we
don't
know
yet.
E
So
I
think,
but
the
question
at
hand
for
me
would
be.
Can
we
can
we
deploy
a
3k
reference
architecture
in
staging,
which
is,
I
think,
two
thousand
dollars,
or
something
like
that
a
month
for
the
foreseeable
future,
for
our
testing
endeavors.
H
Yeah,
I
think
that
should
be
fine.
I
don't
think
we'll
get
any
questions
asked
about
that.
So
I
can
just
talk
to
joey
about
that
for
now
and
then
once
we
know
when
we're
gonna
deplete
your
production,
you
can
also
put
that
in
but
yeah.
That
was
my
only
remaining
question.
That
was
if
it
still
makes
sense
to
do
staging
if
we
expect
like
delays
and
actually
pushing
to
production
at
some
point.
E
I
think
the
other
thing
that
I
and
maureen
can
correct
me
or
scarbec.
If
this
is
wrong,
is
I
thought
for
a
long
time
that
staging
is
not
important?
It's
just
our
staging
environment
right,
and
so
why
would
we
care
and
have
vr
right,
but
I
think
the
the
opposite
is
true.
If
staging
becomes
unavailable,
we
can't
go
to
production.
It
blocks
the
entire
company,
so
having
the
r
capabilities
for
staging
separately
is
actually
also
very
valuable
right.
So
it's
not
that
the
only
value
is
realized
by
having
geo
in
production.
D
A
Like
it's
as
simple
as
that,
whether
they
know
it
or
not,
right
everyone
stops
if
staging
is
not
working.
A
Cool,
would
you
mind
writing
that
in
the
issue
and
when
I
say
the
issue
I
mean
in
2b,
so
in
issue
m
stuff
34
just
like
there
is
a
discussion
about
staging
in
our
architecture.
If
you
give
us
the
go
ahead,
if
you
give
us
go
ahead,
we
can
we
can
move
on
that.
A
All
right
well
I'll,
take
that
as
a
no.
We
have
some
action
items
that
will
take
and
please
continue
doing
the
failovers
in
staging
just
for
your
own
practice.
This
is
more
for
henry
who
might
watch
the
recording
as
well
and
we'll
see
each
other
next
time
have
a
good
one.