►
From YouTube: 2020-10-15 GitLab.com k8s migration EMEA
Description
Discussing Git https migration
A
A
A
B
C
B
B
B
A
C
B
C
A
A
B
B
B
B
B
B
B
C
B
Have
the
min
pods
is
set
to
30,
which
is
a
change
that
we
made
this
morning
first,
I
first
thought
was
thought
we
would
do
50
to
have
like
the
full
minimum,
but
I
realize
that's
going
to
blow
out
the
number
of
notes
like
it.
It
basically
blows
out
the
number
of
notes
to
30
30
vms
across
the
three
zones,
and
since
we
aren't
yet
putting
all
the
traffic
on
the
kubernetes
cluster,
I
want
to
start
with
30
to
see
how
it
behaves
and
then
we'll
increase
it
to
50.
If
we
need
to.
B
So
the
reason
why
I'm
setting
this
floor
is
just
so
that
when
we
shift
the
traffic
initially,
we
don't
run
the
risk
of
overwhelming
it,
but
we
will
be
doing
a
gradual
transition
anyway,
so
I
would
say
once
we
do
the
full
transition.
What
we'll
probably
want
to
do
is
back
it
off
to
set
a
floor.
That's
you
know
something
that
will
allow
us
to
scale
up
and
down
with
traffic.
A
B
B
B
So
what
I'm
doing
here
is
saying:
I'm
looking
at
type
equals
git,
I'm
looking
for
only
things
that
are
coming
from
kubernetes
and
I
don't
want
to
look
at
the
canary
stage
right
now
and
I'm
just
filtering
out
readiness
checks.
So
these
are
the
logs
and
then
we
have
the
get
overview.
The
git
overview
doesn't
help
us
too
much
just
because
it's
not
possible
to
well
with
this
dashboard,
it's
not
possible
to
differentiate
kubernetes
and
non-kubernetes
traffic.
B
B
B
This
is
workhorse
percentiles,
50th
and
95th
percentile.
You
can
see
that
it
looks
like
vms
are
doing
much
worse
than
kubernetes.
So
far
like
on
the
95th
percentile
we're
seeing
all
these
like
little
spikes
that
go
up.
I
know
that
we
believe
that
the
git
fleet
is
under
provision
right
now,
and
that,
like
this,
could
be
related
to
that.
We
aren't
sure
it'll
be
interesting
to
see
what
happens
to
these
two
graphs.
B
As
we
start
moving
more
traffic
over
right
now,
we
only
have
10
percent
of
traffic
in
the
kubernetes
cluster,
so
I
wouldn't
put
too
much
stock
into
it.
Another
thing
that
sort
of
conflates
this
and
skyrack
I'm
curious.
What
your
thoughts
on
this
are
is
that,
right
now
on
the
git
vms,
we
mix
both
ssh
and
get
https
traffic
on
the
kubernetes
cluster.
We're
only
doing
git
https,
so
it
might
be.
It
might
be
interesting
for
us
to
do
a
more
apples.
B
B
Yeah,
so
I'm
thinking
about
that
right
now,
I
think
I
think
the
transition
to
kubernetes
is
for
ghbs
is
going
to
go
on
until
monday
or
tuesday,
because
I
just
want
to
be
super
slow
and
careful
with
it.
Here's
a
similar
graph
for
rails
and
again
it's
interesting.
We
see
this
same
spike
now
for
the
rails
rails
is
only
showing
get
https
because
git
ssh
well,
that's
not
true!
Actually,
yeah,
there's
only
authorization
requests
are
going
through
rails,
but
I
would
say
most
of
these
requests
are
coming
from
get
https.
B
The
blocker
I
found
was
that
we
don't
have
the
metrics
yet
and
thanos,
so
our
rps,
you
can
see
here
like
as
soon
as
I
started,
to
shift
traffic
into
the
grenades
cluster.
This
started
to
go
down
and
that's
just
because
we
have
prometheus
running
in
the
clusters,
but
thanos,
which
is
our
aggregator
for
metrics,
which
serve,
which
is
like
feeds
into
the
dashboards,
wasn't
configured
yet
for
the
zonal
cluster.
So
that's
fixed
now,
so,
hopefully
the
next
run.
We
will
see
the
rps
stay
constant.
B
This
was
a
bit
concerning
load,
balancer
component
error
rates.
I
need
to
dig
into
this
a
little
bit
more,
so
this
started
to
creep
up
as
I
started
to
move
traffic
over
and
oddly,
it
seemed
like
the
errors
were
coming,
not
from
kubernetes
but
from
the
virtual
machines.
So
I
don't
know
what
exactly
is
going
on
there,
I'm
going
to
do
a
smaller
jump
on
the
next,
because
I
already
like
when
I
already
reverted
after
this
I
started.
I
started
to
see
this
climb.
B
A
B
That
is
pretty
much
it
yeah
you
can
see
here
like
we
were
seeing
mostly
500
errors
and
they
were
actually
coming
from
the
virtual
machines.
When
I
dug
into
the
500
errors
it
wasn't,
it
wasn't
clear
yet
it
wasn't
clear
to
me
yet
what's
causing
them,
so
I
just
need
to
look
into
that
a
bit
more
and
yeah.
So
I'm
kind
of
waiting
waiting
for
this
query
to
reach
there
we
go
so
this
looks
like
it's
canary.
Let's
take
out
canaries.
B
C
Okay,
with
the
addition
of
the
zonal
clusters
and
the
addition
of
the
new
thanos
related
work,
that's
been
coming
down
from
the
observability
team.
I
have
a
worry
and
concern
that
our
existing
metrics
and
our
logging
mechanisms
are
not
going
to
keep
up
with
the
changes
that
we
have
yet
to
place
under
run
books,
and
I
have
this
fear
that
eventually
we're
going
to
have
some
form
of
outage
and
we're
not
going
to
have
a
quick
and
easy
way
to
decipher
where
the
problem
might
lie.
C
Due
to
this
fear,
I
kind
of
want
to
see
us
try
to
focus
on
tech
debt
in
the
future.
Like
I
don't
know,
what's
coming
up
next
after
we
finish
what
we
came
with
sidekick
and
after
jar
finishes
up
to
get
https
stuff,
I
don't
know
what's
really
next,
but
I
would
love
to
see
some
improvements
made
to
our
dashboards
and
I
would
love
to
see
some.
We
also
have
the
helm
upgrade,
which
has
been
stuck
forever.
C
I
would
love
to
see
us
get
out
of
the
helm,
2
version,
because
it's
kind
of
crappy
I
don't
know
how
to
steer
us
in
the
direction
to
focus
on
tech
net.
I
know
we've
got
an
epic
that
captures
this
data,
but
I'm
hoping
this
is
also.
You
know
the
opposite
of
our
okr,
but
I'm
wondering
what
do
we
need
to
do
to
try
to
figure
out?
What
can
we
do
to
ensure
confidence
in
our
runbooks
and
our
monitoring
to
ensure
that
now
that
we've
got
our
multiple
clusters?
B
Yeah,
I
think
what
would
help
for
me.
Starbuck
is,
since
you
have
a
fresher
set
of
eyes
on
this
stuff,
maybe
kind
of
gives
us
an
idea
of
what
you
want
to
see
because
yeah.
I
was
talking
to
amy
about
this
in
our
one-on-one
this
week
and
I
think,
as
soon
as
this
migration
is
complete.
We're
gonna
we're
gonna,
improve
both
the
architecture
of
review
and
the
documentation
and
the
run
books,
and
I'm
not
sure
what
so
some
some
ideas
that
I've
had
is.
B
One
is
like:
how
do
you
configure
your
workstation
to
troubleshoot
each
zonal
cluster
with
cube
ctl,
so
that
would
be
the
first
thing
right.
Next
is
the
endpoints
for
prometheus.
I
mean
those
are
kind
of
the
same.
You
just
substitute
out
the
cluster
name,
but
we
can.
We
can
write
that
down
somewhere.
C
C
Google's
routing
traffic,
for
example,
I'm
just
thinking
off
the
top
of
my
head-
we're
going
to
have
the
cluster
in
usd
1b
spin
up
a
crap
ton
of
pods
and
also
spin
up
some
notes
to
match
it
and
we're
not
going
to
understand
why
it'd
be
good
to
know
if
we
had
some
sort
of
saturation
alerting
or
just
some
sort
of
dashboard
to
look
at
to
figure
out
how
well
distributed
our
traffic
is
occurring
that
way
in
times
of
dire
need
or
outages.
We
have
that
ability
to
say.
Oh
look.
B
C
C
B
Yeah,
it
was
my
understanding
we're
going
to
just
deprecate
the
mix
in
dashboards
and
fold
what
we
want
into
the
general
dashboards.
So
that's
it's
a
little
bit
more
work,
but
first
we
need
to
figure
out
what
we
need
and
what
we
want
to
see
yeah.
So
I
don't
know
if
we
can,
like
maybe
work
with
the
absorbability
team
on
this.
C
A
few
additional
things
that
I
would
like
to
see
that
one
revert
that
you
had
yesterday
late
your
time.
I
created
an
issue
about
that,
because
I
thought
it
was
kind
of
odd
that
we
had
a
spike
in
nodes
in
one
zone
and
not
the
others
and
I'd
like
to
figure
out.
If
we
could
address
that
in
some
way
shape
or
form.
B
C
C
B
So
so
what
I
have
then,
is
like
we
take.
We
add
a
region
filter
to
the
general
dashboards.
We
take
a
look
at
the
pod
info
dashboard
and
incorporate
that
into
the
general
dashboard,
and
then
we
take
a
look
at
maybe
even
maybe
whatever
information
we
need
about
nodes.
We
incorporate
that
into
the
general
dashboard.
A
D
A
B
B
A
Yeah,
I
think
that
makes
a
lot
of
sense,
so
we
could
just
pull
in
anything
that
we
think
like
review
this
dashboard
or
write
this
document.
We've
got
a
few
things
on
the
board
already,
so
I'm
kind
of
assuming
we
will
focus
and
get
like
all
the
get
stuff
get
ssh
get
https
completed.
So
if
there's
issues
outstanding
for
those
would
be
great
to
get
them
onto
the
delivery
billboard.
A
A
B
A
A
B
B
A
But
I
do
think
a
readiness
review
would
also
still
be
useful
because
definitely
like
gets
people
to
review
it,
and
I
say,
even
though
there's
lots
of
documentation,
I
think
if
people
expect
to
be
able
to
find
a
readiness
review
and
they
live
in
a
certain
place.
I
think
it
could
be
a
nice
entry
point
into
other
documentation.
B
But
to
me,
readiness
reviews
are
like
for
the
moment
like
they're,
not
something
that
we
maintain.
We
don't
go
back
and
we
up,
we
don't
go
back
and
update
them.
I'd
rather
I'd
rather
just
do
documentation
for
the
architecture
of
the
multi-cluster
and
troubleshooting,
and
also
keep
it
up
to
date.
Okay,.
A
I
mean
yeah.
That
also
sounds
like
a
great
thing
to
do.
Do
we
do
you
think
you'll
get
the
same
level
of
review
like
because
what
I
think
the
difference
to
me
is
that
the
documentation
is
kind
of
a
telling
everyone.
Here's
how
it
is,
whereas
a
readiness
review
is
a
little
bit
more
inviting
critique
into
here's,
how
we
have
it
set
up.
C
A
B
C
I
hate
logging
right
now,
like
jarv,
showed
his
screen
for
a
split
second,
and
we
saw
just
nothing
but
new
lines
because
of
the
way
tailing
or
logs
in
the
state
of
networks,
and
I
realize
the
distribution
team
has
this
in
their
backlog
to
work
on.
Is
there
any
way
we
could
try
to
figure
out
how
to
like?
C
B
A
A
A
A
A
C
C
A
A
B
C
C
A
B
A
C
B
C
C
B
B
B
B
C
B
I
think
what
I
would
do
is
probably
delete
the
indexes
and
we
can
see
if
they
are
recreated,
but
I
think
that
they're
just
junk,
but
I
don't
know
what
created
them.
That's
really
strange
and
if
you
and
if
you
see
vlogs,
going
into
pub
sub
shell
and
gpro
now
then
like.
Do
you
see
your
easier
kubernetes
logs.
C
B
B
B
B
C
D
C
B
B
B
We
have
these
index
names
and
then
this
this
name
corresponds
to
creates
is
a
glob.
It's
a
glob,
it's
actually
star
git
lab
underscore
git
lab.
Where
course
star
we
put
the
star
before
and
after,
and
it
looks
for
slash
var,
slash,
log,
slash
containers,
and
then
that
name
and
then
it
goes
into
this
index.
B
B
C
B
C
B
Cool
marin
since
you're
joining
a
bit
late,
just
to
give
you
a
quick
update
on
good
https,
we
currently
have
around
10
of
the
traffic
going
into
the
kubernetes
cluster,
saw
that
there
was
a
problem
with
thanos
getting
the
metrics
in
for
the
zonal
clusters.
So
that's
fixed
now,
and
I'm
going
to
be
like
slowly
ramping
up
the
traffic
a
little
bit
more
today,
but
the
expectation
is
that
this
is
not
going
to
complete
tomorrow.
B
D
B
B
D
About
I
mean
I,
I
support
in
doing
this
a
bit
more
careful
because
it's
a
huge
possibility
for
for
an
outage.
What?
What
kind
of
confidence
do
you
need
to
get
in
order
to,
for
example,
jump
from
10
traffic
to
like
25
and
then
to
50
instead
of
10
15
20.
like
what?
What's
apart
from
that
spikiness
that
you're
not
sure
about
like
what
else
yeah.
B
That
I'd
like
to
and
also
just
I
mean
I
saw
that
I
think,
once
we
have
proper
metrics
going
to
thanos
properly,
then
I
was
losing
some
visibility
because
of
that
I
was
just
looking
at
logs,
so
we'll
see
I
think
my
target
is
to.
I
was
hoping
to
have
50
of
the
traffic
in
today.
It
looks
like
that
might
happen
tomorrow
and
then
we'll
we'll.
Let
that
sit
over
the
weekend
and
then
go
to
100
early
next
week.
D
C
B
I
don't
know
what
happened.
I
don't
think
this
was
ever
working
man
like
when
we
tag
when
we
tag
cng.
It
picks
up
the
latest
version
of
workhorse,
so
we're
using
the
tagged
version
of
workhorse.
That's
fine,
but
it's
actually
we're
using
the
tagged
image
the
tag
cng
image,
but
it's
actually
the
latest
version
of
workhorse.
That
was
available
at
the
time
that
that
cng
is
tagged.
D
C
D
C
D
There
is
an
issue
they
have
created
one,
but
there
is
a
desire
to
fix
it.
It's
more
about
the
order
of
it
like
do.
We
stop
the
world
now
to
fix
this
or
take
on
some
risk,
so
we
decided
to
talk
about
what
the
risk
is
and
we're
deciding
to
take
a
bit
of
risk,
but
then
the
two
corrective
actions,
as
we
roll
things
out,
are
that
and
also
the
shutting
off
traffic
between
clusters.