►
From YouTube: 2021-01-28 GitLab.com k8s migration EMEA
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
C
C
Cool,
so
welcome
to
today's
demo,
who
has
something
they
would
like
to.
D
Demo,
I
do,
are
we
jumping
right
into
it
or
is
there
anything
else
we're
going
to
discuss.
D
D
I'm
basically
going
to
repeat
what
I
did
this
morning,
though
I
have
a
little
bit
more
information
and
I
also
emailed-
or
you
probably
saw
that
I
emailed
google
about
this
problem
as
well,
so
this
is
for
a
little
bit
of
background.
This
there's
an
issue
currently
with
our
websockets
deployment.
D
We've
we
removed
the
nginx
ingress,
so
we're
connecting
right
to
the
service
endpoint
and
we're
seeing
problems
when
pods
are
cycled
we're
seeing
errors,
and
this
is
very
reminiscent
of
an
earlier
problem-
and
this
is
something
that
jason
was
also
helping
us
out
with
where
we
saw
an
issue
with
the
nginx
ingress,
where
connections
were
being
held
open
and
we
were
seeing
errors
as
bike
as
pods
were
cycled
and
we
tweaked
some
parameters
in
the
nginx
ingress
and
that
problem
went
away.
D
We
then,
after
that,
we
were
seeing
problems
with
when
engine
x,
controller
pods
were
being
cycled.
So
what
we
did
is
we
threw
a
lot
of
resources
at
the
nginx
ingress
so
that
the
pods
never
get
cycled.
D
What
we've
done
since
then,
is
that,
instead
of
running
a
gcp
internal
lb
in
front
of
engine
nginx,
now
we've
taken
out
nginx
and
we're
running
a
gcp
internal
lb
right
in
front
of
our
nodes
nodes
that
are
servicing
websockets
so
in
front
of
our
rails
nodes.
What
I'm
going
to
show
is
what
happens
when
a
pod
is
removed.
A
E
D
D
G
D
God
there
we
go
okay,
we
are
hitting
this
endpoint
here.
This
is
the
service
endpoint
for
the
gitlab
web
service
websockets
service.
This
is
a
google
load,
bouncer
internal
load
balancer
and
I'm
hitting
just
like
a
readme
in
a
project.
This
is
on
pre-prod
in
the
lower
right
hand
corner
you
can
see
like
the
requests
coming
in
and
in
the
upper
right
hand
corner.
You
can
see.
I
have
k9s
running
so
the
pod
is
xghtk
and
you
can
see
in
the
lower
right
hand
corner
these.
D
I
can
get
the
endpoints,
which
shows
that,
for
this
node
we
have
xghtk
is
the
endpoint
for
web
server's
websocket.
So
everything
looks
good.
We're
able
to
talk
to
the
websocket
service.
This
isn't
a
websockets
connection,
but
it
doesn't
matter
because
you
can.
I
can
demonstrate
this
by
doing
just
regular
http
requests
I'm
going
to
edit
the
service
I
think
hold.
D
G
G
E
You
check
your
zoom
version,
I'm
just
curious
sure
it
is
five
three
one.
Oh.
E
D
Yeah,
this
really
sucks,
because
I
I
don't
know
I'll,
try
to
leave
and
come
back.
C
C
D
For
now,
right,
try
not
to
touch
too
many
things
here.
So
I'm
going
to
make
a
change.
That's
going
to
result
in
like
a
new
pod
coming
up.
Can
you
see
that
a
new
pod
is
being
brought
up?
Oh
okay!
Maybe
if
I
don't
touch
anything
it'll
it'll
keep
working,
so
this
takes
a
little
bit
of
time,
but
what
we
hope
to
see
is
that
traffic
will
shift
from
this
xghtk
pod
to
the
nu1
ztw
pod
and
we
won't
have
any
500s,
but
that's
not
going
to.
E
E
And
what
boot
up
time
are
we
talking
about
these
days
about
a
minute
or
two
okay,.
D
So
please
like,
if
you
can
still.
E
Two
seems
really
long
why
I
think
it
stopped
jarv
again,
I
don't
see
anything
moving.
Wow.
D
D
Basically,
if
you
were
to
look
at
if
you,
if
you
were
to
do
cube,
ctl
get
endpoints,
you
would
see
that
both
endpoints
are
listed.
This
new
one
isn't
healthy
yet,
but
it
will
be
soon
as
soon
as
it
comes
up.
D
And
hopefully
you
still
see
we're
servicing
requests.
You
see
some
of
like
the
new
pod
blogs.
Are,
you
know,
we're
starting
to
see
some
of
some
activity
from
it?
So
first
thing
to
notice
here,
hopefully
still
moving
xg
htk
isn't
terminating,
but
look
at
that.
It's
still
receiving
traffic.
That's
strange.
D
F
D
But
that's
that
doesn't
come
into
play
here
because,
as
soon
as
the
termination
starts
right,
the
q
proxy,
it
should
stop
traffic
that
right
now
we
have
zero
out
of
two
pods
zero
out
of
two
containers
ready
and
we're
still
receiving
traffic.
Now
we're
still
processing
requests
because
we're
in
this
period
of
time,
where
puma
this
is
the
blackout
window
where.
D
F
So,
let's
understand
what
takes
a
pod
out
of
a
service
because
there's
there
seems
to
be
this
view
that
if
it's
terminating
it
should
just
be
removed.
D
D
H
D
F
D
D
D
Yeah,
so
the
google
load
balancer
has
attached
to
it.
Let
me
I'm
going
to
stop
my
screen
share
again
I'll,
be
right.
D
D
So
the
gcp
tcp
load
balancer
has
the
nodes
attached
to
it,
so
those
nodes
are
always
healthy
and
the
requests
are
going
to
one
of
the
nodes
which
then
you
know
then
you're
dealing
with
like
the
the
q
you're
doing
the
q
proxy
right
and
then
that
gets
directed
to
whatever
pod
is
healthy
right
now,
if
you
see
my
screen
like
we
have
this
pod,
that's
still
in
the
terminated
state
and
still
receiving
traffic,
and
this
has
been
like
over
a
minute
now.
D
So
I
I
don't
know,
I
sent
an
email
to
google
and
I'll
be
interested
to
see
what
they
come
back
with
about
this.
D
I
don't
know,
I
I
guess
it's
possible
that
we
have
connection
reuse,
but
for
over
two
minutes
like
does.
That
is
that,
is
that
really
a
possibility.
F
F
D
F
Have
to
figure
out
what's
going
on
that
that
is
being
held
open
and
effectively.
You
have
a
sticky
connection
through
the
ip
stack
from
node.
A
to
pod
z,
because
that's
what's
happening
is
that
traffic
is
still
being
routed
there
and
there's
not
something
that
says
get
off
of
me
go
away
and
it's
just
keep
getting
routed
there.
D
D
Yeah
I
mean
I
can
do
that
now.
I
I
just
like.
I
just
hate
the
screen
sharing
stuff
because
now
like
as
soon
as
I
click
anywhere,
I'm
probably
lost.
My
screen
share.
F
D
E
F
H
F
F
C
Do
you
need
any
help
on
the
dependency
external
dependency
issue.
B
Java-
and
I
had
a
conversation
about
that
yesterday-
we're
going
to
try
to
iterate
on
the
solution,
so
I'm
going
to
try
to.
I
didn't
get
a
chance
to
do
this
yesterday,
due
to
other
things,
but
I'm
going
to
try
to
work
on
a
first
iteration
of
that
today
and
then
I
may
wait
until
an
improvement
comes
into
a
git
lab
to
make
it
more
better
in
the
future,
but
we'll
see
how
that
goes
over
the
course
of
time.
B
Hopefully,
I
should
be
able
to
complete
that
within
a
day,
just
a
matter
of
getting
that
reviewed
if
all
goes
well
sounds
good.
If
anything,
I
could
use
help
on
jason
you've
seen
me
putting
in
thousands
of
merge
requests
into
the
home
chart.
They
don't
have
a
priority
label.
Some
of
them
are
labeled
as
production
requests,
because
we
use
those
home
charts
it'd
be
great
if
we
could
just
get
those
reviewed
so
that
we
could
at
some
point
when
we
get
a
chance
upgrade
our
own
helm
chart
that
we're
consuming
that
way.
B
At
this
point,
I
think
I've
got
all
the
charts
in
that
we
control.
I
think
that
one
question
remains
is
whether
or
not
we
want
to
do
the
nginx
chart,
because
I
think
that
requires
a
lot
of
touching,
because
we
forked
it.
B
The
others
that
we
inherit
like
postgres
and
radius
and
grafana,
I
think,
based
on
the
conversation
that
you
and
I
had
very
quickly
yesterday,
that
will
probably
not
touch
those
which
I
think
is
fine.
I
need
to
note
that
on
the
issue
just
so
that
we
have
that
documented
though-
and
I
have
not
done
that
so
I'll-
make
sure
I
do
that
after
this
call.
F
Great
the
the
items
that
we
have
not
forked,
therefore
do
not
control
the
source
of
we
can
propose,
mrs
upstream,
but
they're.
Adding
labels
is
unfortunately
not
sufficient
to
fork.
So
sorry,
andrew,
you,
don't
use
those
pods
in
production
anyways.
So
at
least
that
kind
of
helps
we
can.
We
can
get
started
on
that
road,
but
not
just
yet
as
to
the
the
large
number
of
mrs.
Yes,
we
have
seen
them.
We
know
that
there
we're
trying
to
get
through
all
of
the
reviews.
F
D
Okay,
so
I
think
we've
confirmed
then
at
least
like
from
what
I
see
doing
a
curl
in
a
loop
looks
like
I'm,
not
sure
if
you
can
see
my
screen
or
not,
but
it
looks
like
that
does
flip
over
immediately.
So
it
sounds
like
to
me,
then
the
errors
that
were
seen
on
websockets
probably
are
like
held
open
connections,
and
we
just
don't
handle
that
gracefully.
B
One
is
there
something
that
we
could
leverage
inside
of
say,
aj
proxy,
that
could
try
to
recognize
that
a
pod
is
it's
not
there's
no
way
to
do
that
that
she
probably
has
no
information
about
pogba.
G
I
I
just
just
on
that
point.
I
mean
having
spent
like
eight
years
working
on
web
sockets
and
but
I
don't
know,
action
cable,
but
I
very
much
doubt
that
they
haven't
built
action.
Cable
with
that
in
mind,
like
you,
get
a
signal
in
javascript
that
the
connection's
gone
and
if
your
proxy
and
your
intermediate
proxies
are
all
set
up.
G
Fine,
the
browser
will
get
an
event
that
says
the
connections
down
and
like
every
implementation
that
I've
seen
of
this
will
immediately
try
to
reconnect,
and
so
there's
no
need
to
signal
to
the
client
to
to
to
start
trying
again.
You
know
in
the
time
frame
that
you
do
that
it
might
actually
connect
back
to
the
same
one.
So
the
simplest
thing
to
do
is
just
to
retry.
F
Yeah
you're
a
100
are
correct
about
the
client
side.
My
concern
isn't
the
client
side.
My
concern
is
that
the
server
side
needs
to
say
go
away
while
we're
in
the
the
grace
period
of
termination,
while
we're
in
the
blackout
seconds
for
the
rails
application,
we
need
to
be
telling
active
connections,
stop
it
go
away
so
that
they
know
to
reconnect
once
the
stop
action
has
occurred
and
we're
into
that
window
after
sig
term
has
been
issued.
That's
when
we
should
start
saying
I'm
going
to
finish
your
requests,
but
you
need
to
go
away.
F
That's
what
the
server
needs
to
say
that,
because,
while
the
client
is
still
connected
and
neither
party
has
dropped,
the
tcp
socket
it's
being
held
open
by
the
kernel
on
the
endpoint,
we
actually
have
to
cause
that
to
terminate
if
we
do
it
with
with
some
configuration
of
coupe
proxy
we're
affecting
everything.
We
do
not
want
to
do
this.
F
F
G
G
You
know,
because,
even
if
you
told
the
client
to
disconnect
you
know,
a
good
percentage
of
your
clients
are
going
to
be
malfunctioning
or
going
through
a
tunnel
at
that
exact
moment
and
they're
not
going
to
do
what
you
tell
them
to
do,
and
you
shouldn't
trust
them
to
do
that
anyway.
So
the
best
thing
to
do
is
to
you
know,
you
don't
need
a
drain,
you
know,
and
especially
with
action
cable,
because
action
cable
is
broadcast,
it
doesn't
have
like
individual
cues
for
individual
clients.
D
About
it,
andrew
there's,
there's
another
element
here
that
you
may
not
have
had
in
other
implementations,
which
is
having
workhorse
in
between
where
these
connections
are
proxy
through
workhorse.
So
is
that
why
we're
seeing
a
bit
of
ugliness
in
our
error
rates?
Basically,
like
you
know,
rails
rails
dies
before
workhorse
and
then
workhorse
gets
like
a
503
service
unavailable
returns
a
503
service
unavailable
because,
like
rails
just
goes
away.
G
Yeah
I
mean
we
that's
a
very
good
thing
to.
That
would
be
a
very
good
thing
to
test
like
we
should
have
some
sort
of
integration
test
that
that
makes
sure
that
workhorses
is
handling
that
correctly,
but
you
know
certainly
nginx
and
other
you
know
front
side
proxies,
have
or
reverse
proxies
at
least
manage
that
perfectly
fine.
So
if
they
can
do
it,
then
workers
can
do
it
too.
D
G
I
mean
what
you
could
do
is
with
with
like
on
gdk
or
something
like
that
or
just
locally
running
processes.
Just
you
know,
set
up
a
open,
a
websocket
connection
just
for
the
clients
to
to
to
the
back
end
and
then
to
action,
cable
and
then
do
a
sick
term.
On
on
on
puma
and
and
see
I
mean
I
would
imagine
that
workhorse
would
just
drop
it
like
that's.
You
know,
there's
not
that
many
things
that
it
can
do.
F
F
That's
not
the
case.
There
is
one
two
or
five
at
most.
We
need
to
be
aware
of
that
fact.
So,
while
the
answer
can
be
just
terminate
the
thing
anybody
who's
not
operating
in
that
individualized
process,
behavior
is
going
to
be
heavily
affected
by
that
choice.
So
when
I
say
we
need
to
tell
the
application
I'm
saying
during
the
the
blackout
window,
we
need
to
identify
that
this
is
an
active
connection
that
is,
from
action,
cable,
shut,
it
off
and
then
finish
answering
anything
else.
G
Yeah,
I
mean
that's,
that's
that's
a
really
good
point.
I
haven't
considered
that
from
a
charts
point
of
view.
Do
you
think
there's
any
appetite
for
making
that
by
default
a
separate
workload,
because
there
are
other
reasons
why
like
not
having
your
websockets
coming
into
your
main
web
application,
is
a
very
good
idea
or
do
you
think
that's
people
aren't
gonna
it's
just
too
much
extra
complexity.
F
I
it's
it's,
it's
not
that
we
can't
separate
it.
It's
a
matter
of
resource
versus
scale
right
at
our
scale.
It
makes
hundreds
of
percentage
to
do
it
the
right
way
in
the
way
we're
doing
yeah
in
somebody
who's
a
thousand
or
even
two
it
it's
like
we're
generating.
What's
possibly
an
extra
two
to
you
know
one
two
cpu
and
a
couple
of
gigs
of
ram
right.
F
F
Thinking
in
the
cloud
native
sense,
if
we
go
to
the
omnibus
side,
I
know
of
thousand
user
systems
that
are
just
really
big
vms
and
we
don't
spawn
action,
cable
for
this
and
rails
for
this
and
rails
for
this
right.
We
don't
have
the
rails
applications
segregated
into
individual
api
routes.
We
just
don't
it's
not
that's
not
impossible.
It's
not
something.
We
can
look
into
it's
that
it's
not
reasonable
for
smaller
instances,
and
we
have
to
think
about
not
only
us
at
millions
and
millions
of
users,
but
at
the
people
with
15
users.
G
D
Service
what
I
mean,
what
do
we
do
in
the
meantime,
I
mean
we
need
to
investigate
this,
but
what
do
we
do
in
the
meantime
about
our
slis
for
error
rates
on
the
websocket
service?
You
know
like
it's.
We.
H
D
D
That's
fine,
but
that
doesn't,
but
we're
still
going
to
see
errors
when
the
issue
is,
is
that
we
see
these
errors,
but
we
just
get
reconnects
and
everything
as
far
as
we
understand
is
hunky-dory
on
the
client
side,.
F
G
Job
sorry,
I
I
was
concentrating
on
several
things
when
you
were
talking
earlier,
but
like
this
over
here
can
is
my
screen.
Sharing
now
yeah
yours
is
working,
I'm
jealous
yeah.
It
won't
last
for
long
so
like
this
over
here
isn't
related
to
this
is
web
sockets
right
this
over
here
this
load
balances
spike.
It
doesn't
seem
to
be
related
to
deployment.
Is
it
at
1400,
utc.
G
I
thought
a
lot
of
them
were
that's
that
other
problem
with
the
client
header
that
was
being
sent
through
by
like
one
ip
address
where
he
was
sending
a
lot.
Did
you
are
you
aware
of
that
incident?
Yeah.
D
Problem
anymore,
we
can
look
at
century
to
see
if,
if
these
are
500
errors,
but
if
you
look
at
it,
the
workhorse
error
rate
is
much
lower
right,
so
it
doesn't
look
like
it
looks
like
this
is
like
the
load.
Balancer
complaining,
not
actual
errors,
yeah.
D
D
G
Just
just
while
I'm
here
before
I
go
away,
this
is
why
I
like
to
have
web
sockets
separated
from
the
this
is
a
classic
example,
and
it's
just
right
in
front
of
us.
This
is
why
I
like
to
have
websockets
separated
right,
so
here's
our
base
load
at
like
20
requests
per
second
and
then
for
some
reason,
it's
just
spiking
up,
and
you
know
we
hardly
been
using
this
stuff.
Yet
so
wait
until
you
know
we're
really
using
this,
and
this
is
spiking.
You
know
five
times
the
load.
D
Yeah,
I
know
I'm
very
happy.
We
decided
to
create
a
new
service
for
this
and
also
to
have
its
own
node
pool.
I
don't
know,
I
see
a
deployment
that
happened
about
10
20
minutes
ago,
so
maybe
maybe
the
spike
is
due
to
that
I'll
dig
in
to
it
a
bit
more.
F
G
No,
but
they
don't
get
jason,
they
don't
get
500s,
though,
because
the
websockets
it
connects
it
does
the
the
upgrade,
and
that's
the
moment
that
you
get
a
200
back
when
that
gets
disconnected
it's
not
a
status
500,
it's
it's!
It's
just
a
socket
connect.
It
disconnected.
The
status
code
is
200
on
those.
So
that's
just
something
that's
worth
being
clear
about
and
clarifying
when
you
get
disconnected
without
expectation
on
a
websocket,
it's
still
just
a
it's
not
treated
as
an
error
by
a
cha
proxy
or
by
nginx
or
anything
else.
B
F
G
For
a
reason
yeah
we
we
did,
we
did
exponential
back
off
on
those
or
randomized
retry,
just
to
kind
of
spread
it
a
little
bit,
but
I'm
sure
actually
action.
Cable
is
a
pretty
widely
used
thing.
I
mean
I
would
be
surprised
if
it
didn't
have
some
of
those
things
in.
G
D
F
G
E
D
I
think
jason.
I
think
I
like
your
suggestion,
the
most,
which
is
to
reduce
this
grace
period.
It
feels
like
it's
a
safe
thing
to
do
for
websockets
anyway,
and
we
can.
We
can
try
decreasing
it
see
if
that
helps
with
the
aerospike.
G
Just
one
more
thing
on
that
on
that
thought:
I
saw
that
heinrich,
or
somebody
mentioned
that
you
can
send
commands
through
the
websockets.
G
That's
something
an
action
action
cable
can
do
and
that-
and
I
think
they're
not
doing
it
yet,
and
I
think
it's
a
very
bad
idea
to
do
that,
and
one
of
the
reasons
would
be
you
know
if
somebody
sends
a
command
and
they're
busy,
creating
a
merge
request
and
they've
issued
that
through
the
websocket
first
of
all,
we
don't
get
any
logs,
our
metrics
don't
work
anymore,
but
then
the
third
reason
is:
if
we
it,
we
can
no
longer
do
the
the
the
short
terminate,
because
you
don't
know
if
that,
if
that
request
is
still
ongoing.
G
B
F
I'd
agree
with
that
and
that's
the
danger.
Websockets
are
really
cool
from
a
technical
standpoint,
but
if
they're
used
in
the
wrong
way,
you
lack
visibility,
you
interrupt
controls
and
the
design
has
to
be
done
right
and
I
can
tell
you
doing
that
right
is
not
easy.
Having
done
web
sockets
in
the
past
myself,
it's
tcp.
G
On
top
of
http,
and
you
got
tcp
and
http
again
and
then
tcp
on
top,
but
also
any
any
reason
that
they
say:
oh
it'll
be
faster
or
anything
like
that.
That's
totally
now
no
longer
true
because
of
streaming
like
http,
2
and
http
3
anyway,
so
you
know
you're
no
longer
setting
up
a
connection
for
every
request
or
anything
like
that.
So
if
they
say
oh
but
it'll,
be
so
much
faster,
you
can
say
well,
it's
so
much
faster
with
http
203
anyway.
C
Cool
okay,
great,
so
just
the
only
other
bit
I
wanted
to
run
through,
just
especially
whilst
we
have
you
here.
Jason
is
just
quick
look
at
our
blockers.
C
Hopefully,
hopefully
you
can
see
the
blockers,
so
I
saw
jason.
You
got
pinged
on
this,
removing
the
duplicate
messages.
Do
you
know
what
might
be
involved
or
like
when
that
might
likely
the
chance
change
that
scarborough
mentioned
when
we
might
likely
be
able
to
get
something
either.
F
I
don't
know
about
this:
charts
change
and
I'll
be
honest.
I
haven't
had
a
chance
to
go.
Look
at
this
before
just
a
few
minutes
ago.
My
days
for
the
last
two
have
been
very
busy
with
interviews,
so
I
had
a
lot
of
spare
time.
I
can
answer
that
question
in
about
an
hour.
Okay,.
C
F
D
I
just
wanted
to
say
thank
you
to
jason
for
joining,
really
appreciate
it.
It
was
very
nice
to
have
you
here.
Yeah,
the
regional
dashboards
are
really.
B
D
Yeah
and
something
I
I
mentioned
to
andrew
and
might
be
interesting
for
scarbeck
as
well-
is
that
when
we
start
moving
the
api,
we
can
overload
this
region
label
to
compare
the
vm
slis
versus
the
kubernetes
sli,
because
for
vms
we
set
the
region
label
to
us
east.
It's
just
like
set
the
same
everywhere
for
the
kubernetes.
We
have
it
set
to
usc,
1,
bc
and
d.
D
G
But
certainly
here
like
exactly
what
you
said,
because
this
sli
comes
from
the
aj
proxy,
which
is
on
vms,
and
so
it's
us
east
and
then
the
three
other
ones
are
usc
1
a
b.
What's
its
bcd,
the
one
thing
that's
really
weird,
at
least
by
I
was
looking
at
this,
and
it
didn't
seem
right.
G
I
don't
know,
is
the
three
are
identical
job
like
like
really
like,
there's
tiny
differences
but
like
I
was
expecting
more
variety
between
the
three
clusters,
but
they
are,
they
have
the
same
dips
and
the
same
spikes,
it's
very
suspicious
to
me.
That
does
seem
suspicious
yeah.
I
mean
there
are
differences,
they're,
not
identical,
but.
I
D
G
C
D
Yeah,
I
don't
know
if
you
read
github's
latest
blog
post
on
how
they
do
deploys,
but
this
is
sort
of
like
where
I
think
we're
gonna
eventually
go
where
they
said,
like
they
started
out
with
the
canary
stage
at
two
percent
and
they
realized
okay,
that
isn't
enough.
They
needed
like
ten
percent,
and
I
think
for
us,
it's
going
to
be
33.
D
Where
we'll
have
one
cluster
go
first
and
then
we
can
measure
you
know
using
the
region
label,
we
can
kind
of
see
what
the
sli's
are
for
that
cluster
and
then
move
forward,
and
that
would
be.
That
would
be
great.
E
D
Been
a
long
day
it
has
been
a
long
day,
yeah,
that's
true,
but
we'll
be
able
to
move
forward.
So
much
quicker,
we'll
just
say:
forget
it
we'll
just
move
forward
and
everything
will
be
fine.
E
H
A
D
It
in
delivery
I'll
check
it
out,
but
I
was
supposed
yeah.