►
From YouTube: 2020-07-30 GitLab.com k8s migration APAC
Description
Demo of the webservice pod on pre/staging. Discussion of Vault and 1.16 Kubernetes upgrade.
C
C
B
B
This
is
one,
though,
at
the
moment,
where
we're
working
around
on
the
catch-all
shards,
so
we're
going
to
separate
out
the
cues
on
catch-all
and
do
as
much
of
that
as
we
can.
So
that's
all
in
progress
at
the
moment,
so
this
will
unblock
the
next
stage
of
that
and
then
jakob's
working
on
the
removing
nfs
well
removing
enough
the
nfs
dependency
on
pages
for
us
to
progress.
There.
B
E
Yeah
so
for
logging,
we're
not
we're,
not
blocked
right.
Now,
I'm
a
little
bit
worried
when
we
start
taking
production
traffic
for
git
that
we're
just
going
to
be
flooded
with
crappy
logs
and
we
don't
have
good
filtering
mechanism,
so
I
would
say
possibly
a
blocker
I
mean,
and
it
sounds
like
that.
Well,
I
don't
know:
jason
is
speaking
for
the
distribution
team,
but
he
seems
willing
to
incorporate
this.
E
You
know
contribution
that
has
a
sidecar
that
wraps
the
logs
in
json
and
then
indicates
like
which
log
file
each
logline
comes
from.
That
will
give
us
the
flexibility
to
do
the
filtering.
We
need
to
do
so.
I
would
say
it's
still
a
blocker
for
now,
but
it's
not
like
it's
definitely
not
preventing
us
from
getting
started
with
the
get
https
and
websockets
stuff
in
production,
but
it
may
prevent
us
from
finishing
it.
E
So,
just
a
little
bit
of
background,
we
enabled
git
https
in
the
well
actually
both
websockets
and
did
https
in
the
kubernetes
cluster
for
pre-product
staging.
So
when
you
do
a
git
clone
or
any
git
operations
on
staging
using
https,
you
should
be.
Those
workloads
are
being
serviced
by
the
web
service.
Part.
A
A
E
Yeah
that
I
mean
we,
we
have
this
new
feature
called
action,
cable
which
we
discussed,
which
hasn't
been
turned
on
yet
which
uses
web
sockets.
We
have
an
existing
feature,
which
is
the
terminal,
the
the
interactive
terminal
which
uses
websockets
and
maybe
you've
seen
that
before
you
can.
You
can
pull
that
up.
If
you,
you
can
actually
pull
that
up
on
our
production
cluster.
If
you
go
to
the
cagedworkloadscape.com
project
on
ops.
A
C
E
D
E
E
C
E
C
E
I
imagine
what
we'll
do
is
we'll
start
off
with
canary
and
then
we'll
probably
just
add
the
gke
load
balancer
as
a
single
back
end
in
h.a
proxy,
so
that
it
gets
a
small
percentage
of
the
production
traffic
and
then
we'll
kind
of
observe
it
from
observe
it
from
there
and
then
we'll
shift
over
all
traffic.
Eventually,
we
can
take
a
look
at
like.
E
E
E
You
can
see
that
you
can
see
that
we're
doing
the
we're
tailing
the
logs
in
the
pod,
that's
going
to
standard
out,
so
we're
seeing
log
lines
from
multiple
log
files.
This
is
one
of
the
things
that
I
hope
this
issue
will
resolve.
Is
that
we'll
be
able
to
determine
you
know
which
log
line
comes
from
which
log
file
we're
most
interested
introduced,
interested
in
the
production,
json
log,
but
there's
still
like
unstructured
logs
in
here
we
have
the
offload
there's
just
like
a
lot
of
stuff
to
sort
out.
E
Yeah
I
mean,
I
think
I
think
this
is
the
reason
why
we
have
this
blocker.
I
think
what
we'll
do
is
once
we
so
what's
going
to
happen,
is
that
each
of
these
log
lines
will
be
wrapped
in
json.
So
we'll
be
able
to
say,
like
this
log
line
came
from
the
auth.log
and
then
we
can
either
drop
it
or
send
it
to
a
you
know
like
right,
not
drop
it,
but
not
send
it
to
elasticsearch
or
send
it
to
a
special
index.
E
C
E
Application
yeah
yeah,
so
I
think
there
is
an
issue
I'll
take
an
action
to
define
it
to
see
whether,
like,
I
think,
yeah,
I
think
I
think
possibly
doing
something
like
we
could.
We
could
add
to
structure
logging
like
a
field
that
indicates
where
the
log
comes
from.
That
would
be
the
best
option:
pops
possibly.
C
E
E
C
C
E
E
Where
is
the
here?
It
is
so
it's
looking
for
this
path
for
workhorse
logs,
so
what
I
should
be
able
to
do
is
just
see
if
that
exists.
It
does
so.
This
is
the
pod
box
that
it's
looking
at,
so
you
can
see
that
the
logs
are
here
and
then
this
should
be
forwarded
over
to
elasticsearch.
I'm
not
sure
why
I
can't
see
them
yet
in
elasticsearch
I'll.
Take
a
look
at
that
this
morning,
but
I
would
say,
like
vlogging
should
be
working.
A
E
A
You
want
to
yeah
so
give
a
five
minute.
Vault
update
the
short
answer.
Is
we
have
a
production,
a
quote?
Unquote
a
production
and
a
non-production
vault
instance
both
up
and
running
they're
vpc,
peered
they're
in
their
own
special
gitlab
approach.
Sorry,
google
projects
in
their
own
isolated
zones
they're
both
peered
with
their
respective
networks,
so
non-productions
paired
with
things
like
pre
and
staging
production,
is
only
paired
with
production.
A
I've
done
all
the
ci
jobs,
all
the
the
boiler
plate
and
all
the
bolting
together
so
that
merge
requests
to
start
basically
using
vault
could
be
added,
so
my
kind
of
in
terms
of
the
project
management
side.
The
things
I
have
left
to
do
before
my
definition
of
done
for
this
stage
of
vault
is:
I
need
to
hook
it
up
to
prometheus
and
I'm
about
90
of
the
way
there.
A
I
just
need
to
have
a
quick
chat
with
some
some
people
from
the
monitoring
team
just
to
wrap
my
head
around
how
some
of
the
monitoring
stuff
works,
so
I'll
get
prometheus
running
there
I'll
run
up
some
some.
You
know
monitoring
rules,
I
guess
making
sure
vlog
goes
down.
We
know
about
it
and
then
I
will
do
run
books
so
how
you
know
if
you
get
an
alert
about
volts
or
whatever,
how?
A
What
do
you
do
and
then
more
or
less
I'm
considering
the
first
part
of
vault
done,
and
I
will
do
a
readiness
review,
which
is
purely
just
to
sign
off
of
the
architecture
of
vault
and
the
setup
I
have,
and
you
know
how
it
works
and
the
run
books
and
the
monitoring
get
consensus
from
the
team
that
you
know
we're
happy
with
that.
There's
nothing
outstanding,
there's
nothing
else.
A
A
C
A
They're
not
used
by
chef
they're,
just
all
by
themselves
and
there's
about
10
files.
We
have
in
there
that
have
like
a
few
lines
of
code,
each
and
they're
only
used
at
ci
job
time
to
create
those
secrets.
I
think
that
was
a
prime
candidate
for
just
switching
over
to
because
from
the
helm
file
example,
you
simply
just
replace
instead
of
like
that
whole
gkms,
and
about
those
really
long
awful
lines
that
we
have
at
the
moment
to
grab
buckets
and
decrypt
it.
You
replace.
B
A
A
A
Yeah,
I
think
the
key
is
is
as
soon
as
I
kind
of
as
I
said
done
that
first
part
and
that
that's
ready
to
go
is
we
tackle
the
chef
problem?
But
if
I've
looked
at
the
work
you
did
job
with
gkms
and
the
current
setup.
I
think
we
just
create
another.
We
extend
that
ruby
code
because
you,
you
wrote
it
in
a
pretty
good
fashion,
where
it's
like
extensible
right,
so
it
can't
be
just
extend
another
class
there,
which
basically
just
calls
vault
get
key
or
whatever
it
is
instead
of
gkms,
but.
C
E
C
E
It's
just
another
back
end
for
the
gitlab
secrets,
cookbook
right
now
it
has
chef
vault
and
gk
ems.
So
it's
like
an
another
shim
that
you
could
just
add.
I
I
would
say
I
I
don't
know
if
it's
worth
spending
too
much
time
on
this,
because
when
we
move
all
of
the
front
end
over
the
only
thing,
that's
left
using
chef
secrets
will
be
postgres
and
redis
and
italy
and
yeah,
maybe
yeah.
E
Maybe
it's
worth
transitioning
that
to
vault
as
well,
but
yeah
I
mean,
maybe
maybe
it
would
be
worth
just
time
boxing
to
see
how
tricky
this
will
be.
I
I
guess
like
I.
I
also
don't
want
to
give
each
right
now
we
have
the
problem
where
every
single
node
in
an
environment
has
the
keys
to
the
kingdom
for
that
environment
and
I'd
like
to
go
about
it.
The
right
way.
E
This
time,
where
we
limit
access
to
secrets,
it's
a
bit
tricky
with
omnibus,
because
omnibus
is
like
grouped
all
together
and
there
aren't
a
lot
of
secrets
that
can
be
separated,
but
like,
for
example,
I
think
giddily
is
a
good
example.
Getaly
doesn't
need
access
to
the
postgres
database,
but
it
still
has
access
to
the
postgres
secrets,
and
this
is
something
that
I'd
like
to
try
to
segment.
If
we
can.
A
B
A
Mention
is
there:
was
a
ticket
floating
around
and
I'll
see
if
I
can
find
it
again
about
making
the
omnibus
so
the
gitlab
rb
file
have
relative
volt
integration,
so
in
the
gitlab
rb
you
would
write
like
colon
slash
and
then
you,
you
know
if
you
run
a
vault
agent
on
the
machine.
The
volt
agent
talks,
the
google
meta,
our
data
server
to
get
machine,
roll
and
gets
policy
and
everything,
and
therefore
we
don't
solve
it
in
chef.
A
D
A
D
E
A
A
C
E
Okay,
so
I
guess
the
the
point
where
we'll
probably
where
we
might
run
into
problems
is
the
point
where
we
turn
on
tls
verification
right.
I
think
up
until
that
point,
it
doesn't
matter
the
certificate's
complete
completely
wrong
and
I
think
console
like
without.
I
think
I
think,
without
tls
verification
set
to
true.
I
think
things
will
work.
So
the
tricky
part,
then,
is
like
turning
that
flag
on
on
the
master
console
yeah.
E
C
E
E
E
E
D
E
A
A
E
A
Which
will
essentially
initiate
a
failover,
so
I
will
be
able
to
at
least
determine
if
all
the
other
nodes
can
connect.
I
think
that's
a
good
sign
like
before
I
do
the
leader
as
long.
If
all
the
other
non-leaders
still
work,
then
I
think
that's
okay
and
then
I
can
just
target
and
like
roll
it
out
to
different
nodes.
C
A
C
A
E
Yeah,
what
I,
what
I'm
unsure
about
is
whether
turning
the
verify
on
and
doing
a
reload
does
that
cause
a
connection
to
drop
and
reconnect
that
I
don't
think
it
it
might
not.
So
if
it
doesn't,
then
those
old
connections
will
be
fine
and
then,
when
they
restart
when
the
connection
you
know
they'll
use
tls
but
yeah.
I
think
I
think
also
be
careful,
because
I
think
our
chef
cookbook
might
do
a
hard
restart
on
all
config
changes.
I'm
not
I'm
not
sure
if
it
does
a
reload
it
just.
E
It
does
a
reload
okay,
so
that's
good,
so
maybe
maybe
this
is
fine,
then
you
know
it
just
does
I
guess.
If
the
reload
doesn't
force
connections
to
re
reconnect,
then
it
should
be
a
simple
really
of
just
turning
on
tls
verification
everywhere
and
like
and
just
forcing
a
reconnect
to
yeah
or
just
waiting
yeah.
But
we
should,
I
guess
before
we
before
we
close
out
the
change
request.
We
should
probably
do
a
hard
restart
of
console
just
to
make
sure
that
yeah
yeah
cool.
C
A
C
C
A
Is
one
all
of
our
monitoring
broke?
Basically,
all
the
dashboards
stopped
working
and
everything
because
right
down
the
bottom
of
the
release,
notes
not
even
under
the
please
look
at
this
section.
Just
one
of
the
lines
down
the
bottom
was
they
dropped
the
label
prometheus
labels
that
we
were
using
so
container
name
becomes
container
pod
name
becomes
pod,
so
everything
we
had
just
kind
of
and
the
reason
and
the
thing
is
this
has
been
running
in
staging
for
weeks
and
the
dashboard
were
broken
for
weeks
and
staging
the.
E
Night,
no
one
noticed
that
these
are.
These
are
like
the
the
mixing
dashboards
right,
like
the
the
these
are
the
mixin
dashboards,
which
are
always
in
like
a
half
state
of
brokenness,
regardless,
like
I
think
we're
only
looking
at
them.
I
like
to
just
deprecate
those
dashboards
all
together
and
just
write
our
own
like
it's.
It's
not
very.
A
C
E
A
A
A
The
only
thing
I
could
find
on
it
is
people
complaining
to
google
about
it
and
google's
saying
that
it's
fixed
in
a
newer
version
of
six
1.16
of
kubernetes
that
came
out
three
days
ago,
so
that
pod
is
still
crash
looping.
Once
again
it
was
in
staging
and
crash
slipping
for
weeks.
It
doesn't
serve
any
real
purpose
for
us
because
we
use
a
different
login
structure,
but
I
am
going
to
look
at
like
upgrading
and
fixing
that
problem
some
point
in
the
future
and
it's.
A
E
E
So
so
for
sidekick,
it's
definitely
safe.
It's
like
I,
I
yeah,
I
think
it's.
I
think
it
should
be
safe
for
for
everything.
I'm
I'm
always
a
little
bit
more
worried
about
git
ssh
and
get
https,
because
you
have
these
very
long
lived
connections.
You
know,
like
you
know,
like
sometimes
like
these
clones,
take
a
lot
longer
than
your
typical
web
request.
A
E
B
B
A
Yeah,
so
I'm
pretty
confident
besides
the
niggly
bits
I
found
everything
else
should
be
fine,
like
we've
got
pretty
good
monitoring
coverage,
we're
getting.
No,
you
know
user
alerts
and
no
like
customer
issues.
So
I'm
pretty
happy
with
that.
Okay,
I've
already
fixed
the
monitoring,
so
the
monitoring
is
fixed.
I
said
there's
this
other
pod
problem,
but
that's
not
an
issue.
That's
major
in
any
way,
and
in
fact
it
didn't
look
like
google
drag
themselves,
drag
their
feed
on
it.
So
I'm
not
really
too
worried
about
it.
So
I'm.