►
From YouTube: Operating Enterprise Grade Kubernetes Clusters at Salesforce on Bare... Anubhav Dhoot & Mayank Kumar
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Operating Enterprise Grade Kubernetes Clusters at Salesforce on Bare Metal - Anubhav Dhoot & Mayank Kumar, Salesforce
Many enterprises today don’t have the luxury of public cloud managed offerings, or VM’s in their private data centers. In the absence of these, it’s challenging to operate enterprise grade Kubernetes clusters and manage it seamlessly. We will talk about specific challenges in bringing up a highly available kubernetes cluster on bare metal, securing it, upgrading it, monitoring it 24/7, and doing all of that using Puppet, while maintaining SLA for our customers. We will cover a lot of lessons learnt in managing a fleet of 2600+ hosts across 20 + data centers over the last 4 years, including battle stories like dangers of etcd initial-cluster-state config, hairpin networking issues, and setting up service accounts with key rotation. While we are well on the path to using managed K8s offerings in Public Cloud, this will be a cautionary tale for teams managing their own k8s clusters.
https://sched.co/bGGS
A
All
right
welcome
to
the
talk
on
operating
enterprise,
great
kubernetes
clusters
at
salesforce
on
bare
metal,
I'm
anubhav,
dhoot
senior,
director
of
engineering.
I
manage
a
team
that
provides
the
kubernetes
based
platform
for
salesforce
teams,
am
joined
in
this
talk
with
mayankumar,
who
is
the
architect
on
the
same
team.
A
A
This
is
really
doing
kubernetes
the
hard
way
as
compared
to
the
managed
kubernetes
engines
out
there
we'll
do
a
deep
dive
into
our
architecture
and
how
we
operate
our
stack
over
the
last
four
and
a
half
years.
We
have
developed
a
bunch
of
war
stories
of
issues
and
pitfalls
that
we
face.
We
would
like
to
share
with
you.
Hopefully
you
can
learn
from
them
as
well.
A
A
A
So
what
we
provide
to
our
customers
is
a
github
style
deployment
pipeline.
They
check
in
their
code
into
git,
along
with
that,
they
also
check
in
a
simplified
manifest
that
describes
their
application.
The
ci
pipeline
runs.
It
validates
their
application,
manifest
packages
it
up
and
deploys
it
to
all
clusters
as
a
goal
state.
A
So
when
the
goal
state
lands
up
in
every
cluster,
we
have
an
operator
that
keeps
listening
for
this
goal
state
as
soon
as
it
gets
a
new
goal.
State
it'll
go
convert
that
crd
into
deployments,
so
that
gives
us
our
customers
ability
to
deploy
across
the
world
across
all
of
our
data
centers.
Within
minutes,
over
the
last
four
and
a
half
years,
our
git
mono
repo,
where
they
check
in,
have
already
reached
100
000,
commits
last
year
alone.
We
had
20
000
commands,
so
it's
highly
used
inside
our
infrastructure.
A
A
Let's
see?
What's
inside
this,
it's
implemented
in
three
layers
from
right
to
left
the
right
post
layer
uses
rpm
to
install
the
binaries
and,
as
you
said
before,
it
uses
systemd
configuration
to
manage
the
daemons.
So
systemd
is
the
component
will
restart
your
service
if
it
fails
that
layer
is
then
used
by
the
node
puppet
class.
A
A
Also.
We
had
to
implement
features
just
for
hcd
and
kubernetes,
for
example
the
hcd
service
it
waits
before
for
quorum
to
be
formed
before
it
proceeds.
The
service
account.
Support
in
kubernetes
supports
key
rotation,
and
there
is
one
more
detail
that
I
did
not
show
in
the
previous
slide,
so
we
were
forced
to
run
flannel
on
docker,
but
docker
needs
final
running
to
start.
A
A
A
A
A
Also,
the
iteration
cycles
are
very
expensive
on
puppet,
like
any
small
change,
it
takes
in
a
lot
of
effort
to
try
it
out
and,
lastly,
the
way
modules
are
composed
in
puppet
any
base
module
that
you
may
not
own.
Maybe
a
common
os
library
change
can
easily
break
your
module
and
you
will
hear
about
this
more
in
the
subsequent
slide.
A
So
that
concludes
our
deep
dive
into
our
puppet
infrastructure.
Now,
what
did
we
have
to
do
to
operationalize?
This
stack
we
knew
from
the
beginning.
We
have
to
run
our
control
plane
in
high
availability
mode,
so
we
run
multiple
servers,
so
our
xcd
runs
in
quora
mode,
but
kubernetes,
for
example,
the
cubelet
cannot
talk
to
multiple
api
servers,
so
that's
how
we
solve
that
was
using
installing
h
a
proxy
on
every
machine.
A
So
we
configure
cubelets
to
talk
to
the
proxy
as
if
it's
talking
to
the
api
server
and
then
the
proxy
will
load
balance
across
the
api
servers
for
security.
We
hardened
all
our
communication
from
service
to
service
across
machines
using
mtls,
based
on
our
custom
pki
configuration
also,
we
isolated
two
workloads
that
should
not
be
on
the
same
machine
by
using
node
labels.
A
A
A
A
A
This
has
features
such
as
warm-up
period,
so
it
waits
for
a
failures
to
happen
for
a
little
bit
of
time
before
it
alerts
us.
So
it
takes
care
of
momentary
blips.
It
also
has
cooldown
periods,
so
once
it
alerts
us,
it
waits
for
another
certain
configurable
period
of
time
before
alerting
us
again.
We
also
added
snooze.
A
This
is
very
helpful
when
you
want
the
alerts
to
stop
for
a
certain
amount
of
time,
because
you
know
you
about
to
fix
something
and
if
you
forget
to
fix
that
or
it
takes
longer,
he'll
come
back
and
alert
you
again
to
remind
you.
This
need
still
needs
to
be
fixed,
so
watchdogs
can
alert
us
directly
through
pagerduty,
as
shown
on
the
top,
but
they
also
emit
crds.
A
B
B
We
had
made
the
decision
to
use
it
and
we
were
still
learning
about
the
various
knobs
it
offered.
One
of
those
knobs
was
hostpath
one
fine
day.
We
found
that
every
rolling
update
of
a
deployment
could
result
in
parts
getting
stuck
in
container,
creating
more
debugging
into
cubelet
logs,
showed
teardown
failures
and
volume,
setup
failures
for
secrets
and
empty
darwinians.
B
Further
along
following
a
series
of
breadcrumbs
in
cubelet
logs
and
running
ls
off
searching
through
pop
mount
and
using
other
linux
tools,
we
found
that
a
partner
team
had
mounted
the
root
file
system
using
hostpath
inside
their
deployment.
Yes,
you
heard
that
right.
The
same
brute
file
system
also
has
var
lib
cubelet,
which
is
the
default
path
in
which
cubelet
mounts
a
secret
and
empty
disk.
B
A
B
Partner
team
to
mount
very
specific
paths,
they
needed
and
went
ahead
and
added
additional
validation,
the
validation
to
allow
only
a
limited
set
of
paths
using
host
paths.
We
also
went
ahead
and
made
read-only
as
the
default
way
to
mount
host
part
while
using
hostpass
is
an
anti-pattern
we
all
realize
now
back
in
the
days.
This
was
a
lesson
learned
that
it
can
be
dangerous
to
expose
all
knobs
of
a
new
tool
that
we
all
didn't
understand
properly.
B
Using
host
paths
can
also
lead
to
services
that
may
work
on
one
distribution
of
linux,
but
not
another.
So,
let's
move
on
to
the
next
war
story,
so
this
is
about
bridge
networking
failures.
This
story
is
also
about
our
early
days
of
kubernetes,
we
saw
a
customer
was
complaining
with
intermittent
connectivity
failures
with
little
more
debugging.
B
We
found
that
the
failure
only
occurred
when
the
client
and
server
parts
were
residing
on
the
same
post
after
a
lot
of
debugging
reading
through
comments
upon
comments
and
kubernetes
issues,
we
learned
about
the
new
concept
called
hairpin
mode,
pretty
sure
you
have
heard
of
it.
Helping
mode
is
setting
on
the
bridge
interfaces
that
enables
a
packet
received
from
an
interface
to
be
routed
back
on
the
same
interface.
B
The
help
and
mode
did
not
apply
in
this
specific
case,
because
the
client
and
server
pods
were
different.
So
after
some
more
debugging
we
found
out
about
the
ciscl
kernel
flag,
called
net
bridge
bridge
nf
call
ip
tables.
You
can
see
it
on
the
screen
on
the
slides.
This
controls,
whether
or
not
the
packets
traversing
the
bridge
are
sent
to
ip
tables
for
processing.
B
B
B
Since
the
two
fcd
members
were
still
in
quorum,
they
convinced
the
third
xcd
member
to
join
this
new
cluster
and
replicate
their
empty
state
to
their
third
member.
Now.
The
reason
for
why
hcd
got
wiped
out
of
its
data
is
not
clear,
but
the
reason
for
why
third
member
joined
the
new
cluster
and
lose
all
its
state
was
that
flag
initial
cluster
state,
you
can
see
it
on
the
screen.
It
was
still
set
to
new
on
all
members.
B
If
this
flag
was
set
to
existing
the
last
hcd
member
would
not
have
joined
the
new
cluster
since
it
was
already
part
of
an
existing
cluster,
and
we
could
have
some
chance
in
recovering
that
data.
The
xct
cluster
formation
requires
all
members
to
be
in
new
state
initially,
but
must
be
switched
to
existing
later.
To
avoid
this
scenario,
we
use
the
puppet
hiera
mechanism
to
switch
from
new
to
existing
in
a
month
time
frame
when
the
new
data
center
is
ready
to
go,
live
so
people
in
the
public
cloud
with
managed
cloud
providers
we
will.
B
They
will
probably
never
get
to
experience
this,
but
on
bare
metal.
We
need
to
expect
this
some
similar
to
how
disk
and
network
failures
are
part
of
any
distributed
systems
and
be
prepared
for
it.
This
incident
showed
how
the
flags
are
confusing
and
we
started
doing
regular
backups
of
our
all
our
lcd
data
and
also
doing
regular
game
day
exercises
to
try
such
scenarios
and
make
sure
that
our
run
books
are
ready
for
dealing
with
this.
B
So,
let's
move
on
to
our
next
war
story,
and
this
war
story
is
about
memory
lakes.
It's
about
the
kubernetes
cluster,
which
was
rapidly
leaking
pods.
You
may
ask:
how
can
you
really
leak
pods
one
day,
our
on-call
reported
that
the
apa
server
machines
were
going
unresponsive
on.
Looking
at
our
graphs,
we
found
their
memory
usage
steadily
increasing
until
they
would
become
unresponsive
while
debugging
this,
we
were
manually,
restarting
them
to
prevent
the
hc
cluster
from
tipping
over
and
lose
score.
B
B
Further
debugging
led
to
a
mutating
weapon
that
was
incorrectly
adding
a
json
patch.
This
json
patch
in
trying
to
add
a
label,
dropped
all
other
labels,
since
the
labels
were
all
gone,
the
pod
being
created
was
not
accepted
by
the
deployment
controller
and
hence
the
deployment
controller
kept
on
creating
new
pods
to
satisfy
its
desired
goal
state.
This
continued
until
new
pods
filled
up
the
entire
api
server
host
memory
and
crashed
it.
B
We
finally
fixed
the
bug,
but
there
were
a
ton
of
learnings
from
this
and
a
lot
of
debugging
for
sure
so
mutating
the
books
are
a
deadly
weapon
that
must
be
constantly
validated
and
can
read
in
a
cluster.
Also,
adding
limits
is
better
than
rendering
the
cluster
unusable,
and
this
holds
true
for
both
hosts
and
parts.
B
We
had
just
enabled
service
accounts
and
puppet
had
begun
rolling
out
this
feature
across
our
data.
Centers
puppet
uses
a
carrying
mechanism
for
our
bare
metal
hose
based
on
their
puppet
roles
for
masters,
which
includes
api
server,
controller
manager
and
schedulers.
We
can
read
the
changes
in
one
master
before
rolling
it
out
to
all
masters.
B
B
B
The
puppet
calorie
mechanism
had
resulted
in
a
situation
where
a
master
node
carrying
the
latest
puppet
code
got
all
the
service
account
flags
for
api
server
and
controller
manager,
but
the
controller
manager
that
was
leader
was
running
on
an
older
node,
so
the
older
controller
manager
did
not
ensure,
and
you
can
see
it
on
the
screen.
So
the
older
controller
manager
did
not
ensure
that
tokens
for
the
reference
service
account
in
the
pod
being
created,
exists
and
hence
the
port
being
created,
actually
failed.
B
The
admission
check
on
the
api
server,
which
you
see
on
the
left,
this
happened
because
the
api
server
handling
that
request
was
a
new
api
server
in
puppet,
can
remote
and
was
validating
the
existence
of
the
token
we
fast
forwarded
that
feature
so
that
the
puppet
can
remote
is
applied
to
all
the
hosts,
and
that's
all
the
issue.
We
all
learned
that
our
channeling
mechanism
is
not
the
best
way
to
roll
out
new
kubernetes
features,
especially
the
ones
which
span
across
api
servers
and
controller
managers
and
more
thought
is
needed.
B
So
this
is
all
the
ro
stories
we
just
talked
about.
Now
we
are
going
to
summarize
to
summarize
rolling
your
own
kubernetes
on
a
bare
metal.
Non-Vm
infrastructure
is
hard
and
requires
a
lot
of
expertise
and
investments.
Hopefully,
we
have
shared
some
recipes
today
that
you
can
take
home
and
apply
to
your
own
infrastructure.
B
We
have
invested
in
a
fully
automated
highly
available
and
secure
deployment
pipeline
for
docker
at
cd
and
kubernetes
infrastructure
across
all
our
data
centers,
which
has
evolved
over
the
last
4.5
years.
The
whole
infrastructure
has
tight
integrations
with
networking
security
and
monitoring
we
have
invested
in
watchdog,
monitoring,
alerting
and
visibility
pipeline.
You
saw
some
of
that
in
anu
slide
when
he
was
diving
deep
into
the
monitoring
watchdogs.
B
B
We
all
earlier
open
source
two
of
our
projects.
One
is
generic
site
car
injector,
which
enables
you
to
declaratively
inject
side,
cars,
meeting
the
need
of
a
wide
variety
of
our
infrastructure
teams.
This
project
is
currently
used
in
production
by
more
than
seven
teams
for
critical
side
cars
like
pki
secrets
and
kerberos,
and
we
have
some
interesting
ideas
in
this
space.
You
will
hear
more
about
by
the
end
of
this
year,
so
we
also
open
source
loop
last
year,
which
is
our
history
visualization
tool
for
kubernetes
resources,
and
it
helps
in
debugging
and
monitoring.
B
A
Because
you
can
hear
us
so
we're
going
to
try
to
answer
all
your
questions
so
do
we?
How
do
we
do
rolling
update
of
our
kubernetes
clusters?
So,
as
we
said
in
the
talk
we
do
connecting,
we
basically
means
we
pick
one
server
out
of
our
five
control
plane
machines
to
run
the
new
version.
If
everything
looks
good
after
a
certain
amount
of
time
interval,
then
we
start
rolling
the
update
to
the
other
api
servers.
A
B
A
A
Recovery
and
bare
metal-
it's
hard-
I
mean
that's
the
biggest
problem
with
bare
metal.
Is
you
have
no
way
to
quickly
nuke
a
machine
if
something
goes
wrong,
so
you
really
have
to
treat
our
api
servers
as
almost
like
pets
like.
If
something
happens
to
one
of
our
api
servers
out
of
the
five,
then
we
want
to
get
it
recovered
very
quickly.
A
B
A
Somebody
else
has
to
use
terraform
or
ansible
there
are
other
teams
have
tried
ansible.
Nobody,
as
we
know,
has
used
it
for
kubernetes,
as
you
said,
in
our
talk
like
using
puppet
for
kubernetes,
is
a
little
challenging
because
of
lack
of
orchestration.
So
if
you
had
to
start
over
again,
we
probably
might
not
have
picked
puppet
and
might
have
looked
at
something
else
that
gave
us
orchestration
across
api
servers.
If
something
bad
happens,
we
can
stop
the
rolling
update
automatically.
B
A
How
do
you
use
load,
balancer
services,
so
we
run
our
custom
software
load,
balancing
stack
that
does
proxying
across
our
pods,
so
the
custom
load,
balancing
stack
can
go
talk
to
the
api
server
grab
the
list
of
ips
of
actual
pods,
and
since
we
run
sdn
stack
that
gives
us
flat
ips
across
pods
and
then
metal.
It
can
redirect
traffic
from
the
proxy
layer
of
the
load
balancer
directly
to
the
parts
and
that's
a
much
better
way
that
we
have
found
to
run
software
load
balancing
on
top
of
our
infrastructure.
B
B
I
don't
know
exactly
what
the
question
is,
but
I
think
we
we
do
have
a
homegrown
service
mesh
solution
and
we
are
slowly
migrating
that
to
histo
on
first
party
bare
metal,
and
I
mean
istio,
uses
endpoints
and
service.
Kubernetes
services
rely
on
endpoints.
So
if
you
can
be
more
specific
on
that
question,
that
would
help.
A
A
A
A
B
B
So
how
long
have
you
been
running
kubernetes
on
bare
metal
in
production
that
we
are
about
to
complete,
like
probably
five
years
now,
just
started
exactly
around
kubernetes
1.3
when
kubernetes
deployment
was
still
an
alpha
feature
and
we
were
thinking
about
whether
to
bet
it
or
bet
on
it
or
not,
but
yeah.
Now
it's
been
around
four
and
a
half
reaching
five
years.