►
From YouTube: Keynote: Kubernetes at GitHub - Jesse Newland, Principal Site Reliability Engineer, GitHub
Description
Keynote: Kubernetes at GitHub - Jesse Newland, Principal Site Reliability Engineer, GitHub
In this talk, Jesse will provide an overview of the on-premesis Kubernetes deployments that currently power 20% of GitHub's production services. He'll also review the challenges GitHub has faced and overcome so far during their Kubernetes journey, and highlight ongoing and future Kubernetes enhancements that GitHub is excited about.
About Jesse Newland
Principal Site Reliability Engineer, GitHub
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
So
I'd
like
to
start
with
a
quick
story,
a
story
about
software,
a
software
project
that
never
got
off
the
ground
around
four
years
ago
in
the
fall
of
2013,
the
github
ops
team
all
met
in
San
Francisco.
We
just
finished
a
big
data
center
migration,
which
had
bought
us
a
ton
of
time
in
our
race
to
keep
up
with
traffic
with
user
growth
and
with
DDoS
attacks
as
a
team.
We
were
really
excited
about
this
newfound
time.
We
were
full
of
ideas
about
moving
up
layer
of
abstraction.
A
So
at
this
team
meeting
we
started
talking
seriously
about
building
a
platform.
A
handful
of
engineers
on
the
team
had
had
experience,
building
and
running
platform
as
a
service
offerings
and
had
a
design
in
mind
that
used
containers
instead
of
VMs.
We
sketched
out
some
of
the
rough
concepts
and
split
up
some
follow
up
tasks
and
the
team
all
went
back
to
their
home
offices
and
started
researching
the
problem
space.
A
We
picked
a
name
and
started
to
get
more
serious
about
the
work,
but,
as
we
started
to
learn
more
about
the
state
of
container
technology
at
the
time
we
began
to
get
a
better
sense
of
the
amount
of
work
ahead
of
us.
There
was
a
lot
to
do
and
we
started
to
realize
it
was
gonna.
Take
us
quite
a
bit
of
time
to
deliver
something
that
solved
the
problems
in
our
sites.
A
As
we
started,
comparing
notes
with
others
in
the
organization,
we
got
feedback
from
a
few
senior
engineers
that
was
initially
surprising.
They
encouraged
us
to
wait
to
do
nothing.
Let
someone
else
build
this.
They
said
the
designs,
not
bad.
In
fact,
it's
good,
but
it's
worthless
without
execution
and
we're
not
in
the
right
spot
to
ship
something
like
this.
At
the
moment,
this
project
needs
a
team
that
has
more
experience
with
containerized
workloads
and
the
resources
necessary
to
push
the
tech
a
bit
further.
A
This
project
would
be
better
served
by
an
organization
with
experience
managing
more
long-running
projects.
We
want
to
see
this
happen
they,
but
we're
not
the
right
group
of
folks
to
do
it,
especially
not
right.
Now,
instead,
we
need
to
work
on
leveling
up
our
organization
and
our
technology
for
storing
git
repositories.
A
A
A
Now,
I
didn't
tell
you
this
story
to
make
any
sort
of
false
claim
to
any
idea
or
the
platform
or
anything
of
the
sort.
In
fact,
quite
the
opposite,
I
told
you
this
story
to
set
up
a
sincere
expression
of
gratitude
to
everyone
that
was
making
different
decisions
just
a
few
months
later,
deciding
to
dedicate
the
time
the
experience,
the
resources
and
the
emotional
labor
necessary
to
kick
off
the
kubernetes
project.
A
We're
incredibly
grateful
for
kubernetes
at
github
and
want
to
extend
that
gratitude
to
every
single
member
of
this
incredible
community.
Yet
hub
is
standard
in
standardizing
on
kubernetes
is
our
service
runtime
across
our
entire
engineering
organization,
and
we
hope
to
do
that
over
the
next
few
years
we're
making
decent
progress
towards
this
goal.
Right
now,
20%
of
the
services
we
run
are
deployed
on
kubernetes
clusters
inside
our
data
centers,
including
the
service
that
serves
github.com
and
api
doc.
A
When
we
started
the
process
of
migrating
github.com,
the
website
to
kubernetes,
we
hadn't
settled
on
a
design
for
isolating
workloads
running
on
the
same
cluster,
and
our
back
was
still
in
alpha.
So
we
reserved
a
few
clusters
for
the
exclusive
purpose
of
running
this
workload
and
a
few
other
supporting
services,
we're
getting
more
confident
with
our
isolation
design
over
time
and
are
excited
to
revisit
this
soon
to
more
densely
packed
our
workloads.
A
We
also
didn't
have
a
lot
of
experience
operating
clusters
at
the
beginning
of
this
project.
We
did
know
how
to
break
one,
though
we'd
recently
broken
a
cluster
pretty
badly,
while
trying
to
roll
out
an
API
server,
config
change
and
broken
another
one.
When
we
tried
an
upgrade
to
ensure
we
could
hit
our
target
server,
the
reliability
of
github
comm,
we
decided
to
run
its
workload
on
multiple
clusters
in
each
site.
A
We
didn't
end
up
using
any
of
the
existing
Federation
tools
to
do
this,
but
instead
rely
on
our
deployment,
tooling
support
for
deploying
to
multiple
partitions.
In
parallel,
this
multi
cluster
approach
has
been
pretty
useful
recently,
as
it's
allowed
us
to
replace
entire
clusters
without
interrupting
the
flow
of
traffic
or
the
flow
of
work.
A
We've
recently
b'fer
been
preferring
these
tiny
little
things
in
a
configuration
that
gives
us
40,
CPU,
threads
128
gigs
of
RAM
and
to
local
SSDs
for
each
node.
We
packed
tons
of
these
into
each
rack
and
use
node
annotations
to
ensure
that
the
kubernetes
scheduler
automatically
distributes
work
across
these
racks.
Over
the
past
few
years,
github
has
invested
a
ton
of
resources
into
our
data
center
management
practices
and
processes.
A
This
investment
is
paid
off
pretty
well.
We
built
out
some
incredible
facilities
and
of
ship
provisioning,
API
s
and
automation
that
resemble
what
you
might
expect
from
a
public
cloud
providers.
Vm
control,
plane
engineers
can
provision
kubernetes
nodes
from
chat,
but
instead
of
getting
back
at
VM,
our
metal
cloud
deals
out
one
of
these
hooked
up
to
our
high-performance
network.
A
No
configuration
is
managed
by
our
existing
puppet
infrastructure,
which
we
use
to
build.
The
images
used
in
the
provisioning
process,
configuration
changes
and
security
updates
can
be
applied
with
a
pull
request
to
our
puppet
repo,
which
could
be
deployed
to
an
individual
node,
a
cluster
or
an
entire
fleet.
A
So
inside
the
github
production
namespace
on
each
cluster,
there
are
three
primary
deployments:
unicorn
unicorn
API
and
a
thing
called
console.
Service
router
each
pod
in
the
unicorn
deployment
contains
three
containers.
First,
an
engine
decks
container
that
accepts
and
buffers
incoming
requests.
Then
a
container
that
runs
unicorn.
The
Ruby
on
Rails
webserver,
which
processes
requests
sent
over
a
domain
socket
from
engine
eggs,
makes
requests
to
backing
services
like
MySQL
or
spokes,
argot
storaged
here
and
then
renders
HTML
and
finally,
a
proxy
called
fail.
A
Bot
that
accepts
decorates
and
buffers
exceptions
reported
by
the
rails,
app
before
sending
them
to
an
exception
tracking
system,
the
unicorn
API
deployment
powers,
API
github.com
and
is
practically
identical
to
the
unicorn
deployment.
We
deploy
it
separately
due
to
some
differences
in
seasonality,
Computers,
wake
up
and
go
to
sleep
more
frequently
than
we
do
it
turns
out.
A
So
this
third
deployment
that's
running
in
the
get
up,
production
namespace
is
console
Service
router,
which
routes
traffic
from
application
processes
to
services
not
running
in
our
kubernetes
clusters,
like
our
search
clusters
or
our
web
hook
delivery
service.
This
uses
a
combination
of
H
a
proxy
and
console
template
this
service,
and
this
approach
is
basically
a
direct
reimplementation
of
the
way
that
we
solve
this
problem
before
kubernetes.
A
We're
hoping
to
replace
this
hand-rolled
service
mesh
with
envoy
sometime
soon
as
a
part
of
the
project
that
aims
to
improve
the
observability
of
all
sorts
of
service
traffic
to
accompany
our
metal
cloud
service.
We
build
you'll,
be
a
horizontally
scalable
load,
balancer
service
that
can
route
external
or
internal
traffic
to
a
group
of
nodes,
and
we
use
gob,
along
with
node
port
services,
to
get
traffic
into
our
clusters,
currently
using
every
node
in
the
cluster
as
a
back-end,
most
of
the
node
port
services
configured
in
gob
route
to
an
ingress
deployment.
A
A
No
and
then
node
problem
healer,
which
detects
node
conditions
that
are
either
set
by
queue,
paw
patrol
or
node
problem,
detector
and
heals
those
nodes
by
rebooting
them.
A
few
of
our
teams
have
been
building
systems
on
kubernetes
for
around
a
year
and
it's
been
a
joy
to
observe
how
their
workflows
have
changed
over
that
time.
A
So
all
of
these
tools
that
I
just
mentioned
they're
like
real
software
projects
with
tests
and
stuff
they
they
each
have
their
own
repo,
which
contains
the
software
a
description
of
how
it
should
be
packaged
and
a
blueprint
for
running
it
in
production.
A
year
ago,
we
might
have
reached
for
a
bash
script
to
solve
some
of
these
problems
and
to
distribute
that
script
to
our
class
of
nodes.
We
might
have
checked
it
into
our
puppet
repo
I
mean
I,
really
really
like
bash,
but
this
workflow
left
a
lot
to
be
desired.
A
Instead,
we've
ended
up
building
projects
like
node
problem
healer
that
used
kubernetes,
incredible
api's
to
solve
problems.
Node
problem
healer
in
particular,
has
been
a
huge
win
for
our
team,
as
it
fixes
a
class
of
problems
that
sometimes
happen
in
the
middle
of
an
but,
to
be
honest,
I
think
it's
also
been
a
win,
because
the
NPH
abbreviation
makes
us
all
think
of
Neil
Patrick
Harris,
more
often,
which
I
mean
that
cheers
me
up
every
time.
I
see
it.
A
We've
also
built
a
tiny
wrapper
for
kube
CTL,
get
that
we
can
run
in
chat.
It's
been
incredibly
useful
as
we
help
a
ton
of
engineers
learn
how
to
build
systems
using
kubernetes
primitives,
but
these
chat
ops
are
read-only.
We
generally
encourage
engineers
to
use
our
deployment
tools
to
declaratively,
manage
the
kubernetes
resources
that
describe
their
infrastructure
as
much
as
possible.
A
Second,
each
service
and
deploy
target
combination
gets
its
own
namespace,
like
github
production,
for
example.
Next
we
expect
a
deployment
tooling
to
load
and
process
kubernetes,
the
ammo
from
config
kubernetes
environment
in
each
services.
Repo
this
step
can
modify
resources,
filter
them
or
inject
additional
ones.
A
This
step
also
updates
image
fields
to
use
the
image
built
for
the
commit
being
deployed,
as
well
as
some
other
actions
that
are
configured
with
annotations
and
then.
Finally,
we
expect
our
deployment
tooling
to
declaratively
apply
the
modified
resource
config
using
the
same
semantics
as
cube.
Ctl
apply,
so
what's
really
rad
about
this
set
of
conventions
combined
is
that
they
support
changes
to
either
the
application
or
its
kubernetes
configuration
using
the
exact
same
workflow.
So,
no
matter
what
sort
of
change
you're.
A
The
next
stuff
is
everyone's
favorite
part
of
the
pull
request
process
code
review
during
the
course
of
reviewing
a
puller
request.
It's
often
pretty
helpful
for
both
feature
authors
and
reviewers
to
be
able
to
preview
the
effect
of
a
change,
especially
if
that
changed
targets,
UI
elements
sort
of
help.
With
this
we
simply
support
deploying
to
a
review
lab
target.
A
Each
review
lab
gets
unique
URL
that
any
github
engineer
can
log
into
named
after
the
branch
that's
being
reviewed
during
a
review
lab
deployment,
our
deployment
tools
load,
the
ammo
and
config
kubernetes
review
lab
and
make
a
few
modifications.
The
image
field
is
updated,
a
secret
is
injected
for
our
from
our
internal
encrypted
secret
store
and
an
ingress
is
injected
to
configure
the
domain.
Our
deployment
tooling,
then
creates
a
new
namespace
named
after
the
branch
near
deploying
and
applies
the
modified
config
into
that
namespace.
A
So
once
your
PR
has
been
reviewed
and
approved,
the
next
step
is
to
deploy
it
to
production,
but
deploying
a
change
to
a
large
system
like
github
is
inherently
risky
and
we're
always
looking
for
ways
to
reduce
that
risk.
We
we
know
the
github
going
down,
can
have
a
significant
impact
on
the
software
industry.
Right
workflows
are
interrupted.
A
Release
schedules
are
affected,
all
sorts
of
productivity
is
lost,
so
after
looking
really
closely
at
all
the
factors
that
have
contributed
to
outages
in
the
past,
we
added
another
step
to
our
production
deployment
pipeline
a
few
years
back.
The
first
step
of
a
production
deploy
is
now
a
canary
deploy
which
exposes
your
change
to
a
small
percentage
of
traffic.
To
help
you
confirm
that
it
doesn't,
you
know,
materially
regress
the
performance
or
increase
the
error
rate
of
request
flowing
through
the
system
to
support
this
canary
workflow.
A
Our
deployment
tooling
looks
for
deployments
with
a
couple
of
special
annotations
one
declaring
a
deployment
as
canary
a
bowl
and
another
selecting
a
label
to
be
modified
during
a
canary
deployment.
Our
deployment
tooling
clones
any
deployment
mark
as
a
canary
sets
the
value
of
the
label
you've
chosen
to
canary
and
then
deploys
just
that
cloned
deployment.
A
So
once
the
deploying
engineer
is
verified
that
the
error
rate
and
latency
of
the
canary
deployment
hasn't
regressed,
they
then
deploy
their
change
to
production.
This
fans
out
there
changes
to
a
bunch
of
different
targets.
Some
of
the
SAPS
workloads
still
run
on
metal,
so
these
environments
are
updated
in
parallel
with.
Essentially,
our
deployment
tooling
doing
something
like
this
to
apply
the
configuration
from
config
to
rattie's
production.
A
A
A
One
request:
we've
heard
from
a
handful
of
engineers
is
for
persistent
volume
support
in
our
on-prem
clusters.
State
is
pretty
important
for
guitar
right.
Storing
your
data
is
basically
the
core
of
our
product,
but
right
now
our
kubernetes
clusters
can
only
run
stateless
workloads.
A
solution
to
this
problem
is
slightly
complicated
for
us
by
our
metal
cloud
service.
A
Lack
of
a
service
like
Google,
Cloud,
persistent
disk
or
EBS,
but
even
if
we
had
such
a
service
network
storage
won't
give
us
the
performance
we
need
for
some
of
our
systems,
especially
get
many
of
our
systems
are
designed
to
use.
The
SSD
is
available
to
each
node
to
locally
store
a
small
part
of
a
replicated
distributed
system.
A
The
local
storage
management
proposal
includes
a
use
case
covering
persistent
local
storage
that
strongly
matches
some
of
the
systems
that
were
running
today.
Spokes
are
distributed
system.
That
stores
and
replicates
git
repositories
looks
like
a
great
fit
for
this
use
case,
as
does
part
of
our
MySQL
infrastructure,
especially
the
fleet
of
replicas,
that
we
use
to
scale
out
read
traffic.
We're
excited
to
begin
experimenting
with
local
storage
soon,
and
we're
also
excited
to
break
ourselves
out
of
some
open-source
habits
and
out
of
some
bad
open-source
habits
in
the
new
year.