►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Kubernetes at Reddit: An Origin Story - Greg Taylor, Reddit, Inc
Despite being ranked in the top 20 busiest sites in the world, Reddit has been a small engineering organization for most of its 13 years. However: the last few years have seen us multiply in size by 7x, causing our tooling, process, and culture to creak and groan. Discussion, evaluation, and experimentation led to our decision to incorporate Kubernetes. This session takes the audience through the motivations, the story, and the results of rebasing our infrastructure on Kubernetes. Today, Kubernetes forms the core of our internal Infrastructure "product", which is a paint-by-numbers experience that engineers of all backgrounds make use of. The audience will see how and why a small (but rapidly growing) organization operating at a unique scale shifted to Kubernetes, get practical tips and advice, and _maybe_ some dank memes.
To Learn More: https://sched.co/GrRp
A
Hello
well,
I,
know
you're
still
coming
in,
but
we're
gonna
go
on
ahead,
get
started.
We've
got
a
lot
of
ground
to
cover
today.
So
I'll
give
you
a
minute
more
here,
as
you're
coming
in
so
I
am
Greg
Taylor
I
am
an
engineering
manager
at
on
reddit's
infrastructure
team
and
today
I'll
be
sharing
the
origin
story
of
kubernetes
at
reddit.
We'll
cover
some
of
the
historical
context,
go
over
the
motivation
for
our
adoption
and
talk
through
what
we're
building
now.
A
First
of
all,
for
a
moment
for
my
own
gratification
by
a
raise
of
hands,
how
many
are
familiar
with
reddit
is
some
level.
Ok,
let
the
record
show
that
there
are
quite
a
few,
but
for
everybody
else
what
is
reddit
depends
on
who
you
ask,
but
today
we
tend
to
define,
read
as
a
network
of
communities
called
subreddits
and
within
these
sub
reddits
users
are
able
to
share
content
in
the
form
of
text,
images,
videos
or
links,
and
this
is
where
it
gets
interesting.
A
This
is
our
kubernetes
subreddit
located
at
reddit.com.
Slash
are
slash,
kubernetes
and
here,
you'll
find
over
15,000
coober,
not
sharing
articles
tips,
asking
questions
and
sharing
in
victory
and
and
we've
got
the
we've
got
discussion
posts
up
for
each
day
of
Q
Khan
as
well,
so
make
sure
to
stop
by
and
say
hi
and
if
you
haven't
already,
if
you
haven't
already
check
it
out,
I'm
a
little
bit
biased
but
I.
Consider
this
to
be
one
of
the
better
communities
for
kubernetes
on
the
web.
A
So
with
our
briefest
of
brief
introductions
to
read
it
taken
care
of.
Let's
get
started
on
the
main
event:
the
origin
story
of
kubernetes
at
reddit,
so
our
story
begins
back
in
simpler
times,
early
2016,
to
be
more
specific,
so
you'll
see
our
lovable
monolith,
pictured
in
the
middle
standing
alone,
there's
no
other
services
around
it.
We
had
about
20
engineers
in
total,
which
were
split
into
around
three
different
engineering
teams,
and
since
this
story
is
from
the
perspective
of
the
infrastructure
team,
let's
talk
a
little
bit
about
what
life
look
like
for
us.
A
Additionally,
we
handled
most
of
the
non-trivial
debugging
when
things
do
go
wrong,
since
we
had
enough
access
to
poke
at
the
dustiest
corners
of
the
infrastructure
and
the
codebase,
and
this
arrangement
served
as
well
for
well
around
a
decade
until
the
middle
of
2016
change
was
a
foot
high.
Okay
bear
with
me
for
a
minute
where
this
is
relevant.
Anybody
here
watch
the
original,
Power
Rangers.
A
We,
if
you
can
imagine
or
see
ghosts
as
their
Steve,
saying
that
that's
totally
how
it
went
down
so
started
the
year,
maybe
a
little
bit
less
than
20
engineers
ended
the
year
over
60
engineers
and
it
only
picked
up
from
then
we
didn't
slow
down,
and
in
2017
and
2018
we
kept
adding
new.
We
kept
adding
new
teams
and
they
kept
that
a
new
server.
You
know
just
all
sorts
of
interesting
things
are
going
on
there.
A
So
as
far
as
the
trajectory
of
this
into
the
now
present
were
arrayed
around
under
200
engineers,
so
that's
a
rate
around
a
10x
increase
from
beginning
of
2016,
which
is
pretty
substantial
for
us.
Anyway,
as
our
ranks
grew,
we
began
having
difficulty
in
scaling
the
development
of
the
monolith
with
our
rapidly
growing
team,
and
while
there
are
plenty
of
examples
of
organizations
successfully
developing
against
the
monolith,
our
particular
model
was
fragile
and
full
of
sharp
edges
that
had
accumulated
over
almost
ten
years.
It
was
difficult
to
make
changes
without
unexpected
breakages.
A
There
was
layers
and
layers
of
caching
and
all
sorts
of
things
in
there
nasty
bits.
So
development
was
much
like
this
was
much
like
the
development
equivalent
of
the
butterfly
effect,
but
with
a
lot
more
explosions
and
as
a
result
of
this,
our
productivity
was
not
scaling
as
well
as
we
had
hoped
as
the
or
group.
A
Split,
oh
I'll
start
picking
apart
the
monolith
into
a
service-oriented
architecture
or
an
SOA.
Previously,
as
a
smaller
organisation,
the
additional
complexity
of
an
SOA
was
unattractive
to
us,
but
as
a
larger
and
still
rapidly
growing
organization,
with
a
difficult
monolith
we
saw.
This
is
the
best
path
board.
We
felt
that
this
would
give
us
a
better
separation
of
concerns
and
understood
that
this
would
come
with
a
with
a
few
new
problems
to
deal
with.
With
the
decision
made.
A
They
plunged
headfirst
into
our
brave
new,
service-oriented
world,
and
lo
and
behold,
all
of
our
problems
were
solved.
We
stood
up
our
own
death,
star
services
and
trotted
off
into
the
sunset
thanks
for
joining
me
today.
Folks,
that's
all,
and
then
I'll
see
you
later
or
maybe
not
so
we
started
standing
up
new
services
and
we
hired
four
and
we
built
out
new
engineering
teams
and
those
new
teams
stood
up
their
own
services
and
before
we
knew
it
routing
all
of
these
new
services.
A
Through
the
infrastructure
team
for
production
ization,
the
problematic
we
had
became
a
bottleneck
when
I
say
we,
the
infrastructure
team
service
owners
had
to
wait
for
us
to
stand
up,
configure
new
infrastructure
for
them,
and
also
we
handled
most
of
the
incident
response
and
production
debugging.
Since
we
had
the
most
permissions
can.
A
We
trained
of
the
few
more
infrastructure
oriented
teams
to
use
our
standard,
tooling
or
standard
infrastructure
operations
willing
to
do
some
of
this
work
on
their
own,
and
these
teams
were
able
to
gather
a
measure
of
Independence
that
relieved
some
of
the
pressure
on
us,
but
is
this:
this
is
the
solution,
our
problems
here.
Expecting
everybody
to
you
know
moonlight
as
an
infrastructure
engineer.
Give
me
like
some
nod:
there's
some
shaking
of
heads.
Oh
man,
I
got
like
yeah
no
I.
Thank
you
spirited.
A
Yes,
this
wasn't
something
we
could
do
for
the
entire
the
organization
and
the
truth
of
it
is
that
not
all
teams
want
to
or
are
able
to
operate,
the
full
stack
for
their
service.
There
is
really
a
spectrum
of
infra
comfort
to
contend
with,
for
example,
when
faced
with
a
prospect
of
doing
infrastructure
work,
some
teams
will
eagerly
dive
in
no
problem.
Others
will
soon
throw
their
laptop
in
the
trash
leave
the
building
and
change
careers
to
something
more
pleasant.
Why?
A
If
you
might
imagine
so,
this
summarizes
our
situation.
We
were
lacking
the
ability
for
engineering
teams
to
fully
own
their
service,
so
we
work
to
establish
Redis
definition
of
service
ownership,
and
we
define
this
as
saying
that
a
service
owner
at
reddit
is
empowered
and
expected
to
develop
their
service
from
start
to
finish,
to
deploy
their
service
early
and
often
and
once
in
production,
they're
expected
to
operate
their
service.
This
includes
owning
the
availability
and
performance
of
said
service,
and
after
hearing
all
this,
you
may
say
to
yourself.
A
Yes,
Greg
is
that
sounds
great,
but
how
do
we
get
there?
Well,
we
have
to
get
away
from
expecting
engineering
teams
to
piece
together
services
from
a
few
dozen
building
clocks,
many
of
which
are
outside
their
technical
background
and
instead
appoint
engineering
teams
at
a
well-manicured
path
and
concept
to
production.
That
sounds
lovely.
A
Doesn't
this
path
through
production
should
be
explicitly
defined
and
assume
no
prior
exposure
to
our
stack
will
expose
the
smallest
possible
technical
surface
area
to
the
user,
avoiding
the
need
to
learn
a
dozen
new
tools
and
technologies
and
to
execute
on
this
vision.
We
needed
to
package
up
our
knowledge
process,
best
practices
and
other
things
into
a
more
accessible
form,
I'd
like
to
introduce
you
to
infrared
our
reddit
infrastructure
product
built
on
kubernetes.
The
rest
of
this
talk
will
go
into
more
detail
about
the
theory
behind
in
the
implementation
of
infrared.
A
So,
to
summarize,
the
mission
for
infrared,
a
service
owner
should
be
able
to
develop,
deploy
and
operate
their
service,
and
we
must
allow
for
this,
regardless
of
an
engineer's
background
and
infrastructure
and
you're
gonna
hear
me
say
that
those
previous
two
defendants
is
quite
a
bit
here
so
bear
with
me
and
also
understand
that
these
are.
These
are
just
central
to
our
philosophy.
A
It
may
look
a
little
bit
different
at
your
company,
but
have
something
like
this
and
stick
to
it,
so
we've
defined
a
mission
statement
or
whatever
you'd
like
to
call
it,
and
the
idea
here
is
that,
if
an
engineer
passes
the
bar
for
engineering,
they
should
be
able
to
own
a
service
and
we'll
start
by
zooming
and
I'm.
Just
the
develop
portion
of
this
underpinning
much
of
our
offering
is
a
reddit
service
specification.
A
In
order
for
us
to
build
an
accessible
and
productive
experience
for
engineers,
we
need
our
services
to
be
of
a
similar
shape,
and
that
means
that
we
have
to
establish
some
standards
to
be
followed,
regardless
of
the
language
or
choice
of
frameworks.
We've
standardized
things
like
our
RPC
protocol,
how
we
set
and
retrieve
secrets
and
how
we
emit
metrics,
isn't
races
and
also
how
we
structure
our
logs.
A
We
have
encoded
our
standards
into
a
language,
independent
service
specification
and
from
there
we
have
service
frameworks
for
each
language
in
use
at
reddit
and
all
of
these
person
per
language
service,
but
frameworks
all
implement
that
specifications
we're
talking
about
here.
If
you
like
in
this
illustration
of
this,
you
can
see
our
Python
implementation
of
the
service
spec
on
github,
it's
under
our
edit
organization,
and
it's
called
base
plate
and
mind
you.
This
is
very
opinionated.
A
It
is
probably
not
gonna
be
useful
to
you
out
of
the
box,
but
this
may
inspire
you
or
otherwise
inform
similar
processes
happening
at
your
respective
organization,
so
check
it
out.
So,
with
her
aunt,
our
service
specification
in
hand,
the
beginning
of
our
development
process
starts
with
a
code
generator.
Then
we
use
something
called
cookie
cutter
by
Audrey
Greenfield
anybody.
You
know
what
that
is:
okay,
well
just
a
few.
Basically,
this
allows
us
to
turn
our
service
stub
that
follows
our
service
spec.
This
gets
assistance
as
far
as
a
hello
world
type
state
of
development.
A
The
generator
project
also
includes
supporting
pieces
like
a
docker
file
and
CI
and
deploy
configs,
and
since
all
new
services
follow
our
service
specification,
we
are
also
able
to
generate
home
charts
for
the
new
service.
Our
reddit
service,
home
charts
are
not
tied
to
particular
languages
or
libraries,
and
that's
very
important
for
us.
In
defining
that
specification,
we
can
mix
and
match
languages
with
these
generated
home
charts,
and
this
is
a
much
better
experience
for
the
developers
and
by
offering
automation
to
kick-start
a
new
service.
A
Our
service
bootstrapping
process
is
much
easier
and
it
also
avoids
the
expectation
that
all
engineers
know
how
to
write
a
home
chart
from
scratch.
Wait
you'll
see
here,
that
is,
that
is
a
very
inhumane
to
subjective
thing.
You're
here,
your
engineers
too,
so
consider
you
know
whatever
whatever,
which
you
like,
be
it
abstraction
or
automation,
or
both
strongly
consider
it,
and
now
that
we
have
a
new
service
and
it's
corresponding
home
chart
ready
we're
this
time
start
development.
A
Before
this
purpose,
we
use
Google
scaffold
to
facilitate
that
process,
and
this
gives
us
an
edit
rebuild
refresh
loop
that
is
accessible
and
familiar
to
our
engineers
and
as
far
as
some
of
the
thought
process
behind
this
in
our
development
process,
some
of
the
major
considerations
are
that
it
should
be
accessible
without
a
deep
understanding
of
kubernetes.
It
must
be
a
similar
to
production
as
possible.
A
It
should
reuse
the
same
helm,
charts
and
docker
images
in
development
staging
and
production,
and
this
is
a
recent
one
that
we've
we've
had
a
really
hard
time
dealing
with,
but
we
must
avoid
exhausting
our
standard,
dev
laptops
to
resources
and
the
primary
culprit
here
is
memory
for
us
and
as
an
additional
interesting
note,
we
haven't
historically
used
mini
cube
locally,
but
are
instead
starting
to
shift
more
of
this
development
flow
to
a
remote
clusters.
Due
to
the
resources
required
to
run
the
full
tapestry
of
services.
A
I
would
love
to
chat
with
you
a
little
bit
more
out-of-band.
So
come
talk
to
us
after
the
presentation,
if
you'd
like
so
with
development
cover.
Let's
talk
about
how
we
build
test
and
deploy
our
services.
We
use
something
called
drone
to
run
tests
and
build
artifacts,
and
the
artifacts
were
mostly
in
the
form
of
docker
images,
and
there
was
a
time
where
we
were
driving
to
pull
driving
deploys
via
drone
in
a
helm
plugin.
A
When
things
go
wrong
and
we're
really
excited
about
the
future
extensibility
that
all
of
the
api's
and
spinnaker
offer
to
us,
so
we're
not
bound
to
the
spinnaker
UI
long
term
and
we
can
do
all
sorts
of
fancy
stuff
to
illustrate
Howl's
comes
together.
Let's
talk
through
a
typical
staging
or
production
deploy
flow.
First,
a
developer
pushes
to
a
services
canonical
repo.
Then
our
CI
system
runs
the
services
tests
and
publishes
a
docker
image
from
here.
We
offer
a
choice
of
two
standard
deploy
flows.
A
Service
owners
can
elect
to
have
CI
triggered,
deploy
automatically
or
they
can
take
matters
into
their
own
hands
and
rely
on
a
manual
trigger
we're
likely
to
offer
additional
flows
and
new
triggers
in
the
future.
For
example,
chat,
ops,
the
CLI
some
web-based
thing
other
than
the
spinnaker
UI
doesn't
really
matter,
and
one
of
the
nice
things
about
spinnaker
is
that
we
can
lean
on
the
API
to
do
interesting
in
new
things,
we're
not
beholding
them
to
its
default
UI.
A
Now
that
we've
covered
deploys,
let's
talk
about
how
we
facilitate
service
operation.
This
is
where
the
meat
of
things
is,
and
this
is
probably
where
I
suspected
a
lot
of
the
interest
is
so
I
will
ramble
on
a
bunch
for
you
today.
So
first,
an
important
bit
of
groundwork.
The
infrastructure
team
worked
with
the
rest
of
engineering
to
define
a
contract
that
codifies
what
is
expected
from
both
parties.
Service
owners
are
expected
to
learn
the
barest
of
basics,
about
kubernetes
and
they're
expected
to
develop,
deploy
and
operate
their
own
services.
A
The
infrastructure
team
is
responsible
for
provisioning,
maintaining
and
scaling
the
underlying
kubernetes
clusters,
and
if
a
service
requires
Amazon
resources,
the
cache
a
database,
the
infrastructure
team
is
best
equipped
to
provision
and
operate
those
it's
not
as
simple
as
apt-get,
installing
Postgres
memcache
or
cassandra
at
our
scale,
you're
going
to
a
really
bad
time.
So
it's
you
don't
want.
A
We
don't
want
all
of
us,
our
engineering
teams
to
have
to
take
their
lumps
like,
like
we
have
on
the
infrastructure
team
and
finally,
the
infrastructure
team
supports
and
advises
the
rest
of
the
engineering
organization
on
how
to
design
reliable,
performant
and
scalable
services.
We,
as
an
engineering
organization,
are
at
our
best
when
each
team
is
spending
the
most
of
their
time
on
the
problems
that
they
have
been
tasked
with.
A
So
we
spend
a
lot
of
time
thinking
about
what
a
productive
experience
looks
like
for
a
service
owner,
and
we
don't
want
engineers
to
have
to
struggle
to
figure
out
what
to
do
next
or
aimlessly
slog
through
kubernetes,
your
own
or
spinnaker
documentation.
We
instead
provide
a
library
of
paint-by-numbers
guides
to
cover
common
tasks
and
problems.
For
example,
here's
how
you
create
a
new
service,
here's
how
to
have
your
application
assume
a
specific
set
of
permissions.
A
Here's
how
to
enable
auto
scaling,
here's
how
to
deploy
an
experimental
release
that
gets
a
fraction
of
production
traffic
and
if
we
are
succeeding
in
our
goal,
questions
of
how
do
I
do
X
are
responded
to
with
a
link
to
an
existing
guide
or
how
to
and
most
importantly
of
all.
In
addition
to
all
of
this,
we
run
training
sessions
in-person
and
distribute
recordings
org-wide
as
they
happen.
It's
not
enough
to
forego
Nettie's
over
the
wall
at
your
engineers
and
expect
good
things
to
happen.
A
And
since
we
create
a
kubernetes
namespace
per
service,
it
then
becomes
a
matter
of
granting
the
service
owners
full
mission
to
the
namespaces
for
their
services
that
they
own.
The
end
result
is
that
service
owners
are
empowered
to
get
hands
on
with
their
own
services,
but
are
unable
to
go
spelunking
and
cube
system
or
other
know
the
regions
within
our
clusters,
but,
like
they
say,
with
great
power,
comes
great
explosions.
A
Unfortunately,
we
can
prevent
whole
classes
and
mistakes
from
happening
with
some
forethought
and
in
cases
where
mistakes
are
made,
we
can
also
limit
some
of
the
damage.
For
example,
resource
limits
on
CPU
and
memory
can
reduce
the
likelihood
of
resource
starvation.
Network
policies
reduce
the
likelihood
of
unauthorized
access,
and
we
make
these
easy
to
include
by
virtue
of
baking
these
into
our
home
chart
generator
coming
back
to
the
home,
chart
generator
you're,
seeing
this
is
an
important
thing.
A
Another
issue,
another
issue
we've
seen
on
inside
and
outside
of
kubernetes
is
a
misbehaving
service,
hammering
it's
downstream
dependencies,
retry
storms,
just
random
spamming,
happening
all
sorts
of
other
fun
things
in
bears.
Oh
my,
and
to
address
this
rolling
out,
sto
and
envoy,
which
will
allow
us
to
enforce
service.
The
service
rate
limits,
add
circuit,
breaking
and
we'll
get
a
little
bit
more
on
the
observability
front,
as
well,
so
again
feel
free
to
come
and
talk
to
us
about
this
after
the
presentation
we're
still
early
on
but
happy
to
compare
notes.
A
Another
thing
that
can
make
for
a
long
day
is
inadvertently
exposing
a
service
to
the
world
or
running
a
questionable
docker
image.
We've
we've
been
experimenting
with
the
CNCs
open
policy
agent,
which
will
allow
us
to
centrally
define
policies
to
prevent
these
and
numerous
things.
And
finally,
we
want
to
grant
all
of
the
permissions
needed
for
a
service
owner
to
operate
their
service,
but
nothing
more
right.
A
Now
we're
managing
our
Beck
policies
with
per
cluster
II
Amal,
manifests
and
git,
but
open
policy
agent
will
give
us
a
more
pleasant
way
to
centrally
manage
these
I
think
there
are
either
going
to
be
some
sessions
on
open
policy
agent
or
there
already
have
been
check
it
out
super
promising
stuff.
But
despite
our
best
efforts,
they
are
going
to
be
happy
little
accidents
and
all
right.
We
don't
want
here,
yeah
yeah,
that's
that's
exactly
how
it
happens
and
since.
A
In
this
bold
and
beautiful
New
Age
of
service
ownership,
the
service
owner
gets
paged
when
they
decided
to
climb
up
on
their
truck,
and
you
know,
lose
control
and
then
hit
if
tow
phone
pole
or
whatever
so
because
they're
then
expected
that
I
diagnose
and
resolve
the
issue,
or
at
least
take
a
first
whack
at
it.
And
if
the
service
owner
ends
up
stumped
or
suspects
an
underlying
an
issue
with
the
underlying
infrastructure.
A
The
infrastructure
team
is
available
via
an
escalation
chain
and
speaking
of
the
infrastructure
team,
we
are
a
page
when
there
are
issues
with
the
kubernetes
clusters,
caches
databases
or
other
bedrock
pieces
that
require
specialized
knowledge
to
operate.
If
the
service
owner
were
to
page
us,
Bukka
saundra,
for
example,
we're
probably
going
to
end
up
getting
paged
anyway.
A
So
again,
keeping
us
focused
on
the
things
that
we're
best
equipped
to
handle
and,
as
an
added
note,
we
have
a
detailed
incident
response
plan
that
the
whole
engineering
organization
uses
incidents
are
announced
tracked
an
end
with
a
blameless
post
mortem,
which
is
very
important.
As
somebody
who
took
Reddit
down.
It
was
one
of
my
first
acts
as
an
employee,
so
there
is
that,
like
the
think,
I
want
to
credit
the
folks,
a
page
of
duty
for
making
their
excellent
instant
response
guide
publicly
available.
A
It
had
a
good
deal
of
influence
on
how
we
operate
here.
So,
if
you
haven't
seen
it,
please
do
check
it
out
is
excellent
and
when
things
do
go
wrong,
our
service
owners
appreciate
you
having
a
full
suite
of
tooling
to
assist
in
quickly
determining
root
cause
without
insight
into
how
our
services
are
performing.
A
A
trivial
problem
can
lead
to
guessing
games
are
long
or
war
long
and
fruitless
journeys,
as
our
protagonists
repeatedly
find
that
Pinterest
piece
Princess
Peach,
is
in
another
castle,
and
since
our
reddit
service
specification
dictates
what
Diagnostics
are
to
be
admitted
to
where
and
in
which
formats
our
users
get
a
solid
observability
story
out
of
the
box.
This
includes
a
nice
nice
set
of
metrics
and
a
default
alerting
story,
tracing
exception
and
error
tracking
and
a
cinch
central
log
aggregation
and
analysis.
Our
per
language
service
frameworks
take
care
of
plumbing
ever
anything
up.
A
Instead
of
leaving
engineers
to
buy
overalls
and
curse
loudly
under
the
sink
on
their
I'm,
probably
gonna
sound
like
a
broken
record,
but
a
footnote
here
is
that
the
tooling
is
supported
by
training
and
documentation.
Quite
a
few
of
these
tools
require
some
specialized
knowledge
to
crack
in
practice
the
master.
So
we
have
to
be
diligent
in
spreading
that
knowledge,
so
we've
covered
a
lot
of
ground
in
service
operation
land.
Let's
take
a
moment
to
recap:
by
using
kubernetes
in
infrared,
our
internal
infrastructure,
product
service
owners
have
explicitly
define
responsibilities.
A
The
service
owner
is
responsible
for
the
health
and
the
performance
of
their
service.
The
infrastructure
team
is
responsible
for
the
health
of
the
kubernetes
cluster
and
other
bedrock
systems.
Service
owners
have
enough
access
to
fully
owned
their
services,
and
this
allows
them
to
respond
to
incidents,
diagnose
issues
and
make
operational
changes
on
their
own
and
also
to
prevent
and
limit
damage.
Infrared
includes
a
number
of
guardrails
and
safety.
A
We
can
prevent
a
number
of
common
mistakes
through
automatically
enforced
policies,
and
we
also
reduce
the
likelihood
of
a
misbehaving
service
impacting
others
via
our
service
mesh
and
finally,
service
owners
have
the
tools
needed
to
diagnose
issues
on
their
own.
But
what
does
all
of
this?
Well,
don't
what
it
sounds
great
great.
What
is
that?
What
does
it
bias?
Well,
we've
gone
over
how
kubernetes
an
infra-red
enables
service
ownership
or
credit
as
far
as
what
it
bought
us.
A
Instead
of
the
infrastructure
team,
being
a
bottleneck
or
frustrated
and
central
dependency,
we've
stepped
out
of
the
way
and
have
empowered
the
engineering
organization
to
take
ownership
of
our
managed
services.
The
engineering
teams
are
now
able
to
develop,
deploy
and
operate
their
services
mostly
on
their
own,
and
they
can
do
this,
regardless
of
prior
engineering
background
now,
I
will
warn
you.
I
have
been
known
to
drop
the
occasional
dad
joke
and
I
might
be
about
to
do
that
right
now,
so,
just
again,
don't
roll
your
eyes
too
hard.
A
I,
guess
that
you
can
say
that
kubernetes
in
infrared
has
helped
in
for
the
infrastructure
team.
Transform
oh
come
on.
Give
me
a
token
chuckle
I
was
pretty
bad,
wasn't
it
yes.
Thank
you.
I
deserve
that
I'm,
a
terrible
person
I
feel
terrible,
I'm.
Sorry,
everybody,
whereas
we
serve
in
more
of
a
traditional
operations
capacity
in
the
past.
Today
we
spend
more
of
our
time
building
tooling
process
and
automation
that
makes
the
rest
of
the
engineering
organization
happier
safer
and
more
productive.
A
So
we'll
close
with
some
details
on
our
kubernetes
adoption
at
reddit
to
start,
we
operate
about
seven
distinct
clusters
and
are
expected
to
add
about
a
three
to
six.
In
the
next
few
months,
we
tend
to
prefer
smaller,
more
numerous
clusters
rather
than
larger,
but
less
numerous
clusters
about
half
of
our
engineering
teams
are
now
interacting
with
kubernetes
those
teams
have
deployed
and
are
operating
about,
20
production
level
services
on
communities.
A
So
at
that
point,
every
new
service,
our
credit,
will
be
provisioned
on
kubernetes
by
default.
Today,
it's
I
think
it
probably
happens
three
out
of
four.
You
know
before
so
at
the
time,
but
it
will
be
the
official
default
of
quarter.
One
so
very
excited
about
that,
and
this
concludes
the
part
of
our
session,
where
I
ramble
at
you
for
about
half
an
hour,
we're
about
to
open
this
up
for
Q&A
I
wanted
to
leave
plenty
of
time.
A
For
that-
and
it
looks
like
we
have
succeeded
in
that-
but
before
before
we
do
that
I
wanted
to
put
some
of
my
colleagues
on
the
spot.
Well,
though,
first
I
want
to
watch
him
squirm
a
little
bit.
Where
are?
You
are
okay,
we're
fortunate
enough
to
have
seven
fine
individuals
from
Redis
infrastructure
team
in
attendance
today?
Oh
boy
I
can
see
them
sweat
and
they
think
I'm
gonna
call
them
up
here
or
something
it's
great.
B
A
Speaking
of
the
team-
yes,
yes
now
you
know
where
this
is
going.
If
you
seem
to
if
what
you've
seen
today
looks
like
something
you'd
be
interested
in
being
a
part
of,
we
are
hiring
across
a
number
of
functions
check
out
our
job
listings
at
reddit,
comm,
slash
jobs.
Here
are
my
details,
along
with
some
resources,
you're
welcome
to
contact
me
on
reddit,
as
GC
Taylor
or
on
Twitter
as
the
same
name.
Also,
please
stop
by
sched
skid
yeah.
A
Okay,
the
thing
will
remain
ajar
schedules
on
it's
not
by
that
leave
a
rating
and
some
feedback
for
me,
so
I
can
do
better
in
the
future.
Thanks
for
listening,
let's
jump
into
QA,
the
team
and
I
will
hang
around
as
long
until
they
kick
us
out
and
we'll
probably
be
on
the
haul.
So
with
that
said,
yes
have
at
it
yell
it
out
to
me.
Yes,.
A
Are
we
doing
chaos
engineering,
so
this
is
not
a
like
structured,
regimented
thing,
but
we
are
always
breaking
things
in
new
and
interesting.
Okay,
we
do
plan
on
building
a
team
out
in
2019
we're
still
kind
of
working
our
way
up
from
the
bottom,
although
over
the
Pyramid
of
whatever
you
would
like
to
call
it,
but
that's
a
really
good
question
and
we're
excited
to
break
things
in
new
and
interesting
ways
in
the
future.
Yes,
let's
get
something
somebody
back
further.
Yes,
oh
it
was
sorry.
Can
you
repeat
that?
A
Yes,
why
seven
clusters?
So
this
is
a
what
why
seven
clusters
and
how
do
you
define
the
boundaries
of
different
clusters?
This
too
is
a
work
in
progress.
So,
basically-
and
if
you
ask
me
six
months
six
months
from
now-
the
answer
is
probably
different,
so
how
it
works
is
we
have
sort
of
a
catch-all
production
cluster?
A
There
are
certain
teams
that
are
a
little
bit
more
infrastructure
oriented
or
have
workloads
that
don't
fit
super
neatly
into
the
current
picture
set
of
kubernetes
and
what
we
have
in
our
offering,
for
example,
very
bandwidth,
heavy
things.
There
are
ways
to
help
the
scheduler
become
aware
of
this,
but
it's
not
a
it's,
not
a
very
clearly
defined
thing.
So,
in
cases
where
workloads
are
very
different-
or
there
are
data
sensitive
concerns,
you
may
end
up
with
a
different
cluster
and
those
are
maintained
and
operated
by
the
infant
team.
A
The
the
cluster
users
that
are
running
the
services,
the
service
owners,
are
responsible
for
the
health
of
the
service
and
we
maintain
the
clusters
yeah
and
we
perfectly
as
far
as
what
some
of
these
clusters
look
like,
we
tend
to
avoid
doing
them
on
about
like
naming
them
after
teams,
because
those
tend
to
change
over
time.
So
it's
more
about
the
functional
thing
that
they're
of
the
workload.
Basically
so
that's
where
we
stand
today
and
I
realize
that
is
a
is
kind
of
a
higher
level
answer.
A
A
A
The
question
was:
why
did
we
move
away
from
home
file
and
we
actually
still
use
home
file?
For
you
know
cluster
core
things
like
you
know,
cube,
I
am
and
stuff.
That's
a
little
more
operations
oriented
the
problem.
There
was
that
you've
got
a
really
nice
declarative
way
to
deploy
things
to
your
clusters,
but
your
route,
it's
a
very
basic
deploy
flow.
Basically,
whatever
deployments
are
daemon
sets
or
the
other
primitive
support.
That's
all
you
get
so
a
spinnaker.
We
get
something
a
little
bit
more
extensible
beyond.
A
Just
I
mean
it's
not
quite
fire-and-forget
with
home
file,
but
it's
it's
not
far
from
it
either.
So
that's
that's
kind
of
and
again
I
know.
This
is
kind
of
vague
I
hope,
that's
a
satisfying
hamster,
but
we
think
that
in
the
future
we'll
be
able
to
do
a
lot
more
spinnaker.
We
get
some
I
realize
I'm
calling
a
lot
over
here.
Look,
yes,
do
we
host
in
the
cloud
or
are
we
on
pram?
We
have.
A
A
What
do
we
use
for
CD
before
spinnaker?
Yes,
it
was
drone
paired
with
a
helm
file
plug-in
in
there
there's
a
there's,
an
open
source,
one
it's
not
by
us,
but
we
forked
that
I
forget
who
the
author
was
it's
it's
it's
an
it's
very
good.
If
you
have
relatively
simple
needs
and
you
don't
need
all
the
fancy
things
that
spinner
who
can
do,
you
can't
go
wrong
with
home
file?
Yes,.
A
Yes,
what
tools
do
we
use
to
launch
and
upgrade
our
clusters?
I'm
glad
that
you
answered
this
in
there,
the
people
that
are
involved
at
that
I'm
about
to
tell
you
about,
but
some
other
folks
are
also
here.
That
would
love
to
talk
to
you
about
it
more.
If
anybody
else
was
curious,
we
treat
our
core
tenant
of
our
platform
is
that
we
treat
our
clusters
is
very
disposable,
so,
rather
than
upgrade,
we'd
prefer
to
stand
up
a
new
cluster
and
shift
the
workload
over.
A
So
it
looks
like
it's
terraform
packer
and
there's
a
little
sprinkling
of
puppet,
but
it's
not
in
the
critical
path,
so
we're
not
dependent
on
it
for
a
fully
bootstrapping
nodes.
So
that's
worked
out
really
well
for
us.
I
felt
bad
about
for
a
long
time
like
this
feels
terrible,
but
now
that
I've
surveyed
the
scene
a
little
bit,
it
sounds
like
this
isn't
as
uncommon
as
I
feel
in
also
we've
had
some
success
with
this
so
happy
to
talk
more
if
anybody's
curious.
Thank
you
good
question.
Yes,.
A
What
do
we
use
for
making
a
container
registry?
Yes,
so
we
had
started
on
Quay,
but
we
had
some
issues
like
we
needed
to
plug
into
our
basically,
our
organization's
I
am
stuff
a
little
bit
better
and
we
ended
up
shifting
through
Amazon
zcr.
It
is
very
bare-bones
you're,
not
gonna,
get
CVE
scanning
you're,
not
gonna,
get
anything
else,
but
it
is
rock-solid
as
fast
and
it's
already
in
Amazon.
So
it
has
served
us
well,
but
it
is
really
only
something
that
you
can
push
images
to
and
pull
it.
A
A
Another
good
question:
do
we
have
a
dedicated
account
for
at
an
artifact
and
docker
image
distribution
in
building?
Yes,
we
do
it's
creatively
named
the
Builder
environment.
We
talked
a
little
bit
about
this
in
our
home.
Summit
talk
earlier
this
year.
If
you're
curious,
we've
got
lots
of
diagrams
and
whatnot,
but
the
idea
there
is
a
very
lockdown,
restricted
environment,
there's
only
four
asset
building
distribution.
So
that's
only
a
subset
of
the
infant
team
has
access
to
that
and
we
try
to
keep
it
as
minimal
as
possible.
Yes,.
A
Are
we
using
Cassandra
as
our
database?
We
have
yes,
so,
okay,
alright,
so
Cassandra
short
story.
There
is
that
at
a
high
scale
you
can,
you
can
have
a
highly
relate
highly
reliable,
a
highly
scalable
Postgres
setup.
But
it's
it's.
It's
a
pretty
difficult!
There's
a
lot
of
there's
a
lot
of
sharp
edges
to
be
aware
of
there,
because
Sondre
gives
us
a
very
distributed
a
little
bit
more
of
a
safe
story,
a
little
more
of
a
scalable
story
more
when
we're
talking
about
ridiculously
high
offs
and
Cori
through,
but
so
a
lot
of
them.
A
What
we
do
is
just
keep
simple
key
value
stuff,
and
this
is
Raiden
casandra's
wheelhouse.
We
don't
use
any
of
the
fancy
features,
so
we've
got
cache
angry
iconic,
signer
behind
it
and
in
the
future.
It
could
also
give
us
a
lot
more
multi-region
stuff
than
we're
doing
right
now.
I
got
time
for
one
more
I'm.
A
brief
question.
Yes
back
here,
yeah.
A
So,
what's
the
start,
smart
stack,
we
did
use
Airbnb
smart
stack,
so
it's
a
a
proxy
and
two
components
called
nerve
and
synapse
for
a
service
registration
or
than
a
proxy.
It
is
very
much
it's
a
it's
like
the
one
part
of
a
service
measure
doesn't
give
you
any
of
the
rate-limiting
or
the
mutual
TLS
or
anything,
and
we
wanted
more
of
that.
So
we
have.
A
We
have
shifted
from
smart
stack
to
envoy
and
now
sto,
so
that
first
looked
like
swapping
out
envoy
for
H
a
proxy
still
pointing
it
at
our
central
zookeeper
store,
and
then
the
next
step
is
that
we're
gonna
start
we're
gonna
we're
gonna
swap
in
pilot,
for
you
know
again
the
zookeeper
side
of
things
and
magic
will
happen,
and
actually
the
person
that
is
is
gonna
be
helping
with
out
on
the
kubernetes
side
is
here
in
a
tenant.
So
please
come
up
to
I.