►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Kubernetes at Reddit: Tales from Production - Greg Taylor, Reddit, Inc
This talk is the EAGERLY-anticipated sequel to last year's "Kubernetes at Reddit: An Origin story". Whereas the saga's first installment focused on early results, thoughts, and a rough higher-level vision, this year's edition serves as a retrospective for how it all shook out over Reddit's last year of rapid Kubernetes adoption. The audience will hear of successes, share in the heartbreak of production explosions, and gain insight into what has and hasn't worked well for one of the world's busiest web properties. Topics covered include: * A brief recap of InfraRed, our internal Infrastructure product * How org-wide adoption has progressed * Scaling challenges (Infrastructure and Inter/Intra-team) * Fires, near-misses, and outages, oh my! * Successes and celebration * Lingering questions and challenges * The impact of Kubernetes at Reddit
https://sched.co/UaaR
A
All
sorts
of
other
interesting
things.
If
you
have
an
interest
or
a
hobby
or
need
for
support,
camaraderie
or
encouragement,
you'll
find
a
home
in
at
least
one
of
our
hundred
thousand
communities
and
you'll
be
relieved
to
hear
that
we
don't
need
to
know
your
real
name.
What
you
look
like
where
you
went
to
school,
where
you
from
or
here
you
know,
on
reddit
these
things
don't
have
to
matter.
The
focus
is
on
user
driven
communities
and
the
sharing
and
the
discussion
that
happens
within
them.
A
As
an
example
of
one
of
these
communities,
I
will
shamelessly
plug
our
kubernetes
on
our
kubernetes
or
30,000
kubernetes
users.
Administrators
developers
gather
here
to
ask
questions,
share,
victories
and
to
keep
up-to-date
on
what's
happening
around
the
community.
If
you
haven't
seen
it
yet
check
it
out,
something
that
is
unique
about
reddit
is
the
scale
at
which
we
operate.
Our
user
base
of
over
330
million
makes
us
one
of
the
most
busiest
sites
in
the
world.
A
We
got
to
this
point
with
what
has
been
a
very
small
team
up
until
the
last
few
years,
around
250
engineers
are
responsible
for
the
fifth
most
trafficked
site
in
the
US
and
as
small
as
that
is
relative
to
our
peers.
Even
that
250
feels
like
a
sizable
army
of
Engineers
compared
to
where
we
were
even
just
a
few
years
back,
and
you
may
find
yourself
saying
that
this
is
all
interesting
Greg.
But
what
does
this
have
to
do
with
kubernetes
and
to
that
I
would
say,
join
me
in
my
totally
safe
osha-approved
time
machine.
A
If
you'd
like
to
find
out,
let's
go
back
way
back
to
a
time
where
you
could
count.
All
of
Reddit
engineering
on
your
fingers
and
toes
yes,
the
distant
past
of
2016,
not
even
four
years
ago.
This
was
a
much
simpler
time
for
us
had
around
20
engineers
at
the
company,
a
single
monolithic
code
base
and
a
much
slower
pace
of
product
development.
Our
engineering
organization
functioned
in
a
traditional
further
could
over
to
walk
to
the
ops
team
arrangement
and
the
brave
and
cunning
protagonists
in
the
story.
A
Our
infrastructure
team
numbered
around
six
in
total,
the
beginning
of
2016.
It
was
around
this
time
that
we
could
fit
all
of
reddit's
engineers
in
a
single
VW
camper,
but
soon
reddit
aspirations
grew
and
with
it,
the
engineering
organization.
In
early
2016,
we
were
around
20
engineers
by
2017.
We
grew
to
around
80
in
2018.
We
are
over
150
and
today
we
number
over
250.
Our
VW
camper
is
a
bit
more
cramped
today,
but
all
of
this
rapid
expansion
wasn't
without
growing
pains
with
new
engineers,
came
new
engineering
teams.
A
These
new
teams
struggled
to
jointly
develop
our
aging
moneth,
so
we
steel
it
ourselves
and
embarked
on
a
journey
to
microservices
paradise.
Anybody
heard
the
story
before
or
something
like
that.
We
pulled
logic
and
core
functionality
out
of
our
monolith
and
into
distinct
services
as
more
code
and
logic
were
liberated.
The
sky
filled
with
the
course
of
microservices.
The
beautiful
sounds
of
a
binary-based
opera
filled
the
air
magnificent,
but
that's
not
exactly
what
happened.
A
As
the
new
services
came
up,
our
throw
coat
over
the
wall
to
ops
model
became
more
apparently
and
compatible
with
our
new
direction.
We
struggled
to
enable
service
owners
to
safely
and
reasonably
take
on
more
responsibilities
without
expecting
them
all
to
moonlight
as
infrastructure
engineers.
Service
owners
couldn't
handle
their
own
instant
response
to
bug,
issues
do
DB
migrations
and
many
other
things
without
our
help.
As
a
consequence,
we
ended
up
on
the
hook
for
most
of
these
things,
as
you
might
imagine,
this
wasn't
great
for
us
or
the
service
owners.
A
So
we
went
back
to
the
drawing
board
to
rethink
our
approach.
We
sought
to
shift
our
role
in
the
orga
way,
from
an
Operations
focus
and
mores
towards
in
an
empowerment
and
enable
MIT
focus.
Our
challenge
was
to
package
up
our
knowledge,
our
best
practices
and
such
that
other
teams
could
use
it
without
having
to
become
infrastructure
engineers.
A
Our
end
goal
is
to
empower
any
engineer
at
reddit
to
develop,
deploy
and
operate
a
service,
so
we
shifted
our
mentality
from
providing
a
pile
of
tools
and
good
wishes
to
packaging
our
various
offerings
up
into
a
more
accessible
product.
We
chose
kubernetes
as
our
foundation
then
began
assembling
tools,
processes
and
documentation
into
an
ecosystem
that
we
now
call
infrared
I
won't
have
time
to
go
into
detail
about
infrared
today,
but
check
out
last
year's
presentation.
A
If
you'd
like
to
learn
more
about
our
internal
infrastructure
product
fast
forwarding
to
early
2018,
we
worked
with
a
brave
and
patient
set
of
a
doctors
to
prototype
the
offering
services
on
infrared
enter
production
at
non-trivial
scale
in
mid
2018.
It
was
at
this
point
that
kubernetes
began
serving
external
user
facing
traffic.
After
about
a
year
of
exploding
things
and
also
iterating.
We
launched
infrared
to
org-wide
general
availability
in
March
of
this
year.
At
that
point,
infrared
and
kubernetes
became
the
default
environment
for
services
at
reddit.
A
A
As
our
early
adopters
got
more
comfortable
with
kubernetes,
they
began
to
realize
a
degree
of
self-sufficiency
they
were
able
to
handle
their
own
incident
responses,
diagnosis,
issues
and
generally
handle
day-to-day
operations
with
minimal
infrastructure
team
involvement.
Other
teams
around
the
organization
began
to
notice
this,
and
the
hype
train
was
soon
shooting,
along
at
full
steam,
leading
up
to
our
March
leading
up
to
our
launch.
In
March,
we
ended
up
with
a
long
backlog
of
teams
wanting
to
get
their
services
on
infrared.
Unfortunately,
for
us,
the
service
launch
process
was
manual
and
also
still
taking
shape.
A
Further
complicating
things
are
small
and
14
needed
to
assist
with
launches,
while
also
having
enough
scheduling
daylight
to
iterate
and
automate
on
the
offering
itself.
So
we
needed
a
plan
to
avoid
sinking
all
of
our
times
into
launches.
We
set
up
a
schedule
of
launch
windows
that
were
two
weeks
long.
We
aim
to
use
at
least
half
of
the
two
weeks
for
hands-on
onboarding
and
training
with
new
service
owners.
As
we
ran
through
each
launch,
we
had
improve
the
documentation,
refine
our
processes
and
build
automation
for
ourselves
and
the
service
owners.
A
The
other
thing
that
went
pretty
well
was
our
top-line
goal
of
empowering
service
owners
to
control
their
software
development
lifecycle.
As
the
internal
community
grew
and
infra-red
mature,
we
saw
service
owners
doing
things
themselves
that
would
previously
required
in
for
help.
Examples
can
range
from
DB
migrations,
diagnosing
performance
issues
and
most
exciting
of
all
Incident
Response.
In
fact,
especially
motivated
services,
even
began
self-driving,
their
service
launches,
despite
our
manual
process
and
as
well
as
things
worked
out.
The
first
few
months
following
the
launch
were
especially
challenging.
A
Most
engineers
that
we
were
on
boarding
had
no
prior
experience
to
kubernetes.
Until
we
could
reach
a
critical
mass
of
early
adopters,
most
support
had
to
go
through
the
infra
team,
combined
with
the
work
on
infrared
itself.
This
is
a
lot
for
us
to
juggle.
The
other
side
of
this
coin
is
that
we
didn't
have
a
great
plan
for
how
to
support
new
service
owners.
After
their
first
launch,
we
were
too
small
a
team
to
appoint
a
dedicated
rep
for
each
service
owner.
Do
is
a
long
launch
backlog.
A
We
needed
to
be
wrapping
these
up
and
moving
on
to
the
next
launch
as
quickly
as
possible.
This
left
our
new
service
owners
in
a
state
where
they
had
greatly
expanded
superpowers
that
they
didn't.
Quite
know
how
to
use
as
far
as
what
we
do
different
than
if
we,
if
we
had
a
six
second
chance,
we
would
have
started
the
build-out
of
our
site,
reliability,
engineering
organization
much
earlier
in
the
process,
preferably
before
launch
a
well-placed.
A
In
fact,
our
most
prolific
service
owner
divisions
have
have
a
have
an
embedded
partner
in
there
and
our
SOE
program
enabled
by
kubernetes
and
infrared
service
owners,
are
able
to
handle
most
of
their
day-to-day
operations
on
their
own
and
though
there
is
much
work
to
be
done
on
automating
our
launch
process
and
improving
Docs
and
tooling
and
training
before
we
move
on
I
wanted
to
share
a
visual
visualization
of
our
adoption.
This
is
a
graph
of
the
total
count
of
Reddit
authored
services
running
on
kubernetes
over
time.
A
This
might
be
too
small
to
see
in
the
back,
but
you
can
see
the
jump
in
March
of
2019.
That's
the
second
arrow
pointing
down
sergyei
launch
when
we
launched
also
note
that
the
overall
curve
is
pretty
steady.
This
is
due
to
our
two
week
launch
cadence.
Today
we
have
about
45
reddit,
authored
services
on
kubernetes
in
production.
We
also
have
about
20
services
in
the
queue
to
be
migrated.
A
Moving
on
our
next
topic
is
cluster
management.
At
this
point,
we're
consistency
consistently
launching
at
least
one
service.
Every
two
weeks
between
the
addition
of
new
services
and
the
growth
of
the
existing
services.
We
found
ourselves
with
larger
and
more
numerous
clusters
and
truth
be
told.
We
were
still
gaining
operational
experience
with
kubernetes
in
a
high
traffic
setting.
There
were
times
where
we
had
missteps,
while
tuning
scaling
or
performing
maintenance
on
our
clusters.
Another
fun
fact
is
that
our
adoption
of
kubernetes
predated
Amazon's
elastic
kubernetes
service.
A
A
At
the
time
we
employed
multi
a
Z
clusters
for
all
of
our
environments,
and
this
mostly
worked
well
until
we
blew
something
up
or
were
a
victim
of
infrastructure.
Gremlins
upgrades
were
more
scary
and
heist
as
we're
config
updates.
We
had
all
of
our
eggs
and
a
few
large
kubernetes
shaped
baskets,
but
we
had
a
plan.
This
is
very
creative
right.
We
decided
to
split
our
largest
and
most
crucial
multi
easy
clusters
up
into
multiple
single
AZ
clusters.
A
To
summarize
the
last
year
our
multi
cluster
has
paid
off.
It
has
prevented
several
outages,
often
without
significant
user,
visible
impact.
We
now
have
additional
peace
of
mind
when
doing
cluster
upgrades,
config
changes
or
responding
to
incidents,
and
knowing
that
we
have
the
additional
capacity
to
fall
back
on,
has
been
a
great
reducer
of
stress.
If
something
does
go
wrong
with
the
cluster,
our
CDN
and
load
balancer
layer
take
the
unhealthy
cluster
out
of
rotation
and
the
rest
of
the
cluster
group
quickly
scales
up
to
handle
the
extra
traffic.
A
This
is
especially
helpful
when
Amazon,
temporary
temporarily
misplaced
is
in
AZ,
which
tends
to
happen
about
once
every
year
year
and
a
half
so
that
as
well.
The
other
nice
thing
has
been
some
cost
and
latency
savings,
from
not
being
as
sloppy
about
going
across
easy
boundaries
with
network
traffic
with
the
single
AZ
cluster
model,
it's
more
difficult
to
unintentionally
send
traffic
to
other
AZ's.
Today,
the
most
common
case
where
we
do
reach
across
a
Z's
is
for
databases
and
caches.
A
This
big
change
in
our
cluster
management
strategy
hasn't
been
without
some
challenges,
as
you
might
expect,
more
clusters
does
mean
more
administrative
overhead.
It
took
us
a
while
to
figure
out
the
best
way
to
manage
these
nearly
identical
clusters
and,
after
some
experimentation,
we
settled
on
splitting
are
interesting,
but
we
settled
on
templating
our
terraform
for
our
cluster
groups
pass
in
the
availability
zones
that
a
cluster
group
should
cover
and
the
code
generator
rates
out
the
corresponding
tariffs
one.
A
While
this
simplifies
the
process
of
provisioning
new
clusters,
we
do
now
have
to
maintain
more
clusters
to
minimize
the
impact
of
the
additional
overhead.
We
tend
to
only
use
the
cluster
group
pattern
in
environments
that
are
mission-critical.
Everything
else
still
stays
with
the
old
Mustard
multi
cluster
model.
A
If
we
were,
if
we
were
able
to
start
over
with
the
benefit
of
hindsight,
we'd
probably
shift
our
more
sensitive
environments
to
the
multi
to
the
cluster
group
pattern.
Sooner
doing
so
would
have
save
some
pain,
made
us
less
worried
about
making
changes
to
our
clusters
and
what
have
also
kept
our
individual
cluster
sizes
smaller,
as
we
gained
experience
with
operating
kubernetes
atlatl
at
a
higher
scale.
Today,
most
of
our
central
environments
do
use
the
cluster
group
model
for
environments
that
aren't
critical
again.
A
A
A
With
the
full
power
that
the
kubernetes
api
offers,
it's
easy
to
do
things
that
you
might
later
regret
so
skip
this
up
at
night
for
a
little
while,
but,
alas,
we
had
a
plan,
some
of
the
mistakes
that
are
made
in
production
can
be
as
innocuous
is
missing.
A
cost
allocation
tag
does
something
is
dangerous,
is
accidentally
exposing
a
service
to
the
entire
world?
I
don't
know.
Has
anybody?
Has
anybody
done
here?
Anybody
anybody
done
that
here,
yeah
it
happens.
Doesn't
it
that's
fun?
A
And
you
know
our
back
provides
some
safety,
but
it
really
it's
it's
limited
in
scope,
and
it's
not
the
general
purpose
policy
engine
that
we
were
looking
for
and
controlling
for
some
of
these
factors.
Fortunately,
open
policy
agent
already
existed
and
looked
to
be
just
the
thing
that
we
needed.
This
may
be
again.
This
may
be
a
little
bit
difficult
to
see,
but
here
is
an
example
policy
written
in
Rago,
the
domain-specific
language
that
opa
uses
to
define
policies.
A
If
you've
ever
heard,
horror
stories
about
somebody
accidentally
exposing
an
eternal
service
to
the
world
OPA
has
got
your
back.
This
particular
example
prevents
the
creation
of
kubernetes
services
of
type
load
balancer
that
haven't
been
whitelisted.
Opa
comes
with
a
library
of
other
example,
policies
that
cover
a
good
deal
of
ground,
and
it's
also
easy
to
offer
your
own
fast
40.
Today,
OPA
has
done
well
and
enforcing
a
small
and
growing
set
of
protective
policies
for
us.
A
As
far
as
what
didn't
go
well,
in
our
experience
with
OPA,
the
only
gotcha
has
been
with
kubernetes
versions.
Before
1.14
story
time,
we
had
a
kubernetes
1.10
development
cluster
that
saw
a
great
deal
of
pod
churn
during
peak
hours.
The
churn
put
quite
a
bit
of
stress
on
the
control
plane.
After
passing,
some
arbitrary
load
threshold
request
to
the
kubernetes
api
server
began,
slowing
down
and
piling
up.
The
logs
showed
that
the
api
service
request
to
the
OPA
admissions
hook
were
timing
out
and
the
end
result
was
a
pile
up
of
requests.
A
Kubernetes
api
server,
the
API
server
could
no
longer
service
most
in
down
requests
from
control,
plane,
components
or
the
workload.
The
cluster
scheduling
and
cleanup
of
pod
just
reached
to
a
halt.
Removing
the
open
missions,
controller
hook,
caused
things
to
clear
up
immediately.
The
good
news
is
that,
starting
with
kubernetes
one
at
14,
you
can
now
set
configurable
admission
hook,
admissions
hook,
timeouts
that
are
much
lower
than
the
previous
30
second
default.
It's
now
I
think
it's
10
seconds
in
1.14
and
beyond.
Also
the
OPA
folks
have
worked
to
optimize
the
agent
to
good
effect.
A
We
have
not
seen
this
failure
mode
in
our
1.14
clusters,
with
appropriate
timeout
set
with
newer
versions
of
the
OPA
agent.
As
far
as
what
we
do
differently,
we've
had
an
overall
positive
experience
with
OPA.
So
far,
our
death
spiral
issue
was
largely
due
to
a
specific
cluster
control
plane
being
under
sized
with
OPA
only
exacerbating
a
situation
that
was
already
not
great.
If
we
could
go
back
and
do
one
thing
differently,
we
probably
should
have
checked
out
how
the
agent
would
have
fared
on
a
highly
stressed
control
plane.
A
This
would
have
ultimately
led
us
to
assign
a
higher
priority
for
our
kubernetes
1.14
upgrade
so
that
we
could
get
those
configurable
admissions
hook.
Timeouts
today,
OPA
is
running
on
every
one
of
our
19
clusters
and
we
have
policies
in
place
for
most
of
the
things
that
keep
us
up
at
night.
The
other
thing
that's
been
really
handy
has
been
being
able
to
set
new
policies
to
something
called
monitor
mode
to
see
when
they
would
have
fired
in
enforce
mode.
A
This
is
especially
useful
when
you're
on
the
lead-up
to
bringing
incorporating
a
new
policy
or
even
in
ad-hoc
experiments.
Maybe
you
want
to
check
when
what
cases
am
I
using
the
latest
tag,
without
actually
taking
any
action
really
handy
and
now
for
everyone's
favorite
topic.
This
is
why
you
are
all
here
right,
yeah
makan,
when
it
comes
to
llamo,
our
story
is
not
unique.
In
late
2016,
we
were
just
beginning
to
experiment
with
kubernetes
we
started
our
journey
using
static,
manifests
and
get
wrapped
in
some
bash
scripts
to
push
things
to
our
cluster.
A
Anybody
doing
that.
Anybody
done
that
be
honest.
I
know,
you're
out
there
yeah
lots
of
they're
good.
At
some
point,
we
decided
we
needed
to
do
some
light
templating
does
anybody
know
where
this
is
going,
so
he
started
doing
some
some
obscene
things
was
said
and
bash
and
re
mo
scripts.
How
about
this?
You
know
what
done
the
old
said
in
amyl
trick:
okay,
there's
a
few
hands
yeah
very,
very
flippant
like
I,
guess
alright,
but
early
2017.
A
We
had
had
about
enough
of
the
bash
and
said
trick
and
began
thinking
about
how
he'd,
like
our
Manifest
story,
to
look
longer
term
in
the
spirit
of
not
asking
our
service
owners
to
become
kubernetes
experts,
it
was
important
that
we
provide
a
simple
and
minimal
and
interface
as
possible.
We
also
didn't
want
to
write
abstractions
on
top
of
ammo
without
having
first
gained
a
thorough
understanding
of
our
eventual
production
usage
cases.
After
evaluating
options,
we
were
attracted
to
helm
for
its
growing
uptake
around
the
community.
A
We
also
felt
like
we
could
present
just
the
values
override
files
to
our
service
owners
instead
of
the
whole
helm
chart.
This
could
serve
is
a
reasonable
user
interface
for
the
early
goings,
so
further
reduced
oil.
We
to
write
tooling,
to
generate
comp
charts
for
reddit
services
that
use
base
plate
our
internal
service
framework
when
it
came
time
to
deploy
a
service
spinnaker
would
pull
the
helm
chart.
The
value
overrides,
render
the
amyl
and
deploy
it
to
the
cluster
without
using
tiller.
A
The
good
news
is
that
this
arrangement
has
served
us
very
well
as
a
starting
point,
our
hypothesis,
that
we
could
mostly
limit
service
owner
contact
to
just
the
values
override
has
mostly
proven
proven.
True,
with
a
few
exceptions
related
to
how
we
structure
configs,
we
made
the
decision
to
auto
generate
charts
for
each
service
instead
of
having
them
all,
usually
central
mega
chart,
and
we
did
this
to
avoid
jumping
into
the
deep
end
of
template,
ID
amyl,
and
to
reduce
widespread
churn
and
breakage
as
we
evolved
our
patterns.
A
The
other
pleasant,
surprise
has
been
how
well
spinnaker
and
helm
have
worked
together.
Spinnaker
is
v2,
kubernetes
provider
doesn't
care
much
about
where
the
am
yam
well,
that
it's
deploying
comes
from,
which
makes
helm
easy
to
drop
in
spinnaker,
essentially
runs
the
helm,
template
sub
command,
then
pipes
that,
through
its
deploy
service,
we
currently
run
about
a
hundred
production,
deploys
a
day
through
spinnaker,
using
this
parent
and
now
for
the
less
rosy
stuff.
A
That's
we're
really
here
for
right,
while
our
helm
chart
generator
approach
worked
well
in
the
beginning,
we're
to
the
point
where
some
of
the
trade
offs
began
to
become
more
apparent,
more
problematic.
We
now
have
a
few
dozen
helmet
charts
that
are
mostly
the
same,
save
for
a
few
tweaks
and
it's
been
tough
keeping
all
of
our
charts
in
sync,
as
our
patterns
evolve
and
solidify,
our
eventual
goal
was
to
flesh
out
the
helmet,
our
generator
to
the
point
where
we
didn't
need
to
customize
the
output,
but
we
didn't
quite
make
it
there.
A
We
stopped
short
of
that
goal
in
favor
of
a
different
approach
more
on
that
shortly.
Please
stay
tuned
as
far
as
what
we
do
differently.
It's
easy
for
me
to
begin,
using
the
benefit
of
hindsight,
to
give
you
a
like
a
specific
prescriptive
answer,
but
honestly
I
think
we've
benefitted
from
going
from
static,
Hamill
template,
add
amble,
said
in-home
and
establishing
our
patterns
and
in
best
practices
along
the
way
we
used
an
existing
community
standard
in
helm
instead
of
reinventing
something
on
our
own.
Before
we
knew
what
we
were
doing.
A
However,
our
service
charts
had
many
possible
configurations
and
optional
pieces,
rather
than
continue
to
develop
and
maintain
large
complex
service
charts.
We
could
have
benefitted
from
making
our
transition
to
a
programmatic
amo
generator
sooner,
and
that
is
exactly
what
we
are
in
the
process
of
doing.
Instead
of
a
central
and
single
reddit
service
home
chart,
we
wrote
a
star
lark
powered
domain-specific
language
for
deploying
baseplate
services
at
reddit.
The
dialect
is
a
narrow
subset
of
Python,
which
is
convenient,
giving
our
orgs
Python
inclinations.
A
It
is
opinionated
and
only
exposes
parameters
for
the
things
that
tend
to
differ
between
reddit
services,
though
deeper
customization
is
when,
if
deeper
customization
is
needed,
the
DSL
offers
an
escape
hatch
with
which
you
can
provide
your
own
resources.
Another
important
note
is
that
we
do
not
attempt
to
conceal
kubernetes,
mechanics
and
terminology
from
the
developers.
Our
philosophy
has
been
to
compliment,
but
not
conceal,
kubernetes.
A
To
summarize
to
the
full
flow
we
passed
in
a
few
inputs
to
the
resource
generator,
which
then
generates
the
corresponding
animal
for
our
service
and
since
the
DSL
maps
directly
to
the
kubernetes
go
api
types
under
the
hood,
we
get
a
solid
type,
checking
and
validation
story,
while
authoring
the
starlike
files,
which
we
did
not
get
with
with
helm
and
by
shedding
deeply
templated
llamo.
For
the
specific
case
of
Reddit
authoring
services,
we
hope
to
simplify
the
development
and
maintenance
of
this
critical
piece
of
our
stack.
A
As
far
as
where
we
are
today,
we
have
now
battle.
We
now
have
a
battle-tested
library
of
patterns
and
best
practices.
We
encounter
far
fewer
firsts
first
times
these
days
and
baseplate.
Our
internal
service
framework
allows
us
to
make
some
assumptions
about
the
shape
of
our
services,
with
our
codified
best
practices
in
hand
or
writing
a
Starlog
powered
resource
generator
to
replace
home
charts
just
for
reddit
authored
services.
However,
we
wanted
the
resource
generator
fit
as
narrow
a
use
case
as
possible.
This
means
that
helm
is
still
used
for
everything,
but
reddit
authored
services.
A
Now,
with
everybody's
favorite
topic,
covered.
Let's
move
on
to
everyone's
second
favorite
topic:
it's
not
service,
meshes
local
development
environments
with
kubernetes
and
gratuitous
quantities
of
micro
services.
All
right
now
be
honest
with
me.
Is
anyone
here
fortunate
enough
to
be
super
happy
with
your
kubernetes
base?
Local
dev
environment
raise
your
hands
if
this
describes
you,
okay?
That
is
a
that's
a
good
answer.
I
am
not
really
some.
A
Maybe
like
one
hand,
why
should
I
go
out
and
high-five
you
wherever
you
were
I
just
saw
you
yeah,
probably,
but
you
know
I'm
not
here,
to
judge
but
I'll
be
honest
and
I'll
say
you
know
we
haven't
quite
made
it
to
that
promised
land.
Yet
it
read
it.
In
fact,
we've
we've
lift
a
little
bit.
We
went
into
2019
with
a
plan.
We
have
plan
for
everything.
A
A
We
also
needed
to
be
able
to
develop
multiple
macro
services
in
parallel,
multiple
related
micro
services
in
parallel,
and,
of
course,
all
of
this
was
useless
without
some
kind
of
service,
a
story
for
making
sure
service
dependencies
were
accounted
for,
but
we
were
a
little
bit
sketchy
on
on
what
that
would
look
like
going
into
20
team.
The
initial
plan
was
to
start
with
scaffold
and
local
mini
cube.
A
Maybe
some
of
you
have
heard
the
story
before
our
testing
showed
that
this
would
work
well
in
simple
cases
and
perhaps
it'd
be
a
good
foundation
to
start
just
just
to
start
it
and
to
start
building
on,
as
we
got
further
into
our
experimentation
with
scaffold,
a
mini
cube.
We
found
the
experience
to
be
very
pleasant
when
all
of
the
pieces
functioned.
At
the
same
time.
Mini
cube
is
an
easy
way
to
get
a
local
cluster
going
without
much
fuss
and
the
scaffold
build
loop
is
fast
and
solid.
A
However,
early
adopters
ran
into
all
kinds
of
breakage
and
flakiness
driving
up
our
support
load
in
frustrating
the
service
owners
mini
cube,
seemed
to
break
in
new
and
exciting
ways
with
each
release
and
scaffold
had
some
periodic
issues
as
well
yeah.
When
things
went
wrong,
the
users
were
frequently
unable
to
figure
things
out
without
blowing
away
configs
in
cluster
state.
That
was,
that
was
a
bit
of
a
bummer.
We
also
ran
into
a
common
class
of
issues
associated
with
multi
service
development.
How
do
we
stand
up
a
services
full
dependency
chain?
A
How
do
I
fit
multiple
resource?
Heavy
services
on
my
machine
scaffold
didn't
have
much
of
an
opinion
on
how
to
approach
multi
service
development.
So
most
of
this
exercise
was
what
was
left
up
to
us
and,
as
we
attempted
to
build
tooling
around
mini
Kuban
scaffold.
We
quickly
arrived
at
the
realization
that
we
had
probably
lift
with
this
overall
approach.
A
This
is
the
one
case
where,
if
I,
could
you
know
if
I
could
go
back
and
do
things
differently,
I
would
ask
for
a
dramatically
different
course.
Instead
of
attempting
to
build
a
local
development
environment
today,
I'd
focus
us
on
building
against
remote
clusters
instead
and
there
are
trade-offs
associated
with
developing
against
a
remote
cluster,
but
we
found
more
positives
than
negatives
for
our
our
own
users.
In
their
usage
cases,
the
user
doesn't
need
to
hassle
with
heavy
local
vm's
anymore.
A
A
A
When
tilts
and
your
service
up
that'll
already
have
all
of
its
dependencies
in
cluster
without
having
to
stand
up
the
whole
stack
for
each
developer.
If
you
want
to
swap
one
of
their
dependencies
out
to
another
branch,
this
is
possible.
This
will
be
possible
with
a
quick
config
update
and
the
result
of
all
this
is
that
we
end
up
with
a
code
development
loop
that
feels
similar
to
true
local
development.
A
Future
iterations
of
our
dev
environment
may
see
a
more
hybrid,
local
and
remote
flow
where
services
can
be
selectively
delegated
to
the
remote
cluster
instead
of
being
fully
remote
or
we
may
go
heavier
towards
mocking
out
dependencies.
It's
honestly
difficult
to
predict
at
this
point
and
thus
concludes
our
look
back
on
the
last
year
of
kübra
of
our
kubernetes
adoption
at
reddit.
Well,
there
were
certainly
some
struggles.
A
I
characterize
the
last
12
months
as
very
successful
service
owners
are
treaded,
are
now
empowered
to
own
their
services,
development
lifecycle
and
while
there
are
still
some
things
that
require
assistance
from
the
infrastructure
team,
those
group
grow
fewer
with
each
passing
month,
but
there
is
still
much
to
do.
Let's
take
a
look
at
some
of
the
big-ticket
items
on
our
list
for
the
next
year
and
beyond.
First
up
is
our
launch
process.
A
We
now
have
a
very
well-defined
repeatable
launch
process,
but
there
are
still
lots
of
manual
steps
and
some
in
fátima
assistance
is
still
required,
we'd
like
to
make
service
launches
as
automated
and
self-serve
as
possible
and
while
I
ran
out
of
time
to
cover
the
story
of
our
sto
adoption.
I'll
summarize
this
by
saying
we're
a
thrift
shop
trying
to
add
thrift
support
to
a
gr
PC
centric
mesh.
That
is
a
wonderful
time
if
there
are
other
any.
If
there
are
any
other
thrift
users
using
orgs
out
in
the
audience.
A
Please
come
see
us
we
would
love
to
collaborate
next
up
to
continue
to
work
on
our
dev
environment
is
a
top
priority
for
us.
We've
also
got
to
start
optimizing
more
for
cost
and
density.
These
have
been
lesser
concerns
for
us
in
the
early
goings,
but
we
feel
that
kubernetes
has
the
potential
to
crunch
our
fleet
down
quite
a
bit,
and
finally,
I
mentioned
our
budding
essary
organization
earlier
on,
it's
a
crucial
player
in
our
quest
to
scale,
infrared
and
kubernetes
out
to
a
rapidly
expanding
engineering
organization.
A
Last
of
all,
here's
some
information
about
me.
If
you'd
like
to
stay
in
touch,
though
I
am
the
lead
I'm
by
far
the
least
interesting
part
of
all
of
this
everything
covered
today
was
pondered
designed
and
implemented
by
a
great
collection
of
infrastructure
platform
and
security
engineers
who
many
of
whom
are
in
the
tenants,
my
reddit
colleagues,
please
stand
for
a
moment,
and,
let's
embarrass
you
once
again,
yes,
they're
over
here,
look
right
here.
A
And
here's
one
right
here
and
of
course,
if
you'd
like
to
join
the
team,
we
are
hiring
across
all
sorts
of
different
functions.
Check
out
our
listings
at
reddit.com,
slash
jobs,
I
am
going
to
go
on
ahead
and
step
down.
Our
booth
is
on
the
other
side
of
this
hall.
We've
got
about
30
minutes
before
the
the
the
exhibit
hall
closes.
Please
come
and
let's
talk,
I
will
handle
questions
over
there
rather
than
here
and
I.
Thank
you
very
much
for
coming
in
following
along
with
us
today,.