►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Implementing a Consumer Focused SLA for a Kubernetes Based PaaS - Shrenik Dedhia, Box
Box's (internal) Platform as a Service empowers other Box teams to deliver 100's of micro services, on 1000's of hosts, across 10,000's of pods. As they scaled to support a large number of micro services and clusters, they ran into several scaling challenges around both the control and data planes. In order to deliver a production-grade platform, they realized the need for a Service Level Agreement (SLA) for their platform to not only demonstrate availability for infrastructure, but also "value" for a consumer, and serve as a benchmark to prioritize those challenges. In this talk, Shrenik Dedhia will present how their team approached the problem of defining a SLA, principles used, options explored, path chosen, and future work to improve the platform's availability from ~99.4% to ~99.99%, thereby improving the overall availability of micro services that power Box.com.
https://sched.co/UaZZ
A
A
B
B
We
we
picked
it
up
right
away
and
we
and
we
implement
our
platform
on
top
of
it,
and
we
actually
g8
our
platform
internally
and
are
made
of
2016.
We
call
our
platform
sky
net,
which
is
a
reference
to
terminator.
We
wanted
a
platform
to
become
self-aware
at
some
point
and
through
the
years
we've
kind
of
scaled
ourselves
to
kind
of
where
we
are.
B
Today,
we
run
about
150
plus
services
in
in
production
today,
which
is
roughly
75
percent
of
boxes
kind
of
the
stateless
footprint
we
we
run
about
fluid
a
half
thousand
pods
of
hosts
with
1,000
in
the
indeed
largest
of
cluster,
mounts
to
roughly
25%
of
boxes,
compute
kind
of
the
the
datacenter
footprint
we
have
about
nine
production,
grader
clusters
across
of
three
regions,
they're
all
in
in
inactive
bare
metals.
We
have
about
25,000
pods
across
all
of
these
or
clusters,
with
of
9,000
pods
in
our
largest
cluster.
B
Our
platform
has
minimal
abstractions
on
top
of
a
kubernetes.
So,
as
box
teams,
you
know
kind
of
on
board.
On
top
of
our
platform,
they
have
a
bit
of
a
learning
curve
to
learn
about
kind
of
a
kubernetes
api
and
stuff
from
when
they
on
board
we
use
drops.
We
have
a
declarative
config
it
we
use
of
JSON
it
to
to
express
that
of
configuration
and
to
also
compile
that
into
into
cube,
manifests
that
ultimately
get
applied
to
our
our
our
clusters.
B
Yep
other
things
we
have
in
our
platform
are
cross-cutting
concerns.
For
example,
we
have
a,
we
have
a
we
have
a.
We
have
an
internal
CA
that
that
of
delivers
PKI
to
the
pods.
We
use
hashey
core
vault
to
deliver
secrets.
We
use
of
calico
for
network
management
and
for
IPM
we
use
smart
stack
for
service
discovery,
we're
actually
at
a
point
where
we're
pushing
the
scale
of
limits
for
smart
stack
and
we're
looking
into
replacing
it.
B
Back
in
2017
or
2018
timeframe.
We
actually
went
through
a
bunch
of
problems
and
and
scalability
challenges
in
our
platform
and,
as
we
were,
implementing
fixes
for
our
platform
for
those
of
stability
problems,
we
had.
We
had
two
kind
of
thoughts.
We
didn't
know
how
much
the
improvements
were
being
made
by
those
fixes
and
we
didn't
know
how
how
to
prioritize
those
fixes,
because
we
didn't
know
what
the
the
the
the
bar
built
run
here
at,
and
so
we
were
kind
of
in
this
mode
of
hey.
B
B
Do
we
even
support
things
like
saturation
with
Rupert
traffic
etc?
What
is
the
service
level
objective?
Is
there
three
nines
two
and
a
half
I
mean
two
and
a
half
nines
four
nines?
How
do
we
measure
these
service
level
indicators
further
on
was
the
breadth
of
the
off
of
this
problem.
Do
we
measure
our
control
plane?
Do
we
measure
our
data
plane
both
or
something
else?
Do
we
go?
B
B
The
the
first
solution
we
kind
of
thought
of
was
hey.
Let's
just
go
ahead
and
count
the
number
of
minutes
that
we've
regretted
our
voxcom
side
that
this
was
very
course
and
it
it
did
not
account
for
a
large
number
of
other
problems
that
were
happening
within
our
platform.
Basically,
not
all
paths,
not
all
paths,
lag
relations.
What
would
cause
a
good
cause
of
box.com
who
tied
relations
so
we've
we've
been
one
step
ahead.
B
There
were
some
problems
with
this
as
well.
It
did
not
account
for
other
for
other
types
of
errors
that
happen.
While
she
showed
up
in
in
the
API
server
logs
like
for
hundreds
it,
it
did
not
account
for
for
like
scheduler
issues,
it
did
not
account
for
controller
manager
issues.
In
effect,
it
did
not
account
for
the
data
plane,
and
so
we
went
ahead
and
implemented
a
synthetic
test
to
represent
our
data
plane,
availability.
B
The
this
synthetic
tests
were
to
run
every
10
15
seconds,
and
it
would
run
on
a
subset
of
hosts
and
it
would
test
a
subset
of
the
of
the
features
for
the
lifecycle
of
the
part.
To
kind
of
see
when
a
you
know,
kind
of
POD
we
created
is
a
part
of
receiving
the
assets
that
it
needs,
for
example,
of
PKI
and
secrets
et
cetera.
B
Then
again,
this
wasn't
real.
It
wasn't
representative
of
of
our
customers,
it
was
it
was
it
was.
It
was
a
synthetic
and
so
at
this
point
we
kind
of
step
back
and
kind
of
ask
what
are
what
are
the
questions
we
want
to
believe
focus
on
from
oral,
the
conundrum
and
and
two
questions
really
stood
out.
One
was
what
were:
customers
really
care
about
right,
and
the
second
question
was:
how
do
we
leverage
kubernetes
itself,
to
build
build
an
SLA
for
a
platform.
B
Customers
normally
care
about
more
about
about
the
ease
of
use
of
the
of
the
platform
itself.
They
care
about
keeping
their
own
service
up
and
along
and
running
at
at
all
times,
and
they
care
about
how
to
serve
a
hundred
percent
of
the
peak
traffic
at
all
times.
They
don't
necessarily
care
about
the
control
plane
availability.
If
our
control
plane
dies,
they
don't
care,
they
shouldn't
even
come
to
know.
B
A
lot
of
our
customers
have
begun,
telling
us
that
they
actually
don't
care
about
the
kubernetes
api
itself.
Remember
I
told
you
that
our
platform
is
has
very
few
abstractions
and
their
customers
have
a
learning
curve
to
understand
with
kubernetes
people
started
telling
us
that
I
should
don't
want
to
learn
about
kubernetes
like
up
a
whole
lot.
B
B
Pdb
is
more
more
recent
concept.
It
protects
service
from
from
administrative
disruptions
and
when
there's
another
kind
of
it,
isn't
it
isn't
a
feature,
but
it's
a
method
of
how
you
configure
quality
the
the
the
the
quality
of
service
for
your
part
by
by
specifying
the
resource
from
requests
and
limits,
author
containers-
and
it's
you
know
you-
can
you
can
set
your
pots
to
guaranteed
vegetable
or
best
effort.
B
We
choose
guaranteed
the
principle
behind
that
and
actual
guarantee
means
to
set
your
part
of
requests
at
these
at
the
same
level
as
your
part
of
limits
on
a
container
request
and
and
limits.
So
you
don't
give
more
CPU
to
your
limits
than
your
requests.
The
the
principle
behind
this
is.
When
we
looked
at
our
traffic
or
peak
traffic,
we
notice
that
all
of
our
services
were
actually
peak
during
peak
traffic
and
if,
if
all
services
would
attempt
to
burst
and
use
more
CPU,
we
would
cause
a
lot
of
noisy
neighbor
problem.
B
So
our
recommendation
to
our
service
owners,
wars,
use
of
guaranteed
mode,
and,
if
you,
if
you,
if
you
want
to
scale
for
peak
traffic,
then
they're
like
scale
horizontally,
you
know
create
more
replicas
and
and
not
increase
of
CPU
life
memory
on
each
of
your
each
of
each
each
of
your
containers
and
there
might
have
been
other
kind
of
jobs
which
we
didn't
explore
at
the
time
realize
this
was
all
most
of
you
are
here
and
a
half
ago
that
we
kind
of
export
this
and
we're
actually
running
a
version
of
kubernetes.
That's
bad!
B
B
B
Conceptually
our
customers
are
our
tenants.
Our
service
owners
would
define
a
minimum
healthy
number
of
critical
replicas
that
would
be
needed
to
for
the
service
to
be
able
to
serve
a
hundred
percent
off
of
traffic
at
peak
I've
left
out
details
about
how
this
is
actually
configured
on
a
per
cluster
or
a
per
data
center
or
a
poorer
zone
of
region
level.
B
Taking
an
example
of
that
realized
that
the
values
that
I'm
going
to
go
through
here
are
very
suggestive.
The
method
is
also
very
suggestive
from
this
might
not
be
exactly
what
we
do,
because
we're
constantly
changing
and
evolving,
but
I
wanted
to
show
this
to
actually
serve
as
a
basis
for
how
you
would
think
about
building
your
own
SLA.
B
So
a
customer
which
is
our
service
owner.
We
have
to
define
their
critical
replica
threshold,
let's
say
hundred
replicas.
It
would
go
ahead
and
configure
their
there
liveness
and
readiness
probes
to
ensure
that
they,
you
know
they
account
for
their
own
dependencies
and
also
they
would
use
the
guaranteed
quality
of
service.
They
would
ensure
that
the
request
for
limits
match
their
request.
My
term
limits
on
on
the
only
other
resources,
and
then
the
platform
would
come
in
and
and
and
and
begin
adding
protections
on
top
of
that
in
a
layered
fashion.
B
B
B
On
top
of
that,
we
added
a
I
added
some
sort
of
rolling
update
buffer,
hey
at
added
anytime
there's
a
hardware
failure
and
there's
a
disruption
going
on
and
and
there's
an
administrative
disruption
going
on.
I
still
should
be
able
to
do
rolling
updates
so
added
that
buffer
here
and
then
finally,
another
buffer
for
maintenance.
For
example.
A
you
know,
I
want
to
be
able
to
lose
a
whole
rack
or
for
like
maintenance,
and
when
there
could
be
many
more
levels.
B
Right,
how
do
we
measure
the
SLI?
How
do
we
really
measure
this?
So
we
take
the
minimum
of
replicas
we
required
for
H
a
for
every
service,
and
then
we
calculate
the
available
replicas
for
every
service
over
comparing
the
total
available
with
the
with
the
CRT.
In
our
example,
the
total
available
was
Kernan
was
was
101
117
and
his
CRT
was
100,
and
then
we
calculated
these
replicas
availability,
which
is
a
percentage
of
of
the
CRT.
B
So
as
long
as
as
that
service
had
more
than
100
parts
alive
at
any
given
time
between
100
and
an
hundred
seventeen,
we
would
consider
that
or
service
to
be
hundred
percent
available
and
the
moment
that
that
service
would
actually
start
losing
parts
below
hundred.
We
would
start
accounting
for
that
in
our
in
our
CSA
and
finally,
we
would
take
an
aggregate
could
take
a
simple
average
across
all
the
across
the
availability
for
all
the
other
all
services
across
a
AVO
who
give
like
given
arbitrary
period
of
time.
B
B
B
B
Some
people
might
think
this
is
a
feature
we
should
treat
everybody
equal.
This
is
a
platform
and
other
people
might
think
this
is
a
bug
we
should
have
some
monitoring.
We
think
it's
a
bug,
because
we
do
feel
that
we
have
a
class
of
services
that
are
more
or
less
in
the
critical
path
of
customer
traffic
and
so
like
there
is
an
actual
reason
to
create
tiers
of
services.
So
that's
something
we
would
want
to
fix
in
the
future.
Maybe
we
do
that
using
grade
averages.
I,
don't
know
we
haven't
explored
that.
B
But
the
platform
actually
has
the
problems
underneath
some
of
the
for
some
future
work
that
we
want
to
do.
We
would
like
to
leverage
new
features
of
kubernetes,
for
example,
odd
priority
and
preemption
realize
we
like.
We
are
our
central
point
of
CRA
is
that
we
want
to.
We
want
the
platform
to
at
least
maintain
the
the
critical
number
of
replicas
at
all
times.
So
let's
say
we
have
situation
where
we
lose
25%
of
our
of
our
of
our
capacity,
and
we
have
a
hundred
services
from
which
99
services
are
healthy.
B
They
have
all
of
their
critical
replicas
up
and
alive,
but
there's
one
service
which
which
which
which
which
has
lost
half
of
its
parts.
We
don't
have
a
mechanism
that
will
go
and
kill
the
healthy
parts
from
all
the
99
services,
so
that
we
can
make
room
for
the
critical
replicas
of
this
one
service
which,
which
is
unhealthy,
and
so
we
need
a
feature
from
from
kubernetes.
Where
we
can.
B
B
We
want
to
introduce
more
more
chaos
in
in
our
system
when
we
built
our
platform,
a
large
number
of
tenants
actually
moved
over
from
the
from
the
bare
metal
world
over
into
the
kubernetes
from
platform,
and
they
kind
of
brought
along
with
them
some
anti-patterns
and
some
like
bad
architectures
when
you're
in
a
barrel
you're,
not
necessarily
thinking
about,
if
my,
if
my
machine
dies
at
at
it.
At
any
point
of
time,
how
do
I
recover
how
do
I
convert
across
my
kind
of
healthy,
healthy
instances
and,
and
and
and
and
so
forth?
B
Well
deployments-
are
really
controlled
and
and
and
tightly
orchestrated.
So
a
lot
of
the
services
that
we
are
running
today
aren't
actually
fully
like
completely
built
for
hard
shutdowns,
and
so
we
want
to
and
and
really
if,
if,
if
hard
shutdowns
happen,
those
services
take
a
long
time
to
kind
of
recover
and
and
and
and
and
become
healthy
again,
and
so
we
want
to
like
introduce
more
more
chaos
so
that
we
can
help
those
services
improve
that
improve
their
architecture
in
order
to
ultimately
improve
our
RCRA.
B
And
lastly,
we
want
to
have
kind
of
a
stronger
service
on
our
accountability.
We
want
to
implement
some
system
where
all
services
on
a
platform
get
out
of
out
of
box
alerts
so
that
if
their
pods
start
dying,
they
get
paged
and
like
they
need
to
come
back
and
fix
those
or
like
implementing
some
sort
of
a
policy
where
you
know.
B
Developers
can't
access
a
prod
yeah,
we
use
I
mean
our
developers
actually
can
access
prod,
but
they
can
access
prod
through
through
through
specific
controls.
We
have
a
we
have
it.
We
have
jump
machines
bastions,
which
have
you
know
appropriate,
ackles
kind
of
applied
to
them
that
enable
them
to
access
access,
prod
and
they're.
All
like
that's
all
like.
That's
all
in
that's
the
infrastructure
level
access,
and
on
top
of
that
we
have
kubernetes
our
back
and
and
we
use
a
bandwidth,
who's
LDAP
for
authentication
as
well.
B
B
B
B
We
use
we've
built
a
closure
in
like
a
like
our
we
use
JSON
to
actually
compile
their
configuration
and
produce
community
use
of
manifests.
We've
included
a
we've
built
a
closure
in
there
which
actually
manipulates
kubernetes
of
manifests
before
they
are
written
back
into
into
who
get
and
that's
how
we
kind
of
we.
We
expect
the
users
to
order
service
owners
to
express
their
critical
replicas.
B
Yes
back
to
the
same
report,
exactly
so
the
sort
of
the
both
the
the
plain
tent
and
the
and
the
full
manifest
actually
goes
in
those
in
the
same
repository.
So
if
service
order
makes
a
change
with
composite
locally,
there
are
producers
of
set
of
like
changes
in
the
in
the
in
the
release
folder
and
that
gets
commuted
as
one
block.
E
B
B
So
if
you
have
a
system
which
which
which
has
all
of
these
integrations,
which
are
on
the
cob,
which
are
also
key
kind
of
like
a
native,
maybe
then
it'll
be
much
easier
to
kind
of
faint
lab
like
find
out
if
there
are
controls
which
we
can
use
to
automatically
protect
services
from
doing
the
right
thing.
The
other
way
to
look
at
this
is
that,
because
we
have
a
deathly
rate
of
system,
we
can
actually
go
ahead
and
build
those
things
out
even
for
those
integrations,
even
though
even
if
they
aren't
work
a
native.
B
B
The
second
answer
to
your
problem
is
that
this
is
not
a
silver
bullet
use
rightly
pointed
out.
This
is
not
the
only
metric
that
we
want
to
like
of
track
and
and
and
and
think
about.
There
are
lots
of
other
metrics
that
we
use,
for
example.
How
fast
can
how
fast
can
a
change
is
really
make
it
into
production?
B
G
B
On
the
other
hand,
we
do
have
a
separate
capacity
engineering
team
which
actually
monitors
the
both
the
capacity
and
the
energy
utilization
of
the
service
to
get
a
sense
of
where
the
service
is
at
in
terms
of
its
its
its
its
head
room
and
that's
that's
one
way.
We
kind
of
use
to
kind
of
kind
of
assess.
What's
the
right
range,
therefore,
the
folio
replicas
part
to
offer
answer
is
that
we
provided
a.
We
actually
ran
this
program
as
an
opt-in
program.
We
didn't
force
everyone
to
implement
this
right
away.
B
We
implemented
the
controls
for
that
for
this,
and
we
we
expressed
a
label
that
one
could
apply
to
their
deployment.
If
we,
if
we
found
the
label,
it
would
be
treated
as
an
opt-in
and
we
would
actually
start
admitting
those
like
those
those
protections
and
if,
if,
if
a
service
who
didn't
have
the
label,
you
actually
would
emit
of
warnings,
alluding
our
build
that
hey,
you
should
start
opting
in
into
this
program
and
figured
out
the
replicas.
You
need
yeah
kind
of
answer.
Your
question.
H
G
H
B
Most
of
our
so
I
think
I
think
there
were
two
questions
there
ask
your
first
question:
how
do
we
provide
guidance
to
what
the
right
thing
to
do?
Is
there
now
we
actually
have
an
internal
wiki
where
we
will
be
produced
all
of
our
guidelines
and
and
guidances,
and
all
that
you
actually
have
a
full
page
about.
B
Do
something
with
in-memory
checks
that
you
can
quickly
respond
back.
We
have
a
full
set
and
we
and-
and
we
point
all
of
our
service
orders
to
that
page
and
we
constantly
kind
of
improve
it.
We
also
have
a
lot
of
links
in
there
that
point
to
upstream
best
practice
and
and
the
kind
of
sources
that's
the
way
we
kind
of
actually
mitigate
that
we
don't
have
a
formal
way
of
actually
going
into
all
all
of
her
services
and
actually
verifying
whether
they
are
actually
using
it
the
right
way.
B
What
do
our
customers
actually
think
about
this?
That's
an
it's
an
interesting
question.
I
think
it
partly
answers
someone
someone
else's
questions.
We
haven't
fully
completed
this
program,
yet
we
think
about
I,
don't
know.
Maybe
10
to
20
percent
of
our
customers
have
actually
opted
into
this.
So
we
haven't
raised
that
that
that
that
that
critical
mass
where
we
are
you
know
where
we
had
that
that
kind
of
feedback
from
them.
Yet
we
hope
to
finish
this
program
by
the
end
of
next
year,
we're
an
enterprise
company.