►
Description
2nd Day Operations: Your Checklist for Cloud Native World
According to a June 2016 report from the Cloud Foundry Foundation, 16% of enterprises are
already using containers in production, and another 64% are looking to move into production
with containers in the coming year. This massive shift is fueling enterprises to architect and build their next generation applications on several core cloud native principles, including microservices in containers with open source frameworks for analytics and big data.
But building cloud native applications is only half of the challenge. It isn’t until you start running them at scale, dealing with outages and making upgrades that you see the full picture. These are the challenges of 2nd Day Operations, challenges that you do not want to actually bump into for the first time on day two of a production rollout. Navigating your way through a cloud native journey means making sure you’ve considered your 2nd Day Operations plan from the very beginning.
In this talk, Mesosphere’s Ben Hindman will share his 2nd Day Operations checklist for cloud native apps. These are the critical challenges you’ll want to be prepared for, and the surprising realizations that you do not want to be surprised by. He’ll also share his advice from years as a leader in the cloud native movement, working on the front lines with 2nd day operators around the world.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
All
right
and
everyone
hear
me
great
I've-
got
a
bit
of
a
cold,
so
so
we're
in
the
jack.
It's
freezing
in
here
I.
Think
other
people
probably
recognize
that
too.
So
it's
great
great
I
got
about
15
minutes
with
you
and
what
I
wanted
to
talk
about
was
day
to
ops
and
really
what
day
to
ops
means
in
this
new
cloud
native
world
and
in
particular
the
challenges
that
people
are
experiencing
as
they're
moving
into
the
cloud
native
world
when
it
comes
to
actually
operating
the
things
that
they're
trying
to
run.
A
So
you
know
day
to
ops,
we
sometimes
we
joke
about
it
back
at
Mesa
sphere.
There's
no
such
thing
as
day
to
ops,
because
when
you
first
start
to
try
to
deploy
some
of
your
software,
that's
at
the
zeroth
day
by
the
tenth
day,
you're
way,
past
day,
two
and
you're
still
trying
to
get
your
software
deployed
by
the
hundredth
day,
you're
still
you're
way,
past
day.
Two
then
you're
still
trying
to
get
your
software
deployed
by
the
time
you
actually
get
it
deployed.
You
decided,
there's
a
better
technology,
you're
actually
going
to
run.
A
So
you
start
that
deployment
instead
never
actually
get
to
operate
in
the
first
thing
that
you
actually
started
with,
but
no
day
to
ops
definitely
exists.
So
you
know
last
decades
day
to
ops
challenges.
I
think
we're
something
around
software
evolution
I'm
going
to
go
through
a
couple
of
these
just
to
talk
about
what
I
mean
when
I,
when
I'm
referring
to
day
to
ops.
So
you
know
last
deck
is
the
day
to
operations.
Look
like
this
developer
got
their
application
running.
A
Then
they
went
to
the
off
steam
and
they
said:
hey
ops,
I've
got
this
new
library
that
I
started
using.
Can
you
get
it
installed
for
me
and
stuffs
like
okay,
great
or
is
like
hey
ops,
I?
Just
in
this,
the
latest
version
of
the
service
that
I'm
running
I
got
a
new
networking
service.
Can
you
open
up
the
port
for
me
all
right?
So
this
is
a
conversation
that
was
going
back
and
forth
between
devs
and
ops,
and
so
lots
of
great
technologies
came
out
to
really
really
automate
this
to
make
this
move
faster.
A
A
So
more
recently
with
technologies
like
docker.
We
get
this
nice
software
evolution
because
the
operator
turning
out
to
care
about
it,
the
developers
get
to
own
the
software
evolution
themselves.
They
get
to
say
all
right.
You
know
all
install
my
own
libraries
and
I'll
configure
my
own
courts
within
my
docker
container
and
and
the
ops
just
don't
even
care
now,
some
up
folks
folks
out
there
might
say
whoa,
especially
the
security
op
folks
I,
don't
know
how.
How
could
I
feel
about
that?
I
might
want
to
be
in
control
of
what
software
actually
gets
installed.
A
Like
may
sews
DCOs,
kuber
netis
what
CC
and
CF
is
really
trying
to
push
it's
because
you're
seeing
stuff
like
this
you're
seeing
people
actually
use
some
of
these
older
DevOps,
like
tools
with
with
with
containers
to
actually
actually
run
run
their
apps,
so
they're
getting
some
of
the
best
of
both
worlds,
but
they're
still
not
there
just
yet
right.
So
next,
iteration
last
year's
day
to
day
to
day
to
ops
challenges
was
all
right
failures.
So
now
you
know
we're
using
the
puppet
docker
combination.
A
You
know,
as
a
dev
grew
up
to
my
my
operator.
Say:
hey
these
machines
just
died.
Can
you
get
me
running
on
some
other
machines
right
or
elasticity?
I
say
you
know:
hey
I'd
like
to
actually
scale
up
my
app.
Can
you
go
and
provision
those
machines
and
and
get
them
running
for
me,
whatever
whatever
I
need
to
do,
and
this
is
where
this
is.
A
So
now
again,
you're
kind
of
in
this
DevOps
world,
where
you
have
someone
saying
oh
well,
I'll
just
packaged
up
my
app
in
a
container
and
dcs
will
be
responsible
for
rescheduling
my
app
when
the
machines
die
and
if
I
need
to
scale
it
up,
I'll
just
go
use
the
DC,
USC
lie
and
I'll
scale,
my
app
up,
and
the
same
applies
to
Gruber
notice
as
well,
right
and-
and
you
know,
I
think
simultaneously.
While
this
was
happening,
developers
also
need
to
say
what
I
probably
probably
need
to
change.
A
The
architecture
of
my
applications
is
I.
Can't
just
you
know,
rely
on
this
wonderful
magic,
pixie
dust
of
container
orchestration
to
do
the
right
thing
to
do
with
failures
and
to
do
things
elastically,
unless
my
applications
can
actually
work
in
in
those
environments.
So
to
me,
what
cloud
native
actually
really
means
is
I've
changed
my
applications
to
be
highly
available
and
fault
tolerance,
and
now
they
can
actually
operate
in
these
environments,
which
are
not,
which
are
going
to
be
much
bored
more
dynamic
and
controlled
in
a
completely
automated
way.
A
A
These
are
folks
that
are
adopting
container
orchestration
technologies
technologies
like
urban
areas,
technologies
like
T
cos
they
as
an
organization,
need
to
take
their
applications
and
make
them
operate
in
these.
These
you
know
with
these
these,
these
new
challenges,
so
I
kind
of
want
to
talk
to
them.
So
first
service
discovery.
A
We
might
have
the
application
moving
all
over
the
place.
Dns
probably
doesn't
work
because
someone's
still
running
java,
6
and
java
6
caches
dns
resolutions.
So
even
after
you,
you
know
the
thing
moves
to
another
machine.
We
don't
resolve
the
new
IP
for
the
connection
and
we
try
to
connect
the
old
machine
so
that
doesn't
fail.
There's
a
lot
happening
in
the
networking
space,
but
people
need
to
start
thinking
about
what's
the
best
ways
to
actually
solve
the
service
discovery.
Problem
next
is
load
balancing
you
know.
Maybe
in
the
past
it
was
pretty
easy.
A
I
just
fixed
some
external
load
balancers
to
speak
to
these
particular
applications.
Now
those
applications
are
being
launched
very
dynamically.
They
might
be
scaled
up
in
scaled-down
dynamically.
You
need
to
go
to
update
the
load
balancers
to
actually
get
those
load.
Balancers
who
communicate
to
the
right
place
is
compounded
with
a
service
discovery
problem.
You
have
to
think
about
how
you
actually
want
to
solve
that
problem.
What
I
think
is
especially
interesting
from
the
load
balancing
case
and
it'll.
Get
to
some
slides.
I
talked
about
a
little
bit
later.
A
A
lot
of
people
feel
like
networking
is
a
long
ways
away
for
us
being
able
to
solve
as
an
OPS
challenge
in
container
orchestration.
I
think,
there's
a
lot
of
truth.
There
I
also
think
that
there's
been
a
lot
of
progress
recently
I
think
we
can
adopt
a
lot
of
the
technologies
that
came
out
of
the
infrastructure
service
realm
things
with
OpenStack,
but
the
two
big
things
that
really
jump
out
to
me
that
are
going
to
be
day.
A
Two
ops
challenges
for
networking
number
one
is
going
to
be
the
logical
isolation
who
can
see
who
who
can
Doss?
Who
who
can
ping?
Who
will
have
people
call
segmentation
and
then
the
other
one
is
performance,
isolation?
How
many
resources
can
you
actually
get
on
the
network,
both
on
a
single
note,
as
well
as
in
aggregate
okay?
Storage
volumes?
Another
big
one
kind
of
speaks
for
itself,
but
this
is
both
local
storage,
as
well
as
a
remote
storage
security.
A
What's
the
right
way
to
do
security
now,
you
need
to
actually
check
the
vulnerabilities
whether
or
not
there's
vulnerability
is
inside
the
containers.
You're
trying
to
run
you
need
to
have
some
notion
of
authentication
who
are
the
people
that
are
trying
to
actually
do
things
in
these
environments.
It
was
a
lot
easier
when
the
Ops
guys
just
did
everything
and
the
developers
just
talked
to
the
Ops
guys,
because
they
always
had
security
that
they
always
had
the
authentication.
A
They
could
do
what
they
needed
to
do
secrets
so
now,
I
can't
just
install
secrets
on
particular
machines.
I
have
to
figure
out
how
to
get
my
secrets
from
my
credentials
to
some
arbitrary
generic
machines.
Health
now
that
I
might
want
to
move
services
around.
It
would
be
really
valuable
to
under
special,
if
I'm
doing,
load,
balancing,
if
you're,
really
valuable
to
understand
what
the
health
of
these
applications
are
metrics.
A
What
do
my
metrics
look
like
how
do
I
collect
my
metrics,
especially
as
containers
are
moving
all
around
logs
I
can't
rely
on
a
developer
having
to
SS
be
able
to
ssh
a
new
machine
to
just
read
their
logs,
that
machine
might
have
been
completely
wiped
or
other
jobs
might
have
started
running,
and
so
we've
had
to
delete
the
logs
to
make
space
and
kind
of
just
this
general
idea
of
how
people
actually
do
bug.
Can
they
just
SSH
into
machines
and
do
things
or
not?
A
What's
the
best
thing
to
actually
do
there,
so
you
know
what
I
think
is
is
great
about
where
we
really
are,
as
an
industry.
Today
is
well
for
a
lot
of
people
that
are
starting
this
journey
in
cloud
native
and
in
a
cluster
management
and
container
registration.
A
lot
of
this
has
been
focused
on
by
these
technologies.
I
know,
I,
put
checks
in
there
and
I'm
really
calling
them
yesterday's
day
to
ops
challenges.
We're
not
completely
done.
We've
made
a
ton
of
progress,
and
that's
actually
really
good.
A
So
in
the
same
way
that
there
were
challenges
with
just
failures
would
say
the
puppet
and
just
docker
world.
Now
we've
got
container
registration.
We
have
other
problems.
The
industry
is
really
moving
forward
and
addressing
these
problems
head-on.
So
we
know
one
of
the
reasons
why
at
Mesa
sphere,
we
actually
built
the
data
center
operating
system
or
DCOs
is
precisely
to
address
these
challenges
to
precisely
to
address
all
these
things
that
you
really
need
to
do.
When
you
try
to
move
to
a
container
sorry,
a
cloud
native
world
also
can
contain
our
new
world.
A
Okay,
so
one
just
real
quick
thought:
I
want
to
throw
in
there
for
some
of
these
things,
one
of
the
things
wear
many
hats,
and
sometimes
I'm
wearing
my
apache
software
foundation
hat
and
I
started
the
Apache
mesas
project
and
still
the
PMC
chair
so
I'm
thinking
about
it
patchy,
sometimes
I'm
wearing
my
cnc
f
toc
hat
I'm
on
the
CNC
ftse.
Sometimes
I'm
just
wearing
my
mesa
sphere
hat.
A
But
one
of
the
things
that
I
wanted
to
mention
at
cloud
native
day
is
something
that
I'm,
especially
excited
about,
for
the
cns
you
have
to
do
is
really
help
to
define
what
is
the
future
politics
for
cloud
native
we've
all
kind
of
thrown
out
there
about
different
projects?
Oh
where
the
politics
where
the
API,
where
the
API
I
think
things
like
can
contain
our
native
sorry
container.
Networking
interfaces
is
a
great
start
to
start
to
define
what
some
of
those
api's
are.
A
You
know
I'd
thrown
up
to
a
container
volume
interface
before
I
saw
Brandon's
talk
earlier,
where
he
had
container
storage
interface.
So
that's
great,
but
also
things
like
a
container
health
interface,
where
it
would
be
great
if
every
single
container
that
got
deployed
regardless
of
the
system
that
was
actually
running
it,
it
could
be
able
to
determine
the
health
of
that
container
and
our
inner
in
a
very
deterministic
and
standard
way.
Okay.
A
So
what
are
going
to
be
next
year's
day
to
ops
challenge-
and
you
know
these
are
things
that
I
think
are
especially
important
as
people
start
to
get
through
yesterday's
day
to
ops
challenges
and
are
really
thinking
about
how
they're
trying
to
run
run
their
their
their
companies
on
this
technology.
So
a
big
one
for
me
is
is,
and
for
me
and
Mesa
spheres
is
bare
metal
storage.
A
There
are
a
lot
of
folks
that
want
to
be
able
to
run
their
applications
that
require
bare
metal
storage,
whether
it's
for
performance
or
whether
because
there
they
are
the
ones
that
are
actually
providing
the
distributed
file
systems
to
other
people.
Okay,
so
this
one's
this
one's
really
key.
Otherwise
it's
going
to
be
someone
else's
problems.
A
This
one
is
near
and
dear
to
my
heart,
these
these
these
pieces
of
software,
absolutely
fantastic
until
you
get
paged
at
2am,
because
you've
run
out
of
resources
and
the
less
protein
or
the
least
priority
intern
job
is
taking
up
all
the
resources
and
the
high
priority
job
can't
get
scheduled
by
the
system.
This
is
really
really
critical
stuff
and
it's
one
of
the
reasons
why
we
built
in
from
an
early
day
the
concepts
of
quotas
and
reservations
into
the
Apache
Mayo's
project
maintenance.
How
do
you
do
maintenance?
A
A
I'm
kind
of
talk
about
this
one
and
in
just
a
sec,
but
a
lot
of
people
are
trying
to
figure
out
the
best
way
to
do
this,
and
it
really
needs
to
be
addressed
with
first
class
primitives,
not
with
sort
of
some
hacks
and
some
glue
being
put
in
places
to
actually
make
this
work,
auto
scaling
and
auto
provisioning.
Well,
now,
we've
gotten
elasticity,
which
is
great.
A
We
can
just
go
to
our
D,
cos
command
line
and
scale
something
up
or
cube,
control
and
scale
something
up,
but
what's
actually
the
right
way
to
scale
things
up?
What
are
the
right
metrics?
We
need
to
pull
in
to
be
thinking
about
scaling
up.
Maybe
the
amount
of
resources
you're
using
is
not
the
best
metric
and
then
utilization,
and
this
one's
also
really
really
near
and
dear
to
my
heart.
Utilization
is
about
actually
dry
at
the
end
of
the
day.
A
One
of
the
big
promises
that
we're
making
with
these
technologies
is
that
we
will
drive
up
the
utilization
of
your
underlying
infrastructure,
but
to
do
that
in
many
circumstances,
we
have
to
do
really
really
smart
things.
We
can't
just
rely
on
you
launched
a
container
that
says
it's
using
four
cpus
and
four
gigs
of
ram.
If
it's
not
using
those
resources,
we
have
to
do
something
smart.
A
These
are
going
to
be
the
day
to
operations,
challenges
that
people
who
start
to
really
really
think
about
as
they
roll
these
out
to
production
in
really
serious
ways,
so
whoops
wrong
way.
So
just
a
couple
more
slides,
you
know
security
and
resource
guarantees
this
to
us
at
mesosphere.
This
sums
up
a
big
part
of
multi-tenancy
when
we
think
about
multi-tenancy
and
what
we
need
to
provide
for
people
going
forward.
This
is
a
really
really
big
part
of
it.
A
Okay,
one
thing
I
just
wanted
to
throw
in
there
real
quick
for
organizations
that
are
adopting
microservices,
which
is
a
big
part
of
cloud
native.
There's
going
to
be
some
challenges
you
have
to
think
about
as
well.
You
know
one
of
the
biggest
one
that
we
saw
at
Twitter
was
it's
fine.
When
you
have
microservices
that
consists
of
two
or
three
services,
you
can
still
doctor
compose
that
on
your
laptop
and
actually
try
to
run
your
entire
application
right
there.
A
On
your
laptop
and
try
things
when
you
get
to
a
thousand
services,
you're
not
going
to
be
dr.
composing
on
your
laptop
to
try
out
how
your
app
application
actually
works,
you're
going
to
need
to
find
new
ways
to
actually
enable
developers
to
work
in
these
in
these
environments
when
they
have
thousands
of
these
micro
services
that
are
all
trying
to
run
run
simultaneously
then
versioning
becomes
real
issue.
Is
each
of
the
different
microservices
want
to
actually
communicate
with
one
another?
What's
the
best
way
to
actually
manage
all
that
and
finally,
admission
control?
A
Now,
as
I
mentioned
earlier,
with
the
load,
balancing
you've
got
services
that
are
trying
to
communicate
with
one
another.
You
might
want
to
rate
limit
which
services
can
talk
to
which
other
services
and
and
you
need
to
start
thinking
about
the
best
ways
to
address
that,
and
if
people
decide
that
they
want
to
put
in
a
message
buses
to
actually
help
perform
some
of
those
functions
and
we'll
have
so
all
over
again,
which
will
be
totally
great
okay.
A
A
The
example
that
I
give
here
is
people
that
are
trying
to
run
containers
that
are
using
local
disks
or
volumes,
something
like
a
database,
and
they
want
to
make
sure
that
that
thing
gets
rescheduled
on
that
box
every
single
time
we
saw
this
with
the
D
cos,
I'm
sure
people
are
doing
it
with
crew
brannis
as
well.
They'll
use
things
like
labels
or
attributes
to
guarantee
that
a
pod
or
containers
always
get
scheduled
on
the
same
machines.
A
But
what
does
that
mean
that
basically
means
that
you're
back
to
having
last
year's
day
to
ops
challenges
all
over
again
all
right?
So
you
want
to
think
about
that,
and
it
was
a
big
question
which
is:
should
the
operator
really
care
about
this
kind
of
stuff?
Should
they
care
about
trying
to
set
labels
or
any
this
kind
of
stuff
to
actually
make
things
run?
A
Another
one
is
to
be
really
thoughtful
about
as
you're
rolling
this
stuff
out
to
production,
but
more
and
more
scripts
that
you
start
writing
to
act
as
the
control
planes
to
manage
all
these
containers.
This
is
probably
one
of
the
biggest
ones
that
I've
seen
in
a
bunch
of
organizations.
You
show
up-
and
you
say,
oh
to
run
this
app-
we've
had
to
write
this
control
plane
that
does
all
this
stuff
and
everyone's
going
to
be
doing
that.
A
You
know
those
scripts
become
on
the
critical
path,
so
you
better
make
sure
that
they
worked
work
super
well
and
unfortunately,
with
this
model
everybody
ends
up
reimplemented,
more
or
less
the
same
script.
So
when
one
organization
has
a
bug
in
their
script,
somebody
else
doesn't
benefit
it,
because
that
script
wasn't
something
that
everybody
could
could
actually
leverage.
This
is
one
of
the
reasons
why
the
the
D
cos
we
actually
introduced
this
concept
of
DC
us
services,
which
were
these
top
level
control
planes.
A
If
you
don't
have
something
like
that
in
place,
okay,
so
in
closing
you
know,
hopefully
I
didn't
scare
everybody
away
from
from
from
all
this
and
I.
Just
to
me,
I
think.
The
crux
is:
is
that
there's
still
a
bunch
of
day
to
ops
challenges?
We
as
an
industry
continued
to
make
amazing
amount
of
headway
in
the
right
direction
and
the
juice
is
definitely
worth
the
squeeze.