►
From YouTube: Next-Generation Microservices on Kubernetes - by Paul McLoughlin & Sachin Kamboj, Bloomberg LP
Description
Next-Generation Microservices on Kubernetes - by Paul McLoughlin & Sachin Kamboj, Bloomberg LP
"Enterprise deployments are complicated. When managing proprietary technologies, sensitive client data and complex rules for access rights you inevitably arrive at a situation where your PROD environment diverges from your DEV and there is no certainty that your code will work in production. You wrote an update, your unit and integration tests pass, yet your cursor is still floating half-heartedly over the 'RELEASE' button. Sounds familiar?
Time to end deploy-watch-it-break-rollback-fix-rinse-and-repeat approach. Paul and Sachin present DTP-on-kubernetes - the next-generation microservices platform at Bloomberg, allowing you to run several versions of your microservice in parallel against the same requests, diff their output and trace messages through the system. On top of that, enjoy the deployment in seconds, brought down from hours.
Learn what impact DTP is having on Bloomberg and how Kubernetes helped to make this system robust and stable (and of course enterprise-ready)."
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
It's
a
slight
twist
on
a
typical
kind
of
micro
service
idea,
I'm
also
going
to
talk
about
our
use
of
kubernetes
and
specifically
how
we
went
about
evaluating
whether
the
hype
and
the
reality
were
matched
some
of
the
problems
we
ran
into
and
some
of
the
pain
points
that
we
had.
So,
if
nothing
else,
maybe
that
kind
of
latter
half
of
the
presentation
will
give
you
some
gotchas
and
some
answers
to
those
which
maybe
will
save
you
some
time
so
Bloomberg.
Who
are
we?
A
A
It
wasn't
always
called
Bloomberg,
but
he
founded
the
company
with
those
three
other
people
35
years
ago,
where
privately
held
and
one
of
the
nice
things
I
think
about
working
for
this
company
is
a
software
developers
that
will
all
of
those
founders
were
also
software
developers
or
systems
development
people.
So,
first
and
foremost,
we
very
much
are
a
software
company.
A
We
have
90,000
employees,
we
have
offices
everywhere.
It
seems
we
have
around
about
5,000
Engineers,
primarily
based
in
New
York
London,
but
we
have
offices
in
various
other
places.
We
have
the
TV
channel.
We
have
radio
channels,
we
have
bloomberg.com,
which
is
a
great
website
these
guys
over.
There
run
it
and
it's
very
helpful.
If
you're
trying
to
manage
your
personal
finances,
we
also
have
a
magazine
which
we
publish
on
a
weekly
basis,
an
old-fashioned
hardcopy
thing,
and
that
has
a
circulation
of
almost
a
million
as
sent
out
to
150
different
countries.
A
All
of
those
other
bits
there
are
more
the
sideline
they're,
not
the
core
part
of
our
business.
So
if
ever
ever
any
of
you
have
heard
of
Bloomberg,
you
might
have
heard
something
called
a
Bloomberg
terminal
and
the
terminal
is
basically
a
subscription
service.
So
what
people
do
is
they
pay
us
to
get
access
to
our
terminal,
and
that
gives
them
any
kind
of
data
they
won,
which
should
be
kind
of
anything
they
can
imagine
relating
to
a
whole
series
of
different
industries.
So,
like
can
a
lot
of
you
I
guess
we
deal
with
data.
A
If
you
think
about
financial
dates,
are
the
kind
of
obvious
thing
that
springs
to
mind
will
be
something
like
the
price
of
a
ticket
or
a
stock
on
an
exchange
so
something
like
an
IBM
and
Microsoft
at
Google?
What
is
it
currently
trading
at?
We
do
do
that.
That's
only
a
very,
very
small
part
of
what
we
do.
A
So
when
we
come
back
and
think
again
about
the
price
of
IBM
today,
it's
all
the
many
many
factors
which
drive
that
price.
It's
the
second
and
third
level
thinking
that
people
have
and
we
have
to
try
and
second-guess
all
the
kind
of
information
they're
going
to
want
to
add
their
fingertips
to
be
able
to
make
a
determination
on
that
stock.
A
So
the
more
an
important
thing
to
point
out:
it's
not
just
the
current
value
of
these
pieces
of
information.
We
want
point
in
time
data
of
all
this
all
this
information.
So
what
I
mean
by
points
in
time
is
we
have
the
current
representation
or
the
current
value
of
a
piece
of
data,
but
we
want
to
be
able
to
see
historically
how
that
data
is
evolved
both
in
terms
of
its
absolute
value
but
also
its
structure.
So
maybe
the
value
of
a
piece
of
data
is
altered
structurally,
and
it's
obviously
altered
in
absolute
terms.
A
A
An
important
thing
to
point
out
as
well
is
that
the
kind
of
data
we
deal
with
has
to
be
very,
very
high.
Precision
people
will
make
decisions
which
have
a
monetary
value
of
hundreds
of
millions
or
billions
of
dollars,
so
we
cannot
afford
to
being
correct
or
wrong
with
our
data.
So
it's
very
very
important.
The
data
that
comes
through
our
company
is
correct
and
accurate.
A
A
We're
going
to
be
very
interested
in
whether
we're
gonna
be
very
interested.
If
there's
a
hurricane
or
some
kind
of
natural
disaster
expected
we're
going
to
be
very
interested
in
weather
forecasts
and
weather,
for
example,
there
is
enough
wind
in
this
particular
area
for
us
to
consider
selling
wind
turbines
there
I'm
going
to
have
to
manufacture
these
things,
so
I'm
going
to
need
so
a
lot
of
information
about
the
commodity
market.
A
What
kind
of
price
I
should
lock
in
whether
I
should
wait
a
little
bit
whether
I
should
just
proceed,
as
is
I'm
going
to
want
to
know
about
government
legislation
that
potentially
could
be
coming
up
and
affecting
me
I'm
going
to
want
to
know
about
grants
and
rebates,
and
whatever
else
the
the
government
is
doing
I'm
going
to
want
to
know
about
the
state
of
the
economy?
Maybe
people
don't
actually
have
money
to
be
able
to
spend.
Maybe
this
particular
geographic
area.
A
People
are
not
particularly
keen
on
renewable
energy
and
there's
kind
of
a
million
other
different
areas
or
factors
which
could
even
firms
and
be
very,
very
useful
to
me
as
a
business
owner.
So
we
have,
we
have
around
about
3,000
employees
in
our
data
division
and
their
job
is
basically
to
second-guess
what
our
customers
might
want,
and
if
there
is
anything
missing,
then
the
customers
can
ask
the
sales
rep
for
it
and
we
will
go
and
acquire
for
it
require
it
for
them,
so
their
job
is
basically
to
source
as
much
information
as
possible.
A
This
is
the
development
team,
so
you
have
Michael
amico
over
there
in
session
down.
Here
please
come
and
talk
to
us
this.
This
presentation
is
more
about
Cuban
ETS,
but
this
is
a
production
system
which
has
been
running
for
a
number
of
months
at
scale.
So
we
have
a
lot
of
experience
with
Prometheus
and
alternatives,
flew
Indian
alternatives,
configuration
management
tools,
so
please
come
and
pick
our
brains
happy
to
help.
A
The
aim
is
obviously
this
is
this
is
why
I
roll
at
this
conference,
but
qet
states
care
of
an
awful
lot,
the
cross-cutting
concerns
that
we
would
have
and
the
non-functional
requirements
that
are
very,
very
common.
So
it's
very,
very
important
that,
or
it's
very
nice,
that
I
as
a
developer
can
focus
more
on
the
problem
at
hand
rather
than
having
to
build
a
system,
own
servers
and
all
the
other
kind
of
operational
concerns
that
go
around
that
which
take
up
an
awful
lot
of
time.
A
So
the
aim
of
this
platform
was
to
kind
of
minimize
that
cost
and
make
it
so
that
we
had
single
ticket
click
deployment
and
we
could
push
go
down
production
exceptionally
easily
so
microservices.
So
these
these
days
tends
to
be
the
answer
to
all
problems.
We
won't
have
a
debate
about
what
that
means,
but
to
us
it
was
basically
about
the
idea
that
they're
small
they're
focused.
They
do
one
thing
and
do
one
thing
well
in
terms
of
what
we
expected
from
people
who
deploy
services
on
our
platform.
A
We
wanted
them
to
go
and
see
that
they
had
a
minimal,
API
and
very
clearly
defined
dependencies
and
ideally
minimal
dependencies
as
well.
We
also
wanted
them
to
make
sure
they
are
good
code
coverage,
so
a
good
unit
test
get
acceptance
tests
than
those
tests
should
pass,
and
in
exchange
for
that
we
would.
We
would
build
this
platform
that
would
empower
them.
A
A
We
shouldn't
be
overly
concerned
with
only
through
pert
below
a
certain
number,
or
only
latency,
above
a
certain
number.
So
the
thing
that
we
built
was
this
next
generation
micro
services.
So
the
first
point
up
there
is
where
we
start
to
differ
from
the
kind
of
normal
example
of
things
in
the
normal
way,
the
world.
So
we
we
very
clearly
in
very
explicitly
wanted
people
to
test
in
production,
and
that
was
a
huge
thing
for
us.
We
wanted
to
handle
all
the
plumbing.
A
We
wanted,
this
single
click
deployment
or
potentially
some
kind
of
push
on
green
deployment
scenario,
and
we
wanted
all
the
kind
of
the
good
things
that
communities
would
bring
to
the
table.
We
also
wanted
the
ability
to
self
host
a
lot
of
the
services
which
define
our
platform
within
our
platform
and
that's
something
that
we've
also
done
in
the
same
way
that
communities
are
moving
along
those
lines.
A
Testing
in
production
is
still
up
there
and
you
obviously
is,
if
you
know,
if
you
have
conversation
with
a
with
some
people,
they
would
you
know,
look
you
very
strangely
and
say
that
must
must
be
a
bad
thing.
This
is
a
famous
quote
from
Fred
Brooks
in
the
mythical
man-month,
so
all
programs
are
optimist.
Obviously,
just
a
paraphrase-
maybe
it's
the
curious
nature
that
our
profession
requires
and
the
kind
of
uncertainty
involved.
A
That
kind
of
drives
this
and
requires
this
of
us,
but
these
two
highlighted
sentences
at
the
back
of
things
that
I've
said
many
times
and
I've
heard
many
many
times
this
week,
I'm
sure.
So
this
time
it
will
surely
run
and
I've
just
found
the
last
bug
there
are
no
more
don't
worry.
It's
be
fine.
This
time
this,
the
typical
release
cycle
that
I'm
guessing
a
lot
of
people
have.
We
have
some
kind
of
a
production
environment,
and
then
we
have
a
series
of
pre-production
stages.
A
We
write
our
code,
we
write
a
series
of
tests,
we
run
the
tests
and
hopefully
that
all
pass,
if
no
we'll
fix
them
at
some
point
in
our
process,
we're
going
to
be
happy
that
all
the
tests
are
passing
and
we're
going
to
iterate
through
and
we're
going
to
deploy
that
into
the
next
environment
up
so
we
deployed
it
in
there.
It's
going
to
go
through
some
other
kind
of
testing
phase,
at
which
point
again,
we'll
ascertain
is
this
working
on
now.
If
it's
not,
then
we
loop
back.
A
Our
customer
starts
using
our
service
or
our
client
does,
and
then
we
have
the
fabulous
thing
where
we
find
out
that
something
somehow
is
different
and
has
changed
so
we're
not
quite
sure
the
hell's
going
on,
because
we've
done
loads
of
testing
we've
been
exceptionally
well
behaved.
The
still
doesn't
work
so
why?
Why
is
why?
Is
this
the
case
so
differences,
obviously
so
differences
in
the
environment?
A
There
are
the
kind
of
disturbing
differences
which
we
should
never
have,
but
unfortunately
do
arise,
so
differences
potentially
in
hardware
in
the
OS
permissions
may
be
with
the
user
profile
any
number
of
things,
maybe
we've
iterated
through
too
many
times
and
something's
going
wrong.
Maybe
there
are
other
things
running
on
that
box,
which
is
which
is
affecting
our
dependencies
and
causing
us
to
break
in
an
in
production
environment,
and
in
our
particular
case
we
have
a
lot
of
data.
This
data
is
highly
complex
and
highly
interdependent
and
that
often
causes
problems.
A
It's
frequently
the
case
that
we
have
excellent
product
data
in
production,
but
as
you
go
down
through
the
environments,
the
data
degrades
quite
rapidly
and
also
we
have
to
rely
on
an
awful
lot
of
third
party
systems.
Other
groups-
and
maybe
they
just
aren't
as
well
behaved
as
we
are.
Maybe
they
don't
even
have
at
their
version
of
their
system.
A
So
what
we
have
is
we
end
up
in
this
typical
kind
of
release
cycle,
which
we
should
certainly
was
familiar
to
us
and
I.
Guess
is
pretty
common
out
there.
So
we
code
we
deploy.
Maybe
it
works.
Maybe
it
breaks,
we
roll
it
back
and
then
we
repeat,
we
repeat
and
repeat,
the
idea
is,
or
the
fundamental
thing
is
here,
unless
we
can
guarantee
that
our
non
production
environment
is
exactly
identical
to
production.
A
There's
no
way
we
can
be
100%
confident
and
when
we
push
out
to
production,
things
will
will
definitely
work
and
try
as
hard
as
we
can.
It
seems
to
be
impossible
for
us
at
least
so
ensure
that
we
have
some
kind
of
a
pre-production
stage
where
production
is
exactly
identical.
So
this
is
where
we
come
back
to
idea
of
production
and
testing
to
ensure
that
when
we
go
to
production,
we
can
be
confident
that
things
are
going
to
work.
A
So
we
a
little
tweak
to
what
what
people
normally
do.
We're
going
to
describe
it
down
a
second,
so
obviously
we're
going
to
write
our
code,
we're
going
to
deploy
it
and
we're
deploying
a
version
which
is
a
canary
version.
So
you
can
see
it's
red,
and
that
means
that
it's
not
actually
handling
or
delivering
production
responses
out
to
clients.
It's
still
going
to
receive
the
product,
the
production
requests,
but
it's
not
going
to
handle
the
responses
back
out.
A
We're
going
to
send
in
a
bunch
of
sample
inputs
we're
going
to
check
the
output,
see
if
we're
happy
or
not.
If
we
aren't,
then
we're
obviously
going
to
make
some
changes
redeploy
and
that
version
one
is
going
to
kind
of
iterate
at
some
point.
We're
going
to
test
this,
we're
going
to
test
the
inputs,
we're
going
to
test
the
outputs
I'm
going
to
be
we're
going
to
be
confident
the
version
that
we
just
pushed
out
and
proper
and
should
go
out
and
be
handling
responses
back
out
to
our
customers
or
clients.
A
So
we
have
this
one
version
running
in
production.
Something
happens.
We
need
to
make
a
change,
so
we
write
some
more
code,
we
deploy
it
and
we
deploy
a
canary
release
into
production
now,
obviously,
one
of
the
requirements
that
we
have
and
one
of
the
the
kind
of
deals
we
make
with
the
people
who
use
our
platform
is
that
if
you're
going
to
have
multiple
versions
of
a
service
running
in
production,
then
you
have
to
certain
definitely
not
do
certain
things.
A
So
if
this
code,
for
example,
was
placed
in
trades,
this
would
be
very,
very
bad
if
we
had
two
services
placed
in
the
same
trades.
So
it's
important
that
we
have
a
restriction
and
the
idea
is
that
people
should
not
touch
systems
which
have
side
effects.
So,
for
example,
we
must
not
write
to
a
database.
We
must
not
write
to
a
file
system.
We
must
not
send
emails.
There
are
great
many
things.
A
This
may
seem
exceptionally
restrictive,
but
we
don't
think
it
is
and
we'll
come
back
to
that
in
a
minute
or
so
so
we've
we
pushed
our
canary
release,
which
has
some
modified
business
logic
in
it
there,
and
then
we
look
at
that
and
we
again
all
the
requests
are
coming
into
the
system
and
there
is
a
mirroring
system
which
is
mirroring
all
the
requests
in
reasonable
times.
All
the
different
versions
of
this
particular
service.
A
We
check
we
leave
it
running.
Are
we
no?
We
decide
we're,
not
happy,
so
we
make
some
more
changes
when
we
deploy
another
alternate
version
of
this
service.
So
now
we
have
the
publishing
version
of
our
service,
which
is
handling
our
production
requests
in
which
our
users,
our
clients,
consumers,
whoever
are
using,
and
then
we
have
to
Canary
releases,
which
are
running
with
modified
business
logic,
maybe
and
for
some
reason,
we're
not
happy
that
they
should
go
out
again.
A
We
check
the
input,
we're
checking
the
output
for
each
of
the
different
versions.
If
we're
happy,
then
we
can
promote
that
one
to
be
the
publishing
version
and
then
the
other
to
revert
to
being
canary
and
canary
releases
so
now
release
three
is
the
version
of
this
particular
services.
We
just
handling
production
requests,
so
where
are
we
we're
in
a
position
now,
where
I
don't
really
need
a
dev
environment?
If
I'm
writing
this
code,
so
hopefully
we
can
deploy
our
code
within
this
system.
The
platform
is
going
to
take
care
of
all
the
operational
concerns.
A
A
The
important
thing
here
is
our
idea
that
micro
services
must
be
pure
and
there
must
be
no
alterations
to
external
stay.
So
what
I
mentioned
before
was
the
the
obvious
example
about
a
trading
system.
There
are
obvious
examples
about
updating
databases,
sending
emails
touching
the
fast
system,
any
number
of
things
when
we
thought
about
this.
We
thought
about
how
we
structured
our
code,
and
you
know
the
common
paradigm
within
software
development.
Is
you
separate
what
varies
somewhat
stays
the
same
and
in
terms
of
and
taking
that
analogy
a
little
bit
further.
A
We
wanted
to
think
about
the
frequency
of
updates
the
frequency
of
what
varies
and
to
us.
If
we
think
about
a
service
that
gives
us
the
capability
place
to
trade
interact
with
the
file
system
and
mail
system,
anything
like
that
or
interaction
with
those
services
does
not
change
or
does
not
vary.
Typically,
what
varies
is
the
object
of
the
data
payload
that
we
give
to
those
services
all
those
other
systems.
A
A
The
last
bit
here
really
is
you
know
it's
pretty
standard
everybody's
doing
it.
So
how
do
people
write
their
code?
So
we
push
our
code
to
github
or
whoever
wants
to
write
a
microscopes
pushes
their
code
to
get
help.
You
tagged
it.
We
give
people
the
ability
to
set
in
VARs,
have
secrets,
and
then
it's
entirely
on
a
user's
hands
to
say
of
those
canary
releases.
Maybe
we
nuke
one
and
it's
no
longer
active.
Maybe
we
switch,
which
is
the
publishing
version.
A
B
Okay,
thank
you,
Paul,
hello,
everyone.
My
name
is
Sachin
compote
and
I'm,
basically
going
to
discuss
the
nuts
and
bolts
of
our
system,
how
we
built
the
system
using
kubernetes
and
some
of
the
lessons
we
learned
all
along
the
way,
some
of
the
pitfalls
we
encountered
and
our
general
approach,
and
hopefully
you
can
learn
from
that
approach
and
perhaps
not
make
the
same
mistakes
that
we
made
okay.
B
So
when
we
just
set
out
like
building
the
system
that
could
be
used
by
non
engineers
and
engineers,
like
Paul
mentioned,
we
had
a
shopping
list
of
things
that
we
really
wanted
and
what
was
in
that
shopping
list.
Well,
we
wanted
isolation.
This
is
the
basic
concept
of
if
you
have
a
micro
service
that
might
have
a
bug
in
it
or
a
flaw
or
might
be
malicious,
we
don't
want
it
to
affect
any
other
micro
service
and
we
specifically
don't
want
it
to
take
down
the
whole
system.
We
want
rendering
of
dependencies.
B
So
this
is
the
concept
where
we
have
a
micro
service.
It
depends
on
versions,
XYZ
of
libraries,
a
B
and
C,
and
we
have
another
micro
service
that
depends
on
versions
tuv
of
the
exact
same
set
of
libraries,
a
B
and
C,
and
we
want
those
two
micro
services
to
be
able
to
coexist
with
each
other
and
to
be
able
to
run
simultaneously
and
not
affect
each
other.
Oh,
we
wanted
orchestration
the
basic
idea
being
that
we
have
micro
services.
We
want
to
deploy
them
across
different
nodes.
B
We
want
to
make
sure
that
they're
deployed
in
different
availability
zones
and
the
idea
that
we
want
to
be
able
to
auto
scale
them,
depending
on
the
amount
of
traffic
that
we
get
depending
on
the
CPU
usage,
depending
on
various
other
constraints
and
like
Paul
mentioned,
we
wanted
ease
of
deployment,
typically
a
single
deployment,
the
ability
to
run
multiple
versions
of
a
micro
service
in
parallel
and
the
basic
idea
of
having
rolling
updates-
and
you
know,
starting
a
new
version
up
and
then
taking
down
another
worship.
And
finally,
we
wanted
self-healing.
B
So
this
is
the
idea.
If
you
have
a
micro
service,
that's
crashing,
we
should
be
able
to
bring
it
back
up
and
you
know
go
on
from
there
and
hopefully
try
to
figure
out
why
the
micro
service
is
crashing.
So,
in
addition
to
all
these
things,
the
last
three
or
two
points
are
pretty
obvious.
We
want
the
whole
system
to
be
very
stable
and
reliable
and
to
have
high
availability.
Okay,
so
that's
our
shopping
list.
We
are
at
kubernetes
conference
and
kubernetes
is
obviously
the
best
things
sliced
bread.
B
So
we
just
apply
our
system
on
kubernetes.
It
all
works.
We
all
go
home.
Everybody
is
happy.
Everything
works
great
awesome.
Unfortunately,
the
real
world
is
never
as
simple
and
not
exactly
the
same
as
an
ideal
world.
So
whenever
you
have
a
piece
of
technology,
that's
being
touted
as
the
next
best
thing
since
sliced
bread,
it's
been
touted.
As
you
know,
this
is
the
thing
to
use.
It's
always
helpful
to
take
it
with
a
grain
of
salt
to
you
know,
show
a
healthy
degree
of
skepticism
towards
said
and
to
actually
really
test
it
out.
B
Take
it
through
the
paces
and
try
to
figure
out
it's
making
these
you
know
awesome
claims.
Does
it
live
up
to
them?
So
what
I
want
to
cover
in
this
connects
couple
of
slides
is
how
we
really
tested
out
the
system,
how
we
got
comfortable
with
it
and
what
we
learned
along
the
way,
and
hopefully
you
can
learn
from
our
experience.
It's
obviously
there's
no
way.
B
B
B
Okay,
so
back
to
our
shopping
list,
I
would
argue
that
docker
does
a
very
good
job
of
packaging.
Some
great
Linux
features,
like
you,
know,
cgroups
namespaces
capabilities
that
gives
us
excellent
isolation
and
rendering
of
dependencies
and
kubernetes
was
really
designed
for
the
concept
of
having
orchestration
easy
deployment.
Auto
scaling
the
things
like
that,
but
what
we
really
wanted
to
determine
was
if
they
deployed
a
cluster
using
kubernetes
Winsett
stable,
is
it
reliable?
Is
it
so
highly
highly
available
and
before
I
continue?
I
just
wanted
to
mention
two
caveats.
B
The
first
one
is
that
kubernetes
is
a
very
fast
moving
target,
so
anything
that
I'm
mentioning
right
now
and
that's
true
right
now-
is
probably
not
going
to
be
true
a
month
down
the
line,
and
it's
definitely
not
going
to
be
true
or
a
year
down
the
line,
and
the
second
thing
is
none
of
this.
There
are
none
of
the
points
that
I'm
going
to
mention
should
be
taken
as
a
criticism
of
kubernetes
itself.
B
B
So
with
that
in
mind,
I
just
wanted
to
briefly
discuss
our
basic
architecture
so
that
we're
all
on
the
same
page,
and
we
all
understand
you
know
how
it
worked.
So
we
have
any
number
of
data
centers
for
space
restrictions.
I
can
only
show
two
on
the
slide,
but
you
know
we
have
multiple
data
centers
and
we
have
multiple
kubernetes
clusters
running
on
those
data,
centers,
they're,
more
or
less
or
actually
they're
supposed
to
be
exactly
identical.
So
our
communities
clusters
are
supposed
to
be
identical
across
data
centers,
we
have
five
master
nodes.
B
What
this
means
is
that
we
are
able
to
survive
two
node
failures,
and
then
we
have
any
number
of
worker
nodes
and
all
our
data
centers
are
combined
together
using
a
load
balancer
diving
in
a
little
bit
deeper
into
what
our
master
nodes
look
like
at
the
bottom.
You
can
see.
We
have
flannel
D,
that's
basically
to
allow
communication
between
ports
running
on
two
different
nodes.
Like
I
mentioned,
we
are
using
docker
for
isolation,
rendering,
and
then
we
have
cubelet
and
q
proxy,
which
is
running
on
every
single
node.
B
We
also
have
this
thing
called
okay.
It
has
API
server
proxy
and
I'm
going
to
go
into
the
details
of
why
we
need
that
and
what
that
is
in
a
little
in
a
little
bit.
So
keep
that
in
mind
that
there's
some
thing
that
I
haven't
explained
right
now
and
finally,
this
is
a
master
node,
so
we
obviously
need
the
master
components,
which
means
we
have
the
API
server.
We
have
the
scheduler,
we
have
the
controller
manager,
we
have
SCP
as
our
source
of
truth
and
we
have
qpn
s
going
on
to
a
worker
node.
B
B
Ok,
so
now
that
we
have
a
basic
understanding
of
our
architecture-
and
you
know
a
common
set
of
things
to
talk
about-
let's
discuss
about
how
we
evaluated
our
system-
we
are
firm
believers
in
the
maxim
that
the
only
real
way
to
know
how
a
system
works
is
to
try
to
break
it.
So
we
wrote
this
piece
of
tool,
that's
called
powerful
seal,
it's
our
chaos
monkey
and
the
idea
behind
powerful
seal.
Is
it
breaks
things
so
that
we
can
understand
how
things
work?
B
So
what
can
we
do
with
empower
still
still
I'm,
not
sure
if
you
can
read
the
slides,
but
just
to
go
through
them?
It
allows
us
to
view
an
inventory
of
all
the
notes
in
our
system,
so
we
can
see
or
what
are
the
notes
in
the
system
we
can
see.
We
can
group
them
by
availability
zones.
We
can
group
them
by
components,
so
we
can
see
which
ones
are
master
notes,
which
ones
are
worker
notes
and
have
a
good
idea
of
what's
in
our
system.
B
The
other
thing
that
it
allows
us
to
do
is
look
at
the
status
of
various
components
that
are
running
on
those
particular
notes.
So
if
you
wanted
to
look
at
the
status
of
let's
say
the
cubelet
node
that
was
running
on
all
the
nodes
on
a
particular
availability
zone,
we
can
do
that.
We
can
look
at
the
status
of
let's
say
the
API
server
or
the
master
and
collect
the
status
from
all
the
various
components,
and
it
also
allows
us
to
give
some
stability
to
tail
or
collect
logs
across
nodes.
B
So
what
this
means
is
if
we
wanted
to
collect
the
logs
from
all
the
API
servers
from
all
the
masters,
we
can
do
that
if
you
wanted
to
collect
logs
from
let's
say
all
the
cubelet
components
from
all
the
notes
in
a
particular
availability
zone,
we
can
do
that
and
finally,
it
really
wouldn't
be
much
of
a
chaos
monkey
if
it
wasn't
able
to
take
things
down
and
the
part
of
the
reason
why
we
designed
the
system
was
mainly
to
take
things
down.
So
what
we
can
do
is
we
can
take
down
a
node.
B
We
can
take
down
an
entire
availability
zone
and
we
can
also
do
things
like
you
know.
Watch
the
things
go
down
in
real
time,
see
what's
happening
and
once
things
have
gone
down,
we
can
try
it
and
see
you
know,
collect
logs
from
them
and
see
what
really
happened.
So
this
is
some
great
work
that
Miko
did
in
building
this
system.
Okay,
so
now
we
have
powerful
seal.
We
have
the
ability
to
take
down
notes.
We
have
the
ability
to
take
down
components.
We
also
have
the
ability
to
take
down
things
in
production.
B
You
know
run
in
Auto
smash
mode
when
it's,
you
know
continuously
running
and
what
can
be
learned.
So
this
is
tied
off.
You
know
lessons
learned
or
how
we
learn
to
stop
worrying
and
love
kubernetes
I'm
not
going
to
go
through
all
the
lessons
learned,
although
you
know
pain
points,
all
the
things
that
we
experienced
come
talk
to
us.
B
If
you
want
to
know
more
I'm
only
going
to
discuss
three
stories,
and
these
three
stories
are
kind
of
illustrative
of
the
things
that
we
tried,
the
things
that
we
learned
and
some
lessons
that
you
can
do.
Ok,
so
the
first
story.
This
was
very
early
on.
We
started
out
with
you
know,
deploying
a
kubernetes
cluster,
deploying
a
couple
of
pods
running
micro
services
and
then
testing
it
out
and
we're
like.
Ok
great.
This
works.
Let's
take
things
down,
so
we
use
powerful
seal.
We
took
down
one
particular
node.
B
B
So
why
was
this
well?
It
turns
out
that
cubelet
and
some
ways
of
the
kubernetes
components
can't
really
have
deal
with
multiple
API
servers,
which
means
that
even
though
cubelet
accepts
a
command
line
flag
that
allows
you
to
specify
multiple
API
servers.
In
actual
fact,
what
cubelet
does
is
it
talks
to
the
first
API
server
tries
to
contact
it
if
it
can't
reach
the
first
API
web
server.
It
does
not
even
try
to
go
to
the
next
one,
and
this
was
very,
very
surprising
to
us.
B
So
there
are
outstanding
issues
on
github
and
you
can
read
up
on
that.
The
reason
you
know
if
this
this
is
a
really
big
deal
I'm,
you
know,
but
we
were
surprised
that
you
know
other
people
did
not
face
that
and
the
reason
for
that
is
that
the
way
people
deployed
them
they
run
the
API
server.
Behind
some
kind
of
a
load
balancer,
that's
provided
by
the
Google
cloud
engine
or
by
AWS,
which
does
the
proxy
into
multiple
API
servers.
B
B
So
this
is
an
example
of
something
that
we
wouldn't
have
learnt
if
we
didn't
take
things
down
and
figure
the
figure
out-
okay,
our
second
example,
so
we
had
our
you
know
kubernetes
cluster
running
for
a
while
things
were
working
great
and
very
soon.
Once
you
know,
we
found
out
that,
for
some
reason
there
were
ports
running
on
a
particular
node
that
weren't
able
to
communicate
with
ports
running
on
another
node.
So
why
is
that?
B
Okay,
so
on
further
investigation,
we
found
out
flannel
D,
which
handles
the
communication
between
thoughts
that
is
running
on
one
node,
with
pasta
running
on
the
other
load,
crashed
on
a
particular
node
and
was
no
longer
running,
and
specifically,
it
had
died
because
of
a
sick
pipe.
So
first
question:
well,
why
did
it
die
we
found
out?
B
What
really
happens
is
we
had
a
micro
service
that
was
generating
a
shit-ton
of
logs
and
the
micro-service
were,
and
we
were
using
docker,
which
was
actually
using
the
journal,
D
log
driver
and
at
some
point
in
time
journal
D
could
not
handle
that
number
of
logs,
so
journal,
D
crashed,
Journal
D
was
restarted
by
system
D.
However,
what
happens
when
you
have
a
process?
That's
trying
to
write
to
an
open
file
descriptor
and
whatever
is
listening
at
the
other
end
of
the
file
descriptor
crashes,
you
get
a
sick
buy
unless
you
handle
sick
pipe.
B
Well,
in
that
situation,
you
are
going
to
crash
also
how
many
of
you
have
actually
written
code
in
which
you
know
you
try
to
handle
errors
in
writing
to
STD
out
or
STD
error.
I
bet
you
almost
no
one.
We
don't
expect
STD
out
or
STD
error
to
crash.
Yet
we
expect
our
microservices.
We,
the
standard
ideas,
so
just
write
your
logs
to
STD
out
in
STD
air
and
we'll
collect
them
for
you
and
we'll
display
them
for
you
in
a
nice
way.
Okay,
so
plan
OD
was
crashing
now.
B
Here's
the
other
interesting
part
of
the
situation.
It
crashed
with
the
sick,
pipe
and
system
D,
which
is
responsible
for
starting
these
services
and
restarting
them
consider
sick
pipe
to
be
one
of
the
normal
conditions
for
exit,
which
means
that
if
so,
if
for
process
goes
down
because
the
sick
pipe,
it
will
not
be
restarted.
Okay.
So
how
do
we
fix
this
issue?
Well,
number
one.
We
make
sure
plan
of
restarts,
no
matter
what
happens
the
other
way
we
did
it
was
we
developed
a
tool
which
we
called
Google
pinger?
What
this
real
tool
does?
B
Is
it
basically
runs
as
a
daemon
set,
so
it's
running
a
pod
on
every
single
node
and
it's
only
job
is
to
communicate
with
pods
on
other
nodes
and
try
to
ping
them
using
an
HTTP
pang
and
if
it
can
reach
the
other
nodes.
Okay,
things
are
good
great,
if
not
we'll,
it
will
tell
us
about
it.
So
this
is
a
way
in
which
we
can
grasp
the
problem.
B
So
that's
our
next
to
the
lesson
learned
over
here
is
basically,
whenever
you
have
a
complex
system
with
very
with
a
number
of
moving
parts
and
a
number
of
interdependent
components,
you
can
have
a
failure
in
one
component,
which
is
like
microservice.
That's
writing
too
many
logs
or
something
like
that
caused
a
completely
separate
system.
That's
not
a
dependency,
and
you
know
that
has
nothing
to
do
with
you
to
fail,
for
example.
Finally,
okay,
so
our
final
lesson
learned,
so
you
know
right
now:
you've
deployed
a
kubernetes
cluster
or
you've
deployed
a
couple
of
clusters.
B
You
have
been
testing,
you
know
it
for
a
while,
then
dev,
you've
deployed
a
whole
bunch
of
micro
services.
You've
taken
things
down,
you've
put
them
back
up.
You've
learned
a
whole
bunch
of
things
great
now,
you're
ready
to
deploy
to
production
everything's
going
to
work
great
awesome,
not
so
fast,
even
if
you
do
all
that,
it
really
really
helps
to
try
to
understand
how
the
sausage
was
made.
What
are
the
various
components,
how
they're
built,
how
they're
put
together
and
what
issues
or
what
design
decisions
they
made?
B
So
I
just
put
this
light
up
here.
You
know
there
could
be
a
lot
of
discussions
about
this,
but
these
are
you
know,
security
concerns
that
wouldn't
be
obvious.
Unless
you
really
understand
understood
what
the
various
components
were,
how
they
interact
with
each
other
and
how
they
operate.
So,
for
example,
the
communication
between
the
API
server
and
couplet
is
broken.
It's
susceptible
to
man-in-the-middle
attacks.
This
is
something
that
was
really
true
in
with
the
worsens
1.1,
1.2
and
later
versions,
it's
gradually
being
fixed
right
now.
B
Most
of
the
communication
has
been
secured,
but
this
really
tells
you
that
security
was.
You
know,
thought
of
as
an
add-on,
as
opposed
to
being
designed
from
the
scratch
and
they're
things
that
have
been
fixed
along
the
way,
but
ours
we're
still
not
there
where
we
have
under
percent
security.
The
other
thing
is,
you
couldn't
really
completely
disable
the
insecure
port
and
the
key
on
the
API
server.
This
has
now
been
fixed.
It's
not
in
the
current
version
of
kubernetes,
so
it's
not
in
1.4,
but
it
is.
B
It
will
be
available
in
1.5
and
later
versions,
but
one
of
the
key
things
that's
still
outstanding
for
us
from
a
security
perspective
is
that
certificate
revocations
is
not
supported.
What
this
really
means
is
that
if
you
have
a
client
certificate,
that's
stolen
you'd,
really
don't
have
any
choice
but
to
redeploy
all
your
certificates
to
rebuild
the
system
and
go
from
there.
Okay,
so
just
to
summarize
what
our
approach
and
what
we
hope
you
learn
from
this
is
okay,
you
know:
try
to
deploy
coasters,
try
to
break
them.
B
Try
to
take
down
notes,
try
to
you
know,
see
how
it
works.
Secondly,
basically,
when
you
have
a
when
you
have
a
complex
system
with
a
lot
of
various
components,
a
problem
in
one
component-
that's
not
directly
related
to
another
component-
can
take
it
down
so
be
aware
of
that.
And
finally-
and
this
is
where
I
really
want
to
shout
out
toward
the
kubernetes
community
and
to
the
open
source
community-
try
to
understand
how
the
system
is
built
and
since
kubernetes
everything
is
transparent,
everything
is
in
github.
B
You
can
read
all
the
issues
you
can
try
to
understand
how
it
works.
You
can
really
try
to
get
a
feel
of
you
know
what
things
out
or
what
design
decisions
were
made
and
what
things
work.
Okay.
So
with
that
we're
hiring
in
various
different
locations-
London
New,
York
Princeton-
please
contact
Paul
if
you're
interested,
as
we
hope
you
were,
will
be
I'd
like
to
give
a
shout
out
to
Michael
Michael,
who
did
much
of
the
worker
that
represented
over
here
and
with
that
I'd
like
to
open
it
up
to
questions.
Thank
you.
A
Yes,
I,
don't
if
everyone
got
there,
the
question
is
at
some
point:
if
you
touch
these
external
systems,
you
must
have
some
of
some
kind
of
impacts.
I
mean
we,
we
didn't
really
care,
particularly
if,
for
example,
we
spewed
in
logs
on
some
service
that
we
were
calling
now
wasn't
a
consideration
of
us
and
similarly,
it
wasn't
necessarily
a
consideration
of
ours
that
we
call
a
service
and
we
have
some
kind
of
degradation
of
service
in
that
service.
A
A
It's
kind
of
like
a
slippery
slope
to
start
talking
about
idempotency
and
side-effects
and
repeatability
and
stuff,
because
we
didn't
necessarily
want
that,
because
we
didn't
want
the
idea
that
you
could,
for
example,
delete
a
record
in
a
database
and
keep
deleting
that
record
and
that
would
kind
of
be
okay.
That's
definitely
not
what
we
wanted,
so
we
the
the
way
I
always
kind
of
think
about
it.
It's
kind
of
like
almost
embracing
a
functional
paradigm.
We
have
very
clear
to
find
inputs
and
outputs,
and
then
it
becomes
exceptionally
easy
to
test
your
code.
A
It's
trivial
to
test
your
code
then,
and
the
key
thing
to
our
mind,
was
that
business
logic
is
the
thing
that
varies
all
the
time:
it's
not
the
interaction
with
the
database
or
all
these
other
kind
of
libraries
and
services.
It's
just
like
that,
computation
that
you
do
so
separate
that
bring
that
out,
make
it
trivial
to
run
new
versions
of
that
alternate
versions
of
that
and
give
some
really
really
nice
clean
way
to
test
it
amazingly
well,
and
then
we
can
reach
the
confidence.
A
Well,
please
feel
free
to
come
and
talk
to
herself
too.
It's
like
I
said
this
is
a
production
system
has
been
running
for
for
a
year
without
kubernetes
or
eight
months
without
kubernetes,
and
then
the
last
four
or
five
months
on
cuban
et.
So
the
initial
version
was
significantly
more
naive
than
the
version
that
we
have
now
so
Cuba
needs
added
a
lot.
I
would
say
through
that
kind
of
process.
A
We
use
Prometheus
at
the
moment,
but
we
have
different
ideas
or
concepts
of
different
types
of
logging
within
our
system,
so
we
use
Prometheus
for
some
data
and
we
use
other
kinds
of
data
tools
for
other
kinds
of
data
as
well.
So
you
know
if
people
want
to
come
and
pick
our
brains,
please
feel
free.