►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Running Kubernetes on Ephemeral Infrastructure - Greg DeArment & Vlad Ungureanu, Palantir Technologies
Palantir runs millions of Kubernetes pods on thousands of nodes to power data platforms that, in turn, power the world's most important institutions. Palantir's infrastructure engineering team decided early in their Kubernetes journey to make infrastructure failure a feature of the deployment systems: every node gets destroyed and rebuilt every 48 hours, thus turning extraordinary events into daily routine. In this talk, Greg and Vlad present the reasoning behind implementing an ephemeral Kubernetes infrastructure, discuss the consequences for application architecture, and sketch solutions to the problems encountered along the way.
https://sched.co/MPYX
A
Everyone
we're
gonna,
get
started,
so
my
name
is
Greg
Jarmon
I
lead
our
production
infrastructure
group
at
Palantir
technologies
and
I'm,
with
Vlad
who's.
The
technical
lead
on
our
cloud
infrastructure
team
and
today
we're
gonna
spend
some
time
talking
about
what
happens
when
you
run
large
kubernetes
clusters
and
you
destroy
every
node
every
48
hours
we'll
talk
about
why
Palantir
chose
to
take
an
ephemeral
approach
to
our
infrastructure
and
how
we
run
kubernetes
on
it.
A
Before
I
do
I
want
to
briefly
introduce
Palantir
for
those
who
aren't
familiar.
Palantir
is
a
software
company.
We
were
founded
fifteen
or
so
years
ago,
and
our
mission
has
been
to
our
purpose
has
been
to
improve
the
way.
The
world's
most
important
institutions
use
data
to
make
decisions.
A
few
examples
of
what
I
mean
our
software
is
the
foundation
of
cyber
defense
systems
for
large
institutions
in
entire
nation
states
around
the
world.
Our
software
is
used
in
support
of
anti-terrorist
missions
in
the
West,
from
everything
from
intelligence
operations
to
mission
planning
in
the
field.
A
Ferrari
uses
our
technology
to
make
their
Formula
one
cars
faster
and
more
efficient.
If
you
look
very
closely,
you
can
see
our
logo
on
the
bottom
right
fender,
which
is
pretty
cool,
Palantir
and
Airbus,
founded
Sky
wise
as
a
platform
to
improve
the
way
the
airline
industry
uses
data
to
make
air
travel
safer
and
more
economical.
Today,
sky
wise
collects
high-resolution
telemetry
from
millions
of
sensors
across
the
aircraft
flying
around
the
world
and
enable
Airbus
engineers
and
aircraft
operators
to
use
this
data
to
root
cause
faults
in
flight
and
improve
the
way
airlines
operate.
A
Palantir
and
merck,
pharmaceutical
join
our
formative.
Joint
partnership
call
it's
entropy
last
year,
which
is
focused
on
enabling
collaboration
amongst
medical
researchers
around
the
world
towards
improving
the
cancer
treatments
that
they're
working
on.
So
how
do
we
do
all
this
so
pound?
Here?
Software
is
used
by
our
customers
to
integrate
data
and
analyze
data
sets
from
around
across
their
enterprise
and
is
used
by
operators
and
decision-makers
to
make
better
decisions
using
the
data
they
have,
and
we
do
this
by
offering
our
customers
two
data
platforms,
Palantir
Gotham
and
Palantir
foundry.
A
So
I
want
to
talk
a
little
bit
about
the
infrastructure
in
which
we
run
those
platforms
on
behalf
of
our
customers,
so
starting
in
2013,
we
started
offering
our
platforms
as
a
platform
as
a
service
model
to
our
customers,
built
on
top
of
our
first
generation
cloud
architecture
and
AWS.
The
world
has
changed
a
lot
since
then.
So,
two
years
after
that,
or
so
Caretti
x'
1.0
came
out
in
2015
and
as
time
went
on
the
needs
of
our
data
platform
and
the
technology
landscape
continued
to
evolve,
and
so
in
late
2016.
A
We
started
thinking
about
what
would
it
mean
to
rebuild
our
cloud
architecture
for
the
future?
And
what
would
we
change
our
first
generation
architecture
as
we
did
that,
and
there
was
a
number
of
things
that
were
influencing
our
decisions
in
how
we
thought
about
the
problem
at
the
time.
So
the
first
of
those
was,
we
started
running
into
scaling
limitations
and
hurdles
in
our
in
place.
A
In
2015,
there
was,
as
in
a
hypervisor
bug
or
two
that
led
Amazon,
to
need
to
reboot
almost
every
ec2
instance
in
the
entire
world
that
was
running
hvm
virtualization
and
when
this
was
announced,
our
infrastructure
team
panicked.
As
at
the
time
we
had
a
large
number
of
instances
that
had
been
running
for
longer
than
a
year.
We
had
large
volumes
of
the
ephemeral
volumes
and,
to
be
honest,
we
weren't
sure
what
would
happen
when
those
instances
got
rebooted.
A
A
lot
of
this
led
us
to
asking
what
we
thought
at
the
time
was
a
relatively
straightforward
question:
well,
what?
If
we
just
destroyed
and
rebuilt
every
host
in
every
environment,
every
48
hours,
if
we
did
this,
it
allow
us
to
treat
our
infrastructure
as
immutable
and
ephemeral,
which
means
we
wouldn't
need
an
in-place
patching
story
and
a
lot
of
the
challenges
that
we
had
in
the
fears
we
had
about.
Rebooting
instances
would
be
solved
because
we'd
be
doing
this
constantly.
A
A
So
in
January
of
2017
we
started
it
to
a
team
and
an
effort
internally,
we
called
Rubik
after
the
Rubik's
cubes
our
CEO
used
to
solve
and
leave
on
people's
desks
as
a
nod
to
kubernetes
and
Rubik
would
be
our
effort
to
rebuild
our
infrastructure
around
kubernetes
and
in
doing
so
treat
our
infrastructure
as
ephemeral
and
destroy
the
hosts
every
48
hours.
So
today,
Rubik
is
in
production
at
the
majority
of
our
customer
environments
around
the
world,
we're
on
many
many
kubernetes
clusters.
A
As
part
of
this,
we
have
environments
on
a
nine
AWS
regions,
I
think
soon
to
be
ten
and
four
different
continents
across
our
fleet.
We
have
more
than
2,500
nodes
at
any
given
point
in
time:
oftentimes
it's
above
3000
as
the
environments
scale
up
and
down
throughout
the
day.
The
clusters
vary
in
size.
We
have
clusters
from
tens
of
nodes
to
hundreds
of
nodes,
our
largest
oftentimes,
reaching
12
or
1300
nodes
at
the
peak
usage
and
because
we're
destroying
every
host
every
48
hours.
A
It
means
that,
on
a
24-hour
period,
we
tend
to
touch
about
10,000
ec2
instances
between
the
standard,
destroying
and
rebuilding,
plus
the
auto
scaling.
The
primary
workloads
that
we
run
on
top
of
kubernetes
today
are
our
distributed.
Computes
a
lot
of
Apache
spark
jobs,
and
because
these
these
pods
need
to
be
relatively
short-lived,
we
have
a
very
high
pod
churn
across
our
cluster.
So
it's
a
common
that
we'll
see
more
than
a
million
pods
being
run
in
the
course
of
a
24-hour
period.
So
I
mention
that
we
destroy
these
hosts.
All
the
time.
A
I'll
quickly
touch
on
a
framework
we
use
for
doing
so.
So
we
built
a
controller,
that's
responsible
for
destroying
and
rebuilding
hosts
in
every
environment.
It
follows
a
relatively
straightforward
three-step
process.
First
step
is
to
evaluate
all
of
the
ec2
instances
that
are
running
that
are
eligible
for
termination.
We
then
take
the
eligible
nodes
and
we
apply
a
set
of
constraints
to
those
to
make
sure
that
we
don't
do
things
like
take
all
three
entity
hosts
out
at
once.
That
would
be
very
bad.
A
We
down
select
all
of
the
hosts
that
are
eligible
to
the
hosts
that
we
can
safely
terminate,
and
then
we
send
them
to
an
actuator
which
will
be
responsible
for
marking
the
notice
on
scheduled,
draining
the
pods
and
so
forth.
So
we've
been
doing
this
for
a
year
and
a
half
or
so
there
have
been
a
number
of
really
great
benefits
that
we've
been
able
to
see
in
running
this
in
production.
A
That's
running
constantly
would
roll
out
the
mitigation
for
us,
and
it
was
the
thing
that
we
didn't
really
think
all
that
much
about
it
just
happened.
It
was
just
any
other
day.
The
second
big
category
benefits
we
saw
was,
as
we
started,
trying
to
do
auto
scaling.
We
realized
that
a
large
set
of
problems
that
one
would
usually
has
to
think
about
when
you
get
to
doing
auto
scaling.
We
had
already
solved
because
those
problems
are
the
same
types
of
problems
that
you
need
to
solve
when
you
start
treating
your
infrastructure
as
a
femoral.
A
These
are
things
like
the
hosts.
Initialization
time
starts
to
really
matter
your
ability
to
do
identity.
You
know
like
joining
a
domain,
getting
a
certificate,
those
kinds
of
things
and
your
ability
of
handling
networking
connections
gracefully
as
hosts
come
and
go.
So
a
lot
of
these
challenges.
Why
it's
gonna
talk
about
here
in
a
few
seconds,
but
for
us
what
it
meant
was
by
the
time
we
got
to
auto
scaling.
It
was
relatively
straightforward
path
for
us
to
get
it
working.
A
So
you
know
there
was
a
number
of
benefits.
There
was
also
a
large
number
of
challenges.
Vlad
was
that
tech
lead
on
Rubik
for
the
last
15
or
18
months.
He
was
on
the
front
lines
of
solving
his
problems,
so
I'm
gonna
hand
it
over
to
him
to
talk
through
some
of
the
technical
challenges
we
ran
into
and
some
messages
we
found.
B
Folks,
with
four
challenges:
first,
one
is
gonna
be
about
leave
their
election
services.
The
second
one
is
gonna,
be
about
cloud
scaling.
Third,
one
host
boot
up
time,
and
why
doesn't
matter
when
your
thermal
infrastructure
and
then
fourth
one
it's
gonna,
be
about
edge
and
container
networking,
so
about
consensus
and
leader
election
based
services?
It's
really
important
not
to
kill
your
leader
and
in
such
a
service
when
you're
taking
now
hosts,
for
example,
the
most
important
service
that
we
can
think
of
is
HCB.
B
If
you
kill
the
HD
leader
without
any
graceful
termination,
the
API
server
is
just
going
to
504
a
couple
of
for
a
small
period
of
time.
In
order
to
solve
this,
we
took
an
approach
where
we
introduced
a
framework
that
we
call
the
pre
termination
actions
framework.
Here
we
deploy
a
binary
that
can
either
be
a
sidecar
for
a
container
or
binary
and
on
the
actual
hosts,
when
a
binary
starts,
it
does
watch
to
the
API
server
it
watches
for
its
own
node
and
then,
when
terminator
selects
the
node
to
be
terminated
at
the
label.
B
In
our
case
termination
pending
equals.
True,
the
the
binary
is
going
to
detect
that
that
label
was
added
and
execute
certain
action.
In
its
the
case,
it's
going
to
call
the
move
leader
endpoint.
So
then
we
can
terminate
the
HIV
leader
gracefully.
The
API
servers
gonna
still
serve
request.
Happily
in
that,
in
that
time
frame
another
service
that
we
use
in
Rubik,
it's
hash
code
fault.
We
use
it
for
secrets,
management
and
identity
management.
Vault
is
deployed
as
a
free
node,
one
node,
being
the
primary
and
two
other
nodes
being
secondary
nodes.
B
Here
we
chose
the
same
approach
before
killing
the
vault
leader.
We
just
call
it
step-down
endpoint.
This
is
gonna,
move
the
leadership
of
the
vault
cluster
to
another
node,
and
then
we
take
down
the
vault
node
itself,
which
now
is.
It
is
a
standby
one
in
in
the
cluster
other
services
that
are
present
in
the
kids
ecosystem
that
are
doing
leader
election
are
the
controller
manager
and
the
scheduler
for
these
two.
B
So
moving
to
cloud
challenges,
we
found
that
the
clouds
not
actually
infinite,
as
we
believed
at
start.
We
had
some
problems
when
Amazon
launched
their
fifth
generation
instance
types.
We
decided
to
use
them
the
next
week
after
they
were
launched
and
in
many
scenarios
Amazon
wouldn't
even
have
one
or
two
VMs
to
provision
for
us
when
we
were
doing
the
normal
termination
cycle.
B
So
we're
detaching
one
machine
48
from
the
auto
scaling
group
and
then
waiting
for
the
auto
scaling
group
to
bring
another
one
up,
Amazon
with
just
airing
out
that
in
gas
that
they
don't
have
any
capacity,
and
we
should
just
like
wait
it
out
or
swap
the
instance
type
around
November.
Last
year,
Amazon
launched
the
feature
for
the
auto
scaling
group
service
called
multi
instance
type
of
skinny
groups.
This
allows
an
operator
to
add
more
instance,
types
the
launch
template
they're
using
for
their
own
skin
groups.
B
So,
for
example,
you
can
have
free
instance
types
over
there
and
if
Amazon
can't
provision
the
first
time
for
you,
it's
gonna,
it's
gonna
move
to
the
second
one
and
so
forward.
So
this
helped
a
lot
in
deploying
Rubik
to
some
of
our
like
bespoke
Amazon
regions,
where
a
capacity
wasn't
always
available
on
top
of
not
having
capacity.
Always
we
found
out
that
sometimes
you're
running
on
bad
Hardware.
Usually
we
touch
around
10,000
hosts
per
day.
We
terminate
them
bring
another
one
up.
B
B
When,
across
our
fleet
we
turned
around
70,000
hosts
per
week,
so
we
get
approximately
around
100
or
150
events
across
our
fleet.
Overall,
in
this
case,
we
just
changed
or
tweaked
the
terminator
code
to
detect
this
instances
like
early
on
in
the
in
the
selection
process,
and
then
we
just
kill
them
without
going
through
the
normal
pipeline
as
a
real-world
example.
Here
you
can
think
that
one
of
these
instances,
that's
running
on
bad
hardware,
can
be
our
API
server.
B
Moving
next
to
host
boot
up
time
when
you're
running
a
formal
infrastructure,
it's
actually
important
to
design
for
this
problem
in
our
environment,
our
hosts
live
up
to
48
hours,
so
spending
a
lot
of
time.
Just
setting
up
the
hosts
for
doing
actual
work
seems
wasteful
in
the
previous
cloud
architecture
would
take
us
around
40
to
40
minutes
to
bring
a
host
up,
and
that's
just
time
left
on
the
table
not
running
any
and
user
workload
on
each
host.
We
run
for
static
pods
for
various
admin
purposes
like
collecting,
logs
or
monitoring
the
host
itself.
B
B
This
meant
that
we're
spending
a
lot
of
time
on
our
gateway
machines,
just
doing
network
traffic,
pulling
these
images
over
and
over
again
to
overcome
this
problem,
we
deployed
then
in
vbc
in
cluster
image
registry
and
all
the
other
image
pools
are
locally,
that
we
faster
and
you're
not
over.
Subscribing
the
Gateway
machines
for
network
traffic.
Another
flavor
of
a
host
boot
up
problem
can
be
establishing
the
identity
of
the
host
itself.
B
B
Back
in
the
day,
all
our
hosts
were
asking
vault
directly
to
generate
the
certificate
and-
and
the
key
that's
meant
that
the
vault
cluster
itself
was
generating
that
secret
material
and
then
passing
it
on
to
the
host
itself.
This
was
putting
a
lot
of
load
volt
cluster,
just
like
the
leading
entropy
and
doing
a
lot
of
CPU
cycles
just
to
generate
secret
data.
C
B
Our
VPC
design
is
pretty
standard.
Follows
follows
couple
of
industry,
standard
practices
we
deploy
across
free
availability
zones
and
each
is
configured
the
same.
It
has
one
subnet,
which
is
called
the
DMZ
subnet
and
the
subnet.
We
have
our
gateway
machine
that
lives
over
there
and
then
all
the
other
subnets
have
their
default
route.
Pointing
to
this
gateway
machines
would
kill
the
gaiter
machine
normally,
as
would
be
another
node
in
the
cluster,
so
every
48
hour
the
host
dies.
B
This
means
that
all
the
egress
connections
from
the
cluster
itself
to
the
outside
world
are
going
to
be
terminated.
We
need
the
way
to
re-establish
connectivity,
so
we
just
reused
our
pre
termination
access
framework
to
reach
out
to
Amazon
when
the
label
was
added
to
the
Gateway
machine,
scan
the
the
route
tables
see
the
ones
that
will
be
invalidated
and
then
just
move
the
route
to
another
gateway
machine
from
the
other
to
a
disease
that
that
are
still
alive
and
happy.
B
For
this
one,
we
opted
again
for
a
predetermination
action
to
inject
a
couple
of
IP
table
rules
that
were
sending
back
TCP
reset
packets
to
existing
connections,
to
signal
them
that
hey
this
host
is
going
to
die,
I
need
to
connect
to
another
one.
That's
that's
available.
We
also
have
another
IP
tables
rule
that
just
refuses
connections,
so
it
forces
the
client
to
connect
to
another
host.
That's
that's
available,
so
going
to
continue
networking
here.
B
The
journey
has
been
interesting
so
far
when
we
started
rubrics
were
running
calico
as
our
CNI
plugin
we're
running
around
calico
2.5
and
after
some
time
when,
when
we
started
like
scaling
up
and
down
our
cluster,
we
found
out
that
calico
didn't
have
enough
IPS
to
allocate
the
pods.
It
was
just
tearing
out
saying
that
there
are
no
more
IPS
available,
but
we
didn't
have
that
many
parts
in
the
system
to
just
exhaust
our
full
cider
block.
B
We
did
some
investigation
and
we
found
out
that
the
HDD
that
calico
was
using
was
full
of
data
that
was
outdated.
It
had
leases
for
IP
is
that
weren't
available
anymore.
Those
parts
were
like
long
killed.
So
to
overcome
this
problem,
we
just
deployed
the
kid
scrunch
out
that
every
so
often
would
walk
the
HCB
key
space
would
validate
if
the
data
from
A
to
B
reflects
the
real
world
and
just
nuke
all
the
audio
at
IP,
all
all
the
old
IPS
to
to
give
them
back
together.
B
When
we
encountered
this
issue,
we
decided
to
look
in
the
community
to
see
like
how
what
other
alternatives
are
available
for
another
CNI
plugin,
and
we
found
out
the
lift,
CNI
plugin.
The
main
differences
between
these
two
is
that
the
left
one
doesn't
use
an
overlay
network
as
Calico
does
it's
doing
native
VPC
routing,
so
every
part
IP
is
an
actual
IP.
That
amazon
knows
about
so
they're
handed
out
from
your
VPC
cyberspace.
B
We
swapped
and
start
using
the
left
one,
but
this
didn't
come
with
free
the
main
issues
we,
the
main
issue
in
content
here
is
we're
doing
somewhat
bigger
scale
of
events
like
bringing
three
to
four
hundred
nodes
in
service.
At
the
same
time,
the
lift
plug-in
would
reach
out
to
Amazon
to
allocate
Annie
and
I
then
attached
the
United
to
the
host
and
the
Hannah
attached
I
pieced,
that
en
isopods
can
run
on
the
host.
B
So
when
you
imagine
doing
300,
400
create
and
modify
requests
in
parallel
to
Amazon,
you
just
get
the
API
rate
limited
to
fix
this.
We
opted
in
to
swapping
the
order
of
operations.
So
now,
when
a
pod
comes
on
the
host,
we
allocate
any
ni,
and
then
we
reallocate
the
IPS
that
we
know
we're
gonna
use
on
that
network
interface.
We
usually
know
where
the
pod
st
on
each
note
that
that
we
run
so.
B
This
is
pretty
fine
up
up
until
now,
but
we
utilized
after
moving
from
calico,
we
lost
the
ability
to
do
network
firewalling
out
of
the
box,
so
we
decided
to
take
it
down
upon
ourselves
to
build
the
solution.
For
this.
We
build
the
solution
in
house,
but
deploying
it
to
larger
clusters.
We
encountered
certain
scale
problems
mostly
around
node
and
party
formers.
The
solution
basically
deploys
an
agent
on
every
host
and
the
agent
watches
pods
and
watches
nodes
from
the
API
server.
B
But
this
puts
a
lot
of
pressure
on
the
API
server
starts,
sending
back
a
lot
of
information,
the
network
traffic
outbound
from
the
API
service,
just
it's
just
insane.
So
we
went
back
and
looked
around
the
community
to
see
how
people
are
solving
this.
This
issue
and
we
found
the
cilium
project
and
currently
we're
partnering
with
the
folks
from
cilium
to
bring
all
the
good
things
from
the
left,
CNI
plugin.
Basically,
the
E&I
approach
that
the
left
folks
took
and
we're
gonna
start
using
the
left
security
stack
as
well
in
in
the
future.
A
Line
so
you
know,
after
doing
this
interruption
for
about
18
months,
you
know
we
set
out
to
solve
a
set
of
problems
that
we
had
at
the
time
thinking.
This
would
be
a
lot
easier
than
it
ended
up
being
and
for
us
this
was
a
pair
of
time,
a
paradigm
change
in
how
we
approach
infrastructure
and
with
any
paradigm
change
you're
going
to
run
into
a
number
of
unexpected
issues,
and
so
we
talked
through
some
of
the
issues
we
ran
in
today.
There
were
many,
many
more
that
would
be
happy
to
talk
about.
A
If
anyone
is
interested
for
us,
we
found
that
you
know
investing
in
some
relatively
simple
controllers
or
solutions
like
the
pre
termination
action
framework
allowed
us
to
solve
many
of
these
problems
in
a
repeatable
way,
and
it
took
us
probably
longer
to
to
do
that
than
we
should
have
in
retrospect
and
so
for.
Anyone
who's,
considering
you
know,
taking
an
ephemeral
or
a
mutable
infrastructure
approach
and
running
pretend
ease
on
it.
I'd
encourage
you
to
think
about
these
things
upfront.
A
You
know
we
realize
a
lot
of
benefits
by
taking
this
approach,
I.
Think
in
retrospect,
looking
back
over
the
last
two
years,
knowing
what
we
know
today,
we
would
definitely
take
a
similar
approach.
We'd
probably
invest
in
a
few
areas
a
little
bit
earlier
on
in
the
process
than
we
did,
but
we've
been
able
to
realize
a
lot
of
great
benefits
in
doing
this,
so
it
didn't
come
without
cost
as
I've
mentioned.
A
So
that's
what
we
have
I'll
point
out
that
we
have
a
booth
in
the
vendor
showroom
if
you're
interested
these
topics
are
interested
learning
more
we're
happy
to
go
into
a
lot
more
depth
for
anyone
interested.
We
also
are
starting
a
blog
series
on
Rubik's
and
our
adventures
in
kubernetes,
so
that
others
can
learn
from
some
of
our
failures
over
the
last
two
years.
So
we
publish
the
first
blog
post
last
week
and
there
should
be
a
number
more
coming
up
and
we're
happy
to
turn
it
over.
For
anyone
who
has
questions.
Thank
you.
D
A
A
good
question,
so
this
is
one
of
the
first
problems
we
ran
into,
so
we
run
a
lot
of
spark
applications
and
so
anyone
who's
familiar
with
spark.
It's
you
have
a
driver
and
you
have
several
executor
'he's,
the
cost
of
destroying
an
executor.
You
may
have
to
do
a
little
bit
of
repeat
work,
the
cost
of
destroying
a
driver.
A
You
have
to
start
the
entire
job
over
and
so
many
of
these
spark
applications
can
run
for
very,
very
long
periods
of
time,
and
so
we
built
a
framework
for
our
stateful
services
as
well,
where
we
add,
essentially
a
health
endpoint,
sub
notion
of
health,
on
top
of
readiness
and
liveness
and
for
drivers.
We
let
them
say:
please
defer
draining
me,
please
deferred
killing
me,
and
so
our
termination
framework
actually
takes
us
into
account.
A
So
as
we're
filtering
out
all
of
the
eligible
nodes
into
only
the
ones
that
we
can
actually
terminate,
we
allow
some
pods
to
defer
termination
for
up
to
24
hours,
and
so
in
theory,
every
host
only
lives
48
hours,
but
we
allow
at
a
24-hour
grace
period
up
necessary
before
we
before
we
terminate
it
forcefully.
So
some
hosts
can
live
up
to
72
hours,
but
we
basically
take
this
into
account
to
try
to
avoid
that
with
some
limits
so
make.
A
Unfortunately,
no
we
we've
run
into
a
couple
problems
with
storage
over
time.
So
most
of
the
storage
we
use
today
is
is
ephemeral,
so,
like
spark,
jobs
will
spell
to
disk
and
so
forth.
But
that's
it's
like
a
ephemeral
data.
So
it's
fine
for
running
at
CD.
When
the
SD
hosts
get
terminated.
We
built
something
kind
of
similar
to
persistent
volumes
versus
volume
claims
where
we
will
detach
the
EBS
volume
from
the
old
host
and
attach
it
to
the
new
host
that
comes
up.
A
That's
going
to
run
at
CD,
so
that
the
data
will
follow
that
city
around
we
run
at
City
as
a
static
pod,
but
obviously
we
can't
run
it
as
a
stateful
set
since
it
underpins
the
API
server.
So
there
are
some
tricks
like
that
that
we've
done,
which
we
know
we've
had
to
invest
in
that
work
upfront.
We
don't
run
that
we
need
stateful
sets
today
we
tried
a
year
ago
and
ran
into
some
issues
with
auto
scaling
groups
having
capacity
that
comes
up
at
the
wrong
availability
zone.
A
B
And
what
are
your
pipelines
look
like
for
building
the
images
yeah
I
can
get
a
one,
so
we
build
a
foundation
ami,
that's
just
basically
running
Packer
on
top
of
Ubuntu
Bionic
and
from
there
we
publish
that
ami
internally
and
then
every
type
of
host
that
we
run
Rubik.
We
have
I
think
five
of
them,
so
Gateway
machines
are
a
bit
special.
It's
needs
a
bit
special
masters
which
are
API
server,
controller
manager,
scheduler
and
then
just
like
the
plain
worker.
B
So
we
rebate
the
foundation,
a
mine
like
different
packages
that
we
need
or
receiving
different
docker
images,
and
then
we
just
like
publish
that
and
our
terraform
code,
when
it
runs,
finds
these
like
the
certain
type
of
a.m.
eyes
and
just
much
lunch
of
them.
So
as
a
bluff
is
just
running
Packard
of
Ubuntu
and
then
like
we
package
it
internally
afterwards,.
F
A
B
A
B
So,
basically,
a
new
machine
would
hijack
the
identity
of
an
old
one,
and
then
the
controller
manager
would
indeed
the
nodes
or
you're
stuck
with
with
just
an
easy
to
node
running
a
cubelet,
but
the
cubelet
wouldn't
read
register
itself.
So
that's
like
one
form
of
from
relative
problem
scale
problem
I,
think
we
wanted.
We
hit
a
new
scale
problem.
It's
like
a
in
the
networking
stack
like
the
networking
challenges
that
I
discussed
with
calico
other
than
that
just
when
we
launched
workloads
and
all
of
them
try
to
talk
to
the
API
server.
B
A
I
think
the
the
edge
networking
challenges
the
glad
talked
about
like
ingress
and
egress
from
the
cluster.
Those
are
largely
based
on
like
the
ephemeral
nature
of
them.
That's
you
know
it
has
little
to
do
with
scale
itself,
so
I
think
those
anyone
who
does
this,
who
also
has
single
entry
points
and
egress
points
in
their
environment,
will
also
have
to
face
other
networking
like
the
container
networking
challenges
were
some
mix
and
it
sort
of
changed
over
time
so
calico.
It
was
kind
of
the
ephemeral
nature.
The
lift
stuff
we
run
into.
A
Today's
is
more
related
to
it's
more
related
to
scale,
though
the
problems
are
exacerbated
by
the
fact
that
there's
so
much
change
in
the
environment.
It
just
makes
it
harder
and
then
yeah,
it's
sort
of
a
mixed
I
would
say
the
first
year
was
all
ephemeral
stuff.
The
last
year
it's
been
a
mix
of
scale
and
if
I'm
wrong
challenges.
F
G
Very
good
talk
and
I
do
upload.
Your
bond
approach
into
this
I
can
only
imagine
meetings
how
they
went.
So
my
question
is
related
with
provider
cloud
provider
bugs
it
sounds
like
your
approach
in
the
lifecycle
of
machine
can
actually
propagate
them
instead
of
contain
them
into
one,
let's
say
or
two
instances,
and
how
does
this
look
like
in
the
monitoring
and
alerting
for
you.
A
A
A
So
I
think
one
of
the
big
challenges
for
us
with
our
first
reiation
cloud
was
we
had
very
rigid
rules
for
monitoring
and
alerting,
so
when
a
host
would
disappear,
for
instance,
it
would
like
ring
all
the
bells
which
is
so
we
couldn't.
Autoscale
are
first
generation,
first
generation
cloud
and
the
second
generation
one
we
it
with
Rubik.
A
We
moved
to
our
sort
of
a
we
care,
a
lot
less
about
every
host
and
we
care
more
about
an
auto
scaling
group,
and
so
we
will
alert
when
the
of
instances
in
in
an
auto
scaling
group
drops
below
some
threshold,
but
we
don't
care
about
a
given
host.
So
if
a
host
comes
up
and
it's
bad
like
there's
Amazon
hardware
issues,
we
just
kill
it.
It's
just
like.
A
We
don't
even
care
just
like
kill
it
move
on
with
their
lives,
and
it's
only
once
the
number
of
those
problems
crosses
some
threshold
that
we
start
worrying
about.
It
doesn't
make
sense
I.
This
is
true
largely
for,
like
the
large
pools
of
worker
nodes,
we
run
when
one
of
the
routers
is
becomes
unhealthy.
We've
had
issues
over
time
where,
before
the
pre
termination
action
framework,
we
would
have
an
issue
where,
if
the
router
became
bad,
your
only
egress
point
from
the
availability
zone
would
be
down,
so
you
lose
the
entire
AZ.
A
H
B
I
think
so,
usually
our
developers
like
building
images
internally
and
we
use
at
factory
Jeff
our
artifact
internally,
so
we
published
those
in
our
corporate
registry
and
then
on
top
of
that
we
have
a
solution
that
allows
external
clusters
from
the
internet,
to
just
authenticate
with
that
corporate
network
registry,
to
pull
the
images
and
in
our
V
PC
on
our
gateway
machines.
We
deploy
we'll
play
catch
registry.
Basically,
so
the
way
when
the
when
the
gate
of
machine
comes
up,
its
cache
is
gonna
be
empty.
B
For
the
moment
all
the
clients
are
going
to
reach
the
cache.
The
cache
is
gonna
reach
upstream,
it's
gonna
pull
the
limits
and
then
all
the
image
is
gonna
be
cached
on
the
routers
and
it's
gonna
happen
across
like
all
free
AZ's
when
those
machines
cycle
basically
now
we're
looking
into
storing.
Actually
the
cache
on
ABS
drives
and
do
the
same
dance
as
we
do
it.
Look
at
CD
like
it's
gonna,
follow
the
it's
gonna
follow
the
node
when
it
comes
up
and
down
just
to
sustain
on
unavailability
of
the
of
the
central
registry
itself,.
A
Today,
it's
also
helpful
that
we
have
a
relatively
few
number
of
images
that
are
run
in
our
environment,
like
the
vast
majority
of
pods
run,
one
of
five
sparked
images,
for
instance,
and
so
once
ok,
the
cache
warms
quite
quickly
and
it
involves
slowly
over
time,
and
so
we
get
a
lot
of
value
out
of
that,
as
we
run
more
and
more
of
our
infrastructure.
More
and
more,
like
our
micro
services
in
the
environment,
we
expect
that
to
you.
E
A
I
B
B
Think
we've
hit
a
couple
of
problems
with
the
Amazon,
the
VPC
dns
resolver
rate,
limiting
us,
because
we're
doing
so
many
queries
but
I
suspect
those
are
just
because
of
the
empaths
problem
and
the
rest
people
are
hitting
so
we're
planning
to
deploy
the
local
node
cache
to
like
not
do
so
many
queries
upstream
and
just
like
hit
the
hit
the
local
cache
but
other
than
that.
I
think
that's
it,
and
in
this
indiana
space
that
we
had.