►
Description
How Box Runs Containers in Production with Kubernetes - Sam Ghods, Box
Everyone is talking about using containers to deploy microservices, but exactly how all the pieces fit together is still a bit of a mystery. In this talk, we'll go over how Box went from bare-metal infrastructure that took six months to deploy a new service to using Kubernetes and Docker to deploy services in under an hour. Learn the practical details of how Box implemented containers in production, including packaging, deployment, service discovery, secret management and more, and how you can use our techniques to get containers up and running in your own organization.
About Sam Ghods
Sam Ghods is a cofounder at Box where he is responsible for service infrastructure. Prior to his current role, Sam led software architecture at Box for eight years as Vice President of Technology. Previously, he was an engineer and cofounder at two other startups.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
We've
now
today
have
four
clusters:
three
are
in
production,
serving
production
traffic.
All
of
them
are
on
bare
metal
in
our
own
data,
centers
and
they're
about
50
nodes,
each
with
more
being
added.
Every
week
now
and
they're
running
several
micro
services
that
power
box
comm.
So
again,
basically
this
talk
is
all
about.
You
guys
can
just
pick
my
brain.
Ask
any
questions.
A
You
want
I'm
going
to
just
cover
kind
of
a
high
level,
what
we
do
to
enable
our
developers
to
use
kubernetes
to
deliver
production
micro
services,
and
then
you
know
any
of
the
things
you
guys
been
running
into
trying
to
deploy
teddies
I'd
love
to
either
hear
about
or
feel
free
to
ask
questions
things
like
that.
So
let's
talk
about
what
boxes
for
just
a
second,
so
we
provide
sharing
and
collaboration
services
for
everyone
from
individuals
to
large
businesses
and
governments
all
around
the
world.
A
You
need
a
lot
of
compliance
and
security
and
requirements,
and
things
like
that
that
we
really
specialize
in
and
so
as
of
today,
we're
spending
a
lot
of
time.
Thinking
about
how
do
we
not
only
run
stuff
in
our
own
data
center,
but
also
burst
a
public
cloud?
So
that's
why
kubernetes
is
really
very
relevant
for
us.
So
let's
talk
about
how
a
developer
begins
their
journey
in
launching
a
production
micro
service
on
kubernetes.
A
So
first
they
start
with
their
source
code
repo
and
they
have
a
docker
file
in
there
and
then
they
they
use
that
we
use
Jenkins
to
build
a
docker
image,
and
then
we
push
that
to
artifactory.
So
first,
let's
talk
a
bed
which
is
that
we
use
as
our
docker
registry.
So
first,
let's
talk
about
the
docker
file
piece,
because
there's
kind
of
like
this
dirty
little
secret
about
docker
files,
where
you
dr.
A
bells,
expect
that
the
the
image
that
you
build,
your
software
in
is
the
same
one
that
you
are
going
to
run
your
software
in.
So
if
you
imagine
like
you,
may
want
one
image
that
has
your
JDK
and
another
image
that
has
a
JRE
so
that
your
your
run
run
image,
doesn't
have
all
the
the
compilation
or
build
tools
that
you
may
have,
but
that's
generally
not
what
dr.
files
expect.
They
expect
one
docker
file.
A
You
have
the
whole
thing
in
there
you
just
throw
in
all
the
different
tooling
and
then
that
seam,
as
you
end
up
shipping,
so
we
basically
wrote
a
bunch
of
scripting
and
tooling
that
we
haven't
released
yet,
but
if
it
reaches
a
critical
mass,
we
will
where
we
have
a
docker
file,
build
as
an
optional
component
that
basically
puts
artifacts
in
a
known
a
mounted
location
into
the
salt
into
the
image
and
then
a
docker
file
that
takes
those
artifacts
and
is
able
to
copy
them
in.
So
you
see
the
place
that
we
put.
A
A
Maybe
it
is
fine
to
just
shove
all
your
stuff
in
side
of
the
bay
stalker
image,
because,
as
you're
making
layers,
you
can
then
make
a
layer
that
only
has
your
runtime
stuff,
so
you're
shipping
kind
of
a
lightweight
piece,
but
there's
like
potential
security
concerns
and
things
of
having
all
your
build
tooling
in
all
your
production
environments.
So
it's
just
something
to
think
about.
A
We
looked
all
over
if
anybody
knows
of
any
tooling
that
handles
this
for
you
or
makes
this
easy,
we
couldn't
find
anything
the
source
to
image
stuff
that
Red
Hat
is
released
with
OpenShift
is
pretty
interesting,
but
it
doesn't
seem
to
do
this
either,
so
we're
still
wondering
whether
it's,
whether
it
really
you're
worth
the
cognitive
overhead
versus
just
using
a
darker
file,
but
it's
just
something
to
think
about,
especially
if
you're
in
a
high
security
or
highly
sensitive
environment,
then
the
other
thing
I
had
up.
There
was
artifactory.
A
A
I
mean
people
we
needed
had
the
need
for
hosting
of
anything
from
jars
to
gems,
to
you
know,
rpms
to
whatever
it
is
for
a
very
long
time
and
there's
a
pretty
sensible
feature
set
that
gets
built
around
that,
whether
it's
redundancy
or
h.a,
across
regions
or
really
sophisticated
api's
to
query
or
your
registry
or
query
your
binary
repository
and
artifactory
had
all
those
things
and
I
just
added
something
that
made
it
compatible
the
registry
and
we're
liable.
This
is
a
really
full-featured
solution,
we'll
just
go
with
that.
A
Instead
of
waiting
for
all
the
other
registries
to
catch
up
to
that
feature
set,
so
we
would
recommend
just
choosing
someone
who's
done
this
before
I
know,
we
actually
were
using
a
sauna
type
nexus
as
our
artifice,
our
binary
repository
before,
but
they
took
too
long
to
add.
Docker
registry
supports,
so
we
just
switched
over
to
artifactory,
and
so
the
API
is
and
the
extensibility
of
the
registry.
You
lend
a
lot
of
really
interesting
and
creative
tooling.
A
So,
like
the
tooling
I
talked
about,
we
really
want
all
your
components
to
be
able
to
be
very
programmable
against,
because
the
future
we
should
be
in
is
one
where
a
new
like,
let's
say,
heartbleed,
or
something
like
that
releases.
What
we
should
be
in
a
world
where
completely
automatically
all
your
docker
images
rebuild
and
then
kubernetes
basically
does
a
rolling
grade
of
all
of
your
software.
Just
alerting
you
if
something
goes
wrong
or
if
one
of
the
nodes,
pods
or
whatever
doesn't
come
up
so
like.
A
We
should
be
in
instead
of
manually
updating
these
these
things
and
manually
doing
deployments.
So
you
that
program,
ability
for
every
major
component
of
your
infrastructure
and
I
would
call
a
registry.
A
pretty
significant
component
should
should
be
a
high
priority
and
that's
kind
of
proven
by
one
of
the
products.
Artifactory
is
releasing
now
called
x-ray,
which
is
really
really
cool.
It
basically
shows
you
all
your
artifacts
and
how
they
all
relate
to
each
other
and
what
things
contain.
What
so?
Let's
say
you
published
a
jar,
and
then
you
publish
a
docker
image.
A
Well,
you
can
actually
say
well,
these
are
all
the
docker
images
that
still
have
that
jar
and
if
you
imagine,
an
integration
that
you
could
build
between
kubernetes
and
artifactory,
or
really
any
dr.
registry
to
a
certain
extent,
then
you
could
actually
have
it
say
in
in
real
time.
Right
now
on
all
my
clusters
and
all
my
data
centers
again,
regardless
of
cloud
provider
which
instances
of
my
app
are
running
an
old
version
of
of
whatever
package.
It
is
that
I'm
trying
to
update.
A
So
that's
a
really
exciting
thing
about
about
what
x-rays
is
basically
scanning
everything
all
the
time.
It's
not
basing
it
off
of
manual
metadata.
It's
actually
going
in
and
scanning
your
docker
images
and
your
packages
and
coming
up
with
what's
in
each
one
and
creating
all
those
relationships
in
the
graph
inside
your
binary
repository.
So
we're
really
excited
about
that.
It's
in
still
pretty
early
stages.
A
It
only
works
with
a
few
package
types
but
they're
building
it
out
more
we'd
love
for
that
to
be
open
and
I
think
there
are
some
projects
in
this
chibs
on
that
front,
but
but
x-rays,
one
working
at
now
and
then
finally
secure
cloud
hosting
like
we
really
didn't
want
to
deal
with
binary
hosting.
So
we
have
them
run
it
for
us
and
it's
really
cool.
They
have
a
lot
of
tiers.
If
you
pay
enough
money,
you
can
basically
have
them
run
it
in
your
own
AWS
account.
A
So
you
can
really
be
confident
about
a
lot
of
the
security,
so
that's
artifactory,
so
yeah.
So
as
we
talked
about
with
our
source
code,
they
have
a
docker
file
and
potentially
docker
file,
build,
build
your
docker
image
and
Jenkins,
but
just
clicking
a
button
and
some
of
them.
If
you
get
pushed
then
automatically
builds
the
image
for
you
and
pushes
that
to
artifactory
another
quick
thing
about
some
of
the
scripting
we
wrote
here.
A
You
don't
want
them
to
like
futz
around
with
the
different
docker
build
commands
and
the
docker
push
and
do
I
tagged
it
correctly
to
name
it
correctly.
You
know,
because
if
it's
not
standardized
and
how's
my
building
down
the
build
tooling
down
the
line
going
to
be
able
to
find
those
images
to
automatically
deploy
them
and
it
potentially
like
continuous
deployment
type
flow.
A
So
we
use
these
labels
in
the
docker
file
as
metadata
and
you're
able
to
query
those
later
so
it'sit's
a
bit
of
a
hack,
but
it
works
really
well
to
prevent
needing,
like
another
config
file
inside
your
repo
to
describe
what
the
image
should
be
named
or
to
describe
different
parameters.
You
want
to
give
the
image
like
what
tag
you
should
explicitly
do,
so
we
have
like
an
automated
system
for
generating
tags
based
on
the
get
hash
and
things
like
that
so
yeah.
So
that
was
another
thing
I
wanted
to
mention.
A
A
So,
for
example,
it
has
well
I'll
show
the
example
in
a
second,
but
basically
it
explains
all
the
ways
of
all
the
all
the
parameters
of
how
you
might
want
to
run
or
app,
including
command
and
environment
variables
and
command
line
parameters,
or
things
like
that
and
we'll
look
at
one
in
a
second.
But
what's
key
here
is
that
we
are
templating,
so
we
have
a
centralized
repository
that
we
publish
all
these
things
and
there's
templating
that
helps
people,
reuse,
different
parts
of
their
configs
or
pods
or
things
is
we
all
know.
A
Pods
are
a
atomic
collection
of
containers
that
that
are
atomically
deployed
and
represent
kind
of
an
instance
of
your
application.
But
what's
tricky
is
actually
being
able
to
provide
reusable
components
that
people
can
build
their
pods
out
of?
And
you
also
want
a
template
so
that
you
can
issue
slightly
different
specs
to
the
different
clusters
as
necessary.
For
example,
if
you
have
a
non
production
cluster
and
then
you
have
the
production
clusters
or
you
have
different
environments
that
you
want
to
publish
to
dev
staging
prod
and
so
on.
A
You
don't
want
to
have
to
have
text
files
that
have
the
entire
yam
or
json
written
out
every
time,
because
that
gets
very,
very
manual
to
make
sure
that
all
the
environments
are
running
the
same
thing.
So
what
we
do
is
we
use
templating
and
we
use
this.
A
specific
library
called
JSON
net,
which
is
another
Google
project
and
JSON,
is
essentially
a
temple
in
language
for
JSON
and,
what's
great
about
that,
is
that
it
produces
JSON.
A
It
doesn't
produce
text,
so
you
can
use
things
like
mustache
or
you
know
your
favorite
tempting
language
or
whatever
it
is.
But
the
big
problem
with
those
is
that
they
produce
text
instead
of
for
using
JSON.
So
they
have
no
idea
if
you're,
if
you're
a
JSON
or
whatever
you're
outputting
is,
is
malformed
or
it's
syntactically
invalid
things
like
that,
but
JSON
it
will
only
compile
and
produce
syntactically
valid
JSON,
which
is
really
really
nice.
So
this
is
an
example.
A
Json
8
file
for
our
hello
world
app
pulled
directly
from
our
repo
and
at
the
top
you
can
see
so
something
really
important
here:
you're
going
to
have
a
temptation
when
you
first
start
with
templating,
to
make
a
full
abstraction
of
everything
below
of
all
the
kubernetes
core
objects
like
deployment
services,
secrets,
Figg
maps
you're
going
to
want
to
make
your
own
like
box
deployment
or
box
service,
or
your
company
name,
config
map
or
whatever.
It
is,
and
I
think
the
jury's
still
out.
A
But
we
tried
that
first
and
we
quickly
rolled
it
back
and
we
undid
all
our
configs
and
made
them
look
much
more
like
the
native
objects,
because
the
native
objects
are
that
complex
and
configurable.
For
reason.
It's
because
many
of
your
developers
may
want
to
tweak
different
things,
and
the
problem
is
that
each
while
90%
of
all
the
specs
will
look
the
same
at
any
given
time.
The
10%
that's
different
will
likely
be
different
between
each
team
and
engineer.
A
So
we
kind
of
byte
the
learning
curve
and
we
say:
okay,
fine
people
are
going
to
have
to
learn
these
core,
creating
objects
because
honestly
they're
pretty
good
and
they're
pretty
well
designed.
So
we
we
have
the
general
structure
of
these
all
look
like
the
kubernetes
objects.
So
there
isn't
that
much
abstraction
between
the
developer
and
kubernetes.
So
you
got
kind
deployment
and
then
you
have
a
spec.
We
have
all
the
other
fields,
I've
kind
of
taken,
a
lot
of
them
out,
but
now
the
spec
field,
you
have
replicas
three.
A
So
if,
for
those
who
don't
know,
deployment
object
is
essentially
the
way
to
tell
kubernetes
run,
you
know
n
instances
of
my
application,
but
if
I
ever
change
anything
about
the
spec
do
a
rolling
upgrade.
So
that's
different
than
like
a
replica
set
of
replication
controller.
Where,
if
you
change
it,
you
have
to
actually
go
kill
each
one
manually
and
then
it'll
respond
as
a
new
one
deployment
object
is
a
rolling
upgrade
for
you,
so
that's
been
really
great
to
do
and
it
really
helps
with
declarative
configuration
as
you'll
see.
A
So
here
we
have
three
replicas.
We
have
the
template
and
in
the
containers
section
you'll
see
that
under
image
we
start
to
diverge
from
JSON
and
we
actually
call
a
function.
That's
in
a
library
file
in
that
in
this
template,
apposite
or
e.
So
this
we
have
this
library
that
we
have
it's
like
template,
library
called
image,
and
that
has
a
URL
function
which
will
generate
the
URL
for
the
image
and
it
takes
the
application
name
and
then
it
lets
you
switch
on
which
environment
you're
generating
a
spec.
A
For
so,
we
essentially
have
a
make
file
that
iterates
over
these
templates
once
per
environment
and
those
those
environments
get
mapped
to
clusters.
Basically,
so
this
switch
allows
you
to
generate
a
slightly
different
JSON
for
each
cluster,
while
all
the
other
pieces
stay
the
same,
so
then
you
see
command,
that's
going
to
be
the
same
across
everything,
so
we
don't
have
to
change
the
hat,
and
then
you
see
ports,
and
this
is
like
the
normal
parts
of
the
deployment
object.
A
You
see
environment
variables
and
you
see
resources
you
can
control
on
the
resources
you
have
and
what
schools
like
this
is
it
like.
If
you
take
this
and
then
you
also
take
the
service
object
and
the
config
map
and
secret,
which
look
kind
of
similar
to
this,
with
whatever
helpers
that
we
may
have
added,
you
have
the
full
description
of
your
application,
which
again
is
incredible.
It's
like
you,
don't
have
to
go
fuss
with
the
ec2
like
ELB
config,
but
then
locally.
A
You
have
your
hea
proxies
that
you
have
set
up
like
this
is
now
what
we
have
developers
code
to
and
they're
done.
We
as
a
kind
of
DevOps
operations
team,
we
take
care
of
all
the
rest,
and
this
is
kind
of
the
layer
that
DevOps
is
really
needed
to
flourish
in
the
way
that
it
was
originally
intended,
and
one
other
really
cool
thing.
A
A
We
don't
have
to
build
as
many
client-side
libraries,
because
we
can
actually
now
build
just
binaries
that
share
the
same
network
interface
and
potentially
share
filesystem
volumes
as
well,
and
we
can
build
applications
out
of
composing
binaries
together
because
of
the
power
of
docker
and
containers
and
pods,
and
things
like
that.
So
this
is
the
template
and
these
templates
up
top.
So
we've
we
looked
at
that
deployment
lips
on
it,
one
right
there.
A
You
see
that
yeah
great
kind
of
you
have
that
deployment
lips
on
it,
and
that
was
the
one
we
just
looked
at,
and
then
we
have
all
these
other
supporting
template
files
and
that
make
file
goes
through
and
generates
this
app
dot,
JSON
up,
which
includes
all
these
other
ones.
So
this
app
dot
JSON.
It
will
be
generated
once
for
every
environment
that
we
have,
so
this
repo
is
containing
all
these
files,
apps
and
relisha.
Two
of
the
main
sub
directories
inside
that
repo
apps,
is
where
all
your
JSON
8
files
go
and
lips
on.
A
A
But
the
reason
we
check
them
in
is
because
it's
really
really
nice,
especially
for
us,
coming
from
a
puppet
background
to
be
able
to
have
a
full
sense
in
the
code
review
of
the
templating
changes
of,
what's
going
to
happen
to
all
your
clusters,
so
you
may
change
some
parameter
and
deployment
ellipse
on
it.
You
may
expect
it
to
only
affect
your
local
cluster,
but
then,
when
you
generate
them-
and
you
put
it
in
the
code
of
you-
everyone
can
review
that
you're,
not
accidentally
changing
prod,
not
only
humans,
but
also
automated
tools
as
well.
A
So
that's
a
really
nice
thing
that
we
found
is
that
actually
generating
these
final
versions
and
checking
them
in
and
being
able
to
code
review
those
along
with
the
template.
Changes
gives
a
lot
of
comfort
and
security
as
you're,
potentially
making
really
big
changes
to
to
your
templates
and
how
things
are
generated.
So
just
to
be
clear
here
on
the
screen.
We
have
the
first
one
here
is
our
cluster
oops
right
and
the
second
one.
The
second
path
element
here
is
the
environment.
So
we
have
a
local
cluster,
which
we
always
call
your
laptop.
A
We
have
DSP
31,
which
for
us
is
our
non
production.
Cluster
amilia
production
cluster
is
1
2
&
3,
which
are
in
three
different
data,
centers
and
you'll
notice
that
the
environment
is
kind
of
a
variable
that
can
change
independently
from
the
cluster.
So
we
kind
of
want
as
few
clusters
like
kubernetes
installations
as
possible,
because
then
you
get
maximum
utilization
across
your
fleet
and
so
like
you'll
notice,
our
non
production
cluster
hosts
two
environments,
dev
and
staging.
A
A
So
next
we
have
that
centralized
git
repository
that
we
talked
about
that
had
all
those
files
in
it.
The
templates
and
the
generated
JSON
that
describes
100%
of
what
the
infrastructure
is
and
we
use
that
declaratively
and
then
apply
it
to
our
many
clusters,
our
four
clusters,
so
this
applier
originally
started
life
as
nothing
more
than
a
bash
script
that
looped
and
ran
cubes
ETL
apply
on
all
the
files,
the
JSON
files
inside
the
get
repository.
So
we
don't
use
those
templates
files
anymore.
A
A
It
puts
the
new
version,
but
it
intelligently
basically
deletes
fields
that
may
have
been
removed
so
and
it
allows
you
to
merge
your
config
with
other
systems
that
may
be
modifying
those
objects,
so
I
could
go
more
in
depth,
but
just
but
if
you're
interested,
you
can
go
check
out
some
of
the
issues
around
it.
It's
very
interesting
piece
of
software.
We
helped
contribute
a
lot
of
the
original
code
for
how
it
reconciles
changes
and
so
recommend
checking
average
did
but
suffice
to
say.
A
For
the
purposes
of
this
talk,
it
synchronizes
the
state
of
the
git
repository
with
the
state
of
all
the
production
clusters
and
because
of
kubernetes
is
desired.
State
current
state
model
with
spec
and
status.
You
can
do
that.
You
can
basically
sell
tell
kubernetes.
This
is
what
I
want
and
commands
will
take
care
of
it
and
then
commands
report
back
to
you
in
a
different
way,
what's
actually
happening
in
your
cluster,
so
because
of
that
we
can
manage
our
config
declaratively,
which
is
awesome.
So
next,
let's
look
at
what
those
clusters
look
like
in
production.
A
So
remember
the
applier
we've
gotten
that
in
the
top
left-
and
that
goes
over
that
HTP
api.
That's
that
known,
stable
API,
which
is
an
amazing
thing
to
program
too,
because
it
doesn't
matter
where
this
architecture
is
running,
where
your
kubernetes
cluster
is
you
always
just
hit
that
same
API
and
we're
going
to
always
have
one
repo
that
in
a
single
way
expresses
how
all
of
our
service
micro
services
run,
regardless
of
the
cloud
infrastructure?
A
And
it's
now
the
job
of
another
team
to
implement
that
and
make
sure
all
the
criminals
clusters
are
operating
properly
in
all
the
different
places.
So
we
have
that
applier
hitting
the
API
server,
so
here
you'll
see
in
the
lower
left.
We
have
a
few.
We
have
three
bare
metal
nodes
per
cluster
dedicated
to
Etsy
D
and
that's
been
a
great
experience.
So
far,
anytime,
we've
had
issues
it's
for
an
expected
reason
where
we
accidentally
had
one
note
down.
A
This
was
before
production
fortunately,
but
like,
and
then
this
the
one
node
was
actually
down
out
of
the
three
for
like
a
month
before
somebody
noticed,
and
then
we
only
notice
because
the
second
node
went
down
and
then
st
was
like
okay,
I
can't
do
anything
now,
so
we
had
when
we
brought
both
back
up
so
the
stability
and
reliability
we've
been
happy
with.
We
haven't
seen
any
issues
that
would
have
been
able
to
be
pinned
on
at
CD.
So
that's
at
sea.
That's,
that's!
A
We
put
them
on
dedicated
nodes
you
don't
have
to,
but
we
do
just
just
to
keep
things
simple
and
keep
ourselves
a
little
sane
I.
Imagine
as
this
is
all
the
stuff
matures
this
stuff
will
become
more
and
more
co-located
and
more
more
flexible
with
where
it
exists.
So
we
have
dedicated
at
CD
nodes
and
we
install
the
sed
rpm
with
puppet
basically,
but
you
could
do
it
anyway.
What
you
could
spawn
an
ami,
you
could
any
way
you
wanted
to
config
management.
A
You
can
still
do
that
and
but
we're
going
to
move
to
having
those
be
managed
with
a
container
under
cubelet
and
then
so
under
the
first
two
nodes
you
see
here.
Those
are
our
master
nodes.
So
again
you
don't
have
to
dedicate
nodes
to
these
functionalities,
but
you
know
again
for
simplicity
and
as
this
project
matures
that
it
can
help.
So
we
have
two
master
nodes
and
those
do
run:
API
server,
controller
manager
and
scheduler
as
containers
under
the
cubelet
and
the
way
we
run
those
containers
is
we
use
puppet
to
drop.
A
A
yamo
files
on
disk
and
cubelet
is
able
to
not
only
listen
for
api
server
for
how
to
run
things,
but
can
also
listen
to
files
on
disk
for
how
to
run
things
so
or
what
containers
to
run.
Rather,
so,
if
you
blitzie
some
files
on
disk
and
it
picks
those
up-
and
it
runs
those
three
containers
and
what's
cool
is
as
soon
as
those
containers
run,
they're
all
stay.
Also,
the
cubelet
also
knows
that
it
should
also
try
and
find
an
api
server
and
it
finds
the
api
server.
A
So
those
are
our
master
nodes
that
run
those
three
main
master
components
of
kubernetes,
and
then
we
have,
you
know
n
bare
metal
nodes.
So
we've
got
four
here,
but
imagine
you
know
50
or
a
hundred
or
however
many,
and
that
runs
all
of
our
different
components.
So
I'm
going
to
talk
about
a
couple,
more
high-level
topics
and
then
I'll
open
for
any
questions
or
anything.
You
guys
have
thoughts
on
what
I
talked
about
so
far.
So
one
thing
I'll
talk
about
is
ingress.
Ingress
on
public
cloud
is
mostly
a
solved
problem.
A
They
do
it
through
ELB
or
GCP
load,
balancer
and
node
ports.
I
won't
go
too
into
depth
on
that.
You
can
just
look
up
the
documentation
on
that,
but
ingress
on
bare
metal
is
a
very
actively
evolving
thing
and
we
had
to
basically
has
so
solved
this
before
the
ingress
objects
kind
of
took
off.
So
what
we
did
was
we.
We
got
the
service
load
balancer.
So
imagine
this
is
a
dedicated
node,
another
dedicated
Hardware
node
and
we
take
the.
A
We
took
the
service
load
balancer
code
from
the
contributory,
which
is
still
up
there,
which
is
responsible
for
listening
to
the
API
server
for
pods
that
match
a
certain
service,
and
then
it
generates
an
H,
a
proxy
that
Maps
basically
DNS
names
to
that
to
that
to
those
pods
and
then
it
bounces
H
a
proxy
and
then
H
a
proxy
listens
on
an
any
casted
IP
on
the
host
interface.
So
we
now
have
the
ability
to
stamp
out
multiple
of
these
and
have
a
highly
available
horizontally
scalable
in
grest.
A
Here,
because
what
happens
is
the
DNS
name
is
patterned
off
of
our
namespace
and
name
of
the
service
and
the
cluster
that
it's
in
so
like
hello,
world
dev
dot,
the
cluster
name,
DSP
31,
dot
box
on
that,
so
that
that
DNS
name
wildcards
to
this
any
casted
IP.
And
then
this
anycast
IP
is
able
to
pick
up
the
traffic
and
load-balanced
it
onto
the
pods.
A
So
that's
basically
how
we
do
ingress
for
other
in
services,
inside
a
box
being
able
to
contact
and
enter
into
services
into
kubernetes
and
then
for
the
kubernetes
service
discovery
internally
from
cube
to
cube
services.
We
just
do
that
with
the
Q
proxy,
so
that's
basically
octa
proxy,
just
our
internal
name.
For
that.
A
That's
a
most
networks
are
designed,
especially
bare
metal
or
even
potentially,
private
cloud
networks
is
one.
One
server
will
essentially
get
one
IP
and
you're
it's
not
designed,
but
on
the
flipside
kubernetes
networking
architecture
asks
that
each
container
running
on
a
node
gets
its
own
IP,
and
so,
when
you
tell
that
to
the
networking
guys
or
they
just
are
laughing
and
you're
just
like
no,
but
really
we
need
it
and
they
just
keep
laughing
and
then
you're
like
okay.
Well
now,
what
do
we
do?
A
So
if
you
want
to
solve
it
on
your
own,
without
the
help
of
the
networking
guys,
what
you
do
is
you
run
flannel,
which
basically
creates
this
virtual
overlay
that
pretends
for
any
given
pod?
That's
that
that
has
an
IP.
It
can
route
it
to
another
pod
that
has
another
virtual
IP,
basically
so,
and
that
mapping
to
the
underlay
network
is
done
automatically
by
flannel.
So
you
can
have
this
kind
of
mesh
that
lives
on
top
of
your
underlay
network.
A
The
downside
is,
those
IPS
are
not
accessible
outside
the
cluster
if
you're
not
running
flannel
on
an
external
node,
so
we
and
and
the
other
complication,
is
that
it's
just
more
infrastructure
that
you
have
to
worry
about
and
and
it
and
has
a
small
performance
penalty
both
on
performance
and
bandwidth,
so
because
you're
basically
wrapping
the
packets,
but
we
so
we
use
flannel
today,
because
that
was
by
far
the
most
expedient
thing
and
it's
been
pretty
stable.
It's
every
time.
A
We
have
an
issue,
we
think,
oh,
it's
probably
flannel,
and
that
ends
up
not
being
so.
It's
pretty
stable
but
we'd
like
to
move
to
calico
bgp
routing
which
solves
the
same
problem
in
a
different
way
and
then
once
we
do
that,
we're
also
right
now,
looking
at
calico
Ackles
to
be
able
to
isolate
these
pods
from
each
other
from
a
networking
perspective.
So
we're
really
excited
about
that.
A
So
I
think
that's
mostly
it
for
networking
yeah.
Ok!
So
that's
pretty
much.
What
we're
doing
I'm
going
to
take
questions
on
another
technology
were
looking
at
that
we
think
is
really
interesting.
Is
linker
D
for
the
service
discovery
piece,
and
so
we
had
so
that
would
be
backed
by
the
kubernetes
api
server.
So
it's
not
an
entire
service
comfy
platform
and
the
way
we'd
use
it,
but
it
would
enable
us
to
have
a
lot
more
intelligence
about
which
these
different,
how
these
different
components
are
communicating
to
each
other.
So
yeah,
great,
ok,
questions.
A
You
want
it
to
eventually
be
consistent
across
all
of
them,
for
those
I
recommend
putting
in
your
image,
because
it
gives
you
the
most
power
for
rolling
forward
and
rolling
back.
Gives
you
a
really
nice
model
you
put
in
the
image
you
rebuild
it,
and
now
you
do
a
proper
deployment
with
all
the
tools
that
help
you
do
deployments
and
the
risk
that
comes
with
a
configuration
change.
A
Your
next
option
is
basically
dynamic
configuration
which
is
the
config
Maps,
so
we
are
using
those
we
like
them
a
lot.
We
like
that,
we
don't
need
a
separate
infrastructure
for
configuration
anymore.
It's
just
built
right
into
kubernetes
the
challenge
with
config
Maps,
the
challenge
and
the
benefit.
Is
they
currently?
A
So
we
can
do
it
gradually,
because
an
config
changes
are
just
as
dangerous
as
code
changes
and
a
lot
of
contexts,
and
you
have
to
treat
them
as
such
so
yeah,
so
so
what
we're
most
excited
about
is
the
best
of
both
worlds,
where
they're
looking
at
basically
embedding
config,
Maps
or
creating
relationships,
I
forget
exactly
how
they're
doing
it
with
the
deployment
object.
So
that's
an
atomic
unit,
there's
a
lot
of
interesting
stuff
there
about
how
this
API
can
evolve
and
becoming
more
atomic
and
it's
mutations
and
modifications.
But
that's
one
really
interesting
area.
A
A
Yeah
question
on
secrets,
so
secrets
and
kubernetes
are
interesting:
on
the
one
hand,
they're
great,
that
they're
data
based
backed
they're
secure
as
long
as
you
secure
to
your
API
server,
a
net
CD
and
they're
programmatically
access
is
controlled
and
because
of
our
back
and
other
access
control,
you
can
actually
control
which
users
and
teams
can
access
which
secrets
the
Nets.
Not
so
a
good
thing
is
they're
unencrypted
@td
right.
So
that's
that's.
The
security
team
doesn't
like
that
as
much
so
there's
a
couple
ways
we're
dealing
with
it.
A
One
is
we're
looking
at
fault,
so
we
really
like
a
lot
of
the
vault
backed
integrations
now
still
in
kubernetes
Etsy.
That
leaves
it
unencrypted
when
it's
being
used,
but
we're
trying
to
get
our
security
team
around
the
idea
that
they're
protected
enough,
even
though
they
aren't
encrypted
they're
protected
enough
that
it's
that
it
can
be
considered
secure.
A
If
it's
locked
down
and
there's
cool
things
you
can
do
where,
if
there's
some
controller,
that's
responsible
for
managing
the
unencrypted
secrets
at
least
programmatically,
you
should
really
lock
it
down,
so
no
human
could
even
have
access
to
the
secret
resources.
Instead,
it's
controllers
and
automated
systems
that
are
pulling
it
from
from
an
encrypted
source
and
putting
it
into
the
unencrypted
format.
So,
yes,
it
may
still
be
unencrypted
net
CD,
but
if
you
can
lock
it
down
enough
where
it's
really
really
hard
to
get
access
to
them,
then
maybe
that's
acceptable.
A
We're
still
working
through
that.
Another
interesting
thing
we're
looking
at
and
we
have
an
intern
working
on
this
right
now
is-
and
this
is
what
the
thing
that's
really
cool
about
third-party
resources.
So
imagine
a
third
party
resource,
or
rather
a
kubernetes
object
that
lives
in
the
kubernetes
HDD
and
is
accessible
by
the
API
server,
but
whose
definition
is
not
owned
by
communities
but
instead
owned
by
a
separate
controller
that
you
run
in
a
pod
yourself.
Ok,
so
that's
what
a
third-party
resource
is
really
really
cool
and
what
we
did.
A
So
that
way,
we
can
check
in
our
secrets
to
our
source
control
encrypted
per
cluster
and
then
automatically
have
them
decrypted
at
the
final
leg,
which
is
really
cool
and
again
the
more
you
can
put
your
declarative
config
right
now,
we
have
to
imperative
ly,
create
the
secrets
before
we
start
managing
our
config
declaratively,
because
we
can't
check
in
the
unencrypted
secrets,
obviously
into
the
repo.
So
some
of
the
stuff
we're
doing
there
yeah.
A
We,
the
thing
is:
you
have
to
understand
that,
like
we,
this
a
lot
of
this
has
been
in
development
of
the
past
like
year
year
and
a
half
and
the
facilities
we've
seen
in
the
various
passes
that
have
been
on
top
of
kubernetes
OpenShift
days
and
those
many
of
those
they're
really
amazing,
and
they
and
they
do
a
lot
of
great
stuff,
but
the
amount
of
either
constraints
or
coupling.
We
would
end
up
to
those
with
those
platforms
compared
to
the
amount
of
value
we
would
get
for
the
specific
use
cases
we
had.
A
The
balance
just
wasn't
there
yet
in
the
future.
I
wouldn't
be
surprised
if
a
year
from
now
we're
on
OpenShift
or
days
or
one
of
these
passes
right,
we're
very
interested
in
that
space.
We'd
love
to
do
less
of
this
tooling.
On
top
ourselves,
we
would
love
to
migrate
to
that.
We
just
haven't
seen
something
that's
ready
enough
and
provides
enough
for
our
specific
use
cases
that
that
justifies
that
extra
abstraction
layer
and
the
complexity
and
kind
of
to
some
extent
the
restriction
that
comes
with
that.
A
A
A
So
far
we
haven't
seen
any
real
issues.
We
did
see
an
issue
going
from
one
to
O
directly
to
one
three:
eight,
because
there
was
a
bug
there,
but
if
you
just
upgrade
to
the
latest
one
to
X,
then
that
works
fine.
So
you
know
we
upgrade
the
masters
first,
an
upgrade
all
the
nodes
and
then
for
open
sourcing
octa
proxy.
A
We
think
the
shelf
life
is
pretty
short
because
of
ingress
coming
out
and
also
most
of
its
already
open
source
and
service
load
balancer,
and
so
it's
probably
out
with
the
the
effort
but
applier
we
are
actively.
Are
you?
Ok,
you're
bugging
me
on
Twitter
yeah,
so
we
are
actively
open
sourcing
he's
literally
every
single
day
he's
working
on
it
he's
here
somewhere.
If
you
want
I
can
I
can
have
you
bug
him
directly,
but
the
engineer
working
on
that
so
but
yes,
it
is
coming.
I
swear
to
god,
yeah.
A
Yeah,
so
so
it
mostly
goes
through
the
flow
of
source
code
commit
your
shows
code
push
it
then,
once
the
new
image
is
built
sometimes
automatically
sometimes
manually,
you'll
just
bump
the
image
inside
of
the
image
conditional
and
then
essentially
roll
that
out
cluster
by
cluster
we're
trying
to
get
it.
So
we
can
generate
a
set
of
templates
and
then
stage
those
cluster
by
cluster,
and
so
we
have
something
we're
rolling
out
right
now,
internally
in
the
repo,
where
we're
able
to
make
that
distinction.
But
yeah
there's
some
complexity.
A
There
and
yeah,
like
I,
said
like
we'd
love
to
have
this
tooling
be
more
broadly
available.
We
think
that
part
of
synchronizing,
your
specs
with
the
clusters,
especially
different
ones,
is
probably
one
of
the
biggest
areas
of
development
and
some
what
we'd
have
to
do?
What
we've
had
to
do?
The
most
work
for
yeah.
A
Yeah,
so
the
question
around
compliance
and
auditing.
Currently
we're
really
excited
about
the
cube
audit
log,
which
I
think
was
either
released
in
1/4,
is
going
to
be
released
in
1
5,
which
has
a
proper
audit
log
of
all
the
actions.
In
the
meantime,
we
use
like
the
authorization
webhook
to
log
every
single
thing
going
on,
which
was
quite
kludgy,
but
it
worked.
A
So
we
were
manually,
creating
user
accounts
and
then
manually
assigning
those
namespaces,
but
we're
going
to
move
core
OS
done
a
bunch
of
work
on
identity
providers
and
integrating
cube
with
that.
So
we're
going
to
move
to
that
for
single
sign-on
and
like
LDAP
compatibility
and
then
we're
also
going
to
move
to
our
back
from
a
back,
which
is
the
Red
Hat
contributed
or
at
least
modeled
after
authentication
framework
yeah.
A
Yeah
the
question
is:
where
are
we
running
clusters?
So
we
have
three
production
clusters
in
production
data
centers
that
we
have-
and
we
just
right
now-
are
spinning
up
one:
a
production,
one
in
public
cloud,
specifically
in
AWS,
and
we're
also
looking
at
gzp
integer
and
pretty
much
all
the
cloud
providers
as
a
company.
We
want
to
become
more
diverse
in
the
kinds
of
infrastructures
we
support,
and
so
kubernetes
gives
us
the
best
path
for
that.
A
A
I'm,
sorry
yeah
so
logs
and
metrics
I
think
that's
also
pretty
unsolved
from
metrics
we
use
wavefront,
which
we're
pretty
happy
with,
but
unfortunately
it's
a
commercial
SAS
offering
it's
not
open
source,
but
wavefront
we've
been
pretty
happy
with
four
metrics
and
we
just
basically
send
data
points
directly
to
that.
For
logs
we
use
a
combination
of
of
HDFS
and
elasticsearch
and
and
as
well
Splunk
so
but
the
cubes
ETL
logs,
the
integration
for
logs
and
SCD
out
is
really
really
nice.
A
B
A
Yeah
so
security
within
the
cluster
we
use
mutual
off.
It's
basically
our
own
home
world
stuff.
We
just
have
each
service
do
its
own
thing.
We'd
love
to
move
to
something
like
link
or
D
would
help
with
that
or
some
other
projects,
but
for
now
we
just
have
each
walk
that
we
manage
ourselves.
So
ok,
so
I
think
I'll
go
ahead
and
wrap
up
now,
but
I'll
be
up
here.
If
you
have
any
questions
so
feel
free
come
up.
Thank
you
very
much.