►
Description
Embracing Cloud Native at a Thriving, Established Company - Brian Akins, MailChimp
We are in the midst of a major shift at MailChimp. In many ways, we are a microcosm of the industry as a whole: moving from large monoliths to microservices and trying to figure out what that even means. I will discuss the hands-on, real world experiences we have had as we embrace microservice techniques and technologies. I’ll discuss why we choose Kubernetes, Prometheus, and other cloud native technologies. I’ll show our approach to building and operating multiple on premise, bare metal clusters. We’ll talk about our existing development and deployment pipeline as well as our current experimental projects. We’ve had a few false starts and failures and will discuss those to help others possibly avoid the same issues. Finally, I’ll speak candidly about the struggles we’ve had getting organizational momentum for this transformation.
About Brian Akins
Brian is a 20 year industry veteran.He has done a bit of everything - from assembly to CSS racking servers to building distributed systems. For the last few years, Brian has been focused on building and operating infrastructure using components such as containers, Kubernetes, Prometheus, etc. He is currently a staff engineer at MailChimp. Brian lives in the suburbs of Atlanta with his wife and four children.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hi,
what
I
wanted
to
talk
to
you
about
today
is
kind
of
a
journey
of
an
existing
company
into
doing
this.
What
is
this
cloud
native
thing
apologize
a
little
bit
for
the
use
of
buzzwords
ahead
of
time,
but
you
know
you
got
to
get
that
torque
accepted.
So
just
a
little
bit
about
me,
I'm
currently
principal
engineer
at
MailChimp,
I
like
to
say
that
means
I
come
up
with
all
the
bad
ideas,
so
they're
only
good
ones
left
for
the
other
engineers
a
little
bit
more
about
me.
A
Having
said
that,
we're
basically
SAS
were
to
be
the
marketing
department,
if
you
will
for
small
to
medium
sized
businesses.
Our
average
customer
is
a
company
that
is
probably
selling
goods
or
services
on
the
Internet,
but
they
want
to
have
email,
engagement,
place
ads
and
that's
a
type
of
stuff
in
MailChimp
can
kind
of
act
like
that,
give
them
analytics
who
saw
what
who
bought
what
that
type
of
thing.
So
that's
what
we
do
most
people
know
is
for
email.
It's
right
there
in
a
name
to
give
you
an
idea
of
what
we
do.
A
We
send
just
over
a
billion
emails
a
day
and
for
around
the
holiday
season.
If
you
can
imagine,
we
actually
do
a
little
bit
more
than
that.
We
usually
have
an
annual
report,
so
a
weight
into
that
comes
out
before
saying
the
crazy
numbers
of
emails
that
we
send
the
last
few
weeks,
we're
hiring
I.
Think
almost
everyone
in
here
is.
We
have
three
physical
offices
and
we
also
have
some
jobs
that
are
available
for
remote
people.
So
if
this
interest,
you
are
you
think,
I'm
doing
everything
horribly
and
want
to
come.
A
A
We
specifically
didn't
want
anything
nautical
related
because
that's
been
a
little
overplayed
in
this
space,
and
so,
if
you
want
to
look
to
see
what
that
is-
and
this
is
members
of
the
team
that
are
our
current
members,
our
executive
sponsorship
and
some
of
the
folks
who
were
on
the
team
earlier
and
have
kind
of
rotated
off
so
I
want
to
say
a
special
thank
you
to
them
anything
that
is
good
and
correct
about
this
project
and
presentation.
You
say
thanks
to
them
anything
that
is
bad
wrong
and
you
hate
it.
A
You
can
blame
me
for
that.
So
just
a
word
of
warning
and
I
think
most
people
here
are
going
to
do
that
anything
I
tell
you
is
very
specific
to
the
context
that
we
have.
Your
context
may
or
may
not
be
the
same.
So
don't
wonder
exactly
why
we
did
some
things
and
why
we
didn't
do
others
I'm
happy
to
try
it
about
that
offline,
but
a
lot
of
times
is
because
there's
these
years
of
context
at
a
particularly
a
company,
that's
been
around
for
a
while,
where
you
know
kind
of
doing
everything.
A
If
you
want
MailChimp
the
kind
of
public
facing
web
app
which,
if
you're
a
customer
MailChimp
you
come
sign
in,
you
want
to
create
an
email
campaign
schedule
some
ads
or
something
like
that
is
a
typical
lamp.
Application
is
some
model.
It's
a
civil,
very
large,
PHP
application.
We
have
your
normal
things
as
Apache
mod
PHP.
We
use
memcache
for
caching,
do
MySQL
and
fellow
repairs.
You
know
so
somewhat
simplified
diagram,
but
you
know
if
you
kind
of
think
what
was
state-of-the-art
or
it
might
have
even
been
old.
A
You
know
10
plus
years
ago,
cuz,
you
know
wouldn't
MailChimp,
or
at
least
the
current
kind
of
implementation
of
is
about
that
old.
So,
like
say
that
is
a
legacy
type
application,
but
kind
of
very
typical
lamp
thing.
And
yes,
you
can
have
a
successful
money-making
business
on
PHP.
You
know,
regardless
of
what
others
will
tell
you,
so
a
little
bit
a
little
bit
more
behind
the
curtain.
It's
really
not
as
simple
as
that
showed
before.
A
If
you
want
to
look
on
the
left
side
of
this
ass,
more
of
a
logical
diagram,
you're
kind
of
gonna
recognize
the
blocks
there,
but,
as
things
grew
over
time,
there
are
things
that
didn't
quite
fit
into
the
monolith,
and
so
we
have
these.
We
call
them
services,
but
don't
think
about
them.
Services
like
you're
hearing
in
the
buzz
words
and
everything
today.
A
A
lot
of
these
are
kind
of
offline
processing,
stuff,
don't
necessarily
fit
into
the
user
facing
direct
flow
or
production
traffic,
but
they
are
business,
critical
applications
and
they
kind
of
live
off
to
the
side,
and
so
we
kind
of
an
interesting
thing
that
we
have
this
monolith.
But
in
reality
we
have
these
supporting
services
around
it
and
that's
not
actually
true
either
is
we
actually
have
17
instances
of
the
monolith
back
whenever
MailChimp
was
growing
like
crazy?
A
The
easy
thing
to
do
rather
and
the
most
prudent
thing
at
the
time,
rather
than
kind
of
figuring
out?
How
are
we
going
to
support
twice
as
many
customers
on
this
platform
is
like
what,
if
you
just
installed
another
instance
of
the
platform,
and
so
we
did
that
multiple
times,
it's
actually
someone
interesting
the
way
that
we
do
that
once
been
a
lot
of
time
on
that
and
one
of
the
unfortunate
truths
about
that.
A
Since
this
is
all
hardware,
as
you
can
imagine
that
all
the
hardware
is
bought
at
one,
so
all
of
it
is
subtly
different
and
also
the
shards
as
we
call
them
each
one
has
been
a
little
bit
of
an
opportunity
to
do
an
experiment
like
oh
what,
if
we
did,
database
fell
over
this
way.
What
if
we
tweaked
this
this
way?
What
if
we
did
that?
So,
while
the
exact
same
code
is
running
across
all
these
shards,
it
may
not
be
the
exact
same
configuration.
A
This
is
primarily
talking
about
that
big
lamp
monolith
that
we're
talking
about,
but
most
of
these
supporting
services
are
similar
at
least
have
a
common
building
block,
so
kind
of
a
normal
person.
Who's
deploying
code
is
going
to
follow
the
same
flow
regardless
of
the
application,
and
once
again
this
is
going
to
gloss
over
some
details
and
because
the
real
world
is
not
as
clean
as
it
looks
on
slides.
So
here
we
have
our
developer
who's
going
to
work
locally
using
vagrant.
A
We
have
some
custom
vagrant
boxes
that
we
build,
which
kind
of
have
the
supporting
infrastructure
components
in
it.
You
know
my
sequel
database,
you
know
memcache,
you
know
PHP
Apache
at
Sarah.
Do
the
normal
thing
that
you
create
a
PR
gets
pushed
up.
We
have
geniuses
doing
tests,
we
do
code
reviews,
so
this
is
very
typical
if
you
know
kind
of
in
an
enterprise
type
environment.
A
We
have
this
thing
that
is
aptly
named
deployer,
which
it
deploys
code
and
really
it's
just
a
wrapper
around
fabric.
So
underneath
that
is
just
doing
arcing,
and
so
we
are
sync
every
merge
to
master
once
this
past
test,
the
setter
gets
our
sync
to
every
one
of
every
application
server
in
each
one
of
those
shards.
So
like
say,
if
you
think
like,
what's
the
most
simple
way
that
I
could
get
some
things
deployed,
it's
pretty
straightforward
and
it
works
really
well,
but
we
do
have
said
that
every
merge
to
master
actually
goes
to
production.
A
So
what
are
some
engineering
expectations
like
so
I'm?
Just
telling
you
all
this?
To
give
you
an
idea
of
the
context
in
which
we
were
kind
of
like
bringing?
What
is
cloud
native
into
such
an
environment?
Is
that
the
developer,
since
we're
developing
in
PHP
expect
like
if
I
edit,
a
file
I
can
immediately
see
the
results
in
an
API
call
a
browser
or
something
so
that
there
are
kind
of
iteration
time
on
changes
that
they
make
locally
in
our
local
development
environment
is
very,
very
fast.
You
know
they
expect.
A
You
know
it
to
be
basically
instantaneous
and
that
some
subset
of
the
test
can
be
ran
locally,
but
that
gives
them.
You
know
like
just
four
basic
things
like
Lanting,
etc,
just
so
that
they
don't
have
to
push
it
all
the
way.
Through
the
build
process
to
see
like
oh
I,
didn't
follow
the
sky
style
guide
in
that
function
or
something
like
that
and
kind
of
more
from
the
operations
and
infrastructure
side
like
say,
master
is
always
running
in
production.
A
A
A
lot
of
these
are
going
to
be
the
exact
same
reason
is
we
have
multiple
development
teams
all
working
in
the
monolith,
and
so
you
can
kind
of
imagine
if
you
have
two
features
are
not
even
related
but
they're,
both
sometimes
working
on
the
show
same
shared
code,
so
you
have
those
things
and
just
it.
The
monolith
is
a
pretty
large
and
very
old
codebase,
and
so
you
get
this
thing
where
sometimes
to
do
a
quick
experiment
is
a
lot
of
work.
A
You
know
both
on
the
development
side
and
then
on
the
operations
side
to
be
able
to
get
something
out
quickly.
So
that's
one
of
the
big
things
we
want
to
do
what's
the
time
that
we
can
go
from
a
product
idea
to
having
it
out
there
for
users.
You
know
in
a
quicker
amount
of
time,
and
sometimes
those
users
are
gonna,
be
those
external
users.
A
Instead
of
earlier
that,
we're
actually
running
completely,
at
least
this
application
runs
completely
on
bare
metal
inside
of
data
centers,
actually
multiple
data
centers,
and
so
we
have
the
problem
that
are
on
a
normal
day.
The
hardware
is
just
very
underutilized,
but
on
a
you
know,
kind
of
busy
day.
A
But
we're
also
think
about
some
of
these
other
things
like
what,
instead
of
doing
just
an
evolutionary
change,
what
if
we
did
a
larger
change-
and
we
also
have
the
thing
that
like
said
so,
our
operating
systems
all
the
way
up
through,
like
our
PHP
packages,
the
versions
of
those
you
know,
the
version
of
memcache,
etc.
That
we're
using
is
deployed,
be
a
puppet.
You
know
kind
of
the
kind
of
normal
way,
but
then
we're
actually
deploying
our
application
code
using
this
other
method.
A
Is
it
gets
kind
of
difficult
and
we've
have
interesting
use
cases
where
deployments
get
interesting
things
where
they're
not
fully
atomic
and
such
a
thing,
and
it
gets
particularly
interesting
whenever
you
need
to
do
multiple
things
within
a
single
deployment,
because
you're
changing
multiple
things,
and
so
that's
just
from
a
purely
practical
things
being
able
to
do
that,
but
also
in
obviously
some
like
criminais
days.
It's
a
little
bit
easier
to
do
canary
deploys
Bluegreen.
A
You
know
that
type
of
thing,
and
another
reason
is
because
of
these
shorts
that
we
were
talking
about
earlier
with
hardware,
and
those
type
of
things
is
you're
somewhat
fixed
like
this.
Shard
belongs
to
that
hardware,
and
so
as
we're
thinking
through
like
what
is
our
kind
of
dr
and
our
business
continuity
planning
around
that
it
makes
sense
like
if
I
got
an
app
striked.
You
know
that
application
away
from
the
hardware
is
running
on
then
I
can
move
that
application
around
much
more
easily.
Like
can
I
pick
up
that
version.
A
And
then
also
another
thing
is
we're
doing
that
like
how
can
I
do
standardized
metrics?
You
know
that
type
of
thing?
Okay,
so
how
do
I
get
there?
So
we've
done
all
this.
This
is
the
context
we
have.
This
is
why
we
want
to
change.
So
what
did
we
do,
and
this
is
still
in
progress,
but
how
did
it
actually
turn
out
in
the
real
world
because
I'm
guessing
that's
what
most
people
weren't
want
to
hear
about?
A
So
the
phase
one
we
had
since
like
say:
MailChimp
is
primarily
on
on-premise
and
data
center
thing
is
we
had
to
deploy
kubernetes
clusters
Unbearable
into
multiple
data
centers,
so
some
things
that
we
needed
to
do
that
as
you're
deploying
kubernetes.
You
hear
all
these
things.
Is
we
decided
that
we're
just
going
to
treat
kubernetes
like
it's
just
another,
app
that
we
do
so
we're
going
to
actually
manage
kubernetes
the
components
of
it
with
puffit?
That's
what
we
have.
A
A
Like
say
ed
CD,
like
just
another
database,
you
know
with
backups,
etc.
Another
thing
that
we
do
is
we're
using
a
lot
of
the
existing
tools
and
knowledge.
We
are
we're
able
to
leverage
that
and
and
really
from
just
a
purely
practical
standpoint,
probably
just
to
be
honest,
a
lot
of
the
kind
of
interesting
new
ways
of
deploying
kubernetes
are
not
quite
ready.
You
know
like
if
you
played
with
some
of
those
tools.
A
lot
of
them
are
still
like.
Pre-Release
don't
use
these.
A
You
know
they
only
support
single
masters
or
something
like
that,
and
so
then
we
do
the
normal
thing
that
we
actually
have.
Some
knows
that
that
run
are
running
the
control
playing
components.
You
know
that's
kind
of
one
puppet,
a
class,
but
we
also
run
workloads
on
those
as
well
and
we
actually
run
multiple
kind
of
they're,
fully
independent
clusters,
they're
not
we're
not
using
Federation
or
anything
like
that.
A
A
You
know
some
VLAN
and,
like
our
database
servers
live
in
another
VLAN
and
we're
using
network
ackles
to
you,
know
kind
of
control,
access
between
those
different
layers,
and
if
you
were
to
walk
into
one
our
Colo
locations,
you
would
see
you
know,
switches
routers.
You
know
that
type
of
thing
and
you'd
recognize
the
logos
on
them.
A
So
what
we
did
is
that
back
in
network
is
just
what
we
are
using
for
the
the
pod
Network
and
we
just
do
use
host
routes
between
them,
and
if
you
can
imagine
that,
so
you
can
go
into
one
of
our
nodes,
you
can
type
IP
route
and
you're
just
gonna
see
all
the
routing
between
there.
No
overlay
Network,
it's
pretty
it's
pretty
straightforward
for
someone
who's
familiar
with.
You
know,
Linux
routing
that
they
could
just
go
in
there
and
see
it
we're
using
flannel
to
actually
manage
the
hosts
routes.
A
But
in
reality
you
could,
we
could
have
used
the
shell
script
to
do
it.
Finals
just
somewhat
convenient,
because
you
know
if
something
changes,
it
picks
it
up
right
away.
So
pretty
simple,
particularly
if
you're
running
on
Prem
10
occasions.
This
is
one
of
those
things
where
also
as
we're
bringing
criminais
tees
into
an
existing
environment.
A
Kubernetes
supports
like
using
a
an
external
authentication
and
one
of
the
ways
it
does
is
with
oh
I
see
we
use
Google
Apps
for
authentication
for
various
things
as
a
terminating
supports
that
enables
oh
we'll
use
that.
But
if
you
have
to
know
that
at
least
Google's
version
of
OID
C
does
not
support
group
information,
at
least
not
directly
and
like
say,
we
really
want
to
be
able
to
have
that,
so
that
it
fits
into
our
normal
scheme
that
these
people
are
allowed
to
do
this.
A
You
know
service
accounts
were
allowed
to
do
that,
and
so
what
we
did
was
kind
of
extract.
That
code
copy
that
code
out
and
made
our
own
version
of
an
OID,
C,
Authenticator
and
kubernetes
has
a
scheme
where
the
API
server
can
ask
via
web
hook.
You
know
this
user
gave
me
these
credentials
and
an
OID
C
or
just
have
a
token.
You
know
is
this
still
of.
A
Speaking
of
integration.
This
is
somewhat
of
a
bad
metaphor.
This
Fredi
our
mascot.
So
the
big
fraidy
is
like
our
existing
infrastructure
in
the
little
one
he's
integrating
and
with
this
sorry
yeah.
So
we
wanted
to.
But
one
of
the
big
things
about
bringing
kubernetes
in
there
as
we
wanted
really
treated
as
just
another
application
like.
How
can
we
do
this
change
but
still
support
it
so
that
people
are
comfortable
supporting
it
like
they
do
other
things
you
know
no
one
likes
to
get.
You
know
page
at
3:00
in
the
morning
with
an
unfamiliar
system.
A
So
at
least
for
this
like
say,
oh
I
know
how
to
use
IP
route,
because
I've
been
doing
that
my
whole
career
type
thing.
We
need
to
integrate
with
other
systems
like,
say
we're
for
large
shipping,
we're
currently
using
fluent
D
and
I'll
reserve
my
comments,
but
but
we're
also,
but
we're
using
follow,
beat
to
actually
ship
operating
system
logs
because
we
use
a
rather
hefty
log
processing
pipeline.
A
You
can
imagine
how
many
logs
that
we
produce
in
there,
and
so
we
have
a
very
mature
pipeline
there,
and
so
we're
actually
looking
at
now
that
file
beet
has
official
kubernetes
support
to
be
able
to
add
the
kubernetes
metadata
to
the
logs
that
are
flowing
through.
It
is
switching
to
that
and
just
deploying
that
via
puppet,
because
that's
what
the
team
that
kind
of
owns
and
maintains
our
log
processing
pipeline
is
familiar
with.
So
that's
another
thing
we're
like
oh,
how
can
we
treat
this
thing
in
kubernetes?
A
Just
like
we
do
others
things
we
currently
miss
is
where
I'm
gonna
go
also
over
a
lot
to
be
honest,
use,
zabbix
and
then
for
kubernetes.
What
we're
doing
is
using
zabbix
for
kind
of
like
operating
system
down
level,
monitoring,
saying
my
oh
I
threw
a
hard
drive
or
CPU
or
a
got
memory
area,
so
we're
still
using
our
same
tooling
there,
but
we're
using
prometheus
and
alert
manager
for
kind
of
like
kubernetes
and
up
including
the
applications.
A
There's
experiments
that
have
been
going
on
since
before
I
got
to
MailChimp
that
we're
actually
investigating
using
that
combination
of
Prometheus
and
alert
manager
to
even
kind
of
go
further
down
the
stack.
So
another
thing
that
we
needed
to
figure
out
is:
we
have
you
know
secrets
we
have
passwords
for
database
credentials,
Sara
we
started
looking
at
and
everyone's
like.
Oh,
you
need
to
use
this
product.
A
You
need
to
use
that
there's
vault,
there's
this
and
that
and
what
we
do
for
a
puppet
is
once
again
it's
one
of
those
things
that
is
just
very
straightforward,
but
works
very
well.
Is
there
is
this
thing
called
a
Yama
which
is
basically
this
public
private
key
encryption
for
secret
so
that
you
can
actually
check
them
into
a
git
repo?
A
A
So
that
concept,
while
not
the
same
as
I
coolin,
was
already
familiar
to
the
engineers
on
the
team
and
we've
actually
had
to
make
to
most
of
these
projects
had
to
make
upstream
PR.
So
it's
kind
of
one
of
those
to
be
honest,
like
everything
didn't
work,
that
great
out
of
the
box,
and
so
we've
had
to
do
some
things
or
didn't
work
great
for
our
context.
It
might
have
worked
great
for
someone
else's
so
ingress.
What
good
is
it
if
you've
got
this
nice,
big,
kubernetes,
cluster
and
I?
Can
access
apps
in
it?
A
A
You
know
getting
metrics
out
of
that,
so
we're
just
kind
of
treating
that
like
it
was
running
in
one
of
the
other
environments,
but
we
run
traffic,
which
is
another
open
source
project
which
is
as
an
ingress
controller,
and
we
just
run
that
on
a
node
port
service
and
so
H
a
proxy
is
low,
balancing
across
all
other
nodes
within
the
cluster
and
then
those
get
router
to
traffic
traffic
is
just
another
deployment.
We're
not
doing
anything
weird
we're
like
we're
running
as
a
daemon
set
on
each
node
or
anything
so
pretty
straightforward.
A
If
we
do
the
way
the
packets
go
yeah,
sometimes
you
take
a
long
way
around
with
your
packet.
But
it's
very
easy
to
understand
and
at
this
stage
of
our
deployment,
we're
wet
to
worry
about.
Is
it
correct
and
easy
to
understand?
Whether
then
is
it?
You
know,
you
know
a
hundred
percent
performance
we're
willing
to
make
some
of
those
trade-offs.
We
terminate
TLS
at
traffic
and
one
reason
we
went
with
is
two
levels
so
that
we
could
do
that
and
so
that
our
ingress
controller
would
have
access
to
the
full
kubernetes
api.
A
We
had
originally
started
playing
with
helm
and
we
kind
of
read
the
helm,
documentation
and
went
all
in
with,
like
all
these
nested
charts
and
everything,
and
that
felt
horribly
for
us,
and
we
did
some
other
things,
but
now
what
we've
actually
come
back
to?
We've
actually
come
back
to
helm,
but
we're
actually
each
application
in
the
same
source
repository
as
its
application
code.
They
actually
have
a
helmet
art
in
there
as
well,
so
a
gloss
over
a
little
bit
of
that,
and
so
our
actual
usage
of
helmets
we're
only
counting
using
it.
A
Just
for
templated
kubernetes
manifests
okay.
So
what
are
we
doing
on
it?
Space
with
four
now
four
deployments
and
kubernetes?
Is
this?
The
general
shape
of
this
is
the
same
as
what
you
saw
before
is
a
developerworks
in
your
local
development
environment.
I'll
speak
a
little
bit
about
that
later.
Does
a
PR
tests
are
run
by
Jenkins
and
Jenkins
whenever
those
past,
rather
than
you
know,
calling
out
to
this
deploy
thing?
We
wrote
this
command-line
tool
called
Pando,
which
it's
really
just
a
wrapper
around
darker
build
and
some
of
the
helm
commands.
A
So
once
your
build
is
finished,
we
push
the
artifactory,
which
is
our
functions.
Are
there
private
darker
registry
and
then
it
links
helm,
calls
to
the
various
services
and
I
said
before
we're
not
doing
any
Federation,
and
so
you
can
explicitly
target
one
or
more
clusters.
So
if
you
have
an
application
needs
to
run
multiple
clusters,
you
just
say
explicitly:
I
want
to
run
multiple
clusters.
A
Expectations
of
development,
like
say
it
is
a
lower
gross
because
there's
NFS
involved,
and
but
it
works
that
way,
and
since
it's
just
a
flagon
and
deployment,
we're
still
using
the
same
deployment
tools
that
we're
using
remotely.
We
tried
some
of
the
things
if
you've
seen,
for
example,
the
way
that
draft
works
that
it
will.
You
know
automatically
rebuild
a
container
whenever
you
change
files.
We
tried
some
systems
like
that,
but
they
did
not
work
so
well.
It
was
one
of
those
they
work.
A
A
What
we
do
is
like
say:
the
chart
is
embedded
locally,
are
embedded
into
the
same
application
repository
most
developers
do
not
care
what
is
appointment,
so
we
have
are
developing
some
boilerplate
ones
to
be
able
to
say,
like
hey,
I
have
a
generic
PHP
application.
Just
let
me
do
that
copy
it,
and
that
way
we
don't
have
any
kind
of
interesting
dependencies
between
things.
A
That
does
mean
that
whenever
we
move
decide,
we
need
to
make
a
change
to
a
chart
or
whatever
we
do
have
to
go
and
open
PRS
on
a
bunch
of
repositories,
but
once
again
much
more
explicit
versus
magical
and
we
actually
use
the
same
tooling.
Locally.
I
was
saying
that
Pando
command-line
tool
also
is
used
locally
of
when
they're
doing
deployments
and
locally.
We
also
run
an
instance
so
Promethea,
since
it's
pretty
lightweight
whenever
you
don't
keep
many
metrics
around
and
that
way
they
can
iterate
on
their
application
code.
A
If
you
need
to
iterate
on
some
deployment
stuff,
but
also
like
some
metrics,
so
that
I
can
set
some
alerting
thresholds
or
something
they
can
do
all
that
locally,
which
is
which
is
kind
of
nice,
so
call
this
3.5,
because
a
lot
of
this
was
all
happening
at
the
same
time
like
how
do
we
deploy
real
applications?
So
we've
talked
about
this
and
if
you
know
it's
not
this,
it's
not
great
to
just
show
toy
applications.
So
the
first
application
we
chose
was
this
very
simple
application.
We
have.
A
It
takes
snapshots
of
web
pages
for
people.
It
turned
out,
an
application
was
not
so
simple
and
it
turned
into
a
complete
rewrite
and
we
were
developing
the
tools
at
the
same
time
turned
into
a
giant
Pike
shed
and
we
still
have
not
finished
it.
So
so
what
we
really
launched
first
through
a
prior
effort
that
log
process
in
pipeline
I
was
talking
telling
you
about
earlier
this
very
mature.
A
It
was
already
containerized
and
it
was
running
on
a
different
kind
of
kind
of
bear
docker
host
and
we
had
a
team
there
that
were
very
patiently
and
hey.
We
want
to
try
out
this
kubernetes
thing
will
be,
you
know,
will
be
your
test
subjects,
so
they
were
extremely
patient,
but
also
extremely
excited
about
doing
this,
and
this
was
real
load
like
business
critical,
but
not
customer
facing.
A
If
you
can
imagine
that
company
is
ten
years
old,
are
older
and,
like
all
the
little
chat
box,
we
have
and
they're
written
in
all
sorts
of
languages,
Python
PHP
Java,
probably
some
other
oddball
ones,
but
we
need
to
move
all
those
over
very
very
quickly
and
a
lot
of
those
are
maintained
by
different
teams.
Individual
development
teams,
it's
Sara,
so
what
we
did-
and
we
didn't
do
this
for
all
of
them.
A
A
Taking
about
something
that
happened
kind
of
late-breaking.
This
was
last
week,
someone
said
hey,
my
deployment,
doesn't
work,
I,
get
this
error
and
then
I
look
and
it
turned
out
that
in
that
particular
cluster
we
have
just
just
over
1300
CPU
cores
and
they
were
all
requested
by
application.
Those
know
and
kubernetes.
A
If
you
do
resource
requests,
you
say:
hey
I
need
four
CPUs
or
I
need
half
a
CPU
or
something
say
we're
a
little
overzealous
and
what
people
were
requesting
for
CPU
cores,
and
it
was
one
of
those
whenever
we
were
building
this
since,
like
so
we're
kind
of
in
this,
where
production
but
still
kind
of
early
production,
we
don't
need
to
worry
about
capacity
planning.
Yet
we
have
you,
know
1,300
cores
in
that
cluster
it'll
be
fine
and
it
was
not
fine.
A
This
way
actually
went
through
and
looked
at
what
our
water
and
what
is
there
kind
of
like
daily
peak
of
CPU
that
they're
using
for
application,
because
what
we'd
kind
of
done
is
did
a
couple
of
applications
earlier
and
then
you
know
everyone
had
just
kind
of
copied
that,
and
so
we
were
able
to
shrink
kind
of
the
footprint
there
significantly.
So
that's
one
of
those
things
that
in
hindsight,
probably
it's
obvious.
A
Okay,
so
now
that
we're
here
and
we're
not
done,
you
know,
this
is
kind
of
still
some
Plumeri
work
like
what
are
some
future
things
that
we're
going
to
do
is
one
of
the
things
everything
that
we
mentioned
here
is
things
that
could
go
down
and
the
business
would
keep
functioning
you
know.
So
it
was
a
good
way
to
test
the
application
it's
out
on
Carini's
without
having
to
do
a
huge
amount
of
work.
A
A
So
it's
a
little
important,
but
we're
gonna
do
that
that
kind
of
update
of
that
100%
om
kubernetes
from
day
one,
and
so
that's
one
way
to
force
us
to
figure
out
any
other
issues
is
because
this
app
is
down
or
if
this
app
is
incorrect,
you
know,
then
you
know.
Ultimately
we
don't
get
paid.
So
that's
one
way
to
make
sure
that
you
do
something
correctly
and
then
that's
really
because
it
is.
You
know
architectural
II,
a
lot
like
the
monolith.
A
You
know
that's
a
good
thing
for
us
to
start
thinking
about.
If
we
get
through
that,
then
we
can
move
that
big
monolith.
You
know,
which
is
the
real
customer
facing
part
of
that
application
of
MailChimp
over
and
another
thing
is
since
I've
showed
earlier,
some
of
those
supporting
services,
we're
gonna,
look
at
actually
doing
a
service
mesh
in
that
and
use
an
envoy
etcetera.
She
just
told
me
my
time
is
up
GOP
servers
in
PHP
and
I.
Don't
have
time
for
questions
I,
don't
think
sorry,
but
maybe
we
do
any
questions.
A
That's
an
interesting
question
right,
I
think
it's
one
of
those
yes,
but
that's
probably
just
being
realistic.
That's
not
gonna
happen
in
the
next
year,
maybe
not
next
to
if
ever
you
know,
I'm
just
being
realistic
with
you,
because
really,
if
you
want
to
think
about
the
mta's
for
us,
if
you,
if
you
worked
in
no,
is.
A
You
know
there's
a
lot
of
interesting
thing,
particularly
roundup.
He
addresses
etcetera
that
we
would
have
to
solve
and
really
our
kind
of
pain
points
are
more
in
down
that
application
side,
so
yeah
yeah
so
on
a
road
map
but
may
or
may
not
happen.
All
right,
I
think
we're.
Oh
one
more
than
that
I'm
getting
kicked
out.
A
A
Right
now
for
us
I
think
sto
is
a
little
too
magical
with
it,
injecting
iptables
rules
etc,
but
we're
just
beyond
proof
of
concept,
with
just
doing
some
things
with
envoy
there's
been
a
couple
projects
that
it
came
out
recently.
There
are
a
lot
more
interesting
and
I
think
a
little
bit
more
our
fit,
because
this
deals
kind
of
like
all
the
way
over
here
and
we
need
like
here
so
alright,
one
more
and
then
I'm
really
gonna
get
kicked
out
right.
A
We
want
to
treat
those
as
separate
concerns,
I
think
that
will
happen
because
I
don't
think
we
want
one
to
be
blocking
the
other,
but
obviously
it
makes
it
easier
if
you're
already
on
kubernetes
to
break
it
up,
because
you
have
somewhere
to
deploy
that
and
then,
if
it
doesn't
go
well,
you
can
kind
of
under
play
it
for
lack
of
a
better
word
but
yeah.
That
is
part
of
the
plan,
and
really
a
lot
of
it
is
as
we
need
to
go
in
and
like
revamp.