►
From YouTube: Balancing Power and Pain: Moving a Startup From a PaaS to Kubernetes - David Sudia & Toni Rib
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Balancing Power and Pain: Moving a Startup From a PaaS to Kubernetes - David Sudia, GoSpotCheck & Toni Rib, Gusto
By hiding a lot of complexity and allowing a team to move fast and simply "heroku push" applications, PaaS solutions like Heroku are a perfect fit when you are early stage startup. However, what do you do when your business starts to get traction, and your scale or use case begins to stretch the limitations of a PaaS? This talk will share the story of a startup's successful migration away from a PaaS to a self-built platform powered by CNCF technology. We'll share the highlights of our journey, such as how we translated PaaS concepts to our new infrastructure, and explain the series of choices we made, like assembling our platform from Kubernetes and other CNCF components. We will also share some of our difficulties, with the goal that other organisations can avoid making the same mistakes.
https://sched.co/UaWH
A
Yeah,
all
right
so
welcome
to
our
talk.
I'm
Tony-
and
this
is
Dave
and
before
we
get
started
kind
of
just
thought
about
this
right
now
is
anyone
here
from
Heroku
awesome,
okay,
we're
not
gonna
badmouth
them
or
anything.
Just
curious.
Is
anyone
running
on
Heroku
or
a
different
platform
as-a-service
right
now,
awesome:
okay,
keep
your
hands
up
if
you're
thinking
about
moving
into
kubernetes
all
right.
What,
if
you're
actively
moving
into
kubernetes?
Has
anyone
actually
done
that
full
transition?
Yet.
A
Okay,
so
I'm
Tony
and
I'm.
Currently
a
software
engineer
at
a
San
Francisco
based
startup
called
gusto,
but
I
used
to
work
with
David
go
spot-check
until
about
July
of
this
year
and
I'm.
An
application
developer,
full-stack
I
generally
am
working
in
Ruby
on
Rails
and
react
would
have
also
done
some
like
SPARC
and
Scala
stuff
too
I'm.
B
Dave
studia
I
work
on
the
DevOps
team
at
go
spot-check.
We
are
three
people
which
includes
a
manager
who
used
to
be
technical.
That
has
no
time
to
be
any
more
so,
and
my
kind
of
joke
rooted
in
reality
is
that
if
it's
not
feature
development,
it's
probably
us
so
we
handle
building
out
the
infrastructure,
maintaining
and
running
the
infrastructure,
also
we're
kind
of
an
unofficial,
InfoSec
team
and
the
unofficial,
SRE
team,
etc,
etc.
It's.
A
A
quick
plug
for
my
company
gusto:
we
are
the
people
platform,
we
do
payroll
benefits,
HR
and
other
things
like
that
for
small
businesses
generally
any
business
with
less
than
a
hundred
employees.
We
have
about
a
hundred
thousand
customers
around
the
US
and
we
do
have
offices
in
San,
Francisco
and
Denver,
where
I'm
from
and
New
York
City,
and
we
are
also
hiring
so
we're
just
gonna
put
that
out
there
and
we
actually
just
moved
our
monolith
over
to
kubernetes
about
a
few
weeks
ago.
A
B
About
ghosts
by
Czech,
tuner,
plus
customers
we're
mid-sized
startup.
We
are
no
longer
in
that.
Oh,
we
just
have
to
live
phase,
but
we
haven't
quite
gotten
to
the
full
maturity
phase.
Yet
so
200
plus
customers
about
135
employees
we're
gonna
round
for
8
years
in
terms
of
scale,
because
one
of
the
interesting
things
having
this
conversation
outside
of
a
talk
with
people
is,
you
know
the
scale
of
people
making.
These
kinds
of
moves
is
very
different.
Some
people
are
handling
millions
of
responses
at
requests
a
day
we
handle
200,000
ich
task
responses.
B
Today,
our
product
is
a
mobile
field
execution.
You
have
someone
who's
going
out
to
check
stock
at
a
location
or
check
compliance
agreements
with
with
a
vendor
or
something,
and
they
will
go
out
and
it
used
to
be
clipboards
and
someone
to
take
two
weeks
to
make
a
spreadsheet.
Now
they
put
it
into
our
mobile
app.
It
goes
back
for
near
real-time
analytics
at
the
central
office,
so
about
200
thousand
of
those
responses
come
back
today,
but
between
all
of
our
internal
systems
and
applications.
B
A
We're
gonna
start
out
by
talking
about
what
our
application
was
running
on.
We
started
this,
which
is
Heroku
and
when
I
joined
the
company
back
in
September
2016,
all
our
applications
were
hosted
on
Heroku
using
a
Postgres
database,
with
all
our
data
being
sent
over
to
New
Relic
and
eventually
we
kind
of
acted
on
circle
CI
for
some
RC
ICD.
It
started
sending
our
logs
to
similar
logic
for
easy
querying
and
analysis,
and
this
was
like
a
very
easy
world
to
live
in
as
a
developer.
A
B
So
2013
to
2016
no
DevOps,
you
don't
need
it.
Heroku
is
great
for
that
and
and
I
think,
as
we
get
deeper
into
this,
we're
gonna
see
it's
it's
a
powerful
platform
by
starting
to
start
up
tomorrow.
I
might
start
there,
because
it's
still
very,
very
good
for
that.
It
scales
really
fast
any
kind
of
add-on
service.
You
just
attach
it.
You
have
really
powerful
operations
from
that
perspective,
because
everything's
a
button
click
to
get
things
hooked
up.
You
don't
need
to
think
of
the
configuration.
You
know.
B
Thinking
about
Bryan
Liles
talked
this
morning
that
there's
very
much
that
like
yeah,
it's
all
convention,
you
just
do
it
the
way
that
that
Heroku
allows
you
to
do
it.
The
downsides
are
it's
very,
very
expensive
and
there's
a
certain
point
at
which
the
cost
for
the
convenience
you
know,
is
very
worth
it
in
the
beginning
and
there's
a
certain
point
where
you
hit
where
it's
gonna,
it's
gonna
go
the
other
way.
It's
very
cookie
cutter.
B
B
A
lot
of
limitations,
first
for
any
given
application
in
Heroku,
you
can
only
get
10
servers
period,
so
we
were
running
at
10
servers
recently
for
our
main
Rails
monolith
and
Weaver
get
in
a
position
where
we
were
looking
at
signing
more
customers
and
the
those
of
us
on
the
ops
team.
We're
like
yeah,
that's
great!
Also!
Oh,
ok,
ok
can
we
you
know.
B
We
started
having
jobs
that
would
use
more
than
fourteen
and
a
half
gigs
of
ram
and
that
that
broke
things
on
one
of
my
early
days
at
the
company,
we
just
deleted
a
user's
like
export
of
data
because
out
of
a
queue
because
it
was
just
causing
the
worker
to
crash
over
and
over
again,
and
that
was
just
kind
of
standard
practice
for
a
while.
It's
like,
oh,
that
person
is
not
getting
their
data.
B
Are
we
communicating
that
how
okay,
okay
and
then
simcha?
This
is
why
we
had
that
again.
Sometimes
you
just
get
too
big
right,
so
the
other
issue
we
ran
into
was
with
with
Postgres
and
with
their
configuration.
We
had
a
period
of
pretty
massive
degradation
and
slash
downtime
earlier
this
year.
That
was
basically
caused
by
a
pretty
low
level,
setting
in
Postgres
called
shared
buffers,
and
we
were
having
queries
that
were
larger
than
our
shared
buffers
and
it
was
causing
things
to
get
ejected
out
and
read
from
disk,
and
it
was
a
very
bad
situation.
B
Last
about
four
days
when
we
came
back
to
Heroku
again
as
one
of
their
top
10
largest
customers
and
said,
can
you
just
chain
up
this
a
little
bit
for
us
and
they
were?
They
said
we
literally
have
no
way
to
do
that,
so
that
was
that
was
kind
of
the
triggering
moment
when
our
senior
leadership
team
finally
came
back
and
said
yes,
project
approved.
A
Yeah
so
they've
kind
of
mentioned
this,
but
like
day
nine
for
him
echo
spot
check
he
here's
our
DevOps
leave
just
say
this
and,
like
you
know,
I've
done
this
so
many
times
at
this
point,
we're
like
oh,
this
job
is
taking
everything
down
great
I'm,
just
gonna
kill
the
worker.
I'm
gonna
go
delete,
it
do
whatever
bring
it
back
up.
It's
fine
and
like
we
just
kind
of
didn't
even
think
about
it
anymore.
A
At
this
point
and
yeah,
my
team,
like
owned
our
main
web
application
along
with
a
few
other
things,
DevOps
was,
you
know
brand-new.
The
first
DevOps
lead
started
a
week
after
I
did
up
until
then
we
didn't
have
anything
so
like
we
had
a
whole
backlog
of
stuff
to
do
that
kind
of
figuring
out.
This
issue
was
like
not
exactly
the
top
priority
at
the
company
at
the
time.
A
Okay,
so
another
kind
of
thing
that
came
into
play
here
at
the
time
is
what
we
call
like
the
two
phases
of
a
startup,
the
first
one
being
just
throw
money
at
it.
So
you
know
there's
a
lot
of
cash
to
go
around
at
the
beginning
and
when
you
have
a
small
team
of
developers,
sometimes
they're
making
the
trade-off
between,
like
should
I
spend
more
time
on
this
thing
or
should
I
just
pay
to
run
it
at
whatever
and
for
sure
in
early
2018
like
this
is
where
we
were
as
a
company.
A
B
And
that
I
was
hired
in
February
of
2018
and
with
the
mandate
to
we're
gonna
build
the
next
platform,
and
then
three
months
later
they
were
like
and
don't
spend
any
money
on
it.
So
so
we
were
charged
with
this
dual
dual
thing
of.
We
need
to
make
this
platform
more
expandable,
more
scalable,
get
us
to
the
next
level
of
this
company
of
you
know
when
we,
when
we
exit
this
plateau
and
we
and
we
get
back
into
into
huge
scaling,
we're
gonna
need
be
able
to
do
that,
and
also
it
has
to
be
cheaper.
A
We
also
at
the
company,
have
our
QA
engineers,
so
often
less
technical,
don't
really
have
any
coding
background
or
anything
doing
our
deployments
right
so
they're
in
Heroku
and
they're
used
to
being
able
to
push
a
button
to
go
from
staging
to
prod.
So
if
we're
going
to
try
to
move
an
application
over
to
kubernetes,
we
can't
really
just
you
know,
hand
them
the
command
line
and
say
go
for
it.
A
Of
gamal,
so
we
needed
some
solution
for
this
too.
That
was
going
to
be
like
simple
enough
for
a
non-technical
person
to
be
able
to
interact
with.
We
also
knew
that
we
wanted
to
have
this
kind
of
partnership
between
DevOps
and
our
application
development
teams
right
that
I
was
on.
So
we
wanted
some
solution
where
developers
could
sort
of
like
adopt
pieces
as
they
felt
comfortable
so
that
it
just
wasn't
all
on
develops
the
whole
time.
A
B
Another
thing
that
we
set
out
at
the
beginning
is
making
sure
that
we
were
keeping
everything
as
code.
That
was
for
our
own
sanity,
for
knowing
what
we'd
built
there's.
Also,
you
know
one
of
the
things
that
that
Heroku
allows
that
makes
it
difficult
to
track.
What's
going
on,
is
you
can
just
like
Heroku
Rails
C
and
bring
up
a
rails
console
and
do
things
in
production
right?
So
we
did
wanted
to
try
to
keep
momentum
and
not
prevent
people
from
doing
things,
while
also
making
sure
that
we
had
a
better
tracking
system.
B
One
of
our
big
things
is
just
making
sure
that
we're
not
ready
to
go
full-scale.
Everything
is
security.
Lockdown.
You
can't
do
anything,
but
we
want
to
make
sure
that
any
systems
we
built
we
could
build
that
we
could
get
there
from
a
solid
foundation
in
base
the
time
we
also
had
a
platform
team
and
they,
so
these
were
four
or
five
developers
who
were
working
more
on
the
application
level
of
what
is
the
architecture?
Gonna
look
like
on
applications.
B
How
are
we
building
out,
and
so
we
were
working
closely
with
them
and
made
this
kind
of
early
commitment
that
we're
gonna
go
with
cloud
native
computing
foundation,
things
and
that
came
from
sort
of
the
first
decision
we
made
was
we're
gonna
move
to
kubernetes.
You
know
Kelsey
Hightower.
This
is
the
Linux
of
the
cloud.
This
is
the
future
we're
there
and
then
from
there
it
became
a
very
easy
commitment
to
say:
okay,
well,
we're
gonna,
do
Prometheus
we're
gonna,
do
at
the
time
open,
tracing
and
and
exist
within
that
ecosystem.
B
And
now
there's
like
500
things
on
it,
Oh,
1,200
and
five
things
on
it.
Sorry,
the
numbers
right
there
and-
and
it
very
quickly,
gets
to
a
point
where
you
you
just
feel
like
you're
you're
juggling
too
many
things.
You
know,
how
are
we
so
again?
Apps
are
made
for
Heroku,
not
for
containers.
How
do
we
containerize
rails
apps?
How
are
we
gonna
do
metrics?
Are
we
gonna
stick
with
New
Relic?
Are
we
gonna
do
something
else?
How
do
we
get
tracing?
Do
we
need
tracing
logging?
How
are
we
gonna
deploy
all
this?
B
You
know
we
know
how
networks
work,
but
we
haven't
had
to
manage
one
for
a
long
time
and
and
not
in
this
context
and
then,
most
importantly
and
critically,
how
are
we
gonna
train
the
developers
to
use
these
systems
to
get
these
things
out?
There
train
QA,
so
many
problems
and-
and
it
very
much
felt
like
this
and
and
if
you've
ever
read
the
Cat
in
the
Hat.
The
next
page
is
him
falling.
A
Yeah
so
Dave
originally
approached
me
in
October
2018
to
do
this
as
a
hackathon
project
for
our
company's
first
ever
two-day
hackathon
right,
which
the
idea
was
like.
Could
we
just
take
an
application?
That's
already
running
on
Heroku
figure
out
how
to
containerize
it
in
some
way
and
get
something
running
on
kubernetes,
so
to
choose
our
candidate
a
planarian
want
to
like
make
a
little
to-do
app
and
throw
it
on
kubernetes
and
be
like
yay.
A
We
learned
something
we
wanted
something
we
actually
had
on
production,
but
that
wasn't
our
like
full
with
webapp,
because
that
would
have
a
lot
of
consequences
right.
If
we
screwed
up,
we
need
to
basically
like
prove
to
the
company
that
this
was
a
thing
that
we
could
do
so.
In
the
end,
we
went
with
an
app
that
I
was
managing
at
the
time
called
photo
album,
which
was
both
kind
of
a
like
small
side.
B
Yeah,
at
the
time
we
had
teams
that
were
also
building
out
goji
RPC
micro
services
for
the
next
set
of
things
that
also
had
to
be
supported.
So
we're
gonna
start
talking
about
how
we
the
challenges
we
faced
in
doing
this
migration
globally,
but
also
on
this
kind
of
tracer
bullet
application
that
we
did
and
and
how
we
solve
them.
So
one.
C
B
First
things
we
ran
into
immediately
was
we'd
made
this
commitment
to
these
open
standards
and
found
that
they
were
very,
very
poorly
supported.
I
will
absolutely
pay
for
something
that
is
not
our
core
business
domain.
I
want
to
do
that.
I
do
not
have
time
to
run
every
single
underlying
operation.
Three
people
right
and-
and
we've
found
that
okay
great
well
we're
in
Google
Cloud,
that's
where
we
did
ended
up
planning
where
the
stack
drive
will
take
our
Prometheus
metrics
I
mean
just
ship
them
off
there,
and
then
there's
storage
and
everything.
B
So
I
turned
on
a
three
node
cluster.
Just
booted
up
installed
prometheus
nothing
but
Prometheus
in
this
cluster
started
shipping,
the
metrics
off
to
stack,
driver
and
I
did
some
math
based
on
the
the
bytes
of
metrics
that
were
coming
and
I.
Think
I
said
guys.
I
think
this
is
gonna
cost
us
like
$10
a
month,
and
we
want
to
guess
how
much
it
costs
yeah
two
thousand
dollars
a
month
just
just
to
ship,
with
no
applications
running
just
to
ship.
B
The
metrics
from
kubernetes
over
to
stack
drivers
so
and
you'll
find
a
similar
story
with
date
with
data
dog
like
it's,
you
know:
Prometheus
metrics
and
the
cardinality,
everyone
big
charges
and
cardinality,
and
that's
just
not
sustainable
for
this
right.
So
the
first
thing
we
did
is
we're
small
enough
to
DIY.
B
Until
support
comes
we're
in
this
nice
ground,
where,
like
we,
don't
have
a
team
that
can
run
from
a
theist,
but
also
we
don't
have
so
many
metrics
coming
in
that
we
blow
out
from
a
theist
in
every
you
know,
seven
hours
or
something
so
we
we
ended
up,
running
Jaeger,
running
Prometheus
inside
our
clusters.
For
for
the
time
being,
and
then
at
the
same
time
advocating
for
support
of
open
standards,
anytime,
a
vendor
would
come
knocking
at
our
door.
B
I'd
say
great:
do
you
take
prometheus
metrics
without
charging
me
like
an
arm
and
a
leg,
and
they
went
oh
well?
No,
you
know
great
Jesus.
Do
you
take
open
traces?
Can
I?
Send
you
open
traces
instead
of
your
proprietary?
You
know
APM
tracing
thing:
oh
well,
yes,
but
it's
gonna
cost
an
arm
and
a
leg
or
no
right.
So
finally,
I'm
we're
starting
to
see
this
start
to
shift
right.
B
Another
issue
we
had
was
DevOps
was
iterating
as
fast
as
developers
where's.
My
deaf
cluster,
my
dev
cluster
is
the
dev
cluster,
but
that's
also
the
developers
dev
cluster,
and
so
we
were
pulling
the
rug
out
from
under
developers
all
the
time
that
was.
That
was
a
huge
issue
for
us
early
on
having
the
infrared
really
helped
here.
It
was
easy
to
like.
If
we
worked
a
cluster,
we
could
bring
it
up
exactly
the
same
way.
Zng
terraform
get
everything
redeployed
very
quickly,
get
things
back
to
a
stable
place.
B
It
was
also
very
easy
to
bring
up
clusters
that
had
the
exact
same
spec
as
the
one
that
people
were
working
in
as
we
started
to
have
our
own
sandbox
clusters
to
run
in
and
I
could
just
deploy
everyone's
applications
there.
So
I
wasn't
running
a
fake
to-do
app
in
my
sandbox
cluster
and
and
that
kind
of
goes
to
planning
ahead
with
having
a
very
evolutionary
architecture
of
this
thing
can
kind
of
go
wherever
we
need
it
to
go
without
completely
reworking
it.
B
B
You
know,
and
and
the
solution
there
really
is
not
to
try
to
lock
things
down
right.
It's
just
create
same
guide,
rails
and
say:
look
here
are
the
tools
you
can
use.
I
trust
you
to
work
well
within
these
one
of
the
ways
we
did
this
was
we
gave
everyone
sandbox
projects
right
to
work
in
and
we've
said,
look
you
can
play
in
this
understand
that
I'm
gonna
trash
at
the
moment.
You
say
that
you're
ready
to
go
to
dev,
but
you
can
do
literally
anything.
B
You
want
in
the
sandbox
project
and
figure
out
what
you
need
and
then
we'll
go
and
kind
of
giving
developers
the
the
empowerment
to
go
figure
out
the
tools
they
want,
and
then
they
could
come
back
to
us
and
say:
okay,
how
do
we
operationalize
this
rather
than
us,
having
to
sit
and
figure
out
how
to
operate
guessing?
What
we
need
to
operationalize
months
ahead
of
time.
A
Yeah
at
that
point,
like
there's
a
lot
of
new
stuff
to
learn
right.
What
are
you
getting
out?
Flubber
means
with
service,
it's
a
scary
world
for
developers
to
go
into,
and
while
we
were
doing
this,
migration
right
I
was
on
a
team
of
four
developers.
That
already
has
you
know
our
entire
other
web
app
to
manage
two
other
apps
and,
like
we're
doing
our
actual
day,
job
and
kind
of
trying
to
do
this
on
the
side.
A
So
we're
being
asked
to
learn
a
lot
of
like
new
tools
and
infrastructure
and
everything
along
with
what
we're
doing
on
a
daily
basis.
So
the
way
we
kind
of
handled
this
was
like
I.
Was
the
team
champion
right
for
my
developer
team
I've
heard
a
lot
of
talks
on
this
today
and
people
are
trying
to
figure
out.
How
do
we
get
developers
to
adopt
this
kind
of
stuff?
And
the
answer
is
like
get
someone
on
the
dev
team?
Who
will
work
with
you
to
do
this
right?
A
Dave
and
I
were
pairing
all
the
time
we
were
documenting
things
that
we
were
discovering
together.
I
was
then
taking
those
to
my
team
and
running
little
training
sessions
on
the
new
tools
that
we
were
using
getting
people
on
board,
making
sure
our
QA
felt
comfortable
and
everything
and
like
really
this
works
when
it's
a
partnership
between
our
two
teams.
Right,
like
you,
need
both
sides
of
the
organization
to
be
bought
in
the
last
thing
you
want
to
do
is
have
your
DevOps
team.
A
A
We
also
had
been
working
on
this
app
right
that
had
been
running
on
a
platform
where
everything
was
just
provided
to
us
and
we
had
never
actually
written
a
docker
file
for
it
or
anything.
We
hadn't
containerized
this
thing
before
so
trying
to
do
that
like
presented
some
unforeseen
challenges
initially,
which
was
just
being
trying
to
figure
out
what
operating
system
are
we
supposed
to
run
this
thing
on
yeah.
B
I,
you
know
I
come
from
a
NGO
background
and
like
yeah
I
will
just
put
this
thing
on
Alpine
and
it's
gonna
be
great,
so
I
make
an
Alpine
container
and
I
get
the
thing
out.
It's
running
and
and
Tony
says
hey:
why
isn't
this
exporting
PDFs
and
I
go
it
exports
PDFs
and
she
goes
yeah
and
I
said.
Where
does
it
happen?
She
said?
Oh,
it
happens
on
the
box,
so
I
go
back
and
I.
A
So
it
kind
of
took
us
a
few
days
initially
of
honestly
just
trial
and
error
to
figure
out
all
the
packages
that
we
needed
on
this.
We
even
discovered
like
oh
yeah
in
these
PDFs.
We,
the
font,
that
we
actually
use
to
do
this
turns
out
to
be
some
paid
for
font.
That's
just
stalled
on
Heroku
x'
boxes,
so
now
I'm
like
we
don't
want
to
pay
for
that
in
order
to
have
it
so
now
we
need
to
go
back
to
designers
and
be
like.
Can
we
use
a
different
font
that
okay?
A
So
just
like
a
lot
of
little
things
like
this,
where,
honestly,
it
was
very
helpful
to
have
Dave
and
I
literally
sitting
right
next
to
each
other
pairing
it,
because
he
knew
how
to
do
all
these
things
and
I
knew
the
entire
domain
of
the
application.
So
us
working
together
really
like
cut
down
that
feedback
cycle
by.
B
Almost
like
that
is
DevOps
Tony
yeah,
translating
these
concepts
right.
We
have
this
whole
suite
of
features
that
developers
are
used
to
and
what
we
did
to
solve.
This
was
we
went
down
to
a
whiteboard
table
and
we
drew
out
everything
that
Heroku
does
on
columns,
and
then
we
drew
out
all
the
things
we
were
thinking
about,
buying
or
building
down
on
rows,
and
then
we
just
started
putting
X's
in
and
we
had
to
figure
out.
Okay,
what
do
we
actually
have
coverage
on
in
terms
of
our
plan,
but
also
what?
B
A
And
I
mean
one
of
the
great
things
about
Haruka's,
their
command-line
interface
right
you
can
deploy
through
there.
You
can
connect
your
database,
you
can
run
code.
You
can
see
stats
on
your
application.
That
kind
of
thing
so
part
of
what
I
was
doing
at
the
time
for
this
was
just
saying
like
okay,
what
are
all
the
equivalent
g-cloud
in
kubernetes
commands
that
I
need,
and
can
I
document
them
put
them
somewhere
make
easy
helpers
so
that
you
know
by
the
time
we
deploy
this
thing.
It's
not
like
my
whole
dev
team.
A
There
was
also
a
very
interesting
day
we're
doing
this
where
I
was.
Finally,
you
know
routed
all
our
traffic
over
to
our
staging
environment
gke
and
went
to
go
load
up
my
app
there
and
it
was
slow,
like
12
seconds,
to
load
slow
kind
of
thing
when
it's
normally
only
one
or
two,
and
so
I
want
to
looked
at
it
on
Heroku
and
it's
two
seconds
over
there,
and
it
was
the
exact
same
code
pointed
to
the
exact
same
database.
At
this
point,
so
I
was
like
okay,
I,
don't
know.
B
Is
a
symptom
of
what
I
like
to
call
the
Amazon
Web
Services
gravity
well,
and
that
is
that
everyone
is
in
Amazon
u.s.
East
one
any
vendor
you
work
with
launched,
probably
when
Amazon
was
the
only
game
in
town
and
they
launched
in
u.s.
East
one,
and
we
all
remember
when
s3
went
down
two
or
three
years
ago
and
brought
down
the
internet,
because
it
was
only
in
u.s.
East
one,
but
everyone's
only
in
u.s.
East
one
and
we
had.
B
We
know
this
and
we
went
okay,
cool
well,
we're
gonna
put
our
stuff
in
Google's
u.s.
East
one,
but
that's
in
South
Carolina,
and
it's
only
ten
milliseconds
of
travel.
But
this
app
made
several
hundred
database
calls
to
launch
its
main
page,
and
so
there
was
an
extra
ten
seconds
added
on
to
the
to
the
page
load.
Because
of
that
ten
milliseconds
of
difference
so
figuring
out
where
your
stuff
needs
to
go
in
relation
to
your
data
to
your
other
applications.
This
was
this
was
an
experiment
that
we
had
to
run.
B
What
we
ended
up
doing
was
launching
a
regional
cluster
in
Google
and
then
placing
the
app
on
nodes
and
various
zones
and
were
actually
able
to
find
the
Google
zone
in
u.s.
East
four,
which
is
their
North
Virginia
data
center
that
was
closest
to
whatever
zone
Heroku
in
our
in
the
black
box,
had
put
our
databases
in
Amazon's
u.s.
East
one,
because
some
of
the
google
centers
are
closer
to
amazon
than
the
other
amazon
ones,
and,
and
we
were
able
to
solve
this,
and
it
varies.
And
now
we
have.
A
Yeah
I
mentioned
earlier,
like
we
have,
you
know,
non-technical
people
doing
deployments
for
this,
so
you
know
just
throwing
kubernetes
at
them
in
general
is
going
to
be
kind
of
a
non-starter
here
when
they
were
used
to
just
going
promote,
so
we
ended
up
going
with
a
paid
service
actually
for
our
continuous
delivery
pipeline
right.
So
this
was
one
of
the
things
we
decided
not
to
build.
We
actually
got
something
that
has
a
whole
UI
around
it.
A
Where
you
can
watch
the
deployments
happening,
because
you
nice
little
status,
updates
actually
does
way
more
verification
and
check
of
things
than
we
used
to
get
and
still
had
capabilities
for
like
rolling
back
and
stuff,
which
was
kind
of
a
thing
that
we
definitely
had
to
iterate
on
a
few
times.
So
it's
like
how
do
we
recreate
Heroku
is
kind
of
deployment
pipeline
in
what
order
should
we
actually
deploy
all
our
different
pods
and
everything?
And
how
do
we
know
when
we
can
rollback
and
how
do
we
roll
back
safely,.
B
Yeah
routing
issues
who
there's
been
talks
of
various
things
here,
like
once
you
own
your
own
network,
you
gotta,
understand
how
everything
goes
in
and
where
all
the
translation
layers
are.
So
one
of
the
first
things
that
I
learned
about
rails
is
that
rails
doesn't
know
what
TLS
is.
It
knows
to
be
strict,
that's
about
it
and
to
serve
secure
cookies.
B
So
so
one
of
the
and
to
not
accept
things
that
haven't
been
marked
is
TLS
traffic,
but
you
can't
give
certificates,
so
we
needed
to
find
a
way
to
support
this,
to
terminate
TLS
in
front
of
the
rails
app.
He
also
needed
to
find
a
way
that
serviced
all
the
new
G
RPC
apps
that
were
coming
up
in
the
micro
services
that
were
being
built
around
this
thing.
They
were
also
being
deployed
and
what
we
ended
up
doing
was.
First
Tony
I
got
the
fun
job
of
coming
back
to
Tony's,
saying
Tony.
B
You
need
to
upgrade
to
Rails
5.
Before
we
can
do
this.
It
turns
out
that
rails
5.
We
really
wanted
liveness
and
readiness
checks,
probes
and
the
only
way
you
can
get
those
is,
if
you
accept
exempt
your
health
check,
endpoint
from
being
strict
in
rails,
and
that
was
not
supported
until
rails
5.
So
that
was
a
big
thing
and
then
we
found
this
open
source
project
called
Ambassador.
B
That
has
just
been
like
my
favorite
tool
and
the
most
like
solid
thing:
we've
been
working
with
for
the
last
20
months
or
so,
and
it's
a
Python
wrapper
around
envoy
that
acts
as
an
API
gateway
and
it
supports
G
RPC.
It
supports
rest,
it
supports
just
plain
TCP
traffic
it'll
do
termination
it'll
do
termination
on
multiple
SN
is
like
it.
It
is
this
like
it
is
the
Swiss
Army
knife
of
traffic
entering
our
cluster.
That's
allowed
us
to
execute
a
lot
of
these
things.
That's
a
really
great
project.
A
B
And
this
slide
is
a
testament
to
Heroku,
because
you
know
we
had.
We
had
to
put
Kabul
all
this
together
into
into
a
workable
framework.
So
we
start
at
CI.
We
have
circle
CI,
which,
by
the
way,
is
the
reason
we
couldn't
look
at
spinnaker.
We
looked
at
spinnaker
for
five
seconds
and
it
was
like
it
works
with
Jenkins
and
Travis,
and
we
went
okay
well
we're
not
rewriting
every
CI
file.
We've
ever
had,
let's
gotta
find
something
different,
so
circle
builds
things
either
goes
to
artifactory
or
docker
hub
pro
tip.
B
B
B
Harness
so,
if
you've
been
to
any
spinnaker
talks,
you
probably
seen
like
there's
a
pot.
The
pattern
is
a
hard
division
between
CIN
CD
right,
CI
bills.
The
artifact
CD
puts
it
somewhere.
So
that's
what
harness
we
use
it
for
that
puts
the
containers
at
two
gke.
We
build
our
GK
clusters
using
terraform,
so
they're
very
repeatable.
We
can
bring
them
up.
We
can
move
them
around.
B
Situs
is
an
open-source
extension
that
shards
Postgres
very
similar
to
the
test,
but
without
all
the
CNC
F
like
magic
around
it,
we
use
Prometheus
in
terms
of
open
or
advocating
for
open
standards.
Summa
logic
was
our
logging
partner
for
a
long
time.
They
just
released
a
product
that
takes
in
Prometheus
metrics,
based
on
volume,
not
cardinality
and
it's
affordable.
So
we
now
push
our
Prometheus
metrics
out.
There
ambassador
is
a
gateway
at
the
edge,
and
then
we
have
a
CDN
in
front
of
that
Jager
for
tracing
there's
more
coming
yeah.
B
A
But
from
the
developer
perspective,
we're
really
only
dealing
with
circle,
CI
harness
gke
and
sumo
logic
for
part
of
it,
so
those
are
kind
of
some
of
the
pieces
that
we
took
over
and
kind
of
as
a
reminder
here
right.
We
started
this
in
October
2018
as
a
hackathon
project.
We
sort
of
worked
on
it,
just
the
two
of
us
on
the
side.
A
For
a
few
months
right,
we
ended
out
actually
going
to
production
in
April
20
19,
having
moved
this
app
over
there
sort
of
waited
a
few
days
to
tell
the
company
like
any
good
developers.
The
case
stuff
goes
wrong
and
when
no
new
production
issues
that
came
up
right
call
this
success
and
this
thing
has
been
running
in
gke
ever
since
so.
B
Where
some
things
we
gained
out
of
this,
we
still
have
easy
deploys.
Now
we
have
a
lot
of
additional
verification
on
the
backend
checking
logs
and
metrics
after
a
deploy
goes
out
to
make
sure
it's
good.
If
it's
not,
we
roll
it
back.
We
have
much
more
control
over
migrations
and
rolling
back.
We
can
place
the
migration
exactly
in
the
pipeline.
Where
we
want,
we
can
have
it
run
manually.
We
have
a
lot
of
options
there.
We
have
far
more
control
over
our
scale,
because
Google
I
should
go
really
really
big.
B
If
you
haven't
heard
we
can
roll
back
and
forward,
it
will
using
those
containers
to
whatever
version
we
want.
We
have
blue
green
deploys
now,
which
was
not
a
capability
we
had.
Hopefully
we're
gonna
be
able
to
do
canary
soon,
as
I
figure
out
how
to
do
service
meshes
and-
and
we
run
for
like
one
of
the
best
parts
coming
out
of
this
is
as
an
ops
team.
It
was
very
much
like
we
were
just
kind
of
putting
out
fires
and
having
to
do
this.
Proactive
work
work
very
closely
with
the
developers.
A
A
Sometimes
it
was
me
sitting
figuring
out
all
the
commands.
That
would
allow
me
to
basically
connect
to
a
console
similar
to
what
I
was
used
to,
so
that
I
could
hide
it
behind
some
alias.
So
the
rest
of
my
team
could
have
this
and
scheduling
training
sessions
when
we
put
harness
in
place
to
walk
my
whole
team
through
and
let
them
have
questions
those
sometimes
getting
the
hardest
people
to
come.
You
know
lead
trainings
for
us
and
stuff
like
that.
B
On
the
upside,
it's
knowing
what
you
want
to
optimize
for
you
know
on
day,
2
of
trying
to
figure
out
Alpine.
It
was
like
we
need
to
get
this
thing
out,
not
small
right,
let's
just
let's
just
optimize
for
that
and
go
with
Debian
right
and
my
number
one
lesson
in
24
months.
Maybe
30
months
of
working
in
the
cloud
native
community
is,
can
you
wait
six
months?
This
is
the
question.
You
should
always
be
asking
yourselves
if
you
need
a
thing
right
now
today,
absolutely
do
it
go
with
the
best
tool
possible.
B
But
if
you
can
wait
six
months,
you
should
because
six
months
ago
we
thought
about
doing
service
meshes
and
I
was
like
yeah.
This
is
gonna,
be
really
bad,
a
huge
pain.
It
looks
like
well
two
years
ago.
I
was
writing
my
own
envoy
X
DSS,
and
then
they
went
to
version
2
and
then
sto
came
out
and
now
that
looks
easy,
but
it
looks
like
much
better
but
harder
to
install
now.
You
know
I
could
almost
condense.
This
too.
B
Can
you
wait
six
hours,
because
yesterday
I
was
like
wow
link,
Rd
looks
really
cool
and
then
six
hours
later
I
found
super
glue,
which
just
is
just
like:
deploy
sto
and
linker
D
disable
one
of
them
like
it's
just
like
three
command-line
things.
Now
it
this.
We
rapidly
that
that,
if
you,
if
you
honestly,
can
say
yeah,
we
don't
need
mutual
TLS
right.
This
second,
just
just
wait
like
push
it
until
until
you
can,
because
it's
probably
gonna
be
easier.
B
So
the
final
thing
is
I
think
we're
actually
over
time
now,
and
so
I'll
talk
very
quickly
on
this,
as
was
it
worth
it
right,
yes,
and
and
for
a
number
of
reasons,
first,
we're
saving.
Well,
we
were
110
thousand
dollars
a
month,
just
in
Heroku,
plus
some
cloud
bill
and
data
pipeline
and
stuff
we're
now
forty
thousand
dollars
a
month
for
our
entire
stack,
taking
ownership
of
the
databases
and
the
and
the
stuff
was
was
hugely
important.
We
can
scale
more.
It's
just
been
incredible.
Developers
have
more
understanding
of
the
ops.
B
A
E
At
what
point
in
your
in
the
project
did
you,
when
you
were
looking
at
rhe
platforming,
basically
with
a
new
tool,
set
at
what
point,
or
did
you
ever
sort
of
introspect
and
ask
the
question
based
on
whatever
the
capabilities
that
we
have
to
have
today,
because
you're
moving
from
Heroku?
What
did
you
struggle
with
the
question
of?
Should
we
alter
the
behavior
of
what
we're
trying
to
do
with
our
application
to
reduce
toolchain
complexity,
or
did
you
just
say
now?
E
We're
gonna
find
a
tool
to
do
that
exact
functionality
and
then
we'll
just
add
it
to
the
chain
like
that?
How
did
you
struggle
with
that,
because
you
had
to
look
for
a
tool
anyway,
yeah
did
you
attempt
to
reduce
complexity,
or
did
you
say,
nah
we're
fine
with
as
much
technical
debt
as
we
can
get
no.
B
B
That's
you
know,
I
mean
part
of
the
thing
for
us
to,
since
we
are
maturing
like
we
do
have
to
actually
look
at
a
more
comprehensive
security
model
at
a
more
comprehensive
networking
model,
and
so
that
alone,
like
we
kind
of
had
to
this,
was
a
part
of
this-
was
almost
an
excuse
to
be
like.
Oh,
we
kind
of
can't
let
you
do
that
anymore
and
when
it
was
really
like,
we
really
don't
want
to.
Let
you
do
that
anymore,
so
so
they're
definitely
pieces
that
we
dropped.
You
know
when
Tony
was
talking
about.
B
How
do
we
map
the
Heroku
deployments
into
harness?
One
of
the
things
we
did
was
what,
if
this
looked
different
right
like
we
don't
have
to
100%
map,
how
we
deployed
things
in
Heroku
to
how
we
deployed
things
in
in
kubernetes,
one
of
the
things
I
do
that
example,
and
that
almost
expands
capabilities
is
you
know.
Heroku
is
this
concept
of
like
PR
review
apps,
where
you
open
up
a
PR,
just
spins
up
your
copy
of
the
app
separately
from
the
main
one
and
everyone
kept
asking?
B
Well,
how
are
we
gonna
do
that
for
all
50
micro
services
we're
doing
and
I
can't
we
have
a.
We
have
a
dev
cluster
everything's
in
there.
We're
gonna
give
you
your
own
namespace,
you
can
own
it.
You
can
deploy
stuff
in
there,
but
but
this
is
a
fundamentally
different
way
of
operating
and
so
yeah.
There
was
definitely
like
things
have
changed
about
how
we
do
things.
Sir
yeah,
as.
D
B
We
had
a
couple
things
that
were
still
running
an
ECS
from
before
and
we
use
harness
to
do
function,
deployment
where
the
integrate
CI
tarballs
up
the
function
code
and
puts
it
in
artifactory
and
then
in
harness.
We
pull
it
down
and
we
run
the
CLI
command
on
our
on
our
run
box
to
deploy
the
function
code,
which
is
really
cool
because
it
allows
us
to
rollback
functions
because
we
know
that
like.
Oh,
that's,
the
immutable
build
129
of
those
functions.
It
broke
just
redeploy
128
now,
so
it's
it's
really
like.
B
We
needed
more
than
I.
Think
in
a
couple
of
in
maybe
two
years,
we'll
have
our
databases
and
kubernetes
will
probably
be
running
k
native,
so
our
functions
will
be
running
in
kubernetes
and
then
we
could
actually
look
at
like
only
kubernetes
tools,
but
my
big
problem
going
up
to
the
vendor
hall,
sometimes
is
everyone's
like
yeah.
This
is
so
great
for
just
kubernetes
cuz.
That's
all
you
need
and
I'm
like,
and
it's
really
not.
We
have
developers
who
do
other
things.
Yeah
give.