►
From YouTube: How Kubernetes Was the Secret Sauce in Our Globally Distributed Team’s Transition to Microservices
Description
How Kubernetes Was the Secret Sauce in Our Globally Distributed Team’s Transition to Microservices - Dan Farrelly, Buffer
It can be hard to fundamentally change how applications are built at a fast moving SaaS startup. Getting buy in, training and changing engineer’s daily workflows are some of the many challenges. Working as a fully remote team, often these challenges can be magnified. It’s often hard to know where to start when your team is trying to move fast, but you know how much your team will benefit from everything that Kubernetes has to offer. Our remote team had been struggling to scale our development process as we’ve grown. A couple of years ago, our team rallied around the idea that we would ship code the minute it is better than what is in production. With a single monolithic application and a growing engineering team, this became a bottle neck and deploying our tightly coupled app started to create new challenges and problems. We needed a new way for each part of our application to deployed independently and quickly. Kubernetes was the missing piece that has allowed us to stay true to our engineering culture, empowering out team of engineers to move fast while also having a resilient and stable application infrastructure. If you’re trying to figure out how to get your company to start using Kubernetes or want to encourage wider adoption on a fast moving team, you’ll need to have a plan. In this presentation, you’ll learn about one plan that worked and is continuing to work at a 6 year old company whose team has grown from 10 to 80 since they last updated their infrastructure. This presentation hopes to illustrate how Kubernetes has enabled our infrastructure to reflect our organization itself; a distributed group of autonomous people aiming to work together while not getting in each other’s way.
About Dan Farrelly
I'm leading up Buffer's efforts to move to a service oriented architecture, replacing a monolithic php application with lots of Node and php microservices running on Kubernetes. Talk to me about SOA, Node.js services, scaling k8s usage across a team, deploying to a k8s cluster.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
How
kubernetes
has
been
instrumental
in
helping
our
remote
team
and
I'll
walk
you
through
the
before
infrastructure
of
what
we
had
and
still
have
in
our
legacy
infrastructure
and
some
of
our
challenges
that
we
had
and
I'll
talk
a
little
bit
about
our
moves
of
kubernetes
and
some
things
that
we
learned
along
the
way.
Lastly,
I'll
share
where
we're
at
today
and
what
we're
planning
to
do
next,
so
yeah
getting
started
just
a
little
little
break
about
me,
I'm,
the
the
architect
of
buffer
and
buffer.
A
If
you
don't
know,
if
you
haven't
heard
of
us,
we
are
a
software
company
that
builds
a
social
media
management
tool
that
allows
you
to
schedule
and
collaborate
on
social
media
posts
across,
like
all
the
different
social
networks
and
on
hundreds
of
different
profiles,
as
well
as
like
pulling
analytics
and
and
see
how
well
you're
doing
on
that.
This
is
for
publishers
and
and
and
marketers
and
whatnot.
A
I
have
two
words:
if
you
like
to
connect
with
me,
you
know,
shoot
me
an
email
or
talk
to
me
afterwards,
but
I'd
love
to
hear
or
feedback
or
suggestions
or
whatnot
yeah,
so
yeah.
This
is
a
little
a
little
diagram
showing
off.
You
know
what
our
infrastructure
was
in
the
middle
of
this
year.
Most
of
it
is
is,
this
is
still
vaguely
is
still
true.
We
haven't
moved
too
much
with
the
existing
app
but
yeah.
A
This
is
just
a
mile
high
view
really
all
of
our
different
server
environments,
we're
on
AWS
we're
using
elastic
Beanstalk
to
just
deploy
copies
in
these
different
environments.
We
have
one
massive
repo
that
everyone's
duh
is
deploying
up
on,
and
you
know
that
that
repo
is
deployed
to
every
single,
every
single
every
instance.
A
So
it's
a
pretty
tightly
coupled
and
just
different
environments,
execute
different
parts
of
the
of
our
big
repo,
and
you
know
with
you
know,
with
Beanstalk
we
were
having
weird
kind
of
scaling
stuff
where
we,
you
know
not
able
to
find
tune
or
or
worker
instances
and
different
processes
that
we
were
running
on
each
of
our
servers.
So
you
know
this
is
a
little
bit
about
the
buffer
team.
We
are
a
team
of
80,
fully
working
remote
scattered
around
the
world.
We've
pretty
much
heard
someone's
online
24
hours
a
day.
A
If
you
can
see
here,
these
are
our
our
team
structure
by
timezone,
and
you
know
we
kind
of
have
not
every
one
in
each
one
of
these
areas
in
that
city.
That's
just
kind
of
you
know,
as
you
see,
there's
58
cities
that
we
have.
So
it's
we're
kind
of
scattered,
so
it's
a
little
bit
of
a
a
typical
kind
of
company,
but
it's
getting
more
popular
as
you
see
these
days,
so
our
engineering
team
is
roughly
25
of
the
people
that
you
see
here.
A
We
have
15
people
regularly
committing
and
deploying
code
to
that
massive
repo,
that's
getting
shipped
everywhere.
We
have
like
three
kind
of
DevOps
type
engineers,
two
of
which
that
mostly
reside
in
in
Asia
and
Taipei
and
Sri
Lanka,
and
one
that
kind
of
floats
around
between
Beijing
and
France
and
wherever
he
chooses
to
be
so
yeah.
There's,
there's
many
positives
on
on
being
remote,
there's
a
lot
of
cool
things,
but
I
think
it's
it's!
It's
also
key
from
what
I'm
talking
about
it's
highlight
some
of
with
a
remote
team.
It's
really!
A
You
can't
call
you
can,
but
it's
difficult
to
call
a
single
meeting
and
really
get
everyone
on
the
same
page
immediately.
It's
kind
of
hard
to
just
grab
all
the
engineering
team,
rounded
up
and
gets
a
get
a
lunch
together
and
just
and
just
roll
something
out
or
start
a
new
agenda.
You
know
also
without
the
ability
to
tap
someone
on
the
shoulder,
for
you
know,
quick
question
on
something
to
be
harder:
pushing
through
technologies
on
the
team.
We
we
basically
you
know,
we
we
have.
A
Yeah
it's
difficult
because
you
don't
have
direct
access
to
the
people
who
might
be.
You
know
the
the
in-house
expert
on
on
the
technology,
so
you
might
have
to
wait
for
hours
until
I
wake
up
so
there's
different
challenges.
We
have
to
like
really
focus
on
being
a
synchronous
and
all
our
developers
need
to
be
very
independent
as
much
as
they
can.
So
you
know,
there's
also
a
thing
you
know:
that's.
A
We
need
to
give
some
empathy
to
our
developers
because
they
might
work
in
a
place
like
Sri,
Lanka
or
somewhere
on
Eastern
Europe,
or
you
know
when
you
know
somewhere
in
the
Pacific
that
don't
they
don't
have
the
overlap
with
that
and
the
access
that
all
engineers
do
so
it's
you
know
we
need
to
make
sure
we
take
all
those
things
into
consideration
whenever
we
do
anything
at
buffer,
so
yeah.
So
so
these
are
some
of
our
challenges.
A
Like
I
talked
about
buffered,
you
know
kind
of
had
that
classic,
like
lean
startup
kind
of
mentality
started
about
five
years
ago,
and
just
kind
of
like
kept
tacking
things
on
kept
chipping
things
trying
to
grow
like
growth
was,
was
a
number
one
and
kind
of
just
couldn't
pull
her
foot
off
the
gas
to
like
we
re
architect
the
app
and
pay
down
some
technical
debt,
so
that
that
struggles.
You
know
when
you
get
larger
and
you
know
so
it
was
something
we
had
to.
A
We
had
to
address
our
our
engineering
culture,
something
we
take
pride
in.
Is
you
know
we
have
this
the
saying
of
you
ship
code,
the
minute
that
it's
better
than
what's
on
production,
so
we
just
as
our
culture
in
your
DNA
like
we
ship
all
day
long.
So
that's
you
know
we're
deploying
things
24
hours
a
day,
and
you
know,
as
you
can
tell
you
know,
we're
all
shipping
to
the
same
environments
same
code.
There
can
be
a
lot
of
conflicts.
Merges
are
a
little
bit
messy.
A
Sometimes,
and
you
know,
with
our
tightly
coupled
application
like
this.
Is
you
know
things?
Could
there
was
updated
changes
for
hours
earlier,
a
library
someone
might
have
not
known
about
five
hours
later
and
deployed
and
broke
something
we've
gotten
better
at
over
ting,
some
of
those
things,
but
still
it's
it's
it's
a
bit
of
a
bottleneck
as
especially
as
we
we've
grown
our
team
over
the
last
couple
of
years,
especially
two
year.
Two
years
ago
we
were
under
under
20,
so
we've
got
a
lot
more
engineers
and-
and
you
know,
more
deploys
we're
making
bottleneck.
A
People
are
queuing
up
for
deploys
and
we
don't.
We
don't
have
the
sense
of
we
have
more
atomic
deployment
rather
than
shipping
a
bunch
of
people
like
PRS
all
in
the
same
on
the
same
deploys.
So
you
know
we
also
kind
of
the
way
that
we
grew
and
being
a
remote
team.
We
had
limited
senior
people
were
there
and
that
really
days
who,
like
know
how
everything
works
and
I
think
kind
of
tying
into
the
next
point.
A
We
have
no,
we
had
no
logging,
which
made
it
even
harder
for
anyone
to
really
figure
out
what
was
going
on
with
their
application.
There's
a
lot
of
justice
system
intuition
of
senior
people
on
the
team,
so
you
can
see
like
these.
Are
these
are
issues
and
we
got
by
somehow
without
these
and,
like
you
know,
we're
a
very
transparent
company
and
we're
doing
you
know
twelve
million
dollars
a
year
in
revenue
annually
and
that
might
be
small
potatoes
for
a
lot
of
people.
A
But
for
us
it's
it's,
it's
pretty
cool
and
you
know
I,
don't
regret
the
decisions
that
were
made.
These
are
just
kind
of
some
of
the
challenges
that
we
have
today.
So
you
know
I,
don't
know
if
anyone
knows
the
movie
Roadhouse
by
yeah
it
in
with
Patrick
Swayze
from
1989
I
like
to
bear
our
situation
to
that.
If
you
haven't
seen
the
movie
or
don't
know
the
premise,
I'll
try
to
give
a
quick
rundown,
so
I
think
it's
important.
So
basically
the
movie
takes
place
in
a
small
southern
town.
A
At
this
bar
called
the
Double
Deuce,
where
every
night
things
are
just
kind
of
little
out
of
control.
Their
fights
like
bouncers
can't
control
people,
they're
broken
bottles
everywhere.
Like
you
know,
people
are
throwing
things
at
the
band
and
it's
like
the
bar
is
in
a
bad
way
and
I
needed
to
you
know
it
needed
to
be
helped
out.
So
that's
where,
like
Patrick
Swayze
comes
in
he's
he's
this
and
there's
a
moment
straight
from
Google.
He
is
a
hotshot
PhD,
educated
bouncer.
A
Who
knows
exactly
how
to
clean
things
up,
so
we
needed
our
PhD
educated
bouncer,
basically
I'll
not
saying
that
our
instructors
as
bad
as
Roadhouse,
you
know,
but
regardless,
like
we
needed
something
to
something
to
happen,
what
we
needed.
You
know
we
need
to
look
for
own
Swayze
when
you
wanted
to
make
major
leaps.
A
We
didn't
just
want
to
make
incremental
changes
and
do
things
we
wanted
to
just
like
completely
like
you
know,
ten
times
what
we
what
we
had
and
we
needed
a
really
way
like
how
do
we
escape
the
model
if
we
talked
about
it
for
a
year
and
a
half
and
it
was
difficult
like
we
just
didn't,
have
experience
on
the
team.
We
wanted
to
be
able
to
create
and
ship
new
services
really
quick.
You
know
to
dress,
you
know
our
scaling
challenges,
but
a
previous
infrastructure.
A
We
wanted
like
fine
grained
control
over
all
our
services
on
rather
than
scaling
these
big
hunky,
like
ec2
instances
on
on
on
with
elastic
beanstalk,
so
we
we
also
wanted
the
straight
through
to
the
team
culture
like
we
wanted
to
deploy
constantly,
and
we
wanted
people
to
just
deploy
right
from
slack.
The
second
that
they
wanted
to
and
be
able
to.
You
know,
coordinate
their
deploys
right
right
there,
so
we
we
really
really
needed
logging.
That
was
something
that
was
incredible
and
majorly
struggle
to
this
in
the
past.
A
So,
on
yeah
and
I
think
the
last
like
you
know,
we've
been
using
docker
and
development
for
about
a
you
know,
six
months
to
a
year,
depending
on
what
projects
you
were
working
on,
but
we
had
very
little
experience
running
containers
in
production.
We
were
kind
of
you
know
rookies
at
containers,
if
you'd
like
to
say
so,
we
were
just
kind
of
focused
on
shipping
things
and
getting
or
getting
our
single-page
app
working
and
whatnot
and
just
delivering
value
to
the
customer
as
soon
as
possible.
So
yeah.
A
So
you
know
after
this
you
know:
how
are
we
gonna
how
we're
gonna
address
these
challenges?
You
know
we
want
a
lot.
We
valued
it
a
handful
as
the
solutions
and
I'm
sure
the
guests
that
we
landed
on
kubernetes,
it's
our
Tai
Chi
practicing
Patrick
Swayze.
You
know
that
had
all
the
tools
that
we
needed
to
get
set
up
and
get
it
ordered
to
get
everything
straight.
A
It
also
gave
us
a
solid
foundation
to
choose
the
right
solution
for
for
each
challenge
really
for
us
at
this
point
either
like
this,
this
road
house,
reference
is
making
sense
a
lot
or
it
is
completely
obscure
and
you're
just
happy
to
I'm
talking
about
kubernetes
now
so
yeah
so
with
you
know,
we
kind
of
chose
on
kubernetes,
and
then
we
we
decided,
you
know
what
what
are
we
gonna
do
now.
You
know:
what's
how
we're
gonna
make
this
thing
successful?
A
What
would
be
our
guinea
pig
for
this
for
Cooper
for
kubernetes
journey
kind
of
on
maybe,
most
importantly
like
we
had
just
educate
the
team,
like
everyone
just
really
needed
to
know
how
how
to
use
all
these
things-
and
you
know
I
just
had
to
jump
in
without
having
to
read
the
docs
and
learn
all
the
mistakes
that
you
learn
when
you
get
into
it
for
the
first
time.
So
we
kind
of
we
divided
a
plan.
A
high-level
was
pretty
easy.
We
split
it
up.
A
It
was
for
engineers
the
three
DevOps
guys
that
I
mentioned
and
myself
kind
of
headed
up.
This
thing
said
like
we
just
need
to
make
this
successful
start
it
up.
I,
guess
it's
about
the
middle
of
summer,
or
so
each
one
of
us
owned
one
of
these
one
of
these
areas
and-
and
you
know,
collaborate
every
day
when
we're
at
you
know
we
were
having
meetings
in
the
morning
at
night,
so
I
could
catch
them
in
Asia
or
Europe
wherever
they
were,
and
and
really
just
it
was.
A
A
quick
iteration
cycle
was
really
great
to
feel
the
energy
of
what
we
were
doing.
They're
constantly
like
every
day,
making
progress
in
each
area
so
walk
through
a
little
bit
of
the
plan.
How
we,
how
we
kind
of
executed
on
some
of
these
things,
what
we
chose
so
yeah,
first
I
guess
the
most
important
is
getting
cluster
out.
You
know
we
need
to
get
in
the
weeds
with
kubernetes.
We
have
little
experience,
running
containers
and
productions.
So
no
experience
with
kubernetes
more.
You
know
core
contributors
or
anything.
A
So
we
kind
of
just
thought
like
hey:
let's
take
this
off
the
shelf
solution
and
see
if,
if
we
can
make
this
work,
so
you
know
we
thought
we
didn't
really
think
anything
of
it
so
yeah.
Why
not
yeah
sure
we
can
do
this
thing,
so
you
know
we
not
only
just
like
standing
up.
Our
first
cluster
was
important,
but
we
had
to
get
some
other
things
figured
out
early
on.
We
stand,
stood
up
a
lot
of
clusters
just
torn
down.
Did
a
lot
of
testing
figured
out
how
it
worked
internally?
A
A
You
know
we
we
ended
up
using
some
service
accounts
for
just
quick
and
dirty
access,
but
everyone
has
full
access
to
every
namespace.
We
have
four
different
different
teams
in
the
company
kind
of
like
pretty
loose
right
now
and
it's
not
as
secure
as
we
like,
but
you
know
what
it's
allowing
us
to
share
knowledge
across
namespaces
and
being
being
able
to
pop
into
other
people's
areas
and
just
see
like
oh,
hey,
you're
doing
this.
A
This
is
very
interesting
and
kind
of
share
knowledge
throughout
the
team
pretty
quickly
and
be
able
to
help
people
out
so
yeah.
We,
you
know
kind
of
stored
everything
in
a
private
git
server
with
a
lot
of
lot
of
our
resource,
gamma
files
and-
and
you
know
it
was
okay-
it's
it
got
us
at
least
moving.
You
know
it
was
not
ideal
was
not
the
perfect
way
that
we
needed
to
do
things,
but
it
it
got
us
to
moving
and
starting
to
use
kubernetes,
and
you
know
seeing
things
light
up.
A
So
you
know
these
choices
that
we've
made
for
these
I
don't
know
highlight
these
aren't
recommendations
per
se,
but
these
are
things
that
we
chose
and
how
we
balanced
our
time
or
resources
the
cost
to
maintain
these
things.
They
were
the
right
call
for
buffer,
and
you
know
everyone
has
different
needs
and
constraints.
So
you
know
take
these
with
a
grain
of
salt
and
you
know
do
whatever
you
is
best
for
your
team.
A
Really,
you
know
this
also
is
highlight
is
still
an
area
that
we're
pretty
we're
pretty
challenged
with
I
think
you
know,
you
see
a
lot
of
those
vendors
out
there
and
they're
there.
You
know
have
things
about
standing
up
clusters
cluster,
you
know
kind
of
just
provisioning.
It's
still
something
that
you
know
we
we
do
need
to
focus
on
a
lot
more
so
and
I
think
you
know
with
kubernetes
we're
standing
on
the
shoulders
of
giants
as
you'd
say,
but
we
don't
really
know
the
giant.
A
Well,
we
don't
know
the
inner
workings
too
much
so
we're
constantly
digging
in
and
finding
new
things
and
getting
in
and
reading
the
source
and
understanding
how
things
work.
So
it's
it's
been
a
fun
journey,
but
you
know
that
was
this
is
this?
Is
the
first
step
kind
of
pushing
us
forward?
We
we
also
wanted
deploys
to
go
out
really
fast.
A
So
one
of
the
things
we
also
did
was
really
fond
of
deploying
with
bots
right
in
slack,
so
slash,
deploy
whatever
your
services
should
be
as
simple
as
as
you
really
need
to
get
it
out
there.
We
wanted
really
just
super
smooth
for
everyone
to
kind
of
handle
and
be
able
to
to
approach.
So
we
just
a
few
things
together.
A
A
So
we
ended
up
just
rolling
our
own
bots
that
we're
hosting
on
kubernetes
itself,
which
was
super
super
great
and
has
a
little
little
dashboard,
that
developers
kind
of
insert
their
their
git
repo
and
what
doctor
filed
to
build
and
where
to
where
to
ship
it
to
and
and
that's,
and
it's
as
easy
as
that,
it
shows
up
right
away.
So
we,
you
know
I
bought
itself
as
hosts
on
kerbin
eyes,
like
I
said
it
handles
our
own.
A
Its
own
deploys
and
our
deploys
right
now
are
out
in
45
to
90
seconds,
which
is
awesome
compared
with
our
legacy.
App
still
hovering
around
three
to
four
minutes,
and
it's
at
times
struggled
and
was
out
at
like
five,
so
we
we
wanted
to
move
faster,
we're
deploying
all
the
time.
So
you
know
this
is
already
having
an
impact
on
getting
our
new
services
and
getting
bug
fixes
improvements
like
so
much
faster.
This
has
really
helped
all
the
developers
started
to
take.
Take.
Take
advantage
of
this
I
really
really
appreciate
it.
A
It's
it's
really
built
a
lot
of
confidence
into
our
team,
especially
non
developers,
designers
or
people,
editing,
editing,
static,
static
pages,
just
content
on
our
jobs
website
or
something
as
well
as,
like
our
data
analysts,
they've
they've,
extremely,
like
they've,
improved
their
their
iteration
cycle
rapidly.
They
used
to
wait
for
an
engineer
to
get
on
to
deploy
their
updated
script.
That's
generating.
A
You
know
some
of
our
calculations
on
user
behavior
or
finances,
and
now
they're
just
kind
of
shipping
their
content
there,
their
new
scripts,
constantly
they've,
just
absolutely
loved
it
and
there's
there's
so
many
more
so
much
more
that
we
can
do
for
them,
which
is
just
it's
opening
up
a
lot
of
doors
for
even
just
our
data
team.
We
didn't
think
that
they
at
the
beginning
of
this
would
even
take
advantage
of
it
too
much.
A
So
it's
it's
been
awesome
and
you
know
one
of
the
things
that
we
kind
of
after
deploys
was
going
towards
looking
at
logging,
as
you
saw,
this
was
a
real
pain
point
for
us.
You
really
need
to
get
this
part
right
and
we
started
off.
We
knew
we
wanted
to
get
our
logs
in
elasticsearch.
We're
really
fond
of
that.
Just
pulling
things
out
with
kibana.
We
experimented
with
a
few
different
ways
to
do.
A
This,
wanted
to
really
have
just
a
couple
weeks
for
the
logs
on
the
last
search,
one
location
for
all
our
apps
on
all
our
services
and
then
maybe
some
long-term
storage
too.
So
we
also
one
of
the
like
standardize,
our
logging
format
a
little
bit,
so
you
could
search
on
something
a
related
ID
or
something
like
that:
a
user
ID
across
all
your
different
logs
across
all
your
different
applications.
You
can
see
user
behavior
what
was
happening
to
their
data
that
this
is
being
great.
A
We
had
to
you,
know,
standardize
and
really
think
about
some
of
these
things.
You
know
key
was
getting
some
of
this
stuff
out
in
front,
so
when
developers
got
it,
they
just
could
see
something,
read
it
and
just
implement
it.
We
also
abstracted
a
lot
of
this
stuff
just
into
our
own
like
little
libraries
and
have
like
some
documentation,
so
they
could
npm
install
or
pipe.
You
know,
pip
install
our
own
package
and
they
just
kind
of
have
everything
straight
away,
yeah.
A
So
starting
off,
you
know
we
that
our
cluster
up
with
the
kuba
script-
and
we
you
know
just
turned
on
the
fluent
D
logging
into
elasticsearch.
This
was
you
know,
working
during
testing
which
was
great,
and
then
we
upgraded
to
1.3
and
had
coop,
DNS
and
realized.
It
was
a
lot
more
talkative
than
we
thought
it
was
going
to
be,
and
it
just
filled
up
our
volume
on
one
of
our
hosts
and
crashed.
It
crashed
a
minion
so
on
we
were.
A
Rookies,
do
you
know
we
weren't
used
to
running
stateful
containers
and
we
didn't
really
know
about
what
was
going
on
there.
So
we
we
just
thought
you
know
what,
let's,
let's
figure
out
something
else,
let's
not
like
digging
on
this,
so
we
basically
got
rid
of
that
and
we
kind
of
wanted
to
figure
out
it
played
around
with
a
couple
different
ideas
on
how
to
roll
our
own
kind
of
way
to
get
into
elasticsearch
cluster
that
we
hosted
externally
just
on
on
an
ec2
instance,
a
two
instances.
So
you
know
we
won
control.
A
We
used
fluent
D,
so
we
tried
a
couple
different
approaches
and
one
approach
that
is
fairly
popular
you
may
have
seen
before
is
a
beside
car
approach.
Where
your
container,
your
pod,
has
the
two
containers,
your
application
container
and
your
login
container,
which
is
basically
just
fluid
D,
listening
and
tailing
a
log
on
a
shared
volume.
So
your
can
your
your
one
container
writes
to
a
log.
A
It
gets
shared
into
the
into
your
fluent
D
container
and
it
gets
piped
out
to
wherever
we
chose
to
just
throw
the
logs
right
at
Amazon
Kinesis,
and
you
just
hook
in
like
pointing
you
know
kind
of
a
couple
clicks
pointed
right
at
an
elasticsearch
cluster
and
that's
three
backups
and
it
got
us.
You
know
we
may
be
able
to
save
money
when
down
the
road.
A
If
we,
if
we
choose
other
solutions
but
right
now,
it
just
got
us
off
the
ground
and
really
has
just
like
improved
our
you
know,
taking
that
off
our
plate
for
right
now.
So
that's
that's
been
great,
like
we
don't
regret
that
and
we'll
probably
stay
with
that
for
the
considerable
future
as
we
grown.
So
some
of
the
pros
were
this:
we
were
that
it
was
flexible
and
independent.
A
You
know
each
each
pod
could
handle
the
load
that
it
was.
It
was
seeing
so
it
wouldn't.
You
know,
there's
no
single
point
of
failure,
no
dependency
on
a
single
service
running
in
a
cluster.
Some
of
the
cons
were
that
you
know
the
application
had
to
handle
odd
rotation
or
something
so
still
a
lot
of
question
marks
on
that
of
what
would
handle
that
you
know.
A
We
also
had
more
more
complex
pod
configurations
that
were
just
these
big
gamble
files
with
secrets
and
volumes
and
stuff
that
were
kind
of
like
making
it
more
complexed
to
reason
about
on
not
too
bad,
but
it
was
nice
without
him
and
we're
also
running
two
times
the
containers.
So
if
we
were
scaling,
you
know
we
scaled
the
service
for
five
to
fifty
at
one
point
as
we're
scaling
it
up
and
it's
running
a
lot
more
containers,
this
just
kind
of
felt
like
we're
using
excessive
resources
to
do
stuff.
A
So
you
know
we
we
chart
off
with
another
approach,
which
is
pretty
much
inspired
by
this:
the
kind
of
fluent
D
logging
that
comes
with
everything
we
we
used
daemon
set
to
run
fluent
D
on
each
machine
and
tail
The
Container
locks
on,
and
it
was
great
used,
a
little
bit
more
resources.
But
you
know
for
that
pod.
But
overall,
with
the
reducing
the
number
of
containers,
it
was
a
lot.
It
was
a
lot
more
efficient
for
us
and
it
was
also
one
way
we
wanted
to
roll
out
an
update
to
this
thing.
A
It
was
one
place
to
roll.
Instead
of
updating,
you
know
each
one
of
our
application,
configurate
pod
configurations,
because
we
didn't
want
to
have
to
do
that
if,
after
running
hundreds
of
hundreds
of
pods
on
our
rotor
instance,
so
there's
just
some
of
the
things
were
trying
to
look
ahead
to
and-
and
you
know,
we
also
wanted-
add
more
flexibility
and
not
just
have
it
on
logged
standard
out.
So
we
just
we.
A
We
opened
up
TCP
port
and
use
the
TCP
plug-in
on
fluent
D
and
just
allowed
our
our
Deb's
to
to
pipe
their
log
straight
via
that.
If
they
couldn't
log
to
standard
out,
we
in
our
standard
libraries
that
we
have
in
the
company,
you
can
just
flip
a
boolean
and
then
you're
going
to
tcp
instead
of
standard
out,
so
we
wanted
to
make
it
dead
simple
for
people
we're
still
using
kinases
to
so
as
our
transport
kind
of
layers.
So
this
has
been.
B
A
Reliable
has
scaled
as
we've
scaled
up
services,
so
you
know
this
has
been
pretty
pretty
nice
and
you
know,
given
us
a
lot
of
control
with
some
of
these
things,
we're
also
able
to
listen
to
other
logs
that
are
at,
like
structured
of
using
fluid
d
to
say,
like
you,
send
different
logs
to
different
Kinesis
streams.
So
that's
something
that
we've
experimented
with
for
internal
user,
behavior
metrics,
instead
of
just
logging
data,
so
that's
been
really
great
to
just
customize
for
one
day,
which
is
been
awesome
so
on
yeah,
I,
highly
recommend.
You
know
this.
A
This
kind
of
approach
rolling
your
own
kind
of
daemon
set
on
you
know
that
moves
us
on
to
monitoring
alerts.
You
know
this
is
a
kind
of
a
you
know,
quick
little
slide,
but
you
know,
data
kubernetes
was
allowing
us
to
set
resource
limits,
all
our
all
our
containers
and
everything
which
is
awesome.
But
you
know
if
we
didn't
know
how
our
containers
running,
then
what
good
are
the
resource
limits.
So
we
wanted
developers
to
love,
building
new
services
and
easier
to
mock,
easier
to
monitor,
monitor
anything
and
create
alerts
around
their
own
thing.
A
They
were
the
developers
that
created
the
app
we
wanted
them
to
be
able
to
easily
add
these
things
and
control
their
kind
of
applications
so
started
with
Prometheus,
clear
starting
point.
Great
awesome,
awesome
project,
but
we
were
we
wanted
people
who
had
no
experience
with
any
of
this
stuff
to
be
able
to
add,
add
things.
So
we
said
you
know,
let's
look
for
something.
You
know
a
little
bit
easier,
a
little
bit
more
straightforward.
A
You
know,
for
the
average
kind
of
say,
even
a
front-end
developer
can
be,
you
know,
creating
alerts
and
tracking
everything.
So
we
started
then
also
throwing
application
metrics
in
our
logs
and
in
you
know,
we
started
creating
dashboards
and
Kabana,
which
was
great,
but
then
we
wanted
to
trigger
alerts.
So
we
tried
to
use
a
project
called
the
last
alert
and
that
was
again
a
little
bit
more
complex
than
we
wanted
to
so,
and
we
had
two
different
locations
for
all
this
information.
A
So
in
the
end
we
end
up
checking
out
data
dog
and
it
was
pretty
awesome
and
of
course
it
has
cost.
But
this
is
one
of
those
trade-offs
that
we
made
of
saying
this
you
know
was
so
much
easier
for
a
team
and
we
really
just
wanted
at
this
point
to
get.
You
know
just
adoption
for
all
our
all
our
engineers
make
make
things
like
a
breeze
to
to
work
on
and
collaborate.
A
You
know,
so
it's
been
awesome,
leveraging
all
their
integrations
and
dog
stats
de
library
to
pipe
all
your
custom
metrics
right
right
into
data
dog.
This
has
been
awesome,
really
developer
friendly,
exactly
what
we
needed
right
now
to
kind
of
push
this
forward,
and
our
monitoring
overall
has
been
leaps
and
bounds
just
better
than
whatever
we
add
before
we
are
creating
some
off-the-shelf
dashboards
for
people
to
use.
So
it's
been,
it's
been
fantastic
to
see
the
adoption.
You
know
we
kind
of
going
don't
going
from
there.
We
started
creating
some
some
internal
best
practices.
A
You
know
there's
a
lot
of
ways
that
you
can
use
kubernetes
it's
you
know
we
needed
to
figure
out
like
how
we're
gonna
do
this.
How
we're
gonna
do
that
kind
of
some
standardized
things
pave
the
way
for
the
developers,
reduce
the
complexity
for
them
to
build
new
services
and
reduce
the
number
of
choices
that
they
really
need
to
make
just
kind
of
pave
the
way
for
them,
but
give
them
a
little
bit
of
flexibility
on
you
know
we
standardized
various
bits.
A
You
know
also
helped
us
to
create
like
standard
tooling,
if
everything
is
happening,
the
same
way
so
that
that
has
really
helped
for
us,
with
our
deploy
bots
and
everything,
and
some
of
the
things
that
it
does
so
yeah.
So
you
know
we
also
wanted
the
standard
things,
because
you
know
teammates
sleep,
which
in
our
company
happens
a
lot
and
something
that
was
off
somebody's
know
had
a
reason
about
how
to
jump
into
something
pretty
quick.
So
standardization
is
really
great.
A
A
We
use
them
all
a
bunch
of
different
ways
and
you
know
kind
of
just
kick
the
tires
and
everything
and
we're
able
to
figure
out
some
standard
labels
that
we
want
to
use
and
paving
the
way
for
canary,
deploys
and
everything
you
know
for
our
internal
services
and
everything
so
yeah.
It's
been,
it's
been
great,
like
just
making
it
dead,
simple
to
kind
of
like
setting
up
or
different
tracks
and
and
being
able
to
do
these
things
from
like
the
day.
A
One
of
every
service
is
awesome,
so
when
some
developer
says
I
want
to
do
this,
but
I
only
want
to
I
want
to
do
a
canary,
it's
already
there.
For
me,
you
know
so
we've
kind
of
standardized
a
lot
of
these
things
and
start
standardizing
our
monitoring,
metrics
and
everything
and
we've
taken
a
little
page
out
of
kubernetes
books
and
started
started
our
own
internal
special
interest
group
for
our
monitoring.
A
So
we
have
a
couple
people
who
are
kind
of
just
kind
of
a
clone
of
kubernetes
special
interest
groups,
and
hopefully
they
will
push
forward
the
agenda
and
moving
this
continuing
on.
You
know
as
we
as
we
grow
and
have
different
types
of
complex
services
that
we're
running
so
we're
gonna.
Have
this
cross-disciplinary,
there's
gonna
be
DevOps
or
systems
type
engineers
and
product
engineers.
You
know
to
kind
of
be
able
to
guide
this
moving
forward
nod.
You
know
we're
not
finished
with
this
part
yet,
but
we're
always
gonna
be
continuously
evolving.
A
But
to
start
this
has
been
really
a
boon
for
our
developers.
So
one
of
the
things
that
we
we
wanted
to
do
is
figure
out
what
our
guinea
pig
was.
We
call
this
an
exemplar.
You
know
it's
it's.
The
work
for
kubernetes
is
only
is
good,
but
it's
only
great
if
to
actually
put
to
use
so
inspired
by
a
great
book
called
building
micro
services
by
Sam
Newman.
We
set
off
to
build
an
exemplar
if
you're
unfamiliar
this
term,
and
you
have
the
classic
definition.
A
A
person
or
thing
serves
as
an
excellent
model
worthy
of
imitation
or
to
kind
of
paraphrase
Sam
Newman
in
the
book,
a
real
world
service
that
implements
your
best
standards
and
practices
that
you'd
like
to
encourage
and
future
services
built
by
our
team.
We
had
to
create
one
of
these
things.
Basically,
it
was.
It
was
what
we
needed
to
do
so
to
do
this,
we
need
some
several
goals.
It
had
to
be
a
real
production
service.
A
You
needed
to
really
demonstrate
to
engineers
and
and
everything
that
that
it
could
handle
real
traffic,
and
this
is
how
something
was
really
gonna
work
in
the
real
world
on.
It
was
important
that
it
handled
a
solid
amount
of
production
traffic.
It
wasn't
just
something
trivial
that
got
hit.
You
know
five
times
a
day,
so
we
chose
something
that
was
fairly
crucial
to
our
service,
but
had
a
very
defined
scope.
So
you
know
this
is
serve
as
living
documentation
of
all
our
best
practices.
A
Basically,
people
will
look
to
this
service
and
say:
oh
well,
you
know
I
can
look
to
here
and
see
how
how
how
we
actually
did
XYZ,
there's
also
model
from
which
other
services
are
built
so
like
spend
the
time
to
get
it
right.
These
have
been
extra
powerful
because
there
are
also
concepts
and
applications
and
services
that
that
your
team
understands
already.
A
This
was
something
that
was
a
court
couple
that
we
built
work
or
aspects
of
our
of
our
architecture,
that
we
knew
that
everyone
knew
how
this
worked
and
could
understand
and
reason
about
the
code.
So
the
you
know
the
having
perfect
examples
or
demo
repos
are
great,
but
they're
created
in
you
know
pristine
conditions
and
they
may
not
be
relatable
relatable
to
actual
services
on
your
team,
so
I'm
sure
you've
seen
documentation
at
all.
A
A
You
know
also,
lastly,
like
this
really
allows
us
to
kick
the
tires
on
kubernetes
building,
something
that
we
knew,
how
much
traffic
it
would
have
to
serve.
We
were
able
to
experiment
with
things
on
our
team
for
the
first
time.
You
know
that
we're
learning,
kubernetes
resource
limits
and
auto-scaling
rollouts
rollbacks
Canary
deploys.
We
could
also
use
the
existing
services
throughput
that
it
was
seeing
for
existing
kind
of
legacy
monitoring
stuff,
so
we
could
throw
real
production
traffic
at
it
before
it
even
received
real
production
traffic.
A
So
we
could
simulate
that
and
actually
trust
our
kubernetes
cluster.
You
know
we
were
standing
this
thing
up
for
the
first
time
we
you
know,
we
just
really
needed
to
trust
this
thing,
so
this
really
helped
start
it
off.
With
one
kind
of
you
are
on
our
team,
we
found
that
it
was
useful
to
have
different
kind
of
exemplars
for
different
types
of
services,
whether
they
be
a
web
server
or
a
worker
consuming
off
a
message
queue.
A
It
was
really
helpful
to
to
kind
of
push
our
team
forward
and
really
figure
out
everything
the
kubernetes
so
yeah
that
I
mean
after
our
plan.
You
know
that
where
does
that
bring
us
to
today,
we
spent
you
know
the
summer
experimenting
the
last
two
three
months
really
working
with
kubernetes
and
pushing
services
there,
mostly
in
the
last
since
beginning
of
September,
we're
super
stoked
with
all
the
progress
that
we've
made.
Okay,
we
probably
have
about
five
teams
that
are
running
just
twenty
six
deployments,
so
it
still
feels
you
know
some
of
you.
A
This
is
small
numbers,
but
for
our
team
we're
super
pumped
about
this
and
we're
kind
of
have
user-facing
services,
data
pipelines,
data
analysis,
jobs,
internal
services.
It's
been
amazing,
we
needed
an
internal
service
the
other
day,
and
you
know
something
else:
something
got
deleted
incorrectly
off
a
server
that
wasn't
in
version
control,
so
we
just
built
the
new
service
deployed
it
and
we're
able
to
get
it
out
in
like
under
an
hour.
So
it
was,
it
was
writing
it
from
scratch
and
seeing
into
production
is.
It
was
just
unbelievable,
it's
so
powerful!
A
You
know
we've
rolled
out
a
new
service
yesterday,
actually-
and
now
we
it
took-
we
took
50
now
we're
over
50%
of
our
traffic
is
going
through
a
proven
Nettie's
cluster,
which
is
a
major
milestone
for
us.
We
just
hit
over
50,
like
54
percent,
so
we've
been
able
to
start
scaling
down
our
old
infrastructure
now,
which
is
exactly
what
we
wanted
to
do
start
shifting
things
away.
A
We're
getting
things
out
so
much
faster
than
we
ever
have
we're
continuously
shipping
new
versions
at
all
hours
of
the
day,
and
you
know
there's
you
know
we're
there,
our
application
is
it's
not
tightly
coupled
anymore.
These
things
are
independent
and
we
have
alerting
or
on
everything.
So
these
independent
services
are
not
bringing
down
our
entire
infrastructure.
A
You
know
it's
really
great,
not
having
to
coordinate
with
and
lean
on
DevOps
people
that
are
all
the
way
around
the
world.
That's
been
incredible
to
really
free
us
up.
So
what's
next
for
buffering
kubernetes,
you
know
we're
still
kubernetes
rookies
like
there's,
there's
there's
still
a
lot
more
for
us
to
do.
You
know
we
still
have
a
lot
of
challenges
that
I'm
sure
are
pretty
common
with
with
a
lot
of
you
out
there.
A
You
know
we
really
need
to
improve
the
development
to
you
know,
production,
the
story
about
how
do
we
get
docker
compose
or
a
regular?
You
know
docker
file
into
production
and
connect
the
right
services,
and
you
know
so
that's
definitely
something
that
we
really
want
to
improve
that
flow
for
for
our
developers.
We're
still
like
it's
it's
it's
at
a
weird
point:
we're
pushing
there
we're
trying
to
make
it
trying
to
make
it
better,
hopefully
build
some
new
tooling
around
everything.
A
So
you
know
we're
not
sure
if
that
really
means
better
documentation,
tooling
training,
but
we're
also
seeing
how
our
actual
developers
or
product
engineers
are
doing
these
things
and
what
creative
solutions
they're
coming
up
with
and
feeding
that
back
in
so
that's
been
pretty
fun.
We're
also,
like
you
know,
really
exploring
things
like
things
like
cops
for
cluster
upgrades
and
kind
of
rolling
out
new
clusters.
Spinning
things
up
we're
like
I
said:
we
really
need
to
get
better
at
that.
A
A
We're
also
kind
of
we
want
to
get
better
with
our
resource
configuration
and
deploys
kind
of
is
it's
kind
of
awkward,
also
how
we
have
everything
set
up
like
it
works,
but
it's
not
great,
so
we're
really
excited
about
things
like
helm
and
and
we
think
that
it
could
really
be
much
easier
to
reason
about
and
how
we
managed
our
configurations
and
deployments
and
everything.
So
we're
really
excited
about
stuff
like
that,
and
you
know
lastly,
I
think
even
previous
speaker
talked
about
security
in
between
pods
and
and
there
there
is
not.
A
You
know
we
don't
have
any
and
we're
encrypting
things
as
we
send
and
through,
but
we
are
not
running
in
on.
You
know,
TLS
so
having
things
like
Lync
or
D,
or
whatever
are
really
things
that
we're
at
that
stage
of
like
experimenting
with
and
and
kind
of
figuring
out.
What's
the
best
solution
for
us,
so
that's
a
little
bit
of
an
unknown
territory
for
us,
but
we're
really
excited
about
you
know
we're
we're
also.
A
Just
has
feeling
like
Swasey
cleaned
everything
up,
and
you
know
how
to
give
away
the
end
movie
with
Swayze
came
up
and
he
cleaned
everything
you
see
out
like
they're
in
top
shape
now,
and
you
know,
kubernetes
really
showed
us
how
to
clean
up
our
infrastructure
and
gave
us
really
great
things
that
have
improved
dramatically
improved
the
way
we
build
services.
You
know
I
hope
I
gave
you
a
little
bit
of
insight
into
how
our
small
team
of
you
know.
A
A
A
A
Right
so
that
that
will
use
the
bot
to
use
coop
CTL,
you
know
rollout
undo
so
yeah.
We
don't
have
any
automated
like
continuous
delivery
like
roll
backs
or
anything,
that's
something
that
will
hopefully
experiment
with
a
little
bit
based
on
you
know.
Whatever
metric,
we
need
quality
metrics.
So
yeah,
that's
that's
for
around
that.
Anyone
in
the
back
for
load
balancing
we
are
using
elastic
load
balancer
because
we're
running
on
AWS,
so
we're
building
our
certainly.
A
A
Yeah,
so
that's
tough,
because
I
think
there
were
also
even
some
people
that
were
not
in
engineering
say
like
in
a
high
level.
They
were
like
what
is
this
project
that
you're
working
on
and
why
is
it
valuable
because
they
didn't,
they
were
saying
that
we're
gonna
lose.
You
know,
say
time
on
redoing
her
infrastructure
when
we
could
be
shipping
more
products
or
something
like
that.
So
there's
a
there's,
definitely
a
bit
of
trust
at
the
beginning,
half
and
the
training
side
of
it.
That's
mostly
been
on
a
one-on-one
basis.
A
We've
we've
we've
hand-picked
kind
of
some
engineers
that
you
know
have
some
project
projects
and
maybe
the
know-how
to
say
like
okay,
you
can
you
want
to,
and
you
have
the
ability
to
move
this
out.
Some
people
who
are
like
really
will
get
in
the
weeds
with
these
things
and
move
their
services
into
the
clusters.
So
it's
kind
of
in
piecemeal
right
now,
pretty
it's
pretty
slow,
we're
gonna
start
doing
some
more.
This
quarter
we're
going
to
start
doing
some
more
actual,
like
recorded
video
training
because
we're
all
remote.
A
So
it's
like
kind
of
difficult
to
do
that.
So
there's
a
lot
of
ad-hoc
training,
one
on
one
on
video,
about
how
to
get
service
out
and
everything
so
a
lot
of
documentation
and
we're
hoping
that
as
we
slowly
spread
this
out
and
get
just
a
few
engineers
on
board,
we'll
see
like
a
domino
effect.
Those
engineers
will
train
others
and
we'll
continue
to
build
this,
but
like
the
kind
of
documentation
and
everything.
A
So
luckily,
our
team
is
not
gigantic,
so
it's
it's
a
little
bit
easier
to
do
for
us,
but
definitely
sharing
that
knowledge.
Asynchronously.
Is
it's
a
bit
of
a
challenge,
so
yeah
I,
don't
know
if
I
exactly
like
answered
answered
that
but
yeah
it's
it's
not
it's
not
the
easiest,
but
yeah
more
training,
and
you
know,
and
really
showing
people
the
ropes
myself
and
then
a
lot
of
the
abstractions
has
kind
of
reduced
some
of
the
complexities.
A
So
someone
might
not
have
to
think
about
why
labels
are
important
straightaway
if
they
want
to
dig
in
they
can,
but
from
the
beginning,
like
abstracting,
some
of
these
things
away
and
just
giving
them
things
kind
of
on
like
a
platter
like
giving
them
a
how-to,
has
kind
of
helped
at
least
make
that
week
and
then
each
they
can
dig
in
a
layer
deeper
and
kind
of
all
the
aspects
if
that
makes
sense,
yeah.
Okay,
we
have
had
some
incidents
in
downtime.
Oh.
A
I,
don't
yeah,
yeah
I
think
we're
you
know
like
if,
since
we're
so
early
on
in
in
in
that
bit
of
running
stateful
stuff,
you
know,
especially
with
the
the
elasticsearch
example
we
have
decided.
You
know
not
yet
not
yet
until
you
know
like,
we
also
need
to
understand,
say
pets,
that's
a
little
bit
more,
so
we
want
to
do
that.
We
want
to
do
that,
but
we
want
to
experiment
with
some
non
crucial
things
that
are
not
a
totally
essential
like
say,
logging
or
certain
things
that
are
just
internal
services.