►
From YouTube: From Monolith to Microservices with Kubernetes and Linkerd - Mason Jones, Credit Karma
Description
From Monolith to Microservices with Kubernetes and Linkerd - Mason Jones, Credit Karma
After about eight years, Credit Karma had built up an impressive tech infrastructure...based on a PHP monolith. Over the past 18 months we’ve (carefully) adopted Docker, Linkerd, Consul, Kubernetes, and more as we shifted to microservices in order to enable continued engineering innovation. This is the story of our evolution from monolith to microservices, starting with our own homegrown tools. The talk will cover our iterations from basic plumbing to dynamic service discovery; why we started using Linkerd and selected Kubernetes; and how we evolved our systems step by step while continuing to serve 75 million members.
About Mason Jones
Mason leads Credit Karma’s infrastructure services team. His team's work provides a foundation for Credit Karma's microservices to enable our 75 million members and financial institution partners in the U.S. and Canada to use data to guide their financial decisions. Before joining Credit Karma, Mason spent more than 20 years as a technical executive at startups throughout Silicon Valley. He specializes in migrating technical organizations to microservices, securing build pipelines and creating self-service tools to empower development teams. When Mason isn’t shipping code, he manages Charnel Music, an independent record label he founded.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Good
afternoon,
everybody
thanks
for
being
here
late
on
a
Friday
afternoon
after
everybody's
brains
are
completely
full,
but
we
survived
the
snow.
So
we
can
survive
anything
so
I'm
here
to
tell
you
a
story.
Take
you
through
the
journey
that
credit
camera
has
been
on
over
the
last
couple
of
years
from
monolith
to
micro-services,
like
it
says,
I'm
Mason,
Jones
staff,
engineer
heading
up
our
infrastructure,
Services
team,
we're
the
team
that
gets
to
play
with
all
the
fun
stuff
that
we've
really
been
all
hearing
about
this
week:
docker
communities,
linker,
D
and
so
forth.
A
That
I'll
be
talking
a
little
more
about
a
little
bit
about
Credit
Karma.
Our
mission
is
really
to
make
financial
progress
possible
for
everybody.
We
serve
our
over
75
million
members
by
helping
them
understand
their
credit
score,
their
credit
history
ways
to
improve
things,
ways
to
find
better
financial
opportunities,
we're
just
over
800
employees,
now
half
in
engineering.
But
when
our
story
starts
two
years
ago
we
were
half
the
size,
400
employees,
half
in
engineering
and
we'd
already
been
growing
very
very
rapidly
over
the
last
little
while
and
we
were
facing
a
model
with
I'll.
A
Take
you,
through
these
steps,
during
the
talk
and
kind
of
tell
you
about
the
various
phases
that
we
went
through,
but
I'm
actually
curious,
first
to
kind
of
start
off
how
many
of
you
have
a
monolith
that
you're
either
preparing
to
get
out
of,
or
are
in
the
process
of
trying
to
get
out
of
yep.
It's
a
really
really
common
story.
These
days.
How
many
of
you
have
already
done
that
and
you're
just
here
to
gloat
cool?
A
Our
reason
for
moving
to
microservices
from
the
monolith
there
are
lots
of
different
possible
reasons
that
you
all
may
have
for
us.
It
was
really
to
try
to
get
teams
independent.
We
had
learned,
as
probably
a
lot
of
you,
have
learned
that
engineering
growth
paired
with
a
monolith
leads
to
great
sadness,
and
we
wanted
to
get
teams
to
where
they
could
really
be
independent.
A
Do
their
thing
move
at
their
own
speed,
not
deal
with
the
monolith
deployment
cycle
and
the
dangers
that
come
from
making
changes
to
a
monolith,
and
we
wanted
teams
to
be
able
to
experiment
as
well.
It's
it's
much
much
harder
to
just
try
something
out
in
a
monolith,
because
the
side
effects
are
unknown
right.
So
that
was
really
what
we
were
after
here
and
when
we
made
the
decision.
Micro-Services
is
gonna,
be
our
future.
We
tried
to
figure
out.
How
are
we
gonna
get
there?
A
A
First
feeler
I
had
to
figure
out
what
was
going
to
bite
us
right
so,
rather
than
doing
the
tempting
thing,
which
is
to
identify
the
piece
of
our
model,
if
that's
causing
us
the
most
pain
that
we
really
want
to
get
out
of
there,
that
we
really
want
to
to
free
from
you
know
from
the
the
monolith
itself
and
we
chose
a
couple
of
really
relatively
unimportant
services.
Don't
tell
the
teams
that
own
them
and
I
said
that,
but
we
really
wanted
to
play
it
safe.
A
We
knew
that
we
were
gonna
have
a
lot
to
learn.
We
selected
those,
and
we
then
wanted
to
use
the
tools
that
we
already
had
in
hand
to
get
things
started.
We
run
to
make
this
more
fun
on
bare
metal
in
our
data
centers.
We
use
salt
to
provision
our
servers
and
we
run
CentOS,
so
supervisor
kind
of
comes
with
the
package.
So
we
thought
all
right.
Let's
see
what
we
can
do
with
those
we're,
not
gonna
try
to
bite
off
more
than
we
can
chew.
A
We're
not
gonna,
grab
an
orchestration
system
and
spend
six
months
trying
to
figure
it
out
before
we
make
any
progress.
We
really
want
to
just
start.
We
want
to
get
this
going,
so
this
seems
kind
of
weird
I'm
sure,
but
we
said,
let's
set
up
deployment
of
a
service
by
using
salt
to
provision
files
onto
the
servers
where
we
want
to
run
it.
Those
files
are
supervisor
configs
telling
it
do
a
docker
run.
A
We
provision
those
files,
we
kick
supervisors
and
it
does
a
docker
run
for
us.
If
that
container
dies,
supervisors
are
gonna,
restart
it
that's
its
job.
It
does
that
pretty
well
now,
provisioning
files
with
salt
and
telling
supervisor
to
do
a
docker
run
sounds
really
horrible
and
it
kind
of
is,
but
it
got
the
job
done.
We
were
able
to
get
some
containers
running,
get
docker
installed,
get
some
experience
with
that.
Try
to
upgrade
docker
fail,
horribly,
get
some
experience
with
that.
A
We
just
we
learned
the
hard
way
that
upgrading
docker
is
not
always
as
straightforward
as
you
might
expect,
but
this
did
work
for
us.
So
at
the
beginning
of
this
year,
then,
we've
actually
gotten
to
about
15
services
or
so
running
across
20
servers
in
our
data
center.
Using
this
mechanism,
we
built
some
deployment
tools
that
would
interact
with
salt
use,
some
weird
ansible
stuff
as
well
get
some
files
out
there
kick
supervisor
and
allow
teams
to
deploy
to
update
their
images
to
restart
the
containers
it's
working.
A
So
now,
we've
got
some
pretty
good
experience
running
the
stuff
in
production.
We've
figured
out
some
logging
and
some
monitoring
and
and
some
of
those
important
things
and
the
teams
that
are
building
services
are
actually
pretty
happy
with
this.
They
don't
see
the
kind
of
ugliness
underneath
and
too
much
we
do
and
we
feel
it
and
we
see
a
lot
more
services
coming
down
the
line,
and
we
knew
that
this
is
not
a
scalable
thing
for
our
team
to
continue
managing.
A
It's
really
obviously
super
inflexible
architecture,
we're
hard-coding,
where
these
service
instances
running
what
servers
have,
which
ones
we're
human
orchestrators,
honest
to
god,
using
a
google
spreadsheet
to
track
here.
Our
servers
here,
our
services
and
here
what
ports
they're
on
it
works,
but
it's
not
great
right.
So
how
are
we
gonna
get
past
this?
A
The
other
thing
that
was
actually
biting
us
that
was
causing
more
heartache
to
more
people
was
that
teams
would
develop
their
service
on
their
laptop
they'd,
get
it
running,
they'd
go
to
testing,
they
do
their
integration
testing,
it
would
be
working,
they
would
deploy
a
production
and
it
wouldn't
work
the
plumbing
the
moving
parts.
The
configurations
were
really
very
different
in
those
environments,
and
that
was
not
making
anybody
happy.
So
we
wanted
a
way
to
give
the
same
experience
across
all
of
our
environments,
but
again
baby
steps.
A
What's
the
next
thing
that
we
needed
to
do,
we
still
got
a
monolith.
Obviously,
but
we've
got
some
services
going
now.
So
that's
pretty
good.
The
next
thing
to
start
to
make
things
a
little
bit
more
flexible,
a
little
bit
more
automatable
a
little
bit
more
looking
into
the
future
was
service
discovery,
so
we're
running
console
already
in
our
infrastructure
for
other
reasons,
and
it
makes
a
terrific
service
discovery
tour
right.
A
So
we
modified
our
deployment
tool
such
that
after
getting
the
service
onto
the
server
and
getting
a
config
set
up,
it
would
tell
console
about
it.
So
now
we've
got
something
we
can
query.
Tell
us
what's
running
where
that's
quite
an
improvement
and
we've
got
some
decent
health
checks,
because,
prior
to
this,
the
supervisor
would
alert
us
if
something
died
and
it
restarted
it.
A
But
if
the
service
stops
responding,
but
the
process
is
still
there
pretty
invisible
to
us,
we
would
often
not
really
know
about
it
until
someone
reported
that
traffic
was
just
not
flowing
properly.
So
this
is
an
improvement,
but
it's
incremental
right
we're
not
any
closer
to
a
dynamic
infrastructure.
A
So
still
got
our
monolith
got
some
services
running
we're
registering
them
now.
We
know
where
they're
running.
We
know
if
they're
healthy,
that's
pretty
good.
So
what
can
we
do?
Next?
We
know
that
we're
moving
towards
orchestration
at
some
point.
You
know
we're
moving
towards
it
because
there's
a
box
that
says
so.
We
wanted
to
leverage
the
service
discovery
now
to
actually
send
requests
intelligently,
based
on
it,
so
dynamic
routing
is
next.
This
is
what
our
routing
looked
like
at
this
point.
A
If
the
monolith
wanted
to
make
us
call
to
a
service
which
is
our
most
common
case
early
on,
it
would
make
the
request
to
an
engine
X
proxy
running
on
its
server,
that
nginx
proxy
would
forward
it
to
a
VIP
load.
Balancer
hardware
load
balancer
in
our
data
center,
which
has
been
manually
configured
to
know
where
instances
of
the
service
are
not
great.
A
The
VIP
would
then
send
it
to
one
of
the
servers
to
an
engine
X
proxy.
On
that
end,
the
job
of
that
nginx
proxy
makes
a
little
more
sense,
which
was
to
allow
Bluegreen
deployments
our
our
team's
wanted
to
be
able
to
have
traffic
going
to
their
blue
instance,
deployed
to
green
and
then
ramp,
1%
of
traffic
to
it
and
get
a
sense
of
whether
things
were
working
because
keep
in
mind
we're
trying
to
roll
this
out.
We've
got
these
services
going.
A
While
we've
got
75
million
members
using
our
site
and
we're
starting
to
move
some,
you
know
actually
important
functionality
into
these
services,
so
one
percent
of
traffic
is
enough
to
watch
the
logs,
get
a
sense
of
whether
things
are
healthy
or
not.
We
did
this
by
having
our
job
in
our
deployment
tool
update
the
nginx
config
send
a
signal
to
nginx
to
reload
the
config,
and
then
we
could
actually
adjust
the
weighting
on-the-fly
and
safely
so
you're.
Probably
though,
looking
at
this
and
wondering
why,
why
is
there
an
engine
hex
on
the
other
side?
A
And
the
reason
for
that
is
those
green
arrows
represent
traffic
going
between
hosts
in
our
data
center?
The
in
Credit
Karma
security
is
the
first
thing
we
think
about
at
all
times,
and
one
of
our
requirements
is
that
all
traffic
between
hosts
must
be
encrypted
must
be
TLS
in
turn.
X
is
there
to
ensure
that
security
and
for
performance
reasons,
I,
don't
think
I
mentioned
earlier
that
to
make
things
more
fun,
our
monolith
is
not
just
any
monolith.
It's
now
a
seven
year
old
PHP
model
with
PHP
does
not
do
a
nice
job
of
hole.
A
Connections
open
with
the
result
that
every
service
request
would
have
to
open
a
TLS
connection.
Send
the
request,
wait
for
the
response
to
tear
it
down
and
do
that
over
and
over
again
the
latency
was
not
acceptable.
It
was
not
working
for
us,
so
we
put
an
engine
X
proxy
on
that
side
to
maintain
the
TLS
connection
to
the
load
balancers
and
that
actually
got
the
job
done,
but
it
was
another
piece
of
essentially
hard-coded
infrastructure
that
we
were
putting
on
our
servers.
A
So
how
do
we
get
past
those?
How
can
we
make
use
of
console
now
to
actually
do
better
and
more
flexible
and
actually
automatable?
Eventually,
routing
and
linker
D
ended
the
picture
at
that
point.
This
is
probably
the
only
audience
that
I
don't
have
to
explain
the
service
master
to
because
everybody's
been
hearing
about
it
for
the
last
several
days,
I
think,
but
in
our
case
linker
D
represented
a
service
national
deployed
unto
all
of
our
servers
talking
to
consul.
So
it
knows
where
things
are
this
removed
the
need
for
the
manual
zip
load.
A
Balancer
configuration
remove
the
need
for
that
engine
X,
because
linker
D
actually
does
TLS
connections
between
all
the
hosts
for
us
and
it
got
us
ready
for
actual
orchestration,
because
now,
when
we
deploy
a
container
somewhere,
it
will
register
with
service
discovery
and
something
will
make
use
of
that
to
actually
make
sure
requests
reach
it.
So,
at
this
point,
routing
looks
like
this:
instead,
which
is
a
whole
lot
cleaner
and
a
whole
lot
smarter.
The
monolith
sends
a
request
to
linker
D
on
its
hosts.
A
I
think
he
looks
at
that
request
and
says
you
want
service.
Foo
great
I
know
where
those
are
I'll
pick.
One
I
will
send
your
request
to
a
linker
D
on
its
host
and
that
will
then
at
its
end,
send
to
either
blue
or
green,
and
we
modified
our
deployment
job
to
do
the
waiting
between
blue
and
green
to
make
use
of
linker,
DS
name
or
D
component.
We
can
use
the
API
rather
than
actually
touching
configs
and
telling
it
to
reload,
which
is
also
much
much
nicer,
so
we're
making
some
progress
here.
A
We've
still
got
our
monolith,
but
we've
got
a
bunch
of
services.
We've
got
service
discovery
in
place,
we've
got
dynamic,
routing
making
use
of
that.
So
we
can
start
to
look
at
actual
orchestration.
Now
we
had
actually
started
to
explore
orchestration
late
last
year,
late
2016,
just
to
kind
of
get
an
idea
where
we
were
going
to
be
headed
in
the
future.
In
case
it
might
influence
some
of
our
decisions
along
the
way.
At
that
time
the
landscape
was
fairly
different
than
it
is
now
we
were
looking
at
mezzos.
A
A
We
looked
at
nomads
from
Hershey
Corp,
which
was
really
actually
appealing
very
simple,
very
small,
easy
to
manage
small
sort
of
functionality,
but
probably
enough
for
us
at
the
time,
but
it
was
very
new
and
we
couldn't
really
find
good
stories
of
people
running
it
at
scale,
so
we
weren't
really
confident
enough.
At
the
time
we
looked
at
docker
swarm
oddly
enough
kind
of
the
same
story,
not
a
lot
of
production
usage
at
scale,
and
we
were
also
a
little
leery
of
putting
all
of
our
eggs
in
one
basket.
A
You
know
rocket
containers,
look
pretty
interesting
too.
We
weren't
sure
where
things
might
be
going
so
kubernetes
at
the
time
was
also
kind
of
an
open
question.
It
was
moving
super
fast,
but
where
was
it
going?
It
was
kind
of
hard
to
keep
up
with
we
weren't
quite
sure
fast
forward,
just
six
months.
Only
six
months
early,
this
spring
mid-spring,
the
landscape,
had
changed
and
thankfully,
for
us
actually
the
decision
was
super
obvious
and
since
we're
all
here,
we
know
what
the
decision
was
right,
which
was
kubernetes.
A
A
So
we
felt
pretty
confident
that
it
was
gonna,
be
able
to
move
forward
and
grow
with
us
the
way
we
expected,
but
having
decided
on
kubernetes,
we
had
to
figure
out
how
to
install
this
thing,
and
this
was
interesting
because
you
run
mini
Cube
you're
like
hey.
This
is
awesome.
Gonna
go
to
production.
How
do
I
do
that
and
to
make
it
more
fun?
A
We
run
on
bare
metal
in
our
data
centers,
but
we
also
do
run
workloads
in
multiple
clouds,
so
we
needed
something
that
would
run
pretty
much
everywhere,
so
that
ruled
out
cops,
no
bare
metal
support.
There
cube
ATM
is
coming
along
now,
but
at
the
time
it
really
wasn't
ready,
wouldn't
give
us
a
full
H,
Aiken
config,
yet
CD
cluster.
All
the
things
that
we
needed
to
really
be
production,
ready
tectonic
was
pretty
cool,
but
we
can't
run
core
OSN
or
data
center
for
various
esoteric
reasons.
We
looked
through
dozens.
A
We
created
a
spreadsheet
of
installers
and
trying
to
figure
out
the
feature
set
and
what
would
do
what
it
was
kind
of
crazy,
but
we
ended
up
choosing
cos
Matic,
which
is
was
at
the
time
just
adopted
by
a
prenda,
a
smaller
active
community,
but
some
support
good
documentation
and,
most
importantly,
it
did
what
we
needed
it
to
do.
We
would
run
on
bare
metal,
it
would
run
in
the
cloud
it
was
pretty
flexible
for
configuration.
A
So
we've
got
a
cluster
in
production
now
and
we're
ready
to
start
using
it.
And
how
do
we
do
that?
Because
we're
not
going
to
tell
the
teams
yeah?
Can
you
stop
developing
for
a
couple
of
weeks
while
we
pick
up
your
services
and
move
them
over
into
communities
and
change
all
the
tooling?
So
it's
talking
proving
any
and
all
this
that
was
not
going
to
work.
So
we
were
really
back
to
our
initial
baby
steps.
A
Pick
a
service,
that's
not
in
the
critical
path
work
with
the
team
to
move
it
over
talk
to
them
and
say:
can
you
work
with
us
to
move
your
service
over
and
that
was
actually
really
really
the
easy
part?
We
talked
to
a
bunch
of
teams
to
say:
hey?
Could
we
work
with
you
to
make
me
move
your
service
as
the
first
one
into
kubernetes,
and
they
were
all
like
pick
me
pick
me,
because
I
could
see
that
you
know
it
was
gonna,
make
life
so
much
better,
so
that
was
really
nice.
A
The
main
question
that
we
had
to
solve,
though,
was
as
removing
some
of
our
services
into
communities
and
we're
continuing
to
run
our
existing
cluster.
How
do
we
manage
the
routing?
We've
got
services
calling
each
other
in
and
out,
and
your
service
mesh
is
great
if
it's
gonna
run
everywhere,
but
is
it
going
to
run
everywhere
and
we
lucked
out
in
a
big
way
here,
I'd
love
to
say
that
we
planned
this
ahead
and
we
knew
it
was
coming,
but
no,
we
didn't.
A
Linker,
DS,
name
or
D
component
is
what
talks
to
service
discovery
and
linker
D
has
the
capability
to
talk
to
multiple
neighbor
DS.
So
what
we
ended
up
with
was
a
cluster
that
looked
like
this,
where
our
docker
cluster
has
an
Emer
D.
Talking
to
console
our
kubernetes
cluster
has
a
name
or
D
talking
to
kubernetes.
The
linker
DS
everywhere
talked
to
both
named
or
DS,
so
they
know
about
the
Union
set
of
all
of
the
services
they
know
where
they
all
are.
A
So
if
a
service
in
our
docker
cluster
wants
to
make
a
call
to
one
in
the
communities
cluster,
it
sends
the
request
to
its
local
linker
D.
That
linker
D
looks
at
and
says
oh
yeah
you're.
Looking
for
that
service,
I
know
that
they're
a
bunch
of
those
over
there.
They
happened
to
be
in
kubernetes
I'll
pick.
One
I
will
send
the
request
to
the
linker
D
on
its
worker,
which
is
interesting
because
the
linker
D
needs
to
be.
Then
a
demon
said
node
poor.
A
It
needs
to
be
kind
of
known
and
addressable
from
outside,
because
it's
not
just
going
straight
to
the
pod.
Obviously
so
the
request
gets
to
that
link
or
D
that
linker
D
does
its
thing
sends
to
the
blue
or
the
green
instance
of
the
service
just
like
before
go
in
the
other
direction,
essentially
the
same
thing
right:
the
community
service
a
request
to
its
linker
d-link,
or
do
you
look
sad
says
you're
looking
for
that
service
over
there?
That's
that's
in
that
crusty
old
docker
cluster,
but
I'll
get
your
request.
A
There
picks
one
sends
requests
a
linker
D
on
that
server
and
it
gets
to
the
service
and
this
works
at
this
point
now
we're
done
right,
not
really,
but
we're
really
making
progress.
We've
got
orchestration,
we're
in
the
process
of
migrating
and
we
can
see
that
we're
moving
into
the
21st
century.
Finally,
as
far
as
running
services,
there's
one
other
element
of
this
I
wanted
to
mention
just
because
it's
kind
of
an
interesting
and
important
one,
which
is
configuration
of
services
themselves
in
our
old
docker
cluster.
What
we
did
was
leverage
our
old
friend
salt.
A
Again,
the
team
has
a
config
file
for
their
service.
It
sync
it
when
they
deploy.
Salt
pulls
that
config
file
out
of
get
provisions
it
under
the
server.
That's
part
of
the
supervisor,
docker
run
command.
It
mounts
that
config
file
into
the
container,
and
it's
got
its
config,
that's
pretty
horrible.
So
we
took
advantage
of
the
move
to
kubernetes
to
actually
move
things
into
a
better
state
where
configuration
data
for
services
is
now
in
console.
A
We've
already
got
console
out
there
and
we
know
how
to
run
it
and
the
secrets
database,
passwords,
API
tokens
and
so
forth,
or
in
hashing
core
fault
now.
So,
when
services
start
up,
they
pull
a
config,
some
console
to
pull
the
secrets
from
hash,
you've
got
vault
and
there
they
go
getting
services
that
are
started
in
communities.
The
token
that
they
need
to
talk
to
vault
is
an
interesting
thing.
That's
getting
better
with
the
latest
versions
of
both
vault
and
kubernetes,
but
it
does
work.
A
So
in
the
process
of
doing
all
of
this,
we
obviously
learned
a
ton
and
kind
of
some
of
the
high
points
were
number
one.
We
lucked
out
in
our
decision
to
start
small
and
simple
for
sure.
If
we
had
gone
into
this,
taking
the
most
important
piece
of
our
monolith
and
trying
to
chip
it
off
and
setting
up
an
orchestration
framework
and
trying
to
get
that
working,
it
would
have
been
six
months
before
we
made
any
progress
before
we
learned
anything
and
we
actually
would
have
made
the
wrong
decisions
along
the
way
without
question.
A
One
of
the
things
that
I
mentioned
earlier
is
that
we
were
trying
to
get
to
a
state
where
the
environments
were
really
the
same.
The
experience
was
the
same
running
your
service,
regardless
of
what
environment,
you're
in
and
naturally,
kubernetes
being
configuration
based
being
declarative,
made
that
much
much
easier
and
we
actually
had
a
slide
but
had
to
take
it
out
for
time.
We
lever
to
helm
for
this,
which
is
terrific,
helm.
A
You
know
the
Installer
to
rule
them
all
and
just
help
with
making
the
onboarding
of
people
starting
out
with
communities
much
much
easier.
I
think
that
would
be
a
great
thing
and
lastly,
but
but
not.
Lastly,
is
security?
Security
touched
all
of
these
things
as
we
were
going
along
and
as
you're
starting
to
move
into
distributed
services
and
you're,
starting
to
look
at
things
like
service
meshes
and
all
of
these
things,
security
gets
very
interesting
with
linker
D
and
the
way
that
we
set
up
the
service
mesh
with
one
instance.
A
We
had
to
pull
all
of
those
down
put
them
in
our
private
registry.
Scan
them
get
them
approved.
Make
sure
that
they're
cool
with
security
and
yeah
move
on
from
there,
which
makes
upgrades
tons
of
fun
so
we're
still
moving
through
this
we've
still
got
our
monolith.
We've
accepted
that
we're
gonna
remodel
it
for
a
while,
but
it's
getting
smaller
rather
than
bigger,
we've
got
more
and
more
services
going
by
the
end
of
this
year.
A
We
will
sorry
by
the
end
of
2018,
we'll
probably
have
a
couple
of
hundred
services
the
way
it's
going,
but
we've
slowly
moved
our
way
through
tooling
understanding
along
the
way,
what
we're
trying
to
get
out
of
it
and
we
feel
pretty
good
about
where
we're
going
and
tools
like
kubernetes
than
greedy
and
so
forth,
have
made
it
much
much
easier.
So,
hopefully,
some
of
what
we've
gone
through
some
of
our
thoughts
and
decisions
may
be
useful
for
folks
who
are
tackling
this
now
and
thank
you
very
much.
A
Yeah,
the
question
is
about
how
we're
using
helm-
and
we
have
standardized
on
all
of
this,
so
that
every
service
runs
the
same
way
and
the
developers
don't
have
to
think
about
deployments
services
or
any
of
those
objects.
Really
all
they
know
is
that
they've
got
a
service.
It's
called
this.
They
need
X
number
of
instances
of
it
running
and
for
the
most
part,
that's
all
they
have
to
change.
We
allow
overrides,
for
we
run
services
in
both
Scala
and
node.
At
this
point,
so
we
do.
A
Allow
overrides,
for
things
like
max
keeps
eyes
for
Scala
services,
just
because
the
JVM
is
annoying,
so
they
can
put
in
some
of
those.
But
then
it's
a
very
small
file
that
they
have
to
think
about,
and
then
we
layer
on
everything
else
at
deployment
time,
based
on
where
they're
deploying
to.
If
it's
going
to
to
testing,
then
you
know
one
instance
of
it's
going
to
production
another,
and
we
also
have
multiple
data
centers
that
we
deploy
to
in
parallel.
A
So
the
question
is
about
performance
degradation
using
linker
D.
We
didn't
actually
now.
That
said,
when
we
first
adopted
it,
we
were
pretty
primitive
in
our
metrics,
so
it
was
a
little
bit
of
best
guess
and
like
yeah.
This
feels
pretty
good,
but
we've
done
you
know
measurements
since
then,
and
it's
it's
really
pretty
minimal,
yeah
very
acceptable
for
us.
A
Yeah
the
so
the
question
is
whether
we
started
well
whether
these
services
were
pieces
of
the
monolith
or
new
functionality.
We
purposely
started
with
new
functionality,
actually
because
we
knew
if
we
were
trying
to
take
existing
code
and
like
put
it
in
this
new
thing,
it
was
gonna,
be
more
difficult
and,
ironically
also
we
had
as
a
company
started,
to
adopt
Scala
a
bit
and
we
were
building
frameworks
for
these
services
and
we
could
have
asked
all
of
the
developers.
You
know
we're
thinking
about
just
doing
the
framework
for
Scala
and
abandoning
a
PHP.
A
How
do
you
feel
about
that?
And
maybe
not
surprisingly,
everybody
said
yeah,
we
don't
want
to
do
PHP
anymore
scholars
great,
so
it
was
all
new
stuff,
all
all
being
written
in
Scala
and
as
we've
been
carving
pieces
out
of
the
monolith
and
replacing
them,
we've
actually
been
wholesale,
rewriting
them
into
Scala.
So
we
really
have
no
PHP
services
or
anything
in
production.
Yeah.
A
Yeah,
so
the
the
culture
question
yeah
I
was
surprised
actually
that
it
was
fairly
easy.
We
we
have
absolutely
gone
for
the
you
build
it,
you
own
it
mentality
and
there's
been
very
little
resistance
to
that,
because
the
the
trade-offs
of
not
having
to
deal
with
the
monolith
are
well
worth
it
in
everybody's
minds.
A
We've
obviously
had
to
train
people
on
understanding,
pager
duty
and
responding
and
all
of
this,
but
the
the
harder
thing
that
enables
the
culture
has
been
providing
the
tools
that
they
need
to
be
able
to
manage
their
services
properly
again,
going
back
to
security.
Almost
nobody,
our
company
has
access
to
the
production
servers.
There
is
no
association
to
production
ever
for
developers,
so
we
really
had
to
approach
this
from
the
standpoint
of
making
sure
they
have
their
logs.
A
They
have
their
metrics,
they
have
dashboards,
they
have
you
know
the
alerting
capabilities
and
they
have
the
ability
to
really
go
in
and
understand,
what's
happening
when
they're
certain.
That
starts
to
act
up
and
I'd
love
to
say
that
we've
got
there
really
well
solved,
but
we're
still
evolving
it
and
probably
will
be
for
a
while,
because
that
that
tooling
can
never
be
too
good.
A
We
have
a
custom-built
tool
for
deployment
actually
and
again.
That
goes
back
to
the
security
reason.
We've
we've
looked
at
things
like
well
all
the
various
tools
that
have
been
in
the
sponsor
area
for
doing
kubernetes
deployments
and
really
none
of
them
work
for
us
because
of
the
security
access
problems
or
challenges.
I
should
say
so.
A
A
That's
for
sure,
because
we
can
actually
just
we
wrote
a
little
go
utility
that
we
just
called
deploy
tool
because
we're
imaginative
and
it
it's
responsible
for
just
taking
a
command
line,
and
then
it
calls
the
communities
API
to
do
what
we
need
and
that
also
kind
of
limits
access.
You
can
only
do
what
that
tool
will
let
you
do
and
so
from
a
security
standpoint,
I
lock
things
down
a
little
bit
more
as
well.
A
Yeah,
the
development
environment
that
is
still
in
in
flux
right
now,
honestly,
because
we've
gone
from
a
day
where
everybody
could
just
run
a
copy
of
the
system
on
their
laptop
in
a
VM.
Actually,
it's
how
we
did
it
and
now,
with
all
of
these
services,
it's
too
big
to
run
on
a
laptop,
so
we've
had
like
everybody
else,
I've
talked
to
had
to
figure
out.
How
do
you
do
development
with
a
minimal
set
of
the
system
available
and
we're
pretty
much
kind
of
saying?
A
Do
that
as
you
like,
and
you
know,
providing
the
tools
to
make
it
fairly
easy
and
then
figuring
out
which
services
we
can
provide
in
a
shared
way
up
in
the
cloud,
so
everybody
kind
of
uses
one
instance
of
that,
one
that
only
works
for
some
services
that
essentially
kind
of
work
in
a
multi-tenant
way,
because
otherwise,
as
you
know
you
you
know,
someone
makes
a
call
changes.
The
data
then
I
make
a
call.
A
It
doesn't
work
anymore,
so
we're
still
kind
of
evolving
that
a
bit
but
from
from
there
we're
figuring
developers
are
doing
their
unit.
Testing
they're
running
in
their
IDE
they've
got
their
stuff
kind
of
working,
and
then
we
made
it
very
easy
to
essentially
push
a
button
and
spin
up
a
tiny
cluster.
Basically
one
node
cluster
for
now
in
the
cloud
and
then
get
a
URL
to
be
able
to
hit
that
I
actually
do
Cuddy
integration
testing
with
everything
available.
A
The
data
migrations
as
we're
kind
of
tripping
pieces
off
the
monolith
yeah,
it's
painful
we've
done
in
a
couple
of
different
ways,
actually
primarily
depending
on
the
size
of
the
data.
Sometimes
it's
just
too
much
to
like
do
during
a
maintenance
window.
You
know,
because
we
could
in
theory
you
can
deploy
the
service.
The
new
service,
get
it
ready,
wait
for
a
maintenance
window,
do
a
data
migration
and
then
turn
it
on
and
hope
it
works
and
see
how
that
goes.
A
So
there
are
definitely
cases
where
you've
had
to
give
it
a
couple
of
weeks
of
running
until
most
of
the
data
has
been
double
written
and
is
in
both
places
and
then
kind
of
do
one
last
update
during
a
maintenance
window
and
then
we're
good
and
we
killed
the
old
table
but
yeah
it's
it's
painful.
Sometimes.