►
From YouTube: Lessons Learned From Maintaining Continuous Delivery While Migrating From a Static Infrastructure
Description
Lessons Learned From Maintaining Continuous Delivery While Migrating From a Static Infrastructure to Kubernetes [B] - Audun Fauchald Strand & Øyvind Ingebrigtsen Øvergaard, FINN.no / Schibsted
Finn.no is Norway's largest website. Whilst running on static infrastructure, we support 300 microservices, 1400 deploys a week, and 6 minutes from git push to production. But our platform didn't meet the demands of our developers. So we started to migrate our platform from static servers to a dynamic approach, both on premise and in the cloud. We are not yet done with that process, but we have learned a lot. We want to share these learnings with you
The talk will maintain the following parts:
- Introduction to FINN, and its legacy infrastructure
- How we set up kubernetes, and how this has changed in the last 6 months
- On-premise vs GKE
- Tools we made for easier deployment and migration
- FIAAS (Tool for deploying apps with a minimal config for each app)
- HA proxy ingress controller for migrating app traffic
- How to integrate Prometheus and Kubernetes for painfree metrics Experiences
- Migrating apps are difficult
- Kubernetes is complex
- Developers like to make apps quickly
- Monitoring is everything
About Audun Fauchald Strand
Audun has worked as a developer and arcitect for 14 years across many industries. He now works on creating a common runtime for all teams in Schibsted. He has expereienced alot of TLA: gof, mq, ejb, wli, bpel, eda, soa,esb, ddd, k8s
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
B
Name
is
hello:
my
name
is
Inigo
I'm
a
developer,
only
infrastructure
team
and
pin
so
since
we're
talking
about
migrating.
Just
to
give
you
some
context
to
start
with
we're
going
to
first
talk
about
some
history
of
infrastructure
at
fin,
and
then
we
get
into
how
we
use
kubernetes
after
that,
we'll
talk
about
the
platform
that
they
built
on
top
of
kubernetes,
which
we've
called
fin
infrastructure
as
a
service.
B
So
what
is
fin
anyway
finish
on
the
region,
internet
marketplace
where
you
can
buy
and
sell
cars,
real
estate,
bits
and
pieces,
and
you
can
get
hotels
and
everything
tickets
and
find
a
job.
It's
know
as
second
largest
websites
in
terms
of
traffic
after
VG,
which
is
the
biggest
national
newspaper.
Now
Norway
is
relatively
small
population
wise.
B
Actually
it's
around
the
size
of
the
greater
billion
metro
area
Elizabeth
on
internet.
Your
later
life
and
where
you
can
sell
it
with
a
big
fish
in
a
fairly
small
pond
fin
is
owned
by
ships
at
Media
Group,
which
is
a
big
company,
other
poems,
among
other
things,
similar
sites
to
fill
in
several
other
countries
in
the
world.
B
So
there
are
some
vital
numbers
that
when
we
are
about
120
developers
and
these
developers
manage
around
350
micro-services
along
with
a
handful
bigger
applications
and
these
developers
ship
to
production
around
1,000
times
a
week
on
average
and
the
median
time
from
a
developer,
pushing
a
change
to
an
application
to
that
change
being
rolled
out
into
production
is,
is
six
minutes
and
it's
very
important
to
us
that
our
developers
can
deliver
value
quickly.
So
we
have
dashboards
and
KPI
built
around
these
numbers.
A
A
This
is
data
from
our
development
pipeline,
which
is
our
home,
build
system,
the
journey
to
continuous
delivery
of
micro
services,
offense
started
in
2007,
but
on
the
delivery
part,
you
could
really
only
start
to
see
signs
of
that.
If
you
do
deployments
by
week
in
2013
when
we
started
to
grow
and
then
in
the
next
few
years
we
had
a
real
big
growth,
so
I
think
we
have.
We
now
deploy
to
production
of
a
thousand
apps
a
week
regular
around.
We
can
go
up
to
1,200
1,400
and
we
did
this.
A
We
had
a
bunch
of
steps
we
went
through.
First
fin
was
a
monolithic
architecture
and
we
had
we
basically
deployed
it
manually
from
operators,
and
we
wanted
to.
We
wanted
to
change
that
so
that
the
management
haven't
gone
to
a
conference
on
her
Mary
poppendieck,
speak
and
I
wanted
to
achieve
some
of
the
development,
the
speed
she
talks
about.
A
So
they
came
back
and
said
instead
of
releasing
every
third
month
we're
going
to
release
every
month
and
then
on
the
technology
side,
people
started,
we
started
creating
creating
some
shared
with
Java
libraries
to
make
it
easier
to
create
services,
and
we
created
different
servers.
We
had
an
API
server,
a
module
server,
all
our
web
services
and
we
had
some
project
control
development.
A
A
But
then,
when
we
had
150
200
different
services
running
on
the
same
actual
bare
metal
servers,
we
started
getting
problems
because
one
application
started
creating
problems
for
another
application
and
we
learned
that
for
developers
if
they
know
the
guy
who
takes
down
the
wrapping
protection,
it's
okay.
But
if
the
guy
sits
on
a
different
floor,
for
instance-
and
you
don't
say
hello
to
him
on
a
daily
basis,
you
get
angry
and
he
don't
talk
to
that.
A
Guy
I
talked
to
the
introductory
department
anyway,
so
we
we
tried
to
solve
that
by
using
or
to
start
using
OpenStack.
We
equated
virtual
machines
for
this
separate
applications.
So
then,
instead
of
having
200
applications
running
on
one
server,
we
had
200-300
virtual
machines
running
which
was
not
easy
to
operate.
A
But,
of
course
we
want
to
be
cool
developers
and
read
the
Internet
and
we
went
on
happiness
and
hopefully
respect.
It
was
one
simple
answer
of
happiness
on
what
to
do.
If
you
had
to
solve
something
with
programming
doc
and
we
started
using
docker.
Really
we
had
doctor
and
doctor
is
great
I
think
for
developers
it's
easy
to
run
something
on
the
same
Maranello
closed
as
in
production,
but
as
soon
as
he
started
getting
any
kind
of
traction
on
running
docker
in
production.
A
We
saw
that
it
gives
you
a
new
set
of
problems,
because
when
they
wind
up
in
production,
there's
so
many
the
things
you
need
to
do
to
be
able
to
get
the
actual
value
out
of
docker,
because
when
there's
no
longer
any
binding
between
application
and
the
underlying
infrastructure,
you
can.
You
can
start
running
the
containers
for
every
Walter
and
then
you
need
some
kind
of
system
that
hangs
that
forward,
because
he
can't
do
that
manually
and
as
soon
as
the
containers
start
moving
around.
A
You
need
to
some
kind
of
services
cover
to
figure
out
where
the
containers
are
at
all
points,
and
of
course
you
don't
want
to.
As
someone
said
earlier,
you
don't
want
to
be
waking
up,
but
not
just
because
not
contain
the
dye,
see
if
you
can
just
restart
it.
So
we
need
to
have
some
kind
of
life
cycle,
support
and
monitoring
to
help
you
and
when
you
get
have
all
these
services
in
order
develop
all
these
developers,
you
need
you
want
to.
A
You
want
to
control
who
has
access
to
what
so
we've
also
kind
of
authorization
and
authentication
mechanism,
and
you
want
to
control
that.
And
lastly,
we
saw
that
when
we
started
using
docker,
we
we
crammed
local
stuff
into
our
containers.
We
started
putting
all
the
information
and
all
the
business
applications
into
one
container,
and
that
made
it
really
hard
to
upgrade
one
thing
we
had
to
build
a
new
container,
so
we
wanted
some
way
of
aggregating
our
containers
into
jobs
without
building
it
into
one
image.
A
B
So
we've
set
up
kubernetes
from
scratch
on
their
own
hardware,
and
this
was
before
the
cabinet
is
the
hardware
guy
it
existed.
So
there
was
kind
of
a
guy
that
pointed
in
a
general
direction,
but
since
kubernetes
is
so
flexible,
there
were
a
lot
of
details
that
we
had
to
figure
out
for
ourselves.
Things
like
what
bubbling
Linux
are
we
going
to
use
versions
of
the
different
components
are
actually
compatible
with
each
other.
How
do
you
solve
the
community
network
infrastructure
requirements
and
how
do
we
do
a
configuration
and
deployment?
And
all
of
that?
B
Let's
just
say
that
we
learn
a
lot,
but
luckily
the
situation
has
improved
a
lot
since
then,
it's
a
while
ago,
now,
anyway,
we
started
out
running
on
virtual
machines
that
we
provisioned
from
our
OpenStack
cluster
that
we
already
had.
We
used
to
terraform
for
this,
and
then
we
did
the
installing
and
configuration
of
kubernetes
itself
with
puppet
which
we
already
used
as
well.
B
But
after
a
long
period
of
debugging
hypervisor
performance
issues
in
OpenStack,
we
decided
to
just
repurpose
a
few
of
those
OpenStack
nodes
to
be
pure
cabinet,
ease,
notes
and
run
directly
on
the
metal.
So
right
now
we're
running
production
on
a
few,
pretty
large,
bare
metal
nodes
that
are
running
in
our
own
datacenter
and
we're
on
our
development
environments
in
container
engine.
This
is
really
nice.
A
The
main
reason
was
that
we
could
create
something
called
fin
investor
destructor,
as
a
service
I
mean
if
you're
Norwegian,
you
would
pronounce
that
fierce
and
that
basically
is
silliness
in
its
there's
funny,
because
now
we
have,
we
talked
about
migrating
stuff
to
fuse
and
there
is
management
sending
emails,
but
we
need
Morpheus
and
everything.
It's
quite
good.
So
I
really
recommend
trying
to
get
a
silly
name
for
your.
A
But,
as
I
said,
we
had
three.
We
had
130
developers,
I'm
trading,
50
microservices.
We
needed
to
we
needed
to
figure
out
how
to
run
fin
applications
on
kubernetes.
As
we
talk
more
about
what
we
basically
did
was
we
with
this.
We
created
a
set
of
contracts
that
describe
how
the
applications
and
infrastructure
should
talk
to
each
other,
and
we
said
as
long
as
you
had.
He
adhere
to
dis
contracts.
You
can
do
whatever
you
want
to,
and
this
reduced
the
solution
space
for
developers
a
lot.
A
They
don't
that
they
don't
know
how
we
do
networking
they
don't
know
how
to
do.
Storage.
It
dead,
just
know
what
FERC's.
Of
course
they
need
access
to
Cuban
artists
to
be
able
to
debug
and
do
migrations,
but
it's
it's
much
easier.
So
we
really
wanted
our
platform
to
be
opinionated.
We
wanted
it
to
be
clear
and
say
exactly
how
what
Athena
plication
looks
like
right
now,
we're
kind
of
in
the
we
migrated
as
many
applications.
So
we
know
said
this:
is
it
works?
A
You
couldn't
keep
as
unstable
as
you
want
to,
they
would
be
happy
and
now
we're
running
some
of
the
biggest
services
we
have
introduced,
auto
scaling,
so
we
scale
up
and
down
according
to
traffic,
and
now
it's
basically
now
most
developers
see
that
this
is.
This
is
a
better
way
to
go,
so
they
come
to
us
and
want
to
migrate
instead
of
the
other
way
around.
B
Sort
of
three
layered
model
to
describe
house
at
the
bottom.
We
have
a
kubernetes
cluster
which
can
be
our
own
premise,
cluster
or
one
hour,
Google
container
engine
coasters.
Let
me
try
to
make
sure
that
these
clusters
have
the
same
cluster
add-ons
and
so
on,
so
that
the
clusters
behave
in
mostly
the
same
way.
On
top
of
kubernetes.
B
We
run
what
we
call
platform
services,
and
these
are
tools
that
handled
application
needs
like
deployment
logs
and
metric
segregation
on
traffic,
ingress,
etc,
and
these
are
deployed
using
a
continuous
delivery
pipeline
that
we
built
based
on
helm
on
top
of
these
services
and
sort
of
around
these
services.
We
run
our
business
applications
and
these
can
rely
on
the
API
of
our
platform
services,
as
well
as
the
kubernetes
api,
and
these
are
deployed
using
the
file
system
which
I'll
get
to
in
a
second.
B
A
B
But
if
you
want
to
write
an
application
in
Haskell
or
something
you
are
sort
of
on
your
own,
you
just,
but
you
have
these
contracts
that
you
have
to
adhere
to
anyway.
So
you
know
what
you
have
to
do
to
implement
these
contracts.
We
use
a
Yammer
configuration
file
named
file
CMO
for
each
application.
B
This
file
lives
in
the
applications
source
repository
and
is
used
by
our
pipeline
and
deployment
system
to
the
right.
Here
you
can
see
the
the
piace
mo
config
file
for
the
service
that
renders
the
header
on
most
pages
on
our
sites.
It's
about
nine
lines
of
config
and
to
the
left.
You
can
see
the
beginning
of
the
kubernetes
manifest
diamo
for
this
application
when
it's
actually
deployed
into
our
cluster
and
it's
about
147
lines,
but
the
important
thing
about
this
isn't
necessarily
just
fewer
lines
of
config.
B
B
The
underlying
infrastructure,
if
we
have
to
without
the
application
developers
having
to
change
every
application
they
own,
this
makes
them
happy
and
when
every
application
sort
of
looks
the
same
or
at
least
similar
from
an
operational
perspective,
this
means
that
we
can
more
easily
do
changes
in
a
live
production
system
if
we
have
to-
and
that
sometimes
happens
since
most
applications.
Look
the
same,
it's
easier
to
get
better
development
speed
as
well,
since
developers
just
have
less
configuration
to
worry
about.
B
So
development
speed
is
important.
We
we
want
to
push
our
six
million
minute,
medians
I'm
the
production
even
further,
so
we
needed
the
employment
system,
that's
fast
and
reliable,
and
this
is
the
way
we
do
it
with
kubernetes.
We
have
a
deployable
that
has
two
parts.
We
have
a
few
more
config
file
and
our
reference
to
a
docker
image.
So
when
the
developer
pushers
have
changed
to
to
his
application,
these
two
things
are
packaged
by
the
CI
system
and
our
pipeline,
which
is
a
web
application
that
we
built
herself.
B
B
Then
we
have
a
kind
of
operator
that
runs
inside
the
cluster
we've
named
it
EOS
deploy
daemon
suitably
boring
name
it's
anyway.
It
subscribes
on
this
topic
and
it
receives
a
message.
It
talks
to
the
communities
API
server
and
creates
this
required
kubernetes
resources
based
on
the
piace
mo
file
and
when
it's
done
and
all
the
thoughts
are
running
and
healthy,
it
pushes
a
mission
message
back
to
the
pipeline
so
that
it
can
render
a
nice
green
box
in
the
web
front-end,
so
that
our
developer
knows
that
everything
is
fine
in
production.
A
When
I
saw
that
the
first
time
approximately
err
ago,
I
saw
that
it
was
incredibly
incredibly
powerful.
You
get
so
much
for
free
because
we
have
functionality
and
promethease
that
understands
the
Promethean
functionality
and
for
me
this,
it
understands
the
community's
api's
and
use
that
for
service
discovery.
A
And
then
because
we
have
the
metadata
from
Prometheus
and
we
put
some
additional
metadata
using
if
you
deploy
demon
that
flows
through
the
system,
we
have
that
metadata
in
Prometheus
and
we
can
use
that
in
graph
armor.
So
what
you
see
on
the
screen
here
is
basically
a
default
dashboard
that
any
application
that
deploys
to
our
clusters
get
for
free.
It's
just
because
to
me,
we
just
use
the
app
label
in
Prometheus
and
kubernetes
as
to
populate
the
drop-down.
A
So
each
new
app
just
gets
a
new
line
in
the
drop-down
and
you
just
go
to
the
tab.
This
dashboard
and
you
find
our
app
and
of
course,
then
we
can
also
do
things
across
every
every
application,
because
every
all
applications
send
out
CPU
that
are
the
same
way
for
instances,
so
we
can
easily.
Even
though
we
don't
don't
it,
we
could
say
what
team
uses
most
resources.
We
have
all
the
data
and
for
me
this,
we
just
need
to
create
the
query
and
get
someone
who
needs
the
data.
A
The
last
thing
we
built
is
something
called
the
FIA's
canary
app.
This
is
one
application
that
we
deploy
using
as
part
of
the
platform
applications
that
tests
all
the
functionality
it
tries
to
expose
the
metrics
and
see
that
this
comes
into
prometheus.
It
tries
to
talk
to
another
application
in
the
tester
and
it
put
some
logs
and
look
at
look
at
that
in
Kabana
and
this
we
use
as
part
of
the
cluster
deployment
pipeline.
A
A
It's
now
to
the
last
part,
which
turns
out
to
be
one
of
the
most
time-consuming
and
difficult
parts.
Migrating
applications
to
doctor
and
Cuban
I,
basically
make
them
cloud
native
and,
of
course,
easy
we're
doing.
That
is
just
saying
you
need
to
be
12
factor
and
at
least
four
new
applications
that's
quite
easy.
They
can
most
developers
will
do
that
just
by
reading
the
doc
and
it's
quite
easy,
but
we
have
loads
of
legacy.
We
spent
10
years
trying
to
write
applications
and
most
of
them
doesn't
really
fit.
We
have
some
one.
That
means
this.
A
Can
we
have
someone
that's
tied
to
some
tiny
money
detail,
inter
in
a
puppet
setup,
that's
impossible
to
replicate
in
kubernetes,
so
for
some
applications
migrating
to
FERC's,
like
in
our
work
and
for
some
it's
weeks.
The
funny
thing
is
even
the
people
who
spent
weeks
migrating
applications
to
different
kubernetes
tells
us
that
they
think
their
application
has
gotten
better
from
that
work.
It's
less
coupled
to
the
infrastructure
and
it's
easier
to
scale
up
and
down.
For
instance,
yeah.
B
So
adapting
applications
is
just
the
first
part
of
the
journey
for
migrating,
though
these
applications
are
obviously
connected
migrating
an
entire.
This
complex,
distributed
system
like
this
is
difficult,
so
we
needed
to
build
sort
of
some
mechanisms
to
make
it
easier
and
safer
to
do
at
speed
towards
excellence.
B
But
before
we
can
get
to
that,
or
to
give
you
some
context
for
what
we've
done
for
migrating,
we
first
have
to
talk
about
how
we
do
traffic
ingress
and
to
run
traffic
from
the
internet
into
our
cluster.
We
use
an
ingress
controller
that
we've
implemented
ourselves,
it's
based
on
a
GI
proxy
and
we
run
to
a
checkbox
instances
in
an
active/passive
topology
and
they
live
outside
the
cluster
itself
and
we
use
note
port
services
for
for
each
application
that
we
deploy
in
the
cluster
to
make
a
chip.
B
B
Let's
set
up
us
as
a
wildcard
DNS
entry,
and
then
the
traffic
will
be
routed
into
your
application
via
the
English
controller,
without
the
developers
essentially
having
to
do
anything.
But
you
can
also
override
this
in
the
fuel
CMO
config
file
by
specifying
a
different
domain
name
and
path
for
where
the
application
should
be
available.
B
So
basically,
what
we
do
is
rely
heavily
on
using
feature
toggling
in
the
load.
Balancer,
and
the
way
it
works
is
that
the
application
developers
can
create
feature
toggles
in
the
Unleashed
application,
which
is
a
web
application.
And
as
long
as
these
follow
a
specific
naming
scheme,
they'll
be
picked
up
by
the
English
controller
and
the
English
controller
wind
will
be
configured
to
route
traffic
in
a
way
that
you
could.
B
You
could
use
the
feature
toggle
to
say
that
I
want
10%
of
traffic
to
go
into
kubernetes
and
then
the
rest,
wouldn't
the
remaining
90%
will
continue
to
go
into
the
legacy
infrastructure
for
any
given
application,
and
we
can
do
these
logos
in
the
load
balancer
or
sometimes
we
do
it
in
the
client
library.
If
there's
a
service,
that's
only
used
internally
and
and
that's
cool
but
marketing.
A
So
for
conclusions,
kubernetes
isn't
the
same
for
everybody.
If
we
tried
moving
our
mimetic
application
of
the
kubernetes,
but
just
everything
would
just
gone
a
bit
slower.
Basically,
it
wouldn't
given
a
given
us
any
value,
but
we
need
we
need.
You
need
micro
services
and
we
need
small
components
to
properly
get
the
value
out
of
kubernetes
and
if
you're
as
big
as
Finland
is
which
in
Norwegian
terms
at
least
it's
big,
it's
probably
worth
it
to
create
something
on
top
of
kubernetes
to
restrict
the
choices
of
developers.
Also.
A
This
is
an
important
way
of
achieving
the
continuous
delivery
and
the
development
speed
you
need,
if
every
developer,
who
wanted
to
create
a
new
service
at
fin,
SATA
and
thought
about
how
they,
which
ingress
controller
they
will
choose,
we
wouldn't
be
able
to
do
anything.
We
will
just
discuss
technology
which,
of
course,
is
cool,
but
it
doesn't
for
some
reason.
Product
people
don't
like
that
and
metrics
metric
is
the
most
important
thing.
If
you
want
to
migrate
in
a
safe
way,
you
need
to
know
what
happens
in
production
and
it
took
didn't.
A
Take
us
too
long
to
create
kind
of
a
Minimum,
Viable
Product
thing
that
we
could
start
to
use
for
migration.
But
actually
migrating
takes
a
lot
of
time
and
we
I
think
the
sooner
you
can
start
migrating
the
sooner
you
can
get
actual
feedback
on
what
features
you
need
to
create,
which
is
useful
if
you
want
to
be
a
part
of
this
ship
set,
which
is
the
owner
of
Finn,
has
openings
so
go
to
job,
stop
chip,
stay
calm
and
then
it's
time
for
questions.
A
Well,
I
think
I
can
imagine
the
developers
coming
to
me
complaining
that
had
to
rewrite
everything,
Oh
actually
I
can
imagine
the
public
people
deciding
what
the
developers
should
do
coming
to
complain
to
me
that
developers
can't
great
products.
They
can
only
create
applications
again.
So
we
wanted
to.
We
want
to
reduce
things.
You
need
to
do
as
much
as
possible
right
now.
If
you
create
the
doctor
container
and
you
create
a
fire
siamo,
the
pipeline
will
just
deploy
your
application
into
both
systems.
C
C
A
Before
we
started
this
process,
Finn
kind
of
had
made
one
strategic
decision
they
wanted
to
have
you
build
to
run
it
so
for
this
120
developers?
This
basically
killed
the
operations
department
and
created
infrastructure
department,
which
was
both
the
Ops
guys
and
a
bunch
of
developers.
So
then
we
created
a
bigger
infrastructure
department
than
what
we
had
as
an
Operations
team,
and
that
gave
us
some
extra
some
extra
strength
and
made
it
possible
to
do
more.
But
at
the
same
time
we
said
at
this
department
shouldn't
do
ops.
A
They
should
create
tools
for
the
developers
do
their
own
operations
and
then
we
got
in
morally
right.
So
basically
we
moved
all
the
responsibility
to
the
developers
and
then
we
start
creating
tooling
that
made
it
easy
for
them
to
do
it,
and
this
really
works.
We
see
that
not
every
developer,
but
enough
developers
have
this
connection
to
their
production
environment
and
they
get
alerts
and
everything.
So
we
I
think
we
managed
to
achieve
this
ability
run
it
in
a
proper
way,
and
that
makes
everything
easier
because
then
they
also
understand
why
this
is
better.
A
A
Can
remember
one
where
the
applications
kind
of
had
they
had
to
preform
their
caches
before
they
could
start
accepting
traffic,
and
this
pre-warming
was
created
somewhere
deep
in
puppet.
So
it
was
this
extra
call
from
puppet
to
the
application
after
it
started
by
hitting
an
HTTP
endpoint
and
that's
not
possible
to
do
it
in
kubernetes.
So
they
had
to
kind
of
rewrite
their
application
to
do
that
themselves
and
then
that
didn't
really
fit
that
application
was
structured
and
then
it
that
took
a
week
or
two,
which
basically
is
the
worst
case.
We
had
you.
C
B
Had
vulnerable
just
because
their
infrastructure
different
a
lot
of
it
is
because
the
network
is
different.
One
example
is
we
had
an
application
that
didn't
set
a
time
out
on
its
database
connection
pools
and
for
some
reason
the
network
started,
not
remembering
method
connections,
so
it
ended
up
just
having
all
of
its
connections
waiting
for
a
response
forever
and
then
it
died.
So,
let's
just
strange
things
that
you
discover
when
the
infrastructure
changes,
we
also
had
problems
with
DNS
being
slower
and
no
js',
not
caching
DNS
responses,
so
every
request
was
slower.
A
D
A
We
started
to
be
on
the
new
missiles,
but
and
then
we
kind
of
the
people
before
the
hard
way
guide.
You
learn
a
lot
just
by
installing
it
and,
of
course,
even
when
the
hardware
guide
is
there,
you
learn
a
lot
and
then
basically
we
talk
to
developers
and
because,
because
we
have
this
ability
running
culture,
it's
the
developers
will
they
will.
They
will
just
have
to
learn
because
they're
responsible
for
their
own
production
system
and
when
we
change
the
production
system,
they
have
to
learn
and
that's
yeah.
C
A
B
A
A
I
met
them,
both
I
think
we
wanted
to.
When
we
started,
do
W
ability
run
it
the
developers.
They
didn't
have
the
knowledge
that
some
of
the
operator
operators
had
before.
For
instance,
we
have
apps
running
on
some
of
the
module
mode
servers
and
the
user
service
runs
on
mode
one
two
three,
but
not
for
because
of
Christ
has
a
point
but
five,
and
if
you
want
to
restart
that
service,
you
need
to
know
this
and
when
you,
when
you
give
this
responsibility
to
everybody,
it
just
doesn't
scale.
A
Everybody
needs
to
know
too
much
and
we
needed
to
we
needed
to
change
this
tool,
because
the
complexity
of
all
the
micro
services
kind
of
came
back
those
it
was
too
small
distress
on
the
big
foot.
We
grew
out
our
infrastructure
as
it
was
so
we
needed
quite
new
tooling,
and
that
was
the
main
risk.
I
don't
know
if
you
would
call
that
technical
or
manner,
but
we
kind
of
collectively
agreed
that
we
need
to
do
something
here
and
of
course
we
want
to
stock,
or
we
did
Oh.