►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Cloud Native Transformation - Pavel Nikolov & Matt Hope, Fairfax Media
This is a story about a 185-year-old publishing company which goes through cloud native transformation. In less than a year the team manages to switch from releasing their monolith monthly to releasing their microservices many times a day. Pavel will describe the process of adopting more than half of the CNCF incubating projects, creating hundreds of Kubernetes namespaces in multiple clusters, while ensuring excellent automation and observability practices. And with all this, the vast majority of developers in the team do not need to use kubectl, helm or anything other than git and chat bots. Damian will talk about what this transformation means for the business and how it affects the company culture.
To Learn More: https://sched.co/Grb7
A
B
Cuckoo
so
we're
both
engineers
at
FX
media,
now
v,
x,
medias
just
recently
merged
in
with
another
company
in
Australia
called
nain,
so
we're
now,
one
of
the
largest
media
organisations
in
Australia
FX
media
itself
is
bid
over
180
years
old.
We
focused
largely
on
print
media
searching
newspapers,
and
these
days
you
may
have
noticed
everyone's
looking
blogs.
You
mediums
sites
like
that
so
they're,
the
kind
of
people
that
we're
competing
against
in
this
day
and
age
and.
A
Just
just
to
give
you
some
some
perspective
of
what
we
are
working
on.
We
are
working
on
a
huge
publishing
pipeline
that
basically
receives
news
from
various
content
management
systems,
news
feeds
and
then
pushes
this
to
print
media,
mobile
applications
and
some
of
the
most
popular
news
websites
in
Australia
and
just
to
to
note
something
here.
A
regular
website
that
has
millions
of
users
usually
focuses
on
unreliability.
You
want
your
website
to
always
be
available.
Ok,
and
if
you,
if
you
achieve
that
you're
all
good,
but
we
we
need
to
get
one
step
further.
A
We
not
only
need
to
be
highly
available.
We
can't
afford
to
be
still
because
we
are
serving
news.
Ok,
so
we
can't
ever
afford
to
have
like
up
stale
website
because
that's
no
longer
needed,
so
not
only
we
need
to
be
highly
available,
but
we
need
to
always
always
update
our
websites
as
soon
as
possible.
A
So
you
might
be
asking:
how
is
this
talk
related
to
you?
So
if,
if
you
are
here
in
this
conference,
obviously
you're
interested
in
in
the
CN
CF
hosted
projects,
namely
kubernetes
and
and
all
the
ecosystem
around
it
and
regardless
of
whether
you're,
considering
starting
a
transformation
or
you
are
already
into
it,
I
will
I
will
try
to
aim
at
and
I
will
try
to
share
some
of
the
steps
of
our
journey
and
will
present
our
journey
as
group
of
violence.
A
That
were
that
we
visited
ok,
we'll
try
to
tell
a
little
bit
of
story
here,
and
maybe
maybe
some
of
the
islands
that
you
have
visited
or
you
will
visit
will
be
different.
Maybe
you'll
find
some
intersection
points,
but
the
main
idea
is
that
you
know
we
share
our
experience.
Obviously
every
single
island
deserves
its
own
talk,
so
this
will
be
a
high-level
case
study
its
kind
of
overview
thing,
but
hopefully
you
will
be
able
to
give
you
some
kind
of
some
some
good
points
of
view
and
important
takeaways.
B
So
why
is
all
this
relevant?
Why
do
we
actually
talking
about
a
transformation?
We
got
very
very
lucky.
We
got
the
opportunity
to
do
basically
a
whole
business
reboot.
We
got
to
address
and
look
at
what
everything
we
were
doing
from
the
top
down,
and
this
is
not
just
technology,
so
we
were
given
the
opportunity
to
do
a
green
field
deployment
inside
and
a
hundred
eighty
year
old
company.
This
is
not
an
opportunity.
B
Obviously
that
comes
up
very
often,
and
we
had
you
really
were
able
just
grab
it
with
both
hands
and
fortunately
for
us
it
was
a
whole
of
business.
It
wasn't
just
the
technology,
it
was
also
the
business
drivers,
the
editorial
teams
and
a
number
of
other
component
everything
sort
of
really
gelled
together
and
gave
us
this
wonderful,
wonderful
chance
to
steal
a
phrase
from
a
colleague.
We
were
given
the
opportunity
to
disrupt
ourselves
media
these
days
is
hard
if
you're
talking
about
a
very,
very
old
legacy
print
company
you're
against
all
the
new
media.
B
B
Very,
very
early
on
we
needed
stability.
Although
this
was
a
Greenfield
deployment,
we
had
other
teams
that
were
sitting
on
top
of
our
infrastructure.
We
had
development,
we
had
editorial
staff,
our
production
clusters
needed
to
be
stable,
rock-solid
from
pretty
much
day
one,
but
we
didn't
know
everything
and.
A
This
takes
us
to
the
unknown
Island,
okay,
so
with
basically,
nobody
in
our
team
had
any
prior
experience
with
kubernetes
at
enterprise
scale.
Okay,
so
it's
one
thing
to
go
to
the
getting
started:
maybe
build
some
proof
of
concept
but
supporting
something
at
enterprise
scale
with
literally
you
know
a
lot
of
load
from
the
beginning
that
that
is
not
a
trivial
job
and
usually
when,
when
you
start
working
with
that
with
a
new
system,
obviously
we'll
try
to
learn
about
it.
A
You
will
try
to
read
about
it
and
everything
that
you
don't
know
you'll
try
to
find
find
about
it
and
read
about
it
and
test
it.
However,
there
are
a
lot
of
unknown
unknowns.
This
slide
is
about
the
unknown
unknown.
Okay,
so
in
our
team,
as
matt
said,
we
were
going
to
multiple
transformations.
At
the
same
time,
we
had
only
only
one
developer
with
prior
my
experience
and
everything
we
have
is
written
in
Gaul
and
we
didn't
have
it.
A
We
had
only
two
developers
with
prior
experience
in
micro
services
and
we
claimed
to
have
a
micro
service
infrastructure,
and
you
know
we
were
learning
many
things
on
the
go
and
we
had
very
tight
deadlines.
That's
a
typical
business
situation.
We
did
not
have
time,
hey
we'll,
stop
doing
our
business
as
usual
and
we'll
learn
these
new
technologies
who
are
actually
trying
to
learn
on
the
fly
and
with
unknown
unknowns.
The
thing
is,
you
only
know
that
you
don't
know
these
things
when
you
fail.
A
Okay
and
one
example
that
I
can
give
for
this
thing
is
maybe
setting
your
resource
requests
and
limits
for
for
your
reporting
to
Burn
Notice.
So
in
the
documentation,
you
will
see
that
these
things
are
usually
optional.
However,
if
you,
when
you
try
to
deployed
something
in
production-
and
you
want
it
to
be
reliable,
you
absolutely
must
set
resources
a
request
and
limits
for
your
port
and-
and
that's
that's
extremely
extremely
important
if
you
want
to
ensure
good
reliability
and
scalability,
and
that
leads
to
the
next
i1,
which
is
the
proper
preparation
island.
A
B
We
initially
we
had
to
spend
a
lot
of
time,
making
sure
that
our
environments
were
standard,
consistent,
redeploy
able,
repeatable
a
similar
pattern.
This
is
important
for
no
operations,
it's
important
to
developers.
It
means
you
can
take
one
set
of
skills,
the
one
thing
and
just
directly
apply
it
to
another.
There's,
no
crazy
differences.
All
around
the
place.
One
of
the
things
we
did
find
that
actually
worked
out
quite
well
for
us,
was
in
kubernetes
we're
actually
using
a
very
large
number
of
namespaces
one
of
the
common
patterns.
B
I've
seen
in
a
lot
of
other
tutorials
is
having
very
small
number
of
land
spaces
with
lots
of
things
in
them.
We've
gone
the
other
way
and
that's
actually
paid
off
quite
well.
For
us,
we've
got
lots
and
lots
of
small
little
namespaces
and
using
things
like
namespace
annotations,
we've
been
able
to
enrich
and
more
easily
manage
your
infrastructure.
A
And
this
takes
us
to
May,
in
my
opinion,
the
most
the
most
important
slide
of
this
whole
thing,
and
that
is
the
skeptic
Iowa.
Okay,
so
I
can't
stress
this
enough.
No
is
temporary,
yes
is
forever
kind
of
forever.
So
what
I
mean
by
this
is
you
have
to
be
extremely
skeptical
when
you
introduce
new
technology
into
your
production,
stack
right,
yeah,
of
course,
by
all
means.
There
are
so
many
good
projects
in
the
current
infrastructure,
but
initially
we
were
resisted
introducing
serverless.
A
We
introduced
resistant
into
introducing
service
mesh
and
few
other
things,
because
they're
under
heavy
development,
if
you
introduce
them
early
in
your
and
I'm
talking
about
two
and
two
years
ago.
Okay,
so,
if
you
introduce
these
things
into
your
stack
early,
maybe
you're
solving
one
problem,
but
what
you're
actually
doing
you
are
not
solving
a
problem,
you're
introducing
more
complexity.
You
install
it
once,
however,
you
maintain
this
forever,
so
think
about
the
maintain
maintenance.
Afterwards,
you
have
to
update
versions.
This
will
interact
with
your
other
components.
This
set
restrictions
on
your
further
development.
A
So
when
you
introduce
news
new
technology,
new
component
into
your
infrastructure,
think
about
the
consequences
and
only
ever
install
something
and
introduce
something
into
your
production
stack.
If
you
absolutely
must,
if
there
is
absolutely
no
other
way
of
solving
the
problems,
so
probably
I
think
I
saw
this.
I.
Think
I
saw
this
title
somewhere
in
Twitter
about
pull
requests
merging.
It
can
be
applied
to
many
things,
but
think
about
the
consequences.
Maybe
you
have
a
problem,
it's
very
easy
to
just
install
something
that
solves
the
problem,
all
good,
and
then
you
know
some.
A
It's
somebody
else's
problem,
however,
think
about
maintenance.
You
think
about
what
happens
next
and
then
next
step
of
our
journey
is
the
secret
type.
Okay.
We,
like
you,
have
some
general
idea.
You
want
to
go
through
this
massive
transformation
and
install
your
application
in
your
CUBAN
these
clusters,
and
then
you
have
this
secret
island.
It
doesn't
matter
what
type
of
business
you
have
and
what
type
of
applications
you
have.
You
need
secrets.
Your
applications
need
to
interact
with
some
kind
of
Secrets.
It
doesn't
matter
if
it
is
database
connection,
strings
tokens
certificates,
you
name
it.
A
You
need
some
kind
of
secret
make
sure
that
your
first
hello
world
application
actually
works
with
secret
from
day.
One
I
can't
stress
this
enough
and
try
to
try
to
add
secrets,
revoke
secrets,
rotate
secrets
and
if
it
all
works,
then
on
and
only
then
you
can
proceed
with
your
development.
This
is
something
that
you
can't
leave
for
day.
Two
you
can't
do
later.
This
is
something
that
you
need
to
do
from
the
very
beginning.
There
are
plenty
of
projects
in
the
echo
system.
A
B
Also
I
thinking
about
things
that
will
bite
you,
if
you
don't
think
about
it
from
day
one
credentials,
access
control,
our
BAC
and
that
kind
of
things
it
is
very,
very
difficult
to
just
have
a
cluster
that
was
built
with
no
access,
control,
setup
at
all
and
then
try
and
turn
access
control
on.
If
you
want
to
imagine
it
well,
just
take
my
word
for
it:
we've
been
bitten
before
it's
a
nightmare.
It
is
horrible.
After.
B
But
these
kind
of
things
require
a
lot
of
planning.
There
are
quite
a
lot
of
preparation
and
you've
got
to
really
think
quite
hard
about
how
to
provide
the
right
levels
of
access
control.
You've
got
operations,
engineers
who
want
me
to
do
X
you've
got
developers
who
might
need
to
do
y.
You
need
to
provide
the
right
level,
so
people
can
do
their
job,
so
you're
not
getting
in
the
way.
One
of
the
things
we've
found
is
quite
useful
for
that
kind
of
tooling.
These
things
like
chat
BOTS.
B
A
B
So
observability
is
very
important,
knowing
what's
happening
in
your
cluster,
with
your
applications.
Knowing
what's
going
on,
is
very
important,
knowing
when
sites
down,
why
it's
down
very,
very
important
things
and
there's
lots
of
very
good
talks,
there's
lots
of
really
good
papers
and
things
out
there.
These
are
some
of
the
tools
that
we've
been
using
that
have
been
incredible.
These
are
tools
that
we
spent
a
lot
of
time
either
using
developing
being
back
to
things.
To
keep
in
mind,
though,
is
there's
lots
and
lots
of
different
things.
B
You
might
need
to
collect
lots
of
different
metrics.
You
need
to
collect
data
about
your
infrastructure,
obviously,
but
collect
everything,
because
you
never
know
when
you
might
use
it.
One
of
the
things
that
we
do
in
our
office
now
is:
we've
got
you've,
got
dashboards
and
status
screens
absolutely
everywhere.
One
of
the
things
that's
really
useful
is
we've
actually
got
a
status
screen,
showing
our
current
costs,
so
our
cost
day-by-day.
So
when
there's
something
that
goes
hint
you,
we
can
see
that
very
quickly.
B
A
You
obviously
have
some
kind
of
applications
and
these
applications
are
doing
something,
otherwise
you
wouldn't
have
them
so
just
measure.
How
often
this
thing
happens,
how
much
time
it
takes,
how
many
messages
or
requests
you're
receiving
or
I
don't
know,
maybe
the
the
input
size
or
the
output
size
and
just
to
give
some
examples,
a
lot
of
people
are
just
using
their
system.
Metrics,
okay,
maybe
requests
per
second,
let's
just
take
this
example.
Is
this
a
good
metric?
A
Maybe
you
have
labels
with
a
status
code
if
it
is
HTTP
requests
like
status
code,
200
status,
call
500,
you
have
success
error
rate.
Is
this
good
enough?
Well,
it's
not!
You
might,
you
might
have
totally
good
set
of
only
successful
requests,
but
you
need
underlying
business
metrics
for,
for
example,
for
our
business.
That's
publishing
an
article
how
many
articles
we
have
published
in
the
past
minute
our
day,
whatever
we
might
have
only
successful
requests,
but
no,
if
nothing
gets
published,
is
this
good?
A
No,
so
you
need
to
be
able
to
cross-reference
business
processes,
be
things
that
happen
in
your
application
that
developers
responsibility
also
in
in
the
tracing
world.
It's
not
enough
to
just
have
distributed
tracing
in
your
system
and
then
say:
hey
we're
using
open
tracing
we're
using
this
in
that
system.
You
need
to
have
spawns
for
every
single
thing
you
use.
For
example,
you
need
to
add
some
kind
of
wrapper
to
your
database
driver.
A
It
doesn't
matter
what
databases
you
need
spans
for
databases,
or
maybe
your
search
system,
or
maybe
your
caching
system
or
third
party
service,
HTTP
requests
or
G
RPC
request.
Whatever
think
you're
doing,
you
need
to
add
proper
middlewares
for
all
the
drivers
and
all
the
cross
up
references,
including
in
up
processes,
so
that
you
can
visualize
every
single
request
and,
in
addition
to
tracing,
obviously
you
need
logging.
Your
logging
needs
to
be
leveled
when
you
receive
millions
and
millions
of
requests
at
some
point
you
get
flooded
by
successful
requests.
A
You
only
interested,
let's
say
in
the
error
requests,
so
we
need
some
kind
of
level.
Obviously
you
need
structured
logging
so
that
you
can
filter
by
service
by
process
by
in
all
by
job
and
and
and
metrics
yeah.
You
need
metrics
for
everything,
but
that's
developer's
responsibility.
Developers
can't
just
say:
hey
it's
the
developed
problem,
they
need
to
add
metrics,
you
have
to
add
it
in
your
in
your
app.
A
So
then,
with
with
without
being
said,
this
brings
us
to
the
left
next
island,
which
is
the
micro
island.
Okay,
does
your
team
speak
micro-services?
Ours
did
not
when
we
started,
we
had
only
one
or
two
developers
with
prior
experience
in
micro
services,
and
you
know
this
is
this
is
very.
This
is
this
is
again
related
to
the
unnormal.
Knows
you
start
doing
something:
maybe
you
read
some
articles
and
what
most
people
think
and
that's
very
wrong?
What
most
people
think
is
that
they
have
this.
A
This
monolith,
this
existing
system
and
many
people
when
they
hear
micro
does
something
small.
So
we
have
to
split
it.
Okay,
divide
and
conquer,
we'll
split
it
into
smaller
parts,
and
everything
will
be
good.
That's
that
that's
really
really
wrong,
because
if
you
have
an
existing
system
with
a
problem-
and
you
split
it
in
two-
you
have
two
systems
with
a
problem
and
then
you
further
split
it.
You
have
it
just
basically
exponentially
increase
the
systems
with
the
problem
in
a
micro
cyst
in
a
micro
service
architecture.
A
Ask
yourself
this
question
to
check
if
you
actually
have
that
you
have
a
micro,
sir,
every
microbe
server
has
its
own
database
or
some
data
stores.
Maybe
maybe,
if
it
is
it's,
if
it's
staple
ask
yourself
this
question,
how
do
you
get
the
current
state
of
the
world
like
a
snapshot
of
the
world
from
more
than
two
services
at
the
same
time
and
ensure
reliability
in
on
scalability?
Think
about
data
consistency?
It
starts
getting
very
interesting,
the
more
features
you
added
it
is
very
interest
and
then
how?
A
So
without
being
said,
we
have
micro-services
and
they
need
to
talk
to
each
other.
Okay
from
day
one,
we
went
with
a
HTTP
kind
of
restful
architecture,
luckily
very,
very
quickly.
Like
few
weeks
later,
we
switch
everything
to
G
RPC
and
that
that's
paid
off
really
really
well
for
us,
and
HTTP
was
created
for
service
to
browser
communication
and
we
have
service
to
service
communication
so
that
that
was
a
very
key
decision
in
our
architecture
and
it
allowed
us
to
add
a
lot
of
useful
middlewares,
a
lot
of
useful
things
and
end
it
around.
A
And
on
top
of
that
we
are
using
prometheus
metrics.
We
are
using
logging,
middlewares
we're
using
tracing
middlewares
and-
and
we
have
a
lot
faster
ever
like
always
running
connections
and
the
discoverer
allowed
us
to
scale
a
lot
of
like
a
lot
of
requests
per
second
to
to
achieve
scalability
that
we
need
for
our
type
of
business
so
serialization
and
how
your
services
communicate
is
very
important
and
then
I
get
to
my
favorite
slide.
A
Ok,
when
you
start
a
new
greenfield
project,
ok,
you
have
two
options:
option
number
one
is
to
pick
a
micro
service
framework
for
your
favorite
to
ram
language.
In
our
case,
that's
go
like
the
data.
That's
absolutely
doesn't
matter.
You
can
pick
any
programming
language
you
want.
So
one
option
you
have
is:
basically
you
can
pick
your
favorite
framework
or
maybe
the
most
of
the
framework
with
the
most
stars
in
github,
and
you
can
go
with
it,
but
remember
our
restrictions.
We
had
to
deliver
features
extremely
fast.
A
Most
of
the
frameworks
are
kind
of
agnostic
about
most
of
the
technologies.
You
have
a
lot
of
things
that
you
can
tweak
and
adjust
and
configure,
and
that
requires
a
lot
of
prior
knowledge
by
a
lot
of
people
in
the
team,
and
we
did
not
have
this
time.
The
other
option
that
we
have
is
basically
we
have
some
kind
of
boilerplate
some
kind
of
project
that
it's
already
pre-configured
highly
opinionated,
and
it
has
you
know
everything
you
need
in
order
to
start,
so
we
decided
to
go
with
option
number
two.
A
We
created
a
small
application
and
that's
more
application.
Headers
level
clogger.
It
had
appropriate
health
check
in
life,
let's
probe
for
kubernetes
it
had
tracing
it
had
metrics.
It
had
examples
of
how
you
add
your
infrastructure
is
called
in
in
your
favorite
cloud
provider.
It
has
a
built
CI,
CD
pipeline.
It
had
pretty
much
every
single
production
already
feature
you
would
want
in
an
application.
A
Ok,
so
this
way,
every
single
one
of
the
developers
in
our
team
could
just
copy/paste
discipline,
just
work
this
thing
and
start
working
on
on
endpoints
on
actual
service
logic,
without
worrying
of
any
of
the
other
more
complicated
things.
So
we
had
people
who
are,
for
example,
using
open
tracing
without
fully
understanding
how
it
works.
We
had
people
who
are
using
metrics
without
fully
understanding
how
it
works.
They
didn't
have
to
set
up
anything.
They
were
using
from
their
own
proper
logging,
with
proper
configuration,
proper
tracing
with
proper
configuration,
all
the
observability
practices.
A
Basically,
they
had
examples
that
could
they
could
copy
paste,
so
we
were
delivering
features
and
half
of
the
team
was
not
like
hooli
understanding
how
the
CNC
F
components
worked,
but
they
just
worked
so
mostly
discus.
I
think
this
has
paid
paid
off
pretty
well.
We
had
one
bad
example
like
we
had
one
flaky
unit
test
and
that
required
that
we
send
hundreds
of
to
request
that
you
know
fix
this
everywhere.
A
So
it
wasn't
all
you
know,
smooth
and
perfect,
but
I
think
that
that's
one
of
the
reasons
that
we
managed
to
turn
this
project
into
into
into
what
we
expected.
Ok,
this
allowed
us
to
obscure
our
team,
but
there
are
other
things
that
Matt
will
mention
later,
but
that's
something
that
you
know
this
highly
opinionated
initial
problem
allowed.
A
Our
teams
that
did
not
have
prior
experience
in
that
programming
language
did
not
have
prior
experience
in
in
such
an
architecture
did
not
have
products
with
in
October,
natives
and
coop
cattle
or
any
of
the
tools
that
we
use
didn't
absolutely
understand
how
helm
worked
and
they
were
actually
using
it.
So
that
was
a
very,
very
fundamental
decision
for
us.
We
ensured
that
we
add
examples
for
all
the
types
of
metrics
with
an
actual
experiment.
Explanation,
if
you
have
similar
use
case,
use
this
type
of
metric.
A
The
actual
features
that
they
deliver
and
those
features
were
immediately
available
for
other
applications.
It
doesn't
matter
if
it's
mobile
applications
web
sites,
whatever
we
have,
they
were
available
to
consume,
so
it
doesn't
mean
look.
Your
journey
might
be
different
right,
I'm
talking
how
we
started
more
than
two
years
ago
and
with
what
we
had
at
the
time,
with
the
restrictions
that
we
had
at
a
time.
That
was
a
very
key
decision
for
us.
A
Maybe
if
you
have
a
more
experienced
team
or
you
have
a
prior
experience
with
micro
services-
and
you
know
what
you
are
doing-
ok,
maybe
you
can
go
with
some
more
more
popular
sort
of
micro
service
frame.
However,
we
decided
to
go
this
way
and
we
do
not
regret
it.
Ok,
this
is
something
that
has
paid
off
and
with
that
being
said,
we
had
a
little
bit
of
impostor
syndrome
in
the
team
because,
like
you
get
this
huge
existing
team
and
you
throw
them
at
this
new
new
architecture.
A
Ok-
and
you
tell
this-
is
what
we
do
from
now
on
and
then
many
people
have
questions
okay
and
and
then
we
had
to
focus
on
knowledge.
Sharing,
how
do
we
obscure
the
whole
team
in
flight
while
we're
delivering
with
extremely
tight
schedules?
Okay,
we
had
very
tight
deadlines
for
all
our
features
and
Matt
will
share
a
little
bit
about
our
knowledge
haiwa
how
we
manage
to
obscure
our
team
in
the
context
of
the
CNCs
ecosystem.
B
So
one
of
the
key
things
that
was
really
useful
was
that
openness
that
we
had
across
the
time
as
part
of
this
Greenfield.
Everyone
was
on
what
brainwashed
into
this
new
world
that
we
had.
We
had
this
new
awesome
solution
and
everyone
really
wanted
to
learn
more
about
it,
but
you
can't
just
no
suddenly
become
a
go
developer
overnight
and
write
new
micro
service
apps
unless
maybe
a
path.
So
it's
a
lot
of
work
to
get
there.
So
we've
had
a
number
of
internal
things.
B
One
of
the
things
that
I've
found
really
really
useful
internally
is
regular
brown
round
bag
sessions
and
we're
running
brown
bags
internally
across
the
whole
division.
We've
got
people
who
do
product
research
who
tell
us
about
how
they
do
their
jobs.
In
turn,
we've
got
developers.
We've
got
operations
engineers
talking
about
what
they
do,
how
they
do
it,
so
everyone's
got
a
better
understanding,
but
it's
also
key.
We've
got
people
who
specialized
in
or
we've
got
a
lot
of
knowledge
in
our
CI
CD
platforms
and
they
run
workshops
on
how
to
better
use
it
caveat.
B
Now,
we've
now
got
developers
who,
independently
decided
on
I,
wanted
to
add
a
new
component
to
the
CI
CD
platform
and
they've
gone
off
and
done
it
because
we've
got
these
skeletons
because
we've
got
these
other
tools
there.
They
don't
need
permission,
they've,
just
gone
off
additive
and
being
able
to
take
some
real
advantages
of
it.
We've
got
things
like
they'll
level
up
sessions,
our
internal
training.
B
B
So
we've
got
our
internal
internal
training.
We've
got
structured
and
unstructured
training
and
giving
people
that
opportunity
to
learn
to
play
to
potentially
learn
and
playing
things
that
aren't
their
exact
jobs.
But
this
sort
of
to
the
side
has
been
really
beneficial
in
building
stronger
teams
and
understanding
what
the
other
guys
are
doing,
because
if
you
understand
as
an
Operations
engineer
what
the
devs
are
doing,
you
can
do
your
job
better.
B
They
can
do
your
job
for
you
if
you're
really
lucky
see
sit
back
and
reap
the
rewards
of
the
devs
doing
all
the
work,
brilliant,
absolutely
brilliant,
but
we've
got
this.
We've
been
pushing
really
hard
for
this
DevOps
culture,
so
in
our
environment,
if
the
development
teams
build
it,
then
they
own
it
if
they
break
it,
it's
their
job.
To
fix
it.
Largely
speaking,
each
of
the
development
teams
gets
paid.
B
You
you,
your
incident,
arms
and
out
of
hours
calls
if
their
applications
fails,
which
is
absolutely
brilliant
for
me,
because
I
used
to
get
woken
up
all
night
long
from
things,
I
couldn't
do
anything
to
control.
You
know,
as
an
Operations
guy
I
get
allowance
for
things.
I
can
control
development
teams
get
your
arms
for
things,
they
can
control
and
you
know
sometimes
they
might
need
help.
Sometimes
they
might
need
more
assistance
and
we
can
help
with
that,
but
that
sort
of
culture
of
not
just
throwing
something
over
the
fence
sharing.
B
These
kind
of
benefits
have
actually
led
into
significant
and
measurable
business-type
improvements,
so
nowadays
we
were
able
to
go.
Where
are
we
there?
We
go,
so
our
old
application
was
a
very
big
monolith
which
had
about
a
three-week
release
cycle.
So
if
you,
the
feature,
was
running
a
little
bit
late,
you're,
potentially
waiting
six
weeks
nine
weeks
to
get
your
feature
into
production,
we're
a
news
company
where
we
need
to
sometimes
be
able
to
turn
around
things
much
much
much
faster
than
that,
and
it
just
was
not
possible.
B
So
these
days
were
actually
releasing
I
think
the
last
count
I
saw
for
some
of
our
stats,
we're
releasing
up
to
90
or
100.
Every
single
day
we've
got
people
that
are
working
on
applications.
They
released
three
or
four
times
over
the
weekend,
just
because
they
can
they
would
get
their
application
out
and
pick
something.
So
it's
been
really
beneficial
for
us,
but
the
culture
has
really
changed
because
of
that
ownership.
A
During
during
this
transformation,
we
have
seen
a
significant
growth
in
our
in
our
pageviews
in
the
number
of
subscribers
that
we
get
and
I
think
that
partly
this
is
due
to
to
the
this
the
fact
that
the
CNC
F
transformation
allowed
our
developers
to
focus
on
features
not
not
how
they
get
the
features
deployed.
They
were
actually
focused
on
features
all
the
time.
In
addition
to
that,
they
were
more
motivated.
They
were
actually
getting
sometimes
paid
to
work
on
open-source
project,
or
they
were.
A
They
were
working
with
with
things
that
are
more
exciting,
more
more
up-to-date
and
like,
let's
be
honest,
everybody
really
likes
working
on
things
that
are
more
modern.
Nobody
likes
working
on
still
technology
that
you
know
ancient
and
what
I
want
to
mention
is.
There
is
no
such
thing
as
I'm
done
with
my
kubernetes
deployment.
Ok,
this
is
not
a
destination
like
I
deployed
everything,
and
now
what
do
we
do
like?
Go
home
and
you
know
have
some
rest
doesn't
just
doesn't
happen.
It
is
just
an
ongoing
journey,
so
you
still
have
to
keep
maintaining.
A
You
still
have
to
keep
taking
care
of
the
whole
ecosystem
versions.
Change
really
really
quickly
things
change
quickly,
so
you
have
to
keep
up
to
date.
You
have
to
update,
you
know,
take
care
of
security
updates,
maybe
some
of
the
things
that
you've
implemented
have
already
changed
and
have
been
standardized
or
maybe
it's
another
project
have
been
adopted
or
graduated
the
the
scenes
you
have
ecosystem.
So
it
is
an
ongoing
process
and
I
think
that
it's
it's
an
ever
never-ending
journey.
A
A
A
B
So
the
question
was,
you
know
now
that
we've
been
through
at
least
this
stage
of
the
journey,
because
it's
not
a
destination,
we're
not
at
the
endpoint
yet
would
we
do
anything
differently,
hopefully
not
to
be
honest,
I
think
everything's
gone
really
well,
for
us,
we've
been
really
lucky:
we've
got
great
opportunities
to
do
this,
I
mean
the
opportunity
to
do
a
Greenfield
inside
of
a
legacy
company.
It's
not
something
that
comes
around
very
often
and
honestly.
Everything
we've
we've
got
has
improved
as
a
result,
so
we're
doing
pretty
well
I.
Think.
C
A
So
the
question
was:
what
was
our
timeline
for
this
project?
In?
In
general,
we
were
rewriting
multiple
huge
mass
hits
that
we
couldn't
afford
to
go
down.
For.
That's
that's
a
very
difficult
question,
because
for
some
projects
we
had
a
deadline
of
let's
say
two
months
for
all
our
projects.
We
like
we
had
different
deadlines
by
different
teams
within
the
organization.
Some
projects
we
just
completely
rewrote,
and
basically
it
was
a
DNS
switch
to
the
new
one
for
others.
A
We
were
replacing
section
by
section
on
the
website
and
were
multiplexing
some
URLs
to
the
new
system,
some
to
the
old
one.
In
total,
we
managed
to
do
everything
from
start
to
finish,
basically
moving
everything
into
kubernetes.
We
didn't
I
believe
less
than
a
year
and
a
half
like
here
in
four
months,
but
we've
spent
on
this
like
two
years,
so
it
keeps
going
there
we're
constantly
adding
new
things.
It's
two
more.
B
So
the
question
was
around
sort
of
making
sure
that
your
scale
ooh
you
your
projects
or
your
work,
that's
forked
off
your
skeletons,
don't
drifts.
If
significantly
am
I
understand
correctly,
we
were
actually
quite
lucky,
because
the
teams
themselves
was
a
really
small
number
of
developers
that
were
actually
working
in
concert.
So,
instead
of
adding
awesome
new
feature
to
the
skeleton,
they
might
add
an
awesome
new
feature,
the
skeleton
know
from
their
project
and
then
move
that
back
up
into
scale,
and
so
other
people
got
an
advantage
out
of
it.
B
B
B
So
the
question
was
around
the:
what
was
the
size
about
the
relevant
team's
work
on
the
project?
The
team's
actually
grew
over
time,
so
initially
the
operate.
The
pure
operations
team
was
two
to
three
people.
I
wasn't
around
at
that
time,
fortunately
or
unfortunately,
depending
on
how
you
look
at
it.
Now
there
were
how
many
developers
initially
directed.
A
B
B
A
So
yeah,
that's
a
very
that's
a
very
good
question.
I
believe
we
were
soon
running
out
of
time,
but
I'll
try
to
answer
so.
The
question
was:
how
did
we
create
our
skeleton?
Was
it
a
template
or
was
it
something
else?
It
was
a
very,
very
naive
project
that
that
was
just
a
HelloWorld
implementation,
with
one
HTTP
handler
and
one
T
RPC
Handler,
just
the
very,
very
minimal
kind
of
thing.
We
were
planning
to
create
a
command-line
tool
that
you
just
do
like
new
service
and
then
the
name.
A
B
So
the
question
was
just
basic:
standardizing
on
a
single
helm.
Chart
that's
true.
We
we
do
that,
but
that's
largely
for
our
internal
applications,
which
all
have
a
very
similar
pattern,
so
obviously
we're
using
lots
of
different
helm
charts
when
we
pull
in
other
applications
from
other
sources,
but
the
current
current
helm
chart
we've
written
is
very,
very
generic
and
open.
So
it's
actually
working
out
really
well
for
us.
In
earlier
days,
we
did
have
multiple
different
helm,
sharts,
but
centralizing
more
made
our
lives
a
lot.