►
Description
Kubernetes from Dev to Prod at GoEuro [I] - Subhas Dandapani, GoEuro
The GoEuro team uses Kubernetes across the whole application lifecycle, from Development to CI to QA to Production, using a variety of ways from Minikube to Hyperkube to GKE. Here are some insights from the team's use of kubernetes, maintaining standards, linting, validating and sharing k8s files, integrating with non-kubernetes raw docker and compute resources, and a lot more to transition to a fully kubernetized development to production experience.
About Subhas Dandapani
Coder, ex-ThoughtWorker, Blogger, Microservices consultant for startups and enterprises, Co-maintainer of devopsbookmarks, Do Node|Java|Ruby|DevOps|µServices, Practice XP|KISS|Lean
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
I'm
Subash
I'm,
an
engineer
at
guru:
Guerrero
is
a
europe-wide
travel,
aggregator,
travel
search
and
booking
engine.
We
have
a
operate
across
12
countries.
We
cover
thousands
of
stations,
200
plus
providers,
10
million,
plus
monthly
searches.
It's
just
search,
as
you
know,
not
sessions,
huge
number
of
volume
and
one
of
one
of
the
more
successful
startups
out
of
Berlin,
not
the
most
but
one
of
the
more
successful
startups
and
I'm
the
I'm,
an
engineer
there
leading
the
CI
CD
platform
team
and
the
transition
to
kubernetes
and
doctor.
A
So
our
transition
journey
goes
back
like
so
we
went.
We
scaled
up
from
50
engineers
and
eight
to
ten
services,
250
engineers,
50
services,
new
one
starting
every
one
or
two
weeks
we
scaled
up
from
10
coordinated
releases
to
150,
plus
independent
releases
happening
every
week.
That
number
is
not
definitely
an
upper
bound.
We
are
just
warming
up
and
we
went
from
legacy
infrastructure
to
see
ICD,
docker
and
kubernetes
through
all
kinds
of
legacy.
A
Problems
like
we
had
all
sort
of
integration
patterns,
file,
driven
integration,
shared
database,
rest
message,
queues,
metrics,
driven
integration
like
one
up,
writes
to
graphite
or
another
app
read
from
graphite.
Like
you
name
it,
you
got
it
compile-time
additional
artifacts,
like
probably
you
know,
and
across
the
spectrum.
You
also
had
all
sort
of
problems,
like
you
know:
dev
hacks,
for
ops
problems
and
ops
hacks
for
dev
problems,
like
all
the
things
that
you
would
expect
when
devs
and
ops
work
in
silos
yeah.
A
A
A
We
want
to
distribute
services,
so
we
just
have
the
rebel
or
docker
client,
and
this
hyper
VM
runs
a
docker
server
which
is
patched
and
if
you
specify
certain
build
arguments
like
if
you
want
build
I'm.
Caching,
like
if
you
want
to
mount
volumes
during
build,
remember
that
we
are
operating
with
lots
of
legacy
services,
you
just
use
Bulldogs
for
it
and
at
the
same
time,
if
you
just
point
your
docker
command
line
to
proper
docker
mission
or
mini
cube,
it
still
works
perfectly
fine
except
a
little
slower.
Yeah,
and
this
is
this.
A
Vm
is
always
our
first
point
to
begin
upgrading,
docker
or
kubernetes
versions
from
dev
to
prod,
like
it's
just
mini
cube
in
the
cloud,
but
this
is
where
I
just
want
to
start
with
slow,
so
developers
like
every
application
repository,
has
an
ops
folder
like
again
I'm,
showing
you
where
we
are
and
then
I'll
later
work
backwards.
So
every
application
has
an
ops
folder
which
contains
all
the
ops
requirements
for
that,
including
the
docker
file.
The
helm,
chart
CI
dotty
AMA,
we'll
see
more
about
that
soon.
A
A
A
It
started
out
as
a
fun
name,
but
today
I'm
just
kind
of
stuck,
so
it's
nothing
but
yama
kubernetes,
sharing
again
before
helm,
and
so
what
we
did
was
this
was
when
helm
was
transitioning
from
the
zero
point,
X
to
1
dot,
X
release.
So
what
we
did
was
we
adopted
the
helm
chart
standard
so
and
we
just
rendered
the
helm
chart
as
a
service.
So
you
can
call
an
API
I'll.
A
Just
show
you
up
next,
so
you
can
say:
HTTP
y8
s,
dot,
internal
and
just
give
it
any
repository,
and
it
will
check
out
that
repository
and
vendor
the
helm
chart
and
give
you
back
the
output
so
cube.
Cuddle.
Has
this
really
nice
but
hidden
feature
that
you
can
use
HTTP
URLs
when
you
are
applying
with
cube
with
cube
control,
and
we
just
use
it
to
the
max
and
so
and
why
we
do.
A
That
is
because,
and
we
also
dog
food,
the
same
thing
for
our
own
services,
like
all
of
our
DevOps
services,
that
we
deploy
in
all
the
clusters
are
also
using
the
same
way.
It
is
tool
and
the
the
widest
service
also
supports
additional
rest
options.
You
can
pass
in
parameters
to
the
templating,
but
we
just
expose
a
slightly
smaller
subject,
like
all
the
repositories
are
supposed
to
render,
with
only
one
variable
called
as
env
and
the
env
is
like.
We
have
predefined
environments,
development,
testing,
mock
QA,
pre,
prod,
prod,
EU
and
or
production
regions.
A
So
there
is
a
certain
set
of
environments
that
every
service
must
support
and
outside
of
that
everything
else
is
additionally
configurable,
but
they
should
work
out
of
the
box.
So
in
addition,
the
ml
sharing
service
also,
when
rendering
the
helm
charts
it
prints
out
a
whole
lot
of
annotations
to
every
resource
like
including
the
get
commit
ID
from
which
it
was
rendered
from
the
parameters
using
which
it
was
rendered
from.
We
can
always
trace
back
running
services
to
source
from
how
they
were
generated.
A
So
this
is
our
development
landscape.
So
we
have
ops
folder
in
all
the
reports:
docker
file,
CI
ml
and
everything
else.
We
have
the
ml
sharing
service
and
the
hyper
VM
to
develop
in
the
cloud
where
developers
can
just
say.
Cube
control
apply
all
the
services
that
they
want.
So
this
is
the
from
the
development
and
let's
move
on
next
to
the
integration
cycle.
A
So
we
have
the
CI
dot
Yama
file
in
every
repository
and
we
decided
not
to
reinvent
any
new
CI
contract
or
you
know
a
new
different
Yama
language.
So
instead
what
we
did
was
we
adopted.
The
gitlab
CI
ml
design
like
get
lab,
has
a
really
nice
yomel,
syntax
that
allows
for
pipelines
and
stages
and
cron
jobs
and
everything.
So
we
adopted
get
lab
CI
ml,
but
then
we
converted
those
into
CI
jobs.
A
On
our
own
terms,
like
we
filled
out
everything
else
like
all
the
plumbing,
we
basically
made
the
choice
of
CI
tool
independent
for
the
developers
like
we
today
we
use
Jenkins,
build
pipelines
because
build
pipelines,
support
manual
checkpoints.
Tomorrow
we
could
fish
to
go
CD
and
no
developer
would
notice
and
we
fill
in
everything
else.
Pr
checks,
manual,
checkpoints,
cron
jobs,
distributed,
build.
Caching,
the
current
signed
or
unsigned
docker
builds
like
all
the
stuff
that
you
have
come
to
expect
from
da
Christ
environment,
slack
notifications,
release
procedures,
filling
out
Excel
sheets.
A
All
of
those
are
automated
inside
developers
just
declare
their
jobs
in
this
format.
So
they
say
this
is
the
image
that
I
want
my
bill
to
run
on,
and
these
are
my
jobs,
and
these
are
the
stages
in
which
my
jobs
runs
on
and
for
every
job
if
they
can
also
specify
an
environment
which
is
you
know,
QA
production,
pre-production
and
so
on.
A
So
this
is
the
debt
lab
amyl
syntax.
We
will
come
to
the
credential
spot
later
and
we
build
out
the
rest
of
the
whole
pipelines
and
the
contract
for
the
pipeline's
is
really
simple.
Every
build
runs
in
an
agent
which
is
a
hyper
vm.
Like
that's
why
I
started
with
the
hyper
VM
in
development.
All
the
Jenkins
agents
are
hyper
Williams.
In
addition,
they
have
Jenkins
agent
running.
There
is
nothing
else
different.
A
It's
like
they
are
operating
on
the
same
dev
mission
that
they
are
used
to
so
and
the
other
contract
of
the
build
is
that
when
your
bill
starts,
docker
and
cube
control
are
readily
configured
just
start
using
it
like
you,
don't
have
to
get
the
credentials
or
anything
2.2,
QA
or
pre
prod.
So
if
you
say
that
environment
is
QA
well,
you
get
a
very
connection
to
QA
when
your
job
runs.
A
If
you
don't
specify
an
environment,
each
job
is
running
in
its
own
agent
with
its
own
isolated
kubernetes,
so
they
can
use
it
for
doing
end
to
end
tests
and
so
on,
and
the
other
thing
is
that
all
jobs
are
recreated
from
source
every
two
hours.
There
is
no
possibility
of
clicking
and
configuring
the
snowflake
CI
and
we
encourage
early
integration
tests.
A
Cube
CTL,
apply
y8s
any
service
that
you
want
and
hit
it
with
your
end-to-end
tests
and
we
take
jobs
all
the
way
to
production
like
we
don't
drop
them
off
in
pre,
prod
and
then,
like
DevOps,
doesn't
take
over
releases
at
any
given
point.
We
let
applications
go
all
the
way
to
prod
by
themselves,
but
we
ensure
certain
amount
of
safety
which
are
it's
covered
in
the
upcoming
slides.
A
So
this
is
the
closer
look.
So
if
you
don't
specify
an
environment,
the
agent
is
the
hyper
VM.
You
can
start
applying
and
running
your
ETA
tests.
Well,
if
you
specify
the
environment
as
preview
24
hours
well,
what
it
does
is
that
when
the
build
starts
it
dynamically
provisions,
a
new
cube
or
Nettie's
hyper
vm
and
points
you
are
built
to
that
and
then
runs
your
job.
Normally
like
you
can
still
do.
Dockable
cube
cereal
apply.
Accept
the
agent
is
accept.
You
know
the
the
results
persist
like
even
after
the
job
is
finish.
A
You
still
have
the
preview
VM
running.
It's
not
cleaned
up
like,
unlike
the
other
jobs
which
use
ephemeral
agents
and
they
are
cleaned
out
after
every
build.
This
preview
vm
is
dedicated
to
your
pool
request
and
it
stays
after
your
job
is
complete
and
you
can
use
this
to
show
previews
to
product.
You
can
use
this
to
do.
Debugging,
like
you
know,
you
can
find
out.
Why,
and
into
and
test
is
failing
and
so
on?
Yes,
so
and
but
the
build
contract,
the
most
essential
part
is
the
build.
A
Contract
remains
the
same
if
you
specify
the
environment
as
QA,
pre,
prod
or
any
of
those
shared
production
clusters
that
multiple
teams
are
using
right,
you
still
get
the
same.
Build
contract
like
just
do
Q
CTL
apply
to
deploy
and
we'll
talk
about
security
later
so
that
you
know
the
new
do
keeps
it
will
apply.
You
should
make
sure
not
to
mess
up
with
any
other
application,
and
there
is
also
some
amount
of
composition
that
you
can
do
like
it's.
A
It's
still
the
same
job,
if
you
say
stage
PR
those
jobs
are
going
to
end
in
the
PRS.
If
you
say
when
manual
right
on
normal,
build
pipelines
when
manual
means
it's
manually
triggered,
but
on
a
pull
request.
Well,
when
manual
means
that
you
have
to
comment
just
comment
on
the
pull
request
say
trigger
my
job
and
it
gets
triggered.
It's
still
like
a
manual
trigger,
and
then
you
can
say
the
environment.
As
per
you
24
hours,
you
get
a
provisioned
environment
and
mix
and
match
this,
and
you
can
do
product
demos
on
demand.
A
You
can
do
github
flow,
like
you
know,
from
the
PR
job,
you
can
say:
name
is
production,
a
manual
trigger
and
you
can
deploy
to
production.
So
if
you
want
to
follow
github
flow
style
of
going
from
PRS
to
production.
Well,
you
can
do
that
as
well,
so
people
mix
and
match
this
to
achieve
the
desired
effect
in
their
pipelines.
So
we
have
pretty
much
a
lot
of
patterns
covered
database
migrations.
It
took
us
a
lot
of
time
to
figure
out
how
to
run
database
migrations
using
cubes
ETL,
one
yeah.
A
So
basically
the
Viator
service,
it
just
accepts
content
type,
I
mean
except
headers.
So
you
can
say,
give
me
the
migration
in
a
JSON
format.
Give
me
the
migration
definition
of
the
container.
You
know
JSON
format
and
you
can
run
it
with
cubes
ETL.
So,
basically,
instead
of
docker
running
migrations,
you
can
run
cube,
CTL
run
migrations
and
they
have
access
to
all
the
cube.
A
cube
root
is
defined
services
and
endpoints,
and
everything
else
you
can
do
in
to
end
testing.
A
Shard
e
to
e
services
like
there
are
three
major
e
to
e
suits
in
the
organization.
All
of
them
are
also
treated
as
services.
They
also
have
their
CI
pipelines,
they
publish
docker,
artifacts
and
other
teams.
You
know
cube
CTL
run
them
using
waitis,
so
so
they
just
thought
they
just
build
and
run
their
own
services
and
they
run
the
e
to
e
tests
on
top
of
them.
We
also
have.
We
were
also
wondering
if
we
should
add
special
syntax,
like
whatever
you
seen
before.
Here
is
all
perfectly
valid:
gitlab
CI
ml.
A
So
if
you
tomorrow,
if
you
switch
to
get
lab
CI
nothing
drastically
changes
and
we
were
wondering
if
we
should
add
special
annotations
for
booting
additional
services
like
selenium
or
my
sequel
or
so
on,
like
how
Travis
CI
does.
But
then
we
decided
against
it
like
you
can
just
cube
CTL
apply
whatever
you
want,
so
we
did
not
need
any
additional
abstractions
for
it.
A
A
So
the
last
part
of
this
is
how
do
we
actually
deploy
all
these
services
once
the
once
they
cross
the
CI
CD
pipeline?
So
so
we
developed
the
simple
cube
or
Nettie's
api
gateway
proxy.
It's
called
as
or
Thetas.
It
takes
only
a
few,
a
couple
of
hundred
lines
of
node.js
code
to
proxy
the
kubernetes
api.
It's
so
simple
that
you
know
we
don't
even
have
to
open
source
it.
You
can
bring
one
up
just
by
playing
around
with
it,
so
it's
deployed
in
every
after
and
it
proxies
to
the
kubernetes
master.
A
So
people
don't
deploy
to
kubernetes
directly.
They
go
through
this
API
gateway
and
what
and
each
namespace
gets
its
own
username
and
password.
So
each
team,
rather
like
each
team,
gets
its
own
username
and
password
and
that
username
and
password
is
tied
to
one
specific
name
space,
so
they
can
deploy
only
to
that
namespace
and
every
other
operation
is
restricted.
So
today
we
got
to
know
that
in
occupant
is
one
point:
six
has
well
based
access
control,
so
we
will.
We
can
easily
replace
this
service.
We
can
remove
this
after.
A
We
have
ported
the
same
rules
over
to
cube
or
entities
or
matter
what
we
do.
In
addition,
outside
of
security
is
also
compliance
like
we
don't
want
to
run
services
that
do
not
have
CPU
and
memory
limits
in
production
like
this
is
not
just
role
based
access
control.
This
is
more
about
compliance
right,
so
we
we
require
certain
annotations
in
every
single
pod
and
every
single
namespace.
So
all
of
this
we
enforce
through
the
proxy.
A
So
whenever
you
try
to
apply
a
new
deployment,
we
make
sure
that
it
has
all
the
normal
rules
that
you
come
to
expect
in
cube
realities.
So
again,
the
approach
has
been
that,
instead
of
trying
to
create
an
abstraction
instead
of
trying
to
design
our
own
Yamal
file,
we
rather
do
compliance
and
validation.
A
Like
we
help
developers
to
write
the
correct
kubernetes
AML's,
we
whitelist
the
resources
that
they
can
use
so
right
now,
you
know
because
kubernetes
has
a
lot
of
API
s--
and
we
don't
want
this
to
become
like
a
Wild
West,
where
everybody
uses
some
sort
of
beta
or
alpha
API.
So
we
gather
just
whitelist
that
only
these
resources
can
be
used
in
production.
A
We
provide
a
lot
of
value
additions
in
the
cluster
servers.
Every
server
has
a
DNS
registration
agent.
That
sort
of
automatically
creates
a
DNS
entry
for
all
the
clusters.
We
provide
everything
else
that
you
come
to
expect
in
production
heap,
stir
pod,
restart
counters
logging,
shipping
container
logs,
and
if
the
container
logs
are
structured,
we
parse
and
ship
structured
logs
as
well.
A
So
we
sort
of
support
both
plain
text
and
structured
log
output,
and
for
now
we
use
node
ports
for
routing,
and
but
we
are
moving
to
ingress
and
we
are
also
working
on
like
whenever
you
create
a
service.
We
would
also
like
to
create
automatically
external
health
checks
for
it.
Just
the
same
way,
we
create
an
external
DNS.
A
We
also
want
to
create
external
health
checks,
like
Kingdom
spy
checks
for
all
the
services
we
offer
additional
services
like
sidecar
containers
like,
as
we
said,
we
deal
with
a
lot
of
legacy
services
and
some
of
them
require
like
geo
databases
which
are
like
two
gigabytes
or
three
gigabytes
in
size,
and
we
can't
build
it
into
the
image.
So
there
are
a
lot
of
additional
artifacts
that
every
service
requires,
and
we
just
provide
sidecar
containers
that
do
our
sinking
of
these
artifacts
to
a
host
folder
to
a
host
mounted
folder.
A
So
so
we
provide
sidecar
containers
for
additional
artifacts
and
we
also
provide
endpoints
for
region.
Specific
services
like
if
a
service
says
I
want
to
connect
to
my
sequel.
We
make
sure
that
this,
my
sequel
endpoint,
is
correctly
configured
for
that
service
in
every
single
region,
basically,
whatever
resource,
whatever
persistent
resource
that
DevOps
provides,
is
exposed
as
kubernetes
endpoints,
and
we
do
CD
for
our
own
stuff
as
well,
and
we
were
multi-region
focused
from
the
beginning.
We
did
not
try
to
hide.
You
know
there
are
two
production
regions
for
beau
euro.
A
We
did
not
try
to
hide
it
from
the
developers.
Developers
are
supposed
to
understand
and
configure
their
application
for
multiple
regions
from
day
one
most
of
the
time
like
90%
of
the
time.
The
configuration
is
the
same
except
certain
stateful
applications,
but
we
really
don't
want
to
hide
it
from
our
developers
so
more
and
more
annotation
driven
services.
So
some
of
our
services
have
problems
like
you
know,
suddenly
crashing
under
load
like
we
want
to
capture
their
heap
dumps
when
they
crash.
A
We
would
like
them
to
just
specify
that
they
need
heap
dumps
and
we
want
to
take
care
of
it.
Take
care
of
that
concern,
rather
than
them
having
to
write
the
old-style
cron
jobs
in
production
environments.
We
have
to
give
them
a
substitute
and
we
would
like
to
drive
it
through
annotations,
like
randoms,
auto
scaling
with
metric
queries
like
I,
think,
with
the
recent
release
of
kubernetes.
They
have
also
added
support
for
pluggable
metrics.
A
So
this
is
pretty
much
or
landscape.
We
use
the
hyper
vm's
only
for
development
and
CI,
all
or
shant
clusters,
or
in
GAE
in
Google,
Cloud
and
Google
container
engine.
We
use
our
Thetas
for
security
and
compliance
and
we
offer
a
lot
of
value-added
services.
So
this
is
where
we
are.
This
is
what
this,
what
it
took
us
to
get
here
and
so
sorry,
actually,
what
did
it
take
us
to
get
here
to
this
point
from
a
really
legacy
world
right?
So
the
first
thing
is
that
containers
are
a
cultural
change
right.
A
A
We
have
to
do
a
lot
of
presentations,
a
lot
of
emails,
internal
blogs,
internal
emails,
call
it
lobbying,
and
by
the
time
we
actually
started
this
out
a
lot
of
internal
prototyping
as
well
a
few,
a
few
hard
projects
that
we
had
to
prototype
and
prove
to
everyone
that
this
works
and
it's
better
and
by
time
we
actually
started
for
the
company.
Everybody
we're
like
we
needed
this
yesterday.
Where
were
you
all
this
time?
So
so
we
we
decided
to
do
all
this.
At
the
same
time
see
ICD.
A
We
made
a
unfortunate
sighs
across
the
whole
company.
We
were
doing
coordinated
releases,
a
lot
like
the
ten
plus
releases
before
before
see
ICD,
and
instead
we
now
made
it
that
you
know
all
services
have
only
git
commit
us,
the
version
that's
about
it.
So
there
is
no
semantic
versioning
for
services.
We
use
an
anti
questioning
only
for
libraries.
All
services
just
have
git,
commit
us
diversion
and
that's
about
it,
so
doc
rising
Cuban
writing.
A
So
this
was
the
main
focus
of
our
whole
effort
right
and
we
could
not
have
done
this
without
support
from
across
the
organization.
So
this
part
really
mattered.
We
had
to
take
support
from
the
teams
from
the
product
owners
from
the
managers
from
the
business
people.
Business
people
were
willing
enough
at
a
point
to
say
that
we
are
ready
to
take
risks
and
risks
mean
downtime.
Like
you
know,
when
you
are
doing
such
a
huge
change
and
when
you
suddenly
have
a
two
or
three
hour
downtime
or
revenue
loss,
then
it
reflects
really
badly.
A
It
puts
a
black
mark
on
your
whole
effort,
so
we
had
to
have
everybody
on
board
before
we
started
this
out
and
we
did
not
try
to
change
the
whole
world
at
the
same
time.
Like
you
know,
in
the
conference
we
see
a
lot
of
things,
a
lot
of
cool
things
happening
like
you
know,
Prometheus
monitoring,
start
logging
fluently
a
lot
of
things
happening
right,
but
we
did
not
want
to
jump
into
everything
at
the
same
time,
this
was
already
begin
of
as
it
gets,
and
the
second
thing
that
we
did
is
wherever
possible.
A
We
try
to
migrate
applications
as
they
are.
We
did
not
try
to.
You
know,
make
them
12
factor
in
the
process
of
migration
like
that.
That
could
have
just
we
couldn't
have
put
a
timeline
to
it
at
all,
then
so
so
we
we
profiled
a
lot
of
our
services.
There
were
some
services
that
were
close
to
12
factor.
There
were
some
services
which
were
not,
but
they
had
a
very
small
dependency
graph
and
then
like
a
few
services
which
were
really
hard
legacy
challenges,
and
we
always
split
the
migration
that
we
were
tackling
all
three.
A
At
the
same
time,
like
you
know,
we
had
one
prototype
project
with
a
really
hard
legacy
service.
We
proved
that
this
can
be
done
with
that
service
like
it
was
a
SBT
play
framework,
stateful,
singleton
application
and
a
very
core
piece
of
our
business,
and
we
started
with
that,
and
we
proved
that
we
can
do
for
that
and
then,
at
the
same
time
we
had
smaller.
We
were
also
focusing
in
bulk
on
these
services
as
well,
so
we
were
trying
profiled
approach
and
always
balancing
out
which
services
were
you
know
getting
to
see.
A
I
seen
EQ
British
and
everything
because
we
didn't
want
to
come
to
the
legacy
challenges
in
the
end
and
suddenly
hit
a
whole
new
wall
yeah.
So
what
this
means
is
that,
if,
if
some
of
these
services
took
five
minutes
to
boot
up,
if
they
had
a
six
hundred
megabyte
container
image,
if
they
had
all
those
dev
and
ops
hacks,
we
don't
care,
we
are
not
fixing
it
like.
A
A
Also
multiple
approaches,
like
you
know,
we
containerized
a
few
services
without
actually
serving
traffic
for
people
to
test.
We
did
shadow
comparison,
traffic,
routing
check
the
metrics
check
the
logs
like
check
the
metrics
and
logs
of
the
service,
which
is
not
serving
all
the
new
customers
right.
So
we
did
all
of
these
to
migrate
most
of
our
services.
A
The
other
thing
that
we
learned
is
work
closely
with
the
feature
teams
like
if
it
means
that
you
have
to
get
out
of
the
DevOps
desk
or
team
or
whatever
you
are
sitting
close
together.
It
means
yes,
like
just
get
out
of
the
desk
and
go
and
be
part
of
the
developer.
Journeys
like
go
and
be
part
of
the
feature
teams
and
help
them
work
with
them.
Let
them
on
their
success.
Let
them
go
and
present
to
their
company
what
they
have
done
and
how
things
are
better
for
them.
A
Now,
it's
okay
to
own
the
failures,
but
it's
okay
to
you
know,
distribute
the
success
and
avoid
the
same
mistake
that
we
have
done
for
all
the
years.
Like
you
know,
dev
and
ops
silos.
We
don't
want
to
repeat
that
when
migrating
to
Cuba
nice
right,
we
just
don't
want
to
do
it
under
a
new
name
and
the
other
thing
that
we
learned
again
more
and
more
learnings
is
that
embrace
standards
pick
a
standard.
A
A
So
we
always
enforce
a
subset
through
validation,
linting,
whatever
possible
means
through
testing,
sometimes
unit
tests,
and
so
that's
how
we
evaluate
a
standard.
And
then
later
we
choose
to
decide.
Okay,
whether
let's
now
move
this
out
across
the
board
and
whatever
we
develop,
we
just
dog
food.
Add
it
to
ourselves.
First,
like
devops,
first
used
it
to
deploy
our
own
services,
and
we
had
a
very,
very
early
on
decision
that
we
will
create
very
little
command-line
tools
and
wherever
possible,
docker
and
cube.
Ctl
should
be,
or
like
official
docker
and
cube.
A
A
So
what
cube
realities
enabled
us
like
Cubana
T's
was
what
enabled
this
transition
for
us.
Like
you
know
previously,
you
know
you
had
to
create
cloud
images
and
then
do
terraform
and
then
manipulate
physical
resources
like
when
cloud
first
came
cloud
was
called
virtual
and
now
that
kubernetes
is
here,
cloud
is
now
called
as
physical
and
cuban
tesis
called
as
virtual.
So
I
don't
know
the
lines
between
what
physical
and
virtual
means
anymore.
A
So
so
so
it's
like,
so
we
got
from
physical
to
virtual
resources
purely
because
kubernetes
makes
it
really
easy
for
a
developer
to
write
a
Ammal
and
manipulate
this
say
they
are
their
own
resources.
The
same
way
DevOps
used
to
manipulate
the
cloud
like
instead
of
an
instance
group.
We
were
using
a
deployment
instead
of
a
load
balancer.
We
are
using
a
service
and
instead
of
disk
images,
it's
docker
images,
so
we're
just
translating
one
on
one
cloud
setup
into
a
cube
or
a
tea
set
up
and
yeah.
A
We
just
moved
from
Packer
build
to
cube
CTL
apply,
and
this
is
safe
enough
that
we
can
give
full
control
to
our
developers,
because
this
one
nobody
really
feels
comfortable.
You
know
distributing
to
every
team
because
it's
very
hard
to
control,
and
it
also
involves
real
resources,
which
cost
money
so
cube
or
Metis
is
what
really
enabled
us
to
deliver
this
whole
pipe
like
there
are
individual
solutions
in
each
step,
for
AML
sharing
or
for
development,
or
for
CI
or
four-cd
and
for
deployment,
but
cube
realities
is
an
underpinning
behind.
A
A
What
we
would
like
to
do
going
forward
is
that
so
we
started
with
engineering
and
then
we
tackled
CI,
CD,
tooling
value-added
services,
and
then
we
have
to
automate
our
infrastructure
and
once
we
reach
all
these,
once
we
raised
the
bar
across
all
this,
then
finally,
we
can
do
site
reliability.
Engineering
like
once
all
our
applications
can
do
continuous
delivery.
We
make
sure
that
so
once
all
our
applications
have
compatibility
policies
like
if
every
single
release
is
compatible
with
the
previous
write,
then
what
we
can
do
is
we
can.
A
We
can
automatically
make
prod,
mean
prod,
zero
or
prod
one
person
like
right
now
they
are
deploying
200%
of
prod
traffic.
Without
changing
any
of
their
processes.
We
can
just
say
that
whenever
they
mean
I
want
to
deploy
it
to
prod
like
when
they
say
environment
prod,
we
can
deploy
it
to
a
10
percent
or
1
percent
or
zero
percent
broad
cluster
and
from
there
we
can
synch
to
all
the
other
clusters
based
on
metric
stability
and
so
on.
So,
but
we
have
to
cross
all
these
to
do
this.
A
We
also
want
to
do
better
state
management.
So
you
know
the
y8s
is
service
is
good,
like
it's
easy
to
use,
but
at
the
same
time
you
have
to
clean
up
afterwards,
like
you
know,
whenever
you
change
the
deployment
labels
or
you
know,
when
you
rename
the
service,
you
have
to
go
and
manually
clean
up
the
old
service
like
that's,
what
helm
is
really
good
at
helm
is
able
to
convert
this
whole
thing
into
one
atomic
transaction,
clean
up
everything
and
then
run
the
new
services.