►
From YouTube: Evolving a Continuous Delivery Workflow - Gard Rimestad & Øyvind Ingebrigtsen Øvergaard
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Experience Report - Evolving a Continuous Delivery Workflow to Kubernetes using Spinnaker - Gard Rimestad, Schibsted & Øyvind Ingebrigtsen Øvergaard, Finn.no (Intermediate Skill Level)
Schibsted started deploying to Kubernetes with Spinnaker about a year ago. In that time, application deployment has evolved from simple point and click pipelines in Spinnaker to being fully defined as code with custom integrations against Kubernetes. The combination of everything-as-code and using contracts and conventions to abstract irrelevant details, has simplified configuration and led to improved usability and reduced maintenance. Gard and Øyvind will talk about Schibsted's journey, motivations for each step of the progression, key lessons learned, and challenges still to come.
About Gard
Gard is an engineer that lives for traceable, stable and fast delivery pipelines. The weekly number of production deployments is why he goes to work. When he is not working on optimizations of deployment, he goes into his man cave and plays with arduino, raspberry, quadcopter or 3d printer projects. Active contributor on Spinnaker. Previous speaking experience: - 2017 Devoxx Belgium - Spinnaker - get control over your deployments - https://cfp.devoxx.be/2017/talk/RTU-8006 - 2017 Spinnaker summit: - Talk - Spinnaker at Schibsted - Panel discussion - Spinnaker’s Kubernetes Integration - Panel discussion - Operating Spinnaker - 2016 AWS Meetup - Deployments pipelines in Spinnaker - https://www.meetup.com/AWS-User-Group-Norway/events/233922802/
About Øyvind
Developer, ops-er and everything in between-er. Currently interested in leveraging modern infrastructure to build robust systems that support delivering applications continuously. Has been using Kubernetes in production since 1.1. Previous speaking experience: - Kubecon EU 2017 - “Lessons Learned From Maintaining Continuous Delivery While Migrating From a Static Infrastructure to Kubernetes” (with co-speaker) http://sched.co/9Tbo - Booster 2017, Software 2017 - “Moving FINN.no to the cloud” (norwegian, with co-speaker) - Pycon.se 2014 - “Deploying enterprise services 60 times a day using Python.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
A
There
are
over
a
thousand
software
developers
total
across
different
sites,
and
these
different
sites
were
initially
more
or
less
autonomous
in
terms
of
how
they
worked
and
moving
like
this
four
years
has
led
to
similar
problems
being
solved
in
different
ways
in
different
places.
For
example,
some
sites
are
running
on-premise.
Some
are
running
in
the
cloud
and
some
do
kind
of
a
combination
of
both.
A
This
also
means
that
the
different
sites
have
different
levels
of
maturity
and
tool
sets
with
regards
to
how
they
do
continuous
delivery
so
to
drive
convergence
across
the
different
organizations.
Central
protocol
technology
organization
was
established
around
2015
to
try
to
find
common
ground
across
all
the
sites
leverage
economies
of
scale
to
solve
the
important
problems
in
one
way
for
everyone,
even.
B
Though
I
have
been
even
and
I
am
in
the
part
of
the
central
tech
organization,
our
mission
is
to
promote
continuous
delivery
and
build
infrastructure
to
support
it.
We
work
hard
to
make
it
easy
getting
into
a
flow
that
makes
deployments
so
cheap
that
all
master
builds
go
out
to
production.
It's
an
established
fact
that
reducing
the
deltas
on
each
deployment
reduces
the
risk
of
bugs
and
lowers
the
lead
time
on
fixing
them
as
the
developers
or
already
have
the
change
that
fresh
in
mind.
B
B
Maintaining
an
application
should
be
as
cheap
as
possible
and
require
as
little
effort
as
possible.
Central
maintenance
should
be
possible
to
do
with,
without
involving
all
the
users,
for
example,
when
upgrading
a
Cuban
s,
cluster
user
should
only
be
involved
if
the
changes
affect
some
or
requires
changes
in
their
code.
In
order
to
do
this,
we
must
slow
their
footprints
for
all
our
users.
B
Even
we'll
get
back
to
this
later
transparency,
it
should
be
easy
to
see
who
deployed
what
and
one
it
should
be
easy
to
navigate
from
the
outside
in
and
also
from
the
inside
out
by
outside.
In
we
mean
that
from
our
continuous
delivery
tool,
we
should
be
able
to
look
at
an
application
and
look
and
learn
the
footprint
of
the
application.
B
What
resources
are
connected
to
this
application,
and
by
inside
out
we
mean
that
you
can
use
queue,
control
to
look
at
objects
and
see
what
commits
that
made
them
and
what
pipeline
execution
that
created
them
and
that
you
can
browse
to
your
pipelines
and
everything
from
cube
control.
Looking
at
annotations.
B
This
is
our
golden
path
for
deploying
application.
First,
you
put
your
code
in
get,
then
it's
picked
up
by
Travis
and
being
built
in
the
CI
system.
Artifacts
are
being
uploaded
to
our
factory
when
the
build
is
complete,
spinnaker
integrates
against
Travis
and
finds
the
build
and
also
loser,
which
pipelines
to
trainers.
It
can
be
one
or
it
can
be
money.
Then
spinnaker
deploys
to
kubernetes
through
the
AHS,
bridging
our
fee,
our
service
that
even
will
expand
on
a
little
bit
later
in
the
presentation.
B
B
We
have
been
running
spinnaker
in
production
for
two
years
now
and
we're
doing
about
400
the
unique
production
deployments
each
week
with
it
at
the
heart
of
spinnaker.
You
find
the
orchestration
engine
called
Orca
each
pipe
lines.
Each
pipeline
runs
in
anatomic
way,
so
that
there's
only
one
truth
for
the
state
of
one
pipeline
execution,
spinnaker
integrates
against
many
systems
and
builds
a
very
powerful
solution.
By
combining
all
these
solutions
together
and
getting
more
strength
than
you
would
from
each
one
of
the
individual
system
alone.
B
Spinnaker
has
many
kinds
of
stages:
I'm
gonna
mention
a
few
that
is
relevant
to
this
presentation.
First
off,
you
need
your
pipelines
to
be
triggered.
I
mentioned
ci
builds
could
trigger
a
pipeline.
You
can
also
trigger
based
on
other
pipelines,
and
you
can
figure
them
manually
and
rerun
them
manually.
B
There
is
also
stage
called
the
web
boof
stage,
which
is
built
by
shipstead,
which
is
a
custom
stage.
We
look
at
it
that
a
sir
get
out
of
jail
for
free
card
and
you'll
see
how
we've
used
this
to
extend
spinnaker
to
cover
our
needs
with
regards
to
kubernetes
in
this
procession,
presentation
will
also
trigger
CI
jobs,
so
you
can.
If
you
have
integration
tests,
you
can
trigger
a
job
on
Travis
or
Jenkins
or
gitlab
CI,
to
run
the
tests
after
a
deployment
to
make
sure
it
meets
your
business
requirements
as
well.
B
B
Here
you
can
see
a
pipeline
execution
in
spinnaker.
The
this
pipeline
has
been
triggered
by
a
Travis
job.
You
can
see
that
in
the
far
left
corner
it's
a
it's
a
link
to
the
Travis
job
that
triggered
this
pipeline
and
spinnaker
extracts.
All
the
artifacts
that
are
listed
below
from
this
pipeline
execute
this
CI
job.
So
we
can
see
all
the
artifacts.
This
has
been
produced
by
the
CI
job.
Spinnaker
injects
this
into
the
context
of
the
execution,
so
you
can
use
it
later
out
in
the
pipeline.
B
This
pipeline
consists
of
four
stages:
there's
three:
the
pro
stages
and
one
manual
judgment
it
approach
to
death.
First,
then
pre,
then
you
have
manual
judgment
and
then
with
deploy
to
production.
Each
of
the
deployment
stages
consists
of
three
sub
stages
which
make
up
dev
deployment
pre-employment.
B
B
So
we
started
started
off
deploying
into
kubernetes
for
real.
Maybe
one
year
ago
we
were
using
the
first
spinnaker
integration
against
kubernetes
and
we
were
very
early
adopters
and
to
us
it
turned
out
to
be
very
hard
to
manage,
there's
so
many
settings
to
tweak
in
this
integration.
This
results
in
many
unique
pipelines
and
kubernetes
footprints.
B
These
pipelines
had
to
be
set
up
by
pointing
and
clicking
in
the
UI
for
every
application.
This
mate
made
it
very
easy
to
make
mistakes,
even
though
you
were
knowing
what
we
were
doing
and
there
was
no
easy
way
to
trace
the
different
ways
versions
of
the
pipeline.
So
if
a
change
needed
in
the
pipeline
configuration
broke,
it
was
not
all
that
easy
to
revert
it
back
to
a
working
state.
B
Worth
another
challenge
we
ran
into
with
the
first
iteration
of
our
cabinets
for
plants
was
that
user
is
expected
to
be
controlling
more
of
their
infrastructure
footprint
within
the
context
of
a
deployment
than
what
we
were
used
to
coming
from
AVS.
The
infrastructure
was
something
we
deployed
our
services
on
top
of
and
suddenly
with
kubernetes
the
infrastructure
became
a
part
of
the
deployment
ingress
service
and
config
maps
should
be
a
part
of
the
code
and
always
the
pride
together
with
the
application.
B
We
decided
to
build
a
custom
stage
in
spinnaker
to
do
this.
There
already
is
the
generic
stage
in
spinnaker,
which
I
mentioned
earlier,
the
web
group
stage,
so
we
simply
created
a
service
which
runs
inside
the
kubernetes
cluster,
where
spinnaker
can
push
the
manifests
for
it
to
provision
into
the
clusters.
We
call
this.
Our
training
wheels
here
is
where
its
hair
is
how
it
works
for
an
ingress
manifest.
First,
you
store
your
manifesting
it
as
a
yellow
file.
Then,
in
the
build
we
upload
this
to
our
factory
using
a
unique
version.
B
Spinnaker
extracts
the
version
and
all
the
details
about
this
manifest
from
the
CI
build
spinnaker
triggers
the
training
wheels
stage
with
a
concrete
version
of
the
manifest
approval
in
the
CI
build.
Then
the
training
wheels
applies
the
raw
manifest
to
kubernetes.
This
solution
was
made
to
unblock
our
cabinets
adoption.
However,
it
felt
wrong
in
several
ways.
One
thing
that
we
lost
control.
One
thing
is
that
we
lost
control
over
who
can
deploy,
to
which
namespace
and
also
we
built
a
service
to
do
the
low-level
operation
that
spinnaker
is
supposed
to
do
for
us
for
free.
B
A
So
God
does
just
talk
about
two
other
ways
that
we've
deployed
the
applications
and
manifests
to
kubernetes
with
so
both
are
using
the
rocky
we
have
to
say
API
underneath
correct.
This
really
has
also
made
Pio
to
manage
infrastructure
and
applications,
but
in
some
ways
it's
not
that
opinionated
about
how
it
wants
you
to
do
things.
A
A
So
the
gels,
sorry,
the
abstraction
is
really
the
fails.
Configuration
format
that
you
can
see
on
the
far
left
side
of
the
slide,
which
defines
defaults
for
almost
all
configuration
settings
and
we've
tried
to
make
the
default
values
for
each
of
the
settings
same
so
that,
for
instance,
by
default,
you
get
auto
scaling.
A
So
this
configuration
is
typically
stored
as
a
younger
file
in
the
application
source
repository
and
during
the
promet.
It's
used
as
input
to
custom
component
called
a
deploy
daemon,
which
is
responsible
for
hydrating.
The
input
configuration
with
cluster
specific
information
and
transforming
it
to
the
kubernetes
objects
that
are
necessary
to
deploy
the
application.
A
A
A
A
A
Additionally,
the
declarative
power
plants
can
be
expressed
as
templates
that
other
pipeline
configurations
can
build
on
top
of.
So
what
we've
done
is
that
we've
built
use
this
feature
to
build
a
set
of
general
templates
that
the
teams
can
use
to
orchestrate
deployments
for
applications
to
kubernetes.
A
So
if
you
use
one
of
these
templates,
you
can
get
away
with
only
minimal
amount
of
configuration
to
set
up
a
pipeline
for
your
application,
like
illustrated
in
the
middle
here,
where
there's
a
flow
of
deploy,
stages
first,
to
the
dev
environment,
then
to
the
pre
or
staging
environment,
and
then
they
have
a
man,
your
judgment,
step
before
deploy
into
production.
So
you
can
see
an
example
of
that
configuration
that's
necessary
to
set
this
up
above.
A
You
only
specify
the
source
template
the
namespaces
that
we
own
in
this
case
it's
delivery,
and
they
were
also
saying
that
we
want
manual
judgments
so
much
like
EOS.
These
centrally
own
templates
make
it
easier
for
us
to
make
improvements
on
the
deployment,
orchestration
configuration
and
mechanism
and
then
roll.
This
changes
up
to
all
applications
without
having
to
change
the
actual
configuration
inside
each
application
explicitly.
A
This
request
contains
the
application
name
target
namespace
document,
your
friends
and
the
EOS
configuration
most
them,
creates
a
false
application,
custom
resource
or
third
party
resource
containing
this
information,
depending
on
the
kubernetes
version
that
we're
targeting.
Then
the
demister
deploy
a
demon
that
I
mentioned
earlier.
It's
kind
of
designed
after
the
operator
pattern.
So
it's
responsible
for
reading
this
EOS
application
configuration
transforming
it
to
commit
these
resources
and
creating
or
updating
those
resources.
It
will
also
create
our
fails
application
status
resource
which
describes
the
stateless
over
deployment
because
we
are
only
considering
deployment.
A
B
Yeah
so
meanwhile,
while
we
have
been
working
with
the
Fiats
deployments,
spinnaker
has
been
evolving
the
cabanas
integration
and
implemented
a
shiny
new
one.
Instead
of
shoehorning
kubernetes
into
spinnaker
terminology,
they
have
created
a
new
integration
that
embraces
kubernetes
and
brings
the
strength
and
flexibility
of
kubernetes
into
spinnaker.
The
spinnaker
core
team
has
listened
to
the
community
and
rethink
how
the
kubernetes
integration
should
be
done.
B
Spinnaker
manifest
based
kubernetes
deployments
is
the
streamlined,
fully
integrated
solution
to
managing
manifests
that
we
were
missing.
Conceptually
is
solving
the
same
problem
that
we
solve
with
our
training
wheel
solution.
The
difference
is
that
this
is
a
native
part
of
spinnaker
and
lets
us
leverage
the
authorization
mechanisms
in
spinnaker
to
make
sure
that
users
can
only
deploy
to
the
namespaces
that
they
are
entitled
to.
So
our
goal
here
is
now
to
deprecate
and
remove
the
training
wheels
service,
replacing
it
with
the
spinnaker
manifest
based
solution.
B
So
what's
the
key
features
of
the
manifest
based
kubernetes
integration
in
spinnaker,
it
uses
cube
control
to
talk
to
cuban
artists,
and
you
can
specify
several.
You
can
have
several
Cube
control
binaries
installed
on
the
machine,
and
you
can
use
different
versions
and
target
different
versions
of
kubernetes
having
the
right
cube,
control
version
installed.
This
is
important
because
the
previous
version
were
using
a
Java
library
and
it
was
a
global
library.
So
we
were
very
unsure
on
how
it
will
tackle
different
cabinets
versions.
B
It's
easy
to
project
clusters,
just
like
we
do
with
the
other
cloud
providers
like
AVS
in
ships
that
there's
no
coming
new
stages
into
spinnaker,
that
supports
kind
of
hydration
of
manifests
and
stuff
to
that.
So
we
will
look
into
that.
Maybe
include
that
in
our
creation
of
the,
if
y'all
stand,
feels
justin
resources
insertion
a
been
described
earlier,
how
most
by
in
spinnaker
chief
Yas.
Now
with
the
version
two
cabinets
integration,
we
moved
some
of
the
responsibility
away
from
must
and
into
the
spinnaker
native
integration,
because
this
gives
us
several
things.
B
B
B
So,
as
I
mentioned
earlier,
we
rehydrate
the
the
manifests
in
spinnaker
and
what
we're
doing
is
to
inject
CIA,
takumi
data
and
so
on.
In
order
to
surface
this
to
the
end-user,
we
build
some
customizations
on
top
of
the
vanilla,
spinnaker
UI.
We
use
the
public
NPM
packages
of
spinnaker
and
add
or
our
customizations.
On
top
of
that,
as
event
mentioned,
we
are
quite
large
and
distributed
organization,
and
one
of
our
goals
is
to
drive
for
convergence.
B
How
can
we
help
each
other
promoting
endorsed
solutions
internally
in
the
delivery
team,
which
we
are
a
part
of?
We
have
approached
this
by
linking
services
together
in
feeis,
we
provide
paved
roads
to
get
kubernetes
deployments
secret
handling,
using
our
internal
secret
management
stuff
and
also
metrics,
using
the
centralized
Prometheus
set
up
in
spinnaker.
We
have
created
a
custom
UI
to
make
us
the
feels
experience
better
and
to
expose
the
other
tools
we
have
by
binding
them
together
in
a
combined
view.
We
believe
that
we
will
on
the
right
side
there.
B
You
can
see
some
details
about
this
replica
set.
This
is
a
screen
shot
from
our
spinnaker
setup
and
we
identify
that.
It's
a
fewest
application
and
we
show
custom
sections
to
the
users.
We
have
lifted
information
that
we
think
is
important
to
the
end
user,
and
here
the
user
easily
can
see
what
commit
that
is
actually
running
in
production,
get
the
link
to
the
build
and
also
quality
gate
code,
quality
reports,
metrics
and
documentation.
B
Another
thing
is
that
all
these
fields
are
added
by
having
annotation
in
cube
from
an
address
on
the
manifests.
So
when
using
cube
control,
you
can
see
the
same
same
data,
so
we
we
believe
that
we
will
get
a
lot
out
of
building
our
own
tight
integration
in
the
UI
to
make
spinnaker
a
natural
place
to
start
when
debugging
issues
and
trying
to
understand
how
an
application
works
and
combining
these
things
through
annotations,
we
can
direct
the
user
in
in
both
directions
to
investigate
problem.
B
There's
been
don't
work
upstream
to
make
it
possible
to
extend
the
UI
by
just
using
annotations,
and
this
is
an
example
of
that,
so
here
nyan
cat
is
added
to
to
the
UI
just
by
adding
a
special
key
to
the
annotations
and
some
HDR
now
on
the
value.
This
can,
of
course,
be
used
to
more
useful
stuff
like
links
to
admin
interfaces,
specific
dashboards,
the
front
end
of
the
service
of
or
whatever
the
team
or
application
owner
finds
useful.
A
So,
in
terms
of
adoption
on
this
system,
we
have
quite
good
deduction
on
spinnaker
itself,
with
around
400
production
deploys
a
week,
and
the
EOS
is
also
well
proven,
as
the
deployment
system
of
choice
asked
one
of
our
sites,
but
we're
still
in
the
early
stages
of
adoption
of
this
kind
of
integrated
system
with
templated
pipelines
and
integrations
in
spinnaker
and
so
on.
There
are
several
several
teams
and
sites
that
are
involved
in
the
early
stages
of
getting
started.
A
We
have
one
site
that
has
75%
of
their
services
being
deployed
by
us,
but
not
using
spinnaker
for
orchestration
I
mean
I,
won't
start
that
are
running
more
or
less
all
their
services
kind
of
an
earlier
generation
of
our
power
plants
is
called
templates,
which
used
a
first
version
of
the
acutest
integration
in
spinnaker,
with
training
wheels,
and
we've
used
a
lot
of
learnings
from
that
process
to
improve
the
templates
into
what
they
look
like
now.
So
we're
still
working
hard
on
getting
more
adoption
in
terms
of
next
steps.
A
So
we
could,
for
instance,
do
something
like
add
the
link
to
trigger
our
incident
process.
So
you
can
start
up,
and
you
see
that
something
is
wrong.
Are
we
working
on
service
to
service
integration
in
spinnaker
so
that
we
could,
for
instance,
highlight
deployments
that
might
be
affected
by
an
active
incident,
for
example
in
the
underlying
current
adjuster
or
in
the
docker
registry,
or
something
like
that
with
information
regarding
that
incident?
So
you
know
that
you
might
be
experiencing
issues.
A
We
also
have
kind
of
long-term
plans
for
providing
a
single
entry
points,
for
configuring
continues
delivery
for
your
so
that
you
can
configure
build
pipelines
and
deployment
all
in
one
place.
And
finally,
we
also
have
another
long-term
roadmap
to
look
into
how
we
can
provision
databases
and
queues
for
applications
automatically
based
on
the
probe
configuration
rather
than
having
the
nooses
having
to
set
those
on
the
side
which
is
kind
of
what
I
have
to
do
now.
A
So,
to
try
to
wrap
up
and
provide
some
learnings,
it's
nothing
evolutionary,
but
we
learned
that
it's
important
to
continuously
improve
your
workflow.
Designing
up
front
is
hard
you'll,
get
it
wrong,
so
start
working
one
way
and
continuously
improve
based
on
what
works
and
what
hurts
another
thing
we've
learned
is
that
opinionated
systems
might
be
slower
to
develop,
but
can
also
pay
off
in
easier
maintenance
and
no
probability
over
time
and
being
opinionated
about
how
you
do.