►
From YouTube: Harder, Better, Faster, Stronger Pipelines @ Adobe - Larisa Andreea Danaila & Ionut Maxim Margelatu
Description
Harder, Better, Faster, Stronger Pipelines @ Adobe - Larisa Andreea Danaila & Ionut Maxim Margelatu, Adobe
At Adobe, Larisa and Ionut have been building and exercising fully automated CD pipelines with Spinnaker since 2018. Could you improve on that? Over the years, they've asked themselves: • how robust and hands-off are deployment tools when encountering problems? • how responsive to problems are these tools? • how do you optimize the deployment pipelines for speed? • how fast do you onboard a new codebase? Join them to hear how they've addressed these problems by making a seamless switch to GitOps with Argo. Find out how they managed to provide the same levels of speed and quality for high frequency deployments of dozens of multi-region microservices. They also explore in detail the challenges to move existing patterns to GitOps workflows, including: • isolated test environments • automated promotion throughout environments • automated rollback • automated canary analysis • wave deployments
A
B
A
A
A
So
we
targeted
to
move
our
robust
and
thoroughly
tested
pipelines
with
spinnaker
to
argo,
and
why
did
we
do
that?
And
why
didn't
we
reduce
that
technology
sprawl
to
spin
occur
only
well
a
kubernetes
native
and
github
stool
has
many
advantages
over
spinnaker
and
the
move
to
argo
was
backed
up
by
things
like
deployment
rift
with
spinnaker.
Your
deployment
can
drift
away
from
the
git
state.
A
Your
source
of
truth
of
what's
running
in
production
is
somewhere
between
your
last
spinnaker
logs
and
what
your
users
might
have
done
in
the
meantime,
as
opposed
to
a
git
of
stool,
which
would
reapply
the
git
state
whenever
a
drift
is
detected,
there's
also
a
learning
curve
which
can
occur.
There
is
application
code,
there
is
kubernetes
code
and
there's
also
spinnaker
specific
language,
while
developing
on
top
of
argo
is
just
writing.
A
But
the
move
to
argo
wasn't
without
these
challenges
and
the
main
ones
were
no
promotion
out
of
the
box.
No
rollback
out
of
the
box
and
what
about
deployment
configuration
changes?
Nobody
is
really
talking
about
them.
Should
they
be
handled
by
the
githubs
controller
controller
only
or
should
they
be
handled
through
some
kind
of
pipeline.
A
A
Also,
irrespective
of
the
deployment
tool
we
used,
we
always
promote
our
changes
through
environments
and
in
each
environment.
We
do
some
tests
or
health
checks
to
start
to
gain
confidence
in
our
release
candidate,
and
we
first
deployed
to
dev
then
to
stage
and
in
the
end,
to
prod
stopping
the
line.
If
something
fails
and
rolling
back
all
the
previous
environments.
A
If
environments
are
directories
in
the
main
branch
promotion
comes
down
to
orchestrating
commits
between
git
pads,
and
although
there
is
no
promotion
out
of
the
box
with
argo
cd,
you
can
easily
do
that
and
you
have
the
building
blocks
to
do
that.
With
argo
workflows
and
the
solution
we
came
up
with.
Is
this
workflow?
You
see
right
here
where
you
can
have
the
ci
as
part
of
the
workflow.
A
A
A
So
what
we
do
is
in
an
exit
handler.
We
revert
all
the
commits
all
the
deployment
commits
from
the
current
workflow
run
and
in
the
right
of
the
screen.
You
can
see
some
code
of
this
exit
handler
that
will
run
if
the
workflow
failed
and
it
will
run
the
template,
revert
commit
and
what
the
reverse
commit
does.
Is
this
script
right
here?
A
A
If
some
kind
of
promotion
is
required,
developers
need
to
do
that
themselves
by
first
merging
the
change
to
dev,
then
to
stage
and
so
on,
and
if
something
goes
bad.
If
something
fails
in
this
process,
rollbacks
are
in
the
responsibility
of
the
developer
as
well,
and
if
there
are
multiple
edits
in
a
short
time,
it
becomes
even
more
of
a
challenge.
So
the
solution
we
came
up
with
is
have
the
developers
edit
and
do
this
modification
in
some
other
place
in
the
first.
B
Thanks
so
we
saw
why
we
migrated
from
spinnaker
to
argo,
and
we
also
looked
at
some
of
the
challenges
that
we
we
met
along
the
way
adopting
argo.
Now,
let's
talk
a
bit
about
some
patterns,
so
here's
the
thing.
I
think
that
you
probably
are
familiar
with
this
trying
to
build
this
automated
pipeline.
That
takes
you
from
dev
to
stage
production
and
it
it's
really
a
balancing
act.
It
has
to
be
fast.
You
don't
want
your
developers
to
wait,
it
has
to
be
stable.
B
You
don't
want
for
failures
to
occur,
for
no
reason
and
it
has
to
be
safe,
like
you
have
to
make
those
changes
and
make
sure
that
whatever
you
just
change
gets
to
production
in
a
good
order.
So
let's
talk
a
bit
about
some
of
the
patterns
that
we
use.
The
first
thing
we
use,
or
the
first
pattern
that
we've
been
testing
for
a
few
years
now,
is
something
that
I've
seen
other
people
folks
use
as
well.
B
Is
we
take
every
change
that
goes
into
a
pr
and
we
deploy
it
into
a
dedicated
environment
for
that
pr,
and
these
pr
environments
are
isolated
or
semi-isolated?
I
guess
it
depends
on
the
service
and
the
dependencies
and
how
easy
it
is
for
us
to
set
it
up.
But
what
happens
is
that
whenever
a
pr
gets
open,
what
we
do
is
there's
a
provisioning
flow
happening
that
just
spawns
up
this
a
new
namespace
dedicated
for
that
pr,
maybe
provision
some
secrets
create
a
dedicated
argo
application.
B
It
is
also
committed
the
git
and
also
maybe
create
some
dns
entries,
and
once
that
happens,
is
that
whenever
we
have
changes
happening
in
that
pr,
so
whenever
a
developer
goes
in
and
makes
an
update,
we
deploy
that
code
automatically
in
that
environment.
And,
finally,
we
also
run
tests
like
we
run
functional
tests
afterwards.
So
what
this
does?
What
this
helps
us
do
is
actually
continuously
validate
the
changes
that
go
into
pr
before
we
even
consider
merging
that
code.
B
Second
thing
that
you
know
we
have
to
ask
ourselves
is
okay,
but
what
about
impact
like
we
diminish
the
risk
for
failure?
But
what
about
the
impact
of
failure
in
some
of
the
cases
where,
let's
say
maybe
the
functional
tests
don't
catch
a
problem
or
maybe
there's
a
failure
which
just
is
so
deeply
rooted
that
only
appears
after
running
for
hours
and
so
on.
Well,
we
have
a
particular
problem.
Let's
say
in
experience
platform,
we
there's
a
network
of
services,
mostly
for
data
collection
that
are
deployed
across
the
globe.
B
Today
we
have
seven
regions
and
whenever
you
know
we
would,
if
we
were
ever
to
try
to
do
a
simultaneous
rollout
across
the
globe,
then
any
kind
of
failure
would
have
a
great
impact
on
our
users
and
eventually
on
our
business.
So
what
we
do
is
we
use
wave
deployments,
and
this
is
just
like
the
name
suggests.
This
is
about
splitting
your
production
deployments
into
multiple
waves.
So
we
start
with,
let's
say
a
smaller
region.
B
Where
we
deployed
that
region
to
production,
we
let
it
bake
for
a
bit
and
if
everything
checks
out,
then
we
proceed
to
the
next
waves
and
so
on
until
we
have
deployed
to
all
regions-
and
this
is
all
done
using
aggro
workflows
and
the
dag-
it's
nothing
very
complicated.
In
the
end,
the
only
thing
that
I
would
like
to
add
there
is
that
some
teams-
it
all
depends
on
the
I
guess,
the
the
requirements.
B
Some
teams
actually
have
a
wave
zero
where
they
deployed
to
a
dark
launch
environment,
most
of
them
called
it
pre-prod
and
it's
called
pre-prod,
because
it's
it's
actually
identical
to
production
in
terms
of
configuration
and
deployment
configuration.
However,
it
doesn't
really
get
any
customer
traffic
like
it
doesn't
really
get
any
real
traffic,
it
just
uses,
it
gets
synthetic
traffic
and
then
it's
also
used
for
testing.
So
we
use
that
before
we
actually
even
deploy
to
any
real
user,
see
the
code,
that's
all
great.
B
But
what
about
at
the
deployment
level
itself
like?
How
do
you
make
sure
that
the
deployment
is
doing?
Okay,
and
probably
some
of
you
or
maybe
most
of
you,
are
using
our
rollouts
and
there
are
some
very
standard
way
of
doing
that,
like
you
can
use
analysis
on
the
rollout
based
on
permitted
metrics,
at
least
that's
what
we
do
prometheus
is
quite
standard
at
adobe.
We
use
it
for
to
monitor
just
about
anything.
B
You
can
also
do
sorry
about
that.
You
can
also
do
a
man-waiting
analysis,
which
is
basically
a
statistical
analysis
using
confidence
intervals.
It
actually
can
tell
you
if
there's
a
significant
degradation
between
you
know
a
baseline
version
of
your
application
and
the
canary
application.
Basically
what
you're
trying
to
roll
out-
and
this
is
very
useful
if
you
have
multiple
things-
you're
looking
at
and
using
prometheus
analysis,
just
doesn't
cut
it
out.
B
And
finally,
another
pattern
is
using
test
as
a
rollout
gate
sin
and
I
just
want
to
go
a
bit
into
this
one.
So
what
we
do
is
we
create
an
analysis
which
is
backed
by
a
job
like
a
regular
kubernetes
job,
and
the
purpose
of
the
job
is
just
to
run
functional
tests,
and
then
we
create
an
experiment
and
another
experiment.
Of
course
it
has
to
run
against
the
version
of
the
service
you're
trying
to
roll
out,
and
the
experiment
just
uses
the
analysis
that
we
just
defined.
B
So
it
basically
runs
the
functional
tests
and
finally,
in
the
rollout
you
can
sorry
you
can
roll
out
and
use
the
experiment
as
the
first
step,
so
it
actually
gates
the
rollout
on
the
result
of
the
experiment
and
this
what
this
does
is
if
the
tests
fail
and
the
analysis
fails,
the
experiment
fails
and
therefore
the
rollout
will
not
happen.
But
if
the
test
succeeds,
then
the
analysis
succeeds,
the
experiment
succeeds
and
the
role
will
proceed.
B
B
One
one
thing:
what
we
notice
is
that
continuous
diplomas,
typically
equal
or
equate
synchronous
workflows
like
you,
have
to
take
each
commit
and
promote
it
from
dev
to
stage
to
prod
and
so
on
and
so
on.
So
this
becomes
a
problem
in
some
cases
like
if
you
have
a
large
team
working
on
a
repo,
and
maybe
you
have
a
high
number
of
of
changes.
So
maybe
you
have
a
slow
rollout
to
production.
B
You
know
in
order
to
minimize
risk-
or
maybe
you
just
have
a
long
deployment
time
for
whatever
reason-
and
you
can't
you
know
drastically
reduce
it.
These
are
all
actual
use
cases
from
adobe
as
well
and
it
becomes
problematic,
because
if
you
use
this
sort
of
synchronous
workflows,
then
your
developers
will
have
to
sort
of
wait
for
all
their
changes
to
be
deployed
until
their
new
changes
are
deployed.
B
B
It
runs
on
a
schedule,
and
the
difference
is
if
you
want
to
have
an
actual
release
as
a
button,
then
you'll
want
to
trigger
it
manually.
But
if
you
are
just
plagued
by
the
problem
of
having
too
many
commits
and
in
queueing,
then
maybe
you
just
want
to
release
on
a
schedule
like
you
want
to
release
every
three
hours
or
once
a
day
or
maybe
once
a
week.
I
guess
it
depends.
B
B
And
finally,
besides
the
pros
and
cons,
I
guess
one
two
other
things
I
would
like
to
add.
I
think
that
if
you're
using
the
synchronous
commit
promotions
like
continuous
deployments,
what
you
can
do
is
even
extend
it
to
in
case
of
failure
in
case
of
rollbacks.
You
can
actually
extend
it
to
roll
back
the
code
that
was
pushing
the
application
repository
and
that
caused
the
failure.
B
That's
one
thing,
and
a
second
thing
is
that,
if
you're,
using
like
continuous
delivery
and
you're
using
this
asynchronous
state
promotions,
like
you
deploy
from
time
to
time
from
stage
or
pre-proc
to
production,
then
because
you're
pushing
more
changes
like
if
you're
pushing
one
to
ten
commits
if
you're
pushing
all
of
those
to
production
at
once,
and
you
have
a
failure.
You'll
have
a
bit
of
a
harder
time
to
figure
out
what
commit
was
really
the
problem.
B
So,
just
to
do
a
quick
recap:
what
we
noticed
really
early
on
when
trying
to
or
adopting
argo
was
that
we
needed
to
create
promotion
and
rollback
and
bake
it
into
our
workflows.
Two
that
you,
you
know.
We
feel
that
you
have
to
take
the
deployment
configuration
changes
through
the
same
diligence
and
process
as
the
application
code
changes
three,
that
pr
preview
environments
are
really
helpful,
they're
pretty
easy
to
set
up
and
they
help
you
to
identify
and
resolve
failures
long
before
they
are
even
reaching
the
main
branch.
B
Fourth,
that
wave
deployments
can
actually
help
you
to
minimize
the
impact
of
eventual
failures
in
your
production,
rollouts
and
five
that
you
know
you
can
use
progressive
deployments
in
very
with
very
standard
ways
like
aggro
rollouts.
Is
I
really
like
that
recommendation?
It's
really
well
documented.
You
can
use
analysis
pretty
easily
and
you
can
use
that
to
sort
of
easily
roll
out
changes
to
to
your
users
and,
finally,
that
you
know
a
synchronous
day.
Promotions
are
sometimes
a
good
way
to.