►
From YouTube: Intuitive Progressive Delivery Across Microservices in a Depe... Hari Kumar Rongali & Rohit Agrawal
Description
Intuitive Progressive Delivery Across Microservices in a Dependency Graph Using Argo Rollouts. - Hari Kumar Rongali, Intuit & Rohit Agrawal, DataBricks
Progressive Delivery as you all know is controlled deployments to minimize the risk associated with service updates. Organizations use tools like ArgoRollouts to achieve Progressive delivery for individual services. Progressive delivery across microservices with dependencies is a challenge that many companies are working to solve. In this session, We will demonstrate how to use Argo Rollouts in achieving progressive rollouts & automated rollbacks across microservices in a dependency graph using capabilities such as automated analysis & custom webhook notifications. We will also discuss how to use your current complex multi-service monitoring queries in analysis and automate the entire deployment flow across microservices. Another important capability that we will present is ‘dry runs’ that will enable users to identify and validate possible automated rollbacks in production without actually doing rollbacks.
A
B
B
Let's
talk
a
little
bit
about
data
breaks,
so
we
are
a
unified
analytics
platform
helping
our
customers
do
get
insights
out
of
the
data
we
did
800
billion
in
arr
last
year
and
have
seen
like
75
year-on-year
growth.
We
have
offerings
across
all
three
major
clouds:
aws
azure
and
gcp.
We
are
an
active
user
of
our
go
project.
B
A
Intuit
found
in
1983
and
evolved
our
sins
for
the
past
few
decades,
currently
at
12,
plus
billion
dollars
of
revenue
and
100
million
plus
customers,
primarily
a
phonetic
software
with
well-known
products
like
turbotax,
quickbooks,
mint
grade,
karma
mailchimp
and
so
forth.
Intuit
inverts,
around
2
billion
plus
in
r
d,
and
it's
very
tech
heavy
with
several
well-known
open
source
project
coming
out
of
it,
and
one
of
them
is
argo.
A
Now
on
track
to
graduation
as
argo
evolved,
there
are
multiple
projects
come
out
of
it.
Solving
varied
use
cases
and
some
of
the
products
are
argo
workflows,
which
is
a
container
driven
workflow
engine
with
use
case
that
solves
batch
processing
and
map
reduce
and
some
of
the
domains
like
air,
modeling
and
machine
ending
topics.
A
B
Let's
take
a
minute
to
recap:
what
progressive
delivery
means?
Progressive
delivery
is
built
on
top
of
continuous
delivery,
with
more
features
like
gradual
rollouts,
clearing
a
b
testing
matrix
analysis.
In
a
nutshell,
is
these
four
things
releasing
of
a
product
in
a
controlled
manner,
having
complete
control
over
the
blast
radius
during
the
updates
supporting
bring
your
own
matrix
analysis
and
then
finally
having
automated
promotions
or
rollbacks
as
per
the
kpis,
let's
see
how
progressive
delivery
in
a
micro
microservice
ecosystem
works,
we
have
a
stable
version
for
a
service
called
v1.
B
We
are
trying
to
roll
out
a
new
version,
say
v2.
We
do
canary
we
collect
and
analyze
the
matrix.
If
everything
is
looking
good,
we
mark
the
version
to
a
stable.
If
not,
then
we
roll
back
to
version
one
by
shifting
all
the
traffic
back
to
back
to
the
stable
version,
and
then
we
maybe
trigger
a
notification
saying
that
your
new
version
rollout
fails.
B
Now,
let's
see
what
progressive
delivery
looks
like
with
argo
rollouts,
we
have
a
controller
which
interacts
with
the
custom
crd
called
rollouts.
We
have
an
analysis
template
which
contain
a
set
of
rules
or
steps
on
how
to
shift
traffic
and
perform
analysis.
We
have
a
stable
replica
set
that
you
see
at
the
bottom.
Now
when
we
are
trying
to
release
a
new
version
of
the
service,
the
controller
will
create
a
new
replica
set
for
the
canary
parts.
B
It
will
also
create
an
analysis,
run
object
which
will
analyze
all
the
matrix
will
start
gradually,
shifting
traffic
from
stable
to
canary,
depending
on
the
rules.
Both
this
table,
as
well
as
the
canary
versions,
are
emitting
prometheus
matrix
analysis
run,
is
consuming
these
matrix
to
make
decisions
and
based
on
whether
the
analysis
is
successful
or
not.
It
will
either
shift
more
traffic
from
stable
to
canary
or
it
will
roll
back
to
stable
version
and
then
we'll
just
mark
the
mark,
the
canary
as
degraded.
You
can
also
integrate
it
with
service
mesh.
A
A
A
We
can
read
through
it,
sorry
for
masking
some
of
them
for
the
confidentiality
and,
if
you
read
one
of
the
snippet
one
of
the
services
changed
because
of
a
change
induced
in
or
which
is
not
communicated
ahead
to
the
impacted
users,
because
and
the
impact
is
the
login
failed
and
what
they
did
is
restore
the.
What
are
the
impacted
change
to
your
previous
state?
That
is
a
stable
state,
and
these
are
just
a
few
of
them.
A
We
are
using
rollouts
and
approximately
capability
to
reduce
this
fdtr,
but
let's
see
how
we
can
do
in
services
that
are
in
a
graph
model
right
to
move
forward.
Let's
look
at
some
simple
graph.
Let's
say
there
are
four
services:
s1
s2,
s3,
s4
and
they're
in
state
v1
state,
there's
a
stable
one.
Let's
say
there
is
an
induced
change
in
s4
and
because
of
the
change,
some
hypothetically,
let's
say
the
service
one
got
impacted.
A
What
ideally
needs
to
happen
is
it
needs
to
roll
back
to
the
previous
state
so
that
everything
needs
to
be
done
automatically,
usually
in
companies
like
institute
and
all
there
are
some
cool,
coordinated
teams
or
integration
teams
which
work
with
all
the
services
within
that
dependency
graph.
They
relays-
and
they
monitor,
know
maybe
looking
at
the
dashboards
and
everything
one
of
the
main
thing.
Or
is
there
a
lot
of
manual
steps
involved?
A
If
there's
an
issue,
and
in
the
next
slide,
I'm
going
to
show
how
or
use
our
orgo
ecosystem
projects
and
create
an
architecture
or
a
flow
to
solve
this
current
use
current
problem,
let's
read
through
from
left
to
right
down
and
left
so
start
with
a
change
in
s4
and
what
happens
is
a
change
in
s4
triggers
a
canary
and
run
the
analysis
if
it
is
successful,
move
forward
or
if
it
is
failure?
Roll
back,
as
we
mentioned
earlier,
however,
hell
or
rollout
or
positivity
happens
in
a
microservice.
A
This
is
usually
a
capability
that
is
provided
by
rollouts.
Now,
let's
say
it
is
successful.
Now
it
should
go
to
or
trigger
notifications
as
an
events
to
the
downstream
dependent
services,
and
we
are
using
argo
events
to
capture
those
events
and
that
one
integrates
with
workflows
what
workforce
can
bring
in
is
creating
a
batch
of
jobs
for
each
of
these
s1
s2
and
s3.
As
you
seen
here,
each
s1
sd
bad
jobs
are
again
jobs
that
run
kpis
to
measure
these
metrics
against
these
services.
A
A
A
Okay,
thank
you
in
full
screen.
Okay,
so,
as
you
see
here
here,
I'm
showing
an
argo
cd
which
is
running
locally
with
four
services
roads
remove,
which
is
where
I'm
using
a
change,
and
there
is
three
other
services
which
I
am
trading
as
a
dependency
once
and
there
is
a
rollout
and
workflows
also
deployed-
and
this
is
the
workflow
ui,
where
there
are
no
workflows
and
let's
go
through
in
detail
about
what
demo
has
and
what
are
the
specifications
and
also
we'll
I'll
talk
about
what
are
the
templates
that
are
created
in
this?
A
So,
if
you
see
here,
this
is
the
notification
which
is
introduced
to
send
the
notifications.
So
let's
look
more
detail
in
terms
of
what
world
patch
has
so
on
download
completed.
Send
their
note
questions
to
the
downstream
services
which
is
again
captured
by
workforce,
and
the
next
thing
is
the
traffic
routing
using
istio,
and
there
is
the
image
that
is
the
yellow
this
car
and
let's
look
at
the
workforce
as
well.
A
Here
I
have
a
workflow
template
that
is
added,
and
here
in
this
template
I
have
a
series
of
jobs
that
are
defined,
which
actually
runs
this
analysis,
and
if
there
is
issue
whether
it
is
successful
value,
an
action
is
defined
to
roll
back
or
roll
forward.
As
a
part
of
this
workflow
template,
let's
look
at
okay
again,
there
is
no
workflows.
Only
the
template
is
binding
together.
A
Now,
let's
look
at
now:
let's
introduce
a
change
in
downloads
demo,
so
in
the
current
rollouts
demo,
it's
a
yellow,
so
I'll
show
you
the
rendering
part
where
it
should
have
a
yellow.
Color
is
rendered
just
for
the
demo
purpose
and
I'll
show
you
I'll
change
it
to
blue.
So,
as
I
said
earlier,
it's
a
good
ops
model.
So
argo
cd
has
a
git
ops
delivery
system.
A
It
will
recognize
the
github's
change
and,
if
you
refresh
it
has
the
capability
to
show
the
out
of
sync
from
the
user
interface
again,
teams
use
auto,
auto
syncs
and
everything
which
are
again
some
of
the
capabilities
argo
city
already
provides.
But
right
now,
ethernet
at
least
we
make
sure
like
everything
is
pipeline.
Driven
and
as
you
see
it
is
a
yellow
and
deploy
is
the
only
change
when
I
sync
it.
A
There
is
a
traffic
routing
in
the
virtual
service
and
the
analysis
run
is
also
running
and,
as
you
see,
the
blue
and
yellow
are
rendered
in
this
graph,
and
these
are
our
services,
where
the
other
colors,
like
purple,
red,
yellow,
are
defined
just
for
the
demo
purpose,
without
not
much
of
meaning
and
as
we
are
care
about
once
this
microservice
is
successful.
What
happens
after
that
right?
So
that
is
where
I
think
we
can
use
promote
for
to
move
forward
just
to
skip
this
and
all
analysis
still
no
workflow.
A
So
let's
do
the
workflow
or
promote
full
so
that
we
can
capture
is
a
robot
completed.
We
should
automatically
trigger
these
notifications
and
it
will
be
captured
by
the
event
and
for
the
workforce.
So,
as
I
did,
it
is
completed,
there
is
a
workflow
is
created.
As
you
see,
these
are
the
batch
of
jobs
and
if
you
see
the
template
also,
there
is
a
okay.
So
here
what
I'm
doing
is
I'm
producing
an
error
rate?
A
I
want
to
show
there
or
an
impact
because
of
the
change
in
roads
demo
and
what
you're
seeing
the
jobs
I've
configured
for
one
minute,
but
you
can
confirm
for
days
or
even
hours,
based
on
the
use
case,
you
can
run
this
kappa
kps
against
it,
and
if
there
is
an
issue
happens,
what
happen?
Let's
get
a
second
or
so
so
it
will
be
completed,
and
if
there
is
a
change
happens,
what
actually
happen
is
you
need
to
revert
based
on
the
change
history
to
the
stable
state
once
all
those
jobs
are
done?
A
So
if
we
give
a
second
or
so,
this
change
will,
once
the
rollback
is
completed,
you'll
see
smooth
transition
from
okay.
Rollback
is
completed
here
and
you'll
be
able
to
see
this
change
in
a
second
yeah.
So
you
saw
that
bluetooth.
Yellow
is
changed
because
we
triggered
our
go
cd
apis
to
call
move
to
the
previous
state
because
we
saw
change
or
something
impacted
in
the
downstream
services.
Let's
go
back
to
the
argo
ui
and
we'll
be
able
to
see
if
we
refresh
you'll
see
this
diff.
A
B
All
right,
let
me
talk
a
little
bit
about
production
readiness,
so
one
of
the
pain
points
we
had
at
databricks
was
on
configuring,
the
right
thresholds
or
right
set
of
matrix
so
that
it
doesn't
trigger
any
false
positives.
We
recently
added
a
new
feature
in
our
rollouts
called
dry
runs,
which
allows
you
to
mark
a
set
of
matrix
running
in
the
driver
mode.
These
dry
runs
can
help
engineers
to
add
new
matrix
to
their
existing
rollouts
object
without
worrying
about
the
risk
of
impacting
their
current
production
workloads.
B
It
would
only
emit
matrix
but
then
have
no
impact
on
the
final
state
of
the
rollouts.
Here
is
a
quick
example.
If
you
are
able
to
see
this
is
how
you
can
add
more
matrix
to
the
driven
mode.
Matric
name
here
is
a
regex,
and
you
can
specify
how
many
matrix
you
want
to
run
in
the
driver
mode.
It
would
not
impact
any
final
state
of
the
rollout
it
would
just
emit
the
matrix
while
making
some
of
these
matrix
running
the
driven
mode.
B
You
also
want
to
capture
all
the
data
points
and
not
just
the
top
10.
before
or
prior
to
version.
1.2
of
rollouts,
it
would
only
capture
the
latest
10
measurements
and
then
would
garbage
collect
everything.
So
we
also
added
a
new
feature
for
measurement
retention
with
which
you
can
customize
how
many
measurements
you
you
want
to
retain.
You
can
retain
like
100,
you
can
retain
200,
you
can
retain
everything
which
is
getting
emitted.
This
is
it's
a
very
similar
idea.
You
just
like
have
a
measurement
retention.
B
You
specify
the
metric
name,
and
then
you
specify
the
limit
like
how
many
measurements
you
want
to
retain.
We
are
working
on
adding
more
observability
advanced
observability,
so
you
can
just
like
export
all
the
analysis
results
in
in
s3
bucket,
maybe
in
file
system,
and
then
do
any
post
roll
out
analysis.
If
you
want
to
you,
can
read
about
it.
We
just
have
a
very
nice
blog
published
in
version
1.2.
If
you
want,
you
can
read
about
like
dry,
runs
and
measurement
retention
on
the
blog.
B
Let
me
talk
a
little
bit
about
what's
next,
so
we
have
these
cool
features
coming
in
in
the
new
version
of
fargo
rollouts,
there
are
dry
runs,
which
is
currently
a
service
level.
We
want
to
expand
it
to
the
dependency
level
to
get
more
confidence.
We
have
graph
generation
based
on
the
traffic
flow.
We
will
be
adding
anomaly
detection
with
intuitive
scoring.
We
are
very
excited
about
the
traffic
mirroring.
You
can
mirror
traffic
to
a
set
of
up
streams
without
impacting
the
final
user.
B
D
C
B
Right
so
with
dry
runs.
What
you
can
do
is
like,
if
you
want
to
run
the
analysis
and
you
don't
know
what
is
the
current
threshold
like
you're
you're,
trying
to
add
more
rules?
But
then
you
have
no
idea
what
the
threshold
looks
like
you
can
run.
Those
matrix
in
dry
run
to
collect
more
data,
and
then
you
can
fine-tune
your
matrix
like.