►
From YouTube: Automated Progressive Delivery Using GitOps and Service Mesh - Yasen Simeonov & Henrik Blixt, Intuit
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Automated Progressive Delivery Using GitOps and Service Mesh - Yasen Simeonov & Henrik Blixt, Intuit
Intuit is rolling out progressive delivery together with service mesh, using metrics from the mesh to automate analysis on the health of a new release. This is our next step towards more sophisticated techniques like AI/ML-driven analysis. As a FinTech Technology company with tens of millions of users and thousands of services, with many being very seasonal, we were facing a wide variety of service performance and traffic patterns that made patterning analysis templates challenging. This session discusses and demos how Intuit leverages Argo Rollouts in combination with Envoy and Service Mesh to provide automatic and declarative progressive delivery. Attendees will learn about the challenges we faced, our learnings and work we had to overcome those challenges Intuit has been heavily involved in building tools for progressive analysis with Argo Rollouts and we are now rolling this out at large scale together with service mesh. This is the story on how and what worked in production.
A
A
B
So
we're
quickly
today
going
to
talk
a
little
bit
about
how
we
use
service
mesh
and
how
it
looks
at
intuit,
I'm
going
to
talk
a
bit
about
progressive
delivery.
Why
we
use
it,
how
we
use
it
and
then
I'm
going
to
talk
about
how
well
they
work
together
or
how
well
they
didn't
work
together
and
we
kind
of
what
we
experienced
and
realized.
As
we
tried
to
use
these
together
a
little
bit
look
into
the
future.
B
What
we
think
is
coming
next,
what
are
the
big
focus
areas
for
us
and
then
we're
going
to
round
off
with
jason
doing
a
live
demo
so
very
quickly,
but
into
it
since
we're
in
europe,
not
too
many
of
you
probably
know
who
we
are,
because
we
have
95
of
our
business
in
in
the
us,
but
we're
one
of
the
leading
u.s
fintech
companies.
We
have
roughly
10
billion
dollars
in
in
revenue.
B
What
you
probably
know
is
more
for,
if
you
knew,
is
that
our
the
argo
project
came
out
of
intuit,
where
it
came
out
of
a
startup
that
we
acquired
and
then
we
later
donated
argo
to
the
cncf,
we're
big
supporters
of
open
source,
not
just
argo,
but
a
number
of
other
projects
that
we
contribute
to
that.
We
use
basically,
if
it's
in
this
cnc,
but
you've,
probably
all
seen
that
massive
map
right.
If
it's
on
that
map,
we
pretty
much
use
it
and
we
contribute
to
many
of
them
as
well.
A
Yeah
before
going
to
this
slide,
I
just
out
of
curiosity
how
many
of
you
are
here
because
of
the
service
mesh
part
can
how
great
yeah
great
and
how
many,
because
of
the
progressive
delivery
in
targo
whoa
great,
okay,
excellent.
I
think
this
slides
shows
our
scale
and
it
is
important
for
the
service
mesh
part,
because
you
can
see
that
we
have
around
more
than
230
kubernetes
clusters
and
the
service
measure
that
I'm
going
to
talk
about
in
a
moment
stretches
across
all
those
clusters.
A
A
A
So
what
we
did
in
the
past,
we
have
this
api
gateway
that
you
see
on
the
top
and
service
a
cannot
talk
to
service
b.
We
apply
kubernetes
network
policy,
that
services
can
talk
only
to
the
api
gateway,
and
this
is
where
we
apply
all
the
security,
all
the
tls,
everything
that
that
is
needed
and
all
advanced
traffic
management.
A
However,
on
my
slides,
the
api
gateways
are
just
a
circle.
However,
just
for
your
information
in
the
peak
period,
we
process
more
than
a
million
transactions
per
second.
So
in
reality
it
is
not
just
a
circle.
It
is
like
a
huge
set
of
replicas
of
the
api
gateway
and
yeah.
It
is
a
complex
environment.
A
So
for
the
services
we
add
this
envoy
proxy,
and
now
the
traffic
can
flow
between
the
services
directly
from
service
a
to
service
b
without
hooking
to
the
api
gateway.
So
when
you
see
at
this
point,
all
services
have
two
endpoints.
The
initial
endpoint
is
the
public
endpoint
through
the
api
gateway
and
service
b
has
like
service
b,
dot
api.intuit.com,
but
service
b.
Now
because
it
is
mesh,
enabled
has
second
endpoint,
which
is
service
b,
dot
mesh,
for
example,
and
the
client
of
service
b.
In
this
particular
example
service
a
can
decide.
A
I
think
many
of
you
think
that
the
envoy
proxy
at
latency,
in
our
case
it
is
completely
opposite,
because
we
don't
have
to
hook
the
traffic
to.
It
is
a
completely
different
vpc,
completely
different
account
for
the
api
gateway.
Now
the
traffic
stays
either
local
in
the
cluster
or
goes
directly
from
coaster
to
cluster.
We
see
more
than
30
percent
latency
improvement
on
the
service
to
service
communication.
A
A
A
We
significantly
reduce
our
public
cloud
provider,
cost
with
this
and
yeah
so
one
hopeless
for
the
communication.
Also,
we
were
rotating
certificates
and
keys
for
the
services
in
order
to
communicate
with
the
api
gateway
manually
every
90
days
right
now
with
service
mesh.
It
is
a
matter
of
hours
and
it
is
fully
automated,
so
developers
don't
have
to
rotate
certification
keys.
A
Also
in
the
past,
when
the
traffic
goes
through,
the
api
gateway,
the
service
owner
can
can
have
observability
from
the
api
gateway
to
to
their
service.
Right
now
we
have
transactional
visibility.
We
can
see
the
entire
chain
from
service
a
to
service
b
to
service
c.
We
have
this
observability,
so
this
is
overall
our
architecturalist.
A
A
A
So
this
is
for
the
for
the
data
path
for
the
control
plane.
We
have
issue
deeper
cluster,
so
control,
plane
per
cluster
and
on
the
right
side
you
can
see
the
certificate
authority
we
can
see.
You
can
see
the
admiral
which
I'm
going
to
talk
about
in
a
moment
and,
of
course,
integration
with
targo.
So
this
is
more
central
component.
Is
to
these
bird
clusters
just
want
to
mention
about
admiral.
A
If
you
remember,
we
have
service
a
service
b
service
c.
However,
those
services
may
be
in
different
kubernetes
clusters
and
if
you
remember,
we
have
issued
per
cluster,
so
we
have
to
find
a
way
how
to
enable
service
discovery
across
the
clusters,
and
admiral
is
another
open
source
project
that
intuit
donated
to
easter
ecosystem.
Now
we
teased
you
going
to
cncf,
hopefully
admiral
also
will
go
to
cncf
and
admiral
is
responsible
to
synchronize
the
endpoints
to
do
service
discovery
across
the
clusters.
A
Just
one
example
what
we
did
initially
for
our
developers,
we
let
them
menu
manually,
onboard
their
services
to
service
mesh.
They
go
to
the
dev
portal,
they
click
enable
mesh
and
we
send
them
prs
with
the
istio
label
and
their
services
on
mesh.
However,
what
we
did
recently
we
made
this
day
zero
experience.
If
our
developers
create
a
new
service,
we
are
not
asking
them
so
the
service
to
service
communication
always
happens
through
damage,
so
api
gateway
is
only
for
external
communication.
B
B
Oh,
it's
a
half,
maybe
so
I'm
going
to
give
you
a
teaser
on
why
progressive
delivery
is
really
cool.
Why
we
use
it
into
it.
If
you
want
to
know
more
about
progressive
delivery,
please
come
by
the
the
argo
booth
over
in
the
project,
pavilion
and
me,
or
one
of
the
maintainers
would
be
happy
to
give
you
as
much
detail
as
you
would
possibly
want.
B
So
one
of
the
main
reasons
for
using
progressive
deliveries
at
internet
we're
from
the
platform
team.
We
build
the
developer
experience
platform
for
our
5
000
developers.
So
one
of
the
main
reasons
for
this
is
increasing
operational
excellence
because
we
also
run
all
the
services
you
know
for
all
intuits,
almost
100
million
customers.
B
B
When
that
happens,
we
want
to
make
sure
that
we
minimize
the
impact
from
that
change,
but
using
progressive
delivery
we
can
actually
roll
out.
You
know
your
new
canary,
your
new
stack
to
subset
of
users.
Do
some
testing
increase
the
number
of
users
that
see
that
canary
we
actually
reduce?
You
know
the
blast
radius
from
anything
that
might
go
wrong.
B
The
other
one
is
when
something
happens.
It
takes
a
while.
You
know
something
goes
wrong:
a
user
finds
out,
they
call
back.
Some
developer
gets
woken
up,
and
you
know
you
start
trying
to
troubleshoot
what's
going
on
and
then
eventually
figure
out
what
the
heck
went
wrong
and
then
you
start
fixing
it
right
and
that
whole
process,
I
think
most
of
us
have
been
through
that
takes
a
while
and
that
just
leads
to
the
long
mtgr
along
mean
time
to
recovery.
B
So
by
using
the
automation
that
progressive
delivery
offers,
we
can
increase
the
reliability
of
the
process
and
also
reduce
the
time,
because
we
can
automate
a
lot
of
that.
So
if
something
happens
during
one
of
those
steps
would
just
automatically
roll
back.
So,
instead
of
putting
putting
everything
out,
there
wait
until
something
happens
and
user
calls
in
will
just
automatically
run
a
bunch
of
tests
if
they
fail
boom,
we
roll
back
right
away
so
that
just
helps
us
troubleshoot
or
we
don't
even
have
to
troubleshoot.
B
It
just
gets
us
back
to
a
known
good
state,
much
faster,
the
other
one.
The
other
thing
we
noticed
when
we
talked
to
some
developers
was
that
they
they
spend
quite
a
lot
of
time,
figuring
out.
If
they
change
the
mate,
if
a
release,
they
did
it's
actually
healthy,
they
build
their
own
custom
dashboards.
B
They
look
at
logs,
they
look
at
other
dashboards
and
they
just
sit
there
babysitting
their
their
applications
to
make
sure
that
they're
actually
good,
and
they
spend
a
fair
amount
of
time
doing
that
so
by
automating
this
process,
building
more
reliability
and
dependability
on
this
process.
We
can
free
up
time
for
these
developers,
so
they
don't
have
to
babysit
and
make
sure
that
everything
is
working
because
we're
doing
that
automatically
right.
So
they
can
deploy.
Rest
assured
that
the
the
automated
tests
fill
the
function
of
the
dashboard
and
then
they
can.
B
B
B
Once
we
fixed
that
in
order
rollouts
and
actually
now
support
that,
we
realized
that
it's,
it's
actually
pretty
hard
to
figure
out
what
metrics
to
use.
What
aggregations
to
look
at
when
you
figure
out
the
health
of
your
application.
So
now
we're.
Actually
the
goal
is
actually
instead
to
to
make
sure
that
we're
using
service
measure
everything.
B
So
even
if
this
is
actually
a
supported
way
in
in
rollouts
now
it
actually
makes
it
harder
for
the
service
teams
to
figure
out
how,
if
you're,
using
the
http
request,
metrics,
which
one
are
actually
the
ones
that
tell
me
a
better
story
on
application
health.
So
we're
looking
at
using
service
mesh
for
everything,
because
then
we
have
a
single
aggregation
point
through
service
mesh.
B
Another
thing
we
ran
into
was
that
we
were
using
the
admiral
tool
that
you
also
mentioned
earlier,
and
it
actually
creates
the
endpoints
as
an
abstraction
over
the
the
kubernetes
service
names.
Unfortunately,
our
rolex
depends
on
the
community
service
names,
so
that
was
something
you
know.
We
realized,
as
we
were
starting
to
test
is
like
hey
this.
This
is
probably
not
working,
so
we
had
to
fix
that.
It's
now
fixed
and
working.
B
But
we're
now
looking
at
two
migrations
here:
I've
been
talking
about
service,
mesh
and
progressive
delivery,
and
we
have
2
000
services
that
were
there
before
we
made
this
a
day,
zero
experience
and
telling
them
to
do
not
one
migration,
but
two
didn't
really
raise
that
much
excitement.
So
we
spent
a
lot
of
time
like,
like
jason,
said
as
well,
making
it
a
super
simple
experience
like
in
a
developer
portal.
There's
a
drop
down.
B
I
don't
know
if
you
can
see
that
maybe
the
first
two
rows
came,
but
for
you
guys
in
the
back,
there's
a
drop
down
that
says
enable
mesh
and
another
on
the
other
one
drop
and
it
says,
migrate
to
progressive
delivery.
So,
just
by
using
going
into
a
dev
portal
drop
down
click,
you
can
move
your
service
over
to
progressive
delivery
or
or
service
mesh.
B
We've
also
run
a
bunch
of
programs
internally
we're
calling
our
fixed
day
or
game
days
where
we,
basically,
you
know,
put
up
a
challenge
with
teams:
the
team
that
migrates
the
most
services
during
the
day.
You
know
they
get,
they
get
a
price
and
you
check
checkpoints
every
hour
and
just
kind
of
make
it
a
competition
and
just
encourage
the
the
teams
to
do
this.
B
Migration,
instead
of
coming
in
you
know,
with
a
big
hammer
like
thou,
shall
migrate,
we're
just
trying
to
to
spur
them
to
do
it
organically
instead
also
super
important
stopping
the
bleeds.
The
first
thing
we
did
was
the
day
zero
experience
and
then
the
migration
to
make
sure
that
we're
not
just
on
boarding
more
services
that
we
don't
have
to
market.
So,
let's,
let's
stop
the
bleed
first
and
then
you
know
we
can
migrate.
What's
what's
left
in
the
in
in
the
the
infrastructure.
B
B
We
have
other
services
that
are
more
geared
towards
small
business,
so
they
run
all
year,
but
maybe
peak
around
you
know
black
friday
or
not.
So
we
have
big
differences
between
the
traffic
that
comes
into
these
services.
Some
have
just
the
same
amount
of
traffic
all
year.
Some
have
almost
all
the
traffic
in
two
weeks
and
then
there's
almost
nothing.
So
if
you
want
to
do
an
http
based
request,
http
request
based
analysis.
B
B
B
So
traffic
mirroring
traffic
shadowing
traffic
cloning,
whatever
you
want
to
call
it
basically
doing
a
a
a
new
path
where
you
send
off
a
certain
amount
of
production
traffic,
to
a
stack
that
your
users
aren't
on
right,
so
that
you
can
test
those
two
stacks
against
each
other
without
really
affecting
any
users.
So
we're
looking
at
how
we
can
use
that
for
for
argo,
rollouts
argo
rollouts
can
do
the
canaries
based
on
on
pods
and
just
do
do
weight
basically
or
we
can
do
percentage
traffic.
You
know
10
traffic
scale
up
to
30.
B
B
B
So
if
you
can
use
ml
to
figure
out
an
anomaly
score
or
a
good
measure
of
how
the
application
is
doing,
that's
a
much
better
way
of
and
will
give
engineers
even
you
know,
greater
trust
in
the
system.
If,
if
it's
healthy
or
not,
and
you
can
take
that
even
a
step
further,
why
stop
at
the
progressive
delivery?
You
know
if,
if
you
can
figure
out
when
your
application
has
an
anomaly,
why
wouldn't
you
want
to
run
that
all
the
time
right?
B
It
might
not
be
part
of
the
the
delivery
process,
but
it
might
be
something
might
happen
once
you've
fully
promoted
100.
What
about
something
happening
two
days
down
the
line
right,
you
would
still
want
to
hey
something
anomalous
happened.
I
want
to
roll
back,
so
that's
also
something
that
we're
looking
into.
How
can
we?
A
A
A
And
these
two
version
is
the
latest
one
that
I
think
is
available
right
now,
13.3.
I
will
go
back
to
this
in
a
moment.
Meanwhile,
I
would
like
to
show
the
application
that
I'm
going
to
deploy
very
simple,
so
this
is
version
one
that
I
will
deploy
now
initially
and
version
1
just
says:
hello:
valencia,
that's
it
once
this
is
done.
I
will
I
will
deploy
version
2
and
I
artificially
added
some
errors
so
other
than
I
added
hello,
valencia,
with
an
error,
and
actually
I
sent
the
http
500
statoscope.
B
A
A
A
So
what
I
have
in
the
row
out
here,
so
you
can
see
the
strategies
canary
and
taiwan.
I
want
dynamic
analysis
and
the
canary
service
name.
Is
this
one
and
the
stable
service
name?
Is
this
one
and
the
progress
will
happen
like
this?
10
percent
of
the
traffic
will
be
directed
to
the
canary
version
and
we
will
wait
for
one
minute
to
make
analysis.
A
Then,
if
everything
is
good,
you
progress
to
the
30
percent,
we
will
wait
for
30
seconds,
then
the
half
of
the
traffic.
You
wait
for
another
another
30
seconds
and
you
see
that
we
are
doing
traffic
router
routing
with
istio
and
we
modify
the
virtual
service.
I
will
actually
show
you
the
virtual
service
in
a
moment,
so
I
will
deploy
the
second
version.
A
A
A
A
A
A
C
A
Yeah
because,
if
I
add
one
more
component
here,
it
will
take
more
time
so,
if
I
add
a
targo
cd
and
send
prs
to
github
with
you,
that's
one
more
component
that
cannot
latencies.
I
just
wanted
to
focus
on
argo,
rollouts
and
service
mesh.
Otherwise,
in
reality
we
are
using
cargo
cd,
yes
yeah.
Another
question
down.
E
B
I
mean
you,
for
example,
for
the
for
the
analysis
templates.
You
can
basically
use
and
anything
that
can
be
served
as
a
metrics.
I
mean
we,
we
go
to
prometheus,
so
basically
anything
that
you
can
serve
as
a
metric
through
for
me
that
you
can
use
to
base
your
analysis
templates
on.
Oh,
that's,
cool!
Thank
you.