►
From YouTube: Automate Your SLO Validation with Prometheus & Flagger - Sanskar Jaiswal & Kingdon Patrick Barrett
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Automate Your SLO Validation with Prometheus & Flagger - Sanskar Jaiswal & Kingdon Patrick Barrett, Weaveworks
Flagger is a progressive delivery tool that automates the release process for applications running on Kubernetes, by gradually shifting traffic to the new version while validating metrics and running conformance tests. Prometheus is a great monitoring and observability tool. Another way to look at it, is that it’s an easy way of validating Service Level Objectives(SLOs) and Key Performance Indicators(KPIs). Flagger leverages this feature of Prometheus by ensuring that the new version of their application behaves as desired, letting developers be confident about their code, before exposing it to users. In this talk, we will discuss about how Flagger utilizes Prometheus to automate canary deployments using progressive delivery. Furthermore, we will see how Flagger integrates with KEDA ScaledObjects which lets you scale your workloads during a canary analysis using a Prometheus scaler to simulate real world conditions.
A
A
Okay,
okay,
so
how
many
of
you
have
implemented?
Progressive
delivery
clusters,
or
you
know,
played
around
with
Progressive
delivery
like
can
I
get
a
show
of
hands?
Okay,
so
not
not,
as
many
people
are
expecting.
Actually
so,
for
those
of
you
don't
understand
what
Progressive
delivery
is.
Let
me
try
and
convince
you
or
that
you
should
be
practicing
Progressive
delivery
in
your
clusters,
because
it's
a
really
really
good
thing
to
do
so.
Progressive
Derby
is
the
art
of
producing
new
software
into
your
clusters
in
a
safe
and
iterative
manner.
A
Basically,
what
you
do
is
when
you
want
to
introduce
new
software
into
your
cluster
and
expose
it
to
your
end,
try
and
get
very
fine
gained
control
over
the
so
as
soon
so
as
to
if
the
new
version
is
not
working
as
expected,
if
there
are
or
if
it
breaks
features.
The
number
of
users
that
are
impacted
are.
A
A
You
need
some
kind
of
a
Ingress
or
like
a
load
balance
solution
and
which,
in
our
case,
is
a
service
mesh
or
something
like
Ingress,
nginx
right
so
and
most
importantly,
for
those
of
our
talk,
you
need
a
good
observability
stack
right,
something
which
you
can
use,
get
metrics
and
validate
your
slos
and
see
whether
kpis
are
being
hit
for
so
the
purpose
of
this
thought
census
problem
days,
we'll
use
Prometheus,
but
you
can
use
anything.
You
want
really.
A
A
So
you
have
two
Services,
your
new
service
and
your
old
service,
and
you
shift
traffic
from
your
old
service
to
your
new
service
in
a
gradual
grade
based
manner.
Will
this
is
the
main
topic
of
our
talk?
I'll
just
briefly
describe
the
other
two
as
well,
so
the
next
is
Alpha
Beta
Testing
I'm
guessing
this
is
more
popular
amongst
you
guys
it's
a
pretty
old
way
of
testing,
so
you
have
some
beta
users
and
the
way
you
differentiate
between
those
data
users
is
because
they
have
the
traffic
that
they
generate.
A
The
request
that
they
have
are
gonna
have
head
on
it
and
you
use
headers
to
differentiate
between
your
beta
users,
and
the
last
is
blue
game
deployments.
Blue
game
deployments
is
a
bit
different
than
the
other
because
it
doesn't
actually
expose
any
new
traffic
to
your
users,
so
the
new
version
is
not
never
goes
directly
to
your
users.
What
you
do
is
when
you
have
your
old
version
up
and
running.
You
stand
up
your
new
version
as
well,
and
then
you
run
like
load
tests.
B
Meanwhile,
Flagger
is
monitoring
the
metrics
to
see
that
everything's
going
okay
and
as
it
seems
that
everything
is
going.
Okay,
we're
incrementally
adding
traffic
to
the
canary
version.
Two
we
go
from
10
up
to
50
percent
once
we
get
to
50
and
our
metrics
all
still
look
fine,
we're
reasonably
confident
at
this
point
that
the
release
is
good.
B
B
So
in
panel
five,
now
the
upgrade
the
rolling
update
is
complete
and
we've
shifted
all
the
traffic
to
the
new
upgraded
original
deployment.
I'm
going
to
call
it
the
primary
because
that's
what
it's
called
in
flagger
and
in
version
six,
we
tear
down
our
Canary
pods
and
now
all
of
the
traffic
is
being
served
by
the
primary
actually
since
panel
five.
So.
A
Okay,
so
how
many
of
you
have
heard
about
Flagger
as
a
project
or
just
like
you
know,
installed
into
clusters
on
with
it?
Okay
boy,
so
Flagger
is
a
cncf
incubating
project.
It
falls
under
the
flux
family
of
projects.
Flux
is
the
guitars
tool,
so
the
entire
diagram
that
Kingdom
explained
right
in
the
last
slide.
Flagger
is
the
communities
operator
which
automates
that
entire
thing
so
right
from
shifting
traffic
incrementally
to
monitoring
metrics.
All
of
that
is
automated
for
you
by
flag.
A
You
can
see
it
enables.
You
know
safer
releases
right,
and
one
of
the
things
which
Library
does
really
good
is
is
that
it
handles
Disaster
Recovery
very
well.
So
if,
unfortunately,
our
version
2
is
not
a
working
expected,
it
has
bugs
or
something
like
that.
You'd
want
users
to
be
shifted
to
the
stable
version,
ASAP
right
and
that's
what
Flagger
takes
care
of,
so
you
can
deploy
on
a
Friday
and
be
satisfied
that,
if
there's
something
wrong
with
the
new
version,
your
end
users
won't
be
impacted
as
much
right.
A
A
These
are
some
of
the
things
we
support
and
it
has
an
extensive
webwork
mechanism
which
you
can
use
to
run,
load
tests
and
various
acceptance
tests
throughout
every
phase
of
the
canary,
so
that,
if
you
want
to
say
that
I
want
to
run
some
different
tests
at
20
of
the
weight
and
if
I
want
to
transfer
different
tests
when
it's
being
when
the
new
version
has
been
promoted,
Flagger
allows
for
that-
and
this
is
one
of
my
favorite
features
about
Flagger.
It's
not
a
feature,
it's
more
like
the
list
of
Integrations.
A
We
have
so
we
support
pretty
much
every
other
Ingress
service
mesh
out
there.
You
can
just
see
some
of
the
logos
I'm
not
going
to
go
through
all
of
them,
but
yeah.
It's
a
pretty
extensive
list
and
we
also
have
support
for
Gateway
API,
which
is
the
new
standard
for
supposed
to
be
the
new
standard
for
networking
and
load,
balancing
and
kubernetes,
and
as
more
and
more
projects
start
adopting
a
Gateway
API.
We
are
confident
that
we
will
be
working
with
all
of
them.
B
Canary
definition,
this
is
the
main
unit
of
currency
of
Flagger,
so
we're
going
to
define
a
canary
and
we're
going
to
start
with
some
targets.
So
we
have
a
Target
deployment
and
that's
the
deployment
that
I
mentioned
before
that
Flagger
is
going
to
copy
for
us
and
make
a
primary
out
of
it.
We
also
have
a
service.
We
can
have
an
auto
scaler
ref,
the
service
Flagger
will
create
and
we
have
an
analysis
section
which
is
really
the
interesting
part.
B
So,
as
you
see,
we
have
a
schedule.
There's
an
interval,
their
threshold
is
kind
of
unfortunately
named.
This
is
the
number
of
failures
that
are
in
order
to
eventually
land
at
a
successful
release.
Go
on
the
max
weight
is
the
point
at
which
we
should
stop
and
we
start
motion
step
weight
is
every
five
percent
we
add,
and-
and
here
the
metrics
is
really
the
interesting
part
for
today,
because
we
can
add
custom
metrics
here
or
any
kind
of
metric,
so
there
are
predefined
metrics.
B
A
Right
so
I'm
just
gonna,
try
to
tie
it
all
together
and
explain
to
you
in
a
brief
Mano
how
flag
actually
works,
so
you
apply
the
canary
object
and
Flagger
is
basically
reconciling
that
Canadian
object,
but
Flagger
is
a
bit
different
and
other.
It
is
operators
in
the
way
that
you
are
not
continuously
reconciling
the
state
right.
So
a
typical
kubernetes
operator
would
continuously
reconcile
the
state
of
the
cluster
to
match
your
desired
state
right.
A
So
the
way
it
works
is
we
wanted
to
be
as
as
like
pretty
much
loaded
as
possible,
so
that
you,
as
users,
don't
have
to
do
anything
to
do
anything
much
so
what
Flagger
does?
Is
you
give
it
a
deployment?
It
creates
an
exact
deployment
for
you
so
and
it
names
it
primary
deployment,
and
then
you
have
a
canary
deployment,
so
the
primary
deployment
is
a
stable
deployment.
A
A
It
basically
creates
services
for
your
Canadian
performance
and
your
primary
deployments
right.
So
all
you
need
is
to
create
a
primary
deployment
and
Flagger
will
create,
will
create
or
handle
all
of
these
other
objects.
So
what
Flagger
does
is
it
seems.
Oh
I
have
a
query
deployment
and
you
know
it's
a
new
version,
so
what
I'm
going
to
do
is
I'm
going
to
scale
this
up
and
I'm
going
to
scale
that
up
and
then
I'm
going
to
use
an
HTTP
route.
So
HTTP
route
is
a
generic
term
that
I've
used
here.
A
It
can
be
anything
it
needs
to
Virtual
service.
It
can
be
a
SMR
traffic
split,
it
can
be
a
Canadian
nginx
right,
it
can
be
anything.
Anything
which
can
distribute
traffic
to
to
different
Services
can
be
considered
an
HTTP
route.
So
what
flag?
What
Flagler
does?
Is
it
orchestrases
HTTP
route
to
basically
Route
traffic
based
on
weights
to
to
these
two
different
right
and
what
it
does
is
check
the
metrics
based
on
those
metric
validation.
It
will,
you
know,
Drive
the
analysis
forward,
so
this
in
a
nutshell.
B
Okay,
I'm
going
to
talk
about
this
section.
These
are
metrics
in
prom.
Ql.
You've
probably
seen
something
like
this
before,
if
you're
here
at
Prometheus
day,
just
to
point
out
on
the
left.
This
is
a
metric
for
nginx
and
it
has
a
namespace
and
a
Target
interval
setting
we're
looking
for
a
latency
under
a
particular
number.
So
it
has
this
by
Le.
That
means
less
than
or
equal
and
on
the
right.
We
have
another
metric.
This
one
is
for
error
rate.
B
It
looks
a
little
bit
different
I'm
not
going
to
go
into
too
much
detail
about
this
because,
like
I
said,
you've
probably
seen
things
like
this
before,
but
this
one
is
out
of
a
hundred
percent,
so
we're
multiplying
by
100
after
we
divide
the
numerator
by
the
denominator.
Numerator
is
successful,
requests
that
are
not
in
the
500
range
and
then
the
denominator
is
the
number
of
total
requests
so.
B
That's
mine
too
also,
so
this
is
a
metric
template
and
we're
going
to
take
advantage
of
one
of
our
definitions
that
we've
just
written
so
here
we're
plugging
our
not
found
percentage.
So
actually,
we'd
like
to
redefine
the
error
code,
we'd
like
to
say
we're
really
interested
in
404s.
We
don't
want
there
to
be
404s
in
our
release,
so
we're
just
going
to
go
ahead
and
make
this
not
found
percentage
metric
template
and
we
can
plug
it
in
to
Flagger.
A
This
kind
of
this,
this
object,
is
basically
the
products
of
how
a
flagger
you
know,
validates
metrics.
So
if
you
see
here,
you
have
a
type
and
an
address
right
in
the
metric
template
object,
and
this
is
how
you
can
have
multiple
different
observability
Stacks
running
in
parallel,
and
you
can
use
all
of
them
and
basically
have
a
very
complex.
A
A
A
A
A
So
this
is
a
more
powerful
version
of
how
you
can
do
scaling
than
HP
is
so
we
just
wanted
to
highlight
that,
because,
when
you
are
doing,
testing
load
testing
can
also
be
a
very,
very
important
factor,
and
if
you,
if
you
only
have
like
one
deployment
running
for
your
Canary,
it's
your
work,
you're
more
likely
to
run
into
memory
or
errors
or
something
like
that,
because
it
won't
split
with
Google
contents.
Real
world
conditions
we
have
Auto
scaling
enabled
to
you
know,
cope
with.
A
You
know
Argo
setup
like
a
git,
Ops
tool
setup
if
you
specify
the
replicas
field
in
a
deployment,
that's
just
a
recipe
for
disaster.
You
should
never
do
that
in
case
you're.
Doing
that.
Don't
do
that!
Never
do
that,
because
that
can
lead
to
all
sorts
of
problems,
so
you
should
always
have
a
scalar
or
Auto
scalar
setup,
okay,
so
just
to
go
back
quickly
to
the
diagram,
it's
pretty
much
the
same
diagram
but
there's
a
new
addition.
A
There's
a
new
component
in
the
diagram
which
is
see
that
Prometheus
is
like
really
driving
everything
here
right.
If
you
remove
Prometheus
Flagger
will
not
be
able
to
determine
whether
your
version
2
is
performing
expected
or
not.
It
won't
be
able
to
increase
the
time.
Okay
and
Keda
won't
be
able
to
scale
the
deployment
if
the
parameters
is
not
working,
Prometheus
is
Prometheus
or
any
other
observability
tool
is
very,
very
essential
to
how
you
Implement
Progressive
delivery
in
your
clusters.
B
B
B
We
have
a
list
of
objects
that
the
canary
is
aware
of,
and
we
can
see
things
like
the
deployment
primary
and
Canary,
so
here's
our
primary
and
here's
our
Canary.
So
when
we
do
the
upgrade
we'll
see
this
one
kick
into
action
and
we
have
a
list
of
events
and
we
can
see
that
our
we
have
defined
a
custom
metric
here
as
well
one
of
the
built-in
ones.
So
all
right
and
there's
a
yaml
view,
so
we're
going
to
try
and
upgrade
now.
B
B
B
A
So
you
we
have
two
ingresses
one
is
a
one:
is
handling
the
it's,
basically,
configuring
the
engine
to
traffic
to
the
primary
and
right
now
it's
protecting
so
here
five
percent
of
95
traffic
to
the
primary
Department
right.
If,
if
you
were
to
simulate
a
real
condition,
five
percent
of
your
users
would
be
experiencing.
A
B
A
C
A
A
So
this
basically
defines
that
we
we
want.
This
recipe
calculates
the
latency
for
every
request.
That
comes
in
right
and
this
is
our
scaled
object.
So
this
is
what
cada
uses
to
scale
or
scale
your
deployments.
So
it's
pretty
simple.
We
say
that
we
want
to
scale
the
product
for
deployment
10
seconds
what's
happening.
We
want
at
least
one
replica
running
at
all
times
and
we
want
a
maximum
radicals
of
three,
and
this
is
how
you
configure
you
to
listen
to
Prometheus.
A
So
you
say:
I
want
to
listen
to
a
Prometheus
server,
and
this
is
where
it's
located
at,
and
this
is
the
query.
I
want
to
run,
and
this
is
the
Threshold
at
which
I
want
to
start
scaling
up
my
deployments,
pretty
simple
right.
So,
let's,
let's
go
to
the
canary
okay.
So
here
we
have
a
canary,
it's
pretty
simple,
as
we
explained
earlier
as
well
as
it
goes
very
quickly.
So
we
say
that
we
want
to
run
the
check
every
15
seconds
we
want
to.
A
We
want
a
maximum
15
failure
checks
before
we
decide
that
you
know
it's
time
to
roll
back.
This
is
a
bad
version
and
we
want
to
take
a
10
steps
or
steplades,
basically
that
which
means
that
we
want
to
increase
the
traffic
by
10
every
15
seconds,
and
we
want
to
do
that
until
we
reach
a
50
traffic
split
into
The
Canal.
At
which
point
we
can
say
that
our
new
version
is
working
perfectly.
A
A
A
Right
so
right
now,
100
of
traffic
is
going
to
the
primary
and
zero
percent
of
traffic
is
going
to
the
canary
right.
So
if
we
look
at
what
Flagger
is
doing
is
that
now
we
have
our
pod
info
deployment
has
been
scaled
up
from
zero
replicas
to
one
replica.
So
now
flywheel
is
going
to
start
spending
some
sending
some
traffic
to
the
canary
version
right.
A
Right
so
10
of
the
traffic
is
going
to
the
canary
service
and
90
traffic
is
going
to
the
primary
service
right
and
in
the
background,
Flagger
is
checking
the
metrics
based
on
the
Prometheus
server
and
seeing
whether
everything
is
working
fine
or
not.
So
we
can
see
that
as
well.
Let
me
put
forward
yeah
yeah.
A
Right
so
we
can
see
here
so
we
have
something
called
red,
Matrix
and
use
metrics
I'm
guessing
this
is
prom
ql.
So
most
of
you
are
familiar
with
what
red
and
used
metrics
are
in
case
you're.
Not
red
basically
calculates,
like
you
know
stuff,
like
requests,
latency
error
rate
and
use
calculates
CPU
usage
memory,
usage,
bandwidth
Etc.
C
A
Right
so
here
you
can
see
that
we
Canadian
deployment
and
efficient,
so
the
primary
deployment
right.
So
it's
it
depends
on
the
weights
are
configured
so
so
like.
If
we
keep
doing
it,
we're
heading
the
primary
deployment
right,
but
if
I
keep
doing
it,
I
keep
doing
it
now.
I
have
the
Canadian
deployment
right,
the
new
version,
so
this
is
what
your
users
are
experiencing.
A
B
C
A
A
You
can
just
have
headers
for
your
beta
users,
but
then
you
don't
get
this
Progressive
delivery
step.
It'll
just
be
all
your
beta
users
in
one
go
right.
So
just
thank
you
so
I'm
gonna
get
out
of
here.
I'm
gonna
do
a
quick
get
of
the
canary
and
it
says
it's
promoting
right
now.
So
right
now,
50
traffic
went
to
the.
A
This
version
works
great,
so
what
I'm
gonna
do
is
I'm
gonna
take
the
new
version,
the
Canadian
deployment,
6.0.2
and
I'm,
going
to
tell
primary
to
be
at
6.0.2,
which
was
at
6.0.1
previously.
So
by
now
we
should
be
done.
We
are
done.
We're
almost
done
so
we'll
just
do
a
quick
watch
on
the
deployments.
A
And
one
thing
which
I
couldn't
really
show
you
during
the
demo
is:
if
you
see
memory
is
at
three
replicas
right
now
right,
but
when
we
started
it
was
at
one
replica
and
that
is
due
to
the
scaled
object
that
we
defined.
So
Keda
noticed
that
traffic
is
being
generated
to
the
Pod
info
deployments
and
based
on
that,
the
replicas
got
scaled
up
to
three.
And
if
you
see
here,
both
part
info
and
both
part
in
for
primary
is
at
6.0.2
right,
but
the
Pod
info
scale
to
0
to
0
replicas
right.
A
Right
so
all
so,
45
primary
has
100
of
the
weight
and
what,
if
the
community,
has
zero
percent
of
the
weight
right?
So
this
basically
shows
like
this
demo
was
meant
to
show
you
how
you
can
promote
a
new
promote
a
new
version
safely
in
your
production
clusters
by
shifting
traffic
gradually-
and
you
know
like
now-
we
can
say
that
all
all
our
100
of
the
users
are
experiencing
the
primary
version
and
we
can
be
more
confident
about
that
yeah.
So
that's
it
for
the
demo,
thanks
for
coming
guys,
thanks
very
much
for.