►
From YouTube: Kanary - Automated and Integrated Canary Testing Using CRD... - David Benque & Cédric Lamorinière
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Kanary - Automated and Integrated Canary Testing Using CRD and an Operator - David Benque, Amadeus & Cédric Lamorinière, Datadog
The Kubernetes ecosystem provides primitives to easily deploy applications. Now, how might we simplify validation of a new application version before triggering its complete rollout? Canary testing is a popular answer. There are solutions to pilot canary testing from outside Kubernetes. How about driving it from inside Kubernetes instead? Kanary is a new Operator proposing an easy, automated and integrated way to orchestrate the canary testing phase, while controlling the traffic, the validation, and the rollout of a new version. After a quick recap of challenges in managing canary deployments within Kubernetes, this talk will present how Kanary offers a fully integrated and automated canary solution by assembling several cloud-native solutions: Prometheus, Istio, K8s resources and CRD. The talk will end with a demo!
https://sched.co/MPaB
A
Good
afternoon
everyone
hope
you
enjoyed
your
lunch,
we're
about
to
start
in
a
couple
of
minutes
time,
just
let
more
folks
to
come
in.
Obviously
it's
it's
a
long
way
from
from
where,
where
are
you
all
had
food,
so
yeah?
Let's,
let's
see
I
guess
we
give
30
seconds
if
nobody
comes
in
in
30
seconds,
we'll
start.
A
Great
pretty
much
full
house.
Thank
you
all
for
coming
hope.
You
enjoy
your
cube
con.
How
many
of
you
well?
How
many
of
you
is
this
first
cube,
carne,
actually
alright,
Wow
yeah!
This
is
my
eighth
cube.
Corn
I've
been
to
all
of
them
so
far,
I
might
I
might
I
might
want
to
take
a
break
next
time.
I'm
thinking,
but
it's
kind
of
nice
San
Diego,
would
be,
would
be
a
cool
place
to
go
to
I've
not
been
there
yet
anyway.
A
Well
hope
you
hope
you
enjoy
the
rest
of
the
show
and
and
the
talks
here
and
are
the
tracks.
So
this
is
a
CI
CD
track
just
to
be
clear.
Anybody
in
the
wrong
room.
Anybody
here
not
interested
in
CI
CD
and
would
rather
go
to
another
talk
elsewhere.
Okay
seems
like
everybody's
in
the
right
room.
That's
great
I
think
the
the
kind
of
the
distances
in
this
building
are
rather
surprising,
but
it's
not
too
hard
to
find
rooms
fairly.
Well,
signposted
I
think
that's
a
fairly
good
thing.
A
There
will
be
a
panel
I
think
the
next
talk
will
be
a
panel,
so
I'd
encourage
you
to
stay
for
for
the
panel
and
we
have
a
fairly
diverse
set
of
opinions
on
the
panel,
so
I
hope
you'll
all
enjoy
the
panel
coming
up.
Now
we
have
a
talk
about
canary
with
Kay,
as
everything
in
where
in
this
world
is
with
ek,
and
we
have
David
here
and
he's
colleague,
friend,
sorry
yeah,.
B
A
C
It's
gonna
be
with
just
one
so
welcome
to
this
presentation,
obviously
canary
with
okay,
and
it's
going
to
be
about
an
automated,
an
integrated
way
of
doing
canary
testing
in
Cubana
T's,
and
we
are
in
the
track
about
extending
communities.
So
we
will
explain
also
how
we
have
built
this
operator
on
top
of
a
customer
source
definition,
so
I'm,
David,
Bank
I,
am
working
as
a
software
engineer
at
Amadeus,
and
here
is
Cedric
Lavinia
working
at
data
dog
as
a
software
engineer.
Also
so
a
quick
recap
about
what
canary
is.
C
Obviously
you
know
that,
but
when
you're
running
this
routed
system,
you
have
the
application
running
many
instances
of
service
running
in
your
communities
and
when
you
want
to
roll
out
a
new
version
of
your
software,
the
only
thing
that
you
want
is
maintaining
customer
satisfaction
and
don't
break
anything.
So
how
do
you
achieve
this?
You
start
loading
few
instances
of
the
new
version
of
your
application
or
new
configuration
and
check
out
this
behave.
C
You
monitor
this
very
closely
and
if
you
happy
with
the
result
that
this
canary
instances
are
delivering
you
just
roll
out
your
new
version,
if
not,
you
just
stop
your
deployment.
So
today
you
probably
already
do
canary
testing,
ok
I'm,
wondering
here
in
the
room
who
has
built
some
kind
of
made
solution
to
do
canary,
ok,
who's
using
spinnaker,
maybe
about
the
same
proportion
and
any
other
product
you're
doing
canary
with
something
else.
So
we
have
also
some
other
case.
You
know
what
we
are
proposing
is
a
something
completely
integrated
native
Chianti's
solution.
C
What
I
mean
here
is
that
to
perform
the
canary
deployment
that
we're
going
to
the
only
thing
that
you
need
is
cube,
control
and
a
new
llamo
fight
that
is
representing
our
CID.
You
will
see
some
example
later
on.
We
will
offer
you
the
possibility
to
schedule
this
cannery,
starting
now
or
in
one
hour,
and
it's
gonna
work
on
top
of
the
existing
objects,
meaning
that
if
today,
you
have
already
some
tool
to
tuning
working
on
top
of
deployment
on
top
of
pods
and
everything,
it's
gonna
continue
to
work
with
this
solution.
C
There's
not
going
to
be
any
changes
for
for
your
existing
ecosystem.
In
theory,
for
example,
we
preserve
the
deployment
story
if
you're
using
it
and
we're
gonna,
see
that,
with
this
proposition,
you
will
be
able
to
run
multiple
canary
test
campaign
on
the
same
deployment.
At
the
same
time,
this
could
be
useful
if
you
have
different
developers
working
on
different
features
and
they
want
to
test
on
the
live
system
exactly
at
the
same
time,
so
that
we
can,
you
tell
us
more
so.
B
Yes,
so
when
you
want
to
put
in
place
canary
the
appointment,
you
need
two
things
about.
First
things.
First,
is
all
your
scale,
you
canary
deployment,
then
which
traffic
you
will
send
to
this
canary.
After
all,
do
you
want
to
validate
it
and
at
the
end,
you
apply
the
new
version
with
a
solution.
We
defined
several
configuration
for
each
of
this
topic.
B
For
example,
when
you
have
the
Cure
that
retrieve
from
kefka
topic,
no
need
to
receive
traffic,
but
you
can
also
configure
it
as
a
live
traffic
test
traffic,
meaning
that
it's
your
own
traffic
that
you
inject
on
top
of
your
you
are
canary,
or
you
can
also
request
both
live
and
test
for
each
go.
You
can
also
think
about
putting
shadow
traffic
on
this
deployment
of
tag
traffic
for
the
validation.
We
have
several
way
to
validate
it.
B
The
first
one
is
manual,
meaning
that
at
one
point
someone
will
need
to
update
the
customer
source
to
say
this
canary
deployment
is
valid,
but
we
implemented
a
way
of
doing
it.
Doing
it.
Sorry,
with
the
label
watch,
meaning
that
if
one
labels
appear
on
the
deployment
of
the
pod,
the
Canary
pod
automatically
the
Canary
deployment
will
be
flagged
as
failure.
B
You
can
also
send
a
concrete
query
against
a
primitive
server.
We
implemented
the
HTTP
or
custom
endpoint
like
while
you
you
need
to
respect
the
format
and
receive
the
response,
and
we
are
thinking
also
to
implement
metric
server
in
order
to
be
able
to
retrieve
the
mystic
from
other
metrics
provider,
and
the
last
part
is
all:
do
you
apply
as
a
new
version
if
the
Canaries
succeed,
so
you
can
do
wholly
automatically
update
or
you
can
run
it
at
the
high
one
for
another
issue
when
you
want
to
test
your
new
application.
B
So,
let's
see
an
example
of
the
gusto
mobile
source
canary
deployment
in
the
speck
part,
what
you
need
to
provide
is
a
deployment
name
that
you
want
to
test
it.
The
controller
needed
to
reach
with
some
information
on
the
current
deployment
object.
Then
the
son
service
name,
if
you
want
to
send
life
traffic
on
it
and
after
you
have
the
different
topic
that
I
presented
before
the
scale
to
traffic,
the
validation
and
last
part
is
a
template.
B
We
should
also
to
put
validation
period
of
15
minutes,
so
the
canary
will
run
at
least
for
15
minutes
before
deciding
if
it
succeed
or
offers
another
example
with
the
label
watch
here
we
define
that
we
want
to
look
at
labelled,
monitor,
alert,
I
CPU
on
the
pod
that
will
open
your
canary
color
last
example.
This
is
with
a
punctual
query.
You
provide
the
endpoint
for
the
primitive
server,
the
query
that
you
want
to
run
and
the
range
of
the
values
that
your
pod
should
respect.
B
In
order
to
be
valid,
we
haven't
also
another
parameter
that
is
valid
for
the
other
initial
delay,
it's
to
start
querying
primitive
server
and
other
metrics
of
solution
after
20.
Second,
because
some
time
your
application
need
to
be
to
have
a
bit
of
time
like
if
you
have
a
cache
inside
to
have
the
proper
metrics
apply.
B
So
this
is
a
representation
of
your
application.
You
have
a
deployment
on
top
application
set
set
of
pads
and
the
services
will
target
your
application.
So
when
you
create
the
crystal
stops
canary
deployment
automatically,
the
controller
will
watch
your
deployment
retrieve.
Some
informational
fit
create
a
new
deployment
in
peril
of
your
queue
on
the
payment,
with
a
new
template
and
a
new
services,
the
canary
test
services,
when
to
receive
the
test
traffic.
B
If
you
choose
to
put
HPA,
it
will
also
create
the
HPI
with
the
proper
configuration
to
scale
only
this
pod
and
not
the
other
pods.
If
you
choose
to
put
both
configuration
for
the
live
traffic,
you
will
receive,
we
will
configure
your
patent
plate
in
order
to
to
fit
the
level
selector
from
your
services
after
the
validation
pallet.
If
everything
is
going
well,
the
canary
deployment
will
be
a
flag,
a
succeed
and
your
current
deployment,
so
my
app
deployment
will
be
updated
and
you
will
have
the
version
to
select
this.
B
C
For
the
live
demo,
its
scripted,
but
everything
is
live.
Okay,
I
have
a
cluster
that
is
running
in
Cuban,
it
is
in
docker
and
the
application.
The
sample
application
that
I'm
gonna
run
is
an
application
that
is
in
the
repo
of
the
project.
It's
a
simple
server
and
I'm
gonna
run
anytime
the
same
image
but
I'm
gonna
change
the
parameter
to
tweak
the
response
time
or
the
response
code
of
the
application
to
make
it
behave
differently.
C
Okay
and
let's
run
the
demo,
so
the
very
first
thing
is
I'm:
creating
a
namespace
okay
and
in
this
namespace
I'm
gonna
inject
the
operator,
the
cannery
operator
that
is
going
to
watch
after
my
CID.
Of
course,
we
need
role
binding
and
an
deployment
for
this
cannery
object.
And
here
it
is
it's
running
now,
so
you
can
see
that
it's
there
and.
C
I
have
even
my
application
that
is
being
deployed,
so
I'm
gonna
reach
a
state
where
I
have
a
namespace,
my
canary
operator
running
and
my
application
with
three
pots:
okay,
I'm
gonna
expose
also
the
service.
The
the
service
for
the
simple
Sabbat
through
the
Gateway
I'm
running
inside
the
cluster
is
Joe
okay.
So
this
is
our
target.
My
application
and
also
I,
am
gathering
metrics
from
there,
so
I'm
using
the
Easter
service
mesh
and
as
primitives
in
the
in
the
background
run.
So
you
can
see
it.
C
The
cannery
pod
is
here
and
my
pod
for
the
application
is
here
targeted
by
this
service
and
we
should
have
some
traffic.
Not
yet
I'm
gonna
send
the
traffic
in
a
minute,
so
I'm
gonna
run
injection
and
at
the
same
time,
I'm
gonna
tell
you
how
I'm
gonna
generate
the
ceoddi.
Every
time
we
have
a
new
series
like
or
a
new
in
tax
I'm
going
to
deal
with
this.
C
We
have
created
a
plugin
to
ease
the
adoption
and
you-
and
there
is
this-
plug-in-
is
called
canary,
so
you
cube
CTL
canary
and
one
of
the
option
is
generate,
so
you
generate
on
top
of
the
application
deployment
that
is
called
my
app
this
one
and
you
give
a
name
to
your
canary
yeah
I
decided
to
call
it
by
Batman
I'm
gonna
receive
traffic,
both
the
option
that
Cedric
described
so
I'm
gonna
receive
live
traffic
and
test
traffic.
The
service
that
is
targeting
usually
my
application,
is
my
app
I'm.
C
Gonna
run
a
validation
pattern
of
one
minute
and
I'm
gonna
do
a
validation
strategy
that
is
monitoring
response.
Time,
I
and
I
want
my
p99
to
stay
below
310
milliseconds,
okay.
That
could
be
my
SLA,
for
example.
So
this
is
the
command,
and
this
comment
is
simply
going
to
generate
the
the
yaman
file
for
you.
So
here
is
the
CID.
You
can
see
the
kind
canary
deployment
and
we
have
done
all
the
books
that
Cedric
has
described
so.
C
C
It's
going
to
be
scheduled
right
now,
I
don't
put
any
date
and
time
here,
so
it's
going
to
be
triggered
immediately.
The
service,
the
template
for
the
deployment
and,
finally,
the
traffic
and
the
validation
and
because
I
choose
the
option
with
SEO
response
time
it
automatically
created.
For
me,
the
prom
ql
query
that
match
this
kind
of
of
SLA
definition.
C
You
see
here
that
there
is
an
initial
delay
out
of
20
seconds
and
the
query
is
going
to
be
run
every
10
seconds
and
for
a
period
of
1
minute.
So
now
that's
definitively
injected.
So
here
you
see
already
that
the
traffic
going
to
my
application
and
now
I'm
gonna
inject
this
yearly.
So
you
see
immediately
on
the
on
the
right
side
that
there
are
new
pods
being
created,
and
you
see
that
they
are
called
my
app
canary
Batman.
Ok,
now
they
are
running.
C
I
have
a
starting
period
of
20
seconds
where
the
cannery
operator
won't
do
anything.
It's
going
to
and
I
didn't
show
it
to
you,
but
I
tweaked,
it
the
response
time
to
be
out
of
SLA,
and
so
what
happened
is
that
after
20
seconds
here,
the
deployments
that
the
the
canary
statues
changed
I
don't
know
if
you've
seen
it.
C
C
The
canary
bird
is
still
there,
it's
not
receiving
any
live
traffic,
it's
cut
from
the
end-user
stuff,
but
it's
there
for
you
to
investigate.
If
you
want
to
continue
to
send
traffic
to
it,
to
investigate,
look
at
the
logs
and
everything
it's
possible
because
the
service
is
still
there,
the
pod
is
still
there.
The
deployment
is
still
there.
We
can
see
here
if
we
go
to
mesh.
C
That
it
has
received
traffic
for
a
short
amount
of
time
till
it
was
detected
as
a
failing
deployment
and
then
no
traffic
anymore.
Ok,
so
if
I
do
my
investigation,
I'm
fine
I
can
delete
this
and,
as
Cedric
explained
only
deleting
the
CID
would
trigger
the
deletion
of
all
the
object
that
were
created
for
you
behind
the
scenes,
so
the
deployment,
the
test
service,
the
pods
and
everything.
This
is
what
you
see
here.
We
have
the
pot
dominating,
but
we
have
no
more
Canary
service,
no
more.
The
deployment
nor
the
replica
set.
C
C
There's
going
to
be
a
rolling
update
of
the
new
version
being
triggered,
okay,
so
focus
on
this
window
here
here
we
can
see
that
it's
running.
We
have
some
minutes
so
a
couple
of
seconds
to
wait.
What
we
can
do
at
the
meantime
is
going
back
to
the
mesh
panel
here
and
see
what's
happening,
so
there's
still
the
line
for
Batman
that
we
don't
use
anymore,
but
ELQ
is
behaving
correctly.
The
p99
is
still
below
310
milliseconds
I'm,
showing
things
here
with
SEO,
but
this
can
run
without
these
two
at
all.
C
Okay,
we
are
just
taking
the
opportunity
that
is
to
is
providing
some
metrics
and
gathering
metrics
from
there.
But
if
you
are
not
running
the
service
master,
you
can
still
play
with
the
canary
operator
that
we
are
proposing
here
and
target
order
system
to
get
the
matrix.
So
what
happened
in
the
end,
as
I
told
you
that
my
app
is
being
rollouts,
you
can
see
here
the
pod
being
changed
on
the
top
right.
C
Some
old
pods
been
terminating,
you
want
being
triggered,
and
you
can
see
that
here
the
statues
has
gone
from
running
to
deployment
updated.
There
is
one
statues
that
we
haven't
seen
here,
because
it
went
very
fast.
There
is
the
stages
succeeded
in
between.
We
will
see
that
later
on.
So
what
we
can
check
is
that
on
our
application,
I'm
going
to
change
here,
the
the
refresh
rate
and
go
to
five
seconds.
C
So
next
step.
What
I'm
gonna
do
is
remove
this
one
and
I'm
gonna
trigger
the
launch
of
multiple
Canaries
at
the
same
time,
and
not
only
I
would
trigger
sorry
for
that
not
only
I,
we
triggered
the
launch
of
three
Canaries,
so
this
is
one
called
tor.
There's
one
called
Superman
one
called
Wonder
Woman
and
not
only
they
are
freaking.
C
Are
we
running
at
the
same
time,
but
I
perform
multiple
check
at
the
same
time,
meaning
that
for
one
minutes,
I'm
gonna
validate
here
you
see
that
keyword,
quantile,
meaning
I'm,
looking
at
the
response
time
and
at
the
same
time,
I'm
looking
at
the
success
rate
for
one
of
them,
I
think
it's
one,
a
woman
I'm
giving
very
good
response
time,
but
HTTP
response
500.
So
what
we
can
see
here
is
that
everything
has
gone
and
at
some
point
it
has
transitioned
from
fail
already
scheduled
running
and
go
to
the
very
end
of
this
yeah.
C
There
is
one
that
is
still
running.
We
still
have
to
wait
20
seconds
for
some
of
them,
I
hope
that
he
will
go
and
he
Arriba
there
are
two
that
should
go
to
fail.
Yes,
we
can
see
it
now
now,
if
I
go
to
the
mesh
dashboard,
I
can
see
what's
happening
from
a
metric
point
of
view
is
that
here
we
have
our
Wonderwoman
canary
test
that
I
was
replaying
very
good
response
time,
but
with
the
success
rate.
C
That
is
not
that
correct,
so
different
reason
for
failing
the
canary
ear
so
and
in
the
end,
just
like
before.
I
still
have
all
my
object:
they're
not
receiving
traffic,
because
the
period
of
validation
has
ended,
and
if
you
are
focused
on
this
window
just
like
previously,
you
have
not
seen
any
deployment
of
the
one
that
succeeded
that
is
called
toe.
This
is
because,
for
this
use
case,
it's
maybe
just
development
validation
that
we
wanted
to
do
and
no
rollout.
So
I
have
appended
it
here,
an
option.
I,
don't
know.
C
If
you
see
it,
it's
try
run
the
one
that
Cedric
explained.
Do
you
want
to
fall
out
or
not
in
the
end
in
case
of
success,
and
this
is
why
the
very
last
stages
that
you
see
here
on
the
CRV
is
succeeded
and
not
deployment
updated
like
it
used
to
be
on.
On
my
previous
example,
okay
I
opted
to
was
clear.
This
ends
the
demo
and
we
are
going
to
continue
with
some
slides.
C
So
you
have
seen
it
to
start
the
cannery
just
use
the
plug-in
cube,
CCL
generate
and
you
have
your
mo
or
if
you
already
have
your
mo,
you
can
apply
your
llamo
and
then
you
can
go
and
grab
a
coffee
because
the
engine
is
gonna,
run
behind
the
scene.
To
do
all
the
the
flow
to
perform,
the
cannery,
the
cannery
test,
scheduling,
validation,
either
fail
or
success,
and
either
deployment
or
dry
run
mode.
B
So
it's
not
the
first
operator
controls
that
we
build
to
control
some
stuff
in
humanities,
but
for
this
one
we
use
the
operators
decaf
from
the
opiate
of
harm
work
that
is
initiated
from
edit
and
course
Fox,
and
you
really
help
us
to
speed
up
the
development
of
this
solution.
It
really
boost
have
a
simple
operate
project
for
you
and
like
this,
you
can
focus
on
the
logic
that
you
want
to
implement
in
our
case
implementing
a
canary
deployments,
but
you
can
be,
as
you
want
to
implement,
with
CRDi
with
another
concept.
B
B
So,
with
only
with
these
four
commands,
you
are
able
to
create
the
skeleton
of
your
Parata
first.
One
very
simple
is
just
to
create
a
new
folder
to
put
your
card.
Then
you
call
the
new
canary
command.
It
will
generate
the
straighten
of
your
operator
after,
since
you
want
to
create
a
custom
Osos
to
define
your
logic,
you
you
use.
The
ADT
I,
give
the
name
for
your
IP
aviation
and
the
name
for
your
cane.
B
In
your
case,
canary
deployment
and
after
the
the
thing
where
you
will
put
your
logic
is
the
controller
and
for
that
web
and
osakan
add
controller,
and
you
give
the
information
of
which
object.
You
want
to
react
on
inside
the
controller
and
what
you
need
to
do
at
the
end
is
only
to
implement
this
method
that
was
created
for
you.
B
So
we
wanted
also
to
talk
a
bit
about
some
traps
that
we
we
felt
using
this
hammock.
So
lot
of
magic
happened
behind
the
same
I.
Don't
know
if
some
of
you
already
built
some
controller
but
II
I
didn't
talk
about
any
cache
work,
working
hue
and
other
concept
that
you
need
to
understand
normally
to
build
a
controller.
So
sometime
you
you,
you
are
blind
about
some
issues
that
you
have
inside
of
Commerc.
First,
one
that
we
we
face
is
depending
of
the
Cuban
test.
B
Questions
that
you
are
using,
the
the
Kusum
of
a
source
are
not
under
the
same
way
from
the
reaction
went
on
then
the
one
that
10
of
cuban
test
the
status
of
customer
source
is
the
super
sauce,
and
so
all
you
have
did
is
different.
Depending
of
the
version
of
your
command
is
questa,
and
this
is
not
undr
by
by
this
homework.
B
B
Looking
at
the
code
of
the
from
octo
Anderson
what's
happen
at
the
end,
we
saw
that
it
wasn't
at
this
point,
and
the
error
that
was
that
we
face
is
the
log
level
support
instead
of
amok,
so
this
is
fixed
before
it
wasn't
supported.
So
we
try
to
put
some
nice
log
level
and
it
we
didn't
see
what
we
wanted.
So.
C
A
few
tips
also,
if
you
are
used
issue
users
and
then
that
you
are
developing
also
controllers,
invite
you
to
put
this
annotation
side.
Christ
you
inject
Falls.
The
reason
is
simple:
in
theory,
when
you
run
a
control
loop,
it's
not!
You
are
not
exposing
a
service
at
the
same
time,
so
being
in
the
service.
Mesh
does
not
give
any
value
here
when
you
want
to
do
unit
test.
Ascetic
said
that
there
is
a
good
test
framework
and
when
you
want
to
run
your
end-to-end
test,
probably
you
have
to
expose
your
services
you.
C
We
are
running
kubernetes
in
docker
for
the
tests
we
used
to
have
mini
cube,
but
we
found
some
good
advantages
of
going
for
capabilities
in
docker
access,
especially
the
way
you
under
resources
on
your
local
machine,
and
if
you
want
to
target
the
service
inside
this
cluster
in
the
end-to-end
test,
most
probably
you
will
have
to
use
the
cube
CTL
proxy
and
go
through
the
API
server
proxy.
If
you
want
to
run
a
demon
just
like
the
one
I
have
done
with
kubernetes
andhaka,
just
be
aware
that
you
are
not
alone
to
expose
notepad.
C
For
the
moment,
so,
you'll
probably
have
to
use
port
forward
and
and
deploy
some
kind
of
ingress
in
your
communities
in
docker
in
my
case,
was,
is
to
as
a
summary,
so
the
cannery
project
itself,
it's
running
you've
seen
a
demo.
There
are
few
strategy
implemented
already.
We
have
some
extension
that
we
would
like
to
do
so
if
you
are
thinking
about
anything
to
extend
this
project,
please
please
tell
us-
and
this
issue
on
the
under
repo,
give
the
link
here
on
the
github
and
also
you're
welcome
for
any
contribution.
C
C
Think
it's
the
time
for
us
to
thank
one
of
our
colleague,
salvatore
diamine
on
it,
which
had
added
at
some
point
in
time,
explained
us
all
this
stuff
and
now,
if
we
about
to
write
clean
control
loop
outside
thanks
to
him,
somehow
I'm
saying
it's
easy
to
create
an
operator
but
creating
a
control
loop.
You
have
to
put
yourself
in
specific
mindset:
okay,
it
has
to
be
to
able
to
recover
the
loop
anytime.
The
process
need
to
be
stateless,
that
there
are
few
things
that
you
need
to
understand.
C
A
D
C
Already
for
the
validation
time,
you
can
extend
the
project
by
calling
any
external
server
that
would
give
elf
on
the
on
the
pods
okay.
So
this
is
one
part
of
the
answer.
I
would
say
you
mentioned
the
API
gateway,
so
you're,
referring
to
to
the
traffic
block.
So
far,
what
we
plan
is
to
have
better
integration
with
these
two
to
you
manage
tag,
traffic
and
shadow
traffic.
If
you
have
any
ideas
about
how
to
tune
an
ingress
or
whatever,
we
are
welcome
to
take
your
your
issues.
E
So
this
demo,
in
the
entire
setup
that
we
saw,
was
in
the
context
of
a
single
deployment
one
deployment,
one
canary
deployment,
often
in
the
Canary
deployment,
we
need
to
deploy
multiple
ports,
multiple
services,
multiple
components
together.
So
how
does
canary
deployment
work
in
a
in
a
world
where
we
need
to
deploy
several
components
together?
We
need
to
test
the
entire
stack,
whether
it
works
or
not,
and
then
we
take
a
decision.
Okay,.
C
C
F
A
question
concerning
history
of
the
the
the
deployment
she
said
that
the
can
I
reconsider,
keep
the
the
deployment
history,
but
at
some
time,
if
you
delete
the
the
canary,
you
lose
your
canary
history,
there
is
no
completion
state
of
the
canary
to
maybe
keep
it
to
to
to
make
sure
that
you
have
your
deployment
history
link
to
a
canary
that
was
used
at
one
point
in
time.
Do
you
understand
my
question.
B
So
currently
yeah,
when
we
did
the
canary
deployment,
you
don't
have
any
after
story
to
say
which
one
was
was
used.
We
can
seem
to
add
annotation
on
top
of
it,
to
to
give
some
information
about
what,
when
it
was
applied
on
everything
to
improve
the
history
of
the
canary
testing
yeah.
This
is
maybe
a
good
use
case.
Wasn't.
G
It
was
curious
if
you
have
a
way
to
compare
your
canary
replica
set
come
to
the
standard,
are
not
the
standard,
but
the
old
replica
set,
say
you're
running.
If
you
have
a
way
to
kind
of
compare
like.
Let's
say
the
cpu
between
the
two
is
similar
or
if
it's
different
and
kind
of
have
a
way
to
alert
on
that
or
your
metrics
based
off
of
solely
on
the
new
canary.
So.
C
G
C
So
you're
asking
if
we
are
able
to
do
kind
of
comparison
in
terms
of
how
it
does
it
is
there
any
deviation
compared
to
the
actual
status.
Yes,
we
do
have
this
you've
seen
the
in
the
prom
cool
section
here,
I'm
using
one
part
of
the
of
the
the
validation
syntax
is
for
having
a
value
in
a
range,
but
we
are
using
behind
the
scene,
a
library
that
we
created
last
year
for
sacred
breaking
that
is
able
to
validate
deviation
compared
to
existing
code.
C
I,
don't
know
if
you
remember
the
slide
of
Cedric,
why
you
saw
pod
name
and
pod,
query,
Truths
and
so
on.
There
is
a
syntax
here
where
you
can
say:
I
want
query
all
my
pods
that
are
behind
the
service
and
make
a
comparison
of
the
one
that
our
cannery
compared
to
the
rest
of
my
workload
and
if
you
register
any
deviation.
Okay,
so
here
you
express
this
deviation
in
percentage
of
a
metric,
it's
gonna
kill
the
cannery.
So,
yes,
we
can
do
that.
Thank.