►
From YouTube: Pod Anomaly Detection and Eviction using Prometheus Metrics - David Benque & Cedric Lamoriniere
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Pod Anomaly Detection and Eviction using Prometheus Metrics - David Benque & Cedric Lamoriniere, Amadeus (Beginner Skill Level)
Dealing with system stability in a distributed and changing environment is a challenge: a single failing pod can affect the majority of your system responses. From Kubernetes probes to Istio circuit breaker, the CNCF projects provide us multiple means of containing this kind of problem. After a quick review of all these means, understanding in which cases they can be used and their limitations, we will see how to react to problems that can only be revealed by internal application KPIs. Maybe you would have liked to use a service mesh circuit breaker feature, however your traffic is not HTTP based; or, one pod continues to reply with HTTP code 200 alongside incorrect functional content. And yet, you have functional indicators that that could help take an immediate and orchestrated operational response. We will see how to cover such cases thanks to dedicated controllers and Prometheus.
About David
Senior software engineer at Amadeus. Area of expertize: PaaS. Working on: application development, automation, system operability at scale, distributed systems. Languages: Go, C++ mainly. Like: sharing experience, learn and teach. My mission at Amadeus is to implement the transition from our legacy distributed system to cloud native environment using Openshift. First time speaker in public conference. Private talks or meetup talks already done.
About Cedric
Cedric Lamoriniere is a software developer at Amadeus working on the integration of Openshift in the new internal PaaS ""Amadeus Cloud Services"". Cedric is specialized in distributed systems development. His first public speaking experience happens last year at the RedisConf (SF) where he presented a Kubernetes Redis-Operator.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
So
hello,
everyone.
So
let's
start
this
presentation
about
the
project
that
we
have
started
animals,
that's
called
Cuba,
Weiser,
ants
about
system
stability
and
anomaly.
Detection
around
both
so
Cedric
and
I
will
run
this
presentation
we're
working
at
Emmaus,
so
we
have
been
using
a
module
software
last
week.
A
Okay,
let
me
let
me
ask
the
question
in
another
way:
we
took
a
plane
to
come
to
this
conference.
Here
we
are
so
you
have
whole
used
almost
our
software.
We
are
an
IT
company.
We
are
a
company
that
provide
IT
solutions
for
the
travel
industry
and
we
are
a
leader
when
it
comes
to
traveling
by
plane.
So
few
numbers
here,
so
this
presentation
will
be
divided
in
three
parts:
the
first
one,
quick,
slides
around
distributed
systems
and
stability,
then
we'll
see
what
we
have
already
in
the
platform
to
enhance
the
stability
of
the
system.
A
So
when
we
have
big
systems
like
this,
one
I
like
to
talk
about
the
butterfly
effect,
because
you
all
know
this
butterfly
effect
when
we,
you
have
something
very
small
happening,
but
weird
in
one
part
of
the
system.
It
can
really
trigger
some
big
chaos
overall
in
the
system
and
impacting
all
your
clients,
and
this
can
be
even
worse,
depending
on
the
load,
balancing
strategy
that
you
are
using
and
all
your
application
behave.
A
For
example,
if
we
have
a
application
and
be
application,
if
the
a
call
to
be
only
one
time-
and
you
have
only
one
instance
failing
your
client
feel
like
it-
has
a
twenty
percent
failure
rate
on
the
system.
But
if
you
a
layer
needs
to
be
to
make
multiple
calls
to
the
B
layer,
for
example,
and
availability
engine
calling
a
pricing
engine,
and
you
have
to
call
for
multiple
prices,
you
may
have
more
than
25
percent
failure
rate,
because
only
all
the
calls
are
distributed
to
all
the
beam
senses.
A
In
the
industry
we
have
many
quotes
about,
failure
is
the
norm
about
stability
of
the
system
or
big
they
are
and
complex.
They
are
to
operate
today.
I
bring
you
one
new
code
from
a
French
artist,
Paul
Susanne.
We
live
in
a
rainbow
of
chaos,
let's
try
to
keep
the
rainbow
and
remove
the
chaos,
and
for
that
we
have
some
solutions.
So
let's
talk
about
solutions
so
as
always
explaining
in
few
slides
before
we
may
have
to
call
another
service
civil
time.
A
So
here
an
example
with
16
senses
of
P,
one
out
of
six
failure
rate.
If
you
need
to
call
B
only
one
time
per
client
request
from
a
farm
or
if
you
need
to
call
them
six
time,
because
if
you
need
to
call
six
time-
and
you
have
run
rubbing
as
a
load
balancing
algorithm,
then
you
will
have
a
very
failure
rate
of
one.
A
So
if
you
manage
to
do
some
proximity
based
load,
balancing
and
you're
able
to
shut
your
system
and
your
algorithm
that
does
the
load
balancing
managed
to
contain
the
load
balancing
within
this
shot,
you
may
reduce
this
failure
rate
for
your
user.
So
one
of
the
things
that
you
can
do
is
take
availability
zone
as
a
shot
as
a
shot
of
your
system
and
try
to
contain
all
the
collateral
cause
of
your
services
inside
the
validity
zone.
A
More
than
that,
it
will
be
easier
for
you
to
spot
the
root
cause
of
the
problem
in
terms
of
locality.
How
do
you
do
that?
Well,
today,
natively
in
kubernetes,
you
don't
have
solutions.
There
are
already
two
issues
open.
The
very
first
one
here
is
about
trying
to
load
balanced
traffic
to
the
same
node
if
the
service
exists.
On
that
same
note,
the
second
one
is
trying
to
have
load
balancing
topology
aware
to
do
what
exactly
I've
presented
on
the
previous
slide.
A
There
is
even
a
proposal
for
that
in
terms
of
implementation
and,
finally,
even
if
there's
nothing
that
even
kubernetes,
if
you're
running
a
service
mission,
especially
the
one
called
is
that
comes
with
a
proxy,
the
skull,
envoy
and
and
voi.
This
guy
is
availability
zone
where
so
you
will
be
able,
with
a
service
mesh,
to
achieve
what
we
have
on
the
left.
A
If
you
have
applications
that
behave
in
a
bad
way
when
it
comes
to
memory
consumption,
maybe
you
can
use
limits.
If
you
are
doing
you
have
memory
leaks,
then
this
limit
will
prevent
a
container
to
eat
all
the
resources
of
the
system
and
the
cubelet
will
kill
this
container.
So
this
can
in
a
way
protect
you
and
improve
stability,
but
fix
the
bug.
A
Have
checks
in
kubernetes
if
you
type
these
keywords
in
the
documentation,
you
will
reach
the
page
that
talked
about
props.
We
have
two
types
of
probes:
liveness
probe
and
readiness
Rob
different
purposes.
The
first
one
will,
in
case
of
non
Elfie
container,
kill
the
container.
The
second
one
will
just
remove
the
pod
associated
with
the
container
from
the
service
discovery,
so
the
container
will
still
be
there
that
will
not
receive
any
traffic
more
than
that.
All
the
pole,
where
the
container
is
will
not
receive
traffic.
Ok,
it's
active
I'll
check
that
we
have
here.
A
The
activity
is
triggered
by
the
couplet
on
every
container,
where
the
probes
are
defined,
to
define
probes.
You
have
three
ways:
the
first
one
being
TCP
socket
trying
to
open
a
connection.
The
second
one
being
sent
an
HTTP
request.
Every
2,200
response
will
be
okay,
everything
else
is
non
Elfi
and
finally,
you
have
another
opportunities
to
use
your
own
script.
A
A
A
Okay,
if
you,
if
you
do
bugs
inside
the
probe,
you
can
completely
create
a
service
outage,
for
example,
keep
it
simple
means:
don't
try
to
check
your
external
dependency,
you
we
are
in
micro,
service
environment
and
you
are
not
going
to
check
if
your
dependencies
are
there
or
not
behaving
correctly
or
not.
This
is
their
job
also
to
do
that.
For
example,
if
you
reach
the
state
where
you
have
cycles-
and
this
happens
in
big
distributed
systems,
you
may
not
be
able
to
restart
your
system
from
scratch.
A
Even
if
you
have,
you
are
checking
your
dependency
chain.
Another
bad
things
when
checking
dependency
chain
is
that
one
pod
may
have
different
paths,
let's
say
a
B
and
a
C.
If
the
probe
in
a
check
the
the
validity
of
the
C,
we
may
have
a
problem,
because
if
the
C
is
not
healthy,
then
a
tones
non
LV.
But
you
have
cutted
the
AC.
But
at
the
same
time
you
have
cut
the
a
B
that
was
working
correctly.
A
The
circuit
breaker
so
on
the
left
side
of
the
slide,
the
the
pattern
circuit
breaker
that
you
may
all
know
it's
a
state
machine
here
that
we
have
and
we
have
either
a
closer
fit
or
an
open
circuit
or
half
open.
Okay.
How
to
achieve
this
in
our
kubernetes
environment,
you
can
use
a
service
mesh
either
with
link
ID
of
or
east
you
again,
you
will
have
a
proxy
being
injected
in
front
of
next
to
you
or
as
a
sidecar.
A
Sorry
inside
your
pods,
and
this
will
be
the
guy
implementing
the
state
machine
and
let's
take
the
example
again
of
a
and
B
a
does
not
is
not
connected
anymore
to
to
all
the
the
B
instance.
All
is
not
load
balancing
directly
there.
It's
the
proxy
that
is
doing
that
now
and
the
proxy
does
an
analysis
of
what's
happening
on
the
on
the
request
and
especially
on
the
response
tray.
A
If
there
is
a
timeout,
if
it's
returning
an
error
or
something
like
that,
and
if
one
of
the
instances
is
not
behaving
correctly,
then
it
removes
them
it
remove
it
from
the
pull
off
of
potential
end
point
for
be
so
it's
the
proxy
that
is
doing
this
circuit
breaker
stuff
keep
this
patter
in
mind,
because
the
kubera
is
a
project
that
we
are
presenting.
Just
after
is
doing
this,
but
another
level.
A
Still
service
mesh
and
what
they
call
the
retry
policy.
So
here
again
we
have
this
proxy
that
is
doing
an
analysis
of
the
response
of
a
dependency
and
that
is
able
to
trigger
another
call
on
another
instance
in
case
of
error.
So
on
the
left
side,
you
have
the
dashboard
of
these
two
and
you
see
what's
happening
when
you,
you
have
a
be
instance
failing
and
that
you
activate
the
retry.
So
on
the
top,
you
have
the
the
success
rate
for
the
for
the
a
and
at
the
bottom.
A
A
So
all
this
solution
that
we
have
gone
through
quickly,
you
have
a
you
have
seen
that
they
are
making
local
decision.
I
mean
it's
always
in
the
container.
The
probe
is
defined
at
container
level
inside
the
pod
with
the
service
mesh.
It's
always
the
proxy
that
take
the
responsibility
to
remove
an
endpoint
or
not.
Okay.
A
There
is
no
orchestration
of
the
the
activity
of
self-healing
that
is
being
done,
and
another
important
thing
is
that
all
the
stuff
is
based
on
technical
signals
or
KPIs
memory,
consumption
connections,
response
time,
timeout,
analysis
of
the
written
code
and
even
the
circuit
breaker.
That
is
needs
to
do
layer.
Seven
analysis,
okay,
is
only
link
to
return
code
of
HTTP
value
of
the
sorry
HTTP
risk
response.
A
A
First,
we
want
to
take
Orchestra
set
of
orchestrated
decision,
meaning
that
we
don't
want
to
look
anymore
at
container
or
pod
instance.
We're
gonna
act
there,
but
we
are
going
to
take
decision
based
on
what's
happening
overall
on
the
service
and
also
we
take
decision
trying
to
not
make
things
worse.
I
mean,
if
you
have
all
your
instances
that
are
failing,
there's
no
point
in
taking
any
action
of
on
any
of
them.
A
We
wanted
to
take.
This
is
being
able
to
take
decision
based
on
any
other
signal
than
technical
signal
and
especially
business
signal.
Okay,
so
you're
gonna
see
that
in
the
demo
after
we
are
using
a
business
metric
and
it
will
be
the
price
of
of
an
air
ticket,
so
the
Cuba
visor
itself.
The
project
is
a
controller
that
we
are
going
to
run
in
the
in
kubernetes
cluster
inside
it's
made
of
two
parts:
the
the
breaker
and
the
activator.
You
remember
this
a
feedback,
a
pattern
opening
or
closing
the
circuit.
A
A
So
here
is
an
example
of
this
custom
resource.
So
the
let's
focus
on
the
specification,
the
very
first
thing
which
service
are
we
monitoring
and
taking
action
on?
Okay,
the
second
part,
the
breaker.
So
here
we
have
a
different
value.
Let's
say
that
the
global
setting
for
the
breaker,
where
you
say
I,
want
to
take
action,
but
with
some
limits,
I
don't
want
to
make
things
worse.
A
In
case
everything
is
failing,
so
you
can
define
either
in
terms
of
count
or
our
percentage,
the
number
of
pots
that
you
want
to
keep
it's
a
bit
like
pedestrian
budget,
but
at
circuit
breaker
level,
and
then
you
have
the
strategy.
I
told
you
that
they
are
multiple
strategies.
Yeah,
we
see
continuous
value
deviation
and
how
do
you
configure
this
strategy?
We
are
going
to
reuse,
an
external
system
that
would
provide
us
monitoring
metrics,
and
here
we
are
using
Prometheus.
A
We
first
define
were
to
find
Prometheus
and
then
you
can
inject
the
prompt
QL.
Will
this
prompt,
you
all
needs
to
be
done
in
such
a
way
that
it
returns.
We
will
see
that
in
the
demo,
a
value
that
is
1
in
case
everything
is
fine
and
then
a
0.5.
If
we
have
a
50%,
deviation
or
1.5
depends
if
it's
upper
bound,
then
you
can
say
what
is
the
deviation
that
you
accept
on
this
KPI
and
the
breaker?
A
Of
course,
we
take
action
that
if
the
the
metric
returned
by
the
from
QL
exceed
this
as
threshold
the
activator,
it's
more
simpler,
you
have
to
define
the
mode.
We
have
three
modes
to
power,
your
dick
and
for
the
peyote
to
define
the
value
in
them
in
seconds.
This
is
for
the
circuit
breaker
pattern.
When
you
go
to
alpha
open
again,
ok,
trying
to
challenge
again
the
the
pod
or
the
instance
against
tree
traffic,
and
you
put
it
either,
you
break
it
again
or
you
re-inject
it
again
and
so
on.
A
So
this
will
do
up
and
down
for
the
pod,
when
you
use
periodic
we'll
see
that
in
the
demo
another
example,
so
here
a
different
strategy
for
the
breaker
discrete
value
out
of
list,
meaning
that
the
prompt
UL
will
return.
You
an
array
somehow
and
you
define
what
are
the
value
that
you
accept
as
valid
and
all
the
one
that
you
don't
want
to
accept
the
bad
values,
all
the
good
values
and
then
internally
the
the
Cuba
visor
will
do
some
computation
and
I
think
I'm,
not
sure
if
we
have
the
ratio.
A
Now
we
are
missing.
One
parameter
is
the
ratio
of
failure
versus
success
that
you
have
to
define
an
ear
for
the
activator,
but
we
have
a
different
strategy.
We
try
and
kill.
Instead,
it's
a
bit
like
periodic,
but
after
a
number
of
count
of
reinjection
of
the
pod,
you
decide
that
no,
it's
never
gonna
be
good
again.
So
let's
kill
it
and
finally,
I
think
the
last
strategy
we
have
in
terms
of
breaker
it's
custom.
A
You
can
call
any
external
service
that
will
tell
you
hey
this
bodies,
be
not
behaving
correctly
or
this
set
of
pods
is
not
behaving
correctly
and
all
the
Cuba
visor.
Our
controller
will
take
actions
in
terms
of
removing
the
pod
re-injecting
it
and
applying
the
circuit
breaker
pattern,
and
here
again
for
the
activator
we
are
doing
periodic,
but
there
is
a
last
rata
G
that
is
not
from
this
example
for
the
activator
in
retry
and
post.
A
A
So
all
does
that
work.
So
very
simple.
Also,
we
have
a
service
that
is
having
a
selector
and
so
a
set
of
pod
associated
to
this
service.
Then
you
have
a
matrix
server
for
us.
It's
Prometheus
that
scraps
the
metrics
or
technical
metrics
functional
metrics,
whatever
any
KPI
that
you
want.
Of
course
your
pod
needs
to
be
instrumented
for
that,
and
then
you
have
the
cure.
Visor
controller,
obviously
I.
As
a
user
of
the
system,
you
need
to
do
two
things.
The
first
one
is
inject,
the
definition,
the
cr-z
that
we
have
just
seen
before.
A
Okay,
then,
once
you
have
done
that
the
cuba,
cuba
visor
itself
is
running
now,
everything
is
fine.
You
can
manage
the
pod
that
are
associated
with
the
service
and
query
from
it
use
with
the
from
to
have
that.
You
have
defined
and
start
to
apply
the
strategy,
but
nothing
is
going
to
happen.
You
are
in
a
kind
of
dry
run
mod
here
for
the
Cuba
visa.
Why?
A
Because,
if
you
want
to
go
for
a
life
and
taking
real
action
in
terms
of
service
discovery
and
eject
eject
in
some
buds,
you
have
to
append
one
thing
to
the
selector:
that
is,
the
the
the
label
associated
to
proviso:
traffic-
equal,
yes,
and
at
that
time,
if
a
pod
is
considered
non
Elfie,
according
to
the
query
that
you
have
done,
Cuba
visa
will
change
automatically
the
the
the
label
value
and
the
pod
will
be
removed
from
the
service
discovery,
so
demote,
I'm
Cedric,
is
going
to
run.
That
is
the
brave
guy
I'm.
B
B
So,
let's
see
first
what
we
are
already
running
in
our
Christophe,
so
we
use
mini
Cuba
for
the
demo.
You
know
you
can
see
I
love,
yes,
that
we
have
four
different
pods
of
our
services,
so
the
price
of
services.
Then
we
have
a
cube,
a
visor
for
the
ear.
So
it's
a
controller
that
is
running
and
then
we
have
primitives
and
graph
an
oven
in
things
to
the
community,
so
pirata
I
will
start
to
inject
some
traffic.
B
So
this
is
a
price
written
by
the
by
our
services
and
if
you
want
to
see,
oh,
it's
look
like
it's
coming
back
yeah,
so
you
provide
the
it's
a
services.
You
provide
the
origin
and
the
destination
and
to
the
return
some
price
for
it.
So
street
price,
as
I
say,
we
created
also
a
dashboard
to
show
this
information.
So
so
the
first
line
is
the
business
information
that
we
have
so
the
number
of
requests
received
by
each
part
for
the
origin
destination.
B
B
So
let's
come
back
to
the
terminal,
so
yeah
we
see,
we
have
only
the
pods
of
application
and
what
is
a
will
be
interesting.
To
look
at
is
labelled
that
will
be
added
automatically
and
updated
by
Cuba
as
well.
So
first
thing
that
we
need
to
do
to
use
Cuba
visa
is
first
a
credit
cards
or
services,
and
this
is
the
first
configuration
that
we
will
use.
So
first,
you
provide
the
name
of
the
service.
This
is
services
attached
to
our
pod
and
then
the
breaker
configuration
so
we
use
formatives.
B
A
Yeah
we
have
the
the
so
we
are
running
the
continuous
value
deviation
strategy,
so
your
prom
ql
query
have
to
return
a
value
that
is
one
when
there's
no
deviation,
okay
and
it
have
to
group
that
by
by
pod.
So
this
is
what
we
are
doing
here.
If
you
look
at
it
precisely,
you
can
see
that
we
are
trying
to
get
the
over
30
seconds
the
average
price
value
that
is
being
returned
against
the
total
average,
the
average
of
all
the
the
set
of
pots.
A
What
is
interesting,
also
in
this
query,
is
this
and
close
that
we
have
here
that
allow
us
to
exclude,
from
the
analysis
any
pot
that
is
not
receiving
enough
traffic.
Okay,
we
are
making
statistics
that
is
six
our
goods
on
big
numbers.
So
if
a
pot
that
has
just
started
or
not
receiving
any
traffic
is
not
interesting,
interesting
for
the
analysis,
so
here
is
an
example
of
query.
I'm,
pretty
sure
that
you
have
all
prom
cures
and
they
are
experts
here
that
will
find
better
way
of
achieving
this.
A
B
Okay,
thank
you.
So
let's
create
this
cuba
reserve
service
and
what
we
can
see
on
the
top
of
the
screen
is
that
we
have
two
new
ala
bells
I
did
on
each
pod,
so
we
have
the
name
of
the
cuba
reserve
service
and
then
we
have
the
traffic
labelled
e
equal.
Yes,
so
it
means
that
for
the
moment,
two
bars
obscene
that
all
pots
are
LC.
B
So,
since
we
are
good
coder,
we
don't
have
any
animal
in
of
pot.
So
let's
inject
it
so
first
first
things
that
I
done
just
before
is
that
okay,
we
added
the
modify
the
label
selector
on
the
on
the
services
in
adult
who
have
no.
We
can
see
here
the
new
selector
labels
that
is,
traffic,
equal,
yes,
and
now,
let's
inject
some
anomaly
in
one
of
the
pods.
So
for
that,
as
we
can
see
here,
the
price
of
one
pods
return
zero.
B
We
also
implemented
a
quickly
a
program
for
cube
CTL
in
order
to
see
the
status
of
our
customer
sauce
here
and
what
we
can
see
that
we
have
currently
one
pot
bright.
This
means
that
we
have
traffic
equal,
no
here
and
since
I
didn't
comment
it,
but
we
choose
first
periodic
mods
of
also
activator
every
15
seconds
the
traffic
the
pod
will
come
back
on
on
the
services
and
so
the
traffic
will.
We
will
see
as
again
some
error
up
here-
yeah.
B
Yes,
so
it
come
back
to
yes,
so
we
have
again
error-
and
this
is
this-
is
we
don't
this
mod
as
periodic
it's,
because
some
time
we
have
an
anomaly,
because
a
pod
is
overloaded,
because
the
not
maybe
is
not
LC
so
by
removing
these
pods
during
a
period
of
it
is
a
gun
will
be
so.
The
load
of
these
pods
will
decrease
and
you
will
be
come
back
LC
thanks
to
that.
In
all
cases,
it's
not
the
case.
B
If
we
come
back
here,
what
we
can
see
that
this
pod
no
always
return
zero
when
he
received
traffic,
and
so
what
we
can
see
is
that
this
pod
receive
less
traffic
from
the
other
and
the
other
pod
have
an
increase
of
traffic
because
it
they
need
to
serve
the
request
more
requesters
and
before
so
know
what
we
can
see.
He
also
is
a
matrix
bitten
by
cuba,
cuba,
riser,
and
we
can
see
the
pattern.
B
Okay,
yes,
and
no-
and
this
this,
this
Ravel
of
requests
correspond
to
the
eyes
open
pattern
of
defined
by
the
self
is
like
yeah
and
yeah.
We
can
see
that
sometime.
It's
all
one
zero
one.
At
the
bright
spot,
so
for
our
case
that
the
body
is
not
able
to
come
back
normal,
what
we
will
do,
we
will
change
the
activity
method.
B
So
first,
I
will
delete
the
project
that
event
you
and
we
will
update
the
configuration
so
this
time
what
we
will
put
his
retire
and
kill
method.
Father
activity'
for
the
biker
party.
Still
the
same,
we
choose
at
the
same
period,
15
Saigon
and
with
the
maths
retry
count
equal
to
1.
It
means
that
Cuba
visa
will
try
if
an
iPod
Avenue
normally
will
try
first
to
remove
it
and
to
come
back
behind
the
service,
but
I
just
saw
on
the
issue.
It
will
kill
the
body.
A
B
B
So
this
is
great
for
the
for
the
quality
of
service
that
we
want
to
put
the
end
of
services,
but
for
investigating
is
not
very
practical
because
we
don't
have
anymore
the
pollen
stores
to
look
at
it.
That's
why
we
have
the
last
configuration
for
activate
all
the
retry
on
post.
So
this
time
we
added
a
new
parameter
is
a
max
post
count,
and
this
is
a
number
of
pods
that
Piazza
can
put
in
pose
mode
for
yourself.
B
B
B
Smile
so
know
the
traffic.
We
can
see
that
we
don't
have
any
more
error.
We
still
have
the
pods
for
investigation,
and
so
maybe
you
can
create
some
alerts
thanks
to
this
level
and
everything
to
do
with
that
lose
the
team
responsible
of
the
application
will
receive
to
say
a.
Maybe
you
have
something
to
investigate.
We
save
your
ass
by
removing
these
pods,
but
it's
not
the
nerve
for
you
and,
and
so
let's
see
yeah.
So,
let's
last
thing
that
we
want
to
show
you
is
that
Oh.
B
Occupy
visor
is
able
to
know
how
many
counts
it
done
for
each
hit
high
on
the
pot.
So
for
that
we
use
annotation
on
the
pod.
We
have
two
annotation
where
the
red
icons
so
equal
to,
because
first
time
it's
the
first
frost.
Try
on
the
second
well
also
interaction
is
when
it
was
put
in
pause,
and
we
have
also
the
last
time
that
the
pod
was
Mike
and
exists
for
investigation
is
cool,
because
you
can
look
at
the
trace
at
this
average
time
or
the
requested
ECT
spots.
B
B
So
that's
it
for
the
demo
just
a
few
side
before
question
so
first
things
that
here
we
want
to
show
you
is
that
we
are
using
already
the
pattern
of
controller
in
cuba
hiatus,
so
we,
and
so
we
have
a
controller
part
that
is
watching.
Events
on
the
cube
arise
of
service
pistol
sauce,
but
also
on
the
board
and
services
that
is
associated
to
this
supervisor.
Salas.
Inside
this
controller
we
have
two
main
parts
to
activity'
and
the
break
apart.
B
This
is
a
very
thanks
to
that.
We
are
able
to
to
him
to
add
some
new
features
early
and
what
one
of
the
possibilities
that
we
have
is
to
to
change
the
anomaly
detector
that
we
use.
So
in
our
example,
we
use
vomitus,
but
thanks
to
this
interface
part
here,
we
are
able
to
use
all
the
matrix
system
that
can
be
used
in
your
company,
like
you
provide
solutions
that
you
already
other.
A
For
example,
now
in
Amadeus
we
are
looking
at
how
to
leverage
this
system
when
we
are
doing
canary
deployment,
because
you,
canary
deployer,
when
he
deploys
the
canary,
can
look
at
the
annotation
associated
to
the
canary
instance
and
it
if
it
figure
out
that
the
Cuba
visor
is
often
removing
this
spot.
The
traffic
and
the
service
discovery.
Maybe
there's
no
point
in
continuing
the
deployment.
Okay,
so
cannery
deployment
is
just
an
example.
You
probably
already
have
some
ideas
or
you
could
use
that
so
some
key
takeaway
for
this
presentation.
A
There
are
many
ways
of
improving
the
stability
of
the
system.
We
have
listed
some
at
the
beginning
of
this
presentation
and
we
are
proposing
here
another
way
this
this
way
allow
you
to
use
your
business
KPI.
If
you
have
any
it
leverage,
any
external
monitoring
system
like
Prometheus
or
any
other,
that
you
would
like
and
provide
you
some
automation
to
do
some
kind
of
circuit
breaking
on
tops
of
Cuban
artists
concepts.
A
It's
based
on
standard
stuff,
CRD
from
QL,
aerobats
services
and
pod.
It's
clearly
extendable,
not
only
because
it
put
labels
and
annotation,
but
also
the
architecture
itself
of
the
the
software
is
there
to
allow
you
to
plug
some
new
strategy
called
strategy.
If
you
want
inside
directly
the
cube
advisor,
we
are
open
sourcing.
This
ooh.
We
of
course,
are
happy
to
receive
any
comments,
so
please
check
out,
try
it
wreck
it
post
issues
and
in
colors
we
have
maybe
the
time
to
take
one
question:
at
least
one
of
the
question
is
answer
on
that
slide.
C
A
You
said
we
are
changing
the
label
and,
as
such
we
are
reusing
the
natives
service
discovery
mechanism
of
humanities
that
is
associating
a
pod
to
a
service
thanks
to
a
label
selector
the
we
have
changed:
the
label
selector
extending
the
label
selector,
which
traffic
equal
yes
and
we
are
manipulating
the
traffic
equal,
yes
or
no
opposed
on
the
pod.
Changing
this
value
will
remove
the
pot
from
the
service
discovery,
but
not
from
the
replication
controller
or
the
replica
set
okay,
replication
controller
fusing
OpenShift
replica
set.
Thank
you.
So
thank
you
and
please.