►
Description
KCD Italy 2021* - The video of the talks are now available!
"Delay is the deadliest form of denial." C. Northcote Parkinson
We live in times when an application or service lag of even 2 seconds can be too long. Building a reliable Cloud Native distributed system means preventing failures and minimizing their effects to keep the system stable.
Join me to explore graceful recovery from unexpected scenarios using live coded examples of the well-known circuit breaker and retry patterns at the application level and complement those in the infrastructure (Kubernetes) with service discovery, load balancing, and load shedding patterns.
🤓 SPEAKER: Ana-Maria Mihalceanu, Developer Advocate @ Redhat
👉🏼 Here the slide deck: https://www.slideshare.net/KCDItaly/cloud-native-resiliency-patterns-from-the-ground-up-anamaria-mihalceanu
______________________
* KCD Italy 2021 is the first edition of a community-organized non-profit event that will take place on November 17th and 18th and will gather adopters and technologists from open source and cloud native communities for education, collaboration, and networking.
📯 Follow us to be notified about the upcoming MeetUps: https://community.cncf.io/kubernetes-cloud-native-milano/
A
B
Bye,
okay,
so
switching
switching
to
english,
so
the
next
speaker
is
actually
anna
maria
mitch
now,
and
I
hope
that
I
pronounce
the
surname
correctly
so
anna
maria
is
the
java
champion.
Developer
advocate
at
red
hat
and
the
co-founder
of
the
bucharest
software
craftsmanship
community
she's,
going
to
talk
about
resiliency
patterns
in
cloud
native,
so
anna
maria,
we
are
super
excited
to
have
you.
Thank
you
so
much
for
being
here.
The
stage
is
yours.
C
Hello
and
I'm
happy
to
be
here
as
well-
I'm
really
honored
to
be
at
the
first
edition
for
quesadilla,
italy
and
yeah
watched
very
closely.
The
talk
before
me
was
really
really
interesting.
I
do
understand
a
bit
of
italian
thanks
to
my
roots,
so
that's
a
good
plus.
However,
the
talk
will
be
in
english.
C
If,
like
you,
have
any
questions
during
the
talk,
please
post
them
in
the
q,
a
section
I
will
try
to
answer
as
many
as
possible.
So,
first
of
all,
who
am
I
and
why
I'm
here
as
said
earlier,
I'm
java
champion
and
a
certified
architect,
but
to
give
you
and
working
currently
as
a
developer
advocate
for
red
hat.
But
what
I
need
to
share
with
you
is
that
I
started
as
a
java
developer.
That's
true.
However.
C
So
that's
pretty
much
about
myself
and
in
the
talk
of
today
we're
going
to
look
a
little
bit
of
how
we
can
avoid
delays
and
denials
in
our
in
our
applications
and,
of
course,
the
first
place
where
we
would
like
to
like
look
into
is
like
defining
a
little
bit.
The
resilience
now
resilience.
There
is
a
definition
about
it.
I
don't
want
to
dive
into
the
definition
today.
I
just
I
would
rather
have
it
depicted
from
the
way
I
perceived
resilience,
so
our
application
typically
is
like
in
this
normal
behavior.
C
Here
right
we
I
mean
the
developers,
let's
say
the
developers
coded
really
really
well,
it
was
deployed.
It
works
perfect.
However,
sometimes
unexpected
conditions
occur.
You
know
in
our
abnormal
behavior
of
application,
it
turns
differently
and
it
starts
behaving
having
unexpected
events
inside
it.
These
unexpected
events
actually
trigger
next,
a
faulty
behavior
our
application
gets
into
faulty
behavior.
C
However,
thanks
to
detection
control
mechanism,
we
can
recognize
that
malfunction.
We
can
get
the
feedback
and
have
a
feedback
control
that
to
mitigate
that
malfunction
and
afterwards
get
into
a
so-called
degraded
mode
behavior
for
our
application,
so
our
application
will
be
not
responding
as
usual,
but
in
this
degraded
mode
we
also
have
options
and
patterns
and
we're
going
to
look
into
those
like
to
offer
a
similar
experience
to
the
end
user.
That
everything
is
like
going.
C
C
The
first
place
that
we
start
to
look
into
and
that
I'm
gonna
code
a
bit
to
show
you
how
it's
going
on
there.
The
first
place
where
you
start
with
fixing
things
in
in
case
of
failures
and
you're,
preparing
yourself
for
failures,
is
it
application
level
and
probably
most
of
the
java
developers
would
have
read:
martin
fowler's,
blog
and
where's.
C
C
They
work
perfectly.
Most
of
the
time,
and
in
my
first
microservice,
the
one
called
activity
is
actually
a
microservice
where
I
am
consuming
information
from
an
interesting
api,
not
sure
if
you
knew
about
it.
The
board
api
is
the
api
that
gives
you
like
ideas
for
activities
when
you're
feeling
bored
and
you
can
also
select
participants
and
so
on.
C
So
what
I'm
going
to
do
in
my
first
microservice,
I'm
going
to
consume
information
from
this
board
api
and
in
my
second
one,
I'm
just
going
to
have
a
very
nice
ui
that
to
display
these
board
api
activities
and
let's
try
and
we'll
try
to
make
this
these
two
systems
resilient
with
different
with
different
options
available,
so
the
board
api.
Typically,
we
would
say
that
is
you
know
functioning
correctly,
and
if
I'm
going
to
my
activity
microservice-
and
I
am
starting
my
quercus
app-
don't
worry
it's
not
going
to
be
a
java.
C
This
is
just
the
beginning
of
our
patterns.
Things
are
going
to
be
looking
really
really
nice.
For
me,
my
application
is
going
to
start.
It's.
Probably
everything
will
be
great.
So
if
I'm
going
to
my
browser,
sorry
about
this
and
say
localhost,
I
think
it's
on
8082
this
one
and
I'm
going
to
my
dev
ui.
My
app
is
going
to
be
really
really
working.
Fine,
let
me
try
out
an
activity
here
and
say:
music
and
execute
this
okay,
so
I'm
getting
some
activities.
All
is
good.
C
C
So
without
doing
something
in
between
our
end,
user
will
see
this
at
app
level.
There
are
several
options
available
so
just
to
quickly
introduce
you
what
we
can
do
at
the
app
level,
because
the
app
level
is
the
first
place
where
we
can
mitigate
problems,
I'm
just
going
to
add
a
few
things.
First
of
all
I'll
add
a
circuit
breaker
annotation
over
my
methods
to
mitigate
the
problem
and
on
this
one
with
a
circuit
breaker.
What
I
can
say
like
okay,
at
what
percentage
I
can.
C
I
want
to
like
open
the
circuit.
I
want
to
stop
things
from
showing
failure.
To
my
end
users,
I
can
say.
75,
however,
is
a
good
thing
like
tuning
for
me
to
retry,
because
maybe
things
are
not
working
at
some
point
in
time,
but
it
would
be
good
to
like
have
the
circuit
in
half
open
and
maybe
open
it
if
in
half
open
and
then
close
it
afterwards
when
things
are
back
to
normal.
C
So
maybe
I
want
to
like
retry
again
in
one
second
next,
when
it
comes
to
like
consuming
information
or
like
systems
that
do
not
respond,
at
least
at
application
level.
You
should
be
looking
at
like
how
much
time
that
response,
how
much
time
is
taking
for
a
response
to
be
to
be
obtained.
So
let's
mitigate
the
bit
the
timeout
part
and
say
hey.
I
do
not
want
to
like
wait
more
than
250
milliseconds
so
to
see
if
something
is
retrieved
or
not
from
this
consumed
api.
C
Now,
when
timeout
fails,
it
is
great
to
like
it's
great
to
see.
What's
that
something
is
wrong
in
time
of
destroying
a
timeout
exception
which
can
be
used
by
another
annotation.
That's
retried
and
what
we
try.
This
is
like
still
believing
that
things
can
be
fixed
and
I
can
say
max
retries
and
of
course
we
try
it
every
100
milliseconds.
C
So
what
these
three
together
do.
It
happens
the
following
like
when
three
out
of
these
four
fail.
It
will
trigger
this
one,
of
course
three
out
of
this
four
retries
with
a
delay
of
100
milliseconds,
and
it
will
trigger
this
one,
but
the
circuit
breaker
with
this
delay
will
be
like
yeah,
but
I
will
try
again
like
in
one
second,
so
maybe
the
things
are
working
better,
but
this
is
not
enough
for
end
users
and
sometimes
we're
using
full
backs
so
that
we
can.
We
can
show
something
to
the
end
user.
C
You
know
that
recovery
things
that
we
can
simulate
at
least
something
for
the
end
users
until
some
information
is
is
obtained
or
to
give
that
to
the
consuming
or
the
front
end
like
to
show
something
to
the
end.
Users
and
I've
used
the
default
basic
hobby.
For
for
that
one
and
my
default
basic
hobby
is
just
a
returning,
an
empty
hobie.
Now,
let's
go
again
to
the
me
to
the
this
part
here
and
execute
it.
C
Now
we
don't
see
the
a
lovely
exception
that
we
saw
earlier
number
response
is
like
having
participant
zero
price
zero.
It
has
some
information
there
about
this
activity.
It's
not
as
great
as
the
real
thing,
but
at
least
it
gives
the
users
some
information
that
is
more
more
nicer
to
see,
or
maybe
it
gives
more
something
more
appealing
to
be
interpreted
by
front-end
or
other
applications
that
are
orchestrated
in
the
ecosystem
of
my
microservice
right.
So
this
is
the
application
level.
C
C
C
And
of
course,
there
are
some
available
patterns
for
these
proxies.
We
have
service
discovery
and
service.
Discovery
is
something
that
well,
we
can
rely
for
what
is
available
in
kubernetes,
or
we
can
improve
on
that,
of
course,
there's
load
balancing
and
we
can
use
external
load
balancers
even
for
balancing
the
load.
When
you
know
things
are
like
becoming
very,
very
heavy
to
handle
to
be
handled
by
our
applications.
C
So
kubernetes
is
like
giving
a
nicer
layer,
an
extra
layer
of
security
to
our
app
and,
of
course,
we
go
into
more
serious
stuff,
with
circuit
breaking
with
retries
and
deadlines,
even
at
the
infrastructure
level,
and
that's
the
beauty
of
blending
all
together
at
application
and
infrastructure
level
and
well
a
third
pattern.
Probably
many
of
you
have
tried
it
or
you
would
like
to
try.
C
It
is
load
shedding
and
we're
going
to
look
into
these
at
infrastructure
level
and
see
how
we
can
build
upon
our
app
first
of
all,
locality,
load,
balancing
and
failover
the
easiest
to
do
that.
From
my
perspective
is
like
using
istio
eco.
Is
there
to
help
us
with
a
bunch
of
additional
resources
that
can
improve
on
what
kubernetes
is
giving
to
us.
So
in
my
example,
I've
used
the
gateway
configuration
so
that
my
requests
coming
to
my
microservice
would
be
like
afterwards
manipulated
by
the
gateway.
C
I
can
make
sure
that
if
v1
is
not
available,
then
v2
is
actually
going
to
handle
the
request,
and
this
local
dlp
setting
with
a
failover
is
something
that
application
developers
have
been
have
been
searching.
Also
in
regards
istio,
because
some
time
ago
a
friend
of
mine
asked
me
like
hey,
okay,
we
have
this
fallback
at
application
level.
How
about?
How
does
east
you
do
that?
Can
you
see
you
do
like
fallbacks?
C
C
C
First
of
all,
let
me
simulate
how
would
the
part
with
just
with
the
failover
enabled
would
be
looking
like
and
first
of
all,
let
me
go
and
keep
ctl
actually
oc
project,
I'm
using
openshift
for
this.
For
this,
for
these
demos,
but
also
my
examples,
work
in
mini
cube,
I've
tried
them
as
well.
So,
first
of
all,
I
need
to
like
deploy
my
apps,
and
I
have
coded
two.
Several
backends
and
I've
also
have
configured
a
backend
faulty
probe.
C
I
have
my
gateway
still
gateway
and
also
need
to
export
my
gateway,
url
and,
of
course,
let's
apply
the
virtual
service
with
the
failover.
Just
for
those
of
you
like
to
see
and
to
trust
that
I'm
doing
the
right
thing.
This
is
just
with
the
failover,
the
simple
rod
robin
from
v1
to
v2
and
next
one.
Let's
go
with
the
virtual
service.
C
D
C
And
maybe
this
will
be
enough:
okay,
yeah!
It's
it's
red!
It's
good!
Let
me
quickly
go
and
do
some
invocations.
I
just
want
to
make
it
bad
for
a
moment
so
that
you
can
see.
What's
going
on,
I
pretty
much
like
to
simulate
with
kiali
in
my
bad
situations,
and
I
hope
that
in
this
case
it
will
work
so
pretty
much
when
v1
is
not
available.
D
C
C
C
So
if
we
like
to
remove
the
fade
over
and
just
be
like
simple
simple
simulation
of
this-
and
just
you
know,
route
the
traffic
between
the
two-
and
I
think
I
should
be
like
keeping
both
back
ends.
Okay
and,
let's
say
cube,
ctl
get
deployments
just
to
make
sure
that
yeah
everything
is
up
and
running,
so
everything
seems
to
be
okay
if
I'm
invoking
an
endpoint
that
is
not
like
in
the
best
shape
ever.
C
This
one
is
like
giving
me
timeouts
what's
happening
when
concurrent
invocations
occur
over
this
end
point,
you
can
see
that
my
endpoint
responds
with
a
504,
but
it
takes
an
awful
amount
of
time
for
it.
To
like
give
me
some
information,
I
mean
two
seconds
in
this
or
four
seconds
in
this
in
this
lifetime
is
I'll
not
fall
out
a
lot
of
time,
so
this
is
not
very
nice
and
we
can
see
that
it
took
like
23.2
seconds
to
do
all
these
failed
situations,
which
is
not
nice.
C
C
C
Maybe
you
can
respond
with
an
application
deployed
in
another
zone
in
another
cluster
and
for
my
simulation
today
for
failover,
I
also
wanted
to
try.
Scooper
scooper
is
is
working
at
a
layer,
7
regards
connecting
connecting
networks
across
clusters
and
is
great,
there's
also
another
option.
I
think
it's
working
at
layer,
five,
it's
called
submariner,
so
you
can
try
that
one
as
well.
However,
you
have
a
little
bit
more
configurations
with
submariner
than
with
scooper.
C
C
Those
being
said
what
you
see
here
is
that
I
have
another
thing
deployed.
It's
called
front
end
and
yes
in
my
cluster,
let
me
go
here.
I
have
deployed
in
three
different
name
spaces,
my
my
application.
I've
done
the
front
end.
I
kept
it
in
the
benelux,
namespace
or
name
benelux
project
ce
has
only
a
back
end
deployed
and
in
there's
another
one
called
uki
also
deployed
here,
and
all
these
three
are
connected
via
scooper
and
just
as
a
heads
up
with
how
it
was
connected
via
scooper.
C
You
have,
by
the
way
all
the
details
for
my
setup
here
and
also
by
the
way,
with
cube
ctl
command
so
that
you
can
reproduce
it
how
I've
used
it.
Well,
it
was
easy
I
just
had
to
annotate
my
service,
my
back
end
in
all
the
only
clusters
or
the
namespaces
or
clusters,
depending
on
what
you're,
how
you're
trying
to
set
it
up,
and
then
I
pretty
much
initialize
cooper
in
the
namespace.
C
That
has
you
know
the
front
and
I
put
the
console
authentication
to
unsecured,
but
you
can
also
have
like
of
this
cooper
console
where
you
see
nice,
things
secured
as
well,
and
I
just
simply
created
the
token
this
token
what
it
does
for
me.
It
says
like
okay,
you
can
use
this
entrance
token
to
like
connect
your
your
deployments
across
namespaces
slash
clusters,
so
by
just
using
scooper,
init
and
scooper
link
and
others
in
other
namespaces.
C
I'm
able
to
interconnect
my
back
ends
without
doing
anything
anything
really
complicated
regarding
I
don't
know,
certificates
or
making
other
secrets
and
so
on
and
so
forth.
So
scooper
is.
This
lightweight,
of
course,
do
not
forget
to
annotate,
so
scooper
is
working
very
much
with
with
the
annotation
for
this
one.
So
it's
that's
another
important
aspect
of
it,
but
just
by
using
the
same
token.yaml,
I
can
do
this
in
a
endless
part
of
namespace.
So
this
is
the
power
of
scooper
and
just
let's
look
at
it.
C
Okay,
I
think
I'm
in
ben
looks,
I
hope,
no,
I'm
still
an
activity.
If
I'm
going
to
my
benelux,
where
I
have
the
front
end
and
my
routes
here,
if
I'm
going
to
hobby
front-end
for
the
moment,
I
see
that
there's
no
no
workers,
but
what
I'm
going
to
do,
I'm
going
to
ask
for
some
requests
that
will
be
mediated
by
scooper
and
speaking
of
this,
let's
look
a
little
bit.
C
This
should
be
the
yes.
This
is
the
console,
the
one
that's
unsecured
at
the
moment,
and
you
can
see
my
three
deployments.
Currently
no
request
has
been
done.
Let's
make
some
requests
to
our
cluster
so
to
our
deployments,
and
you
can
see
that
scooper
is
going
to
try
to
serve
it
from
benelux,
which
is
closer.
C
C
Okay,
so
you
can
like
see
what's
going
on
with
your
requests
across
across
this
another
thing
that
I
like
with
the
failover
with
scooper,
is
like,
if
I'm
doing,
let's
go
to
the
deployment
and
scale
it
to
zero
like
not
having
it
available
anymore
and
since
I
think
I've
already
have
killed,
killed
this
the
benelux
one
now
the
next
requests
are
gonna
be
requested
to
the
available
places.
So
you
see
that's
getting
things
from
ce
and
he's
not
touching
benelux
anymore,
because
it
has
nothing
available
there.
C
So
failover
can
be
orchestrated,
even
with
other,
with
other
options
in
the
cloud
native
space
and
of
course
I
will
quickly
go
with
you
through
low
chatting
load.
Chatting
of
course
drops
some
traffic
and
grabs
a
proportion
of
traffic
as
the
server
approaches
overload
conditions.
This
is
the
definition
from
the
site,
reliability,
engineering
book,
probably
you've,
read
it
and
you've
went
through
it
and
it's
an
interesting
subject.
C
How
tokenbucket
works
is
that
is
handling
incoming
requests
and
well,
when
incoming
request
is
being
is
coming
to
our
endpoints.
Let's
say
that
request
is
checked
against
the
bucket
size
and
if
the
bucket
size
is
full,
then
that
request
is
either
dropped.
Prioritized
or
sent
with
low
priority
drop
should
be
like
the
last
resort.
Most
of
the
time
things
are
prioritized
by,
I
don't
know
http
headers
or
you
have
some
sort
of
algorithm
to
prioritize
the
request.
C
You
don't
want
to
drop
things
and
you
don't
want
to
drop
requests
of
your
users,
it's
better
to
sometimes
serve
them
later
than
just
dropping
them.
Then,
if
the
full
bucket
was
not
like
reached,
you
can
send
the
request,
because
your
app
is
like
going
on.
Like
nothing
happened,
the
bucket
can
be
refilled
because
you
know
you
are
comparing
the
incoming
request
with
the
bucket
size,
but
this
is
not
like
an
infinite
thing.
So
when
the
bucket
is
full,
that's
it
no
more
request
is
being
handled.
That's
not
the
right
thing.
C
Also,
a
nicer
way
to
like
look
at
rate
limiting
so
istio
is
also
coming
with
options
to
do
to
implement
rate
limiting
and
those
are
available
as
well.
But
you
can
also
do
concurrency
control,
based
on
request
priority
with
with
sdo
and
the
way
to
do
that
is
most
of
time
by
using
either
headers,
and
you
can
use
the
the
baggage
user
agent
like
from
which
browser
the
request
is
coming,
and
you
can
like
prioritize
lower
based
on
that.
So,
for
example,
it
can
happen
that
you're,
maybe
on
end
users.
C
You
don't
have
a
pool,
a
big
pool
of
end
users
using
safari
and
like
asking
for
your
services,
and
in
this
case
you
can
like
give
them
a
lower
priority
and
give
to
the
others
a
bigger
priority.
I'm
not
saying
to
drop
I'm
saying
just
to
like
give
a
lower
priority
or,
like
you
know,
handle
them
differently
so
that
they
are
not.
You
know,
being
put
so
much
load
on
your
on
your
app
when
you
are
like
having
high
traffic.
C
How
about
auto
scaling?
Because
we've
been
talking
about
these
things
of
like
handling,
load
handling
failure,
but
sometimes
when
you're
like
thinking
of
handling
load,
isn't
there
auto
scaling
available
in
kubernetes,
and
why
is
that
not
enough
for
handling
load,
for
example,
to
make
an
application
resilient?
C
C
What
you
can
do
is
when
requests
are
coming
to
your
app
and
where
your
workload
is
increasing,
and
you
feel
that
you're
approaching
the
auto
the
auto
scaling
you
can
have
the
load
chatting
kick
in
to
reduce
the
resource
stress
and
that
will
be
nice.
You
know
for
app
once
the
resource
I'm
load
chatting
is
happening,
but,
as
you
saw
earlier
for
the
token
bucket
algorithm,
there's
a
duration
for
which
load
chatting
is
applied,
so
load
chatting
will
stop
at
some
point.
You
don't
want
like
to
apply
it
for
a
longer
time.
C
That's
when
you
should
like
look
at
the
of
the
situation.
Does
the
increased
traffic
continue,
because
if
the
increased
traffic
is
continuing
over
time,
then
in
this
case
you
should
trigger
auto
scaling
or
you
should
trigger
the
scaling
mechanism
so
that
you
are,
you
know
actually
instantiating
a
new,
a
new
pod
or
or
you
have
a
new
deployment
scaled
there
so
that
you
can
handle
things
with
for
your
users.
C
So
always
like
look
at
how
much
time
the
bad
things
or
the
workload
is
under
stress.
If
things
are
not
that
bad
or
the
low
traffic
is
lower,
then
yata's
killing
condition,
then
you
have
can
get
to
the
normal
situation.
You
can
assume
that
things
are
normal,
so
no
more,
no
chatting!
No
more
of
thinking
about
auto
scaling.
Everything
is
back
to
normal,
so
everything
is
great.
C
However,
this
is
how
I
would
say
that
the
auto
scaling
works
great
with
with
a
load
shedding
and
but
always
keep
in
mind
about
the
conditions
and
how
you
can
reduce
stress
with
both
of
them.
Thank
you
very
much
for
listening
to
me.
The
code
is
made
available
to
umbra
resilience
and
I
hope
I'm
not
over.
A
C
It's
a
quercus
I've
used
quarkx
for
my
app
because
well
it's
cloud
native
and
I've
used
the
native
version
of
fit
so
I've
used
with
growl
and
yeah
quark
is
in
java
17..
So
that's
that's
the
the
magic
behind
it,
but
I
think
there's
also
other
annotations
available
in
other
frameworks
as
well.
So
it's.
A
C
Normally
you'd
say
that
that
would
be
enough,
but
I
would
say
that
ex
extra
safety
net
coming
from
sdo
is
always
great
to
have
it's
what
you
are
like
expose
that
you're
monitoring
your
systems
so
even
sometimes
having
a
small,
a
small.
I
know
unavailability
situation
can
be
really
bad
and
can
be
really
upsetting.
Not
just
maybe
end
users
will
not
see
it,
but
that
translates
sometimes
into
some
business
impact.