►
From YouTube: Making Your Apps and Infrastructure Services Failure-Resilient with... Yaron Schneider & Henry Spang
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Making Your Apps and Infrastructure Services Failure-Resilient with Dapr - Yaron Schneider, Diagrid & Henry Spang, Microsoft
Microservices architectures see a growing number of deployments and with it a larger number of infrastructure dependencies like databases, caches, message buses, secret stores and others. To prevent cascading failures, developers need to make sure that not only their apps are recoverable and resilient to failures, but also the underlying infrastructure that their applications interact with. In this talk, we'll explore how Dapr helps developers apply global resiliency policies across their apps and cloud or on-premises infrastructure services, irrespective of where their apps are deployed or what programming language they are using.
A
A
A
So
literally
any
programming
language
that
understands
each
peer
jerkbc
can
be
used
to
talk
to
debber,
and
so
the
application
will
call
dapper
over
these
protocols,
and
here
are
some
examples
of
how
these
look
like.
So,
for
example,
here
the
application
will
call
localhost
and
it's
going
to
tell
dapper
hey
dapper.
Please
invoke
the
card
application
for
me
on
the
method
new
order,
and
this
is
an
example
of
state,
so
the
application
is
basically
telling
dapper
hey
dapper,
please
fetch
the
item.
A
67
state
for
me
from
a
key
value
store
that
I
configure
you
work
with.
This
is
an
example
of
the
upper
pub
sub,
so
publish
subscribe,
and
this
is
an
example
of
how
an
application
might
fetch
a
secret
from
the
upper
from
a
component
called
key
vault.
So
what
are
these
components
for
state,
publish
and
secrets
dapper
at
its
very
heart,
has
the
concept
of
components
which
are
how
developers
talk
to
these
different
apis
or
ac,
rather
the
actual
implementation
behind
those
apis.
A
On
kubernetes
dapper
has
a
pretty
simple
architecture:
it
has
a
control
plane
that
is
used
to
configure
the
data
plane,
so
the
control
plane
runs
four
main
pods.
These
pods
are
the
dapper
runtime
injector,
which
is
a
sidecar
injector
it'll
inject
the
dapper
sidecar
into
your
application.
Once
you've
annotated
your
deployment
yaml
for
it,
the
sentry
service
is
gonna,
come
up
with
certificates
for
your
application,
which
contain
a
spiffy
compliant
identity.
A
So
you
can
then
use
that
identity
to
tell
dapper
to
apply
policies,
authorization
policies
so
that
one
app
can
only
call
the
apps
it's
been
configured
to
call,
and
the
operator
is
the
component
that
runs
in
the
cluster
that
listens
to
new
components.
So,
for
example,
if
you're
applying
a
component
of
a
state,
server
type
or
a
publish
or
a
pub
subtype,
the
operator
will
detect
that
and
then
update
the
sidecar
with
the
latest
metadata.
A
The
actor
partition
placement
service
is
used
for
a
very
specialized
dapper
building
block,
which
is
called
actors
which
is
used
to
distribute
verse
volumes
of
computing
state
throughout
the
cluster.
Millions
of
these,
the
dapper
can
just
fail
over
and
rehydrate
with
their
state.
If
you're,
not
using
actors,
you
don't
actually
need.
I
have
to
run
that
pod
and
then
dapper
will
essentially,
the
runtime
will
make
sure
that
it
listens
to
all
of
these
components
and
your
application
can
just
talk
to
it
using
regular,
http
or
grpc.
A
Let's
talk
about
resiliency,
so
far,
we've
talked
about
how
dapper
can
give
you
apis
so
that
you
can
talk
to
state
stores
or
publish
messages
or
subscribe
to
messages
or
retrieve
secrets,
but
at
the
heart
of
all
of
these
operations
you
have
something
that
your
application
depends
on.
It's
not
alone
in
the
world.
It's
not
isolated.
Your
application
might
have
a
direct
method
called
to
a
different
service
via
service
and
location
or
if
you're,
getting
an
application
configuration
there's
got
to
be
a
backhand.
That's
fetching
these
or
more
to
the
point.
A
A
But
what's
special
here
is
that,
for
example,
unlike
service
meshes,
which
give
you
these
capabilities
for
service
to
service
calls,
dapper
really
allows
you
to
apply
these
policies
holistically
and
globally
throughout
your
entire
application.
So
it'll
cover
service
to
service
calls,
but
it'll
also
cover
other
application
dependencies
like
databases
and
caches
and
secret
stores,
and
all
of
your
applications
external
dependencies.
Also,
and
with
that,
I'm
going
to
hand
over
to
health
to
continue
deep,
diving
into
that
resiliency.
B
B
So
as
a
compared
to
doing
something
like
this
with
an
sdk
or
you
know
any
other
piece
of
code
where
you're
manually
wrapping
each
of
your
individual
message
calls
what
dapper
does
here
is
it
lets
you
define,
you
know
a
a
whole
policy
that
lets
you
take
things
and
apply
it
across
a
bunch
of
different
sets.
So,
instead
of
worrying
about
your
individual
calls
or
where
things
are
being
set
up
that
way,
you
actually
have
this.
You
know
big
easy
way
to
set
up.
You
know
a
global
resiliency
policy.
B
So
what
those
are
is-
or
I
guess
I'll
go
to
crd
first,
so
that
was
the
yaml
which
you
can
run
as
a
standalone.
The
crd
itself
is
a
you
know.
The
kubernetes
concept
which
your
row
mesh
and
dapper
does
run
on
kubernetes
and
in
that
case,
what
we're
allowing
is
for
also
multiple
crds
to
be
defined
with
your
or
multiple
crds
to
be
fined,
and
then
they
all
get
merged
together.
So
you
can
see
here
we
actually
have
two.
You
know
different,
very
short
policies.
These
are
not.
B
You
know,
full
things
by
any
means,
but
you
can
see
you
know
one
specifies
a
timeout
one
specifies
a
retry
and
then
they
get
merged
into
one
big
policy.
So
this
also
allows
you
to
define
multiple
crds
across.
Even
you
know,
different
teams,
different
organizations,
you
know
a
bunch
of
different
people-
can
provide
the
resiliency
that
they
care
about
and
that
works
with
their
app
and
they
don't
need
to
necessarily
merge
it
all
together
at
once.
B
So
what
does
every
part
of
the
resiliency
structure
mean?
There
are
three
main
policies
we
have
timeouts,
we
have
retries
and
we
have
circuit
breakers.
Timeouts
are
the
easiest
out
of
all
these
guys.
They
simply
let
us
specify
a
duration
and
after
that
duration
has
expired.
We
timeout
and
the
request
fails.
You
know
really
straightforward
stuff,
but
allows
you
to
set
it
instead
of
worrying
about
per
http
call
per
grpc
call
and
which
client
you're
using
you
know
you
can
just
set
it
at
one
level.
It
can
apply
across
all
your
different
components.
B
B
Retries
and
exponential
retries
and
then
inside
of
the
retry
as
well,
you
can
even
specify
internal
adapter.
You
can
specify
if
a
retry
is
if
an
error
is
actually
retriable
or
if
it's
a
more
permanent
error
and
then
finally,
we
have
circuit
breakers
circuit
breakers,
being
one
of
the
more
complicated
sections
of
resiliency,
but
essentially
what
they
allow
you
to
do
is
cut
off
systems
from
traffic
or
reduce
traffic
to
allow
for
recovery
time.
This
is
in
situations
where
you're
having
a
component
or
service.
B
That
is,
you
know,
simply
not
working
appropriately
or
you
know,
causing
a
large
number
of
errors.
Now,
with
these
three
policies,
I
wanted
to
highlight
here
how
they
work
together.
You
know
they
essentially
all
wrap
each
other.
We
have
retry
at
the
outermost
layer,
followed
by
circuit,
breaker,
breakers,
followed
by
timeouts
and
then
finally,
the
actual
wrapped
function
or
code,
that's
being
called
by
the
resiliency
policy.
B
The
reason
we
have
them
in
that
order
is
because
that
allows
the
timeout
to
signify
error
to
the
circuit
breaker,
and
then
it
also
allows
circuit
breakers
to
signify
to
the
retry
policy.
When
we've
gone
too
far,
you
know
if
the
circuit
breaker
is
now
open.
We
don't
want
to
keep
retrying
and
continuously
spinning
our
wheels.
B
So
going
into
the
policies
just
a
little
bit
retries
like
I
said
we
have
constant
and
exponential
policies,
constant
policies,
really
simple:
they
have
a
maximum
number
of
retries
and
they
have
a
duration,
maximum
retries.
How
many
retries
we
have
duration?
How
long
in
between
each
individual
request,
exponential
policies
are
a
little
bit
more
complicated
because
they
have
a
little
bit
more
advanced
behavior,
but
we
can
set
the
max
retries
again
maximum
number
of
retries
that
we
want
to
make.
B
You
can
set
the
initial
interval,
which
is
the
starting
point
at
that
you
start
doing
exponential
backup
at
we
have
a
randomization
factor
which
is
used
to
introduce
jitter.
So
that
way,
you
don't
have
everything
you
know
calling
at
the
same
time,
because
again
you
can
just
have
swarms
of
requests
that
way,
there's
a
multiplier
which
is
the
growth
rate
of
the
initial
interview
interval.
You
know
so
the
bigger
the
multiplier,
the
faster
we
back
off.
We
have
a
maximum
interval,
which
is
the
maximum
amount
of
time.
B
B
A
circuit
breaker
has
three
states
which
are
closed,
open
and
half
open
closed
similar
to
an
actual
circuit.
Breaker
like
in
your
house's
electrical
system,
means
that
everything
is
running
normally
open
means.
We
have
flipped
the
circuit
breaker,
which
is
stopping
all
traffic
and
then
half
open
is
a
state
where
we
will
let,
through
the
max
number
of
requests
that
you
see
here
and
that's
where
we
look
for
a
successful
message
or
successful
request.
B
B
The
next
thing
we
look
at
is
actually
the
interval,
which
is
the
cyclical
period,
that
errors
are
evaluated
in.
So
that
means
that
we
have
a
rolling
window
where
we're
looking
for
a
certain
number
of
errors
or
a
certain
condition
to
be
set,
and
if
that
condition
is
met,
then
that's
when
the
circuit
breaker
opens
so
we're
looking
for
failure
scenarios
here
as
opposed
to
anything
else,
if
you
don't
specify
an
interval,
it
just
aggregates
forever.
B
The
timeout
is
how
long
the
circuit
breaker
remains
open
before
going
back
to
the
half
open
state.
So
if
you
set
a
timeout
of
60
seconds,
that
means
that
all
traffic
will
be
denied
to
whatever
that
target
is
for
those
60
seconds
and
after
60
seconds,
we'll
do
half
open
we'll
let
through
the
number
of
max
requests
and
again,
if
those
succeed
back
to
close,
if
they
don't
back
to
open
and
then
finally
is
the
trip
which
is
the
actual.
You
know
thing
that
we
evaluate
in
the
circuit
breaker
generally.
B
These
are
fairly
straightforward
cases.
You
know
the
and
the
default
that
we
see
here
is
consecutive
failure
count.
So
that
means
instead
of
looking
at
and
again
since
it
is
consecutive
failures
which
what
we're
looking
for
here
in
that
interval.
You
know
you
it's!
You
need
to
actually
have
those
errors
back
to
back.
B
So
that's
how
those
go
in
the
and
function
like
that
and
again
on
the
going
from
half
open
to
closed
again
we're
not
or
half
open
to
open
we're,
not
looking
for
the
full
trip
value
at
the
half
open
open.
That's
looking
for
a
basically
a
single
piece
of
success
or
failure
in
that
regard.
So
that's
not
evaluating
the
trip
again.
B
So
the
entire
resiliency
configuration
is
basically
setting
up
a
string
to
value
map
where
the
string
is.
You
know
the
name
of
the
policy
to
the
actual
policy
itself
and
then
in
the
target.
It's
the
name
of
the
target
to
what
the
policies
look
or
what
the
target
is
looking
for
in
terms
of
a
retry
policy.
B
So
first
we
have
apps
and
we
have
just
an
example
here
app
a
so.
This
means
that
when
any
application
calls
into
application
a
including
itself,
if
application
a
called
its
own
method,
it
would
be
using
these
policies,
so
it
would
have
a
general
timeout,
the
retributive
service
retry
and
the
circuit
breaker
would
be
service
circuit,
breaker.
None
of
those
are
defined
here,
but
that's
basically
how
this
is
going
to
work
out
is
that
you
would
expect
in
your
resiliency
policy,
to
find
a
timeout
named
general.
B
A
retry
named
service
retry,
a
circuit
breaker
named
service
circuit
breaker
components
are
a
little
bit
different
than
apps
because
they
have
a
slightly
different
set
of
behavior.
Here.
You
notice
that
we
have
a
pub
sub
component
that
we're
defining
this
for,
and
it
has
two
different
types
that
we're
defining
for
outbound
and
inbound.
B
Now
pub
sub
and
bindings
are
actually
the
only
ones
that
have
an
inbound
policy,
and
the
reason
for
that
is
you
can
look
at
components
essentially
as
calling
generally
an
external
source,
so
an
outbound
policy
is
calling
into
that
component.
You
know,
for
example,
if
I
have
an
azure
service
bus
component
and
I
wanted
to
publish
a
message,
a
method
or
message.
B
That
message,
you
know,
which
is
another
thing
handled
by
dapper
when
we
have
it,
receive
a
message
and
then
it
calls
into
our
application
they're
two
different
scenarios,
so
in
the
case
of
resiliency
they've
received
two
different
policy
definitions
and
finally,
we
have
actors
again.
These
are
a
little
bit
different
than
the
other
systems
very
similar
to
apps,
except
they
have
two
extra
fields,
which
is
the
circuit,
breaker
scope
and
as
well
as
the
circuit,
breaker
cache
size.
The
scope
for
an
actor
can
include.
B
The
type
of
the
actor
which
you
can
actually
see
is
the
how
we
index
into
this.
We
have
actors
and
then
my
actor
type.
So
that's
actually
how
we
would
use
this.
We
called
an
actor
of
my
actor
type.
We
would
use
this
policy
and
the
scope
can
be
type
id
or
both
and
when
you're,
using
both
or
id
what
it's
looking
for
is
actually
the
individual
actor.
So
you
can
actually
have
your
circuit
spread.
B
B
Actually,
you
don't
want
to
stop
all
traffic
to
all
actors,
but
you
want
to
stop
traffic
to
that
host
while
that
host
recovers
from
whatever
is
happening
or
while
the
host
is
de-provisioned
and
a
new
host
is
reprovisioned
and
then
finally,
we
also
have
the
circuit
breaker
cache
size,
which
again
is
important
because,
as
your
own
stated
earlier,
actors
and
dapper
they're
very
specific
paradigm
of
how
you're
doing
some
development
and
it's
designed
to
distribute
load
and
it's
designed
to
distribute.
You
know
tens
of
thousands
to
even
millions
of
of
actors.
B
It's
also
worth
calling
out
circuit
breakers,
going
back
for
a
second
here
circuit
breakers,
as
we
can
see
from
this
definition,
or
this
image
here
are
actually
stored
locally
to
your
sidecar,
so
each
sidecar
has
its
own
circuit
breaker
and
its
own
circuit
breaker
cache.
I
should
say
so,
as
you
can
see
here,
I
kind
of
define
the
flow
of
an
of
an
app
invocation.
B
If
app
a
wants
to
call
app
b
app
a
calls
into
us
into
a
step
or
sidecar
which
is
going
to
handle
all
of
your,
you
know,
resolution
and
networking
things
of
that
sort
and
know
how
to
call
appb,
but
the
first.
What
it's
going
to
do
is
going
to
check
it's
circuit,
breaker
cache
it's
going
to
look
for
a
a
circuit
breaker
for
app
b
and
it's
going
to
try
and
put
that
request
through
its
circuit
breaker.
B
The
reason
for
this
is
because,
if
app
b
is
broken,
we
you
know
we
need
to
know
about
it
on
app
a
so
we
keep
all
that
data
there
and
each
of
these
apps
interacts
individually
with
their
own
circuit
breaker.
So
that
way,
you
know
we
don't
have
any
of
these
they're,
not
widespread
failures.
Then
you
know
nothing.
No,
you
know
app
a
to
app
b.
Failure
is
going
to
impact.
You
know
app
c
to
app
b.
If
those
two
things
are
okay,
you
can
kind
of
imagine
this
might
happen.
B
If
you're
you
know
doing,
multi-az
cloud
work
and
maybe
az1
to
az2
is
down.
But
you
know
az2
to
az3
is
fine.
You
don't
want
an
easy
one
to
82.
Failure
to
stop
your
entire
traffic
set,
so
circuit
breakers
live
on
the
individual
hosts
and
that
way
we
can
have
some
more
fine
grained
control
of
how
traffic
is
managed.
B
So,
finally,
let's
do
some
target
examples
just
to
make
sure
everything
you
know
is
syncing
appropriately.
So
I
have
a
really
simple
policy
to
find
here.
We
define
a
few
timeouts.
We
have
fast
at
two
seconds
and
slow.
At
ten
seconds
we
have
a
general
retry,
which
is
a
constant
policy
with
a
duration
of
five
seconds
between
requests
and
a
maximum
of
ten
retries.
B
We
have
an
app
b
retry,
which
is
exponential
and
has
a
max
interval
of
20
seconds,
but
you
notice
it
doesn't
actually
have
anything
else
set,
including
maximum
retries.
When
you
don't
set
the
maximum
retries,
we're
actually
going
to
be
trying
to
do
some
more
we're
going
to
try
and
do
essentially
infinite
retry.
So
maybe
at
b
we
don't
care
how
long
it
takes.
We
just
want
that
thing
to
you
know
to
succeed.
Eventually,
we
also
define
a
circuit
breaker,
app,
acb
or
a's
circuit
breaker.
B
Then
we
have
just
our
two
targets
down
here:
app
a
and
at
b.
You
can
see
that
app
a
references,
the
general
retry,
which
is
this
retry
policy
right
here
and
then
it
references
the
timeout
fast,
so
request
here
would
come
out
after
two
seconds
going
into
app
a
and
then
it
also
uses
app
a's
circuit
breaker,
app
b,
however,
uses
app
b's
retry.
So
it's
going
to
retry
forever,
calling
into
app
b
is
going
to
retry
forever,
and
then
we
have
also
the
timeline
slow.
B
B
But
what
we're
going
for
here
is
the
analyzer
right
now
doesn't
have
any
resiliency
set
up
on
it
and
I've
just
you
know.
I've
made
some
new
changes
to
it
and
I'm
pushing
it
out
for
the
first
time
and
now
we're
going
to
see
how
it
runs.
So
we
can
start
my
generator
who's
going
to
start
generating
some
data
for
us.
B
We
should
see
in
just
a
little
bit
so
it
started
generating
data.
You
can
see
that
we're
getting
hitting
our
items.
We
have
found
11
items
to
process.
That's
great.
We
have
some
data
in
there
already,
oh,
but
look
now.
We
notice
that
we
have
13
instead
of
12,
even
though
we've
already
added
one.
So
maybe
something
weird
is
happening
with
this
application,
but
we
can
revisit
it
in
just
a
moment
because
first
I
want
to
show
off
my
resiliency
policy.
B
B
You
can
see
that
down
here.
My
cosmos
db
state
has
an
outbound
policy
where
I'm
just
using
the
circuit
breaker
state
circuit
breaker.
It
has
a
trip
of
consecutive
failures,
greater
than
one
it
has
a
timeout
of
60
seconds
and
has
an
interval
of
30
seconds
and
a
mass
request
of
one
second.
So
what
this
means
is
that,
after
more
than
one
consecutive
failure,
my
app
is
now
going
to
start
triggering
its
circuit
breaker,
which
hopefully
will
find
that
my
app
you'll
see
now
we're
getting
up
to
23
items
in
here
so
shortly.
B
My
application
is
going
to
hit
this
point
where
it's
you
know,
because
of
you
know,
maybe
we
have
a
filter.
That's
incorrect
on
our
on
our
query,
scan
that
we're
actually
getting
too
much
data
and
we're
going
to
start.
You
know
putting
too
much
pressure
on
to
columns
db
instance,
so
without
the
circuit
breaker.
What
we're
going
to
see
is
that
we're
just
going
to
start
failing
a
lot
in
there
right
on
cue.
It
started
failing.
B
B
It
is
again
so
right
now
we're
just
hitting
columns
db
we're
getting
429s,
which
is
caused
to
use
error
code
for
going
too
fast,
and
it's
just
going
to
keep
happening
over
and
over
again,
because
you
know
our
app
is
broken.
Essentially,
we
we
pushed
out
a
bad
deployment,
we're
now
causing
too
much
traffic,
and
what
this
means
in
in
our
our
day-to-day
life
is
that
you
know
we're
gonna
start
running
up
unintentional
cost
on
cosmos
db.
B
So
let's
kill
this
really
quick
and
then
what
we're
going
to
do
is
we're
going
to
go
back
and
we're
going
to
restart
my
application.
But
this
time
we
have
a
config
enabled
that
enables
the
resiliency
feature
so
now
we'll
restart
it
we'll
see
that
it
should
fail
immediately,
which
is
good,
but
then
you
notice
look.
We
failed
our
request,
but
now
we
have
this
our
query
states.
We
failed
to
query
the
state
store
because
the
circuit
breaker
is
open
and
now
you'll
see
that
we
keep
failing
our
circuit
breaker.
B
So
what
this
means
is
that
we're
no
longer
putting
pressure
on
cosmos
db.
It
means
that
our
first
application
doesn't
need
to
worry
about
there
being
too
much
traffic
for
it
to
get
its
job
done
and
then
because
the
circuit
breaker
is
applied
on
the
granule
level
of
the
cosmos
db,
you'll
notice
that
it
actually
means
that
we're
still
processing
messages
on
this
application,
we're
still
getting
our
pub
sub.
This
app
is
still
open
for
service
invocation.
B
If
you
had
to
do
other
things
to
it,
you
know
so
it's
is
isolating
just
the
bad
portion
of
the
code
and
now,
with
the
circuit
breaker
we're
giving
ourselves
time
to
investigate
time
to
find
the
bug
time
to
roll
back
or
fix
it
without
having
to
risk
the
the
failure
state.
You
know
we
don't
have
to
have
any
of
these.
You
know
over
over
charges
or
or
risk
a
bigger
outage,
and
you
can
see
here
that
we
failed
again.
B
B
If
you
wanted
to
or
shorter
one,
you
could
take
more
requests,
but
at
the
end
of
the
day,
what's
happening
is
you
know
our
circuit
breaker
is
stopping
us
from
overloading
the
database
and
stopping
us
from
causing
further
impact
with
our
bad
deployment.
So
that
is
the
demo
any
questions
before
I
stop.
Sharing
your
own.