►
From YouTube: Writing Reliable, Scalable, Fault Oblivious Code On K8s the Easy Way - Kendall Roden & Alice Gibbons
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Writing Reliable, Scalable, Fault Oblivious Code On K8s the Easy Way - Kendall Roden, Microsoft & Alice Gibbons, Diagrid
Speakers: Alice Gibbons, Kendall Roden
Remember when writing connected application code to do something innovative was the hard part? These days developers have to do more than ever, not only to write innovative and elegant code to run their business, but now also have to deal with all the faults, outages, disasters, hot spots, scale points, and concurrency issues that come with building distributed apps and microservices at scale. In this session, you will learn about the best practices to write distributed app code that sings at scale directly from the team that helped bring you Dapr and a number of hyper-scale cloud services in production. It will focus on the topics of resiliency, distributed locking, and optimization in common tasks like async messaging & Pub-Sub, state management, secrets and more. You will also learn about abstractions and implementations to do this the easy way in your favorite programming model and language, while targeting Kubernetes.
A
Hey
thanks
so
much
for
coming
everybody.
This
is
great.
This
is
a
bigger
room
than
I
expected,
at
least
personally,
we
are
going
to
talk
about
building
resilient
apps
the
easy
way
today
and
I.
Don't
know
if
anyone
actually
read
the
abstract
for
this
talk,
but
we're
trying
to
do
a
lot
in
35
minutes.
So
hopefully
we'll
show
you
at
least
one
or
two
of
the
things
that
we
promised
in
the
abstract.
B
A
A
Okay,
so
I'm
going
to
kick
it
off
here.
Talking
about
some
developer
challenges,
I
think
this
is
one
of
these
things
that
we
are
all
here
at
kubecon
talking
about
distributed
systems
and
some
of
the
challenges
we
face
as
developers
in
nit
today,
I
think
every
single
person
in
this
room
would
have
you
know
different
idea
of
what
this
means
to
you.
Every
single
person
could
say
that
they
have
a
different
challenge
that
they're
or
their
their
Dev
team
faces.
A
A
How
do
I
do
consistent
retries
on
my
apps?
You
know
if
something
fails,
if
there's
Network
latency,
how
do
I
ensure
that
goes
all
the
way
through
the
back
end
call
and
doesn't
just
stop
when
you
know
there's
Network
latency.
These
are
just
a
few
of
the
things
that
was
the
reason
behind
the
Dapper
project.
A
A
Today,
we're
going
to
be
focusing
on
a
subset
of
dapper
apis,
we're
going
to
be
focusing
on
publish
And
subscribe,
distributed,
lock
and
service
to
service
invocation.
Dapper
stands
for
the
distributed
application.
Runtime
didn't
mention
that
and
it
can
be
run
on
any
different
platform.
That
you'd,
like
today,
we'll
be
using
an
implementation
running
on
kubernetes
because
hey
it's
kubecon
and
that's
where
you
know
we're
all
here
and
how
does
dapper
do
this?
A
It
uses
an
idea
called
Dapper
components.
These
are
yaml
manifests
that
essentially
you
take
the
connection
information
from
your
applications
and
from
your
infrastructure
and
embed
it
into
your
applications.
So,
instead
of
having
to
import
an
SDK
for
something
like
a
state
accessing
a
state
store,
you
actually
can
just
call
the
Dapper
API
and
access
the
data
that
way
there
are
over
a
hundred
Dapper
components
today,
and
these
are
all
open
source
online.
So
definitely
encourage
you
checking
these
out
and
we
also
have
the
opportunity
to
create
our
own
components
with
Dapper.
B
All
right
awesome,
so
I'm
going
to
do
a
quick
overview
of
the
architecture
today
that
we're
going
to
be
walking
through
just
so.
You
have
a
little
bit
of
context
in
terms
of
what
we're
trying
to
accomplish
and
how
Dapper
is
going
to
make
it
easier
easier
for
us
to
get
the
solution
up
and
running
on
kubernetes.
B
B
Once
we
move
on
from
here
that
order,
API
will
receive
that
payload
from
the
front
end
and
will
essentially
publish
it
to
a
topic
in
this
case
we're
using
a
Kafka
implementation
for
the
message
broker.
This
could
be
swapped
out
with
any
number
of
back-end
components
that
are
available
in
supported
via
dapper.
B
Once
the
message
is
arrives
on
the
top
the
topic
it
will
basically
be
received
by
a
subscriber,
and
this
subscriber
is
called
the
archive
service
and
basically
it's
objective.
It's
to
take
new
orders
that
have
come
in
and
store
them
in
a
storage
account
once
those
are
available
in
this
Azure
storage
account.
Essentially
we're
also
going
to
have
a
job
that's
running
in
the
background
on
a
schedule,
so
every
15
seconds,
this
job
is
essentially
going
to
pick
up.
B
The
new
orders
that
have
come
in
do
some
processing
on
those
and
store
the
output
in
a
state
store,
in
this
case
we're
going
to
be
updating,
loyalty
points
for
people
who
are
active
users
of
our
product.
So
now
I
want
to
dive
in
specifically
to
the
service
invocation
aspect
of
this
architecture.
The
big
thing
here
is
that
we're
using
Dapper
in
order
to
get
a
lot
of
benefits
when
it
comes
to
service
invocation.
B
So
let's
take
a
look
at
how
Dapper
actually
interacts
with
and
helps
coordinate
the
services
calls
within
my
kubernetes
environment
So
within
the
front
end
application.
Instead
of
calling
directly
to
the
backend
service,
we're
going
to
call
adapter
sidecar,
so
in
kubernetes,
dopper
does
run
as
a
sidecar
in
the
Pod.
However,
you
can
deploy
deploy,
Dapper
without
containers
and,
frankly
without
kubernetes,
so
we
just
want
to
make
it
very
clear
that
this
is
just
one
way
that
Dapper
can
enable
you,
via
your
kubernetes
deployments.
B
Now
we
talked
about
the
Dapper
apis
they're
available
in
both
HTTP
and
grpc,
so
you
can
choose
whatever
makes
sense
for
you
once
you
hit
the
Dapper
sidecar,
that's
running
local
to
the
front
end
pod.
We
will
then
have
Dapper
communicate
with
the
sidecar
for
the
backend
service,
and
the
sidecars
will
always
communicate
secure
by
default.
Grpc,
and
you
can
also
get
mtls
here.
What's
also
exciting-
is
that,
in
addition
to
mtls,
we
can
actually
use
spiffy
identities
which
will
allow
us
to
set
Access
Control
policies.
B
So,
for
example,
we
could
restrict
the
front
end
from
posting
to
the
back
end
and
instead
only
allow
it
to
do
git
operations.
This
is
just
you
know,
one
of
many
examples.
In
addition,
we
also
see
that
we
can
enable
distributed
tracing,
not
just
amongst
our
services,
but
all
the
way
to
the
backing
infrastructure
for
our
adapter
components.
Sound
good
makes
sense,
awesome,
I'm,
Levin,
Dapper,
hope
you
guys
are
too
okay,
so
moving
on
service
invocation
is
great
and
a
lot
of
customers
do
use.
B
Direct,
Services
service
calls
and
somewhere
in
their
architecture
at
some
point,
but
obviously,
when
we're
building
distributed
systems,
there's
typically
a
need
for
us
to
have
some
kind
of
communication
mechanism
that
allows
a
little
bit
more
independence
of
our
services
right.
We
want
to
decouple
those
via
some
kind
of
message
broker
or
a
venting
system,
for
example,
if
we're
using
a
competing
consumer
pattern.
So
that's
where
Dapper's
Pub
sub
apis
really
come
into
play.
So
once
that
order,
API
has
been
invoked
from
the
front
end.
B
One
thing
to
call
out
that
I
didn't
mention
on
the
previous
slide
is
that
by
default,
Dapper
does
use
cloud
events
messaging
format,
but
you
can
disable
that,
for
example,
if
you're
talking
from
one
Dapper
app
to
maybe
a
legacy
app
that
has
not
been
Dapper
enabled
and
you
also
can
get
distributed
tracing
across
this
experience.
Not
just
the
service
invocation
calls
all
right.
Has
anybody
heard
this
phrase
at
any
point
in
their
career
in
cloud
computing?
Anybody
failure
is
inevitable.
B
Okay,
I
remember
I
once
like
at
the
beginning
of
my
career
at
Microsoft.
Six
years
ago,
I
remember
somebody
gave
a
presentation
where
they
said:
Azure
never
fails.
This
was
funny
because
I
work
at
Microsoft
was
like
Azure,
never
fails.
It
was
a
bit
of
a
sales
pitch
and
the
advice
that
this
person
was
given
is
that's
the
absolute
opposite
thing.
You
want
to
say
you
want
to
say
it
is
inevitable
that
all
Cloud
providers
at
some
point
could
potentially
fail.
It
is
inevitable
that
some
kind
of
call
at
some
point
will
fail
right.
B
A
pod
can
go
down,
a
node
can
go
down.
Net
Network
latency
can
get
introduced.
You
can
have
transient
failure
in
your
network
calls,
so
you
basically
have
to
build
your
distributed
applications
knowing
that
faults
is
inevitable
and
knowing
that
failure
is
inevitable.
So
it's
great
that
Doppler
provides
us.
This
service
invocation
capability,
these
Pub
sub
apis.
But
what
happens
when
we
run
into
one
of
these
inevitable
failures?
We
have
to
have
a
way
to
recover
from
that,
and
Dapper
now
has
a
capability
which
enables
you
to
do
that
and
that's
called
Dapper
resiliency.
B
So
let's
talk
a
little
bit
about
how
this
works
in
action
for
service
and
vacation
and
then
what's
great,
is
I'm
going
to
stop
talking
for
a
little
bit
and
we're
going
to
dive
straight
into
a
demo
and
get
rid
of
these
slides
sound
good.
Are
you
with
me?
Okay,
just
checking
okay,
I'm,
a
validation
girl.
You
need
to
know
you're
still
here
all
right,
so
we've
got
a
front-end
application.
Let's
say
that
this
front
end
is
not
Dapper
enabled.
B
So
in
this
case
it's
just
a
regular
deployment
running
in
kubernetes,
it's
typically
going
to
communicate
to
some
kind
of
backend
API.
In
our
case,
it's
going
to
be
that
customer
order
service
API
now
because
I'm,
not
using
any
kind
of
dapper
here,
I'm,
just
basically
calling
the
backend
service
for
my
backend
API,
so
nothing,
nothing,
crazy,
Happening,
Here,
No,
mtls
being
given
nothing
like
that.
So
let's
say
that
this
call
is
to
fail
for
whatever
reason,
one
of
the
many
examples
that
I
gave
earlier.
B
That
means
that
that
request
is
likely
going
to
fail
and
return
to
the
front
end
caller,
which
is
one
not
a
great
experience
for
end
users
and
also,
ultimately,
you
know,
because
the
failure
is
likely
transient.
It
would
be
a
lot
better
if
we
had
some
kind
of
retry
capability,
which
is.
This
is
not
new
new
Concepts,
but
it
is
nice
that
you
can
do
this
without
instrumenting.
Your
application
code
and
across
language
is
in
a
consistent
way
and
that's
what
Dapper
provides.
B
So,
let's
switch
the
scenario
and
we're
going
to
have
a
Dapper
side
car
running
alongside
our
order
API.
So
we've
essentially
dapperized
it
as
we
like
to
say
our
front-end
API
or
excuse
me
our
front
end.
Caller
will
also
now
be
Dapper
enabled
when
I
make
the
call
this
time,
instead
of
actually
just
making
the
post
to
the
service.
For
the
back
end,
API
I
will
call
the
service
invocation
in
point
via
Dapper.
So
you
can
see
here,
I'm
calling
the
V1
invoke.
B
Api
I'm,
passing
in
the
name
of
the
of
the
Dapper
app
ID
that
I'd
like
to
Target,
which
in
this
case
is
the
order.
Api
and
I'm
also
specifying
the
method
on
which
I'd
like
that
app
to
be
invoked.
You
can
see
here.
I
also
have
this
small
yaml
file
called
resiliency.
What's
nice
is
I
can
apply
a
resiliency
spec
or
policy,
that's
loaded
at
runtime
whenever
the
Dapper
or
sidecar
gets
up
and
running.
B
B
So
in
this
case,
I'm
saying
hey
after
500
milliseconds
I
want
Dapper
to
return
an
error
that
will
then
kick
off
my
retry
policies
that
I've
specified
specified
in
the
resiliency
spec
and,
in
addition,
I
can
even
have
a
circuit
breaker
policy
that
I
apply
when
the
number
of
retries
has
failed.
You
know
a
certain
number
of
concurrent
requests,
so
ultimately
the
goal
would
be
I
retry
this
and
eventually
it
succeeds
and
that's
what's
returned
to
the
front
end.
You
all
ready
to
see
this
in
action.
B
Okay,
okay,
cool
before
we
do
that.
The
last
thing
I
want
to
call
out
is
that,
in
addition
to
resiliency
policies
for
service
invocation
calls,
I
can
also
accept
resiliency
policies
for
all
of
the
Dapper
components.
Pretty
much.
So
if
you
think
about
that,
that
means
I
can
have
Pub
sub
retry
I
can
have
subscription.
Retry
I
can
even
have
retries
for
actors,
retrieving
secrets
from
a
state
store
reaching
out
to,
or
excuse
me
from
a
Secret
store
or
even
retrieving
state
from
a
state
store.
So
that's
just
something
I
wanted
to
highlight
all
right.
B
B
All
right
can
we
still
see
yeah.
It
looks
good,
so
here's
where
I'm
going
to
start
in
order
to
essentially
inject
the
timeout
that
I
need
right
in
order
to
inject
Network,
latency
I'm
going
to
use
something
called
Azure
chaos
Studio.
This
is
essentially
just
a
wrapper
around
chaos,
mesh
the
open
source
project,
and
this
is
really
critical
because
we
obviously
want
to
be
able
to
test
our
applications
and
their
resiliency
policies
dependent
on
the
type
of
failures
that
can
happen.
B
So
in
our
case,
specifically
we're
introducing
a
delay,
and
this
delay
will
essentially
kick
off
the
timeout
policy
that
we've
set
so
I'm
going
to
go
ahead
and
start.
This
I
should
have
started
it
before
I
did
that
Spiel.
That
would
have
been
good,
but
you
know
we're
okay.
So,
let's
see
if
we
can
get
this
started
all
right,
so
it
should
get
up
and
running
here
soon.
B
What
I'm
going
to
do
now
is
I'm
going
to
switch
over
to
my
terminal,
we're
going
to
close
a
few
things
out,
and
what
we
can
see
is
that
we
have
several
pods
that
are
already
deployed
to
our
AKs
cluster.
Obviously
this
could
be
a
cluster
of
your
choosing,
but
what
I
really
want
to
call
out
is
we're
going
to
start
with
the
non-dapper
enabled
scenario.
So
that's
going
to
be
my
front
end.
B
And
then
I'm
going
to
do
the
IP
address
of
the
front
end.
That
is
not
dapperized,
okay.
So
what
we
should
see
is
that
we're
sending
some
orders
in
and
we're
seeing
the
order
the
order
is
being
received.
What
we
should
ultimately
see
is
a
bunch
of
500s
right.
So
what
essentially
happened
here
is
our
Network
latency
was
injected
and
our
front
end
is
now
receiving
500
errors,
because
it
never
got
the
successful
response
that
it
was
that
it
was
waiting
for.
So
we
can
see
that's
not
an
ideal
scenario
right,
I'm
going
to
quickly.
B
Show
you
too,
what's
neat.
Just
you
know,
for
visualization
purposes.
Is
that
oh,
this
may
not
have
actually
been
running.
That's
okay.
I
was
going
to
show
you
some
live
metrics
that
showed
the
500s,
but
I
think
you
get
the
points
the
logs
are
good,
so
now
what
I'm
going
to
do
instead
is
we're
going
to
flip
over
the
scenario
and
focus
more
on
the
Dapper
enabled
pods.
So
you
can
see
here,
I
have
a
front-end
application
called
front
end
Dapper.
This
is
the
exact
same
application
code.
Nothing
has
been
changed.
B
I
want
to
make
that
very
clear.
I
have
not
instrumented
the
code
with
poly
I'm,
not
using
any
kind
of
resiliency
libraries
or
retry
policies
within
my
application
code,
I'm
just
setting
a
resiliency
policy
via
Dapper.
So
if
you
want
to
see
what
this
looks
like
and
I'm
going
to
zoom
in
just
to
make
sure
everybody
can
see,
you
can
see
here,
I
have
a
resiliency,
spec
and
I'm,
applying
it
to
the
Dapper
resiliency
namespace,
where
my,
where
my
application
workloads
are
running
so
I
want
to
highlight
a
few
different
sections
here.
B
The
first
is
the
actual
policy
section,
so
you
can
see
here:
I
can
set
policies
around
timeouts
I
can
set
policies
around
circuit
breakers
and
I
can
also
set
retries
in
general,
so
the
timeout
that
I've
set
is
500
milliseconds.
This
is
what's
going
to
trigger
Dapper.
To
tell
my
front
end
application,
hey
the
timeout's
been
met.
An
error
has
been
returned
once
that
timeout
has
been
met
once
I
apply.
The
resiliency
policy
we'll
see
that
I
can
use
one
of
many
policies
that
are
specified
in
this
case.
B
B
We
can
also
set
a
circuit
breaker,
and
in
this
case,
what
we're
saying
is
once
20
consecutive
requests
have
failed,
we're
going
to
open
the
circuit
breaker.
Then
we're
going
to
wait
60
seconds
and
once
the
circuit
breaker
becomes
half
open.
We
can
essentially
let
just
a
Max
request
of
one
order
come
through.
B
What's
neat
is
we
can
then
specify
targets
and
targets
and
Scopes
are
really
used
together.
So
in
this
case
our
front
and
Dapper
app,
which
you
can
see
here,
hopefully
our
front
end.
Dapper
app-
will
use
a
specific
Target
when
talking
to
the
customer
order
service.
So
whenever
the
front
and
Dapper
app
is
targeting
the
customer
order
service,
it
will
use
that
General
timeout
policy
and
it
will
use
that
retry
forever
retry
policy
in
terms
of
components.
We
also
set
up
a
retry
policy
for
the
publishing
to
the
Kafka
topic.
B
Here,
all
right
make
sense
all
good
all
right,
so
our
KS
test
should
still
be
running,
which
it
is
so
I'm
going
to
switch
back
over.
The
resiliency
policy
is
already
running
in
our
cluster,
which
is
amazing
and
I'm
going
to
now
use
the
DDOS
if
I
tool
and
I'm
instead
going
to
Target
the
front
end
IP
for
the
Dapper
enabled
service
and
before
I
do
that
I
do
want
to
pull
up
the
log.
So
we
can
see
what's
happening
in
real
time.
B
B
So
here
we
go.
What
we
should
see,
which
you
can
hopefully
see
is
that
on
the
left
side
we
see
the
orders
are
getting
sent,
but
we
don't
see
any
500s
right
because
they're
never
getting
returned
to
the
front
end
instead,
what's
happening
is
that
Dapper's
sidecar
is
going
through
that
retry
forever
policy.
So
the
front
end
user
will
never
see
this
in
an
Ideal
World.
If
we
set
up
the
Jitter
and
all
that
stuff,
we'd
eventually
see
the
the
call
succeed
in
this
case.
B
It
can't
because
we're
applying
this
network
Chaos
on
every
single
call.
The
last
thing
that
I
want
to
show
before
I
hand
it
over
to
Alice
is
that
we
also
set
that
retry
policy
on
the
the
publisher
as
well.
So
if
you
take
a
look
at
our
sorry,
got
a
lot
going
on
here
bring
this
down.
Maybe
you
can
see
a
little
better.
B
So
if
we
take
a
look
at
the
customer
order
service,
if
you
remember
it
had
a
target
for
that
Kafka
pod.
So
when
we're
publishing
orders
and
there's
a
certain
timeout,
it
should
kick
in
that
retry
policy
and
eventually
the
circuit
breaker.
So
we're
going
to
take
a
look
at
the
customer
order
service
and
we
can
see
here
that
we're
attempting
posts
and
we're
getting
back
500s
and
I'm
going
to
stop
this
from
Auto
scrolling.
So
we
can
see.
B
So
you
see
that
the
application
code
is
getting
a
500
saying
we
are
not
able
to
successfully
publish
this
order
and
when
we
look
at
the
order
service,
Dapper
D
logs,
we
should
see
that
the
Dapper
API
is
being
called
and
that
it's
going
through
the
circuit
breaker
process.
At
this
point,
so
the
circuit
breaker
has
now
been
opened
and
now
it's
not
letting
any
of
those
calls
through
all
right,
awesome.
Well,
for
the
sake
of
time,
I'm
going
to
pass
it
over
to
Alice
Alice.
B
Had
you
already
switched
this
over
so
while
she's
doing
that,
I
want
to
talk
a
little
bit
about
what
Dallas,
what
Alice
is
going
to
be
covering
in
the
next
segment.
So
we've
talked
about
how
we
can
make
our
applications
more
resilient
without
changing
our
application
code.
Alice
is
going
to
go
into
another
important
and
critical
aspect
of
having
reliable
applications
and
that's
going
to
be
enabled
through
the
distributed
lock.
Api
that
was
made
available
in
Dapper
1.8
I
always
take.
A
A
All
right,
cool,
I
know
and
as
Kendall
mentioned
right,
we're
showing
a
lot
here
so
I
want
to
make
sure
everyone's
still
with
us.
We're
still
trying
to
do
a
lot
for
resilient
applications.
All
right-
and
this
is
just
one
more
step
in
that
so
on
on
up
here
on
the
screen.
You
can
kind
of
see
our
architecture
and
right
now,
we've
made
it
through
the
back
end.
We've
done
some
distributed
calls
and
they
are
all
resilient
as
Kendall
showed.
A
What
we
want
to
do
now
is
we
have
a
loyalty
job
and
the
job
of
this
service
is
essentially
to
pick
up
all
of
the
orders
that
have
been
processed
and
add
the
loyalty
points
for
each
customer.
This
is
important
right.
We
want
to
we
put
our
customers
first,
so
we
want
those
loyalty
points
to
be
right
and
we
have
decided
to
use
the
distributed
lock
API
in
Dapper
to
ensure
that
strong
consistency
on
the
database,
because
we
have
multiple
replicas
of
the
Loyalty
job
running
and
this.
A
A
So
you
can
see
here
that
we
are
like
using
the
distributed
lock
API
in
Dapper,
and
what
exactly
does
this
give
us
right?
The
purpose
of
a
lock
is
to
ensure
that,
among
you
know,
several
different
application
instances.
You
are
only
going
to
have
one
application
instance
at
a
time
accessing
that
data
and
updating
it
or
doing
a
put
on
whatever
the
data
might
be.
In
this
case,
we
have
again
the
Loyalty
job
and
we
are
deciding
to
update
that
those
loyalty
points.
A
A
We
are
running
again
two
replicas
of
this
app
and
you
can
see
it's
using
the
redis
implementation
of
the
Dapper,
lock
API
and
the
next
one.
Here
you
can
see
that
the
once
the
lock
is
complete
or
once
the
lock
has
finished
the
same
replica
can
actually
release
the
lock
and
then
so
other
replicas
of
the
Loyalty
service
or
the
Loyalty
job
can
come
in
and
do
the
work.
A
I
should
also
mention
that
Dapper
is
a
lease
based
lock,
and
so
it
does
have
a
Timeout
on
it
or
a
time-based
system
where,
after
60
seconds,
you
know,
one
replica
will
release
that
lock,
so
other
replicas
can
come
in
and
do
the
work.
This
also
helps
prevent
failure,
because
if
you
run
into
an
exception
in
that
first
replica
that
has
the
lock
you're
always
going
to
release
it
after
60
seconds
and
one
more
thing
I
should
call
out
about
distributed
lock.
A
Is
that
if
you
had
two
different
applications
and
they're
both
accessing
the
same
data,
but
you
can
access
them
concurrently,
so
from
two
different
apps
Dapper
is
going
to
allow
you
to
access
those
at
the
same
time.
So
both
different
applications
can
take
a
lock
on
that
data
and
maybe
they're
updating
different
pieces
of
it.
Or
you
know
your
system
allows
for
both
instances
to
update
that
data
at
the
same
time.
A
There
yeah
okay,
as
Kendall
mentioned.
You
know
we
are
doing
these
live
so
another
round
of
applause
for
Kendall's
down
on
Prime
yeah
go
all
right,
so
I'm
switching
over
to
canines
here
as
well-
and
this
is
you
know-
the
screen
looks
a
lot
like
Kendall's
did.
This
is
the
same
cluster,
surprise,
surprise
and
the
same
applications
and
you
can
see
on
the
left
here.
A
You
can
see
I
have
multiple
instances
of
this
customer
loyalty
job
running,
and
this
is
that
loyalty
job
I
was
talking
about
right.
This
is
the
one.
That's
updating
those
loyalty
points
we
want
to
make
sure
are
really
consistently
right
for
our
customers.
So
we
have
the
customer
loyalty
job
two
instances
of
that,
and
then
we
have
this
customer
loyalty
job
no
lock.
A
Lots
of
orders-
great
great
customers,
okay
and
you
can
see
I
have
you
know
this
is
my
no
lock
code
and
what
it's
doing
in
both
cases,
it's
going
to
actually
be
processing
these
orders
simultaneously
in
a
lot
of
cases
right.
So
there
is
no
mechanism,
that's
telling
you
know
the
app
hey,
don't
be
accessing
the
same
code
at
the
same
time.
So
a
lot
of
these
orders
will
be
exactly
the
same
on
the
left
pod
and
on
the
right
pod.
A
So
you
can
see
actually
on
some
of
these
time
stamps
they
do
actually
match
up
from
the
goods.
So
you
know
we're
updating
the
Loyalty
customer
id9,
we're
updating
nine
on
the
right
here.
So
again,
this
is
just
the
scenario
where
the
lock
is
not
being
used
and
we
have
multiple
instances,
updating
the
loyalty
points
and
then
saving
them
to
a
database.
A
A
And
this
again,
as
I
mentioned
earlier,
so
this
loyalty
job
is
checking
every
15
seconds
to
make
sure
that
there's
no
orders
left
to
be
processed
right.
It's
looking
in
this
case
within
Azure,
blob,
storage
and
checking
to
see
hey,
you
know,
are
there
orders
for
me
to
process?
What's
going
on?
So
let's
run
another
load
test,
see
what
happens.
A
Okay,
we
see
some
orders
right.
There
you
go.
Things
are
happening,
okay,
so
what
we
have
on
the
left
hand
side
here
is
we
have
again
the
multiple
replicas
of
the
same
application,
so
I
have
replica
one
on
the
left
replica
two
on
the
right
and
you're
going
to
see
some
lock
failures,
and
this
is
good.
This
is
what
we
want
to
see
so
I'm
attempting
to
process
this
one
order
ID
on
the
right
here
or
on
the
left,
and
it
is
not
being
processed.
A
So
this
one
successfully
locked
it
and
then
you
can
see
on
the
right
hand,
side
that
it
wasn't
actually
locked,
so
this
had
failed
to
lock
order.
So
for
each
one
of
these
you're
going
to
see
that
the
order
on
the
left
hand,
side
would
be
processed
and
the
order
on
the
right
would
not
be,
or
vice
versa
right.
A
So
one
pod
is
going
to
be
updating
those
loyalty
points
and
the
other
one
will
not
be
and
this
if
this
is
essentially
going
to
ensure
that
each
one
of
these
customer
orders
gets
updated
in
the
correct
format
and
doesn't
you
know
overwrite
or
clobber
each
other
in
the
database
at
the
end
of
the
day
cool?
So,
let's
check
out
now
what
that
looks
like
in
the
redis
database.
So
if
you
remember
from
my
architecture,
I
had
these
writing
to
that
redisdb,
so
I'm
going
to
switch
over
now
just
to
my
redis
database
today.
A
This
is
just
running
on
my
AKs
cluster
and
I'm,
going
to
check
out
what
those
orders
look
like.
So,
for
instance,
coming
in
here,
let's
check
out
that
no
lock
one
first
okay,
because
I'm
going
to
check
out
no
lock
because
I
want
to
see.
You
know
what
that
looks
like
if,
in
the
case,
I
did
not
use
a
lock
in
my
code.
So
I
sent,
if
you
remember,
two
different
load
tests.
Each
of
them
had
10
orders
in
them,
so
there
should
be
around
20
orders
and,
in
the
right
hand,
side
here.
A
So
on
the
no
locks
case,
I
have
a
ton
of
different
scenarios
and
I'm
getting
different
numbers
every
single
time,
sometimes
I'm
getting
up
to
35,
sometimes
I'm
getting
37.
You
know
these
are
processing
entirely
different
amounts,
as
it
goes
as
the
lock
you
know,
writes
over
and
over
to
this
database
or
as
the
no
lock
app
writes
over
and
over
to
this
database.
But
you
know-
and
you
can
see
33
all
very
different,
but
you
know
if
I
come
in
here
and
look
at
the
case
where
there
was
the
lock
I
think
networklings.
A
Well,
we
have
19.,
19
came
through
Kendall,
okay.
Well,
we
have
now
on
this
case.
We
have
20
orders
that
came
through
for
the
customer
load
service
with
the
lock,
and
this
is
essentially
going
to
be.
If
there
wasn't,
we
had
a
bit
of
network
latency
Kendall
said
so
we
had
a
bit
of
latency
here,
but
there
would
be
20
orders
every
single
time,
so
we
have
a
few
instances
there
you
go
it's
coming
through
now.
Actually
Kendall.
A
But
okay
I
think
it's
just
coming
through
still
so
yeah
we
have
a
few
they're
common.
All
the
orders
are
coming
through
and
at
the
end,
you're
gonna
have
20
orders
for
every
single
instance
of
this
in
this
redisdb.
So
again,
this
is
two
of
exactly
the
same
application,
one
of
them's
using
the
lock
one
of
them
isn't,
and
this
is
updating
the
loyalty
points
you
know,
and
you
do
have
that
correct
order
count
at
the
end
of
the
day,
awesome
all.