►
Description
This talk was part of AsyncAPI Online Conference:
- AsyncAPI Initiative website - https://www.asyncapi.com/
- You can also watch this talk as part of full conference recording with audience questions in the LiveChat - https://youtu.be/oMSzGc5bDr4?t=1017
TALK ABSTRACT:
An architects view on the lessons learnt on implementing event based patterns for large insurance & wealth management clients. Taking a look at the pain that can be caused when event driven goes wrong in a stateful world that happens to be supporting a 24 x 7 business environment. Along with the remediation to try ensure the business and ops teams don't want to run you out of town. This is not a product based talk but more a view from the trenches.
SLIDES:
https://www.slideshare.net/PaulTaylor41/unhappy-path-dealing-with-bad-events-233113597
A
Good
morning
good
afternoon,
and
if
you're
here
in
the
lower
part
of
the
southern
hemisphere,
like
me
good
evening,
I'm
Paul
Taylor
a
strategic
advisor
with
mule
soft
and
it's
my
absolute
pleasure
to
have
the
opportunity
to
talk
to
you
at
the
first
async
API
conference,
for
which
the
async
API
community
should
be
commended
on
putting
together
in
such
a
short
space
of
time
and,
of
course,
making
the
event
both
virtual
and
free.
The
subject
of
my
talk
today
is
the
unhappy
path
and
dealing
with
bad
events.
A
Although
I
I
guess,
the
subtitle
here
could
have
been
things
I
wish,
I'm
known
before
implementing
event-driven
architecture
at
scale.
I
should
also
add
the
lessons
learned
here,
one
on
every
project:
I'm,
not
sure
that
will
reflect
too
well
on
my
abilities
to
design
an
implement
but
a
more
the
product
of
a
long
and
varied
career,
and
hopefully
the
points
I
will
cover,
could
save
you
some
time
and
pain
in
the
future.
A
So
a
word
on
the
content
I
plan
to
present
it's
really
aimed
to
provide
some
pointers
to
meet
the
business
requirements.
You're
trying
to
meet
with
your
event
based
technology
stack
first
I'm
assuming
you
do
have
business
requirements
right,
I
mean
no
one
in
IT
would
implement
technologies
because
it's
new
cool
and
shining
would
they.
But
anyway,
the
serious
point
I
want
to
make
here
is
to
implement
asynchronous
patents.
World
takes
hard
work
if
your
requirements
could
be
met
with
rest.
Request/Response
bans
do
that.
A
The
unhappy
paths
are
a
hell
of
a
lot
easier
to
deal
with
the
synchronous
flows.
When
you
can
send
the
response
back
to
the
consumer,
then
they
offer
asynchronous,
where
you
have
many
decisions
to
make.
How
to
handle
failures.
I
should
also
be
clear
on
what
unhappy
path
and
bad
events
are
mainly
because
they're
the
focus
of
this
talk
and
it
would
be
good
to
level
set
and
I'm
happy
path,
is
the
flow
you
take
when
a
unexpected
error
or
failure
occurs.
Ie,
not
the
normal
happy
flow
and
bad
events
well,
they're.
A
Just
that
events
they're
caused
problem,
either
due
to
their
format
or
content
or
just
bad
timing
when
they
were
meant
to
be
processed
I.
Guess,
to
give
you
some
background
on
me:
I
should
say:
I
try
to
take
life
not
too
seriously
and
in
keeping
with
the
immortal
lines
from
Monty.
Python.
Worse
things
happen
at
sea,
so
keep
that
in
mind
June
for
an
presentation.
I
want
to
start
by
talking
about
an
event,
its
shape,
its
size
and
its
intent,
though
before
I
do
if
anyone
was
watching
is
in
charge
of
an
enterprise,
economical
dateable.
A
Firstly,
you
have
my
sympathy
and
second
and
most
importantly
and
I
should
point
out
that
all
using
this
presentation
on
my
own,
not
my
employer,
so
if
I
upset
you
have
any
my
statements
about
data,
modeling
I
will
apologize.
Now,
if
you
haven't
come
an
enterprise
economical
data
model
before
there
can
be
huge
beasts
and
tried
to
capture
and
define
data
across
all
use
cases
in
the
enterprise.
A
This
means
they
can
sometimes
be
a
little
unwieldy
to
work
with,
to
say
the
least,
and
although
at
a
logical
level,
you
can
see
the
benefits
trying
to
directly
implement
that
info
can
be
an
issue
now.
I
must
admit
my
bias
against
large
scale.
Enterprise
data
model
stems
from
working
on
projects
where
it
was
an
unbreakable
design
principle
that
the
canonical
data
model
had
to
be
used
for
all
interactions.
A
Now
trying
to
get
a
large
data
model
to
play
nice
with
bandwidth-limited
devices
was
a
major
issue,
to
be
honest
and
in
fact,
that
unbreakable
design
principle
had
to
bend,
shall
we
say
just
a
little
may
anyway,
that's
my
faults
under
all
that,
so
drilling
down
on
an
event.
What
should
an
event
contain
to
be
most
useful?
Yes,
the
core
is
the
significant
event
or
state
change,
but
is
that
it?
My
point
of
view
on
this
is
as
a
good
corporate
citizen
or
even
netizen?
A
You
need
to
give
some
full
on
the
value
of
the
event
you
are
producing
and
how
you
can
increase
the
value
of
it
and
help
others
realize
that
value
make
it
too
hard
to
use
and
the
ROI
doesn't
make
sense
you're
going
to
struggle
to
get
traction.
Of
course.
Not
all
events
are
equal
in
value
and
they
associated
care.
You
can
such
kind
of
give
them.
The
reality
is
that
some
vents
are
even
disposable.
A
However,
the
first
point
I'd
like
to
make
in
it
may
seem
obvious,
but
is
make
your
events
specific,
not
generic.
We
often
try
to
make
things
reusable,
but
going
down
this
generic
path
means
you
end
up,
sometimes
over
complicating
or
having
to
constantly
enrich
what
you're
trying
to
achieve
with
additional
metadata.
For
the
event
to
be
understood
or
worse,
you
end
up
doing
both
over
complicating
it
over
enriching.
A
So
as
an
example,
if
a
customer
updates
an
address,
doesn't
undress,
update
event,
make
more
sense
for
you,
then
a
generic
custom
update
event,
which
could
include
a
whole
host
of
different
updates.
It
could
be
you
know
their
their
telephone
number.
Their
email
address,
their
name.
Change,
for
me
is,
is
the
formula
is
often
the
best
practice
where
you
have
that
specific
event,
so,
once
you've
decided
on
the
event,
you
need
to
think
about
the
event
envelope.
What
additional
data
does
it
make
sense
to
include
beyond
the
captured
state
change?
Do
you
new?
A
You
need
to
include
some
time-dependent
data
along
with
that
change
may
be
given
the
context
of
the
change
ie.
The
change
was
set
in
motion
by
a
condition
being
met
or
was
an
event
generated
by
a
third
party
may
be
it
require
authorization.
The
Audion
auditing,
the
governance
team
would
thank
you
for
that.
Impasse
depend
only
event
string
because
it
gives
them
that
source
of
truth.
A
This
can
also
hold
true
for
triumph
of
channel
information.
Was
this
far
a
mobile
channel
or
web
app?
Maybe
thinking
about
what
you
can
enrich
an
event
with
reference
data
can
often
be
useful.
Is
there
a
requirement
for
each
and
every
subscriber
to
that
event
to
end
up
doing
the
same?
Look
up
to
obtain
some
piece
of
reference
data?
Would
it
make
sense
to
actually
inject
that
into
the
event
at
the
time
it's
produced?
A
Also,
you
need
to
think
beyond
the
initial
play
of
that
thing.
Is
there
a
chance
that
this
event
may
end
up
going
down
the
happy
path?
Will
the
event
contain
enough
information
for
you
to
replay
that
event
in
the
future?
You
know,
is
it
the
right
shape
to
support
the
future
requirements
now,
I'm
not
and
I
mean
I'm
not
talking
about
populating
the
whole
enterprise
canonical
data
model.
In
the
event,
very
few
consumers
would
thank
you
for
that.
Not
to
mention
the
infrastructure
network
and
security
teams
be
pragmatic
with
the
event
design,
I.
A
Think
history
has
taught
us.
Religious
walls
never
turn
out
well,
and
the
same
can
be
said
for
event.
Modeling.
My
advice
would
be
passionate
with
your
point
of
view
maintain
desireless
in
principles,
but
stay
pragmatic,
keep
in
mind
those
non-functional
when
designing
the
events.
You
know
governance,
auditing
load
handling
here,
they're
all
need
to
be
considered
as
the
wider
implementation.
You
know,
and
as
I
said,
can
you
handle
the
unhappy
path
with
that
event,
when
it's
replayed
in
the
future?
A
Now
it
may
be
the
version
1
of
your
event,
definition
has
to
change
over
time.
That's
of
course.
Ok.
It
means
you
have
traction,
but
just
as
you
no
doubt
do
with
your
REST
API
designs
currently
tread
carefully
with
those
breaking
changes.
Now
I
fully
expect
async
API
to
help
here
as
more
and
more
tolling
supports
here,
allowing
events
to
be
designed,
understood
and
consumed
in
a
consistent
way.
A
On
a
discuss
or
during
you
know,
and
even
if
you
don't
have
a
need
now,
I
would
expect
at
some
point
on
your
event-driven
architecture,
journey
you're,
going
to
come
across
event
ordering
and
the
need
to
maintain
it.
Of
course,
plenty
of
use
cases
don't
require
strict
ordering,
but
it's
not
difficult
to
think
of
those
that
do
banking
and
insurance,
and
you
could
say
even
more
importantly,
especially
in
these
times
healthcare
they're,
all
obvious
use
cases
that
require
the
strict
ordering
of
events.
A
Ordering
of,
of
course,
sounds
easy,
especially
when
you
have
an
append-only
log.
But
how
do
you
maintain
that
order
down
through
the
consuming
Network?
You
may,
as
example,
have
systems
a
record
that
still
run
on
big
iron
in
the
back
office
that
will
refuse
to
update
and
their
state
changes
are
received
in
a
CERN,
ordered
sequence,
take
insurance
use
case
and
the
events
associated
with
that.
You
know
you
raise
a
claim,
but
you
can't
pay
a
claim
until
the
claim
has
been
accessed.
So
these
events
in
the
claim
process
need
to
be
ordered.
A
So,
what's
the
issue
of
ordering
well,
first,
let's
look
at
the
simplest
way
to
maintain
order
and
the
pitfalls
you
may
come
across
if
we
have
an
ordered
event
stream
or
associated
broker.
That
maintains
order
for
us
and
we
subscribe
with
a
single
consumer
for
all
updates,
then
making
sure
each
event
is
consumed
and
processed
before
moving
on
to
the
next.
A
In
essence,
we've
created
a
end-to-end
single
threaded
flow
for
the
normal
paddle
path.
We
can
assure
ordering
is
maintained,
pretty
straightforward,
but
the
problem
proton
difficult,
c4
start
scalability.
You
can
only
consume
events
as
fast
as
a
single
threaded
single
consumer
to
process
them
for
low
volume.
That
might
not
be
a
problem,
but
what
if
the
new
widget
service
you
introduced
just
goes
viral,
you
may
find
the
producer
crates
events
far
quicker
than
you
can
process
with
this
architecture.
This
would
give
you
an
ever-growing
backlog
and
we
all
know
scaling
vertically
has
its
limits.
A
A
Second,
what
happens
if
you
have
problem
certain
events?
These
are
the
bad
events
I
mentioned
earlier
I.
Let's
say
there
are
data
issues
with
them
or
a
downstream
platform
reaches.
It
refuses
to
process
it
due
to
stealth
they
being
stored.
What
do
you
do?
Stall,
the
event
string,
consumer
page,
the
operations
team
wait
until
someone
comes
along
and
fixes
the
events
manually
and
restarts
the
consumer.
A
It's
actually
how
it's
handled
more
commonly
than
you
would
think,
but
to
be
honest,
it's
usually
a
less
than
optimal
solution
and
can
lead
to
a
lot
of
upset
stakeholders
in
pretty
short
space
of
time,
especially
if
you
bring
the
business
to
a
standstill.
Why?
The
issue
is
sorted
some
of
the
techniques
to
mitigate
against
these.
Well,
if
you
need
more
throughput
that
can
be
achieved
with
a
single
consumer,
you
of
course
first
need
multiple
life.
Consumers
make
sense
right
and
an
identify
to
correlate
related
events.
For
example.
A
A
Then
you
need
a
way
to
ensure
that
those
for
a
single
entity
are
grouped
for
consumption
to
ensure
the
order
is
maintaining
to
end.
You
can
do
this
either
on
storage
by
storing
in
two
different
partitions,
which
then
have
dedicated
brokers
or
use
a
broker
that
hides
in
flight
events
based
on
the
same
identity.
So
if
you
do
have
those
multiple
brokers
and
three
updates
in
an
event
string
for
account
number
42,
for
example,
when
the
first
of
those
is
consumed,
the
other
consumers
are
shielded
from
knowing
the
other.
A
Two
updates
exist
until
that
first
event
has
completed
okay.
So
now
we
have
multiple
users,
so
we
can
get
the
fruit.
What
we
need
and
we've
got
resilience
and
still
retain
ordering-
that's
all
great,
but
what
about
when
events
get
stuck
ie
the
system
of
record
for
account
number
42,
is
locked
and
won't
accept
any
updates.
For
that
account.
You
see
the
problem.
Are
we
back
to
stalling
consumers
and
having
the
business
upset
with
us?
A
Okay,
so
often
we
need
to
consider
what
happened
when
and
why
so.
Turning
to
event
observability
one
of
the
big
problems
when
end
up
down
the
unhappy
path
is
sundering
understanding
how
you
got
there
in
the
first
place,
and
also
you
know,
usually
you
have
to
reach
the
understanding
pretty
quickly,
especially
where
you
have
a
pressing
production
issue
to
deal
with.
It's
very
frustrating
when
you
have
to
rely
on
a
manual
or
government
process
and
then,
after
a
lot
of
effort
scrolling
through
the
logs
to
still
only
together,
answer
hours
or
days
after
the
issue.
A
Even
worse,
though,
is
when
you
don't
have
the
correct
logging
in
place
and
need
to
turn
up
the
debug
log
and
then
just
wait
for
it
to
go
wrong
again.
You
generally
don't
win
friends
with
that.
Ideally,
you
want
to
have
as
much
visibility
as
possible
and,
of
course,
enterprise,
logging
and
visualization
platforms
are
available,
such
as
El
Casablanca
cetera,
and
they
can
go
some
way
to
help
in
if
implemented,
cross
each
layer
in
the
enterprise
a
key
point
of
courses.
A
The
end-to-end
Valencia
vents
becomes
a
key
weapon
in
resolving
unhappy
paths
or
bad
event,
issues
the
quiz.
The
question
does
get
raised
as
to
what
does
end-to-end
practically
mean,
for
example,
do
you
have
the
ability
to
observe
the
event
producers?
Can
you
observe
the
consumers
that
may
fall
to
the
system
of
record?
And
again,
can
you
observe
the
system
of
record,
given
that
many
enterprises
have
a
varied
mix
and
age
of
technologies,
applications
and
platforms?
Rarely
can
you
get
a
complete,
unified,
end-to-end
view?
A
A
Also,
when
building
out
the
observability
give
some
thought
to
spawned
events,
that
is
events
caused
by
other
events.
Do
you
have
a
way
to
link
these
together?
Doing
that
is
incredibly
powerful
in
the
end-to-end
issue
hunt,
you
will
find
various
tracing
frameworks
that
were
developed
by
industry.
Powerhouses
such
as
Facebook
and
Google.
These
are
very
useful
in
helping
you
work
through
your
problems.
Even
if
you
don't
use
the
actual
framework,
they
can
certainly
seed
ideas.
A
Now,
let's
talk
about
a
feedback
loop,
even
in
the
age
of
cloud,
elastic
scaling
and
platform
size
to
handle
Internet
load
events
apply
in
a
break
or,
as
it's
commonly
known
back
pressure
via
a
feedback.
Loop
may
seem
unnecessary,
especially
with
the
event-driven
architecture.
The
consumers
can
just
consume
at
the
level
they're
comfortable
with
right.
Well,
maybe,
if
ever
greenfield
site
and
no
legacy,
you
can
imprint.
A
Reacting
and
implementing
support
for
the
back
pressure
pan
is
often
implemented
during
their
in
the
integration
platform,
so,
for
example,
mule
soft,
obviously
example
of
where
to
do
this
and
where
you
can
actually
understand
the
feedback
from
the
consumer
or
target
system
to
use
back
pressure.
You
implement
a
mechanism
indicating
that
the
receiver
or
subscriber
is
unable
or
unwilling
to
continue
with
the
current
applied
load.
A
Limiting
Shuma
concurrent
fred's
is
a
useful
way
to
limit
the
impact
on
the
system
under
stress,
or
you
can
also
rate
limit
by
throttling
the
transaction
per
second
account
ie.
How
quickly
are
you
pushing
those
events
down
to
the
platform?
Keep
in
mind
that
you
know,
because
you've
got
cloud-based
elastic
infrastructure.
You
can
usually
just
keep
scaling.
You
know
that's
great,
but
you
won't
be
thanked
for
reducing
a
system
and
record
to
a
deadlocked
mess
by
spinning
up
numerous
cloud
instances
you
need
to
think
of
the
bigger
picture
and
the
overall
impact
of
the
enterprise.
A
A
Now
III
topic,
which
always
generates
debate
ya.
Coupling
is
really
a
fun
architectural
issue
and
tight
coupling
is
the
root
of
all
evil
right.
How
often
have
you
heard
that
embed
in
there
or
some
kind
of
variation?
It's
been
a
lot,
but
we're
talking
about
a
vendor
of
an
architecture.
So
what
does
tight
coupling
even
mean
with
an
architecture
by
design?
That's
meant
to
be
decoupled.
A
Well,
for
me,
in
the
end,
architecture
occasionally
need
many
things.
It
could
mean
consumers
bound
to
a
large
set
of
event.
Attributes
making
changes
and
versioning
of
your
event
is
difficult
or
maybe
you've
got
some
kind
of
complex
orchestration
of
events
with
hard
dependencies
between
ie
I've
been
one
has
to
finish,
and
it
has
some
relationship
with
event
2
and
you
could
end
up
in
some
kind
of
infinite
loop.
A
You
know
there's
all
kind
of
hard
dependencies
you
want
to
avoid,
but
the
one
I
wanna
talk
to
you
about
now
is
the
coupling
of
the
system
availability
most
of
the
space
services
to
be
available
when
we
want
them
at
our
convenience,
not
all
the
convenience
of
others.
All
the
companies
we
deal
with
I
know
from
experience.
A
surprising
number
of
people
want
to
pay
their
bills
or
check
balances
at
3
a.m.
in
the
morning.
Don't
ask
me
why
I
just
know
they
do.
A
A
Well,
for
real-time
query
operations,
you
can
of
course
fall
back
to
reduce
functionality
or
use
cut
their
cash
data
and
provide
that
back
to
the
client,
while
the
system
of
record
has
downtime
and
using
a
pattern
like
CQRS,
that
is
command.
Query
response.
We
segregation
can
help
here
by
having
clear
paths
to
handle
both
the
read
in
the
light
using
events.
Of
course,
we
can
decouple
those
write
operations
or
state
changes
and
then
waiter
the
system
record
is
available
before
having
them
consumed.
A
Another
caution
here
you
need
to
make
sure
that
is
clearly
communicated,
that
this
pattern
is
implemented
and
try
to
ensure
that
client
applications
don't
make
assumptions
or
when
state
changes
will
be
available.
So
this
may
mean
moving
towards
a
mindset
of
eventual
consistency
rather
than
distributed
transaction
type
of
implementation,
which
traditional
enterprises
may
be
more
familiar
with.
A
A
A
They
will
try
to
process
that
event
before
it's
moved
to
the
DL
queue,
and-
and
that's
done
so
that
you
can
handle
transition
trend,
transient
type
errors.
So,
for
example,
a
system
a
record.
Maybe
maybe
you
have
a
HDTV
time
out
the
next
time
around
it
works.
You
don't
want
to
necessarily
fail.
Those
have
been
straight
away.
A
So
when
you
do
hit
an
error,
though
it
will
move
the
event
to
the
DL
queue
and
that
can
unblock
a
bottleneck
and
I'll
allow
later
events
to
be
picked
up
for
processing,
but
when
you're
trying
to
process
events
in
order,
it
in,
of
course,
gives
us
an
organ
problem.
We
take
our
example
from
earlier,
but
now
use
dl
q.
Let's
assume
we
have
a
problem
with
event
1
for
a
capital,
T
2.
We
can
a
move
that
bin
to
the
dead
letter
Q.
A
But
what
do
we
do
now
with
events
2
&
3
for
that
same
account
without
full?
From
our
part,
the
default
behavior
is
usually
events.
2
&
3
are
ready
to
be
consumed
as
soon
as
event,
1
is
moved
and
we
then
propagate
out
of
order.
Events
across
the
enterprise,
you
know
exactly
what
we
were
trying
to
avoid
now,
given
that
most,
if
not
all
deal
queues
are
actually
a
generic
cure,
partition
that
has
just
been
designated
to
be
Odile
dlq.
It
actually
works
exactly
the
same
as
the
original
ven
store
as
recon
ordering.
A
So
how
do
we
handle
this?
Well,
one
way
is
to
have
the
consumer
check
if
any,
pending
failures
exist
for
the
event,
identity,
ie
does
account
for
t
to
have
an
event
in
the
DL
cube.
Now,
if
this
isn't
possible,
if
it
isn't
possible
for
you
to
directly
query
against
the
dlq
data
store,
for
example,
your
dlq
might
not
support
a
search
type
operation.
You
may
need
to
maintaining
the
external
operational
database
that
can
be
used
to
track
failures.
A
You
know
updated
each
time,
you
have
a
failure
and
remove
when
it's
completed,
and
that
will
give
you
an
idea
if
there
are
any
pending
failures
that
exist.
If
there
are,
you
know,
what
do
we
need
to
do?
We
need
to
move
the
subsequent
events
and
quarantine
them.
You
know
just
like
us
place
event
two
and
three
into
quarantine
in
the
dlq.
To
ensure
they're
not
processed
out
of
order,
you
may
have
some
corrective
maintenance
done.
A
Someone
may
have
to
come
and
look
at
the
events
figure
out
what
they're
gonna
do
move
them
back
once
they've
been
fixed
or
you
know,
maybe
delete
them.
If
that's
the
appropriate
course
of
action,
it's
not
pretty,
but
it
helps
automate
as
much
as
possible
the
failing
events
and,
of
course,
it's
really
preferable
to
only
have
to
deal
with
one
affected
customer
rather
than
have
all
customers
being
blocked.
A
A
One
of
the
characteristics
of
high
levels
of
automation
is
when
things
go
wrong.
They
can
go
wrong
at
tremendous
speed
before
you
know
it,
you
can
have
a
big
steaming
pile
of
issues
to
resolve
catching.
These
early
can
be
a
challenge,
especially
if
the
failure
is
subtle.
One
example
of
this
I
came
across
was
when
a
downstream
process
to
send
out
communications
to
customers.
Emails
in
fact
had
a
failure
in
send
the
problem.
A
The
first
sign
that
we
had
that
something
was
wrong.
Was
a
number
of
vents
causing
write
backs
to
the
system
and
record
went
to
the
roof?
In
fact,
the
operations
team
remarked
in
passing
how
old
the
cloud
infrastructure
was
holding
up
under
the
load.
So
here
we
have
a
valley
producer,
creating
conformant
events,
but
just
making
incorrect
assumptions
now
giving
this
scenario.
Caching,
these
type
of
issues
can
be
difficult.
They
could
be
viewed
as
edge
cases
and
each
type
of
failure
like
this
may
be
different,
so
difficult
to
code
in
advance.
A
The
best
way
I
come
up
to
handle
this
issue,
but
if
you've
already
solved
it
in
a
better
way,
please,
let
me
know
I'd
be
interested
to
hear
was
monitoring
of
the
vent
load
if
a
Sun,
large
increase
of
usually
fairly
predictable
events
were
seen,
it
would
trip
a
safety
valve
and
stall
the
consumer
until
someone
could
come
and
eyeball.
The
actual
events
see
what
was
going
on,
make
a
reasoned
decision
on
whether
they
should
restart
that
carry
on.
Of
course.
In
the
meantime,
all
events
will
be
installed,
but
they've
just
not
being
propagated
on.
A
Actually
to
to
fix
the
impact
of
this
this
email
issue,
I
mentioned
we
had
to
do
a
mass
replay
once
we
figured
out
what
the
problem
was,
but
this
time
flipping
the
email
invalid
flag
back
to
its
correct
value.
Otherwise
no
one
would
have
got
emails,
which
you
may
think
was
a
good
thing,
but
the
company
questions
only
didn't.
A
That
update
would
have
had
to
done
a
restore
in
from
backups
or
go
and
write
jobs
that
sweep
the
database
and
make
updates
needing
to
figure
out
the
logic
to
tease
out
which
had
been
invalid
updates
and
which
had
actually
been
correctly
updated
previously
in
the
past,
it
would
have
been
a
mess
and
it
would
have
taken
a
long
time
to
sort
out,
because
in
this
case
and
in
general,
the
systems
are
record,
which
there
are
many
of
them,
rely
on
traditional
related
databases
as
a
store
of
truth.
So
they
only
store
the
latest
update.
A
They
didn't
have
a
history
of
ayats,
so
a
point
to
keep
in
mind,
depending
on
what
you're
trying
to
resolve
is.
You
might
not
be
able
to
do
a
straight
replay,
though,
if
you
can
get
away
with
that
is
usually
straightforward,
as
you
can
set
the
consumer
back
to
the
event
you
wish
to
start
from
and
then
just
let
it
roll
forward.
A
A
A
Wasn't
don't
do
events
or
event-driven
architecture
or
don't
use
async
e,
an
API
I'm,
a
big
fan,
it's
more
look
out
for
the
issues,
we've
just
reviewed
and
if
you
come
across
them-
and
hopefully
will
this
help,
you
navigate
them
and
do
events
well,
if
not
the
first
time,
maybe
at
least
with
version
2
of
your
implantation
I
have
found
that
within
reason,
many
memories
fade
on
how
long
it
took
to
deliver
something
but
production
issues.
There
are
more
or
constant
headaches
so
bill
we
can
to
avoid.
A
Remember
you
know
as
much
as
I
like
events,
events
are
a
cure-all
they're,
just
another
tool
in
your
toolbox.
Don't
get
tempted
to
use
them
for
every
use
case.
Does
a
request
response
implementation
also
make
sense.
These
things
need
to
be
considered
help
yourself
by
helping
others
using
a
sink
API
to
convey
the
details.
Intent
of
the
events
you're
dealing
with
this
Lo
is
both
the
barrier
to
entry
speech
development
and
helps
with
the
event
self-serve
ie
teach
them
to
fish.
A
Try
to
design
failure
handling
into
your
viral
environment
early
in
the
process.
It's
much
easier
to
think
about
the
best
way
to
handle
those
at
the
design
stage,
not
at
2:00
a.m.
when
you're
trying
to
get
a
very
upset
stakeholder
back
online
and,
of
course,
in
a
world
where
toilet
paper
seems
to
take
on
special
value
when
we're
dealing
with
a
virus.
A
As
far
as
I
know
effects
breathing
in
temperature
I
guess
you
need
to
always
expect
the
unexpected
well
I
hope
you
enjoy
my
talk
and
found
something
of
interest
that
may
help
in
the
future.
Beyond
the
Q&A
today,
if
you'd
like
to
discuss
further
or
just
connect,
you
can
find
me
on
LinkedIn
I
hope
you
enjoy
the
rest
of
the
tour
schedule
today
and
I'm
looking
forward
to
them
and
I
hope
to
speak
to
you
again,
maybe
even
as
part
of
the
async
API
community.