►
From YouTube: Build a Cloud Native Asynchronous Messaging System For Scale With Redis - Madelyn Olson
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Build a Cloud Native Asynchronous Messaging System For Scale With Redis - Madelyn Olson, Amazon Web Services (AWS)
Speakers: Madelyn Elizabeth Olson
Large scale microservices architectures often rely on synchronous APIs to implement their functionality by communicating with a large number of other services. This pattern works well when all services are aware of when and how data changes. However, as the system size increases, the tight coupling and complexity can become difficult to maintain. A solution for this growing complexity is asynchronous message passing, where services register what types of messages they are interested in and submit messages to be broadcasted to the system. This highly decoupled system allows new services to be added without changing code of the existing components. In this talk, hear from Madelyn Olson, one of the Redis core maintainers and a software development engineer who will deep dive into best practices for implementing this asynchronous message design pattern. She will also demonstrate how to build, deploy, and monitor a highly scalable message broker system by using standard CNCF components, specifically Kubernetes and Prometheus, along with Redis, the “most loved” database.
A
So
hello,
my
name
is
Madeline
Olsen
I'm,
a
member
of
the
redis
open
source
project,
I'm,
also
a
engineer
at
AWS
and
today
I'm
going
to
be
talking
a
little
bit
about
how
to
use
redis
in
Cloud
native
architectures
to
build
an
asynchronous
messaging
system
and
before
I
talk
too
much
about
the
agenda.
I,
like
the
motivation
for
this
talk
is
I
know
a
lot
of
people
use
radius,
just
kind
of
like
quick
show
Advance.
A
Who
here
is
aware
of
what
redis
is
it's
like
everyone
who
here
thinks
rice
is
just
a
cash,
so
those
like
it's
really
tailored
for
you
guys,
which
is
that
a
lot
of
people
know
just
like
they
don't
know
what
else
Reds
can
do,
and
they
often
think
about
it's.
Just
like
an
ephemeral
data
store
you
throw
data
in
it
and
it
can
go
away.
So
the
specific
use
case
I'm
going
to
dive
deep
into
today
is
the
event
broker
kind
of
design
pattern.
A
So
I'll
start
kind
of
introducing
that
explain
how
it
works
in
traditional
service,
oriented,
architectures
fan,
I'll
kind
of
give
a
pseudo
introduction
to
redis.
It
seems
like
most
people
know
so.
I
won't
go
too
deep,
but
kind
of
talk
a
little
bit
about
why
it's
good
at
what
it
does
not
talk
too
much
about
what
it
is
and
then
kind
of
put
Radisson
to
this
architecture.
A
We're
talking
about
and
see
how
it
actually
kind
of
fits
the
that
role
and
then
I'll
show
kind
of
a
simple
toy
application,
kind
of
putting
everything
together
and
I'll
close
off
with
some
best
practices
and
then
we'll,
hopefully,
have
some
time
for
some
q
a
if
people
have
questions
great.
So
why
like?
What?
Why
do
people
need
message?
Brokers,
So,
In,
traditional
service,
oriented
architectures.
You
basically
have
a
lot
of
different
services
and
they're
all
talking
to
each
other
with
tightly
coupled
apis.
A
This
works
pretty
well
some
people
I
know.
Amazon
is
well
known
for
having
this
very
complex
service,
oriented
architecture
because
they
used
to
have
this
big
monolith
had
lots
of
problems
so
they
broke
everything
up
so
service
oriented
architecture
is
great.
The
problem
starts
to
stem
when
that
starts
to
get
very
complex
right.
When
you
have
one
service
calling
lots
of
other
services,
you
end
up
with,
you
know,
dependency
hell
and
a
very
high
latency,
as
all
these
services
start
chaining
together.
A
So
the
idea
behind
that
to
solve
this
is
like
either
a
message,
brokering
architecture.
So
the
idea
behind
here
is
you:
do
all
your
synchronous
work
and
then
you
just
record
an
event
that
all
the
other
services
eventually
read
and
then
they,
you
know,
take
their
actions
based
on
that.
So
this
breaks
the
tight
coupling.
You
still
have
some
coupling,
since
these
messages
need
to
be
still
well
formed,
so
all
of
the
downstream
consumer
services
know
what's
going
on,
but
it
solves
kind
of
those
two
major
problems
we
had
before.
A
So
the
first
is
now
we
get
back
to
having
super
low
latency,
since
all
we
have
to
do
is
do
our
synchronous
work
and
then
record
the
event
into
the
event
the
message
broker,
and
then
we
also
have
much
better
API
slas,
since
we're
just
now
dependent
on
the
synchronous,
work
and
all
of
the
asynchronous
work
kind
of
can
just
happen
in
the
background.
If
a
service
dies,
it
doesn't
affect
the
main
apis.
A
So
when
you
have
a
message
broker
architecture,
it
becomes
very
easy
to
scale
and
add
more
services
in,
since
you
just
need
to
Define
how
these
Services
consume
sort
of
from
this
main
message,
bus
and
for
the
rest
of
talk,
I'm
going
to
talk
about
these
message,
bus
in
two
different
forms.
The
first
is
what
I'm
going
to
call
an
authoritative
log,
so
an
authoritative
log.
You
basically
have
a
single
stream
of
every
single
message
that
ever
was
produced
as
part
of
this
message.
A
Bus,
and
so,
if
you
just
want
to
set
me
up
a
new
service,
let's
say
we
have
some
new
business
intelligence
that
needs
to
know
like.
What's
going
on,
it's
able
to
basically
go
back
in
time
start
from
the
beginning
of
the
log,
consume
everything
to
the
head
of
the
log
and
basically
bootstrap
as
a
new
service.
A
So
this
is
one
type
of
good
use
case.
The
other
really
common
use
case
I've
seen
and
it's
pretty
common
with
redis
and
with
a
lot
of
our
customers,
is
what
I
call
like
ephemeral
logs.
So
the
best
frame
for
this
is
that
you
could
lose
a
message,
and
so
that's
a
lot
of
people
get
kind
of
scared
of
that
they're
like.
Why
would
I
want
to
lose
a
message
from
a
log
and
the
best
place
to
think
about?
This
is
if
late
events
are
the
same
as
the
message
not
showing
up
at
all.
A
If
you
have
some
type
of
you
know,
maybe
you're
delivering
some
type
of
package
and
that
you're
trying
to
give
like
real-time
updates
to
the
end
user.
You
want
to
give
a
real-time
update
of
where
that
package
is.
If
you're
going
to
deliver
two
days
late,
they
don't
care
anymore,
you
can
just
have
basically
the
same
has
dropped.
It
there's
some
other
interesting
use
cases
like
maybe
you're,
okay,
with
inconsistencies
between
your
services,
because
you
have
some
background
reconciliation.
A
A
A
Those
are
you
know,
Cloud
native.
We
want
everything
to
be
performed
and
scalable
and
we
want
to
be
able
to
support
different
types
of
events,
so
not
something
that's
too
specific
to
one
individual
use
case
and
can
Shard
to
different
individual
types
of
logs.
You
don't
want
to
be.
If
you
have
one
specific
log
that
holds
all
your
data,
it'll
eventually
become
the
bottleneck
for
your
service
and,
of
course,
we're
at
kubernetes
conference.
We
want
to
be
industry
standard
and
we
would
love
open
apis.
A
So
with
that
I'm
going
to
suggest
redis
as
a
good
way
to
solve
that
problem.
I
know
a
couple.
People
said
they
didn't
know
what
rest
is,
but
res
is
a
fast
and
simple
in-memory
First
Data
store
it's
completely
open
source.
As
I
mentioned
I'm
one
of
the
maintainers
we
love
active,
we
have
a
very
active
Community,
we're
still
building
a
lot
with
inside
the
and
we're
still
releasing
new
versions.
A
So
the
main
trade
off
the
rest
kind
of
thinks
about
is
that
it
stores
everything
in
memory
first
and
then
optionally
will
persist
stuff
to
disk.
So
it
doesn't
try
to
expose
many
trade-offs
for
end
users
right.
So
it's
not
one
of
those
applications
that
will
try
to
make
it
as
easy
for
the
consumer
to
the
client
to
use,
but
it
wants
to
make
it
like.
Predictably,
high
performance
is
also
well
known
to
be
very
flexible.
A
There's
lots
of
data
structures
and
we'll
be
talking
about
one
of
those
specifically
today,
and
it
also
has
high
availability
and
various
forms
of
durability.
So
the
main
framing
I
want
us
to
use
for
res
today.
Is
that
it's
that
it's
that,
basically
a
it's
a
API
for
shared
data
and
you
can
build
applications
on
top
of
it
right.
So
if
you
have
one
client
who
writes
data,
everyone
else
can
read
from
it.
A
So
this
is
by
far
the
traditional
use
case
for
redis,
and
this
works
great
without
durability
without
replication
and
fits
into
a
lot
of
service.
Warranty
architectures
the
use
case
I'm
trying
to
push
more
for
Reddit
is
generally
is
this
idea
of
Beyond
caching,
which
incorporates
a
bunch
of
different
aspects,
but
one
specific
I
want
to
talk
about.
Is
data
projection?
So
this
sort
of
fits
into
this
message
book?
A
We
talked
about
earlier,
where
you
have
a
primary
database
and
it
emits
a
series
of
logs
based
for
the
updates
from
that
that,
like
for
mutations
and
the
rest,
is
able
to
consume
those
updates
and
then
project
the
some
form
of
the
data
into
redis.
So
the
the
authoritative
data
still
comes
from
redis.
Sorry,
it
still
comes
from
the
backend
database.
In
this
case,
I'm,
using
postgres
everywhere
and
if
Reds
were
to
die,
you'd
basically
be
able
to
go
back
to
the
postgres
database
reproject.
A
All
the
data
and
Eventing
works
really
great
in
this
case
right.
So
one
of
the
things
that
science
was
rested
as
periodic
backups,
so
you
can
basically
take
an
entire
snapshot
of
the
data
set,
save
it
to
disk
and
then
restore
that
snapshot.
So
if
you
keep
track
of
where
you
are
in
the
updates
from
the
message,
bus
you're,
able
to
like
restart
from
that
point
in
time,
so
res
is
a
great,
is
already
great
in
message.
A
A
So
there
are
two:
there
could
theoretically
be
a
third,
but
there's
usually
two
main
use
cases
that
res
has
that
people
use.
The
first
is
what's
called
Pub
sub,
so
publish
And,
subscribe,
clients
are
able
to
subscribe
to
channels
inside
redis
and
when
messages
get
published
to
those
channels,
rather
than
broadcast
the
messages
to
all
the
various
consumers
of
that
channel.
A
So
you
might
see
here
this
doesn't
work
at
all
for
authoritative
Vlogs.
It's
really
just
targeting
that
sort
of
ephemeral
workload
and
it's
really
targeting
just
making
it
as
simple
as
possible,
because
there's
no
data
being
stored
within
redis.
It's
just
immediately
broadcasting
it
out.
So
it's
really
good
for
the
normal
thinking
about
res,
where
you
don't
really
have
any
type
of
schema.
You
just
pushed
it
in
broadcast
the
data
out
and
then
you
kind
of
forget
about
it.
A
Firing
forgets
often
used
to
describe
Pub
sub
the
second
data
structure,
which
was
released
in
redness
five,
which
it's
like
four
years
old
now
as
what's
called
red
streams,
so
red
streams
took
a
live
inspiration
from
Kafka
and
is
basically
a
append
Only
log
of
data.
So
that's
this
sort
of
fits
more
into
our
our
Paradigm.
We
just
had
of
of
an
authoritative
log,
so
you
put
a
bunch
of
events
in
redis
and
then
you're
able
to
cooperatively
consume
them
through.
A
What's
called
consumer
groups
or
an
individual
client
can
basically
keep
track
of
where
they
are
and
then
keep
reading
from
that
point
in
time,
this
data
will
need
to
eventually
be
trimmed
out
of
redis
cool.
So
so,
how
do
these
two
actually
look?
Most
everything
in
redis
uses
commands.
It
doesn't
have
any
query
language
like
something
like
SQL.
So
in
this
case
we
have
a
command
called
xad,
which
is
responsible
for
adding
data
into
streams.
A
And
then,
similarly,
we
have
the
pub
sub
system,
which
does
something
very
similar,
but
since
kind
of
in
the
reverse.
So
consumers
are
the
ones
that
start
the
transaction.
They
say:
hey
I
would
like
to
subscribe
to
this
channel
and
then,
when
messages
are
published
into
the
channel
everyone's
able
to
get
basically
how
many
messages
were
broadcast
and
everyone
that
got
the
consumers.
A
So
we
mentioned
before
that.
We
also
needed
some
type
of
partitioning
of
streams.
So
rennis
has
a
deployment
called
rest
cluster
mode,
which
is
a
natively
sharded
deployment,
and
it
then
partitions
the
data
by
the
crc16
of
either
the
name
which
can
either
be
the
channel
name
or
the
name
of
the
Stream.
A
So
that's
all
well
and
good,
but
there
is
still
an
elephant
in
the
room
that
haven't
really
talked
super
too
much
about
it's,
not
postgres.
I
do
like
postgres
postgres
was
known
to
be
the
most
loved
database,
which
stole
away
redis's
title,
but
it's
actually
durability
of
data,
so
most
people
think
redis
is
not
durable
and
for
the
vast
majority
of
people
that
actually
use
res
it's
not
it's
not
right.
A
A
So
rice
has
not
particularly
great
at
being
durable,
but
it
can
be
so
it
can
be
used
as
its
authority
of
long,
because
it's
still
good
at
that
high
throughput,
high
efficiency
type
workloads,
so
I
will
argue
that
it
can
be
used
for
that
and
in
many
cases,
using
a
durable
store
for
an
event.
Bus
is
better
than
using
an
ephemeral
store
because
it
makes
it
simple.
You
don't
have
to
worry
too
much
about
messages
not
getting
delivered,
but
a
lot
of
event.
A
Bridges
inherently
assume,
like
some
type
of
item
potency
or
having
to
Think
Through.
You
know
how
to
handle
at
least
once
or
at
most
once
type
of
messages.
So
the
ephemeral
workload
still
you
still
have
to
solve
a
lot
of
the
same
problems.
So
even
though,
if
you're
not
using
durable
rest
can
still
work
pretty
well
as
an
event
bus
in
either
mode,
and
there
is
also
multiple
managed
providers
that
provide
redis.
B
A
So
best
practices
for
redis
at
scale
as
I
mentioned
briefly.
Redis
has
a
shared,
nothing
partitioning
scheme.
When
you
use
it
in
its
clustered
configuration
you
can
typically
get
between
100
150
000
operations
per
like
process
per
core
in
typical
non-durable
configurations.
It
falls
to
between,
like
60
to
100
000
operations
per
second
in
more
durable
configurations,
but
you
can
scale
up
to
about
a
thousand
different
shards
which
gives
somewhere
around
like
50
to
150
million
operations
per
second,
which
is
pretty
much
higher
than
anyone
I've
ever
seen
practically
used.
A
We
have
a
couple
of
customers
AWS,
who
kind
of
get
close
to
those
numbers,
but
practically
it
there's
plenty
of
scaling
room
as
long
as
you
have
a
good
way
to
Shard
the
data
starting
with
redis
7.
We
actually
introduced
what's
called
chartered
Pub
sub
in
the
olden
days
before
red
is
seven.
There
was
no
good
charting
for
Pub
sub
messages,
so
in
round
seven,
not
that's
more
or
less
been
fixed.
That
was
contributed
by
AWS
and
then
also
if
you
want
High
availability
for
your
data.
A
A
high
availability
is
not
as
important
for
event
buses,
because
if
you
kind
of
take
an
outage
for
a
while
and
bring
up
a
new
node,
it
will
still
like
that.
You
have
that
built-in
slack,
because
you
don't
need
the
synchronous
operation.
So
it's
not
as
important,
but
you
should
understand
like
when
you
actually
do
need
it
and
if
you
think
about,
if
you're,
adding
replicas
you're
really
optimizing.
A
For
that
case,
when
failures
do
happen
and
99
of
the
time
they
aren't
happening
so
you're
paying
that
cost
so
just
be
very
deliberate
about
choosing
to
run
with
replication.
And
then
the
important
thing
about
rest
is.
It
does
not
try
to
solve
heart,
hotkeys
or
hot
channels
for
you.
So
if
you
do
end
up
funneling
too
much
data
into
one
Shard
that
Shard
will
get
overwhelmed
and
you'll
have
problems
so
and
rest,
as
I
said
before
it
just
doesn't
try
to
solve
those
problems.
A
For
you,
then,
as
I
mentioned
before
the
when
you're,
using
an
authoritative
log,
eventually
you'll
need
to
trim
data,
so
in
that
case,
you'll
typically
take
the
hot
date
and
keep
in
redis
reddis.
As
I
said
memory
in
memory
first,
all
the
data
is
stored
in
memory.
So
you
don't
actually
want
that
because
that's
expensive,
so
you
eventually
want
to
tear
it
to
disk.
A
A
If
your
data
is
set
up
in
such
a
way
that
the
events
all
sort
of
can
get
conducted
together
and
put
into
the
end
at
like
an
end,
State
like
if
you
have
multiple
different
values,
you're
like
continually
updating
it,
we're
at
a
snapshot's,
probably
a
good
way
to
sort
of
compact
all
the
logs
events
together.
So
you
can
kind
of
stick
it
on
disk
and
then
restore
that
that's
a
little
bit
more
specific.
A
It
depends
a
little
bit
on
your
use
case,
so
I've
been
talking
a
lot
about
just
about
redis,
but
since
res
is
an
open
source
project
and
is
one
of
the
most
downloaded
res
sorry
Docker
containers.
It
has
a
lot
of
support
for
all
the
cncf
projects.
I'll,
specifically
talking
a
bunch
about
how
it
integrates
with
Prometheus
what
type
of
metrics
to
look
at
to
make
sure
it
doesn't
kind
of
explode.
Because
I
said
it's
like
it's
a
lot
of
people.
A
Think
of
it's
kind
of
like
a
race
car
like
can
go
really
fast,
but
if
you
kind
of
drift
a
little
too
much
it
just
piles
on
and
explodes.
So
it's
very
important
to
make
sure
you
kind
of
understand
those
failure
modes
cool
so
with
that
I'm
going
to
walk
through
a
short
little
demo
and
kind
of
walk
through
how
you
can
actually
use
redis
in
this
configuration.
A
A
So
in
this
example,
there
will
be
three
main
components:
we'll
be
talking
about:
there's
going
to
be
a
web
app,
a
queue,
that's
consuming
data
and
a
generator
of
data
that
we'll
throw
a
load
in
a
second
I.
Actually,
don't
want
this
thing,
generating
consuming
any
load
for
that
seconds.
I'll
turn
that
off
some
people
might
ask
one
of
my
favorite.
A
A
A
So
there's
there's
a
bunch
of
contents
up
here.
They're
good
to
know
about
all
of
our
events
are
going
to
have
a
very
specific
schema.
You
should
probably
use
something
more
like
protobuf
or
something
but
I
wanted
this,
because
it's
a
little
bit
more
readable
to
at
least
follow,
and
you
once
you
use
prototype
everything
kind
of
gets
lost.
A
A
Redis
is
a
little
bit
more
performant
when
you
send
a
bunch
of
argument
commands
together.
I
do
this
just
for
some
monitoring
aspects
later
in
a
newer
version
of
rice
it'll
be
much
easier
to
see
the
size
of
streams,
but
right
now
there
isn't
so
I
have
to
keep
track
of
that
all
manually.
So
not
super
important,
but,
as
you
can
see
here,
the
only
thing
we're
doing
in
this
order.
Operation
is
just
putting
an
item
in
the
Stream
in
the
real
world.
A
We'd
probably
want
to
be
doing
other
synchronous
work,
making
sure
we
have
capacity
making
sure
the
user
is
authenticated,
but
that's
all
you
know,
business
logic
will
figure
that
out
kind
of
in
a
second
and
what
are
we
doing
with
this
we're
putting
in
a
stream
and
then
we'll
have
two
consumers
for
this?
The
first
consumer
will
be
showing
the
last
item
this
user
has
purchased
and
the
second
is,
we
will
be
showing
the
overall
top
sellers
for
this
given
operation.
A
A
A
A
We
have
a
log
size,
I
put
putting
some
data
in
here
earlier,
so
it's
non-zero,
but
it's
there
and
we're
seeing
latency
about
four
to
five
milliseconds.
A
lot
of
people
are
used
to
write
as
being
sub
milliseconds,
but
I'm
using
rice
in
a
more
durably
stored
configuration,
so
we're
actually
running
to
three
separate
azs
inside
AWS,
which
takes
a
little
bit
more
time
so
that
all
looks
great
oops,
not
fair,
so
I'm
going
to
go
ahead
and
we're
also
going
to
start
putting
some
load
on
here.
A
Wow,
this
refresh
a
little
bit
more
regularly.
Oh
yeah,
one
other
thing
I
wanted
to
show
you
was
that
I
can
prove
that
this
actually
hasn't
been
consuming
yet
because
his
history
is
still
empty
because
we,
the
consumer
of
the
data,
has
not
consumed
the
data.
So
while
everything
is
soaring
to
start
up,
I'll
quickly
show
what
the
consumers
look
like
consumer.
A
The
consumers
are
really
simple,
so
we
have
down
here.
We
have
a
function
that
we're
calling,
basically,
when
an
item
actually
gets
processed,
so
we're
using
we're
still
using
rats
for
everything,
because
why
not
we're
using
a
sword
set
to
keep
track
of
the
top
selling
items
and
then
we're
just
storing
the
last
purchased
item
in
a
hash
value.
A
So
going
back
and
talking
a
little
bit
what
we
said
before
we're
going
to
use
the
X
group
commands,
which
are
the
Cooperative
consumers,
so
we
can
have
multiple
different
nudes
consuming
multiple
different
nodes,
consuming
data,
so
being
very
unread,
is
like
you
actually
have
to
call
an
explicit
function
to
do.
X
group
creates
and
have
to
like
catch
exceptions.
If
you
do
it
multiple
times.
Typically,
redis
is
very
schema-less,
but
not
in
this
case,
we're
still
trying
to
work
on
a
better
way.
A
A
So
the
group
is
created
on
a
stream
and
then
you
read
from
a
group,
and
so
that
makes
sure
that
every
message
within
the
stream
is
only
delivered
once
so
lots
of
fun
stuff
here,
there's
some
just
Plumbing
to
basically
bootstrap
a
node.
It
hasn't
consumed
anything
this
just
means
start
consuming
from
the
beginning
and
then
we're
consuming
up
to
500
elements
and
we're
weighing
one.
Second,
in
case
there
is
no
items
currently
in
the
Stream.
A
The
other
interesting
case
we
have
to
cover
is
when
an
item
gets
every
item.
That's
read
from
a
group
has
to
be
acknowledged
and
if
an
item
is
dropped,
we
need
someone
to
go
and
claim
it.
So
we
have
a
separate
command
called
X,
auto
claim
which
will
allow
you
to
actually
claim
commands
which
aren't
owned
if
they're
idle
for
at
least
one
one
seconds-
and
this
does
mean
in
this
configuration
we're
doing
it
most
once
since
a
node
could
have
read
an
item
gone
off.
A
A
A
We
can
see
that
the
number
of
elements
is
steadily
growing
with
the
log
size
keeps
growing
up
and
up
and
up
we've
added
we've
been
continually
adding
names
to
the
log
and
the
unconsumed
message
kind
of
ticked
up
for
a
second
and
that's
because
we
started
the
producers
faster
than
the
consumers
and
I
kind
of
came
back
down.
It's
been
hovering
around
zero,
not
quite
zero.
A
A
A
I
talked
a
bit
about.
You
know
how
to
make
sure
you're
alarming.
You
definitely
want
to
make
sure
you
have
if
you're
not
consuming
messages.
You
need
some
type
of
lump
on
that,
so
it
depends
on
you
know.
You
should
figure
out
your
application
to
figure
out
how
fast
you
can
consume
messages
and
figure
out
how
long
you're
willing
to
tolerate
messages
not
being
consumed,
because
once
the
qsr
gets
too
long,
it
will
take
a
long
time
for
it
to
actually
drain
back
out,
and
you
can
have
other
problems
as
well.
A
There's
one
interesting
Paradigm
inside
AWS,
where,
if
you're
falling
too
far
behind
what
you
should
start
doing,
is
instead
of
process
processing.
First
in
first
out
you
should
do
the
opposite
start
processing
last
in
first
out,
so
you're
actually
doing
some
successful
requests
and
actually
kind
of
getting
latency
down.
Otherwise,
you
might
just
kind
of
keep
getting
further
and
further
behind.
A
How's
this
going
it's
all
starting,
should
have
probably
figured
out
a
better
series
to
actually
do
this
yeah.
These
are
all
still
creating,
and
maybe
we'll
come
back
to
this
in
a
second
and
I
can
quickly
go
and
catch
the
last
of
the
best
practices,
best
practices
so
yeah.
This
is
the
last
thing
I
want
to
talk
about
when
building
message,
brokering
systems
the
most
important
thing
is
you
really
need
to
understand?
What's
what
needs
to
be
done?
Synchronously
in
this,
you
know.
A
Web
app
example
we're
talking
about,
like
no
user
likes
to
get
a
successfully
purchased
and
then
found
out.
Oh
no
async
had
failed
because
we
didn't
actually
have
capacity
so
make
sure
you
know
what
needs
to
be
done,
synchronously
what
needs
to
be
communicated
to
users,
since
if
something
is
failing,
asynchronously,
you
kind
of
need
to
like
you
need
to
know
how
to
resolve
that,
because
you
didn't
give
a
synchronous
error
to
users
and
make
sure
you
test
failure
modes.
A
As
I
said,
this
is
a
pattern
which
is
really
helps,
but
it
also
can
be
a
huge
Thor
in
the
side.
Reminds
me
a
lot
of
caching.
In
that
case,
a
lot
of
people
have
failures
because
their
cash
Falls
over
and
then
they
storm
their
backend
database
and
sad
things
happen.
So
just
like
I
feel
like
in
a
lot
of
places
with
red.
A
It's
just
making
sure
you
understand
your
failure
modes
and
you
test
them
regularly
an
example
we're
just
kind
of
talking
about
to
try
making
sure
like
just
turn
off
all
your
consumers
seeds
with
see
what
happens
when
you
restart
them
cold
starts,
are
nice
and
easy
to
do
in
kubernetes?
For
the
most
part,
another
important
thing
is
to
like
really
make
sure
you're,
avoiding
poison
pills,
make
sure
you
have
like
schemas
for
events
and
that
everyone
is
like
adhering
to
them.
A
A
We
already
talked
about
this
a
little
bit
make
sure
you
enough
burst
capacity
to
handle
whatever
type
of
load
you
want
to
do,
especially
in
the
res
case,
when
RES
can
actually
burst
into
memory.
Make
sure
you
have
enough
like
spare
memory
capacity.
Cues
are
typically
a
good
use
case
for
redis,
because
you
usually
don't
consume
all
that
much
memory.
You
kind
of,
like
you,
add
items
remove
items,
so
memory
stays
pretty
constant
but
make
sure
you
have
enough
to
actually
burst
into
the
last
thing
is
message
partitioning.
A
We
talked
a
little
bit
about
how
to
name
stuff
in
redis
using
like
crc16.
It's
still
important
to
understand
like
how
you
like,
what's
the
right
Dimension,
to
partition
your
messages
on
most
people,
do
it
by
something
like
a
user
ID
or
a
customer
ID,
but
there's
other
ways
as
well
like,
like
you
know,
based
like
just
logically
grouping
stuff.
Instead
of
doing
something.
D
A
A
hash
on
top
of
the
customer
ID,
so
that's
all
I,
have
for
you
at
the
moment.
I
believe
I'm
supposed
to
show
you
a
QR
code
and
give
you
something
so
I'll.
Just
briefly
say:
like
Thank,
you
Madeline
Nelson,
you
can
follow
me
on
Twitter.
Here's,
the
QR
code,
if
you
need
it
and
yeah
I
think
that's
all
I
have.
A
B
A
For
the
most
part,
I've
seen
like
Grannis,
so
Kafka
really
only
Sports
the
the
logs
more
have
and
it
stores
everything
durably.
If
you
don't
need
that
durability,
that's
the
biggest
reason
I've
seen
people
move
away
from
Kafka
because
rest
tends
to
be
more
efficient
and
has
lower
cost
to
operate.
But
I
would
say
it's
a
lot
about
preference.
A
A
D
A
Yeah,
especially
for
memory
as
I
said,
it
kind
of
really
comes
down
to
your
capacity
planning.
At
some
point,
you
should
stop
accepting
messages
if
you're
running,
if
you
have
no
memory
like
spare
memory
to
actually
like
put
them
so
typically,
you
want
to
start
like
you'll
want
to
reject
them
and
have
the
back
pressure,
push
it
back
to
the
front
end.
So
it
like
things,
stop
adding
more
data
in
so
you'll
want
to
for
capacity
planning.
That's
I
mean
too
it's
very
you
know,
system
dependent,
AWS.
A
A
Cues
are
almost
always
CPU
and
throughput
bound,
not
memory
bound
unless
you're
like
really
far
falling
far
behind
so
try
to
figure
out
like
how
like
how
much
time
are
you
willing
to
keep
accepting
rights
before
you
kind
of
have
a
hard
outage
is
another
thing
we
you
do
at
AWS
like
for
we
have
like
when
we
do
like
authentication.
We
have
like
six
hours,
so
we
usually
say
you
need
at
least
six
hours
of
reasonable
extra
capacity
before
you
should
start
rejecting
requests,
but
obviously
that
becomes
very
different.
A
If
you're
actually
very
constrained
on
memory
or
disk,
then
you
might
need
to
keep
less
Surplus
for
disk
yeah.
You
have
the
same
problem.
If
you're,
using
with
aof,
you
should
figure
out
how
much
you're
willing
to
spill
before
you
start
actually
rejecting
the
front-end
apps,
because
most
of
the
time
message,
Brokers
are
used
to
keep
availability
up
and
have
the
latency.