►
Description
Kubernetes and FaaS have solved most of the challenges of dealing with stateless applications. But when it comes to handling state, it quickly becomes “someone else's problem”. Because of that, we have struggled with the same issues of data consistency and complex failure semantics for the past decade. For stateful applications, we are still far from the smooth development and operations experience associated with serverless.
Stateful Function is a new pattern to solve this problem. It uses containerized, event-driven functions with a stream processor (Apache Flink), not a database, to manage the state and handle the messaging between functions. This talk walks through the ideas behind Stateful Functions and shows how this simple framework solves the problem of consistency and failure semantics within and across functions, at the same time staying true to the serverless experience.
A
A
I'm
not
sure
how
how
much
this
is
known
in
in
the
like
in
this
in
this
meetup
audience
so
party
flink
is
a
is
a
system
for
for
stateful,
self-worth
stream
processing.
It's
it's!
I
think
more.
You
know
coming
out
of
the
of
the
data
processing,
real-time
data
processing,
yeah
big
data
infrastructure
community.
So
I'm
I'm
not.
I'm
not
100
sure
how
how
much
like
whether
you've
heard
of
it
before
or
not.
You
don't
need
much.
You
didn't
need
to
know
much
about
flink.
A
This
talk
is
largely
independent,
even
though
it
it
builds
on
on
top
of
the
system
yeah.
We
we
created
this
as
a
university
spin-off.
If
you
wish
here
in
berlin,
I
think
by
now,
almost
six
years
ago,
founded
a
company
around
it
was
acquired
about
two
years
ago
by
alibaba
and
is
operating
as
a
subsidiary.
A
Now-
and
we
are
we're
continuing
to
develop,
you
know
apache
flinkers
as
an
open
source
system
for
for
big
data
processing,
but
like
we're
more
and
more
looking,
also
into
use
cases
exactly
for
for
stateful,
serverless,
yeah
applications
and
that's
what
I
would
like
to
to
focus
on
in
the
talk
today
and
I'll
also
mention
what
the
elephant
in
the
room
is
in
a
few
slides
into
the
talk
all
right.
You
might
have
seen
these
these
here
before
the
dilbert
comic
and
the
the
tweet
from
kelsey.
A
What
we're
trying
to
do
is
is
really
rethink
the
the
application
layer
and,
and
some
of
the
inherent
paradigms
we're
doing
like
we're
using
there
to
to
make
it
easier
to
build
to
build
like
reliable
and
scalable
applications,
maybe
to
to
break
this
down
in
in.
In
my
own
words,
the
the
thing
that
we're
trying
to
solve
is
really
the
the
problem
that
it's
still
hard
actually
to
build
a
scalable,
consistent,
stateful
application,
and
the
problem
is
like
putting
this
on
an
on.
A
If
we
actually
apply
this
there
can
we
can
we
get
something
that
is
you
know
I
I
just
put
magically
scalable
there.
It's
of
course
not
going
to
be
completely
magic.
Nothing
is,
but
you
know
it's
just
going
to
put
us
in
a
good
and
a
good
starting
position
to
build
these
kind
of
really
easily
scalable
applications
and
and
scalability.
Here.
Is
it's
really
just
one?
A
One
aspect
right:
we're
we're
looking,
of
course,
at
various
concerns
in
that
space,
like
scalable
computation
stage,
state
consistency
is
a
big
thing,
but
also
security,
observability
and
so
on.
So
in
this
talk
I
would
really
like
to
focus
on
on
these
three
parts:
scalable
computation,
scalable,
state
and
and
consistent
state.
This
is
kind
of
what
we've
been
working
on
in
in
the
in
the
apache
flank
community.
A
For
for
a
while-
and
I
think,
there's
some
some
really
good
ideas
in
there
that
if
we,
if
we
actually
start
to
to
combine
them
and
merge
them
with
some
of
the
ideas
that
have
come
out
of
this,
like
yeah
of
them
of
these
of
this
space
here,
modern
application,
architectures
cloud
native
application
architectures.
If
I
think,
if
we
try
to
put
them
together,
we
can
we
can
actually
can
actually
do
something
really
nice.
A
A
The
elephant
in
the
room
for
me
has
has
always
been
the
database
and-
and
I'm
not
thinking
you
know,
like
sql
versus
key
value
store,
but
like
the
general
idea
of
how
do
we
actually
use
databases
and
key
value
stores
today
from
from
our
applications?
And
this
like
request,
response
interaction
that
we
have
with
databases,
you
know
we
do,
we
send
it
a
key
lookup
request.
It
gives
us
the
value
for
a
key.
We
send
it
a
put
put
value
for
a
key
request
that
tells
us
okay
has
worked
or
didn't
work.
A
A
But
here's
here's
a
bit
of
let's
say
maybe
provocative,
talk
that
we
should
actually
think
rethink
this
a
bit
and
maybe
do
something
about
that
if
we,
if
we
really
want
to,
if
you
really
want
to
arrive
at
this
at
this
next
level
of
yeah
stateful
serverless
applications,
I'm
going
to
give
you
a
brief
teaser
in
in
which
direction
some
of
these
solutions
that
that
I'll
be
talking
about
today
are
going
to
go.
This
is
one
of
my
it's
one
of
my
favorite
papers
of
of
all
times.
A
A
A
A
Maybe
the
easiest
way
to
do
this
is
by
me
just
explaining
what
we've
actually
built
with
stateful
functions
and
then
we're
going
to
look
a
little
bit
at
how
how
this
is
actually
different
from
from
some
of
the
our
classical
database-centric
yeah
application
patterns
and
approaches
yeah,
so
the
the
the
one-liner
it's
not
quite
a
one-liner,
it's
two-liner
here
for
stateful
functions
is
it's
a
polyglot.
Event-Driven
functions
for
distributed
stateful
applications,
and
that
is
quite
a
mouthful.
So
let
me
let
me
try
to
break
this
down
what
this
is
all
about.
A
A
A
You
can
think
of
them,
as
you
know,
as
functions
as
with
functions
as
a
service,
aws
lambda,
or
something
like
that
that
you
can
write
in
you
know
in
in
many
languages,
they're
they're
strictly
strictly
event,
driven
invoked
through
messages
and
designed
to
not
not
compute
resources
when
they're
inactive
in
a
lot
of
ways,
you
can
also
think
of
them
as
actors,
except
for
for
the
fact
that
that
they
have
this,
this
virtual
notion
to
them.
So
you
know
when
they're
not
invoked
they're
they're,
actually
not
there.
A
The
other
part
of
of
the
name
stateful
function
as
it
implies,
is
actually
the
stateful
full
nature.
So
so
each
of
this
function
has
access
to
to
local
and
consistent
state.
So
rather
than
saying
okay,
we
have
we
have
these
pure
functions
and
let
them
talk
to
a
database
really
trying
to
to
bake
state
in
here
as
a
first
class
concept.
A
So
every
every
function
acts
as
a
state
pretty
much
as
in
a
local
variable,
so
it
is
invoked
with
with
a
message
that
comes,
for
example,
let's
say
we're:
we
have
a
handler
for
an
e-commerce
platform
and
the
message
is
put
something
in
the
cart.
Then
the
message
is
put
put
something
in
the
cart
together
with
that
something.
A
A
The
other
part
that
is
that
is
here
is
that
these
these
functions
can
also
exchange
messages,
so
they
they
send
messages
to
each
other.
You
know,
maybe
the
shopping
cart
function
might
send
messages
to
the
inventory
functions
in
order
to
you
know,
update
update
the
stock
might
might
send
something
to
the
checkout
process
function
and
so
on.
A
So
again
we
can
really
see
this.
There
is
some
similarity
to
to
actor
programming
here
with
the
difference
of
like
having
this
this
baked
in
notion
of
consistent
state
and
messaging,
and
for
both
of
them.
What
this
abstraction
here
really
assumes
or
or
lets
you
assume,
is
you
can
send
messages
and
they
don't
get
lost.
They
arrive
exactly
once
you
operate
with
state
and
it
is,
it
is
going
to
be
it's
going
to
be
fine
like
you,
don't
have
to
worry
about
transactions
working
in
the
background
about
failures
and
retrying.
A
So
this
these
are
really
aspects
that
we
that
we
get
completely
out
of
out
of
the
picture.
Ideally,
you
actually
work
purely
with
these
functions
and
state.
You
start
everything
by
by
you
know,
by
in
an
event-driven
way
by
by
subscribing
to
certain
event,
ingresses
and
you're
publishing
your
reaction
to
the
outside
world
by
publishing
messages
to
event
egresses
by
really
keeping
keep
keeping
this
this
paradigm.
A
The
only
way
to
interact
with
the
outside
world
is
with
the
ingresses
and
egresses,
and
otherwise
we're
really
just
functions
messaging,
each
other
and
with
stage
we
can
actually
guarantee
an
extremely
high
level
of
consistency
out
of
the
box.
So
as
an
application
developer,
you
have
to
worry
about
about
nothing
there.
Really,
the
the
thing
that's
worth
mentioning
here,
maybe
early,
even
though
we're
going
to
go
into
into
that
in
a
bit
more
detail
later
is
the
run.
Time,
for
this
is,
is
actually
different
than
than
for
many
other.
A
A
The
the
way
that
persistence
here
works
is
that
that
these
functions
and
and
messages
or
the
state
of
the
messaging
is
periodically
snapshotted
to
some
some
mass
storage.
Be
that
something
like
s3
or
google
cloud
storage,
hdfs,
min
io,
or
so
just
anything
that
can
write
a
file
really
so,
rather
than
having
to
deal
with.
You
know
like
scaling
a
separate
database.
A
So
how
how
does
it?
How
does
that
look
from
from
an
architecture
perspective
or
from
a
runtime
perspective?
Now
so
going
going
back
a
few
slides?
We
saw
you
know
you
have
this,
this
idea
of
building
functions
and
messaging,
and
you
know
you
don't
have
to
worry
about
messages
getting
lost
instead
getting
lost.
It
all
sounds
fine,
but
you
know:
how
is
it
actually?
How
is
it
actually
run
in
practice?
A
So
the
way
that
this
is
this
is
deployed
is
basically
with
with
with
a
system
that
we
sometimes
call
something
like
an
event-driven
database,
so
it
takes
the
role
of
an
of
a
database,
but
also
deals
actually
with
with
messaging.
So
it
takes
the
messages
from
the
ingresses.
It
takes
care
of
routing
the
messaging
between
the
function
and
it
takes
care
of
writing
the
messages
to
the
egresses.
A
Behind
a
a
service
or
or
a
similar,
endpoint
like
it
can
be
in
can
be
something
like
aws
api
gateway.
It
can
be
a
kubernetes
service
and
so
on,
so
you
can.
You
can
actually
deploy
these
functions
in
just
in
a
in
a
stateless
container
deployment
and
operate
it
pretty
much
exactly
like
that.
So
all
this
data
and
messaging
really
happens
through
this
upper
layer
here
through
through
apache
flink
and
the
the
lower
part.
The
application
code
really
runs
purely
in
in
in
pure
end
and
stateless
functions
that
you
can.
A
Actually
you
can
actually
operate
them
just
as
any
other
stateless
component,
even
though
you
actually
write
stateful
code
in
there,
you
write
code
that
accesses
local
variables
and
so
on.
You
can
still
actually
treat
it,
treat
it
pretty
much
as
stateless
and
the
the
important
part
here
is
that
state
is
provided
actually
to
these
functions
by
by
this
upper
component
by
this
stateful
function.
Cluster,
whenever,
whenever
a
function
gets
a
message
with,
let's
say
you
know,
handle
handle
this
action
on
your
shopping
cart.
A
It
actually
gets
the
state
of
the
shopping
cart
as
part
of
of
that
message
directly.
So
it
can
manipulate
that
and
then
send
this
with
a
response.
So
after
this
one
message,
processing
is
done,
the
the
container
is
or
the
function
is
again.
Stateless
and
and
invocations
are,
if
you
wish
item
potent
by
default.
That
way,
let's
actually
focus
a
little
on
on
this
aspect,
because
I
think
this
is
maybe
the
the
most
important
difference.
I
think
in
how
this
framework
approaches
things
compared
to.
A
I
think
most
most
other
solutions
that
that
we're
seeing
these
days
so
if,
if
you're,
if
you're
looking
at
a
traditional
database
application,
how
it
usually
looks
is
that
the
application
logic
itself
is
the
is
the
start
of
of
things.
You
know
it.
It
receives
a
request
by
rpc
by
or
directly
http
from
from
the
from
a
client
device,
or
it
picks
up
a
message
from
from
a
message
queue
or
so.
In
any
case,
everything
starts
in
the
application.
A
A
Now
the
the
thing
is
that
the
fact
that
message
and
state
here
are
are
handled
kind
of
with
different,
with
different
systems
or
with
different
protocols.
Even
I
think
is,
is
one
of
the
core
things
that
makes
a
lot
of
things
very
hard
in
order
to
you
know
to
guarantee
especially
consistency
here,
but
also
also
around
scalability
yeah.
You
have
actually
very
different,
very
different
scalability
characteristics
for
for
these
two
sides.
A
If
you
wish
the
idea
of
of
stateful
functions
or
the
idea
of
using
what
we
sometimes
call
event
driven
database
here
is
to
to
turn
this
actually
around
to
actually
invert
the
role
of
the
application
and
the
database
in
this
case
here,
the
like
everything
starts
actually
with
the.
If
you
wish
with
the
database,
the
the
component
that
receives
the
events
from
the
ingress
is
is
the
database.
It
can
actually
take
these
messages.
A
It
can
route
them
to
the
chart
that
owns
the
state
and
from
there
supply
these
messages
and
states
together
to
the
application,
and
then
it
calls
the
application
whenever,
whenever
that
part
is
done,
it
can
also
implement
pretty
sophisticated
scheduling
there
like,
for
example,
making
sure
that
only
one
one
invocation
per
per
state
entry
is
as
active
at
any
point
in
time,
similar.
As
you
know,
with
actor
systems,
you
don't
want
more
than
one
function,
manipulating
a
certain
inventory
entry
at
any
point
in
time.
A
So
you
can,
you
know
you,
can
enqueue
update
messages
and
schedule
them
one
after
another,
and
you
invoke
the
application.
Whenever
you
know
the
circumstances
are
right,
so
the
the
driver
of
of
the
interaction
is,
I
would
say
in
there
in
the
and
then
and
then
classical
application,
a
database
application
architecture.
It's
it's
the
application
itself,
it's
the
one
that
takes
the
message
that
makes
requests
to
the
database
in
in
this
event-driven
database
architecture.
The
the
driving
force
here
is:
is
this
event-driven
database,
the
stateful
function
cluster?
A
It
takes
the
messages,
co-locates
them
with
the
state
and
then
invokes
the
application,
so
the
application
becomes
the
the
reacting
part
rather
than
being
the
acting
part,
and
the
database
becomes
the
active
part
rather
than
being
the
reacting
part.
We
kind
of
we
kind
of
flip
the
roles
between
the
two
and
it
might
sound
a
bit
strange
to
do
this
at
the
beginning,
but
it
actually
leads
to
some
some
very
nice
properties.
If
we
think
that,
through
to
the
end,
let's
actually
make
this
a
little
more
concrete.
How
does
it?
A
A
So
what
we
have
here
is
it's
a
very
it's,
a
very
simple
setup.
Let's
say
we're
we're
using
a
we're
using
our
database
tier
here
or
in
in
this
case
our
fling
stateful
function.
Cluster
has
has
two
two
shards
two
partitions
right:
we
it's
a
very
low
parallelism,
the
scale's
too
much
higher
parallelism,
but
just
to
to
illustrate
the
example.
We
just
have
two
different
different
parallel
charts
in
the
system.
A
A
So
the
the
message
comes
in
that
targeted
at
here
at
a
card
for
kim.
So
if
we
look
at
this
and
if
we
decompose
this
message,
every
message
will
go
to
a
certain
type
of
function.
The
type
here
is
is
card,
and
then
it
will
go
to
a
certain
function
instance
or
function
id
the
id
here
is,
let's
say,
just
the
username
of
the
user.
That
may
not
go
shopping
in
this
case.
A
It
would
be,
it
would
be
kim
and
where-
and
the
message
itself
is
an
is
a
message-
add
to
cart
that
says.
Okay,
I
want
to
add.
I
want
to
add
three
socks
to
this
to
the
shopping
cart.
So
this
message
first
enters
the
system
in
in
one
partition
in
in
partition
a
and
this
one
is
routed
to
partition
a
and
because
this
one
is
the
one
that
actually
owns
owns
the
state
for
kim
or
for
kim's
card.
A
Then
then,
this
one
actually
dispatches
this
to
our
our
shopping,
cart
functions
that
are
that
are
defined
behind
an
endpoint.
Let's
say
it's
behind
a
behind
the
kubernetes
service,
so
we're
we're
basically
dispatching
this
message
add
to
cart.
Three
socks
and
the
state
of
the
current
card,
which
is
currently
empty.
A
The
you
know
this
is
a
very
simple
action
that
the
that
the
function
does
it
simply
manipulates
the
state
of
the
cart
and
then
it
the
function
produces
another
message,
namely
updating
the
inventory.
So
the
the
response
is
a
resulting
state,
let's
say
a
card
with
three
socks
and
a
message
to
the
inventory
type
function.
A
And
here
we
we
basically
then
do
the
same
thing.
You
know
we
dispatch
the
message
we
we
attach
as
the
state
to
the
message
the
current
the
current
state
for
this
for
this
function,
namely
how
much
how
much
stocks
do
we
currently
have
in
stock?
And
then
this
function
is
expected
to.
You
know
just
update
that
and
produce
the
resulting
message,
be
that
you
know
we're
out
of
stock.
Actually
action
failed
or
or
whatever
you
want
here.
A
So
their
application
logic
is
as
pure
as
in
the
functional
programming.
Sense
right,
no
side
effects
here
and
also
the
the
result
purely
depends
on
the
it
depends
on
the
invocation
parameters.
So
we
can,
we
can
pretty
easily
see
if,
if
even
let's
say
a
connection
here
fails.
Let's
say
this:
this
invocation
that
we
have
here
of
the
inventory
service.
If
this
fails,
we
don't
really
have
to
worry
about.
Oh
did
this
go
through.
Did
this
materialize
any
side
effects
or
not?
A
We
we,
basically,
we
retry
it
right,
there's
only
one
owner
for
this
specific
for
this
specific
entity.
Let's
say
the
the
inventory
socks
and
this
one:
can
you
know
it?
Can
it
can
keep
the
ownership
it
can
know?
Did
the
request
fail
or
not
within
this
specific
system?
Here
I'm
going
to
go
into
that
in
in
a
bit
within
the
this
event,
driven
database
there's,
basically
a
a
transactional
protocol
in
making
sure
state
updates
and
messages
happen
and
so
on
which
also
integrates
the
communication
with
ingress
and
egress.
A
So
we
can
actually
give
the
user
the
the
impression
that
that
all
these
messages
and
state
updates
happened
happen
exactly
once
and
where
we
cannot
guarantee
the
exactly
once
semantics,
namely
the
invocation
of
the
functions
itself.
They're,
basically,
item
potent
by
definition,
so
we
don't
have
to.
We
don't
really
have
to
ask
the
user
to
pay
any
special
attention
there.
A
It's
actually
look.
How
would
this
would
this
look
like
if
we,
if
we
actually
set
this
up
one
on
on
the
system
like
kubernetes,
if,
like
what
components
are
involved
there?
If
we,
if
we
build
this,
there's
actually
there's
actually
surprisingly
few
requirements
that
we
need
for
that.
So
one
thing
is:
we
do
need
a
cluster
of
of
link,
stateful
functions
that
can
be
just
the
deployment
I'm
going
to
talk
about
that
in
in
a
bit.
You
don't
actually
need
to
worry
about
persistent
disks
or
anything
there
like.
A
A
A
A
A
It's
I
think
it's
an
interesting,
it's
an
interesting
idea
and
how
to
how
to
look
at
this,
how
to
how
to
develop
these?
These
are
application
architecture
stacks
right
now.
So
what
what
we
really
have
here
is,
if
you
wish
two
to
three
two
to
three
tiers
right,
we
have
the
application
tier,
which
is
just
arbitrary
user
code,
on
something
like
like
state,
stateless
deployment
or
lambda.
Then
we
have
this
state
and
messaging
tier,
which
is
flink
or
something
else
that
follows
this
paradigm,
and
then
we
have
basically
durable
storage.
A
So
the
durable
storage
here
is
a
completely
separate
thing
right,
it
doesn't.
Storage
is
not
part
of
the
of
the
middle
tier
of
this
data
messaging
tier,
the
middle
tier
is
actually
it
doesn't
assume
any
durability
of
its
local
disks
or
anything
it.
It
uses
them
a
lot
for.
Caching,
but
it
really
assumes
durability
only
of
the
of
this
of
this
durable
storage
tier,
which
in
turn
really
just
has
to
store
files.
A
So
this
gives
us,
I
think,
an
interesting
separation
of
concerns
here.
So
we
have
the
the
state
list
here,
which
is
stateless
and
very
fast
inelastic,
where
there's
the
state
and
messaging
tier,
which
is
still
somehow
stateless.
So
you
know,
even
if
all
containers
die
at
the
same
time,
you
do
not
have
data
loss,
but
it
needs
a
lot
of
local
caching
for
performance.
A
So
it's
moderately
fast
in
its
elasticity,
like
you
can
scale
it
in
and
out,
but
it's
it's
not
completely.
You
know
it
doesn't
happen
every
you
don't
do
it
every
second
like
in
a
in
a
way
that,
for
example,
lambda
would
scale.
There's
there's
a
caveat
here
like
this.
The
way
you
know
apache,
link
works
right
now,
scaling
is,
is
something
that
still
disrupts
message
processing
for
a
certain
amount
of
time,
we're
working
on
making
this.
You
know
less
disruptive,
but
at
the
moment
this
really
like
this
really
means
it's.
A
It's
probably
best
used
for
near
real-time
use
cases
rather
than
you
know
very
hard,
real-time
use
cases
and
the
the
durable
storage
tier.
It's
really
something
where
you
know
scalability
is
not
not
much
of
a
concern,
because
it's
all
about
it's
just
about
adding,
adding
more
disks
right
files
just
live
wherever
they
live.
There's
no
there's!
No,
no
real
need
to
to
think
of
you
know
migrating
anything
or
having
any
having
any
issues
with
scaling
up
and
down
it's
it's
kind
of
a
solved
problem
this
this
tier,
I
would
say.
A
A
Let's
actually
make
let's
actually
look
a
bit
at
how
this,
how
this
looks
in
in
action
and
maybe
give
you
an
example
of
an
an
application
that
we
can
build
with
that,
and
the
the
example
we
picked
here
is:
is
a
billing
application,
so
assume
you,
I
don't
know
you
have
your
spotify
subscription
or
something
like
that
and
every
every
month
or
so
you
you
get
invoiced
and
they
they
subject
they.
You
know
they
pull
something
from
you
via
your
credit
card.
A
How
would
that
look
so,
let's
say
the
first
thing
that
that
we
built
here
is.
We
actually
need
to
know
about
what
subscriptions
exist
and
we
do
want
to.
Let's
say
we
want
to
attach
this
payment
service
to
to
some
existing
infrastructure.
Let's
say
you
have
a
database
that
manages
all
your
users
and
we're
going
to.
You
know
we're
going
to
grab
the
messages
from
there.
Let's
say
by
something
change:
data
capture
or
so
and
putting
it
into
into
a
kafka
topic,
so
they
basically
they're.
A
Basically,
all
the
messages
that
that
describe
changes
to
subscriptions,
then
we
define
we
define
a
user
function
here
that
that
takes
these
events
and
and
this
function
has
in
its
local
state,
let's
say
the
id
the
subscription
status.
The
billing
interval
that
you
that
you
picked
and
so
on.
So
it
basically
has
all
the
information
that
it
needs
for
billing,
and
you
know
if
the
subscription
change
comes
that
says
you
know
this
user
actually
is
not
using
the
service
anymore.
Then
this
function
basically
gets
deleted
or
gets
inactivated.
A
So
what
this
function
does
is
as
a
first
thing.
It
doesn't
need
to,
of
course,
be
working
all
the
time
because
subscribe
subscriptions
is
a
periodic
business.
So
we're
using
another
feature
that
I
didn't
talk
about
here
before,
but
that's
that's
actually
message
message
delayed
message
sending
or
timed
messages,
so
what
the
user
actually
does
is
it
schedules
a
message
for
the
next
for
the
next
billing
date
when
when
the
next
payment
would
be
due
so
the
next
time,
this
message
is
actually
received
by
this
user
function.
A
It
actually
creates
a
new
function,
let's
say
a
payment
function,
and
this
this
function's
sole
sole
purpose,
is
to
to
see
one
one
payment
process
through
you
know
like
talk
to
the
credit
card
service,
make
make
sure
it
handles
failures
and
retries
and
eventually
concludes
after
every
tried
three
times
and
it
was
rejected.
Maybe
I'm
going
to
report
the
spec
that
this
month
wasn't
paid
and
then
the
user
function
can
decide
what
to
do
with
it.
So
this
one
actually
talks,
let's
say
to
credit
card
processing-
and
I
mentioned
it
before.
A
Ideally,
all
communication
to
the
outside
world
works
through
something
like
event,
ingresses
and
egresses.
So
in
this
case
we're
actually,
we
actually
say
we're
we're
talking
to
our
to
the
to
the
credit
card
processing,
let's
say
by
two
to
kafka
topics,
one
to
send
the
want
to
send
the
the
request
and
want
to
handle
the
the
response.
A
So
something
that
that
I
think
why
this
makes
an
an
interesting
example.
Is
that
it's
it's
really,
it's
pretty
annoying
for
users
if
they
actually
get
built
twice
for
something,
and
it's
really,
I
think,
annoying
for
the
service
provider.
If
they,
if
some
some,
some
billing
requests,
get
lost
right.
So
in
let's
say
in
case
there's,
there's
a
failure
of
the
of
the
process
that
handles
a
payment
function.
A
A
A
So
the
first
thing
is
actually
defining
defining
functions
that
the
that
the
you
know,
the
the
stateful
function
event
driven
database
should
should
be
aware
of,
and-
and
we
can
see
this
on
the
left
side-
we're
actually
defining
two
two
types
of
function:
a
subscription,
the
subscription
function
and
the
payment
function,
they're
actually
behind
behind
two
to
http
endpoints
through
which
they
are
invoked.
We
defined
some
state
that
they
have.
A
You
know
some
parameters
around
timeouts
and
message
batching
and
so
on,
and
that's
that's
pretty
much
it
on
the
right
side.
We
actually
see
two
of
the
ingresses
that
we
that
we
defined
so
they're
both
of
the
type
routable
protobuf
ingress
on
kafka,
so
they're,
basically
taking
taking
protocol
buffer
messages
from
kafka
and
they're
they're.
A
Taking
the
the
key
from
from
kafka
is
the
as
the
name
of
the
the
function
that
should
be
involved
invoked
like
the
user
id
or
the
payment
id
the
application
itself
could
then
in,
if
implemented
in
python.
Look
looks
something
like
this
right
on
the
on
the
upper
left
side,
where
we're
looking
at
the
subscription
function
here,
we're
basically
saying
that
we
bind
this.
This
function
here
to
handle
everything
that
goes
to
goes
to
the
subscription
function.
They
could.
A
A
Is
it
a
timer
message
as
a
result
of
a
payment,
and
then
you
know
dispatch
it
to
the
or
yeah
invoke
the
right,
the
right
function
here
and
if
we,
you
know,
if
we
look
at
the
at
the
code
here,
the
most
important
thing
actually
to
notice
is
that
there's
there
isn't
really
any
type
of
there
isn't
really
any
type
of
you
know,
failure
handling
code
in
here
in
the
sense
of
what
happens.
If,
actually
you
know
some
state
update
goes
wrong.
What
happens
actually
do
we
have
to
you
know
retries.
A
A
At
some
of
this,
like
in
action,
I
have
a
recording
here.
I
wanted
to
be
on
the
on
the
safe
side,
with
all
the
all
the
remote
talks
and
so
on.
So
I'm
going
to
use
this
recorded
demo.
It's
not
it's
not
quite
live,
but
I
guess
with
the
remote
with
the
remote
setups.
That's
that's
how
it
goes
all
right.
So
what
we
see
here
is
is
actually
a
deployment
pretty
standard
deployment.
We
have
these.
We
have
stateful
function,
master
and
three
workers.
These
are
pretty
much.
A
Our
few
wish
database
processes.
There
are
three,
you
know,
three
actual
processes
that
do
the
message,
routing
and
and
on
the
state,
one
master
to
coordinate
them,
and
then
we
have
two
deployments
for
the
for
the
subscription
function
called
the
subscription
worker
here
and
for
the
payment
functions,
so
they're
they're,
just
regular,
regular
deployments,
kubernetes
deployments
and
behind
behind
the
symbol,
a
simple
I
think,
cluster
ip
service.
In
this
case
this
is
the
the
left
window
on
the
bottom
here
shows
the
log
from
the
from
the
payment
worker.
A
Just
so
we
can
see
log
messages
later
in
other
parts
of
the
demo.
We
don't
see
much
action
here,
so
I'm
pretty
I'm
just
quickly
showing
you
that
something's
actually
going
on
we're
we're
switching
to
the
ui
from
from
apache
link.
That
gives
us
an
insight
about
data
flowing
around
so
we're
seeing
it's
it's
a
moderate
setup
right,
we're
processing
some
500
events
per
second
in
each
chart.
So
1
500,
all
in
all,
so
that's
a
fairly
low
data
rate
for
this
demo.
A
You
know
we
can.
We
can
do
the
the
common
things
we
can
do
with
apache
link
like
look
at
is
the
application
getting
back
pressured.
Can
we
actually
keep
up
with
the
data
rate,
and
so
on
can
get
some
insight
into
into
tasks?
How
often
did
they
fail
like?
What
is
the
aggregate
data
amount
that
they're
that
they're
receiving
in
in
their
messages?
And
so
on
now?
One
of
the
things
that
is,
as
I
mentioned
here
pretty
crucial,
is
actually
consistency.
A
A
A
It
actually
also
scales
down
pretty
nicely
to
to
very,
like
very
small
setups,
like
like
yeah
small
iot
devices,
which
I
think
is
again
actually
because
of
this
interesting
architectural
approach
that
I
mentioned
earlier,
like
this,
the
separation
of
of
having
in
the
the
messaging
and
stateful
tier,
being
independent
of
the
of
the
durability,
tier
and
and
working
with
asynchronous
snapshots.
I
think
it's
a
very
flexible
and
very
scalable
approach
to
do
things
actually
skip
over
over
the
rest,
because
I
think
I'm
running
out
of
time.
A
I
would
like
to
leave
some
time
for
question.
So
it's
briefly
this.
This
project
is
part
of
apache
link,
it's
developed
by
a
handful
of
people
in
in
addition
to
to
me.
So,
actually,
those
folks
do
a
lot
more
work,
so
they
actually
actually
there's
some
more
credit
than
I
do
for,
for
this
work
just
happened
to
be
the
one
presenting
today
and
yeah.
I
hope
I
hope
I
could
give
you
a
a
rough
overview
of
what
we're
what
we're
doing
in
this
project,
and
I
I
yeah.
A
I
hope
I
could
actually
interest
you
a
little
bit
in
this
in
this
idea
of
of
rethinking
rethinking
this
approach
to
to
application
building,
especially
event
driven
application
building,
and
this
like,
if
you
want
to
take
two
things
away
from
from
this
talk,
I
would
say
take
away
that
this.
This
inversion
of
the
role
of
of
database
and
application
is
actually
a
pretty
a
pretty
neat
trick
like
making
the
database
also
responds
responsible
for
the
message,
processing
and
making
it
the
acting
part
and
the
application.
A
This
is
still
you
know,
there's
still
a
lot
of
work
to
be
done
on
that
level.
It's
getting
it's
getting
better
with
every
with
every
release,
but
I
think
it's
very
it's
a
very
promising
promising
architecture,
promising
approach
to
do
things
yeah.
That
is
it.
Thank
you
very
much,
and
I'm
I'm
releasing
the
screen
here
and
handing
it
back
to.