►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Serverless is Interesting But FaaS is Not Enough - Jonas Bonér, Lightbend Inc.
Serverless is interesting because it emphasizes automation of infrastructure, but current Function-as-a-Service implementations are not suited for building general purpose apps. What’s missing is a way to model and manage stateful services. Can we make stateful apps run as if they were stateless? We’ll investigate the inherent challenges with serving stateful services from a scaling perspective while preserving responsiveness by designing for resilience and elasticity, using Akka—an OSS library for distributed stateful apps—as a concrete example. We’ll discuss design principles needed to address problems around data consistency, system integrity, and data locality, why they are key to maintain overall system behavior under failure and load and discuss pitfalls and requirements needed to understand when operating the infrastructure fabric—f.e. Knative—which orchestrates the services.
https://sched.co/MPeO
A
Welcome
everyone
thanks
for
joining
me
in
this
session,
about
bus
service
about
service
and
in
particular,
stateful
service,
a
more
descriptive
title
would
probably
been
towards
stateful
service,
but
once
I
figured
that
out,
it
was
too
late
to
change
the
abstract.
Anyway.
My
name
is
Jonas
blue,
near
I'm,
the
CTO
and
co-founder
of
light
urban
decay's.
It's
a
company
behind
Scala
programming
language.
You
might
have
heard
of
ARCA
play
web
framework
and
some
other
and
some
of
the
tools
and
I'll
start
this
discussion
by
trying
to
frame
it
to
explain
to
you.
A
You
know
sort
of
the
title
of
the
title
of
applications
that
I
think
really
needs
to
live
a
really
really
leveraged
stateful,
stateful
design
and
a
stateful
thinking,
so
you
so
we
can
sort
of
frame
it.
This
discussion
to
the
type
of
systems
that
it's
most
applicable
to
so
starting
with
some
industry
trends,
I
mean
the
last
10
15
years
have
radically
changed
for
us
as
developers
we
use
it
drastically
new
new
world.
You
know
we
started
sir
we've
started
from
you
know.
A
It
was
not
that
long
time
ago
that
we
would
develop
for
single
core
machines,
running
on
C,
no
single
core
developers
or
bus
or
a
single
core
processor
running
a
single
zone.
Single
machines
with
you
know
we're
late
to
see
it
was
you
within
within
seconds
was
fine
and
we
we
had
to
tackle
sir
moderate
amounts
of
data,
but
today
it's
a
radically
different
world
I
mean
distributed
systems
is
here
to
stay.
When
you
embark
on
macro
services,
you
can't
avoid
it.
A
You
know
you're
entering
this
world
of
distributive
systems,
where
there's
a
lot
of
uncertainty
between
the
nodes.
You
know-
and
you
know,
messages
can
fail
and
and
we'd
be
reordered
and
all
these
things-
and
you
know
we
have
a
so
and
we
are
multi-core.
We
have
cloud
computing
with
IOT
and
mobile.
You
know
we
have
this
called
the
whole
sort
of
category
of
streaming
apps
and
big
data
apps
and
an
AI
and
all
of
those
things.
It's
radically
different.
You
know,
and
we
remove.
Of
course,
a
lot
of
apps
need
to
tackle
these.
This
deserve.
A
You
know
never-ending
stream
of
data
in
the
real-time
fashions.
So
we
have
this
whole
category
of
apps
having
to
tackle.
You
know
real
time:
data
centric,
streaming
type
of
applications
serving
students
to
grab
data
while
it
flies
by
and
be
able
to
present
the
user
with
intelligent
answers
almost
immediately
and
finally
we're
moving
towards
a
world
where
we
are
sort
of
automated
operations.
I
I
tend
to
call
them
ops
less.
You
know
some
people
call
it
different
things,
but
this
is
the
world
of
service.
A
The
server
lists
are
really
shown
us
the
path
that
was
possible
when
it
comes
to
these
things-
and
you
know
them
I
think
the
monolith
and
the
three-tier
architecture
has
really
served
as
well,
but
it's
it's
it's
death
is
really
overdue.
I
believe
you
know,
the
classic
three-tier
architecture
is
usually
have
to
has
a
lot
of
problems.
It's
usually
asked
like
to
highlight
to
see
all
these
round
trip
down
to
the
database.
A
You
know
with
transactions
and
locks
sort
of
stop
the
world
type
of
traffic
typo
locks
makes
it
really
really
hard
to
to
build
really
high
throughput
highlight
really
low
latency
type
of
systems,
and
that
might
be
fine,
sometimes
in
your
pub.
But
for
a
lot
of
these
new
new
tackle
use
cases,
it's
actually
offered
to
slow.
You
know
you
have
to
high
contention
too
much
coupling
between
components.
It's
really
hard
to
build
through
truly
available
systems,
because
you
can
easily
get
into
this
problem
with
cascading
failures.
A
Context
means
that
we
can
actually
serve
serve
users
with
intelligent
answers
faster
and
faster
or
or
as
fast
as
as
needed.
You
know,
but
but
there's
a
lot
of
problems
that
comes
with
that
I
mean
data
now
needs
to
be
replicated
between
nodes
for
availability,
realities,
shoving
it
down
to
the
database
all
the
time
and
you
and
you
need
to
fully
embrace
asynchronous
processing,
as
well
as
asynchronous
I/o
and
asynchronous
communication,
and
you
know
we
activists
really
started
to
become
the
framework
or
that
a
lot
of
people
use
or
frameworks
use
in
order
to
conceptualize.
A
You
know,
what's
needed
the
design
practice
and
the
practices
that's
needed
to
tackle
these
new
challenges
to
react
to
manifested
was
created
but
five
years
ago
to
try
to
frame
this
discussion.
So
to
speak,
and-
and
you
know
you
know,
we
move
into
world
or
through
a
world
where
we're
data
is
king,
I
mean
data's,
have
been
important
for
a
long
time
and
the
companies
have
been
storing
down.
You
know
vast
amounts
of
data
down
as
data
lakes,
but
but
a
lot
of
people.
A
A
How
fast
can
you
can
you
deliver
through
intelligence
or
knowledge
about
the
data
that
you
receive,
intelligent
answers
to
your
users
and,
and
and-
and
we
started
to
see,
sir,
this
convergence
between
the
market
services
or
the
app
application
development
world
were
that
that
often
needs
to
be
very
data-centric
in
the
bird
data
focused
as
well
as
the
big
data.
It
also
need
to
need
to
store
reply
with
intelligently
in
in
in
short
and
short
term
space.
A
So
we
see
this
convergence
into
what
we
call
what
I
call
fast
data
applications,
so
this
is
sort
of
the
new
type
of
apps
that
I'm
sure
discussing
that
we
need
to
tackle
and
again
can
fall
short
when
with
this,
with
with
the
current
incarnation
of
service
so
to
speak.
You
know,
I
took
this.
This
quote
from
from
him
from
a
positioning
paper
by
Berkley
that
came
out
a
couple
of
months
ago,
where
this,
where
they
say
that
I'm
and
I
quote,
we
predict
the
service
computing
will
grow
to
dominate
the
future
of
cloud
computing.
A
I
really
believe
this
is
true,
even
though
the
title
might
have
well,
my
talk
might
have
bish,
you
know.
I
was
less
enthusiastic
when
I
when
I
wrote
the
abstract.
Perhaps
you
know
I
really
believe
the
service
is
great,
because
it
gives
us
sort
of
this
unified
solution
to
both
sir
sir
sir,
in
throw
a
one
way
to
catching
operational
costs
as
well
as
development
costs.
You
know,
you
know
you
know,
being
a
beast
or
cost
efficient
pay-as-you-go
scales
scaled
down
to
zero
as
way
as
well
as
scalar
asked.
A
The
pastor
of
the
the
load
varies,
which
is
very
often
will
do
when
you
put
them
on
the
on
on
line.
You
know
with
the
users
are
unpredictable
in
you
know
a
slash
dot.
It
will
used
to
be
turned
when
I
when
I
started,
but
it's
really
developing,
there's
really
becoming
more
and
more
important.
Today
you
know
Black
Friday
or
whatever
you
know,
I'm,
probably
both
unpredictable
load
as
well
as
predictable
load
that
you
can
plan
for.
So
you
know,
an
automation
is
one
thing
that
really
is
really
a.
A
It
has
made
serve
really
really
great
with
when
it
comes
to
service,
automation
of
scale,
automation
of
failure,
handling,
automation
of
recovery
and,
in
general,
the
great
developer
experience.
You
know
to
actually
moving
to
the
cloud
because
that's
hard
and
it-
and
you
know
this-
is
a
lifted
shift
model
where
you
bring
along
along
a
law
along
a
lot
of
your
you
know,
historic
knowledge
patterns
and
and
and
even
tools
and
products.
A
You
know
really
that
doesn't
work
and
the
answer
LS,
you
know
shows
the
way
here,
but
I
can
actually
be
a
simpler,
a
simpler
world,
moving
to
the
cloud,
not
a
more
complex
one
by
cyril,
doing
a
lot
of
the
heavy
lifting
and
finally,
in
the
reduced
time,
complexity,
risk
but
but
I
think
it's
very
important.
You
know
a
lot
of
people,
they
confuse
server
lists
with
function
as
a
service.
You
know,
they're
overlap,
absolutely
I,
think
I,
think
I
think
for
us,
I
think
function
as
a
service
is
a
great
first
step.
A
A
Think
service
really
shows
a
glimpse
of
the
future
of
cloud
computing
what's
possible
and
we
see
a
lot
of
initiatives
like
like
the
KNAT
product,
for
example,
trying
to
expand
on
this
or
the
notion
of
a
survey
list
means
and
and
and
put
that
into
into
sort
of
a
context.
That
is
that
that's
broader
than
then
function
as
a
service,
which
is
great.
A
You
know,
I,
don't
know
I'm,
not
really
here
to
pick
about
functions
or
service
per
se,
but
I
think
I
just
try
to
explain
what
I
think
is
good
and
bad
about
it
anyway.
So
we
know
where
I'm
starting
from
here
so
I
said.
It's
I
think
it's
really
good
that
it's
repaid
the
way
for
service
shown
was
possible,
but
it's
just
one
incarnation
of
the
service
idea,
I
believe
the
first
step,
but
it's
or
showed
us
it's
possible
to
scale
on
demand
from
zero
to
tens
of
thousands
of
requests.
A
You
know
in
a
really
cost-efficient
manner.
You
know
the
economical
scale.
You
can
really
play
at
your
advantage
here.
The
realists
simplify
its
delivery
of
scaleable
and
available
applications.
You
know
makes
it
actually
possible
to
move
to
the
cloud
and
reduce
risk
instead
of
increasing
risk,
so
it
encourages
good
distributed
system
design.
You
know,
forces
you
to
focus
on
a
known
event,
an
event
driven
all
events.
First
to
design
I,
don't
know
if
you
heard
open
of
events
for
DDD,
but
it's
something
that's
proven
to
work
very.
Very
well,
you
know
gives
you
you
know.
A
A
Okay,
applications
of
this
or
use
cases
of
this
is
like
low
traffic
applications
like
like
IT
back-end
systems,
for
example,
stateless
web
applications,
where
you
can
serve
the
content
from
s3,
for
example,
or
embarrassingly
parallel
processing
tasks
where
there's
really
no
coupling
between
your
your
tasks,
but
they
keep
meaning
no
contention.
There's
new.
This
is
really
no
need
to
coordinate,
but
you
can
just
scale
it
out
in
what
is
called
you
know
in
academia.
You
would
embarrassingly
parallel
sort
of
type
of
patterns.
Examples
of
this,
like
is
like
resizing
images.
A
It's
great
for
orchestrating
functions
like
integration,
coordinating
or
a
set
of
third-party
services,
and
also
some
composing
chains
of
functions
to
serve
that
where
you
were
very
sort
composed
and
Beto's
on
data
dependencies
or
driven
through
through
the
need
of
the
data.
Also
job
scheduling,
of
course,
like
a
Crone
type
of
jobs
and
things
like
that.
But
then
what's
bad
with
function
as
a
service
and
I
think
I
think
it's
really
really
hard
to
build
general
purpose
applications.
Most
of
the
use
cases
we've
seen
here,
I
mean
are-
are
usually
part
of
a
larger
application.
A
So
you
see
that
this
means
that
that
that,
for
that
part
of
your
app
it's
a
great
fit
for
service,
but
but
what
about
I
mean
building
general
applications?
I
mean
ideally
I'd
like
to
see
everything
moved
to
the
serverless
idea.
If
you
don't
need
to
have
to
manage
service,
really
means
sort
of
having
knobs
to
turn.
You
know
in
instead
of
all
the
hassles
of
of
distribute
systems
and
operations
of
the
cloud.
A
Some
of
some
of
the
problems
in
more
in
more
detail
is
that
I
I
think
it
retains
a
lot
of
a
lot
of
limitations
with
all
the
three-tier
architecture
that
are
talking
about.
One
of
the
problems
is
that
that's
all
for
the
advantage
I
have
to
say,
but
in
general
purpose
applications.
One
of
the
problems
is
that
functions
are
stateless,
ephemeral
and
short-lived
I'll
be
I'll.
Get
back
to
that
now.
Is
that
there's
no
direct
addressability?
You
always
need
to
communicate
over
some
sort
of
storage,
medium,
which
is
great.
A
You
know
for
durability
and,
and
things
like
that,
but
it
is
actually
possible
to
me
to
make
it
as
as
as
reliable
with
point
to
point
and-
and
you
know,
without
having
to
pay
the
latency
of
always
going
down
the
communicating
over
some
message
broker,
you
know
did
as
I
said.
Different
use
cases
have
different
requirements,
but
always
have
them
do
that
is
quite
bad
for
general
purpose:
the
distributed
systems.
A
Another
thing
is
that
there's
really
no
notion
of
collocation,
where
I
can
physically
say
I
would
like
to
co-locate
processing
and
state
over
there,
so
it
doesn't
need
to
go
fetch
it
every
single
time.
You
know,
you
know,
location
location,
look
how
that
your
reference
is
something
that
we,
where
we
value
a
lot.
You
know
when
we,
when
we
working
with
more
low-level
type
of
programming-
and
it's
I
mean-
and
it's
actually,
the
cost
is
even-
is
even
greater.
With
the
higher
up
collaterally,
we
client,
we
also
have
limited
options
for
managing
and
coordinating
distributed.
A
State
in
a
state
management
is
a
hardest
thing
in
distributed
systems
right
now,
I
would
like
to
have
more
wider
range
of
options
for
managing
these
things,
supporting
more
use
cases
limiting
more
options
from
modeling
consistency
guarantees.
It's
not
a
one-size-fits-all
everything
is
drawing
the
consistence
or
everything
is
eventually
consistent.
You
know,
but
I'd
like
to
you
note,
be
able
to
use
whatever
is
needed
for
the
actual
problem.
A
You
know
strong
consistency
should
not
be
the
default,
in
my
opinion,
in
distributed
systems
because
there's
just
too
much
drawbacks,
even
though
it
makes
the
world
easier
for
us
program
is
when
we're
critical
in
one
we
code.
You
know
it
makes
the
system
more
brittle
more
less
available
as
things
like
that,
on
the
other
hand,
always
having
it
eventually
persistence.
It's
not
good
either
because
there
are
certain
type
or
use
cases
that
require
strong
consistency.
So
more
consistency.
Our
cart
would
be
nice.
Okay,
also
limited
options
for
managing
durable
state
in
general,
okay.
A
So
what
I
want,
if
I
should
try
to
like
capture
it?
Very,
very
briefly,
is
I
like
the
server
less
experience,
but
for
general
purpose.
Applications
I'd
like
to
put
all
our
clients
at
like
meant
that
that's
using
ARCA,
you
know
fast
data
on
server
less,
eventually,
that's
not
an
option
today
you
know
and
I
sir
I'd
like
to
have
you
know
some
sort
of
abstraction
for
stateful
functions,
which
I
will
dive
into
more.
What
I
mean
come
complementing
stateless,
you
know
not
replacing
them.
A
A
You
know
a
service
most
malls
are
turing-complete,
so
you
can
essentially
do
everything
if
you
follow
me,
but
but
being
able
to
support
these
use
cases
in
in
a
really
available
fashion,
or
our
use
cases
like,
for
example,
training
is
serving
on
machine
learning
models
and
essentially
any
dynamic
sort
of
in
in
memory
model
that
that's
constantly
being
updated
and
needs
to
be
served.
Okay,
real
time
to
distribute
the
stream
processing.
A
You
know
concrete
problems
like
like,
like
prediction,
recommendation
engines,
anomaly,
detection,
fraud,
detection
and
these
type
of
things,
modeling
user
session
or
shopping
carts
and,
and-
and
you
know,
caching,
essentially
any
sort
of
in-memory
data
in
memory
serve
construct
that
throughout
lives.
You
know,
requests
that
has
its
own.
A
You
can
model
your
own
sort
of
life
cycle
for
so
it's
available
until
you
explicitly
destroy
it,
and-
and
in
you
know,
things
like
distributed
resilient
transaction
workflows
like
in
event-driven
programming,
talked
a
lot
about
the
saga
patter,
which
is
which
is
which
is
a
really
nice
way
of
modeling.
You
know
distributed
workflows
in
a
reliable
fashion
without
having
to
go
back
to
XA
transactions.
You
know
which
will
doom
us
Modelling
shared
collaborative
workspaces.
You
know
things
like
document,
editing,
blackboards
type
of
systems
and
and
also
being
able
to
in
a
good
way
ma.
A
You
know,
sir
modeling
class
eg
distributed
computing
patterns
like
leader,
election,
etc,
would
be
nice
as
well.
So
some
of
the
technical
requirements
in
order
to
get
here
I
think
is
that
first
we
need
to
have
some
sort
of
abstraction,
some
sort
of
possibility
to
create
stateful
long,
lived,
addressable,
virtual
components
or
functions.
You
know
and
what
I
mean
what
and
what
do
I
mean
by
that
yeah
first
stateful
I
mean
in
memory
durable
resilient
state,
not
necessarily
stay
down
there.
A
You
know
or
not
stay
down
there,
but
stayed
up
here,
but
they're
still
backed
by
durable
state.
Well,
let's
explain
more.
How
long
lived
I
mean
that
it's
actually
outlived
specific?
You
know
an
individual
requests,
it's
more
tied
to
specific
session,
that's
available
until
explicitly
destroyed.
Okay,
addressable
mean
it
means
that
it
should
be
referenced
through
stable
address,
that's
always
available,
even
though
I
send
the
rent
they
address
around,
even
though
you
know
I
them
they
actually,
the
actual
instance
of
the
function
might
be,
might
be
moved
around.
A
I
also
would
like
to
have
sort
of
support
for
a
wide
range
of
communications
and
coordination
patterns,
and
also,
as
I
talked
before,
options
for
managing
consistency.
You
know
managing
distributed
state
at
scale
while
not
giving
up
the
guarantees
that
we
have
set
once
once
set.
You
know
also
things
like
intelligent
and
adapted
placing
or
functions
being
able
to
do
that
in
intelligent
fashion,
I
mean
where
the
state
currently
is,
and
vice
versa.
So
you
can
have
colocation
physical
colocation
of
stage
and
processing
for
really
really
low
latency.
A
A
Oops,
okay,
is
that
my
sign
okay,
so
waste
of
also
one
way
so
managing
end
to
end
correctness,
not
just
thinking
about
each
individual
function,
but
actually
thinking
about
you
know
the
larger
scope,
oh
and
then
being
able
to
reason
about
the
guarantees
that
the
whole
workflow
has
as
a
whole.
I
think
it's
extremely
important
task
complexity
grows
and,
of
course,
I.
Don't
predict
the
performance,
latency
and
throughput.
A
So
you
know
what
one
of
the
great
things
with
function
as
a
service
is
that
it
gives
the
way
away
a
very
nice
way
of
abstracting
overcome
or
overcome
communication.
You
know
you
have
you
have
sort
of
enable
in
a
really
abstract
way,
message
coming
in
and
message
going
going
and
going
out,
you're
in
and
you
know
and
and
the
framework
or
the
runtime
man
is
all
the
nitty-gritty
details
for
making
this
happen.
A
You
know
because
it
really
doesn't
know
anything
about
how
you're
actually
using
the
database.
You
can't
really
do
a
lot
of
scaling.
For
example,
I
mean
if
it
function
response
slowly.
Is
it
because,
because
the
function
is
overloaded
or
is
it
because
the
database
is
overloaded
the
framework?
A
really
really
can't
know?
A
Okay,
so
I
really
don't
believe
that
this
is
is
true.
Service
service
in
general,
I
think
I'm,
sure
unconstrained
database
access
makes
it
really
really
hard
to
automate
operations
and
and
in
a
crud,
create
read,
update,
delete
that
we
use
with
with
with
with
single
databases.
It
is
really
a
problem
here
because
it
makes
it
really
hard
to
understand
the
intention
of
each
each
each
request.
Is
it?
Is
it
a
read
or
is
it
a
right?
You
know:
can
it
be
cached
can't
the
consistence
really
requirements
be
relaxed
or
not?
You
know,
etc.
Right.
A
A
You
know
analyst
up:
options
can
kill
productivity,
we
kill
urgency.
You
know
so
so.
I
think
I
think
that
sir
limiting
the
options
for
state
and
database
management
is
a
connection
would
be
a
good
thing
and
we
can
do
that
by
trying
to
sir
abstract
over
state.
Instead,
okay,
we
already
have
an
abstraction
from
messaging
a
message
out:
okay,
what
if
we
have
an
abstraction
way
of
abstracting
for
state
in
and
state
out
instead
and
having
an
having
sort
of
the
the
framework
manage
all
the
neat
degree
to
details
of
this
before
us?
A
Okay,
you
know,
crud
really
doesn't
work
that
well
in
this
mall,
because
we
can't
pass
the
whole
database
in
and
out
every
single
time.
So
the
question
is:
if
we
can't
use
crud,
then
what
can
we
use
in
experience
to
really
good
models
for
modeling
distributed
state
scale?
You
know
or
our
event
sourcing
and
see
our
duties?
How
many
have
heard
about
event
sourcing
and
knows
what
it
is?
Okay,
almost
everyone,
that's
great
what
about
CR
DTS,
they
stand
for
conflict-free
replicated,
they
attacked
less.
A
So,
okay
I'll,
try
to
just
explain
briefly
anyway
for
those
that
they
don't
know
what
it
is,
so
you
at
least
some
some
context.
Okay
and
let's
say
we
have
an
events,
the
worst
function.
Now,
what
would
that
look
like
like?
We
have
one
user
sending
a
command
because,
with
event
sourcing,
you
tend
to
have
commands
and
events
commands
are
framed
in
the
in
the
imperative
or
phrased
in
the
imperative,
for
example,
create
the
order,
so
an
create
order
command
is
sent
down
to
this
function.
A
A
You
know
the
fact
that
we
already
have
created
an
order,
so
events
are
usually
named
order
created
the
fact
that
the
order
has
already
been
created
and
any
events
or
see
what
we
do
is
that
we
just
store
all
these
state,
changing
events
or
or
for
events
representing
a
state
change
in
an
event
log
durable
on
disk
in
the
order
they
arrived.
Okay,
we
can
also,
you
know,
have
others
subscribe
to
to
to
that
event.
Log
if
other
parties
are
interested
in
knowing
what
you
know
the
updates
or
the
events
created
by
that
function.
A
Okay,
this
is
a
similar
to
functions
as
a
service.
Today,
this
is
a
happy
path.
Then
what
happens
any
of
things,
the
sort
of
fail
you
know
or
if
you
just
need
to
activate
and
passivate
the
function,
so
the
sad
path
when
we
now
recover
from
failure,
the
only
thing
we
need
to
do
is
sort
of
replay
the
event
log
in
the
order.
The
events
arrived.
You
know
they're,
of
course,
optimized.
A
We
never
do
destructive,
updates
or
compare
us
or
update
in
place
like
we
do
with
with
with
sequel
databases,
but
all
everything
that
ever
happened
in
the
system
is
there.
You
know
in
the
order.
You
know
it
happened
even
better.
This
allows
us
to
work
with
the
durable
in
a
memory
state
you
know
without
having
to
do
any
object-oriented
mapping.
We
never
need
to
go
to
our
fetch
the
state
every
single
time
we
want
to
use
it.
Is
there
it's
in
memory,
but
still
backed
by
durable
event?
A
Log
things
go
fail
with
just
three
instance
or
rehydrate.
Okay.
It
also
allows
others
to
subscribe
to
state
changes.
It's
really
good
mechanical
sympathy.
What
that
means
it
is
in
line
with
modern
hardware.
You
know
if
you
have
single
right,
you're,
just
riding
down
very,
very
fast
works,
perfect
with
SSDs.
You
know
down
into
the
event
log
and
you
have,
and
you
can
replay
this
log,
not
just
when
things
go
fail.
You
can
do
it
for
all
it
reasons.
You
know
debugging
research,
you
know
replication,
you
have
multiple
hot
replicas
and
things
like
that.
A
So
the
question
is,
then:
how
does
the
event
sourcing
fit
into
this
state
model?
Yeah?
You
have
state
in
your
event,
log
your
if
the
replay
all
them
in
one
by
one,
the
events.
The
syrup
arrived
up
to
this
point
and
and
state
out
this,
essentially
the
event
that
you
emitted
so
as
representing
the
latest
state
change?
Okay,
so
this
is
great
to
use
as
a
model
to
model.
A
You
know
what
you
normally
will
use
crowd
for
for
your
your
your
source
of
truth,
your
domain
model
that
type
of
persistence-
okay,
I've
also
mentioned
that
C
or
DTS
are
also
great.
I
mean
these.
They
are
also
a
really
good
way
of
mounting
state
but
probably
but
in,
but
for
slightly
other,
the
other
other
type
of
use
case.
Okay,
so
so
it's
very
implements
the
ideal
asset.
Udo
I,
don't
know
if
you
heard
about
that
for
a
funky
funky
acronym
for
sort
of
it
stands
for
it's
really
good
in
really
nice.
A
What
you're
thinking
about
the
state
in
distributed
systems?
So
a
here
stands
for
associative,
meaning
that
grouping
doesn't
matter.
Okay,
that's
that's!
Really!
Good
property.
C
stands
for
commutative,
meaning
ordering
doesn't
matter,
is
of
the
great
property
and
distributed
system,
and
I
stands
for
idempotent,
meaning
the
duplication
doesn't
matter.
You
can
actually
receive
the
same.
This
is
the
same
request
more
than
one
more
than
once
and
see
it
and
see
here.
A
The
T
stands
for
conflict-free
replicate
data
types,
it's
a
fairly
recent
research,
but
we
abuse
it
in
ARCA
and
our
clients
and
customers
abuse
it
for
quite
a
few
years.
Now
it's
a
term
if
you
know
reactive,
if
you
remember
that
database
cellular
that
it
went
away,
it
was
one
of
the
first
actually
to
implement
CEO
duties
in
production
environments.
So
it's
or
CEO
duties
essentially
is
editing,
implements
attitude
that
oh,
it
has
strong
eventual
consistency
is
replicated
and
decentralized,
and-
and
you
know,
and
then
what
the
really
nice
property
is.
It
has
this.
A
It's
always
converged
correctly,
so
you
cannot
oldc
or
do
these
data
types
flowing
around
and
eventually
they
will
always
converge
in
a
correct
fashion,
okay,
which
makes
it
extremely
available
and
extremely
scalable.
Okay-
and
you
know,
the
the
data
types
that
are
catalysts
are
discovered
by
the
research
community
that
we
have
implemented
it
slice
counter
register,
sets
Maps
even
graphs
and
the
nice
thing
with
these.
You
know
as
if
they
are
a
certainty
they
compose.
A
So
you
can
have
a
map
of
counter,
for
example,
or
a
set
of
maps
of
counters
or
registers
or
something,
and
they
all
adhere
to
the
same
type,
tab
properties.
Okay.
So
what
does
that
look
into
our
state
state
model
here
that
yeah?
We
will
have.
He
said
the
event
log.
In
now
we
have
we
have
the
deltas
in
the
deltas,
the
changes
between
the
different
updates
of
the
CRE
T's
and
out.
We
have
the
deltas
out.
A
You
know
so
it's
nicely
abstracts
over
CR
DTS
as
well,
and
sincere
duties
are
great
for
for
doing
distributed,
coordination
of
state,
for
example,
without
ever
going
to
serve
a
durable
medium.
You
can
do
that.
We
have
options
for
that
in
akka
and
many
many
implementations
too,
but
if
you
just
have
enough,
knows
enough
replicas
that
you
replicate
to
it's
available
enough,
so
you
never
have
to
really
persist
it.
A
A
I
mean
that
we
actually
try
to
do
try
to
do
some
work
around
this
as
well
at
Lightman,
so
currently
working
on
POC,
it's
all
open
source,
I'll
link
to
it
in
the
end
on
top
okay,
okay,
an
ATM
on
top
of
kubernetes
on
top
okay,
an
ad
with
an
around
gr,
PC
and
ARCA,
then,
but
but
first
then,
before
I
dive
in
a
little
bit
about
how
we've
done
this.
How
many
knows
well
know
what
our
Chi
is?
A
First
about
about
half
so
I'll
just
explain
it
very
briefly:
sort
of
an
implementation
of
the
action
model.
If
you
heard
about
Erlang
I
mean
that's
like
I
created
ARCA
some
time
ago
and
I
was
heavily
inspired
by
buyer
buyer
line.
So
and
it
has
this
notion
of
ARCA
cluster,
that's
sort
of
decentralized
self-organizing,
very
simple.
To
do
that.
The
Dynamo
model
for
for
for
for
organized
for
having
so
these
these
actors
for
self-organizing
and
making
the
very
very
available
and
and
and
and
scalable,
so
it's-
and
this
is
really
high
proof
high
throughput.
A
We
implement
a
ton
of
different
protocols,
but
the
most
important
thing
is
that
we
have
really
good
support
for
distributed
state
management,
that's
what
ARCA
does
well,
you
know,
and
that's
that
also
one
of
the
hardest
thing
you
can
do
in
distributed
systems.
If
you
want
to
find
out
more
look
at,
you
can
go
to
our
code
IO.
A
Okay,
the
role
of
state
for
serving
is
that
it
speaks
TR
proceed
with
these
functions.
It
rouse
inbound
communication
to
from
the
these
functions
it
communicates
with
the
database
on
behalf
of
the
of
the
of
the
functions
abstracting
away,
all
that
hard
things,
and
it
also
meets
events
to
K
native
events
for
public
consumption.
Okay.
So
if
we
dive
into
Canada
serving,
then
how
does
it
really
look
inside
then
yeah
inside
there?
We
have.
A
We
inject
one
octa
sidecar,
instead
of
the
regular
k
native
side,
for
you
know,
into
into
the
user
function
that
works
the
source,
it's
as
a
proxy
on
behalf
of
each
these
functions.
These
manages
the
state
and
the
communication
on
behalf
of
these
of
all
these
functions,
and
then
one
thing
that
ARCA
does
really
really
well
is,
for
example,
cluster
shorting.
A
So
so
so,
in
this
case
we
have,
we
have
for
each
function
that
we
call
an
entity
then
has
an
entity
key,
anarchic,
arca,
archa,
an
octave,
an
arc,
a
cluster
actually
shirts
or
shorts
these
across
all
the
pods
and
making
sure
the
we
have
Steve's
sticky
routing
to
for
for
each
one
of
these
functions.
So
if
we
have
really
good
they
data
locality
on
each
one
of
these
two
of
these
pods
for
each
request
coming
in
okay,
so
so
so
a
cluster.
A
By
the
way
is
it
sort
of
a
gossip
based,
decentralized,
masterless,
peer-to-peer
tack,
book
of
implementation,
so
Sokka
here
abstracts
in
managing
all
the
distributed
state.
Even
if
you
use,
for
example,
event
logging
see
an
or
see
our
duties
or
any
future
type
of
distributed,
state
type
or
management.
Okay
and
and
of
course,
these
functions
can
be
implemented
in
any
type
of
language,
so
just
just
to
wrap
up.
You
know:
I,
really
believe
that
the
service
is
revolutionary.
I
think
it
shows
blimps
of
as
possible
with
cloud
computing.
A
I
think
fractional
service
is
a
great
first
step,
but
I
would
be
so
devastating
it
was.
It
was
the
only
step,
I
think
it
shows
what's
possible,
but
it's
only
the
first
step.
We
need
to
take
it
where
it
I
mean
to
make
it
live
up
to
its
promise
so
to
speak,
okay
and
and
I
think
service
to
the
row
or
whatever
you
want.
You
want
it
to
you.
Would
you
want
to
call
it
knees
around
time,
programming,
molar
for
general
purpose.
A
Applications
I
want
to
build
everything
in
this
new
within
this
new
style
right
and
we
have
said,
be:
we've
started
working
on
it
with
K
native
our
kanji,
our
PC,
and
we
need.
We
need
your
help.
You
know
it's
it's
open
source,
we
need
feedback,
you
know
our
ideas
or,
if
you,
if
you're
interested
in
hacking,
learn
to
roll
up
your
sleeves
and
hack
on,
we,
we
created
a
little
it's
about
a
month
old.
This
is
this
big
bitly
link
is
to
your
little
little
demo.
A
If
I
would
have
had
an
hour
here,
I
would
have
given
the
demo,
but
now
we
recorded
it
instead.
So
if
you
wanted
to
find
out
a
little
bit
more,
what
we're
thinking
about
what
the
user
experience
could
look
like
and
the
github
repo
is
right
there,
so
you
can
just
check
it
out
and
clone
it
or
and
and
get
started
if
you're
interested
in
these
things
file
issues
or
whatever.
Thank
you.