►
From YouTube: Microservices for the Masses – Integrating a Distributed OS with Kubernetes - Q. Hoole & I. Zhang
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Microservices for the Masses – Integrating a Distributed OS with Kubernetes - Quinton Hoole, Futurewei & Irene Zhang, Microsoft Research & University of Washington
We discuss our experiences integrating an open source distributed operating system (Amino) with Kubernetes to make reliable distributed microservices much easier to write. Writing reliable distributed microservices is notoriously difficult, even with Kubernetes help. In the absence of a coherent and cohesive set of standard open source distributed computing primitives available across all programming languages, things get really hard. We take for granted standard primitives provided by Linux like file systems, virtual memory, threads, network sockets, and locks. Without them, modern application development would be almost impossible. Yet distributed systems require different primitives for things like consistent replication, sharding, distributed shared memory, load balancing, failure handling, and auto-scaling. What should these primitives look like, and how do they work?
https://sched.co/MPXr
A
Right
good
afternoon,
everyone
thanks
very
much
for
staying
right
to
the
end.
To
listen
to
us.
We
hope
to
make
it
worth
your
while.
My
my
group
and
Irene's
groups
have
been
collaborating
for
the
past
few
years,
actually
on
a
pretty
exciting
project
to
take
the
research
that
she
did
as
part
of
her
PhD
thesis
and
expanded
to
a
fully
fledged
open-source
project
within
the
CNC
F.
So
that's
what
we're
gonna
talk
to
you
about
today.
A
A
One
of
my
colleagues,
the
new,
will
give
us
a
demo
of
some
of
the
stuff
in
action.
Hopefully,
that'll
help
you
do
kind
of
some
of
the
stuff
is
a
bit
abstract
and
difficult
to
wrap
your
head
around.
So
demos
can
be
useful
to
illustrate
what's
possible,
we're
pretty
short
of
time.
We
got
a
lot
of
material
to
cover,
so
I'm
gonna
move
fairly
quickly,
and
we
should
have
some
time
being
the
last
session.
I
guess
we'll
have
some
time
to
overlap
with
Q&A
right
so
long
time
ago,
applications
were
pretty
simple.
A
A
Life
was,
you
know,
pretty
good
single
operating
single
machine,
operating
systems,
work
fairly,
well,
Linux,
Windows,
Mac,
etc,
Android
even
and
all
of
these
provide,
you
know
great
abstractions
for
those
kinds
of
applications,
local
processes,
virtual
machine,
sorry,
virtual
memory,
file
systems,
locks.
All
of
these
things
are
kind
of
indispensable
when
writing
applications
on
a
single
machine
and
for
the
most
part
you
know.
If
you
wanted
to
build
an
application,
you
could
do
it
in
one
or
maybe
two
languages,
and
everything
was
good.
A
The
problem
is
that
relatively
suddenly
everything
changed
so
cloud
computing
happened,
everybody
moved
to
smartphones,
mobile,
first
strategies,
etc.
The
ubiquitous
connectivity
happened
with
high
speed,
high
bandwidth
networks
and
suddenly
everything
changed
so
the
types
of
applications
changed.
Suddenly
we
had,
you
know
social
media
and
these
kinds
of
things,
but
in
addition
to
that,
all
of
the
stuff
that
previously
was
single
user
also
became
you
know
collaborative
multi-user
applications.
A
None
of
this
is
surprising,
but
it's
important
to
note
that
there's
some
fairly
big
differences
in
the
the
nature
of
the
applications
and
the
kinds
of
development
that
you
need
to
do
you
know,
and
even
today
my
aquarium
at
home
has
you
know
Raspberry
Pi
installed
in
it,
and
that
thing
talks
to
the
cloud
and
controls
the
temperature
of
my
fish
and
a
variety
of
other
things.
So
it
really
has
become
pretty
ubiquitous
now
in
practice.
Yeah
sorry
I
forgot
about
my
animation,
so
these
things
get,
you
know,
suddenly
get
very
complicated.
A
You've
got
lots
of
machines,
lots
of
people,
sometimes
not
very
reliable
network
connectivity
and
you've
got
to
try
and
write
applications
that
run
in
this
environment
and
unsurprisingly,
people
are
tearing
the
hair
out.
So
one
of
the
reasons
why
there's
the
cloud
native
computing
foundation
exists
is
because
this
stuff
is
hard.
Kubernetes
was
obviously
one
of
the
first
big
tools
that
the
people
used
to
try
and
you
know
wrestle
with
us
complexity.
A
Now
you
can
partition
your
applications
up
into
into
containers
and
use
things
like
kubernetes,
but
it's
still
relatively
difficult,
so
Along
Came,
essentially
the
CNC
F
containers,
kubernetes,
microservices,
etc,
and
this
was
definitely
a
big
step
in
the
correct
direction.
So
we
could,
as
I
said,
decompose
our
applications
into
independently
deployable
containers
and
a
lot
of
this
work
sort
of
started
in
the
early
to
mid-2000s,
but
particularly
at
companies
like
Google
and
Amazon
and,
as
you
know,
become
a
reasonably
mainstream
these
days.
A
Most
of
you
know:
well,
you
know
all
the
things
on
the
slide
and
things
have
got
better.
So
the
reason
you're
here
is
because
you
know
that
all
this
is
good.
Programmatically,
orchestrated,
declaratively,
driven
configurations,
different
languages
can
be
used
in
every
single
container.
You
can
hook
these
together
with
things
like
link,
ID
and
Envoy,
and
this
do
and
you
can
you
know
your
SRS
can
run
these
things
and
your
developers
can
develop
them.
A
Building
these
resilient
scalable
systems
problem
is
that
your
mere
mortal
application
developer
doesn't
necessarily
or
your
company
that
needs
to
build
some
end-user
application.
It
doesn't
necessarily
have
you
know
tens
of
thousands
of
the
best
distributed
system
software
engineers
in
the
world.
So
the
kinds
of
problems
that
people
run
into
very
frequently
are
things
like
distributed:
concurrency
synchronization?
How
do
you,
you
know,
make
unreliable
networks?
Look
reliable
from
a
remote
procedure,
call
point
of
view.
How
do
you
handle
faults,
disconnections
reconnections,
retries,
leader
elections,
sharding
for
horizontal
scaling?
A
You
know
everybody
assumes
that
every
single
single
machine
operating
system,
whether
it's
Linux,
Windows,
Android,
etc,
is
going
to
provide
virtual
memory.
File
systems
locks
these
kinds
of
things,
but
we
don't
actually
have
the
equivalent
things
for
distributed
systems
are
the
ones
on
the
list
on
the
slide.
We
don't
have
a
standard
implementation
on
every
platform
and
we
think
there
should
be
one.
A
So
not
surprisingly,
these
are
distributed
system
problems.
These
are
the
things
that
you
know.
Android
tanenbaum
and
these
kinds
of
people
have
been
teaching
computer
science
and
nerds
for
quite
a
few
decades.
Now,
in
the
same
way
that
you
know
when
you
study
computer
science,
you
learn
about
how
virtual
memory
managers
work
and
you
understand
how
TCP
sliding
windows
work,
but
we
don't
expect
application
programmers
to
actually
like
write,
TCP
sliding
windows
in
their
applications
or
decide
which
memory
needs
to
go
on
disk
and
which
needs
to
go
in
RAM.
A
That
should
be,
you
know,
abstracted
away
which
at
the
moment
it
is
not
really
in
distributed
systems.
So
the
question
is:
what
do
we
do
about
this?
So
we
clearly
need
you
know
people
to
solve
these
problems,
who
are
not
necessarily
application
developers
and
they
are
pretty
specialized.
So
I
just
want
to
do
a
quick
diversion
here
and
talk
about
this
terminology
that
I'm
using
an
application
developer
is
not
necessarily
something
that
everybody
understands,
certainly
not
a
standard
term
in
the
industry.
A
So
I
just
wanted
to
make
it
clear
in
my
mind
what
what
I'm
talking
about
here
so
so
I
think
there's
a
large
amount
of
very
necessary
specialization.
That's
developed
around
this
area.
So,
on
the
left
hand
side
you
have
application
developers
these.
These
are
the
kinds
of
people
who
know
a
particular
application
domain
very
well.
You
know
whether
it's
travel
finance
banking,
social
networking,
whatever
they
know
what
their
systems
need
to
do.
They
know
what
their
customers
want.
They
know
what
functionality
they
need.
A
They
don't
really
give
a
hoot
about
Paxos
or
distributed
systems,
algorithms
or
exponential,
back-off
or
TCP
sliding
windows
for
that
matter,
then,
in
the
middle
there
you
have
system
developer
or
what
I've
called
system
developers
and,
in
particular,
distributed
systems
developers.
So
these
are
the
kind
of
people
who
are
really
interested
in
understanding
and
solving
hard
distributed
systems
problems,
but
they
are
in
extremely
short
supply.
They
also
cost
a
very
large
amount
of
money,
typically
and
more
problematically.
A
They
don't
actually
understand
your
business
needs
really
and
typically
they're,
not
that
interested
in
them
they're
more
interested
in
distributed
systems
problems,
and
then,
of
course,
we
have
the
SR
ease
or
DevOps
people,
and
these
are
also
very
specialized,
also
very
expensive.
Typically-
and
these
are
the
people
that
understand
how
your
actual
application
behaves
in
real
life
when
your
customers
hit
your
application,
what
happens?
You
know?
What
are
the
sharding
systems?
Look
like?
What
are
the
overload
situations?
A
Look
like
how
much
capacity
do
you
need
all
these
kinds
of
things
once
again,
they're
not
necessarily
distributed
systems,
experts
and
they're,
certainly
not
necessarily
application
domain
experts,
but
they
do
understand
how
to
run
systems
well
and
so
I
think
it's
very
useful
to
understand
these
distinctions.
Of
course,
in
some
cases
you
can
get
individual
people
who
have
all
of
these
skills
and
perhaps
fulfill
all
of
these
functions.
But
my
experience
has
been
that
you
know:
sres
are
exceptionally
good
at
running.
A
Very
reliable
systems
and
application
developers
are
should
be
very
good
at
understanding
their
application
domain
and
we
shouldn't
try
and
force
people
to
kind
of
be
jack-of-all-trades
and
not
very
good
at
any
of
these
things.
So
what
do
we
do
about?
This
is
the
question.
So
we
don't
have
time
to
go
into
all
the
details,
but
we
have
this
umbrella
project,
which
we've
called
a
Meno
RS.
A
Basically,
we
think
that
that,
ultimately,
what
we
need
is
a
distributed
operating
system
and
we
think
that
that
will
probably
be
comprised
of
at
least
a
few
subsystems,
one
of
which
needs
to
manage
this
tributed
microsystem
microservices.
And
that's
what
we'll
be
focusing
on
today.
We
think
that
you
know
synchronized
distributed
memory.
Shared
memory
of
some
sort
is
is
also
super
useful
and
we
have
some
prototypes
of
systems
that
we
think
will
be
useful
there.
A
A
You
know
that
we
don't
have
to
go
and
you
know
decide
which
pieces
need
to
go
into
which
machines
the
operating
system
should
should
take
care
of
some
of
that
stuff.
We
need
a
unified
system
that
provides
these
micro
service,
runtime,
environment
memory,
management,
storage,
etc,
and
it
needs
to
run
across
all
operating
systems
and
be
programmable
from
all
programming
languages.
Otherwise
we
don't
really
live
up
to
the
promise
of
of
this
new
world.
A
This
is
a
lot
of
research
has
been
going
on
for
several
years
at
the
University
of
Washington,
and
Irene
will
be
stepping
up
in
a
few
moments
to
dive
us
into
some
of
the
the
work
that
she
did
with
her
team
there,
and
then,
after
that,
work
was
done
and
around
about
the
time
when
Irene
was
publishing
a
PhD
I
came
across
it
and
we
decided
that
this
was.
You
know
worth
turning
into
a
big
open
source
project
that
people
could
actually
use
on
an
ongoing
basis.
The
research
wasn't
intended
for
that
purpose.
A
It
was
really
exploratory
to
figure
out.
You
know
what
the
space
looked
like
and
what
we
needed
so,
in
particular
their
academic
papers
and
things
floating
around,
which
cover
a
lot
of
this
material.
Some
of
the
terminology
just
to
sort
of
warn
you
is
different.
So
over
time
we
discovered
that
certain
terms
we
use
we're
confusing.
We,
for
example,
renamed
this
sapphire
system
to
amino
run
and
a
variety
of
other
things.
A
So
if
you
see
contradictory
terms,
floating
around
you'll,
hopefully
understand
the
origin
listings
right,
so
we're
gonna,
look
at
what
the
goals
were,
the
architecture
how
it
works,
particularly
these
deployment
managers,
which
are
a
pretty
key
part
of
of
the
Amina
run
system,
some
of
Irene's
experience
in
evaluation.
They
did
some
tests
and
came
up
with
some
pretty
interesting
observations
and
data.
A
A
So
we
don't
think
that
it's
a
good
idea
for
applications
actually
know
anything
about
sharding
or
load,
balancing
or
error
recovery,
or
any
of
these
things,
because
they're
not
really
related
to
application
logic.
We
would
prefer
to
separate
that
out
into
different
part
of
the
system
which
can
be
dealt
with
by
expert
programs
in
those
domains
and
run
by
people
who
know
what
they're
doing
with
them.
A
In
addition,
of
course,
we
wanted
to
be
super
simple
to
understand
and
use
must
be
high-performance,
and
it
must
support
all
the
different
programming
languages
that
everybody
knows
and
loves.
I
I've
been
around
long
enough
that
I've
come
to
the
conclusion
they're
trying
to
convince
someone
to
use
a
programming
language
they
don't
like
is
not
very
successful
in
practice.
Very
often
so
I
decided
not
to
try
and
do
that,
and
we
we
definitely
don't
want
to
reinvent
any
wheels
or
compete
with
any.
You
know
existing
systems
that
solve
some
of
these
problems.
A
A
A
We'll
give
you
an
overview
of
the
architecture
and
then
describe
all
of
these
layers
in
a
little
bit
more
detail
at
the
top.
We've
got
the
application
itself,
which
runs
in
a
uniform
environment
that
spans
across
mobile
devices,
cloud
servers
and
edge
devices
and
at
the
bottom
there's
a
deployment
kernel
that
runs
everywhere.
A
The
application
does
and
provides
a
uniform
environment
for
applications
with
basic
deployment
services,
and
you
know
it's
very
convenient
and
easy
to
run
and
manage
the
deployment
kernel
layer
and
therefore
everything
on
top
of
it
on
top
of
kubernetes,
and
we
do
that.
That's
our
sort
of
default
deployment
mechanism,
but
it's
not
required
or
fundamental
to
to
Amino
itself
and
just
to
give
you
an
idea
of
what
a
you
know
in
practice.
Deployment
like
might
look
like
and
I
apologize
for
the
not
very
good
color
scheme
here.
A
A
In
the
middle
there,
and
all
of
these
run
on
top
of
essentially
a
distributed
kernel
which
provides
all
the
heavy
lifting
you
know
the
thing
that
moves
processes
around
and
and
intercepts
various
different
system
calls
and
using
this
kind
of
approach
it
becomes
very
straightforward
to,
for
example,
automatically
migrate
micro
services
to
optimize
for
any
given
property,
latency
or
power
consumption.
Anything
like
that.
A
And
we
want
to
do
all
of
this,
with
support
for
multiple
languages,
I'm
gonna
skip
over
most
of
the
slide.
What
I
would
like
to
mention
briefly
before
handing
over
to
Irene
is
that
we
did
quite
a
bit
of
work
to
support
multiple
languages
and
we
we
used
a
thing
called
DRA
vm,
which
some
of
you
may
have
come
across
I'm
not
going
to
go
into
detail.
A
We
definitely
don't
have
time,
but
but
I
wanted
to
just
mention
that
it's
a
high
performance
polyglot,
meaning
multi-language
virtual
machine
in
the
sense
of
a
JVM,
not
in
the
sense
of
a
of
an
OpenStack
virtual
machine,
and
it
provides
support
for
both
ahead
of
time,
compilation
and
therefore
native
execution
or
just-in-time
compilation.
It's
embeddable
in
other
environments,
so
you
can
actually
embed
it
inside
the
database
to
run
these
things.
A
You
can
embed
it
inside
the
open,
JDK
or
you
can
run
it
standalone
and
the
particular
thing
that
it
buys
us
is
the
ability
to
one.
You
know,
produce
different
parts
of
the
application
in
different
languages
and
yet
be
able
to
easily
transfer
data
structures
and
things
like
that
between
the
different
parts
of
the
language,
but
also
the
the
kernel.
The
deployment
managers
in
the
application
can
also
be
run
in
different
languages
without
the
language
boundaries
getting
in
the
way,
and
just
to
be
clear.
A
Some
of
these
diagrams
we're
going
to
show
you
are
going
to
show
you
know
all
these
layers
and
pieces
of
software.
This
stuff
all
gets
compiled
away
into
a
single
binary,
a
single
process.
So
you
don't
need
to
worry
too
much
about
you
know
inter
process,
communication
and
network
connections
and
all
that
stuff
other
than
obviously
between
machines.
But
as
soon
wherever
you
have
multiple
of
these
layers
on
the
same
machine,
they
all
get
compiled
away
into
nothing
effectively
and
with
no
further
ado
I'm
gonna
hand
over
Tyree.
B
Hi
so
Quinton
told
you
a
bit
about
what
the
amino
runtimes
goals
are
and
what
it's
capable
of
what
I'm
gonna
talk
about
is
sort
of
the
nitty-gritty
technical
details
of
how
it
actually
does
all
of
this
magic.
So
the
key
idea
behind
the
amino
runtime
is
that
we
divide
the
run
time
into
a
deployment
management
layer
and
a
deployment
kernel,
and
the
deployment
kernel
is
the
very
lowest
layer.
Oh,
these
are
our
micro
services
is
the
very
lowest
layer
of
the
amino
runtime
and
what
the
deployment
kernel
does
is.
B
It
does
essentially
those
best-effort
mechanisms
that
you
need
for
running
any
microt
service,
so
it
provides
instantiation
of
your
amino
microservices.
It
automatically
replicates
them
tracks
them,
it
can
move
them
and
it
can
also
find
your
amino
micro-services.
So
you
can
essentially
make
a
call
transparently
from
any
amino
microservice
to
any
other
amino
micro-service
without
having
to
you
know,
do
a
lot
of
complicated
lookup
and
RPC
management.
B
B
So
deployment
manager
is
actually
work
by
interposing
on
RPC
to
and
between
micro
services
and
what
they
can
do
is
they
can
do
things
like
create
multiple
copies
of
a
microservice
and
then
every
time
that
microservice
receives
a
call.
It
will
route
the
call
to
all
of
those
copies
rather
than
just
one.
It
can
also
do
things
like
shard,
a
micro
service
and
then
route
our
pcs
to
that
micro
service
to
the
correct
shard
and
then
finally,
micro
service
calls
and
events.
B
This
interposition,
that
these
deployment
managers
are
doing
they're
completely
transparent
to
the
application
right,
they're
completely
transparent
to
the
micro
services
themselves,
because
they
happen
before
the
RPC
reaches
the
micro
service,
and
so
that
means
that
you
can
actually,
as
an
application
program
or
easily
choose
and
then
change
your
mind
about
what
kind
of
deployment
manager
you
might
want.
For
example,
you
might
start
out
having
just
a
single
copy
of
your
micro
service
running
but
then
decide
you
need
to
later
shard
it,
and
this
can
be
done
without
changing
any
of
your
application
code.
B
So
the
way
that
the
deployment
manager
management
layer
works
is
that,
for
each
micro
service
the
programmer
chooses
a
single
deployment
manager,
so
you
might
choose,
for
some
of
your
micro
services
say
that
you
need
to
replicate
for
fault
tolerance.
You
would
choose
a
replication
deployment
manager
for
some
of
your
other
micro
services.
You
might
choose
things
like
code
offloading
if
you
want
to
move
your
code
somewhere,
where
there's
a
more
powerful
machine,
or
maybe
one
that
has
a
GPU
or
some
other
service
that
it's
able
to
offer.
B
We
offer
a
library
of
deployment
managers
that
cover
as
lots
of
categories
of
what
we
expect
to
be
standard
problems
for
application
programmers
to
solve
today,
so
things
like
caching,
checkpointing,
replication,
sharding
and
scaling.
We
have
a
selection
of
libraries
of
deployment
manager
for
each
category.
So
not
everyone
has
to
do
cashing
in
the
same
way.
You
can
you
do
caching
using
a
lease
you
can
do.
Caching
using
some
sort
of
timeout.
You
can
do
caching
using
an
explicit
lock.
So
this
is
a
list
of
some
of
the
deployment
managers
that
we
have
built.
B
We
continue
to
expand
on
them
and
the
deployment
manager
library
is
very
much
extensible,
and
this
is
entirely
due
to
the
way
that
the
amino
runtime
is
architected.
So
I'm
going
to
talk
next
about
how
each
deployment
manager
actually
works.
To
give
you
an
idea
about
how
SR
ease
or
these
distributed
systems,
people
can
actually
build
additional
deployment
managers
based
on
their
needs
or
their
research,
so
the
deployment
manager
API
takes
each
deployment
manager
and
splits
it
into
three
components.
B
So
each
deployment
manager
must
consist
of
some
server-side
code,
which
these
amino
runtime
will
co-locate,
with
the
micro
service
itself,
some
client-side
code,
which
the
amino
runtime
collocates
with
the
callers
of
the
micro
service
and
then
a
centralized
group
coordinator
component,
which
is
located
on
a
fault-tolerant
service.
That's
provided
by
the
immuno
runtime.
So
to
make
all
of
this
a
little
bit
more
concrete,
I'll
explain
exactly
what
happens
so
you
write
code
for
each
of
these
deployment
managers
and
the
deployment
managers
invoke
the
code
automatically
on
each
RPC.
B
So
you
start
out
with
your
micro
Service
write.
It
runs
on
some
server
somewhere
in
your
kubernetes
container.
Then
you
decide
that
you
want
to
replicate
this
micro
service.
So
how
do
we
go
about
doing
that?
Well,
we
have
a
single
copy
of
our
micro
service,
which
we
place
in
a
kubernetes
container,
and
then
we
create
a
server-side
deployment
manager
that
runs
next
to
it.
On
the
same
deployment
on
the
same
kernel
server,
then
we
can
create
another
copy
which
gets
its
own
copy
of
the
server
the
server
deployment
manager.
B
So
these
are
just
code
that
get
invoked
by
Amino
runtime
on
the
deployment
kernels
when
the
micro
Service
receives
an
RPC,
so
these
managers
they
can
talk
to
each
other
within
a
single
address
space.
So
essentially
the
Amino
runtime
is
providing
transparent
RPC
between
the
deployment
managers
between
the
components
of
a
single
deployment
manager,
then
each
to
reach
each
Microsystems
service.
Each
application
goes
through
a
stub
in
order
to
call
the
micro
service.
B
This
is
where
the
magic
happens
of
the
amino
runtime
deployment
kernel
where
it
looks
up
where
the
micro
service
lives
and
makes
the
RPC
to
the
micro
service,
depending
on
where
it's
been
deployed,
and
so
with
this
stub,
we
we
collate
locate
a
little
bit
of
code
that
we
call
the
piece
of
the
deployment
manager.
That's
the
client
component,
so
each
stub
gets
its
own
bit
of
client
code
client-side
code
and
that
code
gets
invoked
every
time.
B
The
another
micro
service
makes
a
call
to
this
micro
service
and
then,
when
the
call
reaches
the
server
or
the
kubernetes
container,
where
the
micro
service
is
running
first,
we
invoke
the
server-side
deployment
manager
before
we
up
call
into
the
application
itself
or
the
micro
service
itself,
and
then
finally,
all
of
these
can
communicate
with
the
centralized
group
deployment
manager,
which
is
we
consider
a
centralized
and
fault
tolerant
component.
So
this
can
do
things
like
choose
leaders
for
your
replica
group.
It
can
decide.
B
B
So
I'm
gonna
be
a
little
bit
more
detailed
about
what
happens
when
you
actually
make
an
RPC
to
a
micro
service,
that's
running
on
top
of
a
replication
deployment
manager.
So
in
the
beginning
the
deployment
manager
is
going
to
instantiate
three
copies
each
in
separate
kubernetes
containers
of
the
micro
service.
B
B
We
can
do
the
same
thing
for
caching,
so
this
is
perhaps
where
the
latency
is
too
high
to
your
micro
service,
and
you
want
to
move
the
micro
service
closer
to
you.
So
essentially,
what
happens
in
the
caching
deployment
manager
is
sort
of
an
inverse
of
the
offloading
one?
We
have
some
master
that's
running
on
a
cloud
server,
but
that
we
decide
the
latency
is
too
high.
So
then,
instead,
what
we
do
is
from
the
colleague
the
purse,
the
other
micro
service
or
mobile
device.
That's
making
the
call
to
this
micro-service.
B
We
actually
create
a
copy
of
the
micro
service
and
cache
it
inside
the
client-side
deployment
manager.
So
in
that
case
we
can
now
have
much
lower
latency
access
to
our
local
copy
of
the
micro
service.
But
of
course
we
now
we
have
to
keep
that
micro,
so
risk
consistent
or
coordinated
somehow
with
the
master
micro
service.
So
hopefully
this
gives
you
some
idea
of
how
these
deployment
managers
can
transparently
under
the
covers,
do
a
lot
of
the
coordination.
The
applications
today
would
normally
have
to
do
in
the
application
itself.
B
B
So
this
is
actually
something
that
you'll
see
in
the
demo
in
a
minute,
so
we
can
also
compose
deployment
managers.
This
is
essentially
done
through
chaining,
so,
every
time
an
RPC
is
made,
we
go
through
a
chain.
We
could
potentially
go
through
a
chain
of
deployment
managers
in
order
to
achieve
all
of
the
properties
that
we
want
for
a
particular
micro
Service.
B
As
you
can
imagine,
this
does
eventually
get
pretty
complicated,
which
is
actually
why
it's
important
that
application
programmers
don't
have
to
do
this
and
it's
done
under
the
covers,
with
configuration
by
sres
or
distributed
systems.
Programmers
all
right
so
to
wrap
up.
I'm
going
to
talk
a
little
bit
about
some
experiments
that
we
did
with
our
initial
prototype
of
the
amino
runtime.
We
ran
our
experiments
on
a
series
of
Nexus
devices
and
cloud
servers
connected
through
Wi-Fi
and
a
4G
connection.
B
Basically,
so
the
first
thing
we're
gonna
look
at
is
what
happens
in
a
peer-to-peer
multiplayer
game
when
we
use
different
amino
deployment
managers
to
place
the
micro-services
in
different
ways.
So
initially,
oh,
so
the
x-axis
here
gives
the
different
deployments
and
then
the
y-axis
gives
you
the
latency
of
making
each
move
in
the
game.
These
results
are
probably
about
five
years
old,
so
take
them
with
a
grain
of
salt.
B
So
the
first,
the
first
thing
that
we
tried
was
actually
just
placing
the
micro-service
in
the
cloud
and
having
the
two
clients
share
and
access
the
micro-service
together.
So
that
gives
you
roughly
even
latency
for
both
players
for
both
read
and
write.
Now
we
can
actually
just
dynamically
move
the
microservice
if
we
would
like
to
one
of
the
players
devices.
So
in
this
case
we've
moved
it
to
player
Juan's
device.
So
this
has
been
done
without
any
application
code.
Changes
whatsoever.
B
B
We
can
see
similar
results
for
taking
a
physics
engine,
basically
for
a
mobile
game
and
moving
it
between
a
running
on
the
mobile
device
and
running
on
the
cloud.
So
when
we
have
a
phone,
we
are
able
to
run
the
physics
engine
on
the
phone,
but
at
quite
a
high
expense.
If
we
have
a
tablet,
we
have
slightly
better
performance
actually,
and
so
our
extension
is
able
to
run
much
faster.
A
So
yeah
we've
run
terribly
over
time,
I'm.
Sorry
about
that,
but
we're
nearly
done
and
given
that
there
no
more
talks,
I
guess,
that's
not
too
serious,
so
yeah
I
think
we've
we've
looked
at
the
summary,
as
there
are
numerous
things
we
would
still
like
to
do.
This
stuff
is,
is
you
know
usable
at
the
moment,
but
it
I?
Wouldn't
you
know,
deploy
any
banking
applications
on
it
just
yet,
but
it's
getting
there.
We
also
have
there's
some
limitations
to
the
state
migration.
A
So,
for
example,
if
your
application
has
state
that
sits
outside
of
your
micro
Service,
you
know
an
external
file
or
something
sitting
in
the
operating
system
that
doesn't
magically
migrate
with
your
micro
service.
So
we
can
do
some
pretty
straightforward
things
to
make
that
work
better.
There's
some
rough
edges
between
certain
combinations
of
languages.
We,
you
know,
we
support
a
large
number
of
languages
and,
depending
on
the
permutations,
there's
some
slightly
funky
things
that
happen
which
we'd
like
to
clean
up.
A
We
would
like
to
build
some
more
plugins
for
external
systems,
like
things
like
you
know,
a
CD
and
tea
and
sto,
and
these
kinds
of
things
which
clearly
can
be
well
used
inside
a
deployment
manager
and
then
we're
also
very
interested
in
in
Federation's
and
disconnected
edge
computing
scenarios.
So
we've
done
some
work
on
that
area
and
we've
run
some
things
in
those
kind
of
contexts,
but
we're
not
totally
sure
that
everything
works.
You
know
the
original
design
was
not
was
not
explicitly
created
for
that
all
right
feel
free
to
get
involved.
A
This
is
all
open
source.
We
just
open
sourced
it
literally
like
two
days
ago,
one
day
ago,
there's
a
slack
channel
as
a
website,
github
repo
you
can
go
and
have
a
look
there,
and
certainly
getting
in
touch
with
us.
If
you're
at
all
interested
in
this
stuff
and
I
would
like
to
then
hand
over
to
Vanunu's
gonna,
give
us
a
little
bit
of
a
demo,
as
I
said,
to
hopefully
show
everyone
that
this
is
not
just
smoke
and
mirrors.
A
D
I'll
be
giving
the
demo
on
automatic
object
migration.
Now
it's
for
face
recognition,
microservice
left
down.
One
is
not
an
object
monitoring
system,
it's
a
part
of
BK
and
the
right
above
it
run
KS
one.
What
you
are
seeing
is
a
kernel
server,
so
I'll
be
and
the
right
side
top
one
is
a
cloud
kernel
server
and
the
left
right.
Bottom
one
is
an
application.
That
is
the
place
where
we
run
the
client.
D
A
A
Yeah,
that's
a
good
question,
so
so
the
way
it
works
is
the
running
process.
You
can
imagine
it's
actually
a
class,
an
instance
of
a
class
that
is
being
migrated
and
that
has
state
in
it.
So
what
we
do
is
we
freeze
the
state.
We
move
the
state
to
the
other
side
and
then
we
load
that
into
a
virtual
machine
with
that
state
and
then
we
carry
on
running
and
because
we
have
it
built
into
the
deployment
manager
layer.
A
C
C
A
That
that's
a
that's,
actually
a
very
good
question
and
one
that
comes
up
quite
regularly.
You
know:
we've
seen
these
kinds
of
things
for
a
very
long
time,
DCE
and
there
are
lots
of
things
you
know
in
the
90s
that
did
this
I
think
they
were.
You
know,
combination
of
factors
that
just
made
them
not
become
ubiquitous.
You
know
we
didn't
have
public
clouds.
In
those
days
we
didn't
have
mobile
devices,
we
had
warring
UNIX
vendors.
Trying
to
you
know
outdo
each
other.
A
A
The
other
I
think.
The
other
thing
that
that
is
pretty
important
is
there
was
this
kind
of
notion
that
that
you
could
just
take
any
object,
oriented
program
and
just
like
scatter
the
objects
all
over
the
over
the
network,
and-
and
you
know
it
should
be
transparent
and
we
discovered
that
that
doesn't
work
like
it
explicitly
does
not
work.
You
need
to
tell
the
programmer
that
this
is
a
micro
service.
This
is
remote.
This
is
gonna
fail,
it
doesn't
need
to
be
replicated
and
it's
fundamentally
different
than
a
you
know:
object.