►
From YouTube: Apache OpenWhisk: Building a Production-Ready Serverless Stack on/for Kubernetes - David Grove
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Apache OpenWhisk on Kubernetes: Building a Production-Ready Serverless Stack on and for Kubernetes - David Grove, IBM Research (Any Skill Level)
Apache OpenWhisk is a serverless, open source cloud platform that executes functions in response to events at any scale. OpenWhisk can now be deployed on Kubernetes, providing a compelling unified platform for a wide range of cloud-native applications. In this talk, David Grove will describe the challenges of deploying a production-ready, multi-tenant, low-latency and high-volume Functions-as-a-Service offering on Kubernetes. He will outline how OpenWhisk's architecture maps to Kubernetes, and motivate OpenWhisk's bespoke container management strategy for low-latency function invocation via an empirical evaluation of the alternatives. David will also demonstrate how OpenWhisk deployed on Kubernetes provides a compelling runtime platform for executing hybrid applications that seamlessly mix OpenWhisk functions, Kubernetes-managed services, and external services.
About David
David Grove is a technical leader of the Apache OpenWhisk sub-project to deploy OpenWhisk on Kubernetes. He is a Distinguished Research Staff Member at IBM Research and a committer for the Apache OpenWhisk incubator project. He was named an ACM Fellow in 2012 for his contributions to programming languages and for the development of open-source research infrastructures.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Okay,
all
right
we're
good
to
go
all
right,
so
everyone
thank
you
for
sticking
out
to
the
second
to
last
talk
to
coop
con
and
being
here
to
hear
about
our
experiences
with
epic.
She
opened
whisk
and
putting
it
on
top
of
kubernetes
I'm
Dave
Grove
I'm,
an
Apache
open,
whisk,
emitter
and
I've
been
one
of
the
leaders
of
our
community
effort
to
take
open,
whisk
and
flow
it
on
top
of
kubernetes.
A
A
Doing
service
composition
of
service
functions,
we're
gonna,
use
that
as
a
demo
to
show
some
of
the
tools
have
been
doing
and
some
of
the
cool
things
we're
doing
and
some
of
the
future
directions
for
the
project.
Okay.
So
what
is
Apache
open,
whisk,
so
Apache,
open
whisk?
Is
a
production,
ready,
open
source
function
as
a
service
offering?
So
it's
got
a
number
of
pieces.
There's
a
core
fast
runtime
command-line
tools,
their
language
runtimes
for
each
of
the
supported
tool.
It
was
pre
supported
languages.
A
We
have
and
they're
bunch
of
packages
that
provide
bindings
and
convenient
ways
to
use
common
cloud
services.
Apache
open
whisk
is
pretty
aggressively
polyglot.
There's
about
a
half
a
dozen
languages
that
are
sort
of
built-in.
You
can
write
your
functions
any
one
of
these
languages,
but
if
those
languages
don't
suit
you,
you
can
bring
your
own
black
box
container.
So
basically,
you
can
run
almost
anything.
You
want
as
a
function
as
long
as
it
responds.
So
pretty
simple
each.
A
If
you
arrest
REST
API,
the
system
is
designed
to
be
extensible
and
pluggable
between
most
of
our
major
components.
We
have
the
service
provider
and
interfaces
or
SP
is
they
give
us
a
way
to
customize
the
deployment
of
this
thing
and
make
different
choices
for
different
people
who
are
deploying
it?
It's
pretty
easy
to
bring
additional
language
runtimes.
Some
of
the
four
writers
have
different
runtimes,
the
Floyd
and
their
things
add
additional
packages,
SDKs
and
so
on.
There's
also
diverse
ways
so
to
employ
open
whisk.
A
You
can
deploy
it
on
a
bunch
of
container
compass
container
orchestration
platforms,
docker
compose
kubernetes
Mesa
is
open
shifts.
You
can
also
use
ansible
adjust,
deploy
it
on
a
host
of
VMs
or
bare-metal
machines.
There
are,
there
are
hosted
commercial
deployment
to
open
whisk
level
with
IBM
and
Adobe
and
their
additional
companies
that
are
in
various
stages
of
getting
this
getting
additional
host
of
you
from
Ursula
flam
its
out
but
open
whisk
is
also
an
open-source
project.
A
So
it's
an
Apache
Incubator
project,
there's
a
pretty
active
open
source
developer
and
user
community
and
as
a
whole.
The
project
is
working
towards
official
releases
and
graduation
from
incubation
which
we're
hoping
to
have
happen
relatively
soon,
as
we
sort
of
move
up
the
curve
to
maturity.
So
it's
it's
in
production
already
for
multiple
companies,
but
we're
still
an
incubator
project
in
the
open-source
sense,
so
open
a
whisk
on
kubernetes.
What
is
this
project,
particularly?
What
is
this
project
entail?
And
really
there
are
three
major
stages
of
work.
A
The
first
one
was
fairly
straightforward,
so
open
whisk,
the
the
control
plane,
open
whisk,
is
already
was
already
containerized,
and
so
getting
that
control
plane
to
run
on
top
of
kubernetes
was
really
just
taking
these
containers
and
deploying
them
as
using
kubernetes
concept.
Concepts.
Stateful
sets
daemon
sets
and
so
on
and
the
process
of
doing
that.
We
found
places
where
the
existing
code
didn't
really
wasn't,
really
a
great
fit
for
kubernetes,
and
so
we
did
a
bunch.
But
a
bunch
of
adjustment
to
the
open
was
code
base
2.
A
It's
the
the
actual
implementation,
the
configuration
and
deployment
scripts
to
make
it
a
better
fit
for
kubernetes
to
get
it,
make
it
easier
to
deploy
and
do
that
and
the
third
stage,
which
is
we're
really
sort
of
just
in
the
initial
early
stages
of
it,
is
now
they're
on
top
of
kubernetes,
trying
to
think
about
how
we
can
exploit
kubernetes
capabilities
to
simplify
the
open
whisky
implementation.
Give
us,
let's
is
less
to
maintain
more
naturally
fit
into
the
kubernetes
environment
and
so
on.
A
So
all
this
work
is
being
done
in
a
sub
project
of
the
Apache
open
wisp
project,
the
github
repositories,
incubator,
open
wisp,
deploy
cube.
If
you
go
there
where
you'll
find
is
something
that
supports
multiple
versions
of
mini
cube
and
kubernetes.
There's
configuration
files
read
Me's
and
deployment
scripts
for
deploying
the
system
onto
mini
cube
and
single
and
multi
node
clusters
and
as
well.
We
can
half
ago
we
actually
of
a
helmet
chart
they'll,
let
you
deploy
it
as
well,
we're
going
to
do
a
demo
of
this
later,
all
right,
but
first.
A
So
what
are
the
underlying
assumptions
for
open
wisk?
What
are
we
trying
to
do?
And
some
of
these
assumptions
come
around
what
we
mean
by
production-ready.
So
when
we're
building
open
Wisc,
we
really
want
to
build
a
software
stack,
that's
capable
of
operating
at
very
large
scale
at
public
cloud
provider
scale.
So
many
many
millions
of
functioning
applications
a
day.
All
of
the
core
components
in
the
control
plane
are
designed
to
be
replicated
for
fault,
tolerance
and
scalability,
and
we're
also
interested
in
building
a
multi-tenant
system
to
really
make
this
thing
economically
viable.
A
Another
thing
about
being
production,
righty
is
when
you
to
take
open,
wisk
and
I'm
a
provider
I
deploy
it
onto
my
cloud.
It
has
to
fit
into
the
rest
of
my
cloud,
so
it's
got
to
naturally
just
plug
in
there
and
that's
why
we
have
so
many
SBI's
and
plug
points.
So
we
can
take
the
core
codebase,
deploy
it
basically
unchanged,
but
yet
feel
like
we're
part
of
a
cloud.
A
The
cloud
platform
on
which
we're
plugging
into
a
second
set
of
assumptions
come
from
some
of
the
workloads
and
use
cases
were
interested
in
supporting
an
open
Wisc.
We
have
a
strong
emphasis
on
extremely
low
latency
and
minimizing
system
overhead
and
some
of
that's
becoming
from
the
applications
we
want
to
support.
We
want.
A
We
want
to
be
able
to
run
the
backends
for
interactive
applications
on
open
wisk,
so
the
backends
for
mobile
apps
for
chatbots,
and
we
also
want
to
run
applications
where
a
single
top-level
logical
action,
something
the
user
wants
to
do-
will
actually
be
a
pipeline
or
composition.
That'll
spawn
many
function,
requests,
and
also
because
of
the
charging
model
of
server
lists.
There's
a
tendency
for
short
running
functions
and
when
you
have
short
running
functions,
is
really
harder.
A
It's
harder
to
amortize
the
system
overheads
right,
so
you
don't
have
as
many
places
to
hide,
and
so
we
have
an
emphasis
on
low
latency
and
minimizing
our
system
overhead,
all
right.
So
now,
let's
dive
into
the
details
a
little
bit.
What
happens
when
you
actually
invoke
a
function
on
open,
open
wisk?
So
the
basic
open
list
model
is
that
every
user
function
pair
is
going
to
be
executed
in
a
distinct
container.
A
Now
we
may
reuse
one
of
those
containers,
so
a
particular
container
may
execute
multiple
invocation,
multiple
invitations
of
its
unique
user
function
pair,
but
it
will
only
ever
execute
that
function
from
that
user.
So,
given
this
model,
container
scheduling,
caching
and
reuse
are
really
essential
for
us
to
get
the
kind
of
performance,
scalability
and
latency
response
we
want
so
I'm
gonna
do
right
now
is
walk
through
the
the
life
cycle
of
a
user
container.
So
a
request
comes
into
the
system
to
execute
a
function
on
behalf
of
some
user.
A
We
don't
have
a
container
for
that,
yet
so
we'll
have
to
create
it.
I
refer
to
this
as
a
cold
start,
because
creating
container
can
take
a
fairly
long
time,
all
right.
So
once
the
container
is
created,
we
can
then
initialize
the
container.
So
an
open
whisk
most
of
our
but
most
of
the
volumes
of
the
invitations
are
actually
using
one
of
our
predefined
runtimes,
so
they're
using
nodejs
or
Java
or
Swift,
or
go
or
Python
or
PHP
they're,
using
a
language
runtime,
and
so
when
we're
creating
a
container.
A
What
we're
doing
is
we're
creating
a
container,
that's
ready
to
run
some
code
in
a
particular
programming
language,
but
it
doesn't
have
the
users
code
in
it,
yet
so
the
initialization
phase.
What
we
do
is
we
inject
the
user's
code
into
that
container,
and
then
we
call
an
initialization
function
to
allow
the
user
to
prepare
to
actually
execute
their
functions.
This
is
where
the
user,
the
user
code,
has
the
opportunity
to
pre
its
example,
create
database
connections.
They
want
to
cache
or
do
other
kinds
of
initialization
actions.
They
don't
want
to
do
on
every
function.
A
Invocation
now
creating
containers
can
be
pretty
expensive
and,
as
I
said,
a
lot
of
our
container.
A
lot
of
our
workload
comes
from
one
of
our
running
one
of
our
language,
one
of
our
language
runtimes,
and
so
we
have
a
notion
of
stem
cell
containers
and
what
the
system
tries
to
do
is
maintain
a
pool
of
these
stem
cell
containers.
A
Nodejs
containers
PHP
containers
go
containers,
whatever
the
providers
decided
that
are
ready
to
run
so
here
the
language
runtime
is
up
ready
to
go
and
all
it
has
to
happen
is
we
have
to
inject
the
user
code
into
it
and
we're
ready
to
run
now
whether
the
container
was
cold
or
pre-warmed
once
it's
initialize,
the
next
thing
to
do
is
actually
run
the
user
code.
So
that's
calling
the
run
route
on
the
container.
This
invokes
the
user
function,
it
computes
it
return
to
result
and
where
you
turn
that
to
the
user.
A
Now,
when
the
function
returns,
we
actually
leave
the
container
running
for
a
small
amount
of
time.
This
is
our
grace
period.
We're
hoping
to
do
here
is
within
this
grace
period
is
relatively
short
amount
of
time.
We're
hoping
another
request
from
the
same
user
for
the
same
function
comes
in,
so
we
can
reuse
this
container
immediately
and
that's
our
hot
path.
So
here
there's
no
delay
at
all.
The
request
comes
in
there's
already
container.
It
goes
after
that
grace
period.
A
Experience
expires,
though
we're
gonna
suspend
the
container,
and
this
is
really
really
related
to
trying
to
run
a
profitable
service.
So
our
billing
model
for
IBM
cloud
functions,
for
example,
is
we
only
build
a
user
for
the
time
their
function
is
actually
running
so
when
the
container
is
sitting
there
running,
but
the
user's
code
is
exited,
we're
not
billing
them
for
that
and
we're
building
them
at
about
100
100
millisecond
granularities.
A
So
if
we
just
left
the
container
running
for
a
long
time,
without
suspending
it,
clever
users
would
have
a
function
that
came
in
fired
off
some
background
work
and
then
a
future
invocation
would
that
and
go
and
harvest
the
results
of
that
background?
Work
and
users
are
clever.
So
if
we
didn't
have
the
suspension
capability,
they
would
definitely
be
doing
this
all
right.
So
now
the
container
suspended
it
sits
there
for
a
while
and
another
request
may
come
in,
at
which
case
we
resume
the
container,
and
this
is
a
warm
path.
A
So
here
we've
got
a
container.
That's
ready
to
execute
the
code.
All
it
has
to
do
is
be
on
paused
all
right,
but
the
container
may
be
suspended
for
a
long
time
and
nothing
happens
when
that
happens,
it's
idle
and
it
times
out,
and
so
we
go
ahead
and
destroy
the
container
to
make
room
for
other
can
other
other
containers
in
the
system.
So
that's
the
overview
of
the
user
container
lifecycle
as
we
go
from
creation
through
actual
use
to
destruction.
A
Okay,
so
wanted
to
do
now
is
take
you
through
the
critical
path
of
a
function.
Invocation
as
it
comes
to
the
system,
goes
through
our
control,
plane,
executes
and
comes
back
when
these
pictures,
my
convention,
is
that
blue.
The
blue
blocks
are
part
of
the
open
list,
control
plane.
Green
boxes
are
the
user
containers?
Okay,
so
the
function
comes
in
a
request
to
execute
function
f,
with
some
parameters
from
some
actor
outside
in
the
world.
A
The
first
thing
it
has
to
do
in
kubernetes
term
is
go
through
our
ingress,
so
the
ingress
does
the
SSL
termination
and
then
forwards
the
request
to
the
appropriate
controller.
The
controller's
job
as
it
has
role,
has
several
jobs.
One
of
them
is
if,
if
this
is
a
blocking
request,
the
controllers-
that
is
the
part
of
the
system,
that's
responsible
for
maintaining
the
open
to
HTTP
connection
interval.
A
It
also
deals
with
all
the
authentication,
authorization
and
emission
control
operations
that
need
to
be
done
to
make
sure
that
we
actually
want
to
run
this
function
for
this
user.
And
finally,
it
runs
the
load
balancer
and
the
job
of
the
load
balancer
is
to
try
to
select
the
invoker
component.
The
evoker
instance
that's
actually
going
to
handle
this
invocation.
A
So
when
you
go
in
that,
go
a
little
bit
more
detail,
so
each
each
control
is
responsible
for
signing
work
to
a
pool
of
invoker's
and
these
invoker's
in
turn
each
is
managing
a
set
of
user
containers
that
are
in
one
of
the
lifecycle
states.
We
talked
about
before
so
I've
got
my
controller.
It
has
a
pool
of
invoker's.
There
are
some
user
containers
there
in
various
states.
Some
of
them
are
anything
some
of
their
paws.
Some
of
them
are
white,
hot
ready
for
work,
and
the
goal
of
the
load.
A
Balancing
algorithm
is
to
pick
an
invoker
to
run
this
to
minimize
invocation,
late,
latency,
maximize
system
throughput
and
hopefully
find
one
with
a
warm
or
hot
container
for
this
particular
function
and
user.
Now,
because
of
we
need
to
be
fault,
tolerant
and
we
need
to
be
scalable,
the
controllers
are
actually
replicated.
So
we
have
multiple
controller
instances
that
are
all
that
are
sending
work
to
this
pool
of
invoker's.
So
we
have
some
replicated
state
in
the
controller
and
the
system
is
fairly
large.
A
So
one
of
the
challenges
for
our
load,
Baloo,
balancing
algorithm,
is
dealing
with
a
scale,
the
replication
and
rapid
state
change,
and,
as
a
result,
our
experience
has
been
that
we
actually
are
unable
to
use
a
centralized
scheduling
algorithm
that
has
actually
precise
information
of
the
state
of
the
system.
Things
are
changing
too
quickly
to
have
the
thought.
Those
controllers
have
an
accurate
view
of
what's
going
on
so
what
occurred
load
current
load,
balancing
algorithm
does.
A
A
So
if
you
want
to
play
with
open
wisk-
and
you
have
an
idea
for
something-
you
think
could
do
better,
you
could
plug
it
in
here
and
try
it
so
I
encourage
you
to
after
this
talk,
maybe
try
that
out.
Ok,
so
stepping
back
up
to
the
flow
of
the
function
to
the
system
once
they
can
load
balancer
has
picked
an
invoker
wien
queue
it
on
a
row
on
a
work
queue
for
that
invoker
to
pick
up
later
and
there's
one
we
use
Kafka
for
that.
A
There's
one
topic
for
every
invoker:
eventually,
the
invoker
will
pull
it
off
there
and
hopefully
be
able
to
select
a
container
already
has
to
execute
the
request.
If
not,
it
has
to
create
one.
It
invokes.
The
action
awaits
the
results
once
it
has
the
result.
It
does
two
things.
First,
it
immediately
accused
that
back
on
Kafka
to
go
back
to
the
controller
after
it
accuse
it
is
then
asynchronously
extracts
the
user
container
logs
and
forwards
them
to
a
logging
service.
A
So
we
have
to
get
the
logs
out,
but
we
don't
have
to
wait
for
that
to
get
the
result
back
to
the
user.
The
result
flows
back
through
Kafka
back
to
the
controller,
and
if
it's
a
blocking
request,
then
we
return
the
result
on
the
open,
HTTP
connection.
So
that's
sort
of
the
path
of
a
function
of
the
git
as
it
goes
in
and
out
of
the
system
and
talked
a
little
bit
more
about
the
invoker
in
particular
how
its,
how
its
interacting
with
the
underlying
container
engine.
So
this
is
another
place.
A
We
have
an
SPI,
it's
the
container
factory
spi,
and
what
this
does
is.
It
gives
us
a
pluggable
abstraction
for
container
engines.
We
currently
have
three
implementations
of
this
in
the
open
source
project.
There's
one
that
talks
to
docker,
one
that
talks
with
kubernetes
and
one
that
talks
to
mesas
and
we
deploy
when
we
deploy
open
wisk
on
top
of
kubernetes.
We
actually
have
the
option
of
using
either
the
docker
container
factory
or
the
kubernetes
container
factory.
A
I'll
do
now
is
just
explain
what
each
of
those
look
like
and
explain
why
we
might
want
to
use
them.
Okay,
so
the
docker
container
Factory
is
the
simplest
one.
In
this
setup
we
allow
kubernetes
to
schedule
the
pods
that
manage
that
represent
the
open,
wisk
control
planes,
all
those
blue
boxes,
the
control
or
the
invoker's
Kafka
zookeeper,
our
other
system
components,
but
not
the
user
containers
from
kubernetes
perspective.
A
What
happens
here
is
we
run
the
invoker
and
a
daemon
said
as
the
daemon
set
on
all
that
a
worker
knows
that
whether
that
are
labeled
with
the
role
invoker
and
that's
the
only
kubernetes
pod,
that's
actually
running
on
that
node.
Well,
we
do
that
as
the
invoker
talks
to
the
local
docker
demon
on
that
node
and
does
all
the
container
operations
for
us.
So
as
far
as
kubernetes
kids
can
learn.
A
There's
just
one
thing
running
here
and
we've
tainted
the
node
and
the
containers
to
prevent
anything
else
from
going
there
and
then
the
invoker
talks
to
docker.
Does
all
the
Creator
operations
talk
to
the
user
containers
to
initialize
and
run
them?
Ok,
so
that's
not
incredibly
Kubb
friendly
hube
native,
but
it's
a
way
of
making
things
work
and
it
has
some
reasonable
performance
characteristics
which
I'll
talk
about
in
a
little
bit.
A
second
alternative
which
you've
been
exploring
is
the
kubernetes
container
Factory.
A
This
is
a
much
more
kubernetes
centric
approach
to
run
approach
to
running
open
wisk.
What
we
do
here
is
now
we're
running
the
investments
smaller
number
of
invoker's
much
smaller
than
there
are
worker
nodes.
We
deploy
those
as
a
stateful
set,
the
invoker
stalks,
the
kubernetes
scheduler
to
actually
create
and
destroy
the
pods,
the
user
containers.
So
now
our
user
containers
or
proper
pods
they
can
run
on
any
worker
node
whose
role
is
invoker,
and
so
now
we
can
have
an
invoker
managing
pods
managing
user
containers
that
are
scattered
across
multiple
nodes
in
the
system.
A
So
early
we
can.
The
invoker
can
have
a
much
larger
pool
of
containers
and
it
calls
unit
run
thing.
There's
one
problem
with
this
picture,
if
you
noticed
so
kubernetes
doesn't
have
any
notion
of
suspending
a
pod
all
right.
So
what
I've
lost
now
is
have
lost
my
ability
to
stop
people
from
stealing
from
me
and
that's
not
so
great,
if
I'm
going
to
actually
try
to
run
this
for
a
profit,
so
we
get
around.
That
is,
we
add
this
notion
of
invoker
agents.
A
So
the
invoker
agents
or
daemon
said
it's
a
small
program
written
and
go.
It
runs
on
each
of
these
worker
nodes
and
it
all
the
Anoka
agent
does.
Is
it
actually
does
the
suspend
and
resume
operation
so
when
invoker
needs
to
suspend
to
resume
a
user
container?
It
talks
the
invoker
agent
on
the
appropriate
worker
node,
which
talks
the
docker
and
suspends
the
container,
and
it
also
efficiency.
We
also
when
we
have
the
invoker
agents
running.
A
We
can
also
use
them
to
extract
the
user
container
logs
directly
from
the
local
file
system
without
having
to
go
through
the
Cooper
Nettie's
api's.
Okay.
So
that's
the
three
different
options
for
the
container
factories.
What
I
want
to
do
now
is
talk
a
little
bit
about
this
present.
Some
experimental
results,
sort
of
look
at
the
performance
trade-off
of
these
three
options
and
the
goal
is
not
to
sort
of
say
that
is
it's
just
to
understand.
The
current
performance
of
our
container
factory
implementations,
particularly
to
understand
the
cold
start
latency.
A
What
do
these
different
container
factories?
What
does
it
do
for
that?
What's
our
warm
and
hot
breeze
lilies
and
see
so
what
happens
when
I
can
reuse
the
container
and
are
there
any
throughput
effort
differences
between
them?
And,
if
so,
why?
So?
These
experiments
are
done
in
a
pretty
small
cluster.
It's
a
kubernetes
1.8
cluster
with
13
nodes.
I
have
10
invoker's
two
nodes
that
are
dedicated
to
run.
A
My
blue
boxes,
my
control
plane
and
one
know
that
I
use
for
a
load
driver
and
what
that
load
driver
by
having
the
load
driver
as
part
of
the
cluster
I
can
eliminate
any
kind
of
external
Network
variability
than
a
driving
load
from
them
or
my
laptop.
So
for
all
the
all
the
experiments
I'm
gonna
measure
the
full
path,
starting
with
the
ingress
going
all
the
way
through
the
through
the
through
the
flow,
so
ingress
controller,
Kafka
and
Volker
Kafka
controller
back
out.
A
A
Okay,
so
first
talk
about
latency
results,
so
here
the
first
set
of
leads.
The
first
set
of
results.
We're
going
to
do
is
we're
going
to
invoke
a
blocking
function.
One
of
the
time.
This
is
the
only
thing
that's
happening
on
the
clusters.
This
is
an
extremely
optimal
situation.
All
we're
doing
is
invoking
one
function
again
and
again
and
again
10,000
times
in
a
row
and
measuring
what
the
end
and
latency
is
it's
perceived
by
the
user
in
in
milliseconds
and
there's
several
things
to
look
at
here.
A
A
A
second
set
of
the
things
to
look
at
is
why
is
container
use
so
important,
so
here
if
we
have
a
warm
container
and
we're
using
the
docker
container
factory
our
end
and
response
time
for
this
trivial
action
is
7
milliseconds
all
the
way
through
our
control
plane
and
back
in
7
milliseconds,
using
the
docker
container
Factory.
Now,
surprisingly,
with
the
kubernetes
container
factory,
it's
actually
slower,
and
this
doesn't
make
a
lot
of
sense.
Initially,
we
had
to
dig
into
it
fine
to
find
out
why?
A
Because,
if
you
think
about
this,
where
we're
using
the
containers,
the
container
Factory,
the
kubernetes
container
Factory-
is
not
really
involved
in
this
path,
except
for
one
thing,
and
this
in
the
kubernetes
mother
and
the
second
configuration
here,
the
kubernetes
container
Factory
is
actually
extracting
the
user
logs
for
us
versus
in
the
invoker
agent,
where
it's
not
and
what's
happening
here
is
even
though
the
log
processes
processing
is
asynchronous,
we
push
it
off
the
off
the
path.
It
takes.
A
A
really
long
time
for
kubernetes
to
get
those
logs
out
of
that
container
and
from
the
invoker
to
processes
of
and
forward
them,
and
so
the
load
balancer
thinks
the
container
is
idle.
The
answers
come
back.
It
thinks
it
can
reuse
that
that
invoker
in
that
container
again,
but
it's
still
loaded
down
extracting
lodge
from
the
last
10
and
functional
vacations
that
have
come
to
that
container.
So
that's
something
that
could
be
fixed
by
modeling
in
our
load,
balancer
or
ideally
could
be
fixed
by
improving
our
log
extraction
or
kubernetes
log
extraction
capabilities.
A
Okay,
the
second
thing
I
want
to
look
at
is
throughput
results,
and
so
here
the
workload
is
slightly
different
or
again
running
no
op
actions.
What
we're
doing
this
time
is
running
as
many
non-blocking
implications
of
this
trivial
action,
as
we
can
on
the
system
and
so
I'm
showing
normalized
throughput.
We're
higher
is
better,
so
we
have
one
and
things
less
than
one,
and
here
again
the
kubernetes.
The
two
kubernetes
container
factories
do
slightly
worse
than
the
docker
container
Factory,
and
this
is
primarily
due
to
the
higher
light
to
the
log-log
extraction.
A
Latency,
preventing
container
use
loading
down
the
system
and
reducing
overall
throughput,
so
the
key
takeaways
from
this.
First
of
all,
you
know
the
higher
cold-start
cost
you
matter
to
us.
Our
caching
is
fairly
effective,
but
even
if
it
you
know,
if
we're
gonna
have
something
cold
starts,
we
care-
and
we
want
to
make
this
as
short
as
possible
and
there's
a
real
latency
impact
for
those
for
short
running
function.
And
the
second
thing
which
was
surprised
to
me
is
that
this
log
extraction
path
actually
mattered
a
lot.
A
So
we
have
some
work
going
on
now
to
try
to
understand
how
to
better
optimize
our
log
extraction
and
our
logging
architecture
on
kubernetes
to
try
to
get
rid
of
that
problems.
We
really
want
to
be
able
to
use
the
kubernetes
container
Factory
to
take
advantage
with
a
much
larger
pool
of
containers
that
could
be
managed
by
the
invoker
and
also
to
fit
more
naturally
into
the
cluster
scheduling.
A
Okay,
so
that's
sort
of
our
dive
through
how
they
open
wisk
core
works
and
how
we
they're
this
hot
path
of
invoking
functions.
All
I
do
now
is
shift
gears
and
talk
about
serverless
composition
of
serverless
functions
and
and
give
you
a
taste
of
that,
and
so
this
is
a
project
that's
being
driven
out
of
research
and
research.
We
like
to
dream
big
it's.
What
really
trying
to
do
here
is
rethinking
the
entire
cloud
programming
experience
for
the
developer,
we're
coming
out
of
thinking
about
we're,
really
we're
programming,
a
cloud
computer.
A
Can
we
make
this
thing?
Look
more
like
a
computer
for
the
programmer.
Can
we
build
a
computer
like
facade
over
this
distributed
system
and
do
this
by
just
writing
factual
programs
in
ways
that
programmers
know
how
to
write
them
and
have
good
tools
to
make
this
be
available
to
help
us
alright,
so
I'm
gonna
do
now
is
I'm
going
to
a
demo
and
I'm
going
to
use
a
little
sample
program
that
we
use
to
do
this,
and
this
is
something
that
we
use.
A
So
the
idea
here
is
I'm
a
developer
I'm
doing
pull
requests
I,
they
go
into
Travis
CI
test
run.
Unfortunately,
tests
take
a
while,
and
eventually
they
finish,
but
by
then
I've
gotten
distracted
and
I've,
forgotten,
they've
run,
and
so
it
this
year
is.
This
is
a
pipeline
that
takes
a
Travis
webhook,
runs
it
through
a
bunch
of
service
functions
to
process
that
web
book
I
get
some
information
about
it
and
then
post
the
results
of
my
tests.
A
A
So
first
of
all
so
I
have
mini
cube
running
on
my
laptop
here
and
on
that
I've
deployed
open
Wisc
using
my
home
schardt,
so
I
can
do
helm
status
demo
and
there
it
is
it's
running
and
we
can
see
what
pubs
are
are
alright,
so
there's
the
control,
plane,
pods,
the
controller,
the
cat,
the
database,
the
invoker
and
so
on.
Alright,
so
now
we're
going
to
do
is
we're
going
to
bring
up
shell.
This
is
our
function.
A
A
It's
one
thing:
that's
kind
of
a
kind
of
useful
as
a
debugging
tool,
so
now
I'm
in
this
shell.
This
is
an
electron
based
application.
I'm
looking
at
my
deployed
function,
I
can
actually
edit
this
here.
Alright
I
can
hit
the
lip
unlock
button
now
I'm
in
edit
mode
editing
my
deployed
code,
so
I
can
say
this
is
live
if
I
can
spell
live.
A
Ok,
so
now
we'll
deploy
that,
and
if
we
run
it,
we
should
get
a
different
message
this
time,
tada
all
right.
So
this
is
a
it's.
This
is
a
useful
way
to
sort
of
to
get
in
and
do
quick
debugging
for
things
alright.
So
now
for
the
I'm
gonna
do
is
shift
to
a
slightly
2vs
code,
because
we're
gonna
look
at
a
more
interesting
example.
In
particular
we're
gonna
look
at
this
composition
and
there
we
go
it's
good
things
lined
up.
So
this
is
my
Travis's
slack
pipeline.
A
The
picture
on
the
right
over
here
is
a
visualization
of
the
pipeline
is
produced
by
shell
in
a
couple
minutes.
We're
going
to
load
this
app
in
the
shell
and
like
to
see
this
visualization
live,
and
over
here
on,
the
left
is
the
code.
So
we've
written
this
composition.
This
is
in
a
JavaScript
library
that
lets
us
talk
about.
They
lets
us
use
familiar
constructs
to
build
out
this
composition.
So
here's
my
pipeline
here
and
there's
several
things
to
notice
about
this.
A
First
of
all,
I
can
have
local
variables,
so
I
can
declare
a
variable
and
then
I
can
use
that
to
cache
intermediate
results
in
this
pipeline
and
use
the
multiple
times.
I
have
a
sequence:
I'm
composing
a
whole
bunch
of
action,
so
each
one
of
these
lines
is
an
action.
Some
of
these
are
defined
externally,
so
these
are
functions.
We're
gonna
define
in
a
couple
of
minutes.
A
Some
of
them
are
just
pieces
of
inline
javascript
code,
and
some
of
them
can
even
be
function
so
here
this
this,
this
particular
composition
allows
me
to
either
get
the
authors
like
information
from
a
database
or
from
a
local
JSON
file
for
debugging.
So
at
deploy
time
when
I'm
deploying
this
composition
I
this
JavaScript
code
to
build
up
my
deployment
artifact,
we
can
get
in
here
that
we
run
this
get
author
slack
information
and,
depending
on
the
value
of
this
environment,
variable
will
either
connect
to
a
database
or
we
connect
to
a
local
JSON
file.
A
A
couple
other
things
to
see
in
here,
so
one
is
so
that
we
have
one
of
the
uses
of
these
inline
functions
is
my
my
information
coming
in
here.
This
p
has
an
author
name
in
it.
It
could
be
in
any
case
at
all.
My
database
wants
it
to
be
uppercase,
so
I
can
write
a
quick
little
inline
function
in
JavaScript
here
that
goes
ahead
and
converts
that
from
loop
from
whatever
case
it's
in
to
uppercase
I
also
try
catch
blocks.
A
So
here
I've
got
a
try-catch
block
in
this
line
here
and
what
I'm
gonna
do
is
I'm
gonna
read
in
the
database
now,
if
the
author
isn't
hasn't,
subscribed
to
slack
notification,
so
they
don't
want
to
be
bothered
by
every
time.
There's
a
slack
request.
This
lookup
will
fail.
I'll
get
an
exception,
but
I'll
catch
that
in
the
catch
block.
A
Okay,
there
it
is
so
here's
the
visualization,
and
it's
helpfully
telling
me
hey
by
the
way
you
haven't
deployed
these
actions
yet
so,
rather
than
forcing
you
to
watch
me
type
that
I'm
going
to
run
a
script
that
will
deploy
them
for
me,
okay.
So
now,
if
we
do,
oh
yeah
I
need
to
put
that
in
the
background,
so
it'll
actually
run.
A
A
Okay,
this
should
take
about
five
seconds
or
so
hopefully,
and
I
should
get
a
slack
notification
that
this
happened
and
it
didn't
work.
What
happened
all
right
so
what
happens?
We
can
look
at
the
session
flow.
Oh,
no,
it
did
work.
It
just
didn't
right.
Here's
my
new
slack
notification,
but
it
didn't
show
up
on
my
window
over
here.
I
know
if
it
showed
up
on
yours
or
not,
this
may
be
an
artifact
of
being
in
screen,
show
mode
all
right.
So
here's
the
session
flow.
A
So
this
is
showing
that
here's
my
pipeline,
the
things
are
dark
blue
because
it
actually
ran.
If
we
hover
over
any
of
these,
we
can
actually
see
what
the
values
were.
We
can
dive
in
to
look
at
them.
Go
back
to
the
app
visualization
all
right.
We
can
still
get
the
results
of
this
and
why
there?
Okay,
there
we
go.
A
Get
my
session
flip.
We
also
have
a
traceview,
so
here
we
can
look
and
see
a
timeline
of
the
the
execution
of
this.
So
the
overall
thing
took
six
point
four
seconds
we
can
looked
at
various
actions.
The
shading
here
indicates
the
shaded.
The
the
hash
part
is
the
initialization.
The
solid
part
is
the
actual
execution
of
it.
A
We
can
look
down
to
go
down
and
look
at
that
alright.
Well,
this
is
pretty
cool,
but,
as
I
said,
some
authors
don't
actually
some
some
people
don't
actually
care
about
this,
and
we've
done
all
this
computing
to
generate
a
notification
that
nobody
cares
about.
That
seems
like
a
natural
thing
to
fix
with
some
control
flow,
so
let
me
find
vs
code
again.
Alright,
so
we
to
now
is
gonna.
Go
here
go
in
here
and
basically
short-circuit.
This
is
out,
so
we
can
say
composer.
A
A
We
put
the
rest
of
our
code
in
there,
and
this
is
just
plain
all
JavaScript
code
vs
code
understands
it,
so
we
can
indent
it
we're
done.
We
can
save
it,
go
back
to
here,
one
VOC,
it
again
and
hopefully-
and
this
is
this-
is
a
this-
should
actually
go
through
and
the
slack
window
didn't
pop
up.
But
if
you
look
down
in
here,
what
did
it
do?
Oh.
A
A
Okay,
good
now,
my
flow
diagram
has
changed.
Is
that
too
useful
cuz
I
actually
expected
it
exchanged?
And
so,
if
we
invoke
it,
this
time,
I
think
it's
again.
Gonna
fail
to
pop
up.
But
now,
if
we
look
at
the
session
flow,
it
went
down
this
path,
and
now
I
also
have
a
second
sample
file
for
someone
who
doesn't
care
about
this,
and
so
that
happened
almost
immediately,
because
if
we
look
at
the
session
flow,
what
happened
here
is
we
came
into
the
first
few
steps.
A
We
hit
that
if
and
we
didn't
bother
doing
the
computing
to
actually
generate
the
slack
message
for
them.
Okay
and
again,
we
can
look
at
the
trace
and
see
what
happened
and
all
that
alright.
So
that's
the
end
of
the
demo
and
I'm
going
to
do
now
is
shift
back
to
the
PowerPoint
and
kind
of
wrap
up
the
talk.
A
Okay,
so
hopefully
that
I've
shown
you
it's
a
little
piece
of
our
sort
of
vision
for
how
we
can
make
a
more
computer
like
programming
environment
from
the
cloud
we
saw,
we
had
a
desktop
development
environment
that
lets
me
use
either
my
own
editors
or
its
built-in
editor
I
can
quickly
edit
code.
That's
that's
been
deployed
on
the
couch
cloud
and
redeploy
it
so
I
can
just
like
I
changed
that
a
little
hello
message
and
the
first
thing
I
could
stick
in
a
login
printf
and
redeploy
almost
instantly
with
composer
I.
A
Have
a
library
written
in
a
familiar
language
that
lets
that
lets
me
have
variables,
control
structures,
if-then-else,
try-catch,
loops
and
so
on
and
use
those
to
actually
build
a
distributed
program.
That's
running
across
a
polyglot
set
of
applications,
so
I
forgot
to
mention
this,
but
you
know
I
was
using
javascript
and
composer,
but
I
was
invoking
actions
or
written
in
Python
and
JavaScript.
So
there's
not
another
restriction
that
about
what
the
actual
functions
have
to
be
written
in
with
shell.
A
A
So
once
we
have
everything
open,
west
granny
on
top
recruitment
at
ease,
there
doesn't
really
need
to
be
a
sharp
distinction
between
what's
a
function
and
what's
a
container
and
what's
a
micro
service
and
sort
of
trying
to
blend
that
together
to
make
it
easier
for
the
developer
to
go
back
and
forth
between
those
opposed
are
in
shell.
We're
really
excited
about
this.
A
We
think
this
is
potentially
a
major
step
in
the
evolution
of
the
curve
at
about
cloud
developer
experience,
there's
a
lot
of
active
research
going
on
there
and
development
and
all,
and
so
on.
In
terms
of
you
know,
just
to
wrap
up
I'd
like
to
encourage
people
to
try
to
get
involved
with
this
everything
I've
shown
you
today
is
open
source
open
wisk
is
obviously
but
also
Composer
and
shell
are
as
well
and
so
here's
some
URLs
for
web
pages
and
github
x'
and
slacks
if
you're
interested
in
anything
that
we've
shown
today.