►
From YouTube: Operating a Global-Scale FaaS on Top of Kubernetes - Chad Arimura & Matt Stephenson, Oracle
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Operating a Global-Scale FaaS on Top of Kubernetes - Chad Arimura & Matt Stephenson, Oracle (Any Skill Level)
This talk will cover the why and how of building a global-scale FaaS using the serverless Fn Project on top of Kubernetes. Focusing on the process of designing and building the service to operate under the most demanding production workloads, the talk will cover the following topics: - Visualizing the flow of traffic from request through DNS, FnLB, K8s node ports, pods, and back - Tuning k8s for massive-scale short-lived pods/containers - Managing resource utilization and auto-scaling of infrastructure inside and above Kubernetes - Optimization and routing of traffic to “hot” nodes for increased performance - Using Helm and DB/queue operators to manage dependent services - Quirks and other unexpected issues discovered along the way If time permits I will end with a live demo of the system working under load and how it responds.
About Chad
Chad is a 3x entrepreneur, with over 20 years experience leading technical teams and building software, most recently as co-founder and CEO of Iron.io where they grew a multi-million dollar ARR business helping pioneer serverless computing. The company was acquired in 2017 and Chad and team joined Oracle. He is now Vice President of Serverless at Oracle, currently leading The Fn Project, open source serverless FaaS platform and orchestration. He still likes to code, preferably in front of a live audience, and speaks quite often about serverless computing, entrepreneurship, building teams, culture, and other topics.
About Matt
Matt is a Senior Principal Software Engineer at Oracle working on creating a hosted Fn implementation on top of Oracle's Cloud. He built some of the underlying infrastructure components that run parts of AWS while at Amazon as well as the AppEngine java runtime while at Google. Matt is on the PMC for Apache jclouds and contributes to Jenkins, OpenStack, and of course the Fn Project. In his free time he helps design the man at Burningman and flies airplanes.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
I
actually
do
have
some
proof
that
2011
actually
that's
from
2010,
so
the
the
hacker
news
stuff
must
have
come
later,
but
it
shows
that
we
were
kind
of
doing
some
writing
and
calling
a
future
service
and
actually
proved
to
be
sort
of
smart.
We
didn't
really
think
about
it
back
then,
but
now
it's
a
whole
category
of
computing.
In
fact,
it's
a
whole
theme
of
cube
con,
which
I
thought
was
pretty
cool,
yeah
yeah.
B
So
I'm
Matt
Stephenson
I've
been
an
engineer
on
maybe
one
more
cloud
than
chatez
entrepreneur.
Company
into
success
and
I've
worked
on
Oracle
cloud,
Amazon's,
AWS,
Google
cloud
platform
and
OpenStack
I
started
off
in
the
DevOps
community,
building
out
Apollo
and
Amazon,
so
I'm
kind
of
one
of
the
oh
geez
there
and
I'm
also
an
open
source
hippie
working
on
him,
patchy,
J
clouds,
Jenkins
and,
of
course,
the
FN
project
and
I'm
building
out
service
at
Oracle.
So.
A
A
Let's
say
it's
gonna
change
the
world
and
it's
gonna.
It's
gonna
fix
all
your
problems,
you're
gonna
sleep
at
night
and
it's
gonna.
It's
gonna,
be
amazing.
We're
never
gonna
think
about
any
infrastructure.
We're
there
right
now,
but
this
the
problem
with
the
with
this
is
that
the
peak
always
there's
always
a
dissent
at
the
other
side
of
that,
and
so
I
think
we're
aiming
for
a
little
bit
of
a
disillusionment
section
with
serverless.
And
then
hopefully
we
can
come
out
the
other
side.
A
But
another
cool
thing
is
we're
actually
seeing
a
ton
of
adoption.
So
service
is
being
used
in
a
lot
of
different
places,
we're
seeing
use
cases
all
over
the
place.
It's
in
keynotes,
it's
a
whole
category.
Now
it's
really
exciting.
This
is
something
the
new
stack
put
out,
and
actually
this
is
a
year
old.
You
know
it's
kind
of
a
headline
article,
so
I,
don't
probably
not
totally
true,
but
you
know
the
numbers
that
they
use
show
that
service
adoption
is
approaching
or
or
eclipsing
containers.
So
that's
pretty
cool.
A
This
talk
is
about
helping
us
move
towards
this
slope
of
enlightenment.
So
hopefully
we
can
share
some
of
our
experiences,
avoid
the
traffic
loserman
or
at
least
reduce
the
time
that
we're
there
and
move
into
the
slope
of
enlightenment.
There's
a
lot
of
talks,
a
lot
of
great
talks
on
service
architectures
how
to
apply
best
practices,
ci
CDE,
there's
a
whole
bunch
of
really
cool
stuff
out
there,
and
there
will
continue
to
be
really
good
talks
on
that
as
we
continue
to
mature
the
ecosystem.
A
But
this
one
is
more
about
our
experiences
of
launching
of
functions
of
the
service
on
top
of
kubernetes,
because
we
believe
that
a
lot
of
people
are
going
to
want
to
do
that.
There
are
a
lot
of
customers
for
a
lot
of
reasons
that
want
to
operate
and
own
and
and
control
their
own
Cerullo
solutions
on
top
of
kubernetes.
Yes,
it
is
still
server
list
of
the
developers,
but
of
course
there
are
always
servers.
A
So
this
talk
is
mostly
about
peeking,
behind
the
curtains
of
what
we
are
up
to
at
Oracle,
launching
Oracle
functions,
which
is
based
off
in
open
source
project
and
we're
talking
about
helm
and
how
we
wire
these
things
together.
So
we'll
share
some
some
helm
code
and
and
in
the
things
behind
that
we
talked
a
little
bit
about
multi-tenancy
and
what
that
means
to
us
and
how
we
do
isolation
and
some
of
the
gotchas
there
and
then
kubernetes
at
scale.
A
So
there's
considerations
when
managing
lots
of
kubernetes
when
you're
doing
sort
of
big
scale
and
then,
of
course,
will
throw
on
a
bunch
of
things
around
operators,
controllers,
flex,
volume,
drivers
that
we've
written
another
cool,
kubernetes
stuff
that
we've
been
able
to
contribute
back
to
the
ecosystem
along
the
way.
Oh,
my
we
are
building
a
service
and
it
is
based
off
the
FN
project.
A
We're
not
gonna
talk
about
the
FN
project,
a
whole
lot,
but
the
FN
project
is
an
open
source
service
platform
that
enables
developers
to
easily
build
and
ship
code
to
the
cloud
containers
are
the
primitive.
So
it
operates
off
standard
in
and
standard
out
and
any
container
can
be
your
function,
it's
open
source!
You
can
take
it!
You
can
run
on
your
own.
You
don't
even
have
to
call
anyone.
You
don't
stock
to
anybody
to
Oracle
and
you
could
be
off
to
the
races.
A
We're
also
going
to
be
building
a
service,
which
is
what
this
talk
is
about.
A
global
scale
to
us.
This
15
plus
regions
that
we've
announced
big
expansion
plan
across
the
globe
through
ey
19
and
in
FN
oracle
functions,
will
be
in
all
of
those
most
of
those,
hopefully
mostly
all
of
them,
we're
also
backed
by
industry,
hi,
SLS,
Oracle's
known
for
its
enterprise
customer
base,
who
have
very
high
demands
and
high
workload
demands,
particularly
around
SL
A's
and
lots
of
acronyms.
A
That
mean
lots
of
things:
lots
of
people
mostly
around
security,
so
PCI
a
sock,
Hiep,
ISO
and
FedRAMP
DoD
data
centers.
So
this
is
going
to
be
pretty
big,
big
scale
stuff
and
that's
one
of
the
guys-
that's
definitely
building
it.
So
it's
got
a
good
perspective
on
it.
This
is
what
our
service
looks
like
sorry
for
the
Chad
I
just
had
to
call
it
something
when
I
was
deploying
using
a
scripts.
So
I
could
take
this
slide,
but
we
check
this
out.
We
have
all
of
our
stuff
at
the
top,
our
pods.
A
A
It's
like
a
smart
load
bouncer
to
ensure
that
we
have
hot
containers,
so
there's
some
routing
stuff
there
that
we
can
talk
about
and
then
our
UI
and
then,
of
course,
we're
using
my
sequel
and
Redis
it's
pluggable,
so
you
can
use
a
bunch
of
different
storage
engines
and
we'll
talk
a
little
bit
about
the
operators
that
we've
built
from
my
sequel.
That's
really
really
cool.
A
B
So
we
use
home.
Obviously
it
makes
things
a
lot
easier.
One
of
the
biggest
benefits
we
get
from
helm
is
that,
like
just
makes
everything
a
lot
easier
to
wire
up
and
we've
taken
a
lot
of
care
in
exactly
what
we're
wiring
up
and
how
we're
wiring
it
up,
yeah
that
is
kind
of
annoying
at
least
I've
bolded,
this
stuff
up
a
little
bit.
B
B
So
what
else
is
up
with
helm?
We
found
that
the
requirements
IMO
is
not
quite
enough
for
what
we
need
to
do
a
lot
of
the
time.
A
lot
of
that
comes
down
to
like
we
have
a
lot
of
other
operators
that
we
deploy
into
our
clusters
and
because
home
generates
a
single
yamo
file.
Essentially
that
gets
applied
to
the
cluster.
There's,
not
a
whole
lot
of
determinism
as
to
win
certain
parts
of
the
chart
were
the
of
the
chart
that
gets
generated
it
gets
applied.
B
So
the
operators
that
you
may
be
defining
custom
resources
within
your
chart
that
rely
on
another
operator.
You
can't
necessarily
include
that
in
the
requirements
that
yema,
because
that
operator
may
not
have
started
and
registered
the
customer
resource
that
you
need
to
define
in
your
e
animal,
so
there's
few
limitations
with
helm
that
we're
getting
around
mostly
through,
like
programmatic
bash
scripts
and
and
such
for
our
deployments.
B
It
gives
us
the
capacity
to
deploy
many
different
versions
of
our
service,
our
development
clusters,
so
that
we
can
test
very
easily-
and
you
know
just
with
builds
declarative,
builds
or
declarative
deployments,
they're,
nice
and
they're
clean
and
they're
pretty,
and
they
work
most
of
the
time.
But
if
you're
always
painting
yourself
into
that
declarative
corner,
you're
gonna
have
to
do
a
lot
of
stuff
to
get
out
of
it
over
time.
So
that's
why
we
don't
necessarily
use
all
of
the
facilities
that
home
provides
and
I'm
happy
to
answer
questions
about
that
at
the
end.
B
So
what
do
we
do
for
CI
CD?
So
in
different
branches,
we
actually
deploy
the
entire
chart
of
just
our
service
in
two
different
namespaces,
and
that
gives
us
the
ability
to
then
run
using
jobs.
All
of
our
integration
tests
against
those
individual
branches
that
we've
deployed.
What
that
does
is
it
lets
us
very
easily
replicate
tests
between
the
service
that
we
deploy
in
our
production,
testing
and
development
clusters
for
doing
regular,
routine
testing
of
the
service.
While
it's
running
and
those
tests
that
we
want
to
make
sure
that
the
service
passes
before
we.
B
Let
in
any
commits
and
the
home
has
a
lot
of
really
neat
hooks
for
jobs,
I'm
happy
to
answer
questions
at
the
end,
these
three
little
bowled
it
out
guys
here
those
lines
right.
There
are
incredibly
useful.
So
if
you're
doing
CI,
CD
using
home
or
CI
CD
using
kubernetes
at
all,
really
I
would
take
a
look
at
that.
It
makes
your
life
a
lot
easier
see.
So
we've
had
a
couple
moving
onto
multi-tenancy
and
like
thinking
about
how
we
build
a
multi-tenant
service
on
top
of
kubernetes.
B
We've
had
a
lot
of
problems
trying
to
get
things
to
work
exactly
the
way
we
want
with
kubernetes
and
I
think
that
as
kubernetes
evolves,
we're
going
to
be
catching
up
to
it.
But
there's
a
lot
of
questions
that
we
have
like
in
order
to
isolate
each
individual
customer.
We
have
to
have
certain
guarantees
around
like
network
isolation
and
kubernetes
is
almost
there.
B
There's
certain
network
overlays
that
you
can
use,
but
we're
still
trying
to
like
kubernetes,
isn't
quite
there
for
us
and
it
also
doesn't
give
it
gives
some
resource
guarantees,
but
to
avoid
noisy
neighbors
like
the
biggest
problem.
You
usually
have
is
I/o
contention,
and
we
you
really
don't
get
that
using
docker
kubernetes.
Quite
yet.
I
think
that
when
kubernetes
catches
up
to
that
will
probably
be
leveraging
that.
B
But
for
now
what
we
do
is
we
have
an
isolated
chunk
of
our
infrastructure
or
node
scaling
and
node
management,
and
we
have
essentially
all
of
the
services
components
of
the
FN
service
running
within
the
kubernetes
cluster.
But
the
actual
runtime
for
running
functions
themselves
has
managed
outside
of
that
and
the
way
we
do,
that
is
with
our
node
pool
management
or
as
our
node
node
pool
manager,
abstraction
and
I'll
kind
of
look.
I'll
walk
through
a
little
bit.
B
B
So
different
workloads
require
different
types
of
nodes
to
be
deployed
on
and
so
kind
of.
Managing
those
resources
within
the
kubernetes
cluster
requires
some
orchestration
and
management
that
essentially
we're
doing
some
pod
affinity
work
within
turbine
Nettie's
to
make
sure
that
certain
services
are
working
on
certain
shapes
of
nodes
and
at
the
same
time,
we
have
the
same
kind
of
problem
with
the
runners
that
we're
deploying
outside
of
kubernetes
to
run
the
functions
themselves.
And,
of
course,
like
my
experience,
working
on
App
Engine
at
Google.
B
B
First,
your
that's,
probably
going
to
be
the
thing
that,
when
you
hit
when
you
put
this
thing
in
production,
that's
gonna
be
the
problems
that
you'll
hit
before
you're
gonna
run
into
memory
or
CPU
and
most
likely
another
well
one
of
the
big
ways
that
we
leverage
kubernetes
to
make
this
easier,
at
least
for
those
services
that
run
within
the
kubernetes
cluster.
As
we
use
a
lot
of
our
operators,
some
of
those
that
are
built
into
all
of
the
Oracle
kubernetes
clusters
that
we
have
deployed,
we
use.
We
have
operators
for
instantiating
load
balancers.
B
But
you
know
other
cloud
providers
have
that
as
well
like
Google,
Cloud
and
BSS
or
Google
Cloud.
Definitely
that
says
no,
and
then
we
have
a
persistent
volume
manager
operator
that
gives
us
the
ability
to
have
persistent
volume
claims
that
are
attached
directly
to
block
volumes
that
are
instantiated
within
our
infrastructure
as
a
service,
and
we
utilize
faithful
sets
to
make
it.
So
that's
easy
for
us
to
move
that
workload
back
and
forth
and
scale
that
up
and
down
we
have
a
my
sequel
operator.
B
B
Of
course,
I
mentioned
earlier,
we
use
pod
affinity
to
kind
of
segment
out
all
of
our
high
I/o
pods
from
high
CPU
pods,
and
then
we
have
separate
instance
shapes
for
that
within
our
cluster
and
I.
Think
that
that's
kind
of
necessary
for
any
service
of
your
building
and
the
nice
thing
about
kubernetes
is
it
makes
that
really
easy.
B
If
you
want
to
implement
your
own
effin
project,
note
pool
manager
which
would
allow
you
to
run
most
of
run.
Basically
everything
that
we
have
running
today
in
the
demo
that
we'll
touch
on
later
in
a
kubernetes
cluster
aside
from
the
actual
running
of
the
functions
so
it'll
run
on
separate
and
those
run
on
separate
instances
for
at
least
four
Oracle
cloud-
and
this
is
the
extension
points
you
can
use
to
do
that
yourself.
B
The
runner
pool
is
an
abstraction
for
getting
a
list
of
runners
so
as
the
load.
These
are
all
extension
points
from
within
load,
balancer
itself
in
FN
and
that
what
we
do
is
we
essentially
call
back
the
the
note.
The
load
balancer
calls
back
and
is
reporting
back
like
it's
getting
as
many
runners
to
getting
back
runners
from
the
running
pool
and
then
the
runner
pool
as
a
result
knows
where
calls
are
being
placed
and
in
our
implementation
of
all
these
interfaces
are
implemented.
B
Basically,
in
the
same
service,
we
know
how
many
calls
are
being
placed
out
on
the
render
pool
and
we
can
do
that
scaling
within
the
pool
manager
and
this
continued
to
feed
runner
pool
runners
back
to
the
load
balancer,
so
the
load
balancer
always
can
be
able
to
rotate
requests
on
to
different
instances.
So
this
node
pool
manager
works
to
give
us
the
ability
to
scale
up
and
down
the
amount
of
runners
that
we're
providing
and
those
runners
are
just
you
know,
standard
instances
using
Linux
kit.
B
As
you,
we
use
Linux
get
to
build
the
operating
system
that
runs
on
them
and,
as
we
scale
those
we
use
that
to
scale
up
and
down,
we
use
that
for
draining.
We
use
that,
for
being
able
to,
you
know,
take
different
runners
out
of
service
that
we
identify
as
problematic,
and
so
we
can
very
easily
cycle
in
and
out
runners.
So
we
can
make
sure
that
these
individual
running
these
individual
instances
at
a
running
code
aren't
necessarily
staying
in
service
for
very
long.
So
doing,
updates
on
them
is
very
easy.
B
We
can
just
take
them
right
out
of
service
update
and
put
them
back
in
the
service.
It's
not
a
problem,
let's
see
what
else
do
we
have
here?
The
runner
interface
is
just
for
executing
individual
calls
on
the
runner,
and
then
the
runner
call
interface
provides
access
to
the
like
all
the
individual
details
about
the
calls
there
on
the
actual
runners
themselves.
So
if
you
implement
these,
you
can
implement
a
fin
on
whatever
infrastructure
as
a
service
provider.
You
want
and
I'm
back
to
Chad
I
forgot.
A
Differentiation
is,
you
know,
highly
optimized
for
Oracle
cloud
infrastructure,
we're
gonna
on
bare
metal,
running
on
GPUs
integration
with
other
services
that
we
have
and
that
we're
building
on
top
of
Oracle
cloud.
So
we
believe
that
the
differentiation
is
not
necessarily
just
the
just
in
the
FN
code,
because
we
want
other
people
to
go
on
launch
services
on
that
as
well
and
offer
their
own
custom
services.
A
So
node
pool
manager
is
really
cool
that
we're
working
on,
which
is
how
we
manage
all
of
our
infrastructure
behind
the
scenes,
so
sort
of,
like
exponentially
weighted
algorithms,
to
ensure
that
we
have
the
right
amount
of
infrastructure
and
that
we're
placing
runners
or
function
requests
on
the
right
runners.
So
it's
really
cool
stuff.
So
look
out
for
from
some
stuff
there
I'm
the
demo
I'm
going
to
show
is,
let's
see
so
it's
a
little
bit
of
a
spin
on
the
cloud
events
demo
that
some
of
you
probably
saw
Austin
show
at
11:00
a.m.
A
this
morning,
which
was
really
awesome.
We've
been
working
really
hard
to
have
this
multi
cloud.
Multi-Vendor
cloud
event
specification,
be
able
to
pass
messages
from
storage
engines
on
multiple
riders
to
lots
of
endpoints,
which
were
implemented
by
different
functions
frameworks.
So
I
took
sort
of
the
same
stuff
we
built
and
I
actually
took
the
same
image
processes
and
that
we
did
which
we
we
don't
have
a
really
cool,
AI
service
like
say
Google
or
Microsoft.
Yet
so
we
had
to
build
some
tensor
flow
models
that
took
a
few
days
to
build.
A
We
could
recognize
about
75
objects,
so
of
course
we
had
some
really
good
hits
and
some
really
terrible
ones,
but
that's
what
you
get
when
you
limit
the
objects.
So
what
I?
What
I'm
gonna
do?
Is
it
probably
take
too
long
to
actually
deploy
it,
but
we
can
try.
So
if
I
do-
or
let's
do
an
echo
so
FN
API
URL
is
where
my
machine
is
actually
talking.
A
A
Let's
see
where
is
it
multi
Club
trigger
so
I'm
gonna
pass
it
this
payload
here
this
payload
is
taking
two
endpoints
and
excuse
me
in
opening
up
some
go
channels
and
firing
off
a
couple.
Images
I
think
three,
each
to
each
of
those
endpoints.
So
we're
doing
Google
and
an
Oracle
cloud
and
I'm
gonna
use
a
trigger
binary
that
I've
already
built
from
go
and
if
I
have
this
open
properly.
Let's
just
do
it
a
couple
times
here,
then
theory.
A
If
we
have
a
connection
which
I,
let
me
know
if
we
do
yep
there,
we
go
cool.
So
we
start
seeing
images
come
into
slack.
You
can
actually
see
where
it
ran.
So
this
one
ran.
It's
a
person
looks
like
a
Bernie
man
from
a
photo
that
Matt
put
in
there.
It
was
ran
on
Oracle
cloud
and
it
identified
some
people.
We
got
a
hot
dog.
Yes,
it's
not
not
a
hot
dog.
We
have
some
cars
AFM
projects,
so
this
ran
on
Oracle
cloud
and
then
we
have
people
or
person
when
it
recognizes
a
person.
A
A
So
we're
really
excited
about
that
body
of
work
and
really
excited
to
demo
stuff
like
this
and
Austin's
demo
was
really
impressed
with
this
morning.
If
you
haven't
seen
it
make
sure
to
watch
the
video
okay,
so
final
slides
here,
we
are
doing
a
lot
with
kubernetes
at
oracle.
We've
been
talking
about,
we've
been
vlogging
about
it.
Oracle
kubernetes
engine
is
something
that
we've
also
announced,
which
is
a
community
service
based
on
on
top
of
OCI.
A
Today
we're
in
limited
availability
and
will
be
announcing
that
nga
at
some
point
in
the
future,
which
is
the
official
answer,
we're
also
doing
oracle.
My
sequel
operator,
this
is
really
cool.
Basically,
it
helps
you
provision,
D
provision
and
manage
upgrade
and
backup
and
restore,
which
is
like
a
key
point
of
all
of
your
my
sequel
clusters
on
top
of
kubernetes.
So
we've
got
a
bunch
of
work
going
into
the
operator.
A
I
think
we
also
have
some
other
operators
for
like
WebLogic
server
and
possibly
Oracle
database,
but
I,
don't
know
for
sure
on
that
I'm
sure
we'll
do
something
there.
We
also
have
an
out
of
tree
cloud
provider
for
Oracle
cloud
on
top
of
communities,
so
we've
got
all
the
providers
there.
We
have
built
a
bunch
of
flex
volume
drivers
to
ensure
that
we
can
load
storage
on
on
OCI
Oracle
cloud
and
then
terraform
installers
to
install
kubernetes
our
for
kubernetes
on
top
of
Oracle
cloud.
A
Finally,
if
you
want
to
get
involved
with
the
project,
we'd
love
to
have
you
we're
we're
in
slacked
Atif
in
project
at
I/o,
you
can
go
to
effin
project
at
I/o,
we're
really
looking
for
people
to
to
join
use
it
as
users
used
as
providers
we'd
love
to
see
some
work
in
the
provider.
So
we'd
love
to
see
this
launched
on
other
clouds.
We're
talking
to
other
providers
today
to
ensure
that
you
know,
there's,
there's
lots
of
FN
services
out
there
and,
of
course,
we're
hiring.