►
From YouTube: Running Serverless HPC Workloads on Top of Kubernetes and Jupyter Notebooks - Christopher Woods
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Running Serverless HPC Workloads on Top of Kubernetes and Jupyter Notebooks - Christopher Woods, University of Bristol
Jupyter notebooks, deployed on top of k8s, are finding rapid adoption in universities and industry. While k8s can spawn new pods for each notebook session, launching high performance computing (HPC) jobs during dynamic workflows is complicated. Fn is a serverless solution that sits on top of k8s. Individual simulations are deployed as Fn functions that are called dynamically from workflows run within Jupyter notebooks. A notebook running on a lightweight k8s cluster can burst HPC workloads via Fn serverless calls to another cluster running on a bare metal HPC/GPU cloud. Using Fn, we have constructed a distributed access, authentication and accounting layer around a shared object store. This adds security and accountability, thereby making it easy for end users to manage complex multi-cloud workflows involving lots of different types of hardware from within a easy-to-use Jupyter notebook.
To learn more: https://sched.co/GraL
A
There's
a
show
of
hands
who
here
watched
Kelsey's
keynote
talk
this
morning,
because
I
thought
that
was
just
absolutely
fantastic,
not
just
because
I'm
a
Fortran
programmer,
because
the
who
else
here
programs
in
Fortran
yay
you
do
exist
still.
This
is
good
20
years
ago,
that's
still
quite
old
Fortran.
That
was
like
at
least
for
10
95,
probably
Fortran
90,
not
bad.
A
I'm
gonna
try
and
be
brave
enough
and
run
an
online
demo
as
well,
which
you
gone,
yeah
I'm,
an
academic.
We
don't
do
rockstar
music.
We
do
expect
lots
of
maths
equations
and
all
of
this
stuff
leave
now
run
in
terror.
I'm
joking
I
stripped.
All
of
that
out.
We
have
a
nice
fun
I've
put
code
in
there,
so
it's
actually
a
weird
conference
for
me.
So
I
normally
go
to
academic
conferences,
and
so,
if
I
put
code
into
a
slide
in
an
academic
conference,
the
researchers
run
away
screaming
so
I.
Basically
swapped.
A
A
A
Think
a
lot
of
us
were
down
in
the
gardens
last
night,
so
I
managed
to
go
down
to
the
gardens
and
ammo
pop
and
I
was
so
thankful
and
my
wife's
so
jealous
that
there
was
the
Marvel
exhibition
that
was
on
so
like
so
many
photos
of
the
Iron
Man
costumes
I
can
see
a
few
people
thinking
about
still
coming
in.
So
again.
A
If
this
was
like
one
of
my
like
teaching
talks
or
one
of
my
workshops,
we
always
give
people
five
extra
minutes
and
I'm
trying
to
teach
students
about
a
concept
of
timekeeping
and
the
way
I
teach.
My
students
about
the
concept
of
timekeeping
is
with
social
peer
pressure,
and
so
what
we
do
is
basically
once
the
time
is
started.
We
then,
basically
all
turn
around
and
look
at
the
people
and
stare
at
them
as
they
come
into
the
door
and
they
begin
to
feel
like
socially
awkward
and
I'm
hoping
I'm
doing
you.
A
A
favor
and
they'll
have
learned
this
before
they
come
and
work
in
business,
but
typically,
as
students
I
come
with
it's
like
if
they
actually
come
in
earlier
than
ten
minutes
before
the
start,
I'm
doing
well.
So
it's
the
only
way
I
can
sort
of
try
and
fix
this
now
we're
at
the
time.
So,
if
you
do
want
to
now
stare
at
people
in
the
door,
that'll
be
fine.
I
will
I
will
not
mind.
A
And
eat
since
we
have
hit
the
time
and
I'd
like
to
see
anybody
else,
walking
I'm
now,
gonna
actually
start
so.
I
am
Christopher,
I
am
an
academic
and,
as
I
said,
I
really
enjoyed
Kelsey's
talk
this
morning
and
it
wasn't
just
because
of
the
Fortran.
It
was
because
what
he
said
was
really
true
and
that's
that
Tech
is
brilliant,
but
actually
tech
is
not
brilliant.
A
Just
as
an
end
in
all
of
itself,
Tech
is
brilliant
because
it
helps
us
solve
the
big
problems
and,
as
I
said,
I'm
a
scientist
I'm
a
research
software
engineer,
so
big
problems
are
kind
of
most
of
my
career
and
the
really
big
problems
at
the
moment
are
happening
in
the
life
sciences,
so
I
work
in
the
life
sciences
and
what
you
see
up
on
these
screens.
These
are
the
molecules
of
life.
These
are
the
proteins,
particularly
these
two
HIV
protease
and
HIV
reverse
transcriptase.
A
These
are
the
proteins
which
HIV
uses
to
take
over
your
cells,
like
the
Borg,
make
the
your
cells
replicate
more
HIV
and
basically
kill
you,
but
now
science
has
advanced
so
much
that
we
could
understand
how
these
proteins
work
at
the
molecular
level.
We
can
actually
understand
biology.
We
can
understand
disease
at
the
molecular
level.
That's
amazing
what
it
means
we
can
do
is
we
can
design
molecules
like
the
one
on
the
far
right
that
huge
molecule?
A
There
is
a
designed
molecule
that
we
assemble
and
that
molecule
contains
antigens
for
lots
of
diseases
with
the
plan
that
you
can
actually
inject
that
molecule
into
people,
and
it
will
be
like
a
multi
target
vaccine
doing
this
kind
of
stuff.
Solving
the
big
problems
is
why
I
hope
we're
all
here
to
do
and
it's
this
is
the
end
goal.
This
is
what
we
want
to
do
now,
while
our
understanding
of
science
and
research
has
just
advanced
like
anything
over
the
last
50
plus
years.
A
The
way
that
we
use
computers
to
conduct
scientific
experiments
has
barely
changed.
Actually,
since
the
moon
landings
and
indeed
thinking
about
hidden
figures,
they
would
actually
recognize
current
modern,
high-performance
computing
because
it
looks
kind
of
the
same.
You
start
off
with
a
terminal.
Typically
now
on
a
laptop,
they
didn't
have
laptops
back.
Then
you
describe
your
experiment
on
the
terminal
in
text.
You
build
all
of
the
input
data
and
then
you
transfer
it
across
to
your
big
mainframe
supercomputer.
It's
change
from
being
a
big
IBM.
A
All
of
the
data
that's
been
produced
back
to
your
little
laptop
and
you'll
do
more
analysis
in
text.
All
those
pretty
pictures,
I
just
showed.
First,
we
don't
look
at
those
pretty
pictures,
they're
just
for
papers
and
talks.
We
do
everything
in
text
now,
that's
terrible!
If
you
think
about
all
the
advance
we
have
in
tech,
we're
still
basically
running
on
a
1970s
command-line
interface
to
run
the
biggest
most
important
simulations
that
we
actually
do.
A
What
I
would
like
to
do
is
actually
do
science
differently
and
hopefully,
if
the
Wi-Fi
works,
I
will
show
you
what
I
mean.
So
while
this
is
coming
up,
who
here
has
heard
of
Jupiter?
Yes
again
scientific
conference,
saying
Jupiter?
Sometimes
you
don't
get
anything,
so
you
know
that
this
is
a
Jupiter
notebook.
Jupiter
is
fantastic,
it's
running
in
the
cloud,
and
so
what
we're
going
to
do
here
is
rather
than
beer
to
a
boring
text
interface.
A
Instead,
we're
just
going
to
basically
execute
command
cells,
and
these
cells
are
loading
up
the
molecules
that
we
want
to
simulate.
So
we've
now
loaded
the
molecules
we
have
here-
631
of
them
and
now
we're
going
to
define
the
protocol
with
which
we
actually
want
to
run
the
simulation,
and
now
this
line
here
now
we've
loaded
up
the
molecules
we
want
to
study.
We
know
the
protocol
of
the
simulation
we
want
to
run.
We
execute
that
cell.
It's
now
gone
and
run
the
simulation.
It's
bursts
it
out.
A
It's
going,
it's
actually
running
a
Fortran
program
woohoo
and
now
that
simulation
is
running,
can
I
ask
easy
actually
running.
Yes,
how
long
has
it
been
running
for
it's
been
running
for
now?
Oh
point
two,
eight
minutes,
but
it's
live
because
it's
live.
I
can
inspect
this
simulation
as
it's
going
so
I
can
do
things
like
ask.
What's
the
energy
of
this
molecular
system
as
we're
going
now?
Well,
it's
currently
6000
K
Cal's
per
mole,
but
now,
what's
the
energy,
it's
simulating
it's
moving.
A
These
molecules
around
they're
changing
all
the
time
and
we
can
then
ask
more
than
just
the
energy
we
can
say
how
much
time
have
you
simulated
just
now
down
52
Pico
seconds
of
simulation
of
this
particular
molecule.
It's
currently
at
a
temperature
of
28
Kelvin,
the
poor
thing
it's
very
cold
but
as
the
simulation
progresses
is
now
heating
up
to
37
Kelvin
we're
interacting
with
a
live
simulation.
That's
running
on
the
cloud
through
an
interface
that
researchers
actually
can
use,
they're
not
going
to
do
keeping
ET
stuff.
A
A
So,
as
I
rerun,
this
cell,
it's
going
to
pick
up
the
latest
data
from
the
simulation
and
add
that
at
the
end,
two
dimensions
is
good,
but
actually
three
dimensions
is
better,
so
now
just
go
and
get
the
molecules
so
I've
done
now
is
I've
got
got
the
molecules
getting
the
molecules
from
Germany,
because
this
is
where
it's
running
come
on
molecules.
You
want
to
come
into
my
notebook,
they're
trudging
their
way
across
the
internet
as
we
speak.
This
is
where
my
Wi-Fi
kills
me
camera
molecules.
A
Now
the
data
that's
being
produced
on
that
simulation
is
sitting
on
the
cloud,
as
you
just
saw.
I've
just
brought
my
visualization
to
my
laptop,
which
took
a
bit
of
time,
but
the
data's
on
the
cloud
I
should
have
the
computer
running
on
the
cloud
as
well.
Next
to
the
data,
and
that's
what
this
is
doing.
A
The
data
has
been
put
on
the
cloud
is
a
trajectory
of
this
molecule
as
it
walls
through
time,
there's
been
seven
frames
of
it
and
what
I
can
do
here
is
I'm,
basically
getting
a
handle
to
those
frames
and
then
asking
it
to
go
and
run
an
analysis
on
the
cloud
of
a
typical
type
of
analysis
that
you
would
do
if
you're
doing
these
kind
of
molecular
simulations
again.
This
is
what
I
call
service,
as
a
researcher
I
haven't
had
to
think
about
where
everything's
been
happening.
I'm
interacting
in
the
space
of
the
researcher.
A
Thinking
about
these
direct,
these
particular
simulations,
so
I
just
play
from
current
slide.
Hopefully
this
will
come
back
up
magic
of
the
laptop
interfacing
laptops
and
displays
is
always
the
hardest
thing
ever.
If
someone
can
solve
this
problem,
we'll
be
very
happy,
so
that
was
the
demo
and
basically
Jupiter
notebooks
they're,
fantastic
they're,
taking
over
I
would
say:
life
sciences,
Medical,
Sciences,
even
physics,
chemistry,
machine
learning,
data,
science
etc,
and
it's
because
in
one
web-based
notebook
you
describe
the
experiment.
You
have
the
code
to
run
the
experiment.
You
have
all
of
the
code
to
analyze.
A
It
you
have
all
the
2d
visualization
of
the
graphs,
the
3d
visualization
of
the
molecules
they
contain
everything
needed
to
publish
and
reproduce
that
particular
computational
experiment,
and,
what's
more
because
all
of
the
data
is
sitting
with
the
notebook
and
all
the
compute
is
happening
with
the
notebook.
It
means
we
don't
have
to
begin
moving
data
backwards
and
forwards
around
the
planet.
It
effectively
means
everything
is
happening
together
and
when
I
render
something.
The
only
thing
that
needs
to
come
back
to
my
laptop
is
the
renderable.
A
So
this
is
actually
just
a
video
of
me,
rendering
a
massive
molecule
to
just
show
you
it
working
between
my
home,
Wi-Fi
and
Bristol,
and
that
was
a
server
that
was
happening
to
be
in
the
u.s.
and
it's
completely
fine.
Now
notebooks
are
useful
because
they
said
they're
useful
because
they
can
be
published
effectively.
What
we
have
here
is
an
open
reproducible,
executable
scientific
paper,
but
they're
only
useful
if
we
can
actually
put
them
on
the
web
and
have
people
use
them
if
there's
some
compute
sitting
behind
them.
So
how
do
we
do
the
compute?
A
A
It
runs
on
top
of
all
of
the
certified
kubernetes
clusters,
so
I've
run
out
on
top
of
Microsoft
Azure,
Google,
GCP
or
glow
CI
etcetera,
and
it
is
great,
but
there
was
one
major
problem
when
we're
trying
to
run
these
kind
of
workloads
on
top
of
kubernetes,
and
that's
that
our
communities
cluster
is
basically
provisioned
to
run
the
notebook.
So
it
has
lots
of
small
VMs
that
are
ready
to
serve
little
python
kernels,
which
are
serving
each
notebook,
but
I
want
to
do
things
like
run
massive
simulations.
A
Effectively.
This
line
here,
which
was
the
MD
dog
run
that
is
not
bursting
it
within
the
communities
cluster.
Well,
that's
doing
is
calling
in
HTTP
endpoint,
which
is
then
creating
the
resource
for
a
large
HPC
cluster,
transmitting
the
data
to
that
cluster
and
then
running
it
on
that
resource.
So
we've
burst
out
of
communities
into
a
separate
resource
here,
and
we
have
to
do
this
as
I
said,
because
you
can't
really
do
that
kind
of
bursting
within
a
while
within
by
starting
a
new
cube
in
ities
pod.
A
Now
FN
might
be
new
to
some
people,
so
I'm
gonna
have
a
quick
overview
of
where,
if
n
is,
if
n
is
an
open
source,
event-driven
functions
as
a
service
platform,
it's
completely
open
source,
I've
run
it
on
most
of
the
clouds.
It's
a
fantastic
thing
and
the
unit
of
abstraction
is
the
docker
container.
So
actually,
when
Kelsey
was
talking
this
morning
about
I
want
to
docker
container
half
standard
in
run
a
Fortran
program
right,
standard
output,
that
is
FN
and
that's
what
we
do
live
all
the
time
now
so
effectively.
A
What
you
do
is
you
package
up
everything
you
need
into
a
docker
container.
The
HTTP
request
comes
in
and
is
interpreted
in.
The
container
is
standard
in
you
do
whatever
you
want
with
it
with
the
standard
in
running
whatever
program,
whatever
language,
whatever
legacy
apps,
you
have
a
processing
standard.
In
anything,
that's
written
to
standard
output
is
what's
returned
by
the
function
over
HTTP
and
anything
you
write
on
standard
error
goes
to
log.
A
It's
really
simple,
so
just
to
remind
you
any
code
and
all
of
the
Associated
programs
and
all
of
your
libraries
and
all
your
installation.
The
whole
lot
goes
in
the
container.
Http
request
goes
in
as
standard
in
its
run
in
the
container
and
interpreted
normally
standard
out
gives
you
the
output
standard
error
goes
to
logs.
Now.
The
nice
thing
is
is
when
you
call
these
HTTP
endpoints,
they
can
either
be
synchronous,
functions
or
asynchronous
functions.
So
I
call
a
synchronous
function.
A
It
will
run
immediately
and
will
you'll
block
until
you
get
the
return
value
with
an
asynchronous
function.
It
runs
immediately,
but
immediately
returns
a
call
ID,
so
that
then
you
can
basically
just
leave
it
running
because
we're
talking
about
fun
here
there
are
simulations.
These
could
be
functions
which
are
running
for
a
week.
It
is
not
your
5
millisecond
function.
A
Now,
as
I
said,
it
supports
any
language
because
it's
whatever
you
want
to
put
in
the
docker
container,
that
can
read
standard
input,
which
is
like
everything,
but
there
are
function,
development
kits.
They
can
make
it
a
bit
easier
to
do
all
of
the
serializing
to
and
from
Jason.
If
you
want
to
be
more
standard,
and
so
I
tend
to
use
the
Python
development
kit,
just
called
the
FN
development
kit
ftk-
and
this
is
actually
the
hello
world
example-
fair
fan
so
effectively.
What
will
come
in
is
a
post
request.
My
posting
name
is
Chris.
A
Fn
is
basically
serialized
that
into
a
JSON
string,
name
is
Chris,
and
then
in
here
and
I
apologize,
it
is
small.
I
thought
the
screen
would
be
bigger
effectively
what's
happening,
is
it's
just
calling
a
function,
there's
taking
the
JSON
out
and
then
just
printing
out
another
JSON
string,
which
is
hello
Chris,
which
then
comes
out
as
the
output?
A
Now
this
is
a
toy
example
and
I
think
in
all
the
talks,
I
kind
of
see
when
I
talk
about
when
people
talk
about
service,
they
always
talk
about
two
examples:
uploading
things
to
Twitter
or
doing
some
basic
things
we
do
more
than
that
so
effectively,
they're
MDR
on
which
is
running
the
simulation.
That's
going
to
do
an
async
function,
call
across
to
a
docker
container,
which
is
then
started
on
a
big
HPC
node,
and
that
docker
container
contains
the
grow
max
application,
which
is
essentially
a
big
C++
application
for
doing
dynamics.
A
It
contains
a
small
Python
script
which
basically
takes
in
the
Jason
parses.
It
then
starts
the
grey
max
process,
and
then
it
just
runs
as
an
async
call.
Now.
Obviously,
at
this
point,
you're
thinking
crumbs,
you're,
doing
a
function
call
here
when
you're
passing
in
the
arguments
of
this
function
call
distributed
are
all
of
the
molecules
of
the
system
and
the
protocol.
This
could
be
a
really
big
argument
list
like
megabytes
or
gigabytes
of
data
here
and
obviously
that
would
be
really
hard
to
serialize
to
Jason.
A
I've
had
people
who
want
to
serialize
it
to
Jason,
but
that's
a
really
bad
idea.
So,
instead,
what
you
do
is
the
obvious
thing
and
that's
you
copy
the
data
to
an
object,
store
put
it
in
a
bucket
and
when
you
do
the
function,
call
you
just
basically
pass
in
the
bucket
in
which
the
data
is
got
and
the
data
is
copied,
then
into
the
container,
which
is
running
the
function.
But
what
about
the
output?
A
What
we
have
running
in
the
background
in
the
container
is
a
little
spite
and
server
about
200
lines
and
that's
using
watchdog
and
every
time
there's
a
bit
of
file,
that's
being
written
by
the
program
that
bits
of
file
is
streamed,
live
back
into
the
output
part
of
the
bucket
of
the
object
store.
This
means
that
live
my
notebook
can
basically
query
the
objects,
don't
get
the
data
effectively.
This
is
your
POSIX
to
object.
Interface
simulations
run
in
POSIX.
Object
is
used
for
distribution
of
data
around
the
planet,
so
this
is
great.
We
have.
A
You
can
have
different
molecular
dynamics,
programs
behind
different
functions.
We
can
use
different
functions
to
refer
to
different
bits
of
hardware,
so
I
can
call
a
GPU
function
or
a
CPU
function.
I
can
have
many
different
types
of
scientific
application
behind
this,
and
because
the
f-m
framework
is
open,
source
and
cross-platform,
I
can
deploy
these
on
every
single
cloud.
Life
is
good
and
this
will
basically
work
with
any
application,
and
all
of
the
data
is
live,
going
into
object,
store,
which
means
already
in
the
output
is
globally
accessible.
A
I
can
give
a
URL
or
a
DOI
that
particular
bucket
and
immediately
I
can
publish
my
results.
This
is
fantastic.
I
have
a
great
framework
that
allows
anyone
anywhere
in
the
world
to
run
HPC
simulations
just
by
calling
an
HTTP
endpoint.
Oh
dear,
anyone
can
run
simulations
by
calling
one
of
my
HTTP
endpoints
I'm,
going
to
be
poor,
because
if
I
publish
the
endpoint
scientists
are
quite
cunning,
they
will
find
the
endpoint
and
they
will
start
throwing
simulations
at
it
and
get
lots
of
free
compute.
A
So,
essentially,
we
need
a
way
of
not
just
making
this
something
which
is
kind
of
like
an
academic
toy.
There
has
to
be
a
business
model
around
it,
which
means
you
need
to
have
Identity
Management
authorization
and
accounting,
and
that's
exactly
what
we
built.
We
built
on.
Identity
Management
it
kept
multi
cloud
accounting
and
access
management
service
discovery
system,
there's
actually
built
on
top
of
FN
as
well
so
effectively.
Here.
I
hope
this
is
readable.
A
So
it
means
that
then
you
call
the
function
to
do.
Login
you
put
in
your
password
and
username
and
the
one
time
code
and
it's
all
processed
as
a
single
FN
function.
So
it's
going
to
run
that
through.
So
you
can
see
here.
I've
put
the
logging
in
requested
the
login.
It's
giving
me
back
the
QR
code,
then
we
click
on
to
the
page
to
get
the
login
page
and
then
I
put
in
my
username
password
and
because
this
was
a
device's
already
recognized.
I
didn't
need
to
use
Google
Authenticator
to
login
and
voila.
A
It's
now
logged
in
and
your
scripts
can
be
left
pause,
while
it's
waiting
for
the
login
to
happen
now.
This
is
great
because
it
means
that
it's
completely
distributed.
I,
don't
need
to
have
a
login
server
running
all
of
the
time
and
as
many
people
who
want
to
log
in
it
will
just
fire
up
more
and
more
functions
as
necessary
to
handle
the
load.
A
Now,
while
that
all
works
for
login,
we
need
to
do
the
same
thing
for
other
services.
So
what
we
actually
have
is
step.
One
you've
got
login,
which
is
run
through
an
identity
service.
You
then
have
an
access
service
which
is
used
for
service
discovery
to
say,
I've
logged
in
now
tell
me
what
I
have
access
to.
Then
we
have
an
accounting
service
which
is
basically
full
double
entry
bookkeeping
and
we
work
with
the
cloud
companies
to
work
out
how
we're
going
to
be
actually
doing
accounting
across
clouds.
A
So
I've
draw
one
of
them,
but
we
can
have
ten
identity
services
on
ten
different
clouds.
You
know
ten
night
counting
services
are
we
building
a
completely
distributed
system
for
logging
onto
cloud
running
simulations
accounting
and
doing
all
of
that
work?
And
it's
all
because
serverless
is
enabling
us
not
to
run
resource
permanently
effectively
if
no
one's
logging
in
there's
nothing
running,
but
as
soon
as
people
start
logging
in
it
just
automatically
scales
and
a
distributed
manner.
To
handle
everything.
Serverless
is
brilliant.
A
A
We
always
go
through
I
mean
I'm
old
enough
they've
been
through
so
many
tech
cycles
going
backwards
and
forwards
edge
cloud
edge
cloud.
How
many
times
are
we
going
to
keep
doing
that
server
lace,
managed
services
service
just
all
the
time
and
of
course,
service
has
many
issues.
The
biggest
issue
that
I
hate,
the
most
is
cold
starts,
who
here
hates
cold
start
if
they
use
service
platforms?
Okay,
what
share
pads,
whose
boy
is
an
acceptable
cold
start.
A
So
you
willing
to
wait
10
seconds
for
your
function
to
start
so
hands
up
if
you're
willing
to
wait
10
seconds
for
your
function
to
start
okay,
you're
quite
good
five
seconds,
one
second
you're
way
too
tolerant,
I
want
milliseconds.
So
basically
this
is
terrible.
This
is
me
running
a
function
twice,
the
first
time
it
ran.
It
basically
came
up
in
four
seconds,
the
next
time
it
ran.
It
came
up
in
nine
milliseconds,
so
in
90,
milliseconds
now
I
code,
HPC
machines,
I'm
writing
code,
which
is
basically
vectorized.
A
Paralyze
gets
every
last
track
of
performance
out
of
these
wonderful
things
that
Intel
and
AMD
and
others
have
created,
and
in
four
seconds
I
could
do
400
billion
floating-point
calculations.
I
can
simulate
pika
seconds
of
dynamics.
I
can
begin
to
I
dunno
end
world
hunger
in
four
seconds,
with
some
simulations,
and
you
want
me
to
wait
that
time
for
one
function,
to
start
I
mean
for
crying
I
think
even
functions
started
quicker
back
in
the
1970s
on
the
original
mainframes.
This
is
not
progress
and,
of
course,
you'll
say
to
me:
yes,
yes,
Chris.
A
Yes,
it
takes
a
long
time
because
the
container
has
to
be
allocated.
You
need
to
start
your
runtime
in
the
function.
You
need
to
import
your
modules.
You
must
run
the
script.
These
are
stateless,
so
we
need
to
reload
and
unload
the
state.
Bla
bla
bla
bla,
but
don't
worry
because
we'll
keep
it
hanging
around
so
once
you've
done
a
cold
start
once
you
have
a
little
bit
of
time
afterwards
and
the
function
will
be
hot
and
you
won't
pay
it
again.
Hence
why
first
time
four
seconds
next
time,
90
milliseconds.
A
Now,
while
you
all
think
that
that
means
it's
great
we've
solved
the
problem
because
we
keep
it
hot.
It's
not
people
are
paying
cold
starts.
You've
got
to
eliminate
cold
start
because
I'm
calling
functions
that
call
functions
that
call
functions
that
call
functions
if
each
of
them
has
a
one.
Second
cold
start:
that's
10
seconds,
potentially
that
I'm
waiting
spinning
my
wheels
doing
nothing.
A
So
the
first
thing
I
do
to
get
rid
of
cold
start.
Is:
let's
not
have
a
cold
staff
at
every
single
function
in
my
application.
So
what
we
do
is
we
bundle
all
of
the
functions
together
into
one
container.
This
means
that
the
actual
function
I
want
to
call
is
effectively
an
argument,
that's
being
called
so
here.
A
This
is
a
really
stupid
example,
but
effectively
here
there
are
two
functions,
hello
and
goodbye,
and
so
when
I
call
the
function,
I
pass
in
the
argument,
which
is
this
case
Chris,
but
we
also
pass
in
the
function
name,
and
so,
if
we're
asking
for
function,
hello,
it
caused
hello.
If
we
ask
a
function
goodbye,
it
caused
goodbye
very
simple:
these
is
the
bundle
we
use
for
their
entity
service
and
effectively.
What
goes
in
here
is
the
JSON
request,
saying
I
want
to
call
the
function
request
login.
A
Now
the
beauty
of
bundling
all
the
functions
together
into
one
container
is
that
once
that
container
goes
hot,
all
of
the
functions
are
hot,
so
any
pay
that
cold
start
time
once
for
the
whole
bundle
of
functions
equally,
that
container,
once
it's
being
pulled
on
to
the
node
all
of
the
functions
and
on
the
node
I'm
not
going
to
be
pulling
a
hundred
containers.
My
hundred
functions,
because
if
you
do
service
properly
you'll
end
up
with
large
numbers
of
functions,
so
bundling
them
all
together
into
single
containers.
A
Small
containers
enables
you
to
basically
get
rid
of
some
of
that
startup
time
through
working
now.
The
other
thing
it
does
as
well
as
enabling
one
goes
hot.
They
all
go
hot
plus
ones,
on
the
node
they're
all
on
the
node
is
we
say
the
functions
are
stateless,
but
actually
there's
a
lot
of
state
which
is
really
useful
to
cache
between
calls
state
which
we
know
does
not
change
between
those
calls.
A
A
Equally,
there's
data
in
the
object
store,
which
is
read-only,
which
is
cacheable
data,
so
once
I've
read
those
keys,
I,
don't
have
to
read
that
again.
I'll
just
hold
that
within
the
actual
function
and
then
because
we're
using
asynchronous
Python
the
actual
functions
of
n
blue
because
in
a
sync,
is
an
asynchronous
event
loop.
This
means
that
that
one
event
loop
can
actually
serve
multiple
functions
at
once.
So,
rather
than
have
one
event
loop
per
function,
we
now
have
one
event
loop
for
the
whole
bundle
of
functions.
A
Now.
All
this
is
done
is
basically
reduce.
The
number
of
cold
starts
we're
going
to
have
and
reduce
the
amount
of
reloading
things
that
we
know
we're
not
going
to
change.
It
hasn't
actually
got
rid
of
cold
start
to
get
rid
of
cold
start.
You
really
need
to
profile
your
applications,
so
you
need
to
say:
where
is
the
time
being
lost
and
remember?
I,
like
milliseconds
I'm,
not
happy
with
seconds
so
step
one.
A
But
of
course
you
don't
know
what
they
are
until
you
profile
them
so
again
profile
your
application
profile,
your
import.
So,
for
example,
here
profiling,
a
Python
is
really
really
easy.
You
just
set
an
environment
variable
and
then
it
will
profile
all
your
inputs
write
an
output
file
that
can
be
processed
by
tuna,
and
in
these
two
lines
you
get
a
beautiful
icicle
plot.
That
tells
you
that
this
module
we've
imported
here.
Oci
apologies
Oracle,
that
module
took
1.4
seconds
to
import
I'm.
Sorry,
that's
shocking!
A
A
Now
the
reason
that
took
1.4
seconds
to
import
was
because
it
was
importing
everything
that
could
ever
possibly
be
used
by
that
module
in
the
whole
of
time
and,
of
course,
I'm
running
a
function.
I
only
need
certain
functionality
out
of
it.
So
the
way
you
solve
it
is
you
actually
use.
Try
import
so
try.
Import
is
basically
a
little
mechanism
in
Python
where,
instead
of
importing
something
it
will
import
a
handle
to
it,
and
only
when
you
use
it
will
it
actually
import
it,
and
so
I
went
into
the
init
dot.
A
Pi
of
OCI
and
I
changed
every
single
one
of
their
imports
into
trying
ports,
and
that
was
the
extent
this
little
black
light.
You
see
ten
lines
of
code
here
that
was
all
I
did
that
took
about
profile.
Five
minutes
changed
those
lines
of
code,
another
five
minutes,
and
you
probably
can't
see
the
numbers
here,
but
that
changed
the
OCA
import
from
taking
1.4
seconds
to
37
milliseconds
20
minutes
of
my
time
to
stop
1.4
seconds
of
pain
for
every
single
person
in
loads
this
as
developers
and
engineers.
A
A
Because
if
one
of
them
fails,
I
don't
want
to
pay
the
cold
start
time
of
phoning
up
the
supplier
and
waiting
them
to
deliver
me
a
brand
new
compute
power
supply
and
then
installing
it
I
want
to
have
another
second
backup,
one
hot
spare
ready
to
take
over
immediately
and
I.
Think
in
the
function's
world.
We
cling
on
to
this
idea
that
we're
scaling
to
zero
and
you
shouldn't
be
paying
anything
when
it's
not
running
and
what
I'm
telling
you
is.
Actually
you
should
scale
to
one.
A
You
should
always
have
one
function
running
all
of
the
time
and
then,
when
that
function
is
being
called,
that
function
should
spin
up
one
horsepower,
and
so
here
you
can
see
in
this
when
I'm
calling
request,
login,
there's
actually
an
async
function,
call
going
at
the
same
time
across
to
warm
and
what
that
is
doing
is
basically
spinning
up
a
hot
spare
so
that
that's
then
ready
in
case
more
people
come
in
at
this
point
now.
All
of
this
is
not
service.
A
Manifesto
I've
bundled
independent
functions
into
one
block,
I'm,
spinning
up
things
and
having
them
run
all
of
the
time
I'm
paying
for
stuff
that
people
aren't
using
because
it's
running
idle.
But
let's
remember
here:
these
are
things
running
effectively
on
one
core
VMs,
particularly
for
the
identity
service.
Those
one
core,
VMs
cost
me
approximately
1
cent
power
is
the
principle
of
manna.
Food
Service
manifesto,
really
worth
saving
25
cents
per
day
about
a
dollar
something
a
week,
forcing
people
to
have
told
start
times.
It's
not
practicality,
always
beats.
A
Pragmatism,
always
beats
idealism.
I
think
we
need
to
move
away
from
the
cost
benefits
of
server
lists
so
instead
say
actually,
let's
build
systems
which
are
distributed
and
where
people
don't
pay
cold-start.
We
engineer
them
so
they're
always
ready
to
help
people
now
effectively.
What
we've
got
here
and
the
way
I
like
to
think
about
this
is
we're
building
now
systems
where
simulations
and
data
and
logons
and
notebooks
and
our
happening
all
over
the
world
and
as
a
scientist
as
a
researcher
I,
don't
have
to
think
about
where
all
of
that's
happening.
A
It's
a
bit
like
effectively
we're
programming.
What
I
call
the
planetary
supercomputer?
So,
if
you
think,
when
you
program
a
normal
computer,
you've
got
a
console,
GUI
and
you've
got
memory,
storage
and
you've
got
processes,
and
things
like
that.
If
we
think
of
all
of
the
clouds
together
as
one
big
planetary
supercomputer,
then
your
object
store
is
effectively
disk
in
memory.
Those
functions
that
we're
running
are
effectively
processes.
A
They
call
it
the
Netflix
of
simulation
because
they
like
the
idea
just
streaming
simulations
on
demand
and
again
they
don't
think
about
where
the
files
are.
They
didn't
think
about
what
servers
they're
using
when
they're
watching
their
shows
they
don't
care
about
the
infrastructure.
They
definitely
should
not
be
running
a
ku
cuttle.
To
mark
a
cue
cuttle
command
to
netflix
and
chill'
that
night,
that
is
not
what
they
should
be
doing
and
the
same
for
researchers.
We
should
be
working
on
the
parts
that
have
value
and
just
to
demonstrate.
This
is
a
planetary
supercomputer.
A
What
you
have
here
is
basically
the
login
steps.
Working
were
essentially
the
notebooks
are
in
Seattle.
The
one
on
the
left
is
basically
FM
running
on
Google
in
Japan
middle
is
as
you
in
the
Netherlands,
and
then
the
right
is
basically
OCR
in
Germany,
and
the
data
is
actually
held
on
an
object
store
which
is
notionally
in
Germany,
but
obviously
they
do
optimization
effectively.
You
have
this
global
thing
now,
and
this
was
all
running
again
from
Seattle
it's.
This
is
real
time
in
terms
of
the
movies.
A
Obviously,
it's
movies,
because
I
didn't
trust
the
Wi-Fi,
but
it's
that
quick,
it's
really
nice,
so
I'm
just
gonna
finish
an
acknowledge.
Obviously,
I've
got
big
team.
Helping
me
when
I
say
big
I
mean
tiny,
which
is
basically
my
research
software
engineers.
We
get
funding
from
the
government
I'm
so
lucky
that
I
get
funding
from
the
government
to
play
with
cool
things
and
to
think
of
the
big
problems.
We
don't
get
paid
much
in
science,
but
we
get
this
warm
fuzzy
feeling
of
knowing
that
we're
making
the
world
better
and
not
selling
stuff
to
people.
A
Sorry
I
really
want
to
thank.
Basically
the
cloud
sponsors
are
basically
Google
or
called
Microsoft.
They
will
be
massively
helpful,
so
they've
been
giving
us
cloud
credits
but,
more
importantly,
we've
been
sat
down
with
their
engineers
and
saying
to
them.
Actually,
how
can
we
build
a
planetary
supercomputer?
A
Well,
actually,
maybe
it's
better
for
them
to
run
the
computing
here
in
America
at
night
time
when
there's
less
load,
and
so
let's
move
the
data
to
America
in
the
backend,
there's
a
huge
amount
of
ops
that
could
go
on
underneath.
That
would
be
really
cool.
So
with
that,
thank
you
all
for
listening
to
my
talk
conference.
People
for
accepting
me
and
I
am
happy
to
answer
any
questions.
Thank
you.
A
Okay,
so
the
question
is
basically
how
can
I
scale
when
everything
is
all
in
one
container?
So
what
I
doing
from
the
kubernetes
perspective,
for
that
the
scaling
on
HPC
is
not
within
kubernetes.
So
when
we
do
the
call
the
FM
call
to
run
the
simulation,
the
FM
call
basically
goes
across
to
a
functions
service
and
says:
I
need
this
resource
from
the
function
service
and
then
puts
the
container
onto
that.
So
that
means
that
the
resource
which
is
for
Humanity
is
running.
A
A
A
So
the
question
is:
what
is
what
is
trying
ports?
The
Train
port
is
basically
it's
a
little
module
in
Python,
which
you
can
put
in,
so
you
import,
try
import
and
then
all
you
have
to
do
is
rather
than
say:
import
module,
you
say:
object
equals,
try
import
string,
which
is
the
name
of
the
module
and
it
returns
a
handle
to
the
module
and
the
first
time
a
handle
is
used.
It
then
does
the
import
for
real,
so
it
enables
effectively
you
don't
do
it
on
your
top-level
dependencies.
A
A
So
I've
not
considered
that
I'm,
afraid
I
know
very
little
about
so
the
question
was:
did
I?
Consider
Zeppelin
instead
of
Jupiter
I'm,
afraid
I've
not
considered
yet
as
Epling
is,
are
not
played
with
it.
Much
essentially
in
the
in
universities,
at
least
in
the
UK
Jupiter
is
as
one
it's
the
thing
that's
used
by
everybody.
So
that's
the
environment.
We
teach
in
it's
the
environment
they're
familiar
with,
so
we
stuck
with
that.
Okay,
okay,
over
there.
A
So
so
the
question
is:
who
launches
the
HPC
cluster?
The
HPC
cluster
is
basically
launched
by
FN
in
the
cloud
as
bare
metal,
so
with
OSI
I,
one
of
the
nice
things
they've
got.
Is
it's
a
completely
bare
metal
cloud
with
a
hypervisor
effectively
as
Hardware
in
the
switch?
And
so
the
actual
hardware
knows
that
they're
ridiculously
beefy
nodes
anyway,
but
the
network
connection
between
them
is
as
fast
as
InfiniBand
and
indeed
from
the
specs
faster
than
a
mini
band.
So
you
get
these
really
nice
HPC
clusters.
They
can
run
big
workloads.
A
You
can
either
go
through
FN,
but
the
other
thing
we're
playing
with
is
essentially
doing
backends,
which
is
what
are
called
instance
pools.
So
we
started
with
something
with
terraform
to
bring
things
up
and
down,
but
now
there's
basically
instance
pool
interface,
which
will
make
it
a
bit
easier
for
us.
So
there's
a
lot
of
plumbing
underneath
which
I
didn't
talk
about,
but
again
scientist
shouldn't,
think
of
the
plumbing.
It
should
be
completely
transparent
to
them.
A
A
So
the
question
is:
are
the
simulations
running
multi,
node,
MPI
kind
of
jobs
and
the
ones
I
demonstrated
were
not
because
the
nodes
we've
run
on
already
had
a
hundred
and
four
cores
and
so
essentially
didn't
need
to
this
ridiculous?
Actually,
the
HPC
node
shapes
particularly
with
the
AMD
epoch,
the
huge,
but
they
can
run
multi
node.
So
we
have
some
multi
node
tests
that
we've
run
because
ultimately,
all
you're
doing
at
the
function
call
is
saying
give
me
this
resource.
A
A
A
So
the
question
was:
if
they
wanted
to
use
tensorflow
or
cube
flow
with
this
kind
of
integration,
could
it
go
that
way?
I
don't
know
enough
about
how
kublai
works.
What
I
would
say
is
if
you're
looking
at
the
identity
management,
authorization
there's
other
JWT
tokens
for
that
we
could
probably
reuse
those
beds
and
ultimately,
all
of
this
this
is
just
HTTP
endpoints.
So
in
theory
you
could
just
be
calling
those
points
and
working
with
them.
A
A
A
So
the
question
is
two
questions:
one
is:
is
the
triple
AI
stuff
open-source?
Yes,
how
much
a
percent
I'm
an
academic
everything
we
do
is
on
github
live,
so
you
can
see
all
of
our
bugs
as
we're
developing
them
and
it
is
open
source
it's
Apache
licensed,
because
we
really
wanted
to
be
sort
of
very
usable
by
lots
of
universities
and
we're
thinking
of
how
we
can
move
this
into
a
national
system.
The
second
question:
sorry,
I
forgot
them.
Can
you
repeat.
A
So
how
often
the
question
is,
how
often
can
you
update
the
FN
rapper,
the
little
Python
script,
which
runs
the
workflow,
and
how
easy
is
that?
How
and
it's
actually
we
updated
quite
a
lot,
so
the
rapper
we're
talking
about
the
reason
we
can
build
this
thing
quickly
is:
this
is
not
millions
of
lines
of
code.
It's
like
a
few
hundred
to
a
thousand
lines
of
Python.
It's
really
simple
and
FN.
A
Has
this
really
brilliant
developer
interface
that
effectively,
once
you
have
got
your
docker
file
to
describe
the
container
you
just
go,
FN
build
FN,
deploy
and
then
the
whole
thing
just
goes
up
so
I'm
I,
probably
deploy
functions,
live
dozens
of
times
a
day
and
when
you
deploy
it
to
the
service,
it
just
goes
on
to
the
next
newest
version.
So
as
a
developer,
experience
is
really
brilliant.
A
What
sort
of
data
follows
my
research
is
moving
around?
We
live
in
the
petter
to
exer
world
so
effectively
you
can
be
having
like
the
simulation
for
malaysia.
The
small
demo
is
just
like
tens
of
megabytes
just
because
I
want
to
do
a
small
demo,
but
in
terms
of
the
real
simulations,
those
tens
of
gigabytes
to
terabytes
petabytes
of
data,
and
so
one
of
the
things
we're
buildings,
we've
architected
in
a
building
next
year
from
february
is
effectively
19.
I
can
show
a
little
picture
of.
A
It
is
a
system
where
effectively
we're
taking
all
of
the
UK
satellite
data
and
all
of
the
atmospheric
sensor
data
and
that's
all
being
piped
in
in
this
framework
here
into
object,
stores
using
FN
functions,
doing
cleaning
of
the
data
using
FN
functions
to
do
analysis,
jupiter
notebooks,
to
enable
interaction
from
its
researchers
and
also
from
the
public,
and
you
do
things
like.
We've
detected
this
cloud
of
methane
here
we
now
spin
off
CFD
calculations
that
can
backtrack
the
methane
back
to
the
individual
cows.
A
The
purpose
being
is
in
the
UK
for
the
climate
change
of
what
we
need
to
count,
how
many
cows
and
cars
and
trains
and
buses
we
have.
So
all
of
this
will
just
be
spun
up
and
we're
building
this
next
year
and
again,
if
there's
anyone
here
who
would
like
to
work
on
this
project
I'll
be
hiring
in
january
again,
the
pay's
not
good,
but
you're
helping
save
the
world
and
not
sell
stuff.
So
that's
kind
of
cool.
A
A
Yes,
so
we
have
a
project
which
will
be
starting
off
with
basically
EBI,
which
is
looking
at
patient
data
and
gene
data
and
again
the
discussions
where
the
cloud
companies
are.
You
actually
have
object,
stores
the
object,
store
data.
When
I
talked
about.
Oh,
you
could
publish
it
they're,
obviously
full
ACLs
on
that
all
of
the
data
in
this
entire
system
leak
we've
built
all
of
it
to
handle
private
data,
so
everything
is
massively
encrypted
massively
secure
one-time
passwords
on
everything.
A
Data
is
encrypted
on
flight
with
neutral
passwords
data
on
the
object
store
is
encrypted
on
rest,
with
rotating
passwords,
all
of
the
cloud
companies
now
I
say
certified
for
data
compliance
and
indeed
I
think
if
anyone
comes
from
the
UK.
You
know
that
we're
moving
all
of
our
NHS
data
on
the
cloud
so
I
personally
think
that
it's
it's
a
governance
thing.
It's
not
a
technical
thing
anymore.
A
The
code
is
on
github.
The
repo
should
show
you
that
bit
the
repo
is
basically
github
comm,
then
slash
Chris
Wood's
and
though
it's
called
choir.
If
I
try
and
make
that
bigger
yeah,
you
should
be
able
to
see
that
and
that's
basically
for
the
accounting
and
infrastructure
power
which
we
forked
out.
Yes,.
A
So
questions
how
you
use
Python
flash
to
handle
massive
amounts
of
inflow
data,
no,
no
way
no
I,
haven't
and
I
think
that
would
be
really
interesting.
So
again,
it's
kind
of
like
in
the
academic
world,
we've
looked
at
the
cloud
a
bit
like
we're
scared
of
it
that
we
don't
want
to
lose
our
precious
our
precious
HPC
machines
and
I
built
machines
for
a
long
time,
as
I
said
and
I
reached
the
point
where
I
never
want
to
buy
another
cluster
I
never
want
to
install
another
cluster,
never
want
to
deal
with
another
cluster.
A
I
just
want
to
throw
things
in
the
cloud
and
the
nice
thing
about
the
cloud
is
when
you
move
beyond
it's
just
a
cluster
and
actually
view
it
as
a
planetary
supercomputer,
where
things
are
just
flying
everywhere
and
infinitely
scales,
bla,
bla
bla.
You
actually
begin
coming
up
with
these
kind
of
ideas
and
you
say
actually
there's
a
whole
different
way.
We
could
be
doing
computing,
which
technology
is
not
the
point,
because
we
can
do
the
computing
this
way
interactively
LiveCycle.
A
Yes,
so
effectively,
what
that
call
does
is
it
goes
here.
I
have
my
data
I
copy,
the
data
into
an
object,
store,
get
the
bucket
location
for
that
data
and
then
I
call
an
HTTP
endpoint,
which
is
calling
the
function.
Tell
it
the
name
of
the
bucket
and
then
I
get
a
call
ID
back
now
it's
sending
just
the
data
which
is
for
that
molecular
simulation.
So
it's
not
the
notebook,
that's
gone
across,
it's
actually
the
molecules.
A
So
it's
these
631
molecules
here
that
have
been
loaded
Plus
this
protocol,
for
how
you
run
the
simulation
effectively.
That's
just
turned
into
the
input
files
running
the
program,
so
we
don't
need
to
send
that
much
is
as
much
as
the
information
as
the
simulation
house.
One
of
the
interesting
things
is
at
the
moment.
This
is
actually
running.
The
data
is
already
on
the
notebook,
but
we
want
to
then
move
to
a
world
where
you
start
off
with
a
date
on
the
laptop
and
then
you
move
the
data
onto
a
notebook.
A
Actually
it's
here
or
here
or
here
I,
don't
think
it's
a
filesystem
I
do
think
it's
a
way
of
playing
with
objects
and
buckets,
and
we
know
it
works
because
effectively
when
you
use
Google
photos,
you
don't
really
think
about
where
anything
is
I
mean
that
kind
of
similar
interface
for
genes
and
simulations
and
cfd
ones,
and
things
like
that
and
again
I'm
interested
in
people's
ideas
of
what?
How
that
could
be,
do
have
any
other
questions.
A
Okay,
so
this
is
a
very
good
way
to
have
I
compared
this
to
lambda,
and
there
are
two
aspects
of
my
answer
to
this.
One
of
them
is
that,
as
a
researcher,
I've
got
to
build
on
top
of
open
portable
platforms,
so
we
were
bit
really
hard
when
we
all
made
stuff
with
CUDA
and
we
ended
up
being
trapped
into
buying
lots
nvidia
hardware
and
because
of
that
there's
basically
a
moratorium
on.
You
will
not
go
and
use
a
single
vendor
platform,
so
we
have
to
go
on
open
platforms.
A
We
cannot
use
AWS,
lambda
and
design.
For
that.
That's
one
of
the
reasons
we
went
to
FN
because
it's
open
and
portable
we
can
move
it
between
clouds
and
it's
no
problem.
The
second
reason
is
because
the
unit
of
abstraction
should
be
the
docker
container,
so
the
problem
with
lambda
I
think
as
Kelly's
talk
showed.
You
can
kind
of
create
a
zip
file
that
contains
a
static
application,
and
you
can
work
really
hard
to
do
that.
A
But
actually
most
scientific
applications
are
not
one
static,
zip
file,
it's
a
much
more
complicated
environment,
including
your
MPI
drivers.
Maybe
your
GPU
drivers.
All
of
these
extra
bits
and
pieces,
this
huge
application
that
you
cannot
just
put
into
a
zip
file.
You
can
put
it
into
a
docker
container,
and
so
the
unit
of
at
least
scientific
server
list
has
to
be
a
docker
container
and
the
easiest
way
for
us
to
get
things
in
and
out
and
understand.
A
It
is
standard
in
standard
output
and
stand
there
for
logs,
so
I
think
if
we
build
server
as
frameworks
with
docker
container
as
the
root
standard
in
standard
out
standard
out,
I
think
it
would
be
perfect
and
any
server
list
framework
which
has
that
it's
really
I'm
happy
to
move
to
and
if
AWS
would
like
to
move
to
that
I
think
you'd
be
great.
Ideally,
what
I'd
like
to
see
is
because
there
is
already
FN
and
it
works
so
perfectly
that
all
the
cards
go.
A
Let's
go
and
use
manage
FN
services,
rather
than
we
I
mean
K
native,
looks
good,
but
it's
so
far
behind,
where
FN
is
being
blunt,
FN
is
really
working.
Well,
we'll
go
on
watching
their
product,
they've
announced
their
production
FN
service
a
few
days
ago.
So
I
didn't
remember.
I'm
running
my
own
FN
clusters
on
top
of
a
kubernetes,
but
eventually
I
should
be
able
to
just
buy
them
and
not
have
to
care
about
the
underlying
convenience
clusters.