►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Economics and Best Practices of Running AI/ML Workloads on Kubernetes - Maulin Patel, Google & Yaron Haviv, Iguazio
In this session, we will discuss how Kubernetes driven AI/ML building blocks are making AI/ML simple, fast and efficient for data scientists, data engineers, devops engineers and everyday users. We will explore how Kubernetes, Kubeflow and Kubeflow pipeline can help to mitigate complexities and challenges associated with AI/ML. We will demonstrate the use of Accelerators like GPUs and TPU in Kubernetes Engine to make serving compute intensive ML/AI workloads easy, fast and scalable. We will present the real world examples of commonly used AI/ML applications, discuss their performance and share best practices. We will also present how the economics are different when it comes to ML workloads and highlight the unique values Kubernetes brings to enterprises.
https://sched.co/MPaf
A
A
A
A
A
So
as
a
community
interested
in
AML,
how
can
we
make
AI
ml
simple,
fast
and
cost
effective?
We
want
to
make
AML
simple,
so
it's
accessible
to
everyone
in
our
organization.
We
want
to
make
it
fast,
so
you
can
train
the
model
fast,
get
inside
faster
and
drive
business
decision
faster.
We
want
to
make
it
cost-effective
in
order
to
democratize
AML
so
that
it's
accessible
to
everyone.
A
So
when
we
think
of
a
IML,
we
typically
think
about
model
building
and
model
training.
However,
we
need
to
realize
that
amo
is
much
bigger
than
that.
Ml
is
typically
a
small
part
of
an
end-to-end
AML
application.
A
lot
of
skills
are
needed
to
build
any
AML
application.
For
example,
a
data
engineer
may
cleanse
the
data.
Another
engineer
may
extract
features.
Data
scientists
may
build
models,
an
application
engineer
may
convert
the
model
into
an
API
and
finally,
an
end-user
may
use
the
applications
which
are
built
using
those
API.
A
A
That's
why
we
need
cloud
native
KML
platform
we
want
to
kubernetes
has
become
the
de
facto
standard
for
cloud
native
computing
and
we
want
to
bring
the
power
of
kubernetes
to
AI
ml
kubernetes
makes
a
IML
workloads,
scalable,
portable
and
composable,
with
the
help
of
kubernetes
data,
scientists
can
focus
more
on
building
and
training
models
instead
of
worrying
about
underlying
infrastructure.
All
they
have
to
do
is
choose
your
favorite
hammelburg
floor
framework
pack,
the
models
up
in
kubernetes
and
run
them
on
kubernetes
at
scale.
A
Kubernetes
takes
care
of
scaling
the
workload
from
a
single
node
to
thousands
of
nodes.
It
makes
data
scientists
more
productive,
so
why
kubernetes
for
a
mo
kubernetes
provides
three
core
features
that
are
very
attractive
for
AML
workloads.
The
first
is
portability:
kubernetes
provides
cloud
native,
open
and
standardized
api's
with
using
coop
by
using
kubernetes.
You
can
seamlessly
pour
your
workload
from
your
laptop
or
your
cloud
to
your
private
data.
Center.
Second
is
scalability.
With
the
help
of
kubernetes,
you
can
scale
your
ml
workload
from
a
single
node
to
thousands
of
node.
A
Kubernetes
also
supports
hardware
accelerators
such
as
GPU
TPU,
and
also
auto
provisioning
and
auto
scaling,
which
makes
killing
your
workload
simple
and
easy,
and
the
third
is
productivity.
Kubernetes
makes
data
scientists
and
data
engineers
more
productive
by
freeing
them
up
from
managing
their
own
workstation
servers
or
VMs.
It
lets
you
focus
on
your
model
building
and
model
training
instead
of
focusing
in
managing
VMs,
so
the
kubernetes
is
actually
application.
Agnostic,
so
kubernetes
provides
many.
Many
advanced
features
like
scalability
load,
balancing
scaling,
etc.
However,
it
does
not
know
anything
about
mo.
A
This
leaves
a
lot
of
work
for
ml
operations,
teams
in
setting
up
the
infrastructure
downloading
and
installing
ml
frameworks
and
libraries
goo
flow
fills
that
gap.
A
cool
flow
is
actually
a
kubernetes
native,
open
source
platform
to
develop,
deploy
and
manage
portable
and
scalable
machine
learning,
workloads.
A
Coop
flow
provides
higher
level
abstractions
such
as
tensor
flow
job,
which
makes
running
tensorflow
job
or
training
job
much
simpler
and
easier
on
kubernetes
it
integrates
tensorflow,
distributed
training
and
estimator
api's
with
the
kubernetes.
It
uses
kubernetes
to
scale
training
and
leverages
hardware
accelerators,
and
it
benefits
from
auto
scaling
and
auto
provisioning
capabilities
of
kubernetes.
And
with
this
you
can
also
benefit
from
the
why
Brent
ecosystem
of
kubernetes
toolchain
to
develop
your
ml
workflows
and
run
amock
workloads.
A
Tensorflow
also
provides
sorry.
Crew
flow
also
provides
advanced
abstraction
for
serving
so
imagine
that
you
build
a
very
advanced
model
for
movie
recommendation
now.
You
want
to
test
this
model
on
a
small
scale,
with
a
global
traffic
to
study
the
fidelity
of
your
model.
So
if
you
deploy
this
model
in
US,
the
traffic
will
be
overwhelmingly
from
Hollywood
movie
fans.
A
If
you
deploy
this
model
in
India,
the
traffic
will
be
overwhelmingly
from
Bollywood
movie
friends,
so
you
have
a
dilemma
on
one
hand,
your
you
want
your
model
to
get
a
slice
of
traffic
from
the
regions
of
your
interest.
On
the
other
hand,
you
don't
want
to
deploy
an
experimental
model
globally
to
limit
the
blast
radius.
Here,
kubernetes
and
cut
steel
comes
to
the
rescue.
With
the
help
of
kubernetes
and
sto.
You
can
configure
the
load
balancer
so
that
your
model
gets
a
slice
of
traffic
from
the
region
of
interest.
A
So
you
can
test
the
fidelity
of
your
model
from
the
traffic
that
you
are
interested
in
also
with
kubernetes
ecosystem.
For
example,
you
can
use
Prometheus
for
exporting
your
model
matrix.
You
can
also
use
sto
for
telemetry
and
traffic.
Splitting
so
kubernetes
and
sto
make
many
many
model
lifecycle
challenges,
simple
and
easy
to
handle.
A
So
in
ml
world.
There
are
two
distinct
experiences
that
we
have
to
address.
So
there
is
a
d0
experience
which
is
focused
on
model
building,
where
a
hosted
notebook
can
make
a
huge
difference
and
once
a
model
is
built,
then
you
have
to
deal
with
the
production
ization
challenges
where
you
have
to
deal
with
security,
scalability
logging,
monitoring,
etc.
So
now,
let's
look
at
how
kubernetes
helps
to
address
those
challenges.
A
So
in
de
0
we
are
interested
in
building
and
training
models
and
the
designer
is
typically
training
their
models
using
notebooks
and
they
have
their
own
preferences
for
the
libraries
they
would
like
to
use
and
the
free
work
they
would
like
to
use.
So
here
we
can
benefit
from
hosted
netbooks,
which
come
pre-configured
with
the
most
popular
libraries
and
the
frameworks,
and
this
includes
libraries
such
as
fanned
as
a
numpy
and
many
more
and
machine
learning
frameworks
like
Kyra's,
tensorflow,
extra
boot,
etc.
A
So
data
scientists
can
open
their
browser
and
connect
to
the
dope
connect
to
the
notebook
which
are
hosted
and
start
fetching.
The
data
and
building
the
models
directly
into
the
notebook
and
they
can
train
the
model
nor
in
the
notebook
and
the
best
part,
is
they
can
deploy
those
models
once
they
are
trained
directly
under
kubernetes
from
the
notebooks,
which
frees
them
from
learning
intricacies
of
kubernetes.
A
So
once
a
model
is
built
and
trained,
the
next
is:
how
do
we
deploy
into
production
wall?
Here
we
have
to
address
many
production
related
challenges,
for
example,
security,
logging
monitoring,
telemetry
also
data
gravity
and
may
so
forth
and,
as
you
may
all
know,
because
of
the
variety
of
challenges,
lots
of
workloads
today
run
in
on-prem,
environment
and
challenges
related
to
regulations,
latency
data
gravity
and
so
forth.
So
you
know
what
that
means
is
lot
of
models
will
be
running
in
on-prem
environment
as
well
as
in
cloud
environment.
We
call
it
hybrid
model
at
Google.
A
We
are
investing
heavily
in
our
hybrid
strategy
in
the
hybrid
world.
What
we
see
is
you
can
run
your
machine
learning
workloads,
an
on-prem
environment,
using
open
source
tools
like
goo
flow.
When
you
are
in
a
cloud
environment,
we
offer
AI
platform,
which
is
managed
equivalent
of
goo
flow
from
Google,
and
you
can
run
your
models
directly
in
under
the
AI
platform.
So
ai
platform
is
basically
developer:
environment,
integrated
developer
environment
for
data
scientists
and
data
engineers
in
that
world.
A
All
the
pain
of
doing
devops
he's
taken
care
by
the
Google
versus
when
you
are
on
pram.
You
have
to
manage
you
flow
on
your
own,
so
you
can
use
your
favorite
notebooks
and
Python
code
to
talk
to
goop
flow
on
pram,
as
well
as
AI
platform
on
GCP.
However,
the
underlying
api's
that
are
supported
on
qu
flow
and
on
AI
platform
are
slightly
different,
so
the
qu
flow
supports
kubernetes
tile
api's.
An
di
platform
supports
restful,
api
s--
and
often
data
scientists
use
their
local
environment
to
run
their
own
code
and
validate
the
model.
A
So
in
that
case,
data
scientists
have
to
deal
with
many
different
varieties
of
api's.
That's
not
fun!
So
to
address
that
we
are
developing
a
hybrid
machine
learning
SDK,
which
basically
abstracts
away
all
the
intricacies
of
underlying
platform
into
common
set
of
API.
So
we
call
it
goof
floor
faring.
So
what
is
good
flow?
A
Ferenc
goop
flow
firing
is
an
open
source,
hybrid
machine
learning,
sdk
for
data
scientists
to
write
ml
code
once
and
run
anywhere,
not
only
that,
it
also
gives
you
the
ability
to
package
your
data
science
machine
learning
workload
for
remote
execution,
so
it
abstracts
away
the
pain
of
managing
different
underlying
platforms
and
now
I'll
show
you
some
examples.
So
this
start,
you
see
three
different
ways:
data
scientists
have
to
deal
with
their
platform.
A
A
In
the
cool
flow
fearing
wall,
all
the
code
is
abstract
away
into
a
common
code,
which
almost
look
identical.
The
main
difference
here
is
that
back-end
definitions
which
are
provided
as
an
argument
to
goo
flow
training
job,
is
slightly
different
in
each
environment.
So,
as
you
can
see
here,
the
backend
definition
is
local
for
local
environment
yai
platform
for
yet
at
platform,
environment
and
then
scoop
flow
for
a
cooler
environment.
It's
easy
to
imagine
that
this
could
very
well
have
been
a
development
staging
and
production.
A
So
now
let
me
show
you
an
end-to-end
demo
how
this
works.
So,
first,
what
we
will
do
is
we'll
discover
an
existing
model
from
AI
hub
and
GI
hub
is
basically
one-stop-shop
for
storing
and
sharing
your
AI
assets,
for
example,
pipelines
your
models,
your
notebooks,
etc.
This
makes
collaboration
within
an
organization
much
easier
once
we
discover
the
model
on
the
AI
hub.
Next,
we
open
it
and
a
hosted
environment
using
Jupiter
notebook,
and
then
we
are
gonna
train
it
in
a
three
different
ways.
So
now,
let's
go
to
the
demo.
A
A
All
right,
so
what
you
are
looking
here,
we
start
at
the
AI
hub
search
space
and
what
I'm
gonna
do
is
I'm
gonna
search
for
a
cancer
detection
demo.
So
as
soon
as
I
hit
the
search
button,
you
will
find
that
there
are
four
results.
Three
of
them
are
actually
notebooks
and
one
is
actually
the
model
itself.
We
are
interested
in
the
third
result,
which
is
a
cancer
detection
model,
so
we
go
there
and
we
open
the
model
in
a
GCP
hosted
notebook.
So
that's
what
you're,
seeing
here.
A
So
this
is
how
the
model
looks
like
on
the
top
part
of
the
model.
You
have
environmental
setup,
which
is
very
common.
Next.
What
you
see
here
is
basically
we
are
exploring
the
data
so
understand
the
data
and
clean
the
data.
After
exploration,
we
will
go
to
basically
a
visualization,
so
here
you
can
see
some
code
for
the
visualization.
When
we
wrote
the
visualization
code,
it
will
display
some
images,
which
are
basically
positive
and
negative
samples
for
your
cancer
detection
input.
A
Then
we
build
the
model
so
after
the
model
is
actually
built,
we
want
to
fetch.
Basically
the
embeddings
and
those
embeddings
are
fetch
from
the
AI
hub
for
our
model.
These
are
image
and
bearings,
and
now
we
are
going
to
train
it
on
three
different
environments,
so
the
first
environment
is
basically
local
environment.
As
soon
as
we
start
training
on
local
environment,
it
will
start
showing
us
logs.
The
second
environment
is
school
flow
environment.
A
A
So
now
it's
straining
and
you
can
see
the
logs
are
being
displayed.
The
training
gets
finished
and
194
seconds
here,
and
you
can
see
the
loss
and
accuracy
metric.
Next,
we
are
going
to
train
the
same
model
in
the
qu
flow
environment,
so
in
this
case
the
model
gets
packaged
by
qu
flow
faring
and
it's
executed
remotely
and
the
logs
are
streamed
back
to
the
notebook
so
in
the
top
part
of
it.
A
What
you
see
is
that
the
model
is
actually
packaged
as
a
docker
image
and
is
put
in
the
repository
from
where
it
gets
picked
up
by
the
qu
flow,
and
then
the
execution
is
pretty
much
same.
You
can
see
the
logs
in
the
notebook
and
the
third
is
we're
gonna
do
the
same
thing
in
the
a
AI
platform
training
environment.
So
the
key
difference
that
you
will
will
see
here
when
we
do
it
and
the
AI
platform
environment
is
now
the
logs
are
actually
made
available
to
be
streamed
to
stackdriver.
A
So
you
can
actually
I'll
show
you
in
a
second.
In
this
case,
we
provide
you
a
link,
so
it
opens
the
logging
in
the
stackdriver
and
from
there
on,
you
can
monitor
and
do
everything
that
you
want
you
to
instruct
our
environment.
So
what
this
shows
here.
So
this
is
the
link
to
the
spec
driver,
and
now
you
go
to
the
truck
driver
and
open
the
logs.
C
A
A
Okay,
so
let's
quickly
review
what
goofily
opera
fairing
provides
you.
So
it's
basically
the
advantage
of
coop
flooring.
Is
it's
a
data
scientists
focus.
It
works
with
the
data
science
friendly
environment
in
notebooks,
as
well
as
data
science
friendly
programming
languages.
It's
a
multi-platform,
so
you
can
seamlessly
move
across
your
local
environment
as
well
as
your
GCP
environment
or
your
private
data
center
environment.
A
Okay,
so
what
are
the
advantages
of
coupe
flow
fairing
for
ml
of
steam?
So
basically
it
allows
you
to
enforce
your
best
practices
that
you
want
to
enforce
across
your
organization.
Second,
is
it's
open
source?
So
there
is
no
lock-in.
You
can
make
use
of
it
right
away
without
having
to
worry
about
any
vendor
lock-in
and
also,
as
I
mentioned,
with
the
remote
bursting,
you
can
save
a
significant
amount
of
money
on
your
operations
cost.
So
with
that,
let's
move
on
to
the
second
part
of
the
talk
and
I
will
invite
Iran
to
take
it.
B
So
I
have
some
ambitious
demo
and
I
hope
it
will
work
and
all
that,
but
before
diving
it
into
a
demo
of
like
a
full
pipeline
using
cube
flow.
Let's
talk
about
some
some
basics.
What
what
I
want
to
show
you
is
this
nice
yeah?
We
can
build
a
fully
automated
machine
learning
environment
and
leveraging
some
of
the
surveillance
technologies
that
are
now
integrated
into
cube
flow.
B
So
one
of
the
key
challenges
today
is
that
machine
learning
is,
is
pretty
complicated
and
siloed,
so
we
have
some
folks
still
working
on
a
dupe
and
data
lakes
and
spark,
and
all
that
on
on
yarn,
some
data
scientists
will
come
more
from
a
Python
mindset
and
deep
learning.
Machine
learning,
scikit-learn
actually
boosts
tensorflow
work
on.
Sometimes
there
are
laptops,
sometimes
notebooks,
sometimes
kubernetes,
if
they're
slightly
more
advanced,
and
then
we
have
the
operational
aspects
of
where
do
you
deploy
all
that?
B
So
you
have
some
containers
that
come
from
the
machine
learning
world
and
they
just
present
endpoints
that
get
some
vectors
and
process
and
respond,
and
you
have
some
other
containers
that
interact
with
those
containers.
Now
we
have
three
different
disciplines:
three
different
silos
within
the
organization
and
most
of
the
transfer
of
artifacts
and
knowledge,
and
everything
is
done
manually
today.
So
we
need
to
change
that.
So
the
general
idea
is:
why
not
use
kubernetes
a
server,
a
unified
architecture,
not
necessarily
a
single
cluster?
B
You
may
still
have
multiple
clusters
but
serve
a
unified
architecture
for
delivering
such
a
pipeline.
So
you
know
you
do
need
various
ways
of
storage
and
databases
underneath
and
then
you
have
all
those
variety
of
services
that
come
from
those
different
disciplines.
You
can
run
SPARC
today,
not
many.
No,
you
can
just
run
SPARC
presto
hive.
All
those
things
on
kubernetes
today,
but
you
can
also
run
by
torch,
tends
to
flow,
etc
and
other
things
that
are
more
serve
on
the
operational
side.
Permit
is
graph
on
service
functions
and
so
on.
B
So
what
you
can
build
is
some
some
pipeline,
which
is
slightly
more
advanced
and
have
three
components.
One
is
collection
of
data.
Now
most
people
that
come
from
the
data
world
still
think
of
collection
of
data
is
an
ETL
process,
if
you're
being
slightly
more
modern
application.
Collection
of
data
is
also
streams
that
just
feed
in
or
or
a
scrapers.
You
need
some
weather
information.
You
don't
want
to
store
that
in
your
your
own
data,
Lake
or
throughout
the
entire
history.
You
can
just
go
to
an
external
weather
service.
B
It's
the
external,
real-time
stock
trading,
information
service,
etcetera.
So
you
need
automation
for
grabbing
data
and
bringing
it
into
serve
this
this
place
where
you're
processing
the
data.
The
second
thing
that
you
do
once
you've
brought
data
and
preferably
label
that
and
prepare
that
a
bit.
You
need
to
do
machine
learning,
training
and
validation,
and
also
here
you
can
leverage
the
advantage
of
kubernetes
in
scalability.
You
don't
want
to
run
experiment
on
this
notebook
or
this
specific
VM.
B
You
want
to
be
able
to
run
the
same
experiment
or
the
same
training
on
potentially
tens
or
hundreds
of
job,
depending
on
the
size
of
your
task,
and
you
want
to
pull
resources
because
I
still
see
people
just
taking
a
Jupiter
attaching
a
GPU
to
that.
You
know
in
Amazon,
that's
like
$12
an
hour
for
a
quad
node
GPU.
If
you're.
If
you
have
money,
that's
ok,
but
then
you're
wasting
those
resources,
so
the
right
approach
would
be
to
essentially
pull
the
GPU
resources
and
every
time
you
launch
a
job
just
have
a
container
amount.
B
Also
here
many
times,
people
think
about
serving
as
something
that
gets
a
vector
of
floating
boat
numbers
and
respond
with
the
floating
point
numbers,
but
then
there's
a
critical
challenge
here.
How
do
those
floating
point
numbers
get
generated
that
serve
a
dark
secret?
No
one
knows
so.
Essentially
the
real
data
in
the
world
is
geolocation.
This
customer
identifies
its
era,
so
the
first
thing
you
need
to
do
is
feature
assembly
and
only
then
feed
it
into
a
model.
B
And
last
but
not
least,
we
have
to
observe
a
feedback
loop.
We
have
to
monitor
everything
that
happens
during
production
and
we
could
use
things
like
Griffin
and
primitives
to
do
that
and
feed
the
data
back
into
the
pipeline,
and
the
last
thing
that
we
want
is
that
essentially
the
user,
the
consumers
of
that
platform
need
some
easy
way
to
access
the
entire
system,
because
there's
a
lot
of
complicated
things.
You
know
service
measures,
containers
the
amell's.
B
We
want
to
extract
it
through
notebooks,
through
pipelines,
we'll
show
that
and
through
things
like
service
functions,
so
queue
flow.
Essentially,
what
it's
trying
to
do
is
is
bring
all
those
different
projects
for
all
those
different
tasks
within
the
pipeline's
under
one
umbrella,
with
the
serve
simpler
integration,
and
this
is
where
you
can
build
a
real
pipeline.
So
let's
look
at
what
it,
what
do
I
mean
by
real
pipeline.
B
So
the
first
thing
you
need
to
do
is
collect
data
and
the
reason
I
put
three
icons
here
again
to
make
sure
people
understand
that
collecting
data
is
not
necessarily
doing
an
ETL
job
from
Oracle
or
reading
an
s3
Park
a
file
and
loading
it
into
your
data
science
platform.
Sometimes
loading
data
is
screaming
its
Kafka
stream.
Sometimes
it's
fetching
from
some
API
services,
then,
once
we
take
all
that
data,
we
need
to
create
a
feature
vector
and
we
prefer
to
serve
snapshot.
It
version
it.
B
So
when
we
run
this
experiment
again
and
again,
we
make
sure
that
we
we
get
the
same
result.
So
we
need
to
take
data,
then
we
put
it
into
a
pipeline.
The
first
step
in
the
pipeline
will
be
to
do
training,
so
we
need
to
take
some
code,
some
notebook
loaded
in
run,
some
training
generate
an
output,
serve
data
set
and
our
model.
B
The
second
thing,
we'll
do
is
probably
validation
again,
we'll
take
some
code,
that
does
validation,
run
it
in
and
get
some
reporting
and
so
and
then
we
do
the
joined
the
deployment
general
generate
some
artifact,
like
a
container
that
may
be
deployed
in
a
different
cluster,
and
once
we've
done
the
entire
testing
everything
works.
We
may
want
to
get
some
notification,
so
every
step
in
the
pipeline
have
some
sort
of
output
result,
and
this
is
just
the
sort
of
the
data
science
for
machine
learning
pipeline.
B
Another
slide,
and-
and
this
is
where
we
can
actually
use
cube
flow
pipeline.
So
cue
flow
type
pipeline
is
a
relatively
new
component
but
very
powerful.
It
allows
us
to
essentially
create
those
dependency
graphs
with
complicated
policies.
It
allows
us
to
build
the
results,
the
inputs
and
outputs
for
each
experiment
and
even
compare
different
different
results.
One
very
powerful
thing
I
like
about
flow
pipeline,
is
the
ability
to
component
a
component
as
elements.
B
So
today
there
are
components
from
Google
from
Amazon
from
IBM
from
from
Iguazu,
so
people
can
just
grab
components
and
integrate
them
into
their
pipelines.
The
next
thing
we
need
to
think
about
is
not
just
the
offline
server.
The
machine
learning
pipeline,
but
also
the
real-time
pipeline
are
the
serving
pipeline.
And
again,
a
serving
pipeline
doesn't
add,
with
just
having
a
container
that
does
model
serving
a
serving
pipeline
includes
some
application.
B
Let's
say
web
app
or
a
mobile
app
that
bring
some
data
in
and
then
you
need
to
essentially
form
this
feature
vector
that
is
fed
into
the
model
where's.
The
data
coming
from
some
of
it
will
come
from
the
event
itself.
Some
of
it
is
from
like
real
time
sources,
some
of
it
from
historical
sources
that
also
been
used
for
the
training,
and
then
we
have
the
model
serving.
So
you
can
see
all
those
things
always
introduce
complexities
are
so
many
moving
components
that
we
need
to
deal
with.
B
So
what's
a
simple
way
to
address
all
this
complexity
without
building
docker
files,
all
the
time
and
compiling
and
manually
scaling
and
manually
logging
and
manually
securing
everything
is
essentially
adopting
the
service
approach.
So
service
is
a
great
thing.
Everyone
likes
herbalists.
You
know
there
was
even
a
service
day
on
Monday
and
what
is
serving
is
really
about
service
is
about
automating
the
process
of
taking
code
and
operationalizing
it
thinking
about
all
the
aspects
of
logging,
auto-scaling,
forgetting
about
yeah
Mo's
and
focusing
more
about
requirement
specifications
and
also
adding
all
the
instrumentation.
B
So
one
of
the
key
challenges
around
machine
learning
and
data
science
is
the
data.
Scientists
are
not
really
used
to
think
about,
like
exception
handling
and
security,
and
all
those
things
so
having
more
and
more
automated
instrumentation
built
into
the
code
is
actually
a
good
thing,
especially
when
the
gap
is
so
big
between
data
scientists
and
production
code
and
the
other
things
that
are
like
auto-scaling,
rolling,
upgrade
etcetera.
But
what's
the
challenge
around
service
that
it
wasn't
designed
for
data
science
or
machine
learning?
Okay,
what
does
it
mean?
Usually
single-threaded?
B
No
concurrency,
no
support
for
GPUs,
because
that's
like
hardware,
it's
a
stateful
thing,
usually
stateless
application
patterns.
So
when
you
need
to
deal
with
things
that
is
with
state
and
context
and
caches
that
pretty
hard
to
do
most
of
the
frameworks,
think
of
it
in
terms
of
HTTP
when
you're
starting
to
move
to
streaming
and
batch
processing,
the
state
involved
and
checkpointing,
and
all
that
it's
not
really
designed
to
most
of
the
frameworks
and
hard
debugging
and
diagnostics.
So
with
that
in
mind,
we
came
with
an
open
source
project
called
nucleo.
B
It's
already
like
two
years
on
the
road
which
is
very
oriented
towards
data
processing,
analytics
and
machine
learning
and
I'll
show
some
of
those
capabilities
in
a
minute.
What's
different
about
nucleon,
the
lower
levels
is
first,
it's
extremely
high
performance
running
your
code
within
nucleo
is
faster
than
running
your
code
without
nuclear,
and
that's
because
nuclear
itself
deals
with
real-time
performance,
optimizations,
real-time,
CPU
scheduling,
real-time
thread,
scheduling,
garbage
cleaning.
All
that
is
handled
by
the
runtime.
B
Second,
is
all
the
features
around
a
data
processing,
checkpointing
sharding,
rebalancing
integration
with
nvidia
Rapids
for
ETL
and
machine
learning,
integration
with
other
framework
for
GPU
optimizations,
all
the
different
streaming
API.
You
could
think
of
Kafka
kinases,
a
google
pub/sub
esri
event
hub
MQTT
jobs,
etc.
B
B
Other
things
that
to
make
it
more
suitable
for
data
Sciences,
first
automation
around
Jupiter,
this
entire
project
just
called
nuclear
Jupiter,
which
is
all
about
how
to
convert
Python
code
or
your
notebooks
automatically
into
real
time.
Artifacts
that
contain
all
the
aspects
around
the
function,
not
just
the
function
itself,
but
also
the
resources
is
going
to
consume
dependencies
packages,
a
PA,
you
know
endpoints,
etc.
B
The
second
thing
is
the
integration
with
cube
flow
pipeline
that
extend
the
features
of
Cuba
pipeline
for
doing
things
like
parallel
processing,
seeing
steps
and
code
build
steps,
you
can
build
steps
that
essentially
build
the
code.
I'll
show
that
things
like
scene,
processing,
etc
also
again
around
GPUs.
We
do
a
lot
of
work
on
GPUs,
very
close
relations
with
rep,
Nvidia
and
videos
using
nucleo
across
the
board
in
different
places
you
can
actually
with
this
is
a
benchmark
that
was
done
with
the
Samsung
on
a
specific
Samsung
commercial
application,
we're
running
the
same
application.
B
B
So
essentially,
when
you
go
to
nuclear,
you
have
projects
and
within
projects
you
have
functions
and
you
can
create
function.
This
is
a
simple
function
that
does
like
NLP.
Let
me
increase
the
font
so
did
by
the
way,
has
an
embedded
vs
code
contribution
of
Microsoft
you're,
not
forced
just
to
work
with
the
embedded
code.
You
can
always
work
from
different
sources:
image
archives,
git,
repos
zips.
All
that
is
supported,
so
you
can
embed
nuclear
into
se
ICD
pipeline.
B
You
can
configure
many
different
things
and
you
clear
all
the
different
resources
you
know
of
in
kubernetes,
whether
it's
CPUs
GPUs
it
doesn't
have
to
start
from
zero.
You
know
you
in
data
science
or
streaming.
You
don't
really
want
to
start
from
zero,
so
you
can
choose.
What's
the
range
of
resources
it
can
work
with
any
type
of
kubernetes
resource
like
value
secrets,
config
Maps
labels
annotations,
you
could
give
it
the
source
image
that
you
want
to
build
on
top
of
it
the
function.
So
it's
not
like
very
limited.
B
You
can
essentially
say
do
things
like
on
builds
and
from
build,
and
since
you
put
your
entire
docker
file
into
descriptive
approach,
it
has
also
volume
mounts
and
PVC,
so
you
can
mount
functions
into
you
can
mount
data
and
storage
into
functions?
The
other
very
interesting
thing
is
the
triggering
so
essentially
not
just
triggering
from
HTTP.
You
can
trigger
from
a
Kafka
stream.
B
So
when
you're
trading
from
a
Kafka
stream,
the
nuclear
engine
is,
which
is
a
scale
up
scale
out,
a
load
balancing
engine
will
know
how
to
work
with
all
the
shards
on
the
stream
and
will
dynamically
if
it
needs
to
scale
it
checkpoints
rebalances
and
you
don't
lose
a
single
bit
during
that
process.
So
it
allows
you
to
auto
scale
streaming
and
that's
supported
across
again.
Very
large
variety
of
triggers
and
adding
a
trigger
is
very
trivial,
so
there
are
about
14
triggers
right.
B
Now
and
people
can
just
add
additional
triggers,
so
that's
more
on
the
on
this
front,
but
let's
look
at:
let's
look
at
it
from
more
of
an
adult
perspective,
the
right
one:
okay,
so
data
scientists,
don't
like
those
or
you
eyes
that
I
showed
you
they
like
Jupiter
notebooks,
and
what
they'll
do?
Let
me
show
you
a
simple
example:
this
is
the
same.
B
Nlp
function
that
I
mentioned
before
get
some
text,
Ransome,
processing
and
response
as
correction
translation
and
sentiment
analysis
very
few
lines
of
code,
so
you
can
essentially
put
some
magic
instructions
to
like
people
install
to
control
environment
variables,
control
all
sorts
of
spec,
nucleo
spec
elements.
I
didn't
show
that
but
nucleo
spec
is
essentially
a
CRD.
If
you
download
the
view
nucleo
spec,
it's
essentially
served
it's
essentially
a
CR
DT
kubernetes
element.
You
can
do
cube
cut'
all
functions,
you'll
see
that
that's
er
D.
B
But
again
you
can
just
do
all
of
that.
You
could
test
your
function
locally,
using
the
built-in
test
harness
and
you
could
then
just
magic
deploy
and
that
will
automatically
build
that
function
on
the
cluster
and
then
once
it's
done.
I
can
just
curl
into
that
function
and
see
that
it's
working
so
taking
that
and
let's
build
something
more
complicated.
One
of
the
nice
things
that
we've
done
recently
is
the
integration
with
cube
flow
pipeline.
B
So
this
example
in
in
the
cube
flow
website,
where
you
can
see
a
simple
example
of
essentially
taking
a
notebook
through
a
URL.
The
same
notebook
I
just
showed
you
running
a
component
of
deployment.
This
component
of
deployment
is
essentially
loading
the
notebook
extracting
the
code,
as
well
as
the
server
annotations,
the
magical
match,
to
figure
out
how
to
build
the
container.
B
Based
on
that,
it's
building
the
container
providing
me
logs
and
responding
with
then
address
for
that
running
function,
and
then
the
second
component
is
like
a
nuclear
invoke,
which
is
essentially
running
tests
against
the
container.
So,
if
I'm
going
to
run
this
in
a
pipeline
and
just
run
this
code,
what
I'll
see
is
that
the
first
step
of
deployment
is
building
a
container,
pushing
the
image
you
know
doing
all
of
that
and
essentially
responding
when
an
endpoint
and
and
the
response
is
the
invocation
response.
B
So
with
that,
let's
move
into
something
slightly
more
sophisticated.
So
let's
assume
we
want
to
build
a
full
pipeline.
A
full
pipeline
consists
of
data
ingestion
data,
preparation,
training
and
then
transfer
to
production.
So
what
I
build
here
is
four
different
functions:
okay,
like
preparation,
a
generator
is
essentially
simulating
a
real-time
feed.
The
preparation
is
taking
that
fit
and
building
joins,
aggregations,
etc.
On
the
data
the
training
is
using
is
essentially
creating
a
training
model
of
that
thing.
It's
using
by
the
way,
dusk
and
nucleo
and
dusk
and
cycle,
and
serve
an
extra
boost.
B
Now,
one
very
cool
thing:
I
did
here
in
the
definition
we
said
that
nucleus
supports
mounting
support,
shared
files.
So
there's
a
nice
thing
in
the
nucleus,
decay
that
essentially
allow
volume
teleporting.
So
I
can
tell
nucleo
that
my
volume
in
my
notebook
is
going
to
be
remounted
into
the
function,
which
is
what
I
did
here.
So
you
see
that
I
have
my
own
directory.
B
B
Ok
started
which,
when
I'm
doing
this
trick
is
essentially
taking
my
notebook
parsing,
my
notebook,
optimizing
it
extracting
all
the
things
that
give
it
configuration
like
this
volume
mount
or
by
OS.
You
see
it's
done
and
now
I'm
going
to
run
against
that
endpoint,
okay,
I'm
done,
and
what
I'm
supposed
to
see
if
I'm
going
to
refresh
okay,
you
see
another
file,
it's
pretty
cool,
so
I,
essentially
launch
created,
took
my
notebook.
Convert
my
notebook
into
a
container
ran
it
in
a
different
container
in
the
cluster.
B
I
gave
that
container
through
very
simple
annotation
I
told
it
you're
gonna
write
into
my
directory
so
or
my
direct
is
going
to
be
mounted
into
your
container
and
it
just
wrote
into
I:
don't
even
need
to
move
objects
around
now,
I,
just
refresh
and
I.
Have
this
data
I
can
actually
go
and
use
some,
so
I'm
utility
I
have
here
to
read
the
you
know,
parquet
and
it
sort
of
name
to
import.
B
Miss
garrison,
no
so
essentially
I
can
just
go
and
read
the
the
data
which
is
on
the
same
place.
So
with
that,
then
we
did
something
that
does
deployment.
The
next
thing
we
want
to
do
is
this
data
preparation,
so
same
thing:
I
want
to
take
those
parquet
far
so
I
have
I.
Have
your
instructions
for
a
nuclear
that
I
need
to
this
container.
B
To
sense
you
have
all
those
Python
packages,
I
need
this
I
want
it
to
be
based
on
Python,
Jessie
and
I,
won't
disturb
virtual
endpoint
virtual
mount
point
and
some
credentials,
and
you
know
things
were
dusk,
etcetera,
I,
then
I'm
running
all
this
thing,
I
can
run
the
same
code
locally.
We're
just
running
here
served
this
handler,
and
this
test
and
I
could
do
the
same
deployment
from
here
and
again
when
I'll
run
this
thing.
Let's
look
at
the
features,
no
features
if
I'm
gonna
run
against
the
real
time
function.
B
I'm
supposed
to
now
get
features
delivered
directly
to
me
and
I
could
do
exactly
the
same
thing
for
for
the
other
notebooks
by
the
way
it's
using
dusk.
Anyone
familiar
with
dusk
dusk,
is
sort
of
parallel,
Python
processing
library
that
supports
many
primitives
of
data
frame
of
a
panda's
data
frame
as
well
as
scikit-learn.
An
extra
boost.
One
of
the
nice
things
that
were
working
with
in
viña
now
is
support
for
the
GPU,
so
dusk,
kudi
F
is
the
server
equivalent
of
dinner
frame
and
qml
is
the
equivalent
of
scikit-learn
and
extra
boost.
B
I'll
just
generate
the
model
C
and
I'll
go
to
the
model
directory
and
hopefully
I'm
gonna
have
a
model
good
and
then
then,
let's
do
the
prediction
so
again
same
notebook.
Now
one
of
the
things
that
I
forgot
to
mention
on
the
on
the
first
one
that
you
see
this
is
commented
out
because
in
real
life,
I
don't
want
to
go
and
click
shift
enter
and
do
something
I
want
in
real
life.
This
generator
to
actually
work
in
a
cron
job
and
every
one
minute
go
fete
something
from
an
external
service
or
do
something.
B
So
if
I
uncomment
those
two
lines
just
sense,
you
say:
nucleo
config
spec
triggers
cron
with
ten
seconds.
It's
essentially
going
to
add
a
cron
trigger
to
my
function
and
my
function
is
actually
gonna.
Every
second
is
going
to
push
things
so
I
don't
have
to
just
think
about
pipeline.
That's
something
that
I
have
to
invoke.
I
can
put
things
like
streams,
cron
jobs,
HTP
endpoints
as
part
of
my
my
pipeline.
B
So
with
that,
the
next
thing
I
want
to
do
is
essentially
create
an
automated
pipeline,
because
I'm
sort
of
tired
of
messing
with
those
all
those
sort
of
shift
enters
so
I
can
essentially
write
this
specific
notebook.
What
it
does
essentially
loads.
Those
two
components
which
are
part
of
the
queue
flow
repository
actually
only
need
invoke
one
and
then
I'm
I'm,
just
daisy
chaining.
Those
different
four
functions,
all
the
way
from
data
generation,
data,
preparation,
training
and
any
inferencing,
okay
and
one
just
after
the
other
and
then
I'm
just
running
this
experiment.
Okay.
B
So
if
I'm
going
to
take
that
notebook
and
run
this
experiment
again,
then
supposedly
I'm
going
to
see
something
running
here
in
a
second
and
all
this
pipeline
that
I
described
that
is
going
to
run
automatically
okay.
So
you
see
I
got
the
first
step
running,
ok
writing
into
this
directory
and
then
the
step
second
step
is,
is
running
this
and
then
running
the
training
and
eventually
getting
the
inference
thing
done.
So
now,
if
I'm
gonna
go
back
to
my
my
thing:
let's
just
go
to
matrix,
you
see.
B
Another
file
was
generated
a
second
ago,
so
essentially,
what
I
did
I
didn't
know
anything
about
the
ammos,
nothing
about
docker.
The
only
thing
I
did
I
was
in
my
native
environment
of
notebooks.
They
just
put
some
annotations
and
saying
you
know
what
I
really
want.
My
volumes
to
be
mounted
potenti
I
need
the
trigger
of
cron,
maybe
Kafka
I
want
couple
of
GPUs
and
those
are
specifying
my
requirements,
and
this
is
automatically
generating
scale
out.
B
Okay,
one
thing
that
we
want
to
improve
and
we're
working
with
the
Jeremy
and
people
from
other
companies
is
to
try
and
and
create
sort
of
a
metadata
management
layer
within
queue
flow
that
will
simplify
greatly
those
notebooks
so
I'm
going
to
share
all
those
notebooks.
In
probably
will
pull
them
into
the
coop
flow
get
up,
so
everyone
can
play
with
those
again.
We
could
use
any
generic
resource
and
and
we're
hopefully
going
to
make
them
better
and
better
over
time.
B
B
D
B
Q
Placentia
q
po
is
both
what
is
Q
flow.
Q
flow
is
both
a
set
of
components
and
sort
of
sit
under
the
same
pile,
okay
and
also
a
set
of
unique
components
at
our
part
of
the
Q
flow
project.
Like
you
flow
pipelines,
Q
pipelines,
I
haven't
shown
all
the
details,
but
it's
very
powerful
because
in
q4
pipeline
it
also
has
experiment
tracking.
B
D
B
D
D
B
Yes,
I
know
again:
we
need
to
understand.
What's
the
data
like
data
leaks
is
where
you
you
put
your
raw
data,
correct:
okay,
yeah.
You
know
some
of
the
things
I
did
here.
We
have
essentially
our
own
high
speed
cluster
file
system.
So
everything
like
you,
move
really
really
fast,
so
I
click
refresh
and
it
works.
So
you
probably
have
a
fabric
which
is
within
the
cluster
for
scaling
the
job.
If
you're
scanning
a
job,
you
need
a
cluster
of
file
system
cuz,
they
all
work
on
the
same.
B
Okay,
you
probably
don't
want
to
go
to
a
dupe
or
s3
for
running
a
distributed
workload
on
the
cluster
okay,
but
your
raw
data
that
you're
going
to
learn
from
is
probably
somewhere
on
a
story
or
google
object
store,
so
the
the
first
step
is
usually
pulling
from
an
external
resource.
In
some
cases,
if
it's
real
time
data
or
swimming
data,
you
will
keep
it
on
kubernetes.
You
want
not
peel,
it
put
it
on
a
cloud
object,
store
and
then
you'll
have
a
working
set,
which
is
probably
on
the
cluster.
B
D
D
B
I
think
the
other
point,
which
is
very
powerful
on
cue
flow
I,
mentioned
those
components
so
to
go
to
the
cue
flow
pipeline
repo.
You
see
that
there
are
many
built-in
components
and
some
of
them
are
server
data
like
components.
Ok
like
if
you
go
to
Amazon,
you
add
EMR
or
athina,
which
are
essentially
server.
Data
like
so
Cooper
does
allow
you
to
integrate
with
a
cloud
provider
a
data
like
or
your
own
data
like
to
just
create
those
components.
There.
Ok,
ok,.
C
B
The
way
I
position
nucleo
against
gay
native
is
when
someone
asked
me:
can
you
compare
flour,
eggs
and
sugar
to
a
cake?
No
k
native
is
essentially
building
blocks
for
building
your
own
service.
Okay,
he's
K
native
equivalent
to
London?
Oh
okay,
that's
so
that's
that's!
The
first
thing
nuclear
is
a
fully
managed
has
API
is
SDKs
it.
You
know
as
load
balancers
orchestrators,
it's
a
you
eyes,
CL
eyes,
K
native
is
a
bunch
of
components.
Okay,
that's
first
thing.
B
Second
thing:
I
showed
that
nucleo
is
a
real-time
high
performance
engine,
all
those
things
usually
open
fast
and
all
those
who
are
CGI
I
call
them
CGI.
Why?
Because
they're
the
way
they
work
single,
threaded,
HTTP
request/response,
they
don't
have
streaming
built
in,
they
don't
have
checkpointing,
they
don't
have
parallelism
and
nuclear.
Has
all
this
notion
of
micro
spreading
inside,
so
nuclear
single
process
gets
to
400,000
events
per
second
one
process:
nucleus
things
like
GPU
and
GP
resource
optimizations.
It
knows
how
to
essentially
take
the
cuda
cores
and
working
against
them.
B
It
has
stateful
data,
you
know
integration,
so
nucleus
is
designed
for
data
processing
and
machine
learning.
Yes,
it
can
also
do
all
the
other
things
that
lambda
does,
but
the
positioning
is
really
more
wrong.
We
want
to
focus
on
you've
seen
the
integration
around
Jupiter.
You
don't
have
such
a
thing
in
any
of
the
other
service
frameworks.
So
it's
the
idea
is
we
want
to
take
the
notions
of
service,
which
is
essentially
no
DevOps
or
minimal,
DevOps
automation
and
all
that
into
sort
of
data,
science
and
machine
learning.
Thank.