►
From YouTube: Building Cross-Cloud ML Pipelines with Kubeflow with Spark & Tensorflow - Holden Karau
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Building Cross-Cloud ML Pipelines with Kubeflow with Spark & Tensorflow - Holden Karau, Google & Trevor Grant, IBM
Data Science, Machine Learning, and Artificial Intelligence has exploded in popularity in the last five years, but the nagging question remains, “How to put models into production?” In this talk, we present KubeFlow- an open source project aims to answer this. This talk will examine how the intricacies involved in taking your pipeline and running it between clouds, mixing data from multiple sources, and building multi-component pipelines. We’ll examine how to tie together multiple tools to prepare your data and train the final model, as well as how to create a serving system to match. The audience will learn how to use kubernetes as a replacement for YARN simplifying your big data stack and empowering your data scientists to self-serve libraries and avoid being responsible for maintaining 20 different incompatible conda environments.
https://sched.co/MPaZ
A
Okay,
so
I
want
to
be
clear
that
this
is
an
intro-level
talk,
so
if
you're
like
an
existing
cube
flow
user,
this
is
probably
not
the
talk
for
you
and
there
are
many
other
fine
cube
flow
talks
that
will
show
you,
but
if
you're,
if
you're
new
to
cube
flow,
that's
really
great.
We're
really
glad
that
you're
here
and
we
would
also
love
your
email
addresses
so
that
we
can
contact
you
not
spam.
A
You
because
we're
in
Europe-
and
you
have
laws
about
that
stuff,
but
we
can
contact
you
in
accordance
with
local
regulation
to
let
you
know
when
the
introduction
to
machine
learning
with
Q
flow
book
is
ready.
Our
co-author
bailed
on
us
yesterday,
so
we
didn't
manage
to
write
it
at
the
pool,
but
I'm
feeling
really
good
about
writing
it.
Tomorrow,
I
realistically
over
the
next
few
months,
man
clicky
button
stopped
working
where's,
your
where's,
your
mouse,
okay,
cool,
okay,
cool
and
so
here
are
some
of
your
other
options
for
cubes
Oh
talks.
A
Some
of
these
are
the
past,
but
you
know
if
we
don't
scare
you
away
from
cube
flow,
definitely
check
out
some
of
these.
If
you're
more
experienced
with
cube
flow,
these
talks
could
also
be
a
better
fit
for
you,
because
this
is
as
mentioned,
intro
so
yeah.
We
are
building
cross
cloud,
machine,
learning,
pipelines
with
spark
and
tensorflow
and
is
gonna,
be
super
fun.
It's
designed
to
be
a
beginner
level.
A
You
know
when
you're
a
little
hungover
just
like
Who
am
I.
Where
am
I,
it's
really
great
I'm,
a
developer
advocate
at
Google
I
work
on
open
source,
big
data,
which
means
that
my
employer
pays
me
to
work
on
things
that
we
give
away
for
free.
It's
pretty
great
I
love
my
job
I'm
on
the
pachi
spark
pmc,
and
this
means
that
in
the
spark
project
it
is
be
really
hard
to
get
rid
of
me.
It's
like
tenure,
but
I
don't
get
paid.
A
It's
super
fun
if
anyone's
really
interested
in
open
sorts,
I've
started
doing
code,
review,
live
streams
and
they're
also
recorded
and
posted
on
YouTube,
and
you
can
sort
of
see
some
of
the
other
parts
of
the
open-source
process
that
you
might
not
have
experienced
before
and
I
think
they're
kind
of
neat,
because
I
do
them
mostly
in
Apache
spark,
which
is
something
with
thousands
of
contributors,
and
so
it's
a
bit
of
a
different
feel
than
sort
of
like
a
smaller
open-source
project.
If
you've
got
any
feedback.
B
My
name
is
Trevor
grant
I'm
from
Chicago
I.
My
preferred
pronouns
are
P
in
him.
I
do
various
odd
jobs
around
IBM,
we're
still
in
flux
about
what
exactly
it
is
that
I
do
there
I'm
also
a
PMC
on
a
few
Apache
projects,
moo-hoo
streams,
the
community
development
I'm.
Also
a
member
of
the
Apache
Software
Foundation.
B
There's
an
IOT
track
coming
up
at
Apache,
con
North
America,
then
I'm
help
there's
also
gonna,
be
one
at
Apache
cotton
Europe,
which
might
be
more
relevant
to
everyone
here,
but
I'm,
organizing
the
North
America
one.
So
please
submit
oh
wait.
No,
that
CFPs
come
please
come
to
Las
Vegas
in
September
and
see
our
IOT
track.
You
can
find
me
all
around
the
web
on
the
common
handle
of
rock
and
Trevo
blogs
websites.
Github
linkedin,
you
name
it.
Rockin
Trevo
been
doing
it
since
high
school.
B
So
in
this
part
of
the
talk,
we
normally
tell
several
jokes
about
coop
flow
I'm,
not
about
coop
fault,
about
data
scientists
and
we
kind
of
rip
on
them
for
a
little
bit.
This
is
a
35
minute
presentation,
we're
normally
you
seven,
maybe
forty
to
fifty
minutes,
so
we're
going
to
skip
through
them.
Really
quick
I
recommend
that
you
check
out
some
of
our
previous
talks
and
you
can
get
all
of
our
funny
jokes
about
all
of
these
things.
Gpus
make
fun
of
stuff
shout
out.
There's
a
very
good
graph.
Take
pictures
of
this.
A
B
B
A
Think
this
one's
me,
hey,
I,
don't
need
the
microphone,
that's
good!
Ok!
So
is
there
anyone
here
who
believes
that
cue
flow
is
magic
with
shipping
containers
and
tensorflow
okay
cool
when
I
do
this
at
strata.
There's
there's
a
lot
of
people
who
are
like
I
cube
with
kubernetes
flow
tensor
flow,
but
no
it's
not
just
that.
It's
a
whole
bunch
of
other
things.
It
is
the
Hilton
breakfast
buffet
of
machine
learning
platforms.
Do
you?
Have
people
don't
stay
as
husbands
here?
Okay,
okay,
Corral
buffet!
Do
you
not
have
buffets?
A
Okay,
whatever?
So
it's
a
it's
a
collection
of
different
things
that
we
can
choose
from
and
some
of
them
go
together
very
wonderfully
and
other
ones
like
the
shrimp
and
the
mango
when
you
wrap
it
in
ham,
maybe
starts
to
taste
a
little
funny
right,
so
we
can
add
all
sorts
of
things
together
and
we
can
build
something
really
wonderful,
but
we
can
also
build
a
monstrosity
which,
if
you've
had
a
job
you've
seen
a
lot
of
those
before
and
so
here's
some
of
the
components
that
our
makeup
cube
flow.
A
Everything
from
doing
pipelining.
There's
tools
to
do
data
preparation,
there's
tools
to
serve
your
jobs
like
TF,
serving
and
Selden,
and
then
there's
tools
to
move
data
around
and
and
all
of
those
fun
things,
and
you
can
train
many
different
kinds
of
models.
We're
gonna
talk
a
little
bit
about
tensor
flow
models,
but
but
not
too
much
and
you
can
do
anything
in
Python,
but
that's
kind
of
a
hack
because,
but
you
can
do
anything
in
Python,
it's
the
same
sort
of
thing
as
like.
Well,
we
gave
you
a
docker
container.
A
You
can
use
pandas
and
if
you
really
enjoy
writing
shell
scripts,
you
can
still,
of
course,
write
shell
scripts
to
do
your
data
preparation
and
then,
once
you've,
you
know,
thought
your
data
ready,
you've
trained
your
model.
You
can
persist
it
somewhere
right
now.
We
don't
really
know
where
you
can
put
it
in
a
cloud
object,
store
and
call
it
a
day.
I
like
underscore
final
underscore
real.
That's
how
I
know
which
model
is
in
production.
A
More
seriously,
you
you
you'll,
want
to
use
another
tool
for
model
tracking
and
unfortunately,
it's
more
or
less
on
at
this
point,
and
then
when
we
can
serve
it,
we
can
use
cells
in
core.
If
you
like,
using
things
that
explicitly
tell
you
not
to
use
them
in
production
in
production,
you
can
use
flask,
you
can
also
use
tensorflow
serving
or,
if
all
of
those
options
sound
like
not
a
lot
of
fun.
A
You
can,
of
course,
always
write
your
own
custom
logic
on
top
of
it,
because
why
do
something
that
other
people
have
done
when
you
can
do
it
different,
okay,
cool?
So
how
does
this
fit
into
the
data
science
ecosystem?
And
so
research
is
really
great,
and
if
you
can
convince
your
boss
that
the
deliverable
insights
of
your
job
don't
actually
need
to
deliver
anything
that
is
great.
Keep
that
job
for
a
long
time.
A
On
the
other
hand,
at
some
point
your
business
users
will
come
to
you
and
be
like
I
heard
you
had
this
really
good
model.
I
would
like
to
use
it
and
well
you
can
only
stall
them
for
so
long.
At
some
point,
they'll
make
you
put
it
somewhere
that
you
can
use
and
Kim
flew
in
to
solve
this
problem
of
allowing
us
to
put
our
models
somewhere
useful
in
production,
but
also
allowing
these
to
be
repeatable
so
that
our
business
users
aren't
coming
to
us
every
month
and
being
like
hey.
Can
you
update
your
model?
A
It's
still
recommending
Hannah
Montana,
that's
just
not
cool
anymore
right
and
then,
like
you
having
to
go
and
find
underscore
untitled
underscore
seven
dot,
a
Python
notebook
and
being
like
I
think
seven
was
the
good
one.
Eight
eight
was
a
little
rough
and
thank
you
so
much
people
in
the
front
row,
I'm
I'm,
so
glad
I'm,
imagining
that
you're
not
from
Europe
uh-huh.
A
So
thank
you.
Thank
you.
Anyways
and
so
q
flu
helps
us
bridge
this
gap
from
lab
to
production
it.
It
helps
us
get
our
machine
learning
pipelines,
repeatable
able
to
run
and
semi
reliable
with
less
shell
scripts
than
before
and
not
a
guarantee.
Don't
sue
me,
okay,
cool!
So
hopefully
we
didn't
scare
you
away
from
the
idea
of
using
cube
flow.
I
I
think
I
keep
going
here
right,
yeah,
I
think
so,
okay
cool!
So
what
are
we
gonna?
Do?
Oh
right
and
we
have.
A
B
A
First
before
you
just
started:
building
a
tensor
flow
model,
because
tensor
flow
sounds
cool,
okay
and
so
maybe
you're
like
you
know
what
holding
that
sounds
great,
but
I
just
want
to
build
whatever
model.
Everyone
else
is
doing
because
you
know
whatever
exploratory
analytics,
not
not
not
cool
enough
I'm,
not
gonna,
get
a
new
job.
Saying
I
wrote
some
Jupiter
notebooks
I
want
to
build
some
tensor
flow
models
for
you,
I
say:
yeah:
okay,
we
can.
We
can
go
ahead
and
do
this.
There
are
two
separate
entry
points
into
cube
flow.
A
A
Okay,
it
works
and-
and
that's
where
more
things
are
gonna
go,
and
you
can
see
here
that
there
are
multiple
platforms
supported
and
that's
one
of
them
important
things
right.
We
could
generate
something
without
any
platform
specifications
just
deploy
it
on
raw
kubernetes
or,
if
we're
going
to
deploy
on
top
of
GCP,
we
can
generate
a
GCP
aware
project
or
a
micro
software
project
and
it'll
do
a
whole
bunch
of
other
setup
work
so
that
we
can
use
things
beyond
just
the
normal
driven
Ilic
kubernetes.
A
For
us
we
can
generate
it
and
this
and
apply
everything
and
then
because
I
work
on
SPARC,
regardless
of
whether
or
not
you
actually
use
spark
in
your
pipeline,
you
should
install
it,
and
this
is
because
there's
a
tracking
thing
that
you
can
opt
into
and
you
should
definitely
opt
into
the
tracking
and
install
spark
so
that
it
shows
that
spark
is
useful
because
you
know
I,
like
my
job,
okay,
cool
and
so
what?
What
do
we
get?
With
the
chef's
recommended
pairing?
We
get
Jupiter
hub
to
do
our
exploratory
analytics.
A
We
get
some
tensorflow
job
stuff,
we
get
tensor
flow
serving
we
get
PI
torch.
We
get
some
hyper
parameter
tuning
tools,
oh
yeah,
okay,
and
we
get
something
to
route,
our
requests.
We
get
a
nice
web
UI
and
we
get
some
pipelines
and
the
pipeline's
are
really
just
our
go
with
magic
and
it
turns
out,
when
you
add
magic
on
top
of
anything,
you
get
a
nice
UI,
but
also
sometimes
it
doesn't
work
and
and
sometimes
doesn't
work
yeah,
that's
true,
okay,
cool,
and
so
this
this
is
you.
Do
you
remember
the
joke?
There.
B
Whatever
I
got
the
laugh,
so
I'm
gonna
I'm,
just
gonna,
drive
forward
with
it.
What
are
those
pipelines
so
again,
they're
just
a
thin
wrapper
over
Argo
you
get
a
nice
directed
a
slick.
Local
graph.
I
can
never
say
that
right
either,
but
it's
a
bunch
of
docker
containers
needs
performs
its
own
special
little
function.
B
B
One
that
maybe
makes
more
sense
for
an
actual
machine
learning
deployments
here.
Your
data
comes
in.
It
gets
validated
pre
process.
There's
some
training
you
didn't
employ
in
a
model,
there's
lots
of
different
ways.
You
can
do
this,
read
the
docs
or
read
Argo,
but
it's
a
little
out
of
scope
for
our
introductory
talk
to
get
to
deepen,
do
be
aware
that
pipelines
are
a
fun
feature
that
we'll
be
working
soon.
No.
A
Means
me
yeah,
I'm,
spark
cool,
okay
cool,
so
how
many
people
here
have
done
any
machine
learning
with
spark
wow?
That's
a
good
number
of
people,
I'm.
Very
sorry.
How
many
people
here
have
used
spark?
That's
slightly
more
people,
okay,
cool!
So
that's
lovely.
So
this
is
great
because
I
don't
have
a
lot
of
slides
about
spark,
but
that's
that's
cool.
So
in
pure
spark
we
can
do
our
data
prep.
We
can
pull
data
from
all
kinds
of
different
sources.
A
We
can
even
pull
it
from
multiple
cloud
vendors
right
like
if
we
have
some
data
in
redshift-
and
we
have
some
data
in
GCS-
and
we
have
some
data
in
BigTable-
we
can.
We
can
query
all
of
those
and
pull
them
together.
We
can
aaaghh
mentor
data
as
needed
and
there's
a
whole
bunch
of
tools
to
do
feature
preparation
in
spark.
This
is
really
nice.
We've
got
things
to
do
filtering
for
like
kicking
out
invalid
data.
A
We
can
fill
in
invalid
data
with
other
values
to
a
degree,
and
if
you
want
to
see
some
some
code
examples
of
how
to
do
this,
I've
got
something
in
a
project
called
Frank.
The
Unicorn,
which
in
case
you
can't
tell,
is
a
demo
project
based
on
its
name,
but
it
shows
you
how
to
do
is
like
sort
of
a
separate
data
preparation
stage
which
will
just
run
in
spark
and
then
once
we've
done
our
data
preparation
in
spark.
A
You
know
if
we
want
to
use
modern
things,
we're
gonna
end
up
wanting
to
use
other
tools
downstream,
and
so
we
can
use
things
like
tensorflow
ons
bark
to
just
keep
doing
our
tensorflow
job
inside
of
our
spark
job,
or
we
could
just
be
like.
You
know
what
spark
thank
you
very
much
for
your
future
preparation.
A
Please
save
the
results
to
my
favorite
object,
store
and
I'm
gonna
kick
off
another
job
and
keep
going
from
there,
although
that
is
kind
of
slow,
because
we
have
to
write
everything
to
disk
and
then
read
it
all
back
in
and
so
we're
gonna
talk
about.
Why,
even
though
the
first
option
sounds
really
good,
it's
not
because
happiness
is
an
illusion:
okay,
cool,
so
pi
spark,
which
is
the
Python
interface
to
spark,
which
is
what
we
need
to
use
to
get
a
tensorflow
users
pickling,
which
is
Python
serialization.
A
It's
not
very
good,
but
it
exists
and
not
only
does
it
use
pickling,
it
uses
pickling
from
Java,
which
indeed
you
can
do
poorly.
It
uses
sockets
UNIX
pipes
and
JSON
and
I
don't
have
a
funny
image
for
JSON,
because
Jason
is
not
funny
okay.
So
it's
a
Python
interface
of
spark,
it's
more
or
less
built
with
those
components
and
while
it
works
surprisingly
well,
it
has
some
surprising
or
actually
I,
guess
not
surprising,
there's
some
pretty
obvious
performance
challenges.
A
A
few
special
if
you
deploy
on
Windows
but
I've
never
done
that,
and
so
we've
got
our
workers
and
our
gonna
copy.
All
of
this
data
from
the
JVM
to
Python
who
thinks
this
design
sounds
fast:
yeah,
yeah,
okay,
thanks
thanks,
Trevor,
okay,
cool,
so
yeah.
This
means
that
when
we
do
this,
we
do
our
feature
prep
inside
of
spark
and
yeah
we're
not
saving
our
data
to
disk,
but
we
are
having
to
copy
it
from
the
JVM
into
Python,
and
this
is
really
slow.
A
Also,
incidentally,
if
anything
goes
wrong,
you're
just
screwed,
the
error
messages
are
completely
unintelligible.
You'll
get
things
like
pythons
actresses,
wrapped
in
Java
stack,
traces,
wrapped
in
I,
think
it
goes
Python
Java
Java
Python
is
the
normal
sort
of
stack,
trace
collection
that
you'll
get
and
then
you'll
get
between
0
and
K
times
2.
A
Where
K
is
the
number
of
workers
copies
of
the
same
stack
trace,
maybe,
but
they
may
not
be
different
when
we
perform
no
deduplication
for
you,
so
that's
great,
oh
and
incidentally,
if
you
go
over
any
memory
limits,
we
do
nothing
except
just
fail.
Your
containers
will
just
get
deleted
without
any
notice
cool,
so
maybe
you're.
Looking
at
this
and
you're
like
holy,
this
sounds
bad.
Why
are
you
telling
me
how
to
do
this?
I?
Don't
want
to
do
this?
Ok!
Well,
let's,
let's
continue
to
do
bad
things.
A
So
tensorflow,
one
spark
from
from
the
wonderful
people
at
Yahoo
I
believe
I
should
have
checked
that
last
night
anyways.
It
lets
us
run
like
our
tensor
flow
graphs
really
easily
on
spark
data,
and
this
is
this
is
more
or
less
what
it
looks
like
and
we
look.
You
know
if
we're
training
go
ahead
and
call
cluster
train.
This
is
really
convenient.
Really
nice
go
ahead,
run
our
training
job,
it's
just
really
slow
and
if
anything
goes
wrong,
we're
completely
screwed,
but
maybe
we
can
make
this
better.
A
But
to
be
fair,
don't
trust
any
vendor
like
we
all
want
to
sell
you
things,
although
to
be
fair,
this
is
open
source
software.
So
cell
is
a
very
fuzzy
concept
and
there's
this
graph,
which
implies
that
if
we
used
a
patchy
arrow
with
spark,
it
could
be
242
times
faster
and
if
you
believe
that
I
wish
I
had
your
optimism,
it's
not,
but
we
can.
We
can
use
Apache
arrow
to
make
the
data
copy
from
Java
in
the
Python
much
more
reasonable.
It's
a
lot
faster.
A
We
still
have
data
copy
happening
right
because
we're
going
from
JVM
heap
to
to
Python
heap,
but
the
format's
are
no
longer
just
like
completely
I
like
different
we're
not
having
to
translate
the
format.
So
it's
a
lot
faster
here,
not
242
times
faster.
In
my
experience,
maybe
20%
what
I
could
understand?
Maybe
they
lost
a
few
decimal
places
when
they
were
making
their
marketing
materials,
and
it's
actually
pretty
easy
to
swap
this
kind
of
code
around
so
inside.
A
The
core
part
of
tensorflow
on
spark
right
now
calls
for
each
partition.
Does
it's
magic
except
instead?
What
we'll
do
is
we'll
put
this
thing
in
here
and
we'll
give
back
a
bunch
of
zeros,
because
we
forgot
to
report
support
returning
nulls,
sorry,
but
it's
ok,
we'll
just
return
a
bunch
of
zeros,
and
now
our
code
will
be
up
to
242
times
faster,
yeah,
realistically
10
to
20
percent,
okay
cool
and
now
our
design
looks
someone
like
this
yay,
okay,
well,
I
thought
that
was
cool.
A
Now,
incidentally,
if
you
try
and
do
this,
you
may
run
into
the
exciting
opportunity
of
getting
to
manage
your
Python
dependencies
because
spark
looked
as
the
problem
went,
how
about
the
vendors
solve
it,
which
is
the
traditional
answer
to
any
hard
problem,
specifically
hard
problems
which
are
not
fun
to
solve
in
open
source,
so
for,
if
you're
running
spark
on
kubernetes,
we
actually
you
know
it's
pretty
reasonable.
You
can
specify
a
custom
container.
A
You
can
bake
your
requirements
into
that
container
and
then
everything
is
happy
if
your
data
scientists
are
happy
doing
that,
you
have
much
more
systems
background
data
scientists
than
I
tend
to
work
with
I
think
we
need
some
better
tools
here,
but
at
the
very
least
giving
them
a
shell
script
to
produce
a
container
for
them
to
run
on
is
a
lot
less
work
than
the
alternative
which
is
running
on
top
of
yarn
and
ok.
I
want
to
be
clear.
These
stars
mean
please
don't
sue
me
and
I
do
not
or.
A
Yeah
yeah,
no,
no,
no,
like
I'm
on
an
h-1b
in
America,
like
I,
do
not
want
to
have
to
find
a
new
job
and
try
and
transfer
a
work
visa,
not
in
today's
climate,
okay,
but
so,
and
and
these
are
not
the
official
positions
of
any
of
these
companies.
These
are
the
results
of
what
I've
heard.
Customers
have
been
told
from
interacting
with
them,
and
so
people
running
on
top
of
CDH
have
told
me
when
they
go
to
Clara
and
are
like
hey
I,
want
to
run
like
tensorflow
and
have
all
of
these
dependencies.
A
You
know
to
make
that
work.
They're
like
have
you
considered
the
many
benefits
of
buying
a
software
solution
from
someone
else,
preferably
with
a
markup
on
top
of
it
I'm
pretty
sure
I
don't
think
they
mentioned
the
markup
part.
But
you
know
it's
good.
Google
were
like
that
sounds
lovely.
Have
you
heard
of
shell
scripts
and
it
turns
out
when
you
tell
data
scientists
this
they
kind
of
best-case
scenario.
They
go
yes
and
leave
the
room
and
IBM
says
called
clever.
I
got.
A
Call
on
that
one
good
call
on
that.
One
turns
out
managed
yarn.
Oh
I,
don't
know
why
this
isn't
liquor
anyways,
but
thankfully
kubernetes
folks,
your
life
is
easier
and
you
can
avoid
having
to
maintain
200
different
anaconda
distributions
or
20
and
I.
Remember:
okay,
cool
often
when
I
tell
people
how
PI
SPARC
works
and
how
SPARC
works
was
tensorflow
they're
like
Holton
the
system
sounds
terrible.
What
should
I
use
instead
I'm
here,
to
tell
you
don't
worry
everything
is
garbage
now
different,
garbage
different,
but
not
better.
No,
no
very
similar
designs,
some
slight
differences.
A
Apache
beam
has
tighter
integration
with
the
tensorflow
tooling.
However,
it
doesn't
work
particularly
well
with
the
Python
versions,
which
is
what
you
need
for
the
tensor
flow
to
handling
outside
of
Google
Cloud.
So
it's
not
super
portable
they're
working
on
it,
but
also
from
a
data
copy
perspective.
It's
not
any
better.
They
use
proto's
instead
of
arrow,
but
that's
not
that's.
A
Just
like
you're
saying
that
I
run
on
diesel
instead
of
gasoline,
it's
not
a
big
difference
or
you
can
look
at
this
entire
mess
and
be
like
you
know
what
holding
all
of
those
things
sound
like
an
exciting
opportunity
to
buy
a
support
contract.
You
know
what
I'm
gonna
do
I'm
gonna
run
my
spark
job
and
save
the
results
to
disk,
and
if
you
decide
to
do
that,
I'm
very
happy
for
you
I
think
that's
probably
the
best
answer
to
put
something
in
production
in
the
short
term.
A
B
Yes,
yeah
there
we
go
sorry
I'm
messing
with
the
AV
people
by
always
flipping
my
mic
on
and
off.
So
there's
gonna
be
a
why
in
a
house
cloud-
and
this
is
the
part
of
the
talk
where
everyone
about
90%
of
the
people
down
in
the
vendor.
Hall
I
just
shoot
him
right
in
the
back,
including
both
of
our
bosses,
who.
A
B
Performance
is
not
tied
to
my
talk,
so
I
can
do
whatever
I
want.
Clouds
are
becoming
commoditized,
and
that
goes
well.
Yeah,
there's
clout
and
that's
because
a
big
part
of
kubernetes
kubernetes
is
kubernetes
is
kubernetes.
I
got
to
my
hotel
lobby
on
maybe
monday
night
and
another
IBM
er
was
down
there
and
we
were
both
hanging
out
at
the
bar
and
he
was
wearing
an
IBM
shirt
and
I'm.
B
But,
and
in
that
joke
Aires
because
kubernetes
is
kubernetes,
is
kubernetes
and
that's
great.
So
in
theory
the
IBM
kubernetes,
the
google
cloud
kubernetes
the
azure
kubernetes
they're
all
the
same.
It's
they
are
getting
there
there's
there
are
better
and
worse
implementations
currently
but
they're
all
getting
to
be.
You
know
just
the
same,
so
you
should
be
able
to
lift
and
shift
your
jobs.
B
That's
very,
very
good
for
consumers,
because
you
can
just
lift
and
shift
your
jobs
and
that's
bad
for
producers
for
reasons
we're
gonna
get
into
soon,
and
you
can
help
make
this
happen
a
little
sooner
and
we'll
also
have
a
slide
on
that.
So
commoditization
of
clouds
to
kind
of
dig
into
this
a
little
more
commoditization
is
when
there's
no
discernible
difference
in
the
product
examples
of
commodities,
if
you're
not
familiar
with
them.
Gasoline
corn,
soybeans
oil,
things
that
are
made
traded
like
commodities.
B
My
undergrad
is
actually
in
economics,
so
this
is
a
fun
time
to
use
my
undergrad
degree,
which
you
don't
always
get
to
do
in
your
real
grown-up
job
and
perfect
competition.
Producers
of
commodities
are
price
takers.
I
grew
up
in
the
central
United
States.
These
are
some
Iowa
farmers.
They
have
to
big
and
sell
their
corn
or
whatever
the
local
merchandiser
is
paying
that
day,
and/or
they
can
hold
it
and
try
to
maybe
it'll
be
worth
more
minus
the
cost
of
the
storage.
Any
ore.
That's
this
too
far
into
commodities.
B
That's
what
commoditization
does
and
it's
good
for
again.
It's
bad
for
I'm,
sorry,
producers,
but
it's
great
for
consumers,
because
if
you
have
jobs,
you
can
flip
on
your
computer
in
the
morning
and
say
whose
running
shoes
whose
kubernetes
is
cheapest
today
and
move
and
you're
already
kind
of
starting
to
see
this
happening
and
the
way
that
the
cloud
companies
are
fighting
back
is
by
having
these
like
in
very,
very
convoluted
pricing
agreements.
We're
like.
A
B
B
So
we're
we're
getting
that
thing!
If
you
looking
like
a
gas
station,
I've
noticed
in
Barcelona,
you
don't
have
like
multiple
gas
stations
on
like
the
same
corner,
but
in
the
u.s.
that's
a
very
big
thing
where
you'll
have
for
gas
stations
right
there
and
they're
all
charged
in
the
same
price,
because
if
somebody's
two
cents,
a
gallon
higher
nobody's
gonna,
go
to
the
gas
station,
they're
always
gonna,
go
to
the
other
one
and
you're
gonna
start
to
see
that
with
kubernetes
and
cloud,
which
is
great,
this
OP
the
opposite
of
vendor
lock-in.
A
B
Touche
its
monopolistic
power,
there
is
only
one
DB
I'm
picking
on
DB,
because
I'm
less
likely
to
I'm
more
likely
to
get
fired,
not
sued
by
IBM,
but
everybody
all
all
the
computer
companies
do
it.
Oracle
Teradata
Google,
AWS
name
a
cloud
company
they're
trying
to
lock
you
into
their
proprietary
solution,
or
they
have
been
there,
we're
starting
to
see
a
little
bit
more.
Our
vendor
neutrality.
And
that's
what
you
want.
B
You
want
a
vendor
agnostic
deployment
because
that's
gonna
save
you
the
most
money,
it's
a
little
bit
more
expensive
on
the
UPS
front,
but
will
save
you
money
later?
How
can
you
help
make
this
faster
by
designing
apps
that
are
designed
to
be
vendor
agnostic
and
then,
if
there's
a
problem
with
the
vendor
agnostics,
because
you've
got
enterprise
contracts,
throw
a
hissy
fit
at
the
sales
rep
and
like?
Why
isn't
this
working
when
it
doesn't
work?
Yeah.
B
B
More
reasonable
way
would
be
to
train
on
one
cloud
and
serve
on
multiple
clouds,
because
just
like
the
there
there's
commoditization
but
GPUs
will
have
different
prices
than
let's
say
memory
and
CPUs.
So
your
different
rigs
are
also
gonna,
be
differently
priced,
and
then
you
have
portable
pipelines.
B
B
A
Another
thing
that
that
I
think
is
is
a
relatively
real
use
case
is
making
your
pipeline
actually
across
blob.
That's
like
I
trained
here
at
the
end
of
my
training,
I
go
ahead
and
I
upload
it
and
I
serve
it
on
other
clouds
and
okay,
two
minutes:
okay,
hack,
es3
copy
final
psh.
That
sounds
like
a
really
quality
shell
script,
that
it
has
some
inline
Python
inside
of
it
and
then
turns
out
that
yet.
But
so,
if
you
click
on
this
link,
you
can
go.
A
Take
a
look
at
this
essentially
ad
add
an
extra
pipeline
stage
at
the
end
of
your
pipeline,
where
it
takes
the
model,
that's
in
the
persistent
volume
that
was
produced
and
uploads
it
into
s3
or
whatever
at
the
right
cloud
object.
Storage
is,
and
then
you
can
run
your
pipeline
on
KF
cloud
and
except
instead
of
training.
You'll
have
a
stage
which
downloads
the
trained
result
from
the
first
pipeline
on
cloud
1
and
we've
got
2
minutes
left
so
server.
It
was
seldom
great
if
thank.
B
A
And
so,
if
you're
interested
in
sort
of
playing
with
any
of
these
things
here
are
some
code
labs
that
I
think
are
relevant,
so
the
code
labs
are
things
that
are
much
better
organized,
but
they
don't
show
you
all
ready
to
cross
cloud
and
then
there's
this
random
github
repo,
which
has
a
readme
dot
MD
file,
and
the
tutorial
is
about
the
quality
of
what
you
would
expect.
But
there's
also
a
video
of
me
trying
to
do
my
own
code
lab,
which
is
hilarious.
A
It
does
not
go
well,
but
it
does
finish
and
we've
got
30
seconds
left.
So
ok,
there's
some
downsides
to
using
this
it's
out
of
5
software,
but
here's
the
DIY
workshop.
Let's
focus
on
you
should
buy
our
book.
It
does
not
exist
yet,
but
that
should
not
stop
you
from
giving
us
your
email
addresses
and
with
that
there
is
no
time
for
a
20-second
demo,
we'll
be
out
in
the
hallway
to
take
any
questions.
People
have
so
we
don't
get
in
the
way
of
the
next
talk.
Thank
you
all
for
coming.
Thank.