►
From YouTube: Bringing Your Data Pipeline into The Machine Learning Era - Chris Gaun & Jörg Schad, Mesosphere
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Bringing Your Data Pipeline into The Machine Learning Era - Chris Gaun, Mesosphere (Intermediate Skill Level)
Kubeflow is a new tool that makes it easy to run distributed machine learning solutions (e.g. Tensorflow) on Kubernetes. However, much of the data that can feed machine learning algorithms is already in existing distributed data stores. This presentation shows how to connect existing distributed data services running on Apache Mesos to Tensorflow on Kubernetes using the Kubeflow tool. Chris Gaun will show you how this existing data can now leverage machine learning, such as Tensorflow, on Kubernetes using the Kubeflow tool. These lessons can be extrapolated to any local distributed data.
About Chris
Chris Gaun is a CNCF ambassador and product marketing manager at Mesosphere. He has presented at Kubecon in 2016 and has put on over 40 free Kubernetes workshops across US and EU in 2017.
About Jörg
He is a technical lead at Mesosphere
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Let
seen
fear
how
many
of
you
have
actually
used
it
just
getting
a
sense
of
the
audience
like
four
so
yeah.
This
is
a
good
place
to
be.
If
you
haven't
used
cube
pillow
before
how
many
of
you
have
used
tensorflow
before
most
the
audience.
Okay,
great.
So
today
we're
going
to
talk
about
bringing
your
data
pipeline
into
the
age
of
machine
learning.
My
name
is
Chris
gone.
I
am
a
product
marketing
manager
at
mesosphere,
I've
been
in
the
kubernetes
community,
a
very
long
time
on
the
CN
CN
CN
CF
ambassador.
A
If
you're
interested
in
what
that
program
is,
if
you've
seen
that
machine
walking
around
with
that
face
on
it,
you
could
ask
that
device
and
they
could
tell
you
what
a
CNC
F
ambassador
is,
but
basically
we
advocate
for
CN
CF
technologies
like
kubernetes
and
core
DNS,
and
the
rest
of
them
that
we've
learned
about
in
this
conference.
Previous
to
that
I
was
an
analyst
at
Gartner.
I
covered
public
cloud,
particularly
public
infrastructure,
as
a
service
to
follow
me
on
Twitter.
B
Yuk
so
I'm
a
technical
community
lead.
What
does
that
mean
that
I'm,
basically
picking
up
community
projects
and
make
sure
that
they
work
well
with
Apache
missiles
and
then
DCOs?
So
when
we
start
up
new
projects
like
tends
to
flow
like
you,
flowin
end
up
with
me
and
then
I
work
with
a
community
of
getting
that
going
originally
when
the
cur
developer
of
Apache
missiles
but
yeah
more
recently,
I
really
enjoys
this
machine
learning
area
data
science
in
general,
so
I
believe
this
is
why
it's
a
good
food
to
preside.
B
Co-Presenter
is
Chris
here
and
yeah,
so
most
hands
actually
went
up
when
we
said
when
we
asked
like
who
knows
tense
flow,
but
maybe
just
to
recap
kind
of
CI
dear.
Why
deep
learning
right
now
is
becoming
so
popular.
First
of
all,
we
got
more
compute
power
and
we
got
more
data
and
that
actually
means
that
this
traditional
machine
learning
pipeline.
Imagine
we
want
to
have
this
classifier
darker,
CAD
or
hot
dark
non
hot
dog.
What
happened
in
traditional
machine
learning?
B
We
actually,
we
had
a
very
smart,
highly
paid
person
sitting
there
and
he
would
extract
features
and
with
his
features
and
could
build
like
an
SVM
or
some
kind
of
classification
mechanism
to
actually
classify
those
images.
And
then
we
had
our
output,
hot
dog,
non
hot
dog
dog
cat
with
machine
learning.
This
actually
requires
this
extra
effort
and
with
steep
learning
we're
replacing
this
human,
basically
with
stator
and
more
machinery.
B
So
instead
of
having
someone
who
also
had
might
have
a
bias
and
deciding
what
kind
of
features
you
might
want
to
extract,
we
actually
just
leave
that
up
to
the
machine
because
from
if
we
have
enough
data,
we
are
much
better
in
figuring
it
out
from
the
data.
What
are
the
best
features
to
actually
decide
this
dark
non
doc
question
here,
so
how
does
it
typically
look
like?
B
And
then
we
have
our
model
and
we
deploy
to
production,
so
we're
actually
getting
new,
previously
unseen
images,
and
then
we
search
them
and
already
in
this
slide.
This
is
something
when
I
look
at
conferences
or
when
I
look
at
blog
posts.
A
lot
of
people
focus
on
like
the
first
site
training.
What
are
the
best
images?
What
is
like
the
best
models
to
use,
how
do
I
train
best
and
how
to
optimize
my
hyperparameters
and
that's
all
very
important
but
like
in
production,
use
cases.
The
second
step
of
how
do
I
serve
this
model.
B
How
do
I
actually
builds
us
into
complete
line
is
at
least
equally
important,
tensorflow
of
almost
all
hands
to
end
off
so
I.
Don't
want
to
say
too
much,
but
it's
basically
a
library
often
used
to
build
deep
learning
models.
It's
not
only
deep
learning
models.
Actually,
it
can
also
do
more
traditional
machine
learning
and
the
nice
part
about
it
that
we
kind
of
have
an
abstraction.
B
So
it
can
optimize
my
learning
process
for
a
set
up
where
I
have
GPUs,
where
I've
CPUs
when
I'm
distributed
when
I'm
non
distributed
so
either
as
a
data
scientist
writing
my
model
I,
don't
have
to
worry
about
it
if
I
use
to
write
api's
if
I
use
the
right
api's
and
just
if
we
look
back
how
tensor
flow
has
been
trending
on
github
stars,
so
here's
is
first
when
it
jumps
up.
This
was
actually
when
Google
put
tensorflow
on
github
and
you
can
see
it
steadily
increasing
and
it's
really
an
impressive
growth.
B
I
was
just
at
developers
tensorflow
developer
summit
last
month
in
Mountain
View,
and
it's
really
impressive.
So
this
was
like
the
largest
conference.
I've
always
felt.
Q
Khan
is
large
or
skating
much
larger,
but
that
was
really
impressive
for
such
a
young
project
to
see
such
a
big
conference
coming
up
there,
and
you
can
also
see
like
if
we
take
get
up
stars
as
kind
of
a
metric,
how
it
compares
to
other
projects.
B
For
me,
honestly,
they're
really
good
other
projects
as
well,
but
it
simply
shows
that
there's
a
lot
of
developer
power
and
also
for
a
lot
of
users
in
in
the
tensorflow
realm
tensorflow.
How
does
that
look
like
if
I
do
that
as
a
single
developer
being
a
single
developer,
and
this
is
what
I
actually
really
like
about
it?
I
simply
downloads.
The
libraries
and
I
can
run
that
on
my
own
machine
and
listen
if
I
download
the
pre-trained
model.
B
But
if
we
actually
turn
over
and
we
leave
our
own
desk-
where
it's
simple,
but
then
we
actually
want
to
put
that
in
production,
it
becomes
a
bit
harder
because
we
all
these
problems,
which
would
like
non-existent
on
my
laptop
we
have
when
we
want
to
do
that
in
production.
So
we
have
to
consider
where
do
I
store
my
data
tip.
If
we
want
to
do
a
machine
learning
deep
learning
right,
we
need
datasets
which
don't
fit
on
a
single
machine,
so
I
need
some
kind
of
distributed.
B
Data
storage
I
need
to
prepare
data,
typically
datasets
I
get
from
anywhere
in
my
production
cluster
in
very
rare
cases,
I
might
use
them
directly,
but
often
I
actually
I
have
to
prepare
cleanse
them,
which
we'll
see
in
a
second
then
of
the
data
scientists.
Work
starts
actually
have
to
explore
the
data
and
once
I
have
an
kind
of
idea.
What
I
want
to
train
I?
Actually
I
need
to
kick
off
like
a
large
job,
utilizing
a
large
class.
B
My
data
set
is
large
and
even
in
the
distributed
set
up,
this
can
take
days,
and
while
this
is
running,
it's
really
important
to
keep
monitoring
in
mind
tensor
board.
It's
really
nice
to
monitor
kind
of
C
training
progress,
but
a
distributed
infrastructure.
It
already
kind
of
points
it
out.
I
also
want
some
monitoring
for
this
overall
infrastructure,
then
also
debugging,
tfd
Bach,
it's
nice
I,
really.
How
many
of
you
have
seen
that
it's?
Actually
there
it's
an
alpha
or
better
version
included
intensive
board
in
the
latest
version.
B
So
previously,
like
TF
debug
used
to
be
this
command
line,
saying
which,
for
us
it
was
helpful,
but
it
was
kind
of
annoying
to
use
and
in
the
latest
versions
they
actually
puts
that
into
a
tensor
board
as
well.
So
that's
a
great
step
for
debugging
forward
and
then
the
last
part,
and
as
mentioned
earlier
also
one
of
the
important
parts
actually
model
serving
is
when
I
puts
this
model
into
production
and
they're
a
bunch
of
challenges.
They're
like
how
do
I
get
my
data
in
there
like.
How
do
I
trigger
requests?
B
How
do
I
choose,
which
model
I
want
to
serve
I
kind
of
need?
Zero,
downtime
deployments
for
my
model
serving
after
all,
as
this
training
process
is
not
a
one-time
thing,
it's
actually
an
iterative
thing
when
I
get
more
data
when
I
refine
my
models
so
also
as
a
model
serving
side
kind
of
has
to
reflect
that
all
right.
B
So
with
that
the
typical
workflow,
if
I'm,
want
to
run
tensorflow
on
a
distributed
set
up
or
just
the
training
part
it
becomes
more
complex.
So
I
have
to
pick
a
number
of
machines.
I
have
to
install
tensor
flow
on
them
and,
if
I'm
not
using
the
right,
api's
I
actually
have
to
adapt
my
coaches.
That
I
would
actually
just
encourage
everyone.
Stick
to
there's
no
new,
a
high-level
api's,
because
they'll
automatically
be
able
to
distribute
your
code
they'll
be
automatically.
B
They
can
utilize,
also
TP
use
if
you're
running
on
Google
infrastructure,
so
it
kind
of
abstracts
away
from
where
I
have
to
write
my
own
entire
cluster
set
up.
But
even
in
that
scenario,
I
still
have
to
specify
the
class
respect
with
those
IP
addresses
for
my
distributed
setup
and
then
I
have
to
deploy
my
Python
code
or,
however,
I
wrote
it
onto
all
files,
and
then
I
can
finally
kick
off
the
training
and
also
like
down
here
in
this
node
setup.
There
are
different
types
of
nodes:
I
might
have.
B
Worker
nodes,
I
might
have
parameter
servers,
so
it's
a
kind
of
art
of
how
to
set
it
up
efficiently.
Is
that
I
get
the
best
output
out
of
my
cluster
so
and
distribute
a
job
running
for
days?
What
do
we
expect
they're
going
to
be
failures
and
that
can
be
depending
on
the
failure
type.
This
can
be
rather
annoying,
as
if
I
have
my
hard-coded
cluster
spec
with
all
those
IP
addresses.
B
I
might
actually
have
to
restart
the
job
simply
because
nodes
have
gone
down
and
if
half
the
notes
are
down,
my
performance
is
gonna,
be
pretty
bad.
This
is
actually
these
failures.
This
is
actually
true
for
any
distributed
system,
and
you
should
keep
that
in
mind
with
also
the
other
big
data
tools
we'll
be
talking
about
here
was
that
I
would
actually
hand
over
to
or
hand
back
to
Chris.
Actually,
who
is
going
to
talk
about
the
entire
pipeline
from
start
to
end
yeah.
A
They
often
had
some
magic
behind
it,
and
it's
some
of
these
external
parts
that
you
would
have
that
may
be
part
of
a
public
cloud
or
something
so
I
just
wanted
to
go
through
the
entire
pipeline,
so
that
you
could
be
successful
in
setting
up
from
your
database
all
the
way
to
your
model
serving
so
in
a
typical,
deep
learning
pipeline.
You
have
your
data
and
streaming.
This
is
where
your
housing,
your
data,
it
could
be
in
the
public
cloud
or
it
could
be
in
an
open
source
tool.
A
There's
several
ones
that
we'll
cover
today
that
are
extremely
popular,
and
then
you
have
your
users,
so
in
one
of
the
tools
that
we'll
investigate
some
of
this
is
given
for
you,
but
there's
other
users,
preparation,
type
software
that
you
can
use.
There's
also
spark
which
we
saw
today
during
the
keynote
and
that's
a
good
way
to
do.
Data
preparation
and
we'll
have
a
live
demo
of
that
as
well.
Some
data
preparation
that
we're
doing
before
we
ingest
it
into
into
the
tensor
flow,
and
then
we
have
the
frameworks
and
the
clusters.
A
So
the
cluster
is
obviously
kubernetes
that
it's
landing
on
and
there's
many
different,
deep
learning
frameworks
out
there
and
then
the
models
and
the
model
serving
so
in
kubernetes
world
there
is
a
tool
called
cube
flow.
A
few
of
your
hands
went
up.
You've
used
this
before
this
allows
you
to
get
kubernetes
up
and
running
very
easily.
I'm
sorry
tend
to
flow
up
and
running
very
easily
on
kubernetes.
A
So
what
does
it
include?
It
automates
the
distribution
of
deep
learning,
and
so
it
installs
tensorflow
on
kubernetes,
with
that
it
also
includes
jupiter
notebooks.
It
currently
works
only
on
kubernetes.
If
you
have
other
distributed
systems,
it
won't
work
on
that,
but
it's
open
for
commits
and
then
it
requires
to
get
it
set
up,
requires
some
knowledge
of
case
on
it.
It
is
a
tool
for
packaging
and
deploying
to
kubernetes
so
I,
you
know,
I'm
a
marketing
guy
and
I
was
able
to
use
it
in
a
day
and
figure
it
out.
A
A
The
solution
are
these
open
source
tools
or
some
public
cloud
options
that
are
available
like
HDFS
Kafka
for
stream
and
Cassandra
for
distributed
databases,
so
these
are
all
I
would
say
in
the
big
data
market,
which
has
been
growing
since
the
first
inception
and
I
would
say,
mid,
2008,
2007
type
timeframe
and
why
have
them
been
growing
because
we're
creating
more
and
more
data
every
day,
there's
also
large
and
existing
data
streams
that
we
need
to
connect
to.
So
this
is
a
lot
of
times.
A
You
want
to
use
something
that
you're
familiar
with
when
you're
working
with
10
to
flow
you've
already
used,
HDFS
or
you've
already
use
Kafka,
and
you
might
not
necessarily
want
to
go
to
one
of
the
public
cloud.
Ap
is,
and
their
proprietary
tools
also
there's
often
strict
data
governance
around,
especially
in
European
companies.
I
was
actually
talking
to
a
European
company
the
other
day
at
cube
con
and
as
soon
as
they
analyzed
the
data
with
their
deep
learning
tool.
A
They
actually
had
to
erase
it
and
that's
because
the
the
government
actually
doesn't
allow
them
to
keep
around
the
results,
because
it
is
customer
data,
so
data
preparation.
So
if
you're
doing
a
demo
data
set,
it's
almost
like
a
hello
world,
it's
very
easy
to
do,
but
that's
the
exception
to
the
rule.
Oftentimes.
You
have
to
do
a
lot
of
data
cleanup,
the
time
stamps
might
be
off
or
you
might
be
collecting
it
at
different
frequencies.
So
you
have
to
match
it
up.
A
A
I
believe
our
demo
specifically
is
on
flink,
so
we'll
see
doing
some
of
that
in
in
our
demo
and
then
model
management,
so
the
ability
not
just
to
create
the
models
but
actually
whole
distribute
them
and
actually
get
them
out
to
your
clients,
and
so
there's
several
open
source
tools
for
that
as
well.
We
already
covered
HDFS
MongoDB
is
another
one,
and
this
is
a
it's
something
that
we
will
demo
as
well.
A
Meses
has
a
long
history
of
working
with
these
big
data
frameworks
in
very
large
environments,
thousands
of
No
and
mission-critical
type
organizations,
and
so
what
missus
does
is,
though,
almost
treats
your
infrastructure
like
kubernetes
treats
pods.
So
you
could
say,
I
want
this
amount
of
infrastructure
and
it
gets
it
up
and
running
and
then,
if
some
of
the
infrastructure
goes
down
it
actually
in
a
declarative
model
checks,
oh
the
desired
state
does
not
match
the
reality
of
the
state
of
what
I
want.
A
I,
don't
have
enough
kubernetes
worker
nodes,
for
instance,
and
it
could
easily
replicate
and
spin
up
another
Cooper
Nettie's
worker
node,
and
so
there's
a
lot
of
cumin
container
management
solutions
that
work
on
on
mesos
kubernetes
is
one
of
them.
Marathons.
Another
is
one
that
has
been
released
two
weeks
ago
by
titus
I'm,
sorry
by
Netflix
called
Titus
and
I
actually
use
mesos
to
spin
up
200,000
clusters
of
titus
per
day.
I
have
no
idea
why
they
need
200,000
orchestration
clusters
per
day,
but
that's
what
they
said
that
they
do
so
with
the
Misses
architecture.
A
The
way
that
it
works
is
that
we
have
an
agent
on
each
one
of
these
nodes
and
they're,
offering
resources
to
all
these
different
schedules
who
have
their
own
way
of
doing
things.
So
kubernetes
scheduler
is,
you
know,
scheduling,
pods
in
one
way,
then
SPARC
schedulers
doing
it
in
another
and
in
Kochba
is
doing
in
another.
So
they're
asking
for
resources
and
the
massive
scheduler
says:
oh
okay,
this
is
kubernetes
needs
six
worker
nodes,
so
it
gives
it
the
amount
of
CPU
and
memory
that
it
needs,
and
with
that.
B
Devil
can
I,
just
maybe
at
like
one
more
sentence
here.
So,
as
you
mentioned,
like
missus,
is
treating
all
those
systems
like
kubernetes
is
treating
your
pods.
So
this
is
why
it's
just
like
two-level
scheduling,
so
Maysles
is
kind
of
easier
scheduling.
Missus
only
cares
about
resources,
but
then
the
way
in
which
different
systems
like
human
at
ease,
like
HDFS,
need
to
treat
tasks,
there's
actually
a
second
level,
and
this
is
very
application
specific.
B
So,
as
you
mentioned,
HDFS
needs
a
totally
different
treatment,
and
this
is
first
of
all
when
deploying
it,
but
more
importantly,
when
they're
actually
failure
scenarios,
if
a
pot
and
your
next
pot
fails
I'll,
simply
reschedule
that
and
I
don't
necessarily
care
where.
But
if
my
HDFS
data,
no
failing
or
if
both
my
HDFS
name
notes,
are
gone,
I
actually
needed
more
complex,
a
recovery
step,
and
so
this
is
kind
of
what's
those
schedulers.
If
you
want
to
compare
them
to,
operators
like
operators
for
distributed
systems
are
pretty
good
at
all
right.
B
We
see
demo
so
when
being
on
the
cloud-
and
this
is
actually
where
cube
flow.
I
really
like
it
is
when
I'm
running,
for
example,
on
Google
infrastructure.
So
when
running
on
Google
infrastructure,
I
have
actually
managed
spark,
I
can
use
for
data,
cleansing
data
preparation,
and
so
that
might
read
from
cloud
storage.
It
will
read
my
date
her
and
it
will
just
might
write
it
back
into
the
cloud
storage
here
then
I
can
use
my
typical
cue
flow
workflow,
so
I'll
spin
up
a
Jupiter
notebook
I
then
explore
my
data.
B
I
will
train
my
model
using
tenza
flow.
I
can
use
tensor
boards
to
actually
monitor
the
progress.
I
can
use
TFT,
Bach,
2d,
debugging
and
then
there's
actually.
There
you've
got
like
two
options
in
queue
flow
itself
for
serving
so
I
then
can
surf
my
models,
and
so
once
I
have
my
model
serving
up
they're,
actually
in
the
way
of
hitting
a
tryst
request.
So
there
are
also
different
streaming
solutions
in
those
cloud
settings.
Now.
The
question
is
just
what
happens
if
I'm,
not
in
one
of
those
cloud
settings?
B
What
can
I
do
so
if
we
are
open
source
and
also
if
we
want
to
be
independent
of
those
cloud
tools,
the
typical
solutions?
I
see
out,
there
are
spark
for
a
data
cleansing,
so
spark
I
typically
see
that
if
have
like
batch
workloads.
So
if
I
already
have
my
data
stored
in
HDFS
and
then
I
just
flows
through
it
and
HDFS,
it's
a
very
scalable
storage
and
then
for
streaming
Kafka.
A
very
good
solution
for
is
actually
streaming
the
request
and
then
hitting
my
attend
surf
low
serving
there's.
B
Actually
something
has
anyone
heard
of
C
rendezvous
architecture
yeah,
so
this
is
actually
by
Ted
Dunning
from
a
mapper,
and
this
is
actually
heavily
relying
on
kafka
boughs
for
hitting
different
serving
end
points.
So
the
question
is:
what
do
I
do
if
I
have
different
models
and
I
don't
actually
know
which
one
gives
me
the
best
result,
because
this
training
process
is
mentioned,
it's
iteratively,
so
I
end
up
with
like
many
hundreds
of
models
and
as
I
need
to
pick,
which
ones
are
best.
B
B
So
this
down
here
this
is
basically
cue
flow
and
it's
very
powerful
and
very
helpful
and
everything
on
top.
This
is
then
up
to
you
what
you
want
to
use
depending
on
your
infrastructure.
If
you
are
an
AWS,
you
can
use
the
AWS
tools
if
you're
in
Google,
you
can
use
the
Google
tools
if
you're
on
Prem
you.
B
Actually
you
need
to
come
up
with
alternative
solutions,
so
this
is
kind
of
the
open
source
version
of
this
which
might
even
if
you
consider,
even
though
you're
running
in
the
cloud,
it
might
be
a
good
choice
simply
because
you're
not
tied
to
particular
cloud
vendor
and
yeah.
This
brings
us
to
the
demo
and
for
this
demo
I
said
there
are
many
many
options,
but
as
we
are
working
on
Miso's
and
TCS,
we
kind
of
choose
that
for
that.
So
how
much
time
do
I
have
left
five
minutes
with
questions?
B
Okay,
so
here
we
actually,
we
have
a
cluster
where,
like
all
those
tools
running
in
on
AWS
in
this
case,
but
this
can
also
be
an
on-prem,
so
I
actually
have
a
system
where
I
have
kubernetes
running.
This
is
pure
kubernetes
and
if
we
look
at
that,
I
already
deployed
a
cute
flow
on
it
as
it
takes
a
while
it's
in
another
namespace,
but
this
is
just
the
open-source
kubernetes
queue
flow.
B
So
here
I
can
see.
This
is
my
default
very
simple.
It's
a
getting
started
acute
flu
example
which
I
deployed
here
and
it's
not
so
bad
which
case
on
it,
and
then
we
have
HDFS,
which
is
holding
a
some
of
our
input
data.
We
have
captive
streaming
flink
for
also
data
cleansing,
so
so
actually
there,
let
me
just
switch
back
to
the
slides
from
one
second.
B
So
if
you
recall
this
picture
here,
SPARC
is
really
great
if
I
have
like
a
batch
processing.
So
if
I
have
my
data
on
HDFS
and
then
I
run
a
batch
job,
I
cleanse
it
and
prepare
it
for
my
workflow,
but
often
I
even
have
like
a
stream
of
data
coming
in.
So
if
I
have
this
event
stream
coming
in
and
then
I
want
to
store
this
data,
what
I
also
seen
people
do.
Is
they
really
use
like
a
streaming
system?
Spark
could
do
that
as
well.
B
But
flink
is
my
personal
favorite
there,
but
personal
choice,
so
they
actually
use
flink
and
they
pre-process
the
data
and
then
they
actually
write
it
into
HDFS
here,
okay,
so
having
that
set
what
what
I
would
do
now,
I
would
actually
just
deploy
a
spark
job
to
cleanse
my
data,
so
it
would
will
read
from
HDFS
and
right
back
to
HDFS
and
I
have
my
cheat
sheet
here.
So.
B
B
B
So
if
we
now
have
a
look
at
the
spark
UI,
we
should
be
able
to
see
that,
hopefully
our
data
cleansing
job
has
been
running
successfully,
so
it's
still
launching
so
those
of
you
who
know
how
spark
works.
It's
launching
our
driver
here
and
still
processing
our
data,
but
we'll
just
leave
that
open
and
continue
with
our
demo.
So
far,
next
step
is
we
actually
we
want
to
access
as
the
default
cube
flow,
and
for
that
it's,
our
nice
port,
forwarding.
B
B
And
here
we
come
up
with,
like
our
Jupiter
hop,
how
many
of
you
know
Jupiter
hop
so
so
Jupiter
notebooks
as
Chris
mentioned
earlier.
They
are
like
a
great
way
for
data
scientists
to
kind
of
explore
their
data
sets
I
mean
if
I'm
a
data.
Scientist
I
want
like
an
easy-to-use
interface
and
Jupiter.
Hop
is
a
way
of
easily
spawning
so
Jupiter
notebooks,
and
here
we
go
and
so
I
already
spawned
one
like
spawning
a
Jupiter
notebook.
It's
a
huge
docker
image,
I
think
it's
like
a
10
gigabyte
darker
image,
so
it
takes
a
while.
B
B
B
So
this
is
the
default
amnesty
example
and
what
it
does
it
actually
downloads
the
dataset
from
from
the
internet.
It's
like
the
default
tensorflow
one,
but
we'll
change
that
in
a
second
or
depending
on
how
much
time
there
is
left
five
minutes
with
questions,
we
can
easily
change
that
to
also
move
it
somewhere
else
and
so
executing
this
now.
B
So
this
takes
a
while
to
run
and
yeah.
Those
warnings
are
accuracy
example,
but
it's
the
official
example
I
wanted
to
stick
to
the
official
queue.
For
example,
it's
using
a
deprecated
you
eyes,
but
don't
be
alarmed
by
this.
It's
just
at
a
precaution,
warnings
and
one
thing
why
I
personally,
we've
been
talking
about
tensorflow
quite
a
lot,
but
actually
there
are
other
tools
which
makes
this
easier.
So
one
thing
which
is
a
little
annoying
for
me
with
tensorflow
up
until
very
recently,
is
like
the
sessions
run.
B
So
what
tensorflow
does
I
actually
I
built
a
very
static,
compute
graph,
and
then
I
run
this
compute
graph
over
and
over
again
in
the
1.7
versions
they
introduced
eager
execution
and
other
tools,
for
example,
pi
torch
I
really
like
that.
As
a
feature
of
Pi
Church,
they
actually
allow
me
to
do
that
on
the
fly
so
I
can
actually
computer
get
my
my
entire
define
my
entire
graph
on
the
fly,
which
means
I
can
also
output
after
each
step
and
I
can
debug
after
each
step
and
yeah
it
ran.
B
We
have
an
okayish
for
this
number
of
iterations
output
accuracy,
and
now
this
takes
a
bit
longer
so
I
now
I
can
basically
switch
in
switch
in
my
HDFS
back-end,
and
this
is
very
important
for
it
tend
to
flow
for
multiple
reasons.
So
there
are
actually
three
spots
in
this
program
why
I
might
want
to
have
set,
and
first
of
all,
it's
for
import.
So
what
we
do
here.
B
This
is
a
very
small
data
set
which
has
been
downloaded
in
choose
the
darker
image
running
Jupiter
here,
but
if
I'm
going
at
more
scale,
I,
actually
I'd
probably
want
to
resit
from
HDFS
in
my
cluster,
the
second
thing
is
sends
the
model
which
are
output
and
I
write
somewhere,
and
these
searching
and
most
important,
probably
or
others
are
also
important,
but
also
very
important
thing
is
the
metadata.
So
how
many
of
you
have
used
tensor
board
so
tensor
board?
It
needs
somewhere
from
where
to
read
those
checkpoint
files
and
they're.
B
Hdfs
is
a
great
option,
so
just
when
I
need
some
kind
of
distributed
file
system
where,
while
this
job
is
running,
it
can
actually
write
those
metadata
checkpoints
and
then
from
where
tensor
board
can
read
them
and
later
on.
Also
other
tools
can
read
this
metadata
to
visualize
that,
as
we
only
have
two
minutes
left,
I
would
actually
go
over
to
questions
I'll
be
around
and
I'm
happy.
So
the
next
step
is
basically
we
exports
it
to
HTS.
B
A
And
the
slides
will
be
online
if
you
look
under
the
talk,
I
believe
they're
there
and
all
the
steps
are
in
there.
Everything
that
you
saw
is
all
open
source.
So
if
you
want
to
go
and
do
it
yourself,
we
could
probably
put
these
commands
in
there.
We
don't
have
them
right
now,
but
that
should
be
simple
to
do.
I'll
put
them
in
the
deck
after
I'm
done
so
are
there
any
questions
on
getting
tend
to
flow
up
and
running
and
kubernetes
any
of
the
open
source
tools
that
we
saw?
Yes,.
B
So
for
production,
or
the
workflow
I
typically
see
is,
is
you
go
in
your
data?
Scientists
actually
train
that
they
explore
so
Jupiter.
For
me,
it's
more
of
like
for
exploring
data
and
getting
a
feel
for
the
model,
but
once
you
have
sent
a
production
scenario,
you
should
also
have
this
definition,
this
pison
file.
This
should
be
somewhere
versioned
in
github
or
whatever
artifactory
version
storage.
You
have
and
then
you're
then
you're
actually
a
firing
off
that
job
against
the
cluster.
B
So
also
this
example,
you
can
get
their
tricks
in
which
you
can
get
Jupiter
against
a
distributed.
Cluster
in
this
scenario
is
actually
just
using
the
resources
inside
the
container.
You
can't
connect
it
against
an
external
cluster,
but
like
for
the
large-scale
learning
you
actually
you
want
more.
So
if
you
go
back
to
this
pipeline,
like
this
user
is
just
like
the
initial
and
analytics,
but
then
once
we
had
to
framework
in
cluster
level,
you
actually
you
wouldn't.
Typically
you
wouldn't
launch
that
from
Jupiter
directly.