►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Modern Data Science in a Cloud Native World - Samuel Kreter, Microsoft
We now live in 2018, where the meaning of Big Data keeps getting bigger. Yet, the tools most people are using with their data requires a huge amount of experience to understand and scale. We are also facing a time where it is necessary to track the flow of data for better understanding and compliance with GDPR. I am going to walk through how to take advantage of Kuberentes and other Cloud Native technologies with the open source project Pachyderm to create data science pipelines that are easy to develop, test, deploy and scale. I will also cover how to use Data Versioning throughout the process to track data changes and understand exactly how your data is changing. Talk Outline: 1. Introduce the basic concepts of Data Pipelines and Versioning. 2. Create and test a simple model. 4. Scale it up to a production sized workload and automatically have changes deployed in the pipeline.
To learn more click here: https://sched.co/FuJu
A
Cool
I
guess
we
get
started.
So
thank
you
guys
so
much
for
coming.
That's
aha
hongou
seem
wait,
doll
so
hi
today,
so
we're
gonna
talk
a
little
bit
about
modern
data
sciences,
they're
modded
data
science
and
I
of
science
pipelines
and
how
we
can
take
advantage
of
coupon,
IDs
and
other
cloud
native
technologies
to
make
this
process
a
lot
more
reliable
and
faster
C
cool.
Before
we
get
started
all
the
code
examples
everything's
gonna
be
on
my
github
I'll,
put
out
sway
there
I
tweet
about
it
too
and
stuff.
A
So
first
thing
is
the
process.
We
start
with
data
science.
We
have
a
business
need
or
problem
discovery
customer
issues.
Then
we
go
into
the
development
phase
we
do.
This
is
where
we
get
that
raw
data.
We
understand.
What's
going
on,
we
iterate
through
and
build
all
models
and
try
to
make
the
most
accurate
model
as
we
can
and
finally,
and
most
importantly,
in
most
cases
is
we
get
to
production.
We
hit
that
point
where
that
model
actually
impacts
the
user
and
it's
the
whole
process
so
we're
looking
at.
A
We
want
to
look
at
we've
gotten
pretty
good
at
this
development
phase.
There's
a
lot
of
tools
and
technologies.
We
have
now
to
go
either
if
we're
more
advanced,
to
go
deeper
and
deeper
into
our
data
to
have
a
full
understanding
or
on
lowering
the
barrier
entry.
So
anybody
can
take
advantage
of
machine
learning
and
data
science.
So
the
only
issue
with
that
is
this
is
in
the
entire
process
if
I
have
the
best
model
of
all
time,
the
most
accurate
model
that
doesn't
do
me
any
good
if
it
doesn't
actually
touch
the
customer.
A
A
Until
cloud
providers
came
now,
we
can
get
vm's
relatively
cheap
and
easy,
but
then
we
have
a
problem
of
managing
those
resources,
as
we
all
know
from
this
confidence,
that's
where
kubernetes
comes
in.
We
now
don't
have
to
think
about
virtual
machines
and
understanding
those
it
can
just
manage
those
for
us,
but
the
problem
is,
as
we
sell
from
that
pipeline
before,
and
data
science.
There's
a
a
lot
more
involved
in
that,
and
so
now
we
need
a
data
infrastructure
management
layer
to
help
go
through
that
pipeline
and
understand
the
data
going
through
it.
A
Number
one
is
a
data
version,
so
version
control
system
for
data
which
is
extremely
powerful
and
we'll
talk
about
that
in
a
little
bit
until
it
has
language
agnostic
pipelines,
which
means
I,
can
set
up
these
declarative
pipelines
just
like
a
pod,
spec
or
deployment
spec
of
the
state
that
I
want
and
then
use
containers
to
transform
our
state,
and
so,
let's
take
a
little
look.
A
common
example
of
one
of
those
pipelines
is
something
like
this:
all
pipelines
in
pachyderm
have
a
name.
A
Then
you
have
the
most
important
section
as
the
transformation,
so
this
transformation
section
we
give
a
container
we
conversion
this
container,
which
is
extremely
important,
and
we
also
give
it
a
command.
So
anybody
who
knows
pod,
specs
or
deployment
specs
this
should
look
very
familiar,
and
then
we
have
an
optional
section,
which
is
parallelization.
We're
gonna
go
deeper
into
this
later,
but
this
allows
you
to
specify
if
I
want
multiple
replicas
of
my
pods
running.
So
this
gives
a
lot
of
power
and
pipelines
to
be
able
to
do
multi
processing.
A
And
finally,
we
have
an
input
section,
and
this
allows
us
to
directly
figure
out
how
we
want
to
expose
our
data
to
our
containers
and
we'll
see
a
lot
of
the
power
of
that
later,
as
well.
So
now,
looking
at
a
pipeline
like
this,
it's
not
as
complicated
anymore.
We
have
to
have
these
manual
batch
scripts
to
set
up
and
manage
it
manually.
A
We
can
have
our
JSON
files
live
in
our
codebase,
alongside
where
our
infrastructure
is
then
reviewed
and
understood
by
our
team
going
forward,
so
something
this
pipeline
can
be
set
up,
just
like
Cooper
naze
deployments
in
seconds,
and
so
now
before
we
really
dive
deep
into
understanding
how
to
implement
these
pipelines
in
pachyderms.
We're
looking
at
for
data
science
principles
to
help
us
along
to
make
it
a
little
bit
more
reliable
in
our
processes.
So
the
first
principle
is
just
autonomy.
A
But
we
also
want
to
focus
on
being
able
to
manage
so
as
a
data
scientist
to
not
have
to
have
an
extreme
experience
in
data
science
as
well
as
infrastructure,
so
don't
need
20
years
of
experience
in
distributed
systems
to
be
able
to
set
up
these
good
pipelines.
At
the
end
of
the
day
we
have
specialized
skills.
We
want
to
be
able
to
focus
on
those
specialized
skills
and
doing
that
it
should
be
pretty
obvious.
Now
is
we
use
containers
so
containerization?
A
This
is
pretty
obvious
in
other
containers,
but
for
us
we
wanted
to
talk
about
so
if
I
have
a
model,
I'm
training
that
model
that
should
be
a
container
if
I
have
evaluating
that
model
and
Ramez,
we
want
that
to
be
a
separate
container
as
we'll
see
later
setting
up
these
pipelines
having
these
separate
containers
gives
us
a
lot
of
granularity
and
understanding
our
process
and
how
data
flows
through
that
process
and
number
two
number
two.
We
want
to
parameterize
our
data
inputs
to
that
container.
A
So
what
this
means
is
we
want
to
be
able
to
have
exactly
what
data
is
going
into
our
container
and
where
it's
pulling
from-
and
this
is
because
pachyderm
exposes
data
to
these
containers
and
something
called
the
packet
and
file
system.
The
PFS-
and
this
means
this
has
two
advantages
in
packing
number
one
is
when
I'm
testing
my
containers
locally
I'm
reading
from
a
CSV
or
a
local
file,
it's
pretty
common,
and
now
my
production
environments.
A
My
code
is
doing
the
exact
same
thing
is
reading
from
these
files
to
understand
what's
going
on
in
the
data,
so
that
means
instead
of
having
a
production
database
and
then
locally
I'm
doing
files
and
having
to
change
these
interfaces.
There's
a
lot
of
problems
prone
with
that,
where
this
means
my
local
development
environment
is
as
close
as
possible
to
my
production.
Environment.
Number
two
allows
us
to
take
advantage
of
that
parallelization
that
we
saw
in
our
deployment
specs
so
now
pachyderm,
if
anybody's
familiar
with
Hadoop,
are
these
MapReduce
problems.
A
A
So
now
principle
two
is
reproducibility
and
this
idea
of
having
data
versioning
and
another
one
of
those
really
interesting
concepts,
because
you
can't,
if
I,
asked
anybody
in
this
room
when
the
last
time
they
started
a
coding
project
without
get,
and
it's
it's
almost
crazy.
Yet
something
like
data,
that's
arguably
more
volatile
than
code.
A
A
I
can
now
branch
off
data
from
my
coworker
and
have
a
full
understanding
of
exactly
how
this
is
going
on
or
for
production,
as
I
can
do
different
sanity
checks
and
debugging
in
production
understand
exactly
how
my
dad
is
flowing.
I
was
talking
to
one
group
of
data
scientists
that
they
would.
They
were
doing
predictive
models
on
a
user
database
and
every
single
data
scientist
there
and
every
single
different
operation.
They
would
make
a
copy
of
that
database,
and
that
seems
if
this
is
something
that's
very
familiar
to
you.
A
This
is
something
that's
very,
very
hard
to
work
with.
Not
only
do
you
have
stale
data,
we
don't
know
how
long
ago
this
data
have
been
has
been
copied
from
the
main
source,
but
also
as
we
make
these
transformations.
What
transformation
has
been
had
to
that.
So,
when
you
build
these
models
off
of
that
data,
you
don't
know
how
it's
been
affected
or
how
it's
going
to
affect
your
in
the
production.
A
So
now
this
leads
into
probably
one
of
most
impactful
things
that
having
these
pipelines
and
having
version
data
is
it's
something
called
data
provenance,
and
this
is
essentially
when
you
have
a
version
control
data
posit
ori
that
feeds
into
your
pipeline
and
then
feeds
out
into
another
control.
I
can
now
track
that
data
through.
That
means,
when
you
have
a
predictable
result
at
the
ends,
so
I
have
a
prediction
for
my
model.
A
I
can
now
track
that
through
every
single
iteration
of
my
data,
all
the
way
back
into
the
raw
data
that
was
put
in
there-
and
this
has
some
amazing
effects
on
your
workflow
and
we
want,
as
I,
can
retry
after
failure.
So
now,
there's
pipelines
if
I
have
a
failure,
happen
I
know
I,
don't
have
to
send
it
through
the
entire
pipeline
again
setting
up
these
manual
processes
where
a
lot
of
cases
I
have
timing
between
my
data
now
I
can
just
pack
it
in
automatically
knows:
okay
that
failed.
A
Here
we
start
just
processing
the
new
data.
I
can
do
roll
backs
say
I
have
some
bad
data
that
gets
some
biased
data
that
gets
put
into
my
process
and
goes
to
my
pipeline.
Now
all
I
have
to
do
to
redo.
The
entire
pipeline
is
remove
one
commit
and
then
it
can
manage
going
through
the
entire
pipeline
flow
and
that's
extremely
powerful,
no
more
having
to
go
back
and
try
to
find
old
versions
that
were
saved.
A
You
can
now
just
do
this
automatically
into
your
CID,
CG
pipelines
and
finally,
but
most
importantly,
is
it
gives
clarity
into
your
organization.
So
now
you
can
see
exactly
how
data
is
affecting
your
organization
exactly
what
things
are
happening.
So
that
means,
if
I
have
bias,
I
can
track
bias
and
my
algorithms
and
everything
so
the
final
thing
we're
looking
at
is
automation
is
how
do
we
really
go
through
and
employ
these
data
science
pipelines
into
our
society,
CD,
workloads
and
cut
native?
A
All
these
workflows
number
two
is
reproducibility
that
we
can
now
track
exactly
the
changes
made
to
our
data,
which
is,
at
the
end,
a
very
our
most
important
thing.
As
data
scientist
and
machine
learning,
next
we
have
data
provenance
to
be
able
have
clarity
in
our
organization
to
have
a
full
understanding
of
how
everything's
happening
and,
finally,
to
automate
the
process
to
remove
that
manual.
Work
from
what
we're
doing
so
that,
let's
look
at
example,
so
we're
gonna
lose
a
very
common
data
set
the
iris
data
set
so
good
for
classification.
A
Anybody
who's
not
familiar
with
it,
there's
four
different
petal
lengths
or
features,
and
then
classification
can
we
do
hyper
pyramidal
tuning,
so
we're
gonna
have
our
all
data.
We
want
to
split
that
data
into
training
tests
build
a
whole
bunch
of
different
models
with
different
parameters,
and
then
we
want
to
evaluate
each
model
and
finally
select
that
model,
so
in
the
pachyderm
terms
or
whatever
pipeline
is
really
gonna
look
like
is
we
have
a
data
repository
for
all
data
that
can
allow
us
to
track
the
changes
that
we
make
and
how
we
add
it.
A
That's
gonna
go
into
a
pipeline
that
splits
us
into
training
and
testing
that's
going
to
go
into
another
pipeline.
That's
gonna
use
the
training
data
to
train
a
whole
bunch
of
different
models
with
different
parameters
passed
as
well,
and
those
parameters
the
same
way
or
version
control.
Finally,
we're
going
to
evaluate
those
models
and
then
select
the
best
models
based
on
the
threshold
and
anybody's
used
to
setting
up
these
kind
of
data
science
pipelines.
It
can
be
very
complex
and
there's
a
lot
to
go
into
it.
A
So
here
I
have
my
four
different
pipeline
specs.
We
can
see
so
here's
my
model,
all
I'm
doing
is
I
have
a
simple
Python
script.
That's
SVM
for
anybody's
curious,
we'll
go
deeper
into
that
in
a
minute,
but
I
loop
through
the
input
data
that
I
have,
in
this
case
my
raw
data
file
or
the
CSV
that
has
my
training
data
and
then
my
parameters
that
I
want
to
test
it
against
and
run
the
command.
The
most
important
thing
here
is
this
parallelization
spec.
A
So
in
Pike
return,
there's
two
ways
to
paralyze:
I
can
either
set
a
set
number,
so
maybe
four
replicas
or
in
this
case
I'm
using
some
quality
coefficient.
So
this
is
taking
advantage
of
the
resources
in
my
cluster
in
this
case,
since
I
have
two.
It's
gonna
run
two
instances
of
this
for
any
node
in
my
cluster.
In
this
case,
I
have
three
nodes
in
a
cluster
and
we'll
see
the
difference
of
that
and
then
I
expose
my
raw
data.
A
So
in
this
case,
I
want
my
training
data
every
single
container
and
then
I
want
to
partition
or
shard
my
parameters
of
which
parameters
to
go
and
so
they're
very
similar
for
the
other
different
pipelines.
Now
you
go
through
those,
but
all
this
required
to
set
this
up.
It
is
these
simple
commands
now
pachyderm
uses
very
similar
to
creeping
IDs
has
a
PAC
control
here.
All
we'd
have
to
do
is
create
our
repositories.
A
Then
we
add
the
data
in,
in
this
case
I'm
single
file
for
my
parameters
that
I
then
split
into
multiple
files.
This
just
helps
to
when
I
do
the
distribution,
z'
and
then
finally
I
create
the
pipelines.
So
now,
instead
of
having
each
manual
step
going
through
there,
that's
all
it
takes,
and
so
let's
take
a
little
look
at
actually
implementing
that,
so
there
I'm
just
running
the
script,
it's
creating
the
repository,
adding
the
data
and
then
finally,
it's
going
to
create
the
pipeline's.
A
So
that's
the
coolest
part
about
all
of
this
is
this
is
just
kubernetes.
So
all
the
tools
that
you're
used
to
with
tracing
and
we're
gonna
see
in
a
minute
Kubb
control
is
it's
just
spinning
in
pods
in
your
cluster.
So
if
you
want
to
go
deep
into
that,
you
have
four
control.
You
can
see
everything
in
your
cluster,
so
we
see
here
that
it
spins
up
a
different
pod
for
an
entire
pipeline.
Now
here
we
have
a
few
options.
You
can
either
have
it
these
pods
always
running.
So
your
pipeline
is
ready
to
go.
A
If,
if
you
have
some
streaming
data
or
quick
data,
that
you
need
a
constantly
process
or
you
can
have
a
setup
to
just
when
you
need
it
to
just
to
send
up
resources
when
it's
needed
and
then
spin
down
those
resources
with
a
not
so.
This
gives
a
lot
of
advantages
to
like
the
virtual
Kubla
project,
so
I
can
spin
off
those
resources
into
a
service
architecture,
and
so
once
we
see
those
pipelines
started
running,
we
can
use
the
PAC
control
to
start
looking
at
these
jobs
being
executed.
A
So
in
this
case
we
see
that
our
first
job,
the
splitting
training
testing
it
saw
that
there
was
data
added
to
our
raw
data
repository.
So
then
it
started
executing
the
pipeline
in
this
case
splitting
into
training
testing
data.
Next
it
started
to
train
our
models.
I
think
there's,
there's
77
models
that
it
shows
and
it's
going
through
and
actually
in
training
our
data
through
there,
and
so
it
detected
that
there
was
new
training
data,
and
so
it
executes
that
pipeline.
A
So
no
more
waiting,
we
have
seen
a
lot
of
data
science
pipelines
go
where
you
wait
eight
hours
or
try
to
make
it.
You
have
to
make
those
timings
because
of
the
version
control
it
will
automatically
track
through
there,
and
so
now
that
finished.
So
once
the
models
were
finished
and
added
to
the
model
repository,
it
automatically
kicked
off
the
evaluation
pipeline,
and
so
now
it
can
evaluate
all
of
those
pipelines
and
move
forward
and
then
once
that's
finished,
it'll
go
into
doing
the
selecting.
A
But
so
this
is
nice,
but
that
doesn't
take
advantage
of
the
automation.
This
is
still
very
I.
Don't
want
to
have
to
manually,
go
out
and
update
my
pipelines
each
time
so
now
I'm
gonna
go
through
an
example
of
being
a
developer.
In
this
case
you
don't
you
can
be
a
data
scientist
and
not
have
to
really
focus
on
crew.
Benes
concepts
or
anything
I
can
push
up
to
github
run
those
different
run
my
CIA
tests
and
do
everything
and
then
have
it
being
deployed
onto
my
cluster.
A
So
we
can
see
what
that
looks
like
so
here,
I
have
a
PR
with
some
small
model
changes.
Then
I
can
push
it.
It
automatically
triggers
a
build
and
my
CI
CD
pipeline
in
this
case
it's
going
to
pull
down
the
source
code,
build.
My
containers
then
push
that
container
to
my
container
repository
and
it
versions
these
through.
A
So
I
can
now
track
the
builds
in
the
source
code
exactly
what
calls
that
container
and
then
it's
going
to
do
all
their
it's
gonna,
replace
in
my
pachyderm
files
and
do
my
deployments
out
to
my
cluster
and
what's
really
cool
about
this
is
pachyderm
was
already
at
matok.
We're
gonna
know
that
the
model
has
changed,
and
so
it's
gonna
start
processing
base,
starting
at
that
point
in
my
pipeline,
so
it
doesn't
have
to
we
don't
to
reprocess
all
the
data
going
through
there.
A
A
To
be
a
little
faster
and
then
once
all
of
that
bill
goes
through,
we
can
do
those
releases
where
we
released
automatically
to
the
development
environments
and
going
through
this.
It
gives
us
a
much
faster
deployment
process
with
our
models.
Now
we
can
constantly
test
in
production.
What
we're,
what
our
model
is
doing,
what
it's
like?
A
Well,
what
confident
we
can
release
in
a
staging
or
production,
and
so
here
we
see
that
the
this
was
our
old
pipeline
that
was
going
through
it
succeeded
and
we
can
it's
automatically
detecting
the
changes
in
our
pipeline
and
executing
those
jobs
and
builds,
and
so
a
really
nice
thing
about
this
is
so
here's
the
Python
script,
they're
the
model
that
I'm
actually
using,
and
so
it's
an
SVM
I
take
in
my
parameters.
My
inputs,
like
we
talked
about
before.
A
We
want
to
really
parameterize
that
input
and
output
data
as
files,
then
I
simply
running
the
in
training,
the
SVM,
and
so,
as
a
data
scientist
I,
don't
really
have
to
understand
even
go
through
docker
and
those
things
yet,
I
can
push
it
up.
It'll,
train
and
it'll
build
my
docker
image
and
then
push
it
to
the
container
registry.