►
From YouTube: Compliant Data Management and Machine Learning on Kubernetes - Daniel Whitenack, Pachyderm
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Compliant Data Management and Machine Learning on Kubernetes - Daniel Whitenack, Pachyderm (Intermediate Skill Level)
Data scientists, machine learning engineers, and researchers are under increasing pressure to provide explanations for how they are processing and managing user data. In particular, the EU's GDPR regulations taking effect this year are forcing organizations to rethink their data management and processing strategies. In this talk, we will demonstrate a data management and processing methodology/framework that is helping organization deploy compliant workflows on top of Kubernetes. The framework, based on the open source Pachyderm project, gives data scientists automatic tracking of changes to data and of all the various pieces of data and processing that lead to particular results. This, along with access control strategies and anonymization (which will also be discussed in the talk), gives organizations a framework that is easy to manage, scalable for AI/ML workflows, and compliant.
About Daniel
Daniel Whitenack (@dwhitena) is a Ph.D. trained data scientist working with Pachyderm (@pachydermIO). Daniel develops innovative, distributed data pipelines which include predictive models, data visualizations, statistical analyses, and more. He has spoken at conferences around the world (GopherCon, JuliaCon, PyCon, All Things Open, Spark Summit, and more), teaches data science/engineering at Purdue University (@LifeAtPurdue) and with Ardan Labs (@ardanlabs), maintains the Go kernel for Jupyter, and is actively helping to organize contributions to various open source data science projects.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Okay,
well,
thank
you
guys
for
for
showing
up
to
a
talk
on
compliance.
I
figured
I
might
show
up
and
be
the
only
one,
but
it's
good
to
see
you
here.
So
my
name
is
Daniel
white
neck
I'm.
The
lead
data
scientist
at
a
company
called
pachyderm,
which
is
a
company
built
around
the
open
source
project
pachyderm,
which
I'll
talk
a
little
bit
more
about
later,
but
today
I'm
going
to
talk
about
compliant
data
management
and
machine
learning
on
kubernetes
and
really
more
specifically,
what
I
want
to
kind
of
show.
A
A
So,
let's
start
by
talking
about
the
the
what
and
the
why
of
compliance,
so
I'll
have
to
give
a
disclaimer
here.
I
am
NOT
a
lawyer
and
I'm
not
trying
to
give
you
or
your
company
legal
advice
in
any
sort
of
way,
but
I
I
have
been
in
the
data
science
world
for
quite
a
while
and
I've
definitely
seen
some
trends,
especially
recently
around
around
compliance
and
regulations.
A
So
that's
really,
where
I'm
coming
from
with
this,
and
probably
most
of
you
guys
when
you
think
about
compliance,
think
maybe
first
about
like
PCI
or
HIPAA
compliance
as
related
to
fine
finance
and
and
health
care
right
and
there's,
certainly
compliance
requirements
in
certain
industries
and
specific
to
certain
types
of
data.
That's
not
really
the
compliance
that
I'm
going
to
be
talking
about
today.
What
I?
A
What
I've
seen
and
what
you've
probably
seen
in
recent
times,
is
that
there's
kind
of
ever-increasing
regulation,
around
kind
of
general
data
processing
and
algorithmic
decision-making
and
that's
kind
of
in
the
spirit
of
gdpr
here
in
the
EU
and,
of
course,
all
of
the
conversations
around
Facebook
and
Cambridge
analytic
and
the
in
the
US-
and
you
know,
there'll
be
more
and
more
of
this
stuff
so
to
kind
of
nail
that
down.
Let's,
let's
take
a
little
bit
of
an
example.
A
A
But
you
know,
there's
going
to
be
a
legal
imperative
in
a
lot
of
cases,
and
so
essentially
what
I'm
getting
at
here
in
terms
of
compliance,
it
is
not
really
related
to
like
anonymization
or
privacy,
or
something
like
that
necessarily
there's
a
lot
of
great
talks
on
that
and
in
other
places.
But
what
I'm
talking
about
is
really
kind
of
in
in
the
spirit
of
GDP
our
like
and
in
the
spirit
of
kind
of
this
moral
imperative
to
to
make
transparent
and
explain.
A
A
Okay,
so
now,
let's
think
about
what
are
some
like
guidelines
and
what
are
some
properties
of
machine
learning
pipelines
that
would
kind
of
make
benefit
them
in
terms
of
being
able
to.
You
know,
explain
all
of
the
data
and
processing
that
took
place
to
create
a
certain
result.
How
do
we
actually
put
that
into
into
practice?
A
So
in
the
example
of
the
autocomplete
kind
of
suggestion
case
right,
you
know,
I,
don't
work
at
Google,
so
I
can't
say
exactly
how
they
do
that,
but
I'm
guessing
it's
probably
more
than
like
a
single
Jupiter
notebook
right
like
there's
a
few
steps
involved
in
that
process
to
actually
get
to
that
end
result-
and
this
is
where,
like
the
complication
in
terms
of
being
transparent,
gets
really
really
tough.
So
we
all
like
to
talk
about
and
focus
on
like
machine
learning,
training
and
the
inference
pieces
of
what
we're
doing
right.
A
That's
pretty
much
95%
of
the
blog
posts
and
and
talks
about
this,
but
really
you
know
we
have.
We
have
a
ton
of
work
that
goes
on
before
that
in
terms
of
pre-processing
and
model
selection.
We
have
a
lot
going
on
around
training
and
in
terms
of
transforms
and
model
optimization
and
export,
and
again
we
have
more
transforms
and
potentially
post-processing
after
inference
right.
A
So
this
is
really
the
the
problem
of
kind
of
being
transparent
in
in
this
pipeline
is
that
we
have
all
of
these
different
pieces
of
data
and
processing,
and
if
we
want
to
be
transparent
about
you
know
how
we
actually
got
to
a
certain
result.
Then
you
know
my
suggestion
and
what
I
think
I
see
as
kind
of
the
trends
here
as
far
as
what
we
need
for
the
sort
of
pipeline
is
a
framework,
an
agnostic
framework
and
infrastructure
agnostic
way
to
deploy,
manage
and
define
our
workflows.
A
So
each
one
of
those
pieces
in
that
previous
pipeline
might
be
a
totally
different
framework
right
like
and
a
totally
different
team
working,
even
a
certain
team
using
like
Python
to
do
some
stuff
and
tensorflow
and
other
team
using
our
to
do
visualization
and
all
this
stuff,
and
each
of
those
pieces
might
have
different
resource
requirements
right.
Some
of
it
might
run
on
GPUs,
others
might
run
on
CPUs.
A
It's
scheduled
in
different
ways
and
all
of
this,
so
somehow
we
need
to
be
able
to
have
a
unified
way
to
define
all
of
that,
because,
if
not,
then,
if
we,
if
we
transfer
from
one
system
into
another,
then
we
completely
lose
the
context
of
you
know
the
provenance
for
what
happened
before.
The
second
thing
we
need
is:
we
actually
need
a
versioning
system
for
all
the
data
models
and
processing
in
our
workflow.
So
in
terms
of
machine
learning
and
in
data
science,
it's
not
enough
to
have
versioning
of
code.
A
So,
yes,
you
should
still
version
your
code
and
you
should
do
that
like
you're
doing
it,
but
it's
not
enough
because,
depending
on
what
data
we
put
into
our
machine
learning
code
right,
we're
gonna
have
vastly
different
results,
and
you
know
we
have
all
sorts
of
parameters
that
come
in.
We
have
certain,
you
know,
representations
of
pre-processing
transforms
and
all
of
this
stuff.
So
it's
really
like
far
from
enough
to
have
just
versioning
of
code.
We
need
some
concept
of
these
were
the
datasets
we
used,
and
these
are
how
they
changed
over
time.
A
A
Because
again,
all
of
these
things
are
stitched
together
in
a
certain
way
right,
even
if
we
were
versioning
a
certain
data
set
that
doesn't
give
us
any
context
as
to
what
it
was
input
to
to
create
what
other
piece
of
data
which
was
input
to
something
else
which
produce
something
downstream,
and
this
is
really
where
this
idea
of
data
provenance
comes
in.
We
want
to
be
able
to
say
for
this
particular
result.
This
was
the
model
binary
that
was
used
to
produce
this
particular
result
and
that
model
binary
was
trained.
A
Based
on
this
training
data
set
was
a
result
of
these
pre-processed
training
data
training
data
sets
which
came
from
this
original
raw
data.
All
the
way
back
up
the
the
pipeline
and
I
should
mention
too.
So
if
you
were
in
Jeremy
Louise
stock
earlier
today,
he
kind
of
emphasized
that
we're
working
as
a
community
towards
like
CI
for
models,
and
this
this
is
definitely
relevant.
To
this
point.
A
We
want
to
be
able
to
kind
of
have
an
audit
trail
of
how
this
model
went
through
the
pipeline
to
eventually
get
trained
and
deployed
and
used
in
inferences
okay.
So
that's
all
the
bad
news.
It's
really
hard
to
do
this
stuff
and
we
need
to
do
a
lot.
Ok.
So
now
the
good
news
part
we're
at
we're
at
cube
con.
A
So,
in
terms
of
like
this,
you
know
framework
and
infrastructure
agnostic
way
to
define
to
define
our
workflow
and
deploy
it.
Obviously,
you've
already
heard
about
cube
flow
today,
so
cube
flow,
provides
us
these
standards
packet
and
provides
us
some
other
things
which
I'll
mention
in
a
in
a
moment
in
terms
of
the
the
versioning
system
for
the
data
and
the
models
and
the
processing
in
our
workflow.
That's
that's
going
to
be
handled
by
by
pachyderm
and
also
you
know.
We
need
something
somewhere
to
house
our
data
and
a
storage.
A
A
A
So
that
includes
like
things
like
tensorflow
and
Jupiter
and
other
things,
but
really
this
is
a
community
effort
and
there's
a
lot
of
people
like
the
the
Selden
team
and
the
pachyderm
team,
and
other
teams
that
are
integrating
their
cloud
native
layers
or
projects
into
cube
flow
to
fill
in
all
of
these
pieces
that
are
needed
right,
cube
flow
provides
the
standards,
but
really
it.
It's
seeking
to
kind
of
gather
all
of
these
projects
to
get
together
to
give
people
the
tools
they
need
to
do.
What
they
need
to
do.
A
So
the
data
pipeline
pieces
are
all
containerized
and
deployed
as
pods
on
kubernetes
and
then
take
these.
You
know
data
sets
or
these
data
repositories
as
as
input
now.
The
really
interesting
thing
about
pachyderm
is
that
all
of
those
data
repositories
are
completely
version
controlled,
so
you
can
kind
of
think
about
it
like
git
for
data,
so
you
have
a
version
collection
of
data.
A
Let's
say
the
training
data
that
changes
over
time,
and
you
can
always
go
back
and
look
at
the
history
of
that
that
that
version
set
of
data
is
input
to
a
processing
stage
which,
under
the
hood,
is
run
as
as
pods
on
kubernetes
that
outputs
other
data,
which
is
also
version
in
pachyderm
and
might
be
input
to
another
stage.
And
so
you
can
already
start
to
see
that
as
we
string
these
things
together,
everything
is
version
and
very
rigorously
defined
all
the
way
along
the
pipeline
right.
A
So
we
can
always
go
back
and
have
that
data
provenance.
That
idea
that
this
result
came
from
exactly
this
model,
which
was
trained
on
exactly
this
data,
which
came
from
exactly
this
raw
data,
etc.
It
also
provides
certain
things
like
access
controls
for
data,
which
kind
of
gets
to
the
who
processed
our
data
question.
A
Finally,
I
want
to
announce
a
just
kind
of
hear
that
that
pachyderm
is
currently
making
it
a
priority
to
kind
of
contribute
and
and
and
be
more
integrated
with
cube
flow.
So
we
now
have
a
case
on
it
package
to
deploy
pachyderm
as
as
part
of
cube
flow.
So
that's
that
first
link
there.
So
thanks
to
Jeremy
Jeremy
Louie
for
work
on
that,
we
also
have
a
proposal
for
some
more
integrations,
which
I'll
mention
a
little
bit
more
here
in
a
second.
A
So
you
might
have
training
data
on
pre-processed
data
and
actually
version
models
right,
so
actually
versioning
all
the
model
binaries
that
you're
producing
and
then
each
of
these
stages
is
defined
via
one
or
more
con
gainers
that
actually
perform
the
processing.
So
this
is
all
managed
via
the
pachyderm
project.
A
But
then
you
know
data
scientists
again
want
to
use
a
variety
of
tools.
So
pachyderm
provides
this
rigorous
way
to
define
this
tract
and
version
pipeline,
but
we
also
want
to
provide
a
way
for
data
scientists
to
access
the
tools
that
they
need,
for
example
like
serving
frameworks-
and
you
know
distributed
tensorflow
and
and
that's
where
cube
flow
comes
in.
So
we
can
actually
from
pachyderm
reach
out
and
have
the
training
be
done,
for
example,
by
a
TF
job.
A
So
that's
where
the
kvc
project
comes
in
and
allows
us
to
both
use
the
versioning
version,
control
of
pachyderm,
but
also
have
this
ability
to
cache
things
for
say:
TF
jobs
and
I'll
mentioned
some
more
acknowledgments
later,
but
this
is
work
being
done
by
by
Intel
and
the
integration
with
with
pachyderm.
So
overall,
this
is
kind
of
what
the
architecture
looks
like
kubernetes
again
at
the
base,
some
persistent
volumes
for
both
metadata
and
maybe
cache
data,
and
then
on
the
top.
A
We
have
you
know
queue
flow,
related,
pods,
pachyderm,
related
pods,
and
this
version
data
in
the
backing
object,
store
which
might
be,
might
be
rook.
So
all
of
these
things
are
existing
projects
that
you
can
go
and
spin
up
today,
so
you
don't
have
to
wait
to
do
this
yeah.
It's
certainly.
There
are
like
some
rough
edges
in
certain
places,
but
this
is
something
totally
doable
today
and
people
are
running
pachyderm
you
all
around
the
world.
People
are
starting
to
run
queue
flow
all
around
the
world,
we've
seen
clusters.
A
A
A
What
I
showed
here,
exports
that
model
to
sell
them
for
serving
so
I'm
actually
going
to
start
at
the
inference,
part
and
work
our
way
back,
because,
ideally,
if
we're,
if
we
have
a
compliant
data
pipeline,
we
should
be
able
to
start
at
the
end
and
connect
and
stitch
together
all
of
the
pieces
backward
to
give
us
that
full
data
provenance
that
connects
a
result
with
all
of
the
data
and
processing
that
led
to
that
particular
result.
So
here,
I've
deployed
again
a
REST
API
via
Selden.
A
That's
going
to
use
that
that
trained
model
to
perform
an
inference
on
this.
This
text,
which
is
like
an
example,
github
issue,
okay,
so
I'm
gonna-
send
that
off
to
the
to
Selden.
It's
going
to
return
me
an
inference,
so
here
it,
it
summarized
the
github
issue
as
how
to
how
to
to
use
the
the
use
which
is
probably
not
the
best
summarization,
although
maybe
is
a
good
summarization
for
like
80%
of
github
issues
and
but
I.
A
But
let's
say
that
it's
not
a
good
summarization
right
like
there's
something
about
this
result
that
we
don't
like,
maybe
the
repeated
words
or
whatever
it
is
right,
and
now
we
want
to
say,
okay,
what
were
all
the
things?
What
was
the
model
that
led
to
this
result?
How
did
we
train
that
model?
Where
did
the
data
come
from,
and
so
let's
read,
let's
retrace
those
steps.
First
of
all,
let's
see
if
we
get
the
pods
in
the
cube
flow.
Namespace
we'll
see
Selden
here.
A
A
So
here
that's
going
to
give
me
these
docker
images,
so
you
can
see
that
there's,
there's
two
containers
running
in
the
pod,
one
is
Selden
and
the
other
one
is
this
container.
That
includes
my
model
prediction
code
and
if
we
look
at
that
model
prediction
container,
we
can
see
that
okay,
it's
an
issue
summarization
container,
but
we
can
also
see
that
it's
tagged.
Okay-
and
this
is
the
important
part
so
here
this
is
the
image
we're
using
for
inference.
A
It's
tagged
in
a
certain
way
and
it's
actually
tagged
with
the
job
ID
that
created
it
in
my
pachyderm
pipeline.
So
let
me
show
you
that
piece
now.
So,
if
I
go
over
here,
this
is
the
dashboard
for
pachyderm,
and
this
is
the
pipeline.
That's
actually
trained
pre-processing
training,
my
model
and
exporting
my
model
to
Selden,
so
here
I
can
see
that
I
have
some
raw
data.
A
If
my
example
github
issues,
those
are
being
split
into
training
and
test
sets
they're
being
processed
pre-processed
in
certain
ways
to
remove
stop
words
and
things
I'm
then
training
that
model
that's
producing
a
serialized
model
again.
Each
of
these
stages
has
input
and
output,
then
I'm,
actually
creating
with
Selden
this
kind
of
build
artifact.
That
includes
a
docker
image
and
a
bunch
of
things
that
I'm
gonna
build
into
the
service
that
I'll
deploy
and
then
finally,
an
export
pipeline
that
actually
just
builds
that
docker
images
image
and
pushes
it
out.
Okay.
A
So
that's
what
the
the
pipeline
looks
like
generally,
but
if
I
look
here,
for
example
in
my
in
this
training
data
repository,
remember
that
each
one
of
these
sets
of
data
is
version.
So,
if
I
look
here,
I
can
see
that
there's
versioning
semantics
associated
with
this
data
I
may've
certain
number
of
commits
into
this
repo,
which
are
changes
to
that
data.
There's
a
certain
number
of
files,
there's
one
branch,
all
of
those
things
now
remember.
We
we
have
this
job
ID,
that's
being
used
for
our
inferencing
right.
A
How
do
we
connect
that
back
to
this
pipeline?
That
at
some
point
produced
that
image
that
we're
using
for
inferencing
well
I've
used
a
bunch
of
I've,
actually
run
this
a
bunch
of
times.
So
here's
all
the
model,
the
pipeline
jobs
that
I've
run
for
training
I've,
run
it
over
and
over.
So
how
one
of
these
corresponds
to
the
one
that
is
being
used
for
our
inferencing
and
that's
connected
via
this
ID,
which
was
automatically
generated
when
I
built
the
image.
A
That's
being
used
for
my
inferencing
in
production
right
and
the
actual
model
that's
being
used
for
my
for
my
inferencing,
and
these
aren't
just
like
metadata
associated
with
what
that
look
like
I
can
actually
get
the
the
actual
data
associated
with
those
things.
Okay,
now
the
other
piece
of
this
is
okay.
That's
good
I
can
go
back
and
see
the
model
that
was
you
that
I'm
using
for
for
my
inferencing,
but
really,
if
something's
going
wrong,
I
need
to
explain
all
the
way
back
up
the
pipeline.
A
So
this
in
just
like
with
a
few
tools
and
plugging
things
together
with
cube
flow
I,
was
able
to
deploy
this
whole
machine
learning
pipeline
that
uses
the
tools
I
want
to
use,
including
tensor
flow
for
training
and
Selden
for
serving
but
I
stitched
it
together
in
a
way.
That's
that
is
rigorously
tracked
and
gives
me
this
idea
of
data
provenance
all
the
way
from
inference
back
to
the
raw
data.
A
So
just
to
kind
of
finish
up
things
here,
I
want
to.
You
know
just
take
an
aside
and
acknowledge
this
was
not
you
know
done
by
it
by
me
myself.
So
again,
the
great
team
at
Intel
is
working
on
this
kvc
project.
Jeremy
lui
worked
on
the
at
Google,
worked
on
the
case
on
it
package
for
pachyderm
and
and
various
other
ones.
A
So
all
of
the
resources
from
this
I
posted
these
slides
are
on
the
schedule.
I
also
tweeted
them
out,
so
you
can
find
all
the
links
there.
You
can
run
this
example
locally.
You
can
find
the
docs
and
the
public
slack
channels
and
the
repos.
Please
again,
you
know
this
is
all
a
community
effort.
It's
all
out
in
the
public,
so
definitely
reach
out.
If
you,
if
you,
if
you
want
to
get
involved
and
or
have
questions,
so
thank
you
guys
very
much
and.
B
A
Definitely
yeah
so
the
the
pipeline's
impact
erm
really.
The
the
only
thing
that
we're
opinionated
about
is
that
you're
able
to
define
your
processing
stages
via
containers,
and
then
you
have
data
that
it
can
be
stored
in
an
object
sort
which
is
essentially
anything
so
yeah.
We
have
a
lot
of
people
doing
like
MapReduce,
II
style
data,
aggregations
and
M
log
collection.
A
C
Thank
you
for
your
talk.
I
may
have
misunderstood
a
few
things
on
your
slide.
So
excuse
me
if
my
question
is
really
bizarre,
but
so
when
you
were
tracing
back
from
your
inference,
data
back
to
the
source
like
a
raw
data
training
data,
in
example,
you
were
showing
git
commit.
So
is
the
assumption
that
your
draw
data
is
also
committed
in
git,
because
if
we
are
doing
big
data
processing,
obviously
that
would
not
go
back
and
to
get
so.