►
Description
Pachyderm: Unlock the Power of Kubernetes for Big Data - Joey Zwicker, Pachyderm
Pachyderm is an open source big data analytics platform completely deployed on Kubernetes. Pachyderm leverages K8s's jobs API to process massive data workloads and build streaming pipelines. Pachyderm's hallmark feature is version-controlled data including viewing branches, commits and diffs for petabyte-scale data sets.
In this talk we'll demonstrate how Kubernetes and Pachyderm empowers data science teams to collaborate on a shared and unified data infrastructure. Everything is run on Kubernetes including streaming data ingestion, machine learning pipelines, to automatic service deployment using Rolling Updates.
Our talk will discuss how Pachyderm couldn't exist without a large swath of advanced Kubernetes primitives and includes demo where we stream data through the system and watch Kubernetes automatically schedule analytics containers and parallelize the data processing. This demo is inspired directly by how production users are managing data in Pachyderm today.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Awesome
well,
thank
you
for
joining.
My
name
is
joey.
I'm
one
of
the
founders
of
pachyderm,
and
so
my
talk
today
is
gonna,
be
geared
towards
a
little
bit
of
a
different
problem
that
what
most
of
the
focus
of
this
conference
has
been
on.
You
know
and
we're
gonna
really
be
talking
about
kind
of
data
science
and
what
those
challenges
look
like
now,
we're
all
at
a
kubernetes
conference.
So
you
know
you're,
probably
either
using
kubernetes
or
thinking
about
using
kubernetes
as
part
of
your
infrastructure
and
most
likely
that's
for
application
deployment
right
and
management.
A
But
today
I'm
going
to
kind
of
paint
a
little
bit
of
a
different
picture
and
show
you
how
pachyderm
actually
thinks
that
kubernetes
can
be
pretty
amazing,
fit
for
your
data
infrastructure,
so
just
to
set
the
stage
for
all
of
this
and
and
give
you
a
view
of
like.
Why
are
we
talking
about
data
science
in
the
first
place?
A
How
many
of
you
sense
of
like
what
pachyderm
is
so
the
inspiration
for
pachyderm
really
came
from
my
co-founder
running
the
Hadoop
cluster
Airbnb
and
we
realized
just
how
insanely
difficult
it
is
to
manage
both
from
a
data
infrastructure
perspective,
as
well
as
a
data
science
perspective
where
these
two,
where
the
teams
are
interfacing
with
each
other.
So
with
that
in
mind,
pachyderms
set
out
with
like
to
solve
one
specific
goal,
and
that
was
to
enable
large
organizations
to
better
manage
their
data
infrastructure.
A
Now,
of
course,
that's
not
actually
answering
the
question
of
what
is
pachyderm.
Pachyderm
is
an
open
source
distributed
processing
framework
built
on
kubernetes,
and
you
know
just
to
climb
clarify
this
up
front
cuz.
We
get
a
lot
of
questions
about
this.
There's
no
Hadoop
code
in
pachyderm,
right,
like
in
that's
kind
of
mentioning
here
in
a
lot
of
ways.
We
are
alternative
to
Hadoop
and
a
lot
of
our
early
customers
are
choosing
us
instead
of
a
dupe
because
we
fit
in
kind
of
the
kubernetes
and
containerized
frameworks.
A
So
how
does
data
science
and
data
and
pressure
really
relate
to
containers
and
kubernetes
right?
So,
let's,
let's
start
by
drawing
an
analogy
to
things
we
know,
kubernetes
is
a
very
good
at
and
that's
managing
micro
services.
We
all
know
and
understand
the
benefits
of
a
micro
service
architecture,
but
I'm
gonna
kind
of
talk
through
a
few
of
them
here.
The
first
is
modularity
right,
so
keeping
every
service,
small
and
modular
with
clean
API
is
to
talk
to
each
other.
A
Well,
it
really
helps
us
understand
exactly
what's
going
on
in
that
piece
of
the
application,
and
it
helps
us
maintain
an
dated
over
time.
Also
flexible,
tooling,
right
services
can
be
written
in
any
tools
using
any
languages
and
run
anywhere
in
the
cluster.
Well,
because
they're
all
containerized
right.
This
is
exactly
what
microservice
is
all
about
and
like
how
this
gets
to
be
really
powerful.
A
Also,
services
are
highly
scalable
right.
You
can
throw
a
load
balancer
in
front
of
it
and
have
your
application
automatically
scale
dynamically,
based
on
what
incoming
traffic
looks
like
as
well
as
what
cluster
resources
are
available
and,
finally
reproducibility.
You
know
you
want
if
I
build
something
locally
I
want
it
to
make
sure
it
runs
like
this,
and
it
runs
in
the
salt
in
the
cluster.
A
The
same
way
and
I
want
to
be
able
to
share
my
application,
what
I've
built
with
another
developer
and
know
that
it's
going
to
run
exactly
the
same,
we're
we're
talking
about
the
exact
same
piece.
So
this
all
applies
to
microservices
right
and
at
first
glance.
You
might
think
that
data
infrastructure
is
a
pretty
different
problem,
but
actually,
if
you
start
to
kind
of
break
it
down,
you
realize
that
many
of
the
data
entry
challenges
that
you
see
can
can
kind
of
be
framed
in
the
same
way.
A
There
yeah
so
so,
let's
start
with
modularity,
so
this
is
a
simplified,
but
like
totally
reasonable
data
infrastructure
that
we
see
in
the
wild
all
the
time-
and
you
know,
data
processing
as
I'm
sure
you've
seen
before
like
involves
all
these
different
steps.
It
may
be
eto
database
dumps,
machine
learning
pipelines
all
over
the
place
and
each
of
these
pieces
can
have
totally
different
requirements
and
SLA
s,
and
sometimes
we
even
managed
by
different
teams.
A
The
next
piece
is
flexible,
tooling,
so
the
data
infrastructure
world
is,
you
know
full
of
libraries
that
are
all
like
great
and
very
specific
things,
whether
it's
tensorflow
or
OpenCV,
and
so
of
course,
you
want
to
use
the
best
tool
for
the
job,
but
you
need
a
way
to
manage
all
these
dependencies
think
I'm
running
a
shared
cluster.
This
is
exactly
the
same
problem
that
containers
solve
in
the
micro
service
world,
which
is
again
a
large
suite
of
tools
where
you
need
to
manage
those
dependencies
highly
scalable.
A
If
you've
ever
done,
like
some
sort
of
data
traffic
management,
you
probably
have
seen
like
this
form
of
a
task
dependency
graph,
write.
Different
tasks
have
different
SL
A's,
and
so
it,
of
course,
totally
makes
sense
to
have
each
of
them
Auto
scale
based
on
new
influxes
of
data.
New
processing
needs
what
cluster
resources
are
available,
and
so
because
of
that
again,
the
data
infrastructure
also
needs
to
be
able
to
scale
up
and
down
and
be
highly
available
as
you're
going
through
all
of
these
pieces.
A
So,
as
you
can
see,
like
data
infrastructure
has
many
of
the
same
challenges
that
application
enforcer
has,
and
it
makes
a
pretty
compelling
case
to
try
to
apply
some
of
these
same
principles
and
so
kind
of
zooming
back
out.
That's
exactly
what
pachyderm
has
done
so
to
take
a
step
back.
Pachyderm
started
to
think
about
what
the
specific
data
science
challenges
were
and
so
we're
designing
pachyderm.
There
were
a
few
main
goals
in
mind.
A
The
first
was,
of
course,
as
we
talk
about
any
tools
right,
we
want
allow
data
scientists
to
use
whatever
libraries
and
tools
are
best
for
their
job.
The
second
is
distributed.
Processing
right.
If
you're
gonna
do
any
sort
of
data
at
scale,
you
clearly
need
to
have
a
distributed
processing
framework,
so
you
can
scale
these
things
up
and
down,
and
the
third
one
is
actually
version
control
for
data,
and
this
is
really
what
we
consider
the
hallmark
feature
of
pachyderm
and
I'll
actually
dive
into
the
details
of
it
for
a
sec.
A
But
the
most
important
piece
here
is
that,
while
building
a
product
that
could
solve
these
problems
and
thinking
about
it
in
the
framework
of
how
would
I
build
data
infrastructure
as
a
set
of
micro
services,
kubernetes
consistently
had
the
answer.
That
was
just
the
key
and
really
made
our
life
easier.
A
So,
let's
dive
in
them
in
one
by
one.
So
any
tools
right
data
scientists,
as
with
developers,
come
with
just
a
crazy
broad
range
of
skills.
They
might
have.
They
go
anywhere
from
like
a
statistician
right
and
those
are
and
uses
Mathematica
all
the
way
to
develop
or
EDA
scientists
that
use
more
Python.
Scala
to
you
know,
hardcore
ETL
engineers
who
love
use
everything
in
C++
and
you
know,
go
as
well,
and
we
need
all
of
this
bill
to
run
on
one
unified
infrastructure,
but
more
important
than
any
of
this.
A
What
kubernetes
really
brings
is
the
ability
to
create
the
right
interface
between
the
data
science
teams
and
the
data
infrastructure
team
is
Right.
The
infrastructures
job
is
to
allow
the
data
scientist
to
be
able
to
do
their
work
and
interface
that
effectively
and
containers
and
kubernetes
are
exactly
that.
Great
abstraction
that
gives
us
a
fantastic
separation
of
concerns.
Data
scientists
are
now
responsible
for
everything.
That's
inside
the
container
right.
A
So
we
can't
distribute
right
so
data
science
at
scale
clearly
needs
to
be
distributed
and
then
scheduled,
based
on
the
computational
needs
and
clusters
resources
available.
Once
again,
kubernetes
had
a
great
answer
to
this
and
there's
actually
their
jobs
API,
which
serves
as
a
perfect,
primitive,
so
I'm
gonna
kind
of
demonstrate
this
through
an
example.
So
let's
say
we
have
a
bunch
of
images
sitting
in
object:
storage,
maybe
something
like
s3
right
crew,
Bay's,
let's
pachyderm
spin
up
many
different
containers
in
parallel
right
and
assign
a
different
shard
of
data
to
each
of
them.
A
So
each
of
those
containers
is
running
the
exact
same
code
which
might
just
be
open,
CV
doing
let's
say
like
some
sort
of
edge
detection.
Isn't
it
easy
example
and
what
Packer
will
do
is
we
will
take
each
of
these
images
and
in
this
example,
take
one-fourth
of
the
images
and
give
them
each
to
each
different
containers?
So
now
each
container
has
a
process,
a
kind
of
a
stream
of
images.
In
parallel.
This
works
great
in
kubernetes
jobs,
because
guess
what?
A
If
container
3,
goes
ahead
and
and
dies,
because
containers
sometimes
do
that
kubernetes
will
automatically
restart
that
and
packet
ER
and
basically
has
the
logic.
Then
it's
a
Oh
contain
three:
is
the
one
that
died.
I
know
what
data
container
three
needed,
so
I'm
gonna
go.
Give
that
container
to
the
new
give
that
data
to
the
new
to
the
new
container
that
just
came
up
and
be
able
to
reprocess
that
data
and
pick
up
pick
up
where
you
left
off,
and
so
now
you
have
an
ability
to
scale
this
out
as
much
as
you
need.
A
This
could
be
for
containers,
or
this
could
be
a
hundred
giving
1/100
of
the
day
to
each
of
them,
and
you
can
paralyze
out
how
this
data
is
processed
by
just
like
having
your
analysis
in
a
container
the
other
side.
The
key
thing
to
point
out
here
is
you're,
actually
not
dealing
with
stateful
containers.
Right
really,
all
that's
happening
here
is
you've
got
a
container
with
some
code
in
it.
A
It's
reading
data
out
of
object,
storage,
transforming
in
some
way
and
then
writing
it
back
and
so
in
this,
in
this
sense,
like
you,
are
transforming
data
but
like
it's
you're,
not
dealing
with
stateful
containers
and
they
can
all
be
scheduled
and
managed
wherever
and
you
basically
to
treat
your
object
store.
Just
like
a
CDN
for
your
data
that
all
of
your
all
of
your
containers
can
access
and
then
the
final
piece
is
version
control
for
data
right.
A
So
this
is
one
of
the
cornerstone
features
of
pachyderm
and
we
think
actually
like
drastically
improves
the
data
science
workflow
in
a
number
of
ways.
Version
control
for
developers
is
what
lets
us
like
collaborate
on
a
large
shared
code
base
and
not
rip
our
Harrod
all
the
time.
Imagine
if
you
went
into
an
organization
or
you
were
interviewing
there
and
they
and
they
said
oh
yeah,
we
don't
use
version
control
for
any
of
our
code.
You
probably
wouldn't
take
that
job.
Well.
A
A
It
could
be
a
month
later
and
I
can
say,
hey
I
want
to
run
this
transformation
over
what
my
users
table
looked
like
a
month
ago
and
that
commit
that
snapshot
of
what
my
users
table
look
like
a
month
ago
is
just
there
and
available,
and
every
single
data
scientist,
if
you
say
hey,
run
this
over
the
users
table,
knows
you're
talking
about
the
exact
same
state
of
data.
The
next
one
is
instant
revert,
and
we
see
this
in
data
science
organizations
all
the
time.
There's
a
lot
of
time.
A
A
really
common
example
of
this
is:
let's
say
you
have
a
large
pipe
machine
learning
pipeline
that
and
the
day
builds
a
fraud
model
right
and
you've
got
that
fraud
model
that
you're
now
pushing
out
as
a
service
to
have
fraud
to
have
traffic,
hit
your
hit
your
server
and
like
decide
whether
or
not
it's
fraud.
Well,
if
something
goes
wrong
with
your
input
data
or
maybe
your
code
and
you
deploy
a
fraud
model,
that's
incorrect.
The
ability
to
be
able
to
say,
oh
just
like
instantaneously,
revert
back
and
roll
out
the
old
Florida
model.
A
From
yesterday,
when
I
go
figure
out
what
the
heck
went,
wrong
becomes
incredibly
powerful
and
always
knowing
how
that
has
changed
over
time
and
let's
track
that
gets
really
powerful.
The
last
piece
is
also
is
data
lineage,
and
this
is
kind
of
a
concept
that,
in
the
data
science
world
is
starting
to
get
a
lot
of
traction
right
now,
also
like
thought
of
as
provenance,
and
basically
what
this
lets
you
do
is
you
lets
you
say
here
is
my
like
raw
input.
A
Data-
and
here
is
all
the
you
know-
might
be
50
steps
of
transformations
that
went
down
to
then
finally
produce
this
final
result
and
it
might
have
taken
in
data
from
a
ton
of
different
sources
and
pulled
together.
All
these
different
things
I
need
to
be
able
to
track
every
single
step
of
that,
because,
at
the
end
of
the
day,
I
get
to
a
result,
and
I
can
look
at
that
graph
and
say:
there's
no
hey.
This
is
right.
A
How
do
I
go
back
and
debug
all
that
and
audit
that
whole
trail
of
how
my
data
has
changed
and
all
these
different
pieces
of
code
that
would
have
transformed
it
and
when
you
think
about
having
your
data
version,
including
all
these
intermediate
steps,
you
now
have
the
ability
to
really
do
that
and
it
makes
the
iteration
cycle
for
developing
data
science
just
incredibly
much
faster
and
very
powerful.
So
data
versioning
comes
with
this
challenges
too,
as
you
might
expect,
the
biggest
one
is
that
you
for
tracking
changes
and
dips
of
files.
A
This
creates
a
lot
of
metadata
that
you
need
indexed
and
readily
available
for
fast
look
ups.
Well,
as
you
might
expect
what
you'd
want
to
use
for
this
is
a
database
to
store
all
of
your
metadata,
but
this
causes
a
whole
bunch
of
problems,
but
once
again
kubernetes
came,
you
know,
showed
up
and
saved
the
day
because
they
released
pet
sets
and
persistent
volumes
and
pachyderm.
Basically,
that's
exactly
how
I
would
do
it.
A
We
have
a
metadata
database
that
stores
all
that
information
and,
like
basically,
talks
about
controls,
the
metadata
around
how
data,
how
files
are
versioned
and
difft
and
that's
run
as
a
pet
set
Whitney
to
the
president
of
volume,
and
this
now
becomes
highly
available
in
kubernetes.
You
get
to
run
it
right
there.
If
you
need
to.
You
can
always
take
that
and
sync
it
back
to
something
like
s3
or
your
object.
Storage.
If
you
want
to
make
sure
it's,
it's
super
persistent,
but
kunai's
does
a
pretty
good
job
of
it
on
its
own.
A
So
let's
take
a
deep
breath
for
say
and
kind
of
stand
back
and
think
about
the
kind
of
world
view
we've
we've
built
here.
We
want
to
be
able
to
run
your
data
infrastructure
like
the
way
you
run
applications
right.
There
is
a
world
where
that
can
really
be
possible.
That
includes
modular
services,
for
your
data
infrastructure,
with
clean
abstractions
between
to
them
flexible
tooling,
so
your
data
science
team
can
use
whatever
it
wants
and
combine
and
compose
all
these
pieces
together.
A
So
that
way
you
can
track
how
things
are
changing
over
time
and
actually
have
make
your
to
have
your
data
reproducible
in
addition
to
your
actual
code,
and
we
want
to
do
all
this
with
one
unified
infrastructure
and
because
of
the
power
of
kubernetes,
we
actually
can
so,
of
course,
if
you're
excited
about
building
your
data
infrastructure
in
this
way,
and
you
like
the
idea
of
saying
okay
I'm
used
to
running
my
application
infrastructure
as
a
set
of
micro
services.
Let
me
think
about
the
my
data
infrastructure
in
a
similar
format.
A
You
should
absolutely
check
out
kubernetes
and
like
try
thinking
about
setting
your
you're
set
up
this
way,
and
you
should
also
look
at
packet
er
because
we
basically
build
a
framework
for
doing
it
in
exactly
that
sense,
cool.
That's
the
end.
I
wanted
to
give
a
lot
of
time
for
questions,
because
I
tend
to
get
a
lot
and
I
went
pretty
fast.
There.
A
Alright,
so
the
question
was
where
one
weird
one
do
something
that
store
like
store:
huge
log
files
and
such
for
us.
We
actually
do
that
in
object,
storage
as
well,
and
I've.
I've
found
that
most
most
object
stores
are
actually
quite
able
to
handle
that
in
a
reasonable
sense.
These
files,
you
can
always
break
up
these
files
in
in
ways,
if
that,
if
that
was
a
problem,
what
have
you
experienced
specifically
that
causes
problems
for
that.
A
A
A
That's
overlaid
on
top
of
that,
and
so
in
fact,
from
a
data
science
code
perspective,
your
code
doesn't
really
have
to
be
super
smart
in
terms
of
doing
anything
with
versioning
pachyderm
expose
the
data
to
you
in
this
versioned
way,
and
so
let's
say
you
know,
have
a
huge
log
file
and
you've
appended.
You
know
1%
more
logs
at
the
end.
A
Yeah,
so
when
you
have
a
continuous
stream
of
logs
coming
in
basically
the
way
pachyderm
handles
this
is
you
can
just
make
commits
at
whatever
frequency
you
want
and
commits
actually
are
very
low
overhead
overall,
and
so
we
have.
We
have
production
users
making.
You
know
a
commit
every
minute
or
commit
every
30
seconds,
and
so
you
can
make
commits
all
the
time,
and
now
your
log
will
just
constantly
appending
and
broken
up
into
these
30-second
chunks,
each
of
which
gets
processed
downstream.
A
You
you
can
you
can,
if
do
that,
if
you
want,
you
don't
have
to
pack
it
er
pachyderm
like
there's,
if
you,
if
you're
just
appending,
loglines,
constantly
there's
very
little
overhead
to
doing
that.
As
you
know,
aggregating
all
the
logs
for
an
hour
and
making
commits
versus
doing
a
commit
every
minute,
just
as
that's
coming
through
sure.
A
Yeah,
so
the
pack
denote
itself
were
just
kind
of
the
coordinator
for
all.
This
is
just
a
pot
in
kubernetes,
so
you
can
scale
that
out
and
run
that
as
much
and
you
thin
the
cluster
and
then
pachyderm
itself
scales
like
great
with
kubernetes
jobs.
Right,
you
can
spin
up
thousands
of
kubernetes
jobs
all
at
once
and
kubernetes
will
schedule
all
those
and
figure
out
all
where
you
go.
So
all
we
get
all
the
scaling
benefits
from
kubernetes
and
pachyderm
itself.
Just
basically
coordinates
this
and
assigns
shards
of
data.
Two
sets
of
containers.
A
It's
great
for
parallel
workloads.
You
know
if,
if
you
have
things
that
can't
be
paralyzed
very
well
right,
so
some
machine
learning
models
right,
like
you
can't
paralyze
that
you're
not
gonna,
be
able
to
get
the
benefits
of
running
this
in
distribute
fashion.
But
you
can
still
throw
all
of
this
as
a
single
big
beefy
box
within
pachyderm,
and
that's
totally
fine
right
and
so
pachyderm
works
really
well
actually
for
like
quite
heterogeneous,
like
huge
datasets,
where
you
might
have
this
piece
over
here.
A
That
is
all
has
to
be
single
thread
right
on
like
big
beefy
boxes,
and
this
thing
over
here
can
be
really
paralyzed
right.
You
might
have
a
map
job,
a
map
style
job
that
can
be
incredibly
distributed,
but
then
you've
got
to
reduce
where
you
got
to
aggregate
everything
all
together
and
like
that.
Both
of
those
setups
work
perfectly
fine,
a
packet,
and
we
give
you
same
style
map
and
reduce
primitives
that
you
might
be
used
to
from
Hadoop
to
run
exactly
that.
B
A
That's
a
great
question:
the
question
was
basically
that
as
an
inference
I'd,
you
guys
kind
of
have
a
good
picture
of
pachyderm
but
oftentimes
presenting
this
to
the
data
scientists.
They
have
a
hard
time
kind
of
wrapping
their
head
around
all
of
that
yeah.
So
at
the
kind
of
early
stage
we're
at
now,
that's
definitely
something
we've
seen
occasionally
oftentimes,
especially
data
scientists,
who
kind
of
lean
a
little
bit
more
towards
the
developers.
Side
of
things
find
this
to
be
a
great,
a
very
powerful
interface.
A
One
of
the
things
we're
starting
to
build
now
is
the
like
actual
like
natives
as
a
supporting
Jupiter.
No
prefer
you
can
just
have
a
Jupiter,
no
Park
run
that
directly
in
the
container
and
as
far
as
the
data
scientists
are
concerned,
their
interface
is
exactly
the
same.
The
only
difference
is
now
in
that,
like
code,
repo
gives
ours
also
docker
file
there.
That
builds
the
whole
thing
into
a
container
right
and
so
making
that
story
very,
very
clean
for
data
scientists
definitely
is
something
that
we're
continuing
to
make
better.
A
That
exact
same
thing
is
now
like
what
you
send
off
to
the
data
infrastructure
and
that's
going
to
run
in
the
cluster
it
could
be
distributed.
That
now
starts
to
feel
very
powerful
right
and
that
that
Python
script
by
the
way
when
they're
running
on
the
laptop
was
probably
reading
and
writing
just
local
files.
That's
actually
how
pachyderm
works
itself.
I
was
talking
about
this
with
a
gentleman
here
earlier.
A
Pachyderm
basically
takes
the
data
in
the
distributed
file
system
right
and
now
after
storage
and
mounts
it
locally
as
a
local
file
system
inside
the
container
right
and
that's
how
we
divvy
up
what
data
goes
into
what
container,
and
so
the
idea
that
that's
that
Python
script
still
just
reads
and
writes
local
files
and
that's
like
that's
the
only
cognate
overhead
they
have
to
think
about
in
terms
of
accessing
their
data.
That
starts
to
be
a
pretty
pretty
powerful
way
to
think
about
things.
A
That's
a
great
question
so
right
now
the
the
auto-scaling
pieces
of
this
are
pretty
pretty
primitive.
What
you
know
pachyderm
will
basically
parallel
eyes
for
each
pipeline
for
each.
You
know
state
task
in
that
dag
you
can
set
a
paralyzation
factor
and
pachyderm
will
try
to
spin
up
up
to
that
many
containers
in
terms
of
how
much
they
want
to
paralyze
that,
in
terms
of
how
kubernetes
now
wants
to
manage
its
resources,
that
really
comes
to
like
how
you
ought
to
scale
the
underlying
resources
that
kubernetes
manages
right.
A
A
Yeah
in
terms
of
like
how
you
specify
your
your
pipeline
and
packet
or
like
that's
it
and
then
you've
got
your
and
that
that
JSON
file,
you
know,
specifies
the
name
of
your
pipeline.
What
image
you
want
to
use
and
the
Registrar
you
pulling
it
from
you
know
what
functions
are
gonna
call
within
your
code
within
the
container
and
then
things
like
the
peril
Asian
factor.
How
do
you
want
your
data
sharted
right?
A
If
it's
a
it's
a
map,
style
step
like
the
images
example
I
talked
about
you,
don't
care
which
images
go
where
right
you
just
want
them.
You
can
think
lease
all
spread
out
if
it's
something
more
kind
of
reduced
style,
where
you
need
a
entire
file
or
a
group
of
files
all
need
to
be
seen
by
a
given
container.
You
specify
exactly
that
kind
of
stuff
right
there
in
the
JSON
file
and
packing
and
we'll
make
sure
to
keep
all
those
things
together
as
it
shards
it
out.
A
A
Let
me
push
that
over
to
a
spark
cluster
that
I
can
spin
up
right
now
to
process
all
that
data
and
then
spark
will
write
that
data
back
in
a
pachyderm
and
then
continue
downstream
from
there,
and
so
this
is
something
that
is
very
easy
to
do.
Pachyderm
kind
of
we
think
about
it.
All
the
time
as
kinda
has
on
the
adapters
on
all
ends
is
that
you
can
basically
have
a
pipeline.
It
just
reads
and
writes
data
out
from
anywhere
right,
and
so
it's
very
easy
to
spark.
An
already
read
out
of
object.
A
B
A
Totally
yeah,
so
so
the
question
here
is
basically
around
like
how
pachyderm
thinks
about
diffs
and
how
that
kind
of
translates
into
doing
what
we
think
of
as
incremental
processing
and
your
your
comment
is
exactly
right,
like
not,
everything
can
be
done
effectively
in
an
incremental
way
right.
I
I,
mentioned
kind
of
in
passing.
During
the
talk
that
oh
yeah,
we
can
say,
oh
there's,
like
5%
new
data
and
I,
don't
actually
need
any
of
the
other
data.
Let
me
just
process
that
new
5%,
because
pachyderm
is
aware
of
what
data
has
changed.
A
A
This
is
like
pachyderm
can't
solve
math
for
you,
but
it
lets
you
optimize
it
whenever
there's
opportunities
to
do
that
and
they're.
Actually,
surprisingly,
a
lot
of
opportunities
and
a
large
swath
of
the
things
that
that
you
do
in
a
tipple
of
data
science,
workflow
do
fall
under
things
that
can
potentially
be
distributed
as
well
as
can
be
incremental,
there's
kind
of
a
class
of
things
that
we
think
of
as
like
online
algorithms.
A
We
will
basically
take
take
one
tenth
of
the
images
and
we
can
just
use
like
hashes
of
the
filenames
to
do
this
and
split
them
up
into
shards,
0
of
10
or
0
nitrite
and
split
those
all
out
and
then
we'll
basically
say
like
okay.
This
container
here
right
is
now
responsible,
its
container
1
and
this
is
correspond
scible
for
that
1/10
of
the
images
and
if
that
container
dies,
we
now
know
the
okay.
A
That
data
now
needs
to
be
scheduled
on
the
new
container
that
kubernetes
brings
up
and
so
pachyderm
itself
kind
of
is
the
part
that
does
the
the
sharding
in
terms
of
figuring
out.
Here's,
how
many
containers
I
have?
How
do
I
want
to
split
the
data
Cabrera's
jobs
as
far
as
the
the
jobs
for
me
was
concerned?
It
just
knows
that
I
had
got
10
containers
and
all
of
them
need
to
run
to
completion.
I
need
actually
run
to
cleats
in
10
total
times.
In
order
for
me
to
call
this
whole
whole
job
done.
B
A
A
Oh
Oh,
so
yes,
okay,
so
pachyderm
kind
of
does
that
for
you?
So
you
don't
have
to
define
your
your
your
pod
spec
for
your
for
your
pipeline
and
you
basically
specify
the
the
pipeline
part
in
pachyderm
and,
like
I
said
that
specifies
an
image
and
then
pachyderm
will
spin
up
those
those
pods
and
containers
that
now
like
get
run.
So
you
don't
actually
have
to
specify
the
pod
details
as
part
of
the
pipeline,
both
in
kubernetes
and
pachyderm.
A
A
It's
completely
reliant
on
kubernetes
scheduler
yeah,
so
I
mean
that
was
kind
of
one
of
the
big
design
primitive
packages
like
let's
not
build
everything
from
scratch
right.
You
know,
Hadoop
had
to
build
all
of
the
distributed
systems,
primitives
and
schedulers
and
zookeeper,
and
all
these
pieces
and
then
kind
of
go
solve
the
big
data
problem
and
Packers
approach
to
this
has
been.
You
know.
Kubernetes
gives
us
almost
everything.
We
need
to
be
able
to
solve
this
from
a
distributed
systems
and
scheduling
perspective.
A
So
so
the
versioning
itself
is
done
on
files
in
an
object,
storage
right.
So
basically
the
we
have
a
couple
different
flavors
of
versioning
I'm
happy
to
dive
into
more
of
these
details.
Offline.
If
you
want
basically,
you
know,
we
have
pile
versioning
right,
so
you
we
know
if
you've
like
over
it
in
a
file
if
you've
added
a
new
file
if
you've
deleted
a
file.
A
Similarly,
we
also
have
a
pen
based
versioning,
so
if
you
add
new
lines
to
the
end
of
a
file,
we
know
that,
like
this
is
now
the
diff
right,
so
file
foo
and
commit
one.
Was
these
hundred
lines
and
file
foo?
If
you
redo
that
commit
to
is
these
hundred
lines
plus
another?
You
know
maybe
another
50
lines
of
something
there,
and
so
that's
how
that
dipping
is
done.
We
are
in
the
process
of
building
now
the
ability
to
do
kind
of
more
sophisticated
things
like
line
base
dipping
for
text
files
or
something
like
that.