►
From YouTube: Realizing End to End Reproducible Machine Learning on Kubernetes - Suneeta Mall, Nearmap
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Realizing End to End Reproducible Machine Learning on Kubernetes - Suneeta Mall, Nearmap
Industry adaptation of data-science has grown rapidly in the last few years. The probabilistic nature of this space requires the right tools and techniques to ensure that the answers produced are reliable. Models are derived from data, which is almost always evolving, massive (as in deep-learning), and requiring clean-up and pre-processing before use. Reproducibility, reporting, tracking and management around the tasks of 1) data - collection, pre-processing, often feature engineering and 2) model – training, tuning, evaluation and serving are essential. With tools such as Pachyderm, Kubeflow, Katib, ModelDB, Seldon and Argo, an automated end-to-end reproducible machine learning framework can be built on Kubernetes. This talk will detail how the aforementioned tools can be used to build an automated, reproducible machine learning framework.
https://sched.co/UacQ
A
Hello,
everyone
thank
you
for
coming
to
the
stop
already
been
introduced.
I'll
skip
to
the
next
slide.
Happily,
new
map
is
a
provider
of
high
resolution,
aerial,
imagery
and
location
content.
We
are
headquartered
in
Sydney,
Australia
has
mentioned.
We
currently
capture
urban
areas
of
United,
States,
Australia,
New,
Zealand
and
Canada.
We
specialise
in
capturing
frequently
and
providing
crisp
and
current
image
images.
Now
recently,
we
hopped
on
to
AI
bandwagon
in
our
experience,
has
been
nothing
but
phenomenal.
Rei
team
has
been
doing
lots
of
exciting
work,
building
derived
cont
content
on
top
of
our
imagery.
A
Some
of
these
are
building
outlines
and
things
in
stuff
detection.
Things
like
swimming
pool
power,
pole,
solar
panel
and
stuff
like
vegetation
roads.
Now
this
isn't
just
rnd
machine
learning.
We
are
talking
about
petabyte
scale,
machine
learning
or
AI.
In
fact,
we
have
been
churning
through
petabytes
of
aerial
imagery,
generating
semantic
content
of
every
ground.
We
have
covered
in
a
very
short
time
to
put
figures
to
that.
We
are
talking
about
burning
2,000
GPUs
for
two
weeks
to
generate
semantic
content
over
a
million
kilometres,
quite
idea
at
a
resolution
of
5
centimeter
per
pixel.
A
In
that
small
time-
and
that's
utilizing
only
preemptable
instances,
because
obviously
we
won't
cost
to
be
low,
and
all
this
has
been
realized
only
because
of
communities
because
of
the
abstraction
that
this
platform
provides
and
simplifies
the
operations
that
we
have
been
able
to
realize
that
sort
of
scale.
With
our
AI
workload.
Now
ai
isn't
just
about
model.
It
requires
equal
contribution
from
a
lot
of
skill
set.
It's
not
just
about
there,
a
scientist,
it's
also
about
engineering,
DevOps
and
data
hubs,
and
that's
why
communities
has
been
our
choice
of
platform.
A
Now
I
wanted
to
show
you
how
San
Diago
looks
like
from
the
lens
of
our
machines.
This
is
Santiago
in
green
and
red,
green
for
roads
and
red
for
roofs,
and
it
looks
pretty
beautiful.
I
must
say
how
lot
to
talk
about
all
these,
but
today
we
are
gathered
for
much
more
important
and
I
say
our
urgent
issue,
which
is
reproducibility
in
machine
learning,
but
we
are
going
to
be
talking
about
how
we
are
running
our
inference
pipeline
on
communities
in
Sydney
forums.
A
So
if
any
of
you
are
coming
down
in
Down
Under,
please
come
and
say
hello.
So,
which
brings
us
to
really
today's
agenda
I'll
be
talking
about
briefly.
What
machine
learning
is
why
reproducibility
matters
and
what
are
the
challenges
in
really
realizing
machine
learning?
How
much
reproducibility
then
do
we
really
need
and
what
are
the
tools
and
techniques
that
we
have
at
our
disposal
to
really
write
a
reproducible,
machine
learning,
code
and
infrastructure?
A
And
then
let's
have
a
look
at
how
to
do
that
on
kubernetes
and
there's
an
extension
to
reproducibility
that
is
replicability,
and
that
is
being
robust
to
certain
noises
in
in
the
model.
How
do
we
do
that
and
then
I'll
open
the
floor
for
question
so
very
broadly
speaking,
machine
learning
is
all
about
learning
complex
mathematics
on
top
of
your
data
to
find
patterns
and
associate
meaning
to
those
patterns,
and
when
we
think
about
machine
learning,
we
really
think
about
that
sort
of
workflow.
A
We
we
have
the
data,
we
transform
it
to
get
it
trendy,
then
we
train
it
to
knit
and
we
probably
improvised
it
for
boundary
conditions
and
adversarial,
and
then
we
calibrate
the
model
and
you
evaluate
the
model
and
if
you
are
happy
with
it,
that's
where
we
think
the
model
really
ends,
but
in
reality
the
workflow
is
much
more
complex
and
sort
of
looks
like
this.
The
bug
doesn't
stop
at
obtaining
the
model.
We
have
to
have
a
release
process
in
place.
A
We
have
to
make
the
model
available
for
consumption
and
often
the
the
output
from
the
model
isn't
quite
usable
by
the
end-user.
You
have
to
really
transform
it
to
make
it
consumable.
So
that's
post-processing
and
the
argumentation,
the
bit
that
happens
at
the
end
ends,
probably
where
your
customers
interest
lies,
and
then
we
also
have
a
funnel
that
goes
back
into
your
data
set
to
continuously
keep
updating
your
model
to
improve
it
for
wherever
it's
not
performing
very
well,
it's
also
commonly
known
as
adoptive
sampling.
A
So
this
is
the
workflow
that,
in
my
view,
is
really
there
reality
and
for
the
rest
of
the
session.
We
will
be
speaking
to
this
kind
of
workflow
now,
let's
take
a
pause
and
talk
about
why
reproducibility
really
matters
so
reproducibility
helps
with
understanding
explaining
and
debugging,
and
it's
specially
hard
in
mr.
machine
learning,
when
just
your
output
may
change
just
by
shuffling
your
data.
So
your
data
is
still
the
same,
but
you
shuffle
it
and
you
may
end
up
getting
the
different
results
and
that
that's
why
reproducibility
is
hard
in
machine
learning.
A
It's
all
the
more
hard
because
we
have
deep
learning,
which
are
also
tagged
as
deep
nets.
We
spent
10
years
of
research
really
trying
to
understand
what
these
networked
learn
starting
from
visualizing.
The
higher
layer
features
to
interaction
between
neurons
and
also
building
activation
atlases,
but
still
not
quite
sure
what
the
model
does
and
why
it
makes
some
inferences
that
it
does
and
in
that
circumstance
is
given
the
the
main
brain
that
we
don't
understand.
We
really
have
to
understand.
A
What's
input
and
what's
output
and
having
that
assurance
that
we
will
get
to
the
same
result
becomes
an
important
factor
and
it's
not
just
for
us
for
understanding
and
debugging.
Our
customer
also
expects
the
answers
to
be
verifiable,
reliable
ethical
and
unbiased,
making
your
answers.
Reproducible
makes
them
credible,
as
Lincoln
has
said,
and
it
correctness
is
important
too
in
our
answers.
These
are
example
of
some
of
the
great
AI
failures
of
our
time
tagging
humans
as
gorillas,
giving
wrong
ripples.
A
A
Let's
look
at
each
of
them
individually,
so
machine
learning
computations
nowadays
requires
gigabyte
or
terabyte
floating-point
operations
which
are
really
complex
and
require
specialized
hardware
to
perform
that
sort
of
computation.
But
it's
a
common
knowledge
that,
due
to
asynchronous
computations
on
the
GPU
different
GPU
architecture,
gives
us
different
results.
So
if
you
have
your
data
and
same
algorithm,
if
you
run
it,
you
may
end
up
getting
different
result.
A
Some
of
the
routines
like
backward
pass,
do
not
guarantee
reproducibility
when
drawn
on
GPU
or
even
on
the
path
with
parallelism
now
again,
there's
a
great
representation
by
Duncan
who
has
done
a
lot
of
work
with
determinism
on
machine
learning,
with
tensor
flow
that
I
recommend
watching,
and
it's
not
just
computation
intensive
workloads,
sometimes
even
much
simpler,
deterministic
workload
may
give
you
different
answers.
Why?
Well,
because
it's
software
now!
This
is
a
true
story
that
we
went
through.
A
We
really
is
nothing
but
a
version
change
of
a
pipe
Raj
which
is
a
Geo
special
library
and
upgraded
just
that
particular
library
from
196
2
to
400
and
suddenly
our
location
calculation
for
San,
Diego,
Convention,
Center
well,
somewhere
out
in
Miramar
of
Gold
Coast.
Imagine
giving
that
results
to
to
customers
and
or
even
ordering
your
pizza
and
enjoying
your
dinner
so
and
machine
learning
really
loves
randomness.
If
you
don't
have
a
solution
to
a
problem,
we
we
try
and
randomize
it
and
hope
it
will
sort
out
itself.
A
We
have
randomization
coming
from
algorithm
like
dropout,
random
initialization,
and
we
also
have
randomization
coming
from
process
and
practice
like
argumentation
flipping.
The
images
are
rotating
them
or
introducing
noise
to
make
the
models
more
robust,
but
or
simply
just
shuffling
your
data
to
remove
the
bias
in
the
data,
but
look
at
how
many
seats
I
have
to
set
up,
and
these
are
just
the
global
random
seed
I'm,
not
even
talking
about
the
seed
that
goes
into
the
algorithms.
Are
you
or
your
layers,
and
then
we
have
data?
A
This
is
the
Beast
of
all
the
challenges
that
we
are
talking
about
when
we're
talking
about
machine
learning.
We
are
really
talking
about
large
data
set
and
when
the
data
is
really
large,
we're
talking
about
all
sorts
of
problem
problem
related
to
data
poisoning,
how
a
representation
under
representation
and
as
Polly
has
said,
no
input
is
ever
really
independent
and
same
applies
to
data
as
well.
No,
no
data
point
can
be
seen
in
isolation.
A
So
that's
a
good
example
of
why
reproducibility
really
matters
and
where
the
challenges
are
coming
from
and
as
if
we
didn't
have
enough
problem
in
enough
challenges.
Making
us
not
realized
reproducibility
in
machine
learning,
we
have
concept
drift,
our
concept
of
things
and
stuff
keeps
evolving
as
we
go
over
time.
This
is
there's
a
first
prototype
car
as
we
envisioned
it
and
now
how
we
envision
a
car
when
we
think
about
you
know
really
what
a
car
looks
like.
A
So
it's
not
just
about
getting
to
the
same
answer
on
your
same
model
on
your
same
iteration.
But
now
we
are
talking
about
continuously,
generating
more
and
more
models
who
are
improved.
Data
set
and
also
having
reproducibility
across
everything.
So
it's
not
just
reproducibility,
but
we
need
in
full
automation
and
governance
across
everything
which
brings
us
to
how
much
our
produce
ability
do.
We
really
need.
There
is
a
middle
ground
to
this.
A
If
we
can
say
that
as
long
as
we
can
explain
our
answers
understand
and
reclaim
them
and
have
a
roadmap
to
reaching
to
the
same
conclusion,
then
we
may
be:
okay
today,
it's
possible
to
write
100%
reproducible
code,
but
it
comes
at
a
cost.
A
cost
of
speed
which
may
not
always
be
at
our
disposal
so
having
write
process
in
place
and
are
doing
the
best
we
can
is
is
a
way
to
realize
hundred
percent
reproducibility
and
we'll
we'll
look
over
these
in
a
little
bit
more
detail
shortly.
A
So
this
is
how
to
realize
reproducibility
again.
If
we
can
write
reproducible
reproducible
ml
code
as
much
as
we
can
and
version
control
everything,
then
we
can
have
reproducibility
and
also
explain
the
variances
that
we
are
seeing
in
the
answer
and
debug
them
and
have
more
control
over
it.
Rather
than
just
shrug
and
say,
I,
don't
know
how
I
got
that,
probably
one
of
those
reasons
why
I
didn't
get
the
same
answer,
not
something
we
would
like
to
go
to
customers
saying
so
what
I
have
is
a
simple
example
done
on
a
ox
for
pets.
A
Our
data
set
it's
a
very
simple
semantic
segmentation
model
that
segments
image
pixels
into
three
categories:
foreground
background
and
unknown
and
I've
taken
that
example
in
transform.
This
is
blue
and
to
an
ml
pipeline
and
kubernetes
for
the
rest
of
the
talk.
I
would
be
referencing
code,
snippets
and
files
from
this
repository.
A
So
talking
about
the
first
step,
which
is
writing
reproducible
ml
code.
Oh,
it's
really
very
simple.
We
got
to
have
version
control
on
this
on
our
code,
which
I
believe
hundred
percent
of
us
do
that,
if
not,
please
do
and
then
reproducible
run
time
think
pipe
ROG.
It
happens.
It
happens
a
lot
to
us
as
well.
If
you
don't
have
the
libraries
pin
to
the
same
version,
it
does
cause
some
unexpected
behaviors
and
then
brand.
A
Yes,
please,
please
do
seed
your
random
layers
and
you
can
change
them
when
you,
when
you
do
need,
then
rounding
rounding,
overflow
precision,
these
kind
of
problems
we
need
to
be
aware
of.
There
are
there's
other
bucket,
where
we
talking
about
dependency.
Issues
coming
from
libraries,
just
like
how
I
mentioned
before
backward
pass,
is
not
guaranteed
to
be
deterministic
on
GPU,
even
when
run
on
tensorflow,
and
there
are
many
different
issues
that
sits
in
tensorflow
bucket.
A
That
sort
of
makes
your
code
non
reproducible
now
there's
a
way
to
make
it
reducible,
but
I,
guess
the
choice
is
your
because
it
comes
at
a
cost
of
Spain,
but
everything
that
can
be
achieved.
Things
like
pinning
library
randomness
that
that
that
is
something
that
I
would
recommend
doing
to
realize
a
reproducible
machine
learning.
A
Now
in
now,
how
do
we
do
100%
reproducible
machine
learning
code?
This
is
an
example
for
that
so
train
dot.
Pi
is
a
code
that
I
have
in
that
project.
I
really
have
a
standard
unit
based
network
with
the
vgg
or
mobile
net
backbone.
If
you
are
familiar
with
deep
learning-
and
it's
a
very
simple
example,
but
all
I
have
is
all
I
have
all
randomly
seeded
I
have
all
libraries
pinned
and
then
I
have
global
seed
set
defined
by
set
seed.
That
shown
on
the
screen,
and
on
top
of
that,
I
have
this
routine.
A
That
I
call
sort
set
global
determinism
that
sets
all
the
deterministic
flags
of
libraries
that
I'm
using
in
my
stack
and
that
guarantees
that
all
my
routines
are
deterministic
and
then
I
also
have
the
CPU
parallelism
flags
turn
to
one
and
then
I
patch
tensorflow
for
determinism.
This
goes
back
to
the
backward
pass
example
that
I
gave
now
doing
that
on
top
of
my
code
guarantees
that
every
single
time
I
run
my
code
or
what
the
same
data
set
I
get
100%
of
reducibility
over
it.
A
Now,
if
you
wonder
what
happens
if
I
forget
to
set
aside
or
if
I
don't
try
to
reproducible
code,
then
have
a
look
at
this.
I
have
results
from
two
models:
one
trained
with
100%
reproducible
code
and
the
second
one
is
trained
with
almost
same
as
one,
but
just
one
drop
layer
see
unseeded.
Now,
that's
just
me,
forgetting
one
scene
and
you
can
see
the
difference.
The
first
example
is
really
simple
example
and
you
can
see
only
subtle
difference
on
the
color
of
the
dog,
but
look
at
the
second
one.
A
That's
a
complex
example
for
a
unit
model,
which
is
very
simple,
and
you
can
see
the
difference
on
the
shadow
in
the
prediction
and
the
third
one.
That's
that's
really
hard.
Even
I
can't
tell
if
it's
a
cat
or
a
dog
or
which
animal
that
it
really
looks
camel
to
me,
but
anyhow,
look
at
the
results.
We
are
talking
about
significant
difference
in
the
predictions
and
that's
just
one
seed
everything
else,
including
flags
and
everything
turned
on.
A
So
we
talked
about
writing
reproducible
ML
code.
Now,
let's
talk
about
version
control
and
full
provenance.
First
up
what
to
version
control?
The
way
to
realize
machine
learning,
really
basically
AI
system
is,
is
comprised
of
your
infrastructure
and
your
platform
and
platform
orchestration
layer
which
is
communities
in
this
context,
and
then
we
have
lots
of
libraries
or
software
that
sits
on
top
of
it.
Like
things
like
Google
cube
flow,
pachyderm
selden
are
going
H
to
you.
A
This
is
what
this
example
is
using
to
demonstrate
how
to
realize
and
to
end
reproducible,
machine
learning
and
then
I
also
have
source
code
on
top
with
the
dependency
and
the
container
and
end
data
on
the
on
the
other
side
and
the
process
in
practice.
Sort
of
ties,
everything
together
to
realize
the
in
to
end
reproducibility
and
the
challenges
are
tagged
along
side
of
the
each
of
the
layer
on
which
layer
what
challenges
do
you
see?
More
often
so,
first
up,
let's
look
at
versioning
environment
now.
Ops
is
a
very
kind
of
popular
concept.
A
Now
now
in
cloud
native
and
communities
community,
we
basically
provision
our
infrastructure
from
a
good
resource.
So
every
change
on
to
your
infrastructure
is
version
control
and
is
really
driven
off
your
git
repository-
and
this
is
what
this
example
is
using.
Basically,
what
I'm
advocating
for
is
everything
should
be
version
control,
and
this
is
just
your
code
and
your
stack
runtime,
but
also
your
infrastructure
and
the
example.
A
Really
all
you
need
to
do
is
bring
your
own
kubernetes
cluster,
install
Argus
ad
and
then
just
install.
These
are
go
up
which
are
defined
already
in
the
cluster
and
this
provision,
CEO
Christopher
for
all
sorts
of
cube
flow,
Q
flow,
six,
oh
and
packet
m198,
and
on
the
back
of
it
you
get
lots
of
training
frameworks
like
TF
job
by
torch,
but
pachyderm
Q
flow
are
go-karting,
Seldon,
TF
server.
Many
are
there
serving
frameworks
and
also
a
steel
for
sure
for
serving
excuse
me
and
then
let's
talk
about
versioning
everything
else.
A
What's
everything
is
really
the
process,
the
workflows
that
whole
pipeline
structure
and
the
data
so
just
reminding
you
back
again.
This
is
the
workflow
that
we
are
working
towards
and
each
of
the
step
in
this
pipeline
is
the
workflow
and
the
data
that
it
generates.
The
artifacts
that
it
generates
are
all
the
data,
including
the
data
that
you
feed
to
your
model
for
training.
A
So,
if
I
take
that
pipeline
and
region
from
the
artifacts
perspective,
I'm
looking
at
this
and
to
to
to
explain
more
on
what
that
is,
I
have
this
stream
pipeline,
which
is
a
continuous
funnel
that
keeps
dropping
data
into
my
warehouse
and
on
my
warehouse
side.
What
I'm
doing
is
really
segmenting
the
animals
rip
our
images
and
masks
but
indexed
by
their
ID
and
then
I.
A
That
starts
observing,
and
then
this
to
API
is
really
one.
Is
that
serving
the
prediction
over
gr,
CPR,
rest
or,
and
also
one
endpoint?
That's
a
feedback
in
point
that
consumes
and
the
true
prediction
that
should
be,
and
also
the
prediction
that
model
has
produced
and
that
feedback
and
is
then
fed
back
to
the
data
warehouse
which
obviously,
you
probably
would
want
to
put
a
verification
step
there,
because
it
can
be
exploited
to
a
great
extent
if
exposed.
A
So
that's
why
I
have
the
manual
verification
there,
but
the
point
here
is
that
there
needs
to
be
a
funnel
that
goes
back
into
your
warehouse,
taking
in
all
the
all
the
corrections
that
should
be
in
the
model,
and
so
with
that
diagram
and
with
that
sort
of
flow
of
the
data
and
the
artifacts.
Let's
talk
about
what
are
the
tools
we
are
using
to
realize
that
that
solution
now
using
pachyderm
for
provenance,
packing
them
is
a
great
tool
for
data
provenance.
It
gives
get
like
construct
for
data.
A
Basically,
your
data
sits
in
your
repository
and
everything
is
versioned
on
your.
So
you
can
pretty
much
ask
for
your
files
on
a
particular
version
and
it
D
duplicates
under
the
hood,
and
then
there
is
another
version
of
automated
repository
where
your
pipeline
really
acts
on
your
input.
So
any
input
repository
with
a
particular
version.
It
will
act
on
the
input
data
and
creates
an
output
and
stores
the
output
in
the
repository
with
that
version.
A
So
so
so
then
we
can
use
these
to
construct
to
build
a
complex
diet,
something
like
what
we
look
before
and
have
a
full
provenance
across
everything.
So
in
our
example,
pachyderm
is
going
to
be
the
battle
doing
the
heavy
lifting
of
provenance.
Then
we
obviously
have
cue
flow.
There
are
quite
a
few
talks
around
in
this
conference
around
cue
flow
and
cave
serving
as
well,
that
that
really
tells
us
the
the
adaptation
of
this
tool,
so
III
assume
that
I
probably
do
need
to
talk
too
much
about
it.
A
But
chip
flow
is
really
just
a
it's,
not
a
solution
to
a
machine
learning
problem.
It's
just
bringing
all
the
solutions
together
into
one
platform,
to
solve
machine
learning
problems,
and
there
is
we
really
talking
about
tuning
training
pipelining
in
and
Jupiter,
because
of
course,
no
data.
Science
is
ever
possible
without
a
notebook.
So
so
then
we
have
a
cube
flow
that
we
would
be
using
to
perform
ml
specific
operations.
But
we
have
a
problem
because
pachyderm
n
cube
flow,
don't
gel
very
well.
A
Cue
flow
is
very
kubernetes
native
packing
no
runs
on
kubernetes,
it's
not
very
native
to
kubernetes.
You
can't
really
use
operators
and
have
a
better
integration
between
queue
flow.
You
can't
really
use
operators,
the
teams
I'm
sure
are
working
on
it,
but
as
of
today,
it's
not
possible
and
that's
the
challenge
we
have
in
seamless
integration
and
we
are
going
to
have
a
look
at
how
to
to
really
simplify
that
to
some
extent
so
I'm
taking
tuning
as
an
example
here-
and
this
is
this-
is
a
specification
of
pachyderm.
A
You
really
specify
your
input
and
in
this
case
the
transformation
is
transform.
Is
the
repository
that
we
looked
at
now,
I'm
going
to
slide
back
to
show
you
what
transformation
was
which
is
the
third
repository
there
and
it's
creating
the
training,
validation
calibration
and
test
set?
So
when
we
are
tuning,
we
really
want
the
training
and
validation.
So
we
really
want
the
input
coming
from
the
transformation
step.
A
So
then,
in
pipeline
we
are
saying
from
transform,
give
me
everything
and
then
apply
this
transformation,
which
is
the
tuning
for
me
and
generate
all
the
results
into
my
tuned
repository
now.
This
works
great
wonderful,
I
use,
I
use
ring
in
container
and
it
does
the
a
parameter
tuning
using
a
really
complex,
algorithm
called
hyper
band.
So
the
point
here
is
that
complex
machine
learning
is
possible
in
pachyderm,
but
because
it's
not
giving
it's
not
leveraging
the
benefit
of
the
distribution
and
scale
that
cube
flow
provides.
A
A
So
this
is
an
example
of
the
similar
tuning
sort
of
operation,
but
what
we
are
doing
is
we
are
transforming
we
are
taking
in
transform
repository
as
an
input,
and
then
we
are
applying
the
transformation
step
which
is
even
with
Gotham,
and
what
that
does
is
starts
a
packet
of
experiment
from
from
pachyderm
container
and
injects
the
provenance
information.
So
what
sort
of
version
of
tun
repository
I
should
be
using,
and
what
should
my
output
version
be
and
with
that
it
starts
the
trial
and
trial?
A
So
that's
that's
pachyderm
in
conjunction
with
to
flow
with
with
full
provenance
and
that's
an
example
of
how
we
cut
from
pachyderm
took
you
through
and
come
back
in
here's
another
example
of
realizing
of
serving
the
prediction
with
the
version
generated
by
pachyderm.
So
we're
doing
pretty
much
same
thing.
We
start
up
Seldon
deployment
from
pachyderm
for
injects
the
provenance
and
once
we
inject
the
provenance,
the
Selden
diploma
and
then
takes
takes
care
of
starting
the
part
with
with
the
model
in
it
and
then
the
part
starts
up.
A
A
So,
which,
in
case
of
Sultan
port,
it
will
download
the
files
and
start
making
the
prediction
API
available
onto
that
particular
version,
and
then
the
API
is
just
available
with
both
feedback
and
serving
now
I'm
I'm.
Being
cheeky,
I've
hacked
up
the
reinforced
Luhrmann's
learning
API
of
send
feedback
to
do
the
feedback,
but
it's
certainly
possible
to
add
a
custom
API
on
top
of
Selden
or
means,
and
this
works
with
any
other
deployments
as
well.
A
So
now
talking
about
the
expect,
the
replicability,
which
is
an
extension
to
reproducibility,
and
this
caters
to
the
scenario
where
we
are
not
just
talking
about
getting
to
the
same
answer
on
on
your
cell
data
same
algorithm,
but
also
what
happens
when
your
input
is
a
slightly
change
or
slightly
perturbed,
or
maybe
added
a
noise?
And
that's
when
we'd
start
talking
about
robustness.
There
are
there
quite
a
few
approaches
around
to
make
your
model
more
robust
things
like
which
is
leekens
library,
along
with
some
of
his
colleagues
there.
A
You,
you,
improve
your
model
with
just
like
argumentation,
you
sort
of
improve
your
model
with
introducing
noise
on
the
input
set.
There
are
similar
libraries
like
Google's
and
sell
natural
structure.
Languages
I
think
it's
called
their
full
box
and
there
are
many
more
libraries
that
allows
you
to
robust
your
models,
and
these
are.
There
are
some
great
sayings
by
self
saying
that
model
generalized
across
architecture
and
training
sets
are
the
good
model
and
really
required
in
our
time
where
the
we,
our
answers,
are
really
impacting.
A
You
know,
inducers
think
about
loans
or
a
scenario
or
even
in
the
image
like
if,
for
example,
I
talked
about
going
just
going
back
to
the
example
of
pet
set
here.
So,
if
I'm
doing
our
inference
on
looking
for
polls
in
the
imagery-
and
that's
really
just
a
very
small
artifact
in
a
big
imagery
and
it's
really
hard
to
miss,
if
just
losing
out
one
drop
out
layer
gives
me
that
sort
of
result
with
so
much
of
variance.
A
So
now
this
brings
me
to
what
are
the
other
tools,
so
I
knew
about
DBC
and
ml
flu
that
can
be
used
to
sort
of
realize
ml
flow,
but
I
learnt
in
this
conference
that
there
is
another
way
to
do
that,
which
is
now
using
a
Richter
with
with
two
flow
I
think
it
was
something
new
that
I
didn't
know
before.
So
that's
another
way
to
really
realize
that,
and
that's
really
all.
If
there's
any
question,
I
don't
like
to.