►
From YouTube: Machine Learning Model Serving and Pipeline Using KNative - Animesh Singh & Tommy Li, IBM
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Machine Learning Model Serving and Pipeline Using KNative - Animesh Singh & Tommy Li, IBM
Lifecycle support— including continuous development, training, testing, and deployment of machine learning models—and continuous integration (CI) for AI applications is still in its infancy. We need a solution that enables end-to-end automation of data preparation and model deployment pipelines.In this talk we are going to show how to leverage KNative components to create an event driven AI pipeline. We will leverage OpenWhisk and Kubernetes to provide an event driven platform, and Istio for traffic management and observability to construct a pipeline which will provide interfaces to various open source tools: model training, validation. serving platforms on KubernetesWe will show how we can leverage this AI pipeline to train using advanced batch scheduling in Kubernetes, automate A/B tests and canary testing of models, monitoring concept drifts and accuracy losses etc.
To Learn More: https://sched.co/GrVt
A
Okay,
hello
and
thanks
everyone
for
coming.
This
looks
like
a
mini
keynotes
tunnel
with
the
number
of
people
here
so
I
believe
it's
it's
the
topic
of
the
day's
yeah
in
machine
learning
in
general,
combined
with
serverless
right,
which
is
often
dressed.
Definitely
how
many
people
here
have
tried
to
use
cane
a
day,
more
I've
heard
about
Canadian
quite
a
bit,
so
that's
great
I
mean
before
I
begin.
The
talk
right,
one
of
the
things
I
do
want
to
make
sure
to
convey.
Is
that
you
know?
A
Essentially,
while
we
are
trying
to
do
this
and
build
this
ml
platform,
leveraging
K
native
K
native
is
still
in
very
early
stages.
It's
an
evolving
project
right.
There
are
a
lot
of
bright
points
in
terms
of
the
concepts
and
where
it
intends
to
be,
but
definitely
there
is
a
lot
of
work
which
needs
to
happen
on
other
components
of
Canadian.
A
My
self
is
an
image
saying
this
is
my
colleague
Tommy
Lee.
Together,
we
work
for
IBM
in
a
group
called
Cole,
a
Center
for
open
source
data,
nei
technologies,
and
we
work
on
open
source
projects
throughout
the
AI
lifecycle
around
the
three
pillars
of
code
content
and
community.
We
contribute
heavily
to
spark
Chancellor,
Flo
Cara's,
a
lot
of
other
projects
from
IBM's,
open
source
or
only
a
lifecycle.
We
create
a
lot
of
content
around
them.
A
What
are
we
going
to
talk
about
today?
Essentially
some
progress
in
machine
learning
and
deep
learning
in
general?
Why
we
think
here
in
cloud
or
complementary,
then
one
of
the
questions
right?
Why
do
we
need
something
like
a
native
to
build
an
AI
platform?
Is
it
really
needed
and
then
how
do
we
build
a
transparent
and
trusted
AI
platform
level,
18k
native,
followed
by
a
quick
demo?
A
For
me,
the
first
time
when
I
heard
about
AI
in
general
was
back
in
1997
when
I
was
back
in
India
of
an
IBM's
deep
blue
beat
garry
kasparov
in
chess.
That's
well!
You
know
this
whole
notion
of
machine
learning,
artificial
intelligence.
You
know,
that's
something
you
know
crossed
my
eyes
and
then
2011
was
another
aha
moment
of
an
Watson,
actually
beat
jeopardy
champions
right
and
very
recently,
which
probably
most
of
us
have
seen
have
heard,
is
essentially
deepmind
going
against
and
winning
the
go
championship.
A
A
What
exactly
is
deep
learning?
Essentially,
we
are
training,
artificial,
neural
networks.
If
we
look
at
the
capacity
which
it
has
achieved,
around
25
million
neurons
100
million
connections,
compare
it
with
human
brain
200
billion
neurons,
32
trillion
connections.
So
that
means
what
deep
learning
is
achieved
today
is
only
0.0003
percent
of
what
a
human
brain
can
do,
but
in
spite
of
you
know
achieving
that
less
of
a
capacity
it's
able
to
do
so
much
that's
humongous.
A
What
is
the
typical
neural
network
design?
Workflow,
look
like
right.
A
data
scientist
comes
designs,
the
initial
neural
network.
Then
you
assign
hyper
parameters,
point
it
to
your
domain
data
and
then
train
the
model.
As
you
were,
training
the
model
or
after
you
have
trained
the
model
is
the
performance
meeting
your
needs
know.
Then
you
go
and
reach
week,
your
hyperparameters
start
another
experiment
right
and
you
go
through
this
in
several
loops.
A
If
it
does
meet
you
need,
then
you
have
the
train
model
which
you
actually
go
ahead
and
deploy,
but
your
job
doesn't
stop
here.
Typically,
most
of
the
applications
once
you
deploy
right,
they
perform
consistently
in
case
of
models
when
they're
actually
deployed
in
real
world
when
they're,
actually
predicting
and
doing
in
dancing
against
real-world
data.
You
start
saying
you
know
drift.
You
start
seeing
accuracy
loss,
you
start
seeing
predictions
you
know
which
were
way
off
right.
So
that
means
you
have
to
again
start
another
experiment.
A
Go
through
this
process,
so
one
thing
is
very
clear:
yeah
in
general
and
deep
learning
in
particular
of
very,
very
isolated
and
very
very
repetitive,
so
they'll
eat
cloud
right.
It
requires
the
strength
of
HPC
and
GPUs.
You
want
to
be
able
to
scale
your
AI,
both
clothes
on
tomorrow,
and
you
also
want
to
utilize
various
technologies
to
achieve
high
performance
computing.
So
a
lot
of
innovation,
which
is
happening
around
different
algorithms
and
our
model.
A
So
it's
very
clear
that
to
scale
we
need
to
go
cloud
native
for
a
a
that
means.
You
know
we
need
the
concept
of
micro
services,
containers,
DevOps
automation,
so
that
means
you
know
we
use
Kuban
at
ease.
It's
very
well
established
that
if
you
are
thinking
cloud
later,
you
know
we
are
using
Chubin
at
ease,
but
as
cuban
it
is
enough
for
a
cloud
native
platform
right.
If
he's
look
at
the
layers,
there
are
multiple
layers.
So,
as
Kelsey
Hightower
says,
right,
kubernetes
is
not
the
end
game.
A
It's
a
platform
to
build
another
set
of
platforms,
and
that's
where
you
know
some
other
aspects
come
in.
So
what
else
do
we
need?
Let's
understand
it
from
the
context
of
an
AI
lifecycle
right,
typically,
an
AI
lifecycle.
You
prepare
and
analyze
the
data.
Then
you
create
your
initial
model,
your
sign
have
their
parameters
and
train.
Then
you
validate
your
model
against
robustness,
checks
or
bias,
checks
accept
or
deploy.
Your
model
then
continuously
monitor
it
capture
the
payload,
and
then
you
know
make
sure
that
it's
not
drifting.
The
accuracy
is
right.
A
So
what
we
need
essentially
is
a
transplant
and
trusted
AI
platform,
so
that
you
know
you
are
able
to
see.
What's
going
on
throughout
the
CI
lifecycle
is
the
training
data
changed?
Is
the
data
set,
biased
is
model
training
showing
increasing
loss
or
model
weights,
biased
or
the
classifiers,
which
you
use
they're,
biased
right
on
the
model,
addictions
accurate,
I
directly?
You
would
want
this
to
be
automated
right,
so
what
you
need
is
transparent,
trusted,
automated
and
even
driven
AI
pipeline.
A
So
instead
of
you
going
and
looking
at
everything
which
is
going
on
inside
the
platform,
as
you
are
running
through
this
AI
pipeline,
the
platform
should
be
able
to
tell
you
that
you
know.
What's
going
on
whether
the
model
accuracy
is
low
or
the
data
set
is
biased,
you
should
be
able
to
be
notified
and
based
on
those
notifications,
the
platform
should
be
smart
enough
to
take
certain
actions
to
handle
those
right.
A
So
as
far
as
possible,
you
don't
want
to
bring
a
human
in
this
loop
and
when
we
work
with
enterprises,
it's
very
very
critical
that
you
know
we
need
the
notion
of
auditability
right.
So
that
means
having
the
right
concepts
around
data
versioning
model,
versioning
tracking,
every
input
and
output
is
very,
very
key
to
a
platform
or
a
pipeline
like
this,
and
in
the
end
right.
A
You
know
how
much
is
the
execution
time
looking
at
the
logs
if
things
go
wrong
and
some
in
something
which
is
you
know
so
long
running
as
this
things
will
go
wrong
right,
so
having
a
very
robust
to
be
able
for
data,
DevOps
operators
is
very
very
necessary
now
if
we
translate
it
to
logical
Arctic
architecture.
It
will
look
something
like
this.
Where
we
have
one
actor,
which
is
you
know,
AI
developer
and
data
scientist,
then
we
have
AI
operators
and
for
AI
developers
and
data
scientists
for
the
most
part,
their
world
is
Python
right.
A
They
want
to
enter
it
through
a
Python
world.
You
want
to
give
them
ability
to
actually
craft
these
pipelines,
using
a
sequence
of
Python
functions
and
on
the
backend
side
right.
You
have
your
complex
pipelines
around
vertical
domains
like
on
data
on
training,
side
on
model
solving,
and
then
you
know
the
operations
beyond
the
model.
Sorry,
so
we
definitely
need
more
than
communities
right.
We
need
to
build
data
scientist
code.
We
need
to
be
able
to
orchestrate
that
machine
learning
code,
and
you
know
beyond
that.
A
A
So
what
exactly
is
Canadian,
so
it
provides
a
set
of
building
blocks
to
enable
model
source
centric
container
based
service
workloads
on
top
of
communities
right.
So
there
are
new
set
of
CRTs
which
have
been
created
to
provide
high
level
API
abstractions
around
the
four
key
areas
right
now,
which
are
built
eventing
serving
and
pipeline.
If
we
look
at
build,
what
does
it
means
essentially
source
to
container
orchestration?
A
Now,
if
you
look
at
in
the
context
of
AI
lifecycle,
right,
take
data
scientist
code
and
then
run
it
right,
build
it
so
that
it
can
run
on
top
of
communities,
can
AD
build
components,
are
essentially
build
builder
and
build
template.
A
build
is
a
collection
of
steps.
Each
of
these
steps
can
be
backed
by
a
builder
and
builders
in
general,
or
container
images
right
and
a
build
template
is
parametrized
and
reusable
template,
which
you
can
use
along
with
your
builds.
A
Eventing
is
all
around
the
delivery
and
management
of
events,
so
the
idea
is,
there
are
some
since
certain
sources
which
will
fire
events.
Those
events
go
to
a
channel
and
there
are
certain
clients
which
are
subscribing
to
those
channels
right
which
will
take
certain
actions
based
on
those
events
right.
So
that's
the
concept
around
eventing
and
something
you
know
which
has
come
up
recently
in
the
que
native
world,
the
Kennedy
pipeline,
which
is
essentially
you
know
how
do
you
configure
and
run
CICE
pipelines
leveraging
chip
in
IDs
right,
so
the
concept
of
tasks
applications
runs.
A
So
essentially,
you
know
a
collection
of
toss,
forms
your
pipeline
and
then
use
tensho
to
pipeline.
That's
a
run
and
the
multiple
runs
can
actually
you
know
point
your
experiment,
and
last
but
not
least,
the
most
mature
component
of
K
native
is
essentially
K
native
solving
right.
This
is
the
first
one
where
a
lot
of
work
actually
went
in
right
and
it
is
built
around
the
pillars
of
configuration,
revision,
route
and
service
next
and,
let's
understand
them
a
bit.
A
A
revision
is
essentially
built
on
top
of
deployments
communities,
deployment
primitives,
which
one
builds
on
top
of
replicas
sets.
So
that
means
you
know,
based
on
your
request,
you
can
scale
up,
but
also
which
came
in
something
new
is
essentially
you
know.
If
the
request
goes
down,
you
can
scale
back
to
zero.
Now.
Revisions
are
immutable.
They
are
essentially
snapshots
of
your
code
and
configuration.
So
that
means
you
know
to
actually
create
a
new
revision.
You
will
have
to
change
the
configuration
once
you
change
the
configuration
a
new
revision
is
created.
A
Then
there
is
CRT
around
roads,
so
routes
are
essentially
long-lived.
Stable,
HTTP,
end
points
through
which
end
users
access
your
kubernetes
key
native
applications
or
services
right.
So
roads
can
actually
point
to
a
particular
revision,
or
you
can
actually
split
traffic
between
multiple
revisions.
A
So
we
need
configuration
route
and
revision
within
the
creative
serving
and
to
create
all
these
objects.
There
is
a
higher
level
abstraction
which
has
been
created
in
creative,
which
is
K
native
service,
so
service
actually
creates
a
configuration,
creates
a
route,
and
then
you
know
as
you're,
creating
multiple
configuration
you're
getting.
You
know
different
deviations,
so
essentially
K
Native
uses
humilities
CRTs
to
define
a
new
set
of
API,
surround,
build
eventing
serving
and
pipeline
great.
A
But
do
we
really
need
something
like
K
native
to
build
an
ml
platform
right?
So
for
those
of
us
who
are
actually
you
know,
living
and
breathing
the
machine
learning
world?
Does
it
still
make
sense
that
you
need
something
like
a
native?
Let's
go
through
our
own
use
case.
We
are
being
as
part
of
our
group
in
coda
right.
We
have
been
trying
multiple
use
cases
with
open
source
technologies.
What
will
be
the
right
strategic
platform
right
set
of
open
source
technologies
to
build
our
ml
platform?
A
We
do
have
a
lot
of
hosted
platforms
right
under
our
products
umbrella,
but
looking
at
the
next
gen,
you
know
how
should
it
be
built
right?
So
if
you
look
at
the
first
part,
which
is
essentially
you
know,
how
do
we
create
your
initial
models
right?
There
are
many
tools
available:
Jupiter
notebooks,
our
studio,
anaconda
SPSS
modeler
very
well
established
the
second
part
right
we're.
A
Essentially,
then,
you
are
performing
complex
mathematical
functions,
so
so
many
libraries
you
know
which
have
been
popular
like
scikit-learn
actually
boost
and
now
with
the
advent
of
deep
learning
chance
of
flow
PI,
Taj
MX
net
acceptor
right,
which
are
being
used.
If
you
go
to,
for
example,
you
know
in
China,
paddle
paddle
is
gets
used
heavily.
A
If
you
go
in
Japan
China
is
something
which
is
used
heavily
right
so
also
here
I
mean
a
lot
of
people
like
chance
and
flow,
but
you
know
it
had
static
graphs,
so
by
thoughts,
dynamic
grass
became
more
popular
right.
So
how
do
we
handle
first
all
these
different,
multiple,
open
source
frameworks
in
a
consistent
manner,
for
that
we
launched
something
called
fabric
for
deep
learning
or
Friddle.
A
It
essentially
is
a
deep
learning
distributed
deep
learning,
training
platform
on
top
of
Cuban
eighties.
It
is
the
winner
of
info
world's
best
of
machine
learning
and
deep
learning
software
for
this
year
right
and
it
provides
a
constrained
way
to
actually
chain
your
models,
visualize
your
model,
matrix,
etc.
A
Essentially
Fidel
is
built
using
micro-services
architecture
on
top
of
kubernetes,
and
we
are
leveraging
scalability
resiliency
for
tolerances,
which
comes
from
cuban
at
ease
into
freedom.
We
are
being
testing
on
on
coming
community
communities,
distribution
on
IBM
cloud,
private
or
public
cloud
version
or
d
ing
right,
so
you
can
actually
run
it
on
various
distributions
of
communities
and
the
idea
is
definitely
right.
We
want
to
access
compute
when
the
request
for
a
training
job
comes
in.
We
actually
spread
that
training
across
multiple
containers.
A
So
that
means
a
your
infrastructure
is
only
used
when
needed
and
once
the
training
jobs
you
know
complete,
then
the
infrastructure
is
released
back.
This
is
the
high-level
architecture
for
fertile.
We
essentially
have
control,
plane
micro
services,
starting
with
rest
api's,
going
all
the
way
to
lifecycle
manager,
which
essentially
is
the
brain,
and
then
the
learner
ports
were
essentially
you
know
on
request
when
the
training
job
comes
in,
they
are
launched
and
they
have
certain
Sitecore
or
helper
processes
which
actually
collect
the
log
of
lot
of
the
logs
and
evaluation.
A
A
But
another
thing
which
we
found,
which
was
missing
in
communities,
was
essentially
you
know
the
concept
of
batch
scheduling.
The
cumulative
scheduler
on
its
own
leaves
a
lot
to
be
desired
for
batch
workloads.
So
when
you
are
running
distributed,
deep
learning,
training,
jobs
of
machine
learning,
training,
jobs,
these
are
long-running
jobs
and
you
have
something
which
comes
as
a
high
priority
right.
You
have
to
hold
certain
jobs,
make
sure
the
high
priority
request
gets
in.
A
You
need
the
concepts
of
gang
scheduling,
priority
being
able
to
post
certain
jobs
for
that
IBM
contributed
a
project
in
open
source
called
cube
batch.
It
is
in
communities
incubator.
Now,
it's
being
worked
on
and
heavily
contributed
by
the
community,
essentially
becoming
the
answer
for
batch
scheduling
on
top
of
communities.
So
a
lot
of
the
concepts
I
talk
about
preemption,
you
know
holding
a
job
being
able
to
make
sure
that
we
give
high
priority
request.
A
Another
data
point
is
that
right,
IBM
Watson
workloads.
We
have
it
running
on
kubernetes
right
now
there
are
12
Watson
services
which
are
leveraging
kubernetes
are
only
800
plus
human
ad
services,
and
each
of
these
service
out
of
800
on
an
average,
runs
around
3,000
plus
pods
on
500
plus
nodes.
So,
as
you
can
imagine,
you
know,
we
are
leveraging
kubernetes
heavily
within
IBM
to
run
a
lot
of
our
AI
n
ml
workloads.
A
So
great
we
have
training
accomplished
using
federal
right,
our
model
is
ready,
can
be
trusted.
How
do
we
validate
and
trust
our
model
right
now?
What
do
we
mean
by
trusting
model?
Is
where
is
it
elite
easy
to
understand?
Can
anybody
tamper
with
it?
So,
let's
start
with
vulnerability.
Detection
of
models
is
the
model
vulnerable
to
adversarial
attacks?
Can
somebody
contaminate
the
data
set
and
have
model
building
or
start
giving?
You
know
very,
very
vastly
different
predictions
for
that.
A
We
launched
the
tool
in
open
source
again
call
adversarial,
robustness,
toolbox
or
art
in
short,
and
it's
essentially
your
Python
library,
which
has
a
lot
of
attack
and
defense
methods
implemented
out
of
research
papers.
So
you
can
actually
point
to
your
models.
You
know
check
for
vulnerability
and
if
valer
ability
is
found,
then
your
neck
and
then
actually
implement
one
of
the
defense
mechanisms.
A
So
that
helps
you.
Do
the
robustness
check
right?
Another
topic,
which
is
very
hard
for
model
validation
and
which
is
very
hot
as
well
right
which
is
being
talked
about,
is
the
bias
spot
right?
So
is
your
data
set
actually
to
begin
with,
was
biased
or
the
model
weights
and
the
classifiers
which
you
are
using?
They
are
biased
once
your
models
are
deployed,
it
starts
giving
predictions.
Are
they
biased
right?
A
And
then
you
know
it
has
a
lot
of
bias
mitigation,
algorithms
right
it
works
across
the
entire
lifecycle,
not
only
to
detect
bias,
but
also,
you
know
mitigate
bias
wherever
you
find
it
and,
as
I
said
that
you
can
actually
use
it
to
detect
bias
at
different
levels
of
the
whole
lifecycle,
it
gives
you
metrics,
algorithms
and
explainers
explaining
you
know
why
it
forms
certain
thing
biased
and
then
you
know
algorithms
to
mitigate
that,
so
our
model
is
trained,
tested
and
validated.
Should
we
just
deploy
it
as
micro
services
on
communities?
A
Should
we
need
anything
else
right?
So
a
couple
of
questions
arise
right.
How
can
I
add
and
enforce
policies
on
my
models
right?
So
a
certain
set
of
users,
you
know,
are
able
to
access
a
certain
version
of
the
model
as
I
talked
about
you
know,
this
model
training
process
is
very
iterative,
very
repetitive
right.
So
at
many
points
you
will
have
different
versions
of
the
models
coming
in.
How
do
you
actually
do?
A/B
testing
Cannery
testing
do
launched
a
lot
of
those
questions
arise
for
that
we
use
internally
in
IBM,
sto,
right
and
I.
A
Think
a
lot
of
us
are
aware
is
to
as
an
open
surface
mesh
platform
to
connect,
observe
secure
and
control
micro
services,
and
essentially
you
know
we
want
to
control
the
traffic
we
want
to
discover.
You
know.
Micro
services
provide
load
balancing
around
them
and
it
also
gives
a
lot
of
matrix
logging
and
tracing
capabilities.
The
other
key
aspect
of
sto
is
essentially
that
you
know
there
isn't
secure
TLS
communication
between
micro
services,
so
you
get
a
lot
of
the
security
associated
with
it
last
but
not
the
least.
A
Definitely
you
know
the
policy
informant
piece,
and
how
does
this
chip
work
in
general?
Suppose,
a
micro
service,
a
cos
B
you
deploy
envoy
sidecar,
along
with
the
micro
service
right
and
the
job
of
the
sidecar,
is
to
actually
intercept
any
incoming
traffic
or
outgoing
traffic
right.
So
we
install
a
pilot,
then
you
need
a
lot
of
the
telemetry.
A
And
then,
once
you
have
this
to
you,
then
you
can
have
you
know,
or
this
inference
platform
right,
which
can
then
allow
you
to
launch
multiple
versions
of
your
model
and
then
be
able
to
control
traffic
between
them
be
able
to
split
traffic
between
them
based
on
a
particular
user.
You
can
actually
redirect
traffic
to
certain
users
or
you
can
is
for
certain
policies.
Right
and
the
great
part
is
SEO
is
part
of
Canadian
right,
so
that
eliminates
the
need
for
someone.
A
You
know
to
actually
looking
at
two
different
things,
bringing
them
together
if
you're
looking
at
K
native
stack,
the
idea
is
that
you
know
is
to
you
comes
along
with
it.
So
that's
great,
so
model
is
deployed
right.
What
else
do
we
need
now?
We
need
to
ensure
triggers
and
actions
can
be
defined
right,
based
on
any
vulnerability
detection
data
set
changes,
bias,
detection
except
all
right,
enter
Apache
open
risk,
so
we
use
open
whisk
internally.
A
Essentially
we
have
our
IBM
cloud
functions
platform
based
on
it,
it's
in
open-source
since
the
last
two
to
three
years.
It's
part
of
Apache,
and
you
know
we
have
partners
like
Red
Hat
in
Adobe
who
contribute
to
it
and
the
whole
idea
behind
open
whiskey
is
essentially
you
know,
service
platform
to
execute
on
actions
in
response
to
you
know
certain
triggers
right,
so
you
have
sources
which
are
generating
events
based
on
that
you
know
triggers
actually
encapsulate
the
response.
A
You
can
actually
write
your
openness
functions
in
multiple
languages.
Python
swift
go
node,
mostly
the
ml
volt
Python
becomes
very
important
right.
We
are
actually
also
working
with
pyrin
to
actually
include
some
of
those
capabilities
in
open
lemaise,
and
you
can
actually
run
it
on
chip.
Minetti's,
Miso's
doc
would
compose.
So
that
means
you
know
any
flavor
of
docker.
You
need
and
we
have
a
lot
of
event.
Providers
integrated
within
open
risks.
A
So
that
means
you
know:
periodic
cron
jobs
which
can
actually
emit
triggers
Cloudant,
which
isn't
coach
TB
under
the
covers,
so
any
data
being
stored
in
Cloudant
or
CouchDB
will
fire
in
event.
You
know
message
hub,
which
is
you
know,
cough
corner.
The
covers
mobile
push
github
any
activity
on
github.
You
know
we
can
fire
an
event
and
then
have
actions
which
can
process
those.
A
And
the
greatest
use
news
is
that
open
risk
is
actually
being
developed
to
work
with
k
native
I
mean
just
today
and
yesterday
you
know
you
would
have
heard
announcements
where
IBM
s
AP
Red,
Hat
Google.
You
know
they
are
joining
hands
together
in
terms
of
evolving
k,
native
more
so
on
the
server
left
side,
and
there
is
a
lot
of
work
happening
both
on
the
IBM
side,
as
well
as
with
Red
Hat.
Where,
essentially,
you
know,
we
are
ensuring
open
vest
becomes
a
first-class
citizen
of
K
native.
A
So
the
trainer
model
we
deployed
a
model
we
are
able
to
validate
it
using
robustness,
check
fairness
check
right.
What
do
we
use
for
pipeline
orchestration?
Well,
so
internally,
within
IBM?
Right
now
we
are
using
IBM
DevOps
tool
chain,
which
is
built
on
top
of
Jenkins
on
open
source.
We
are
investigating
a
lot
of
technologies,
pachyderm
or
go
airflow.
All
these
are
interesting
technologies
which
we
are
investigating
for
AI
pipeline
orchestration
right,
k
native
actually
comes
with
its
own
pipeline,
though
the
caveat
here
is,
you
know
it
still
in
very
early
stages.
A
Finally,
once
your
AI
ops
platform
is
ready
right,
how
do
data
scientists
interact,
for
example,
using
jupiter
notebooks,
so
for
that
we
launched
a
project
initially
in
the
jupiter
incubator
system
called
jupiter
enterprise
gateway?
Now
it's
a
top-level
project
in
jupiter,
and
the
idea
here
is
that
when
you
are
working
with
jupiter
on
your
local
laptop-
and
you
know,
if
the
capacity
is
not
enough,
then
you
can
actually
launch
a
builder
kernels
remotely
on
a
Hadoop
cluster
on
a
cuban,
a
DS
cluster
and
leverage
that
capability
right.
A
A
Then
you
know
being
able
to
again
once
your
models
are
deployed,
being
able
to
capture
that
payload
and
look
for
bias
and
other
metrics
there.
You
know
we
can
actually
build
out
a
ops
platform
with
that,
we'll
go
into
a
quick
demo
right
and
what
we
are
going
to
use
in
this
particular
demo
is
essentially
a
whitewash
gender
classification
model
right,
which
is
a
very
simple
model
right,
using
CNN
with
three
convolutional
layers
and
two
fully
connected
layers
built
on
the
utk
phase.
A
Data
set,
and
it
has
a
very
simple
job,
essentially
to
be
able
to
predict.
You
know.
But
then
the
image
uploaded
is
a
male
or
a
female
right,
and
if
we
look
at
the
demo
flow
right
now,
what
we
are
using
is
essentially,
we
are
using
fabric
for
deep
learning
under
the
covers
for
training
on
top
of
cuban
''tis.
A
The
data
is
stored
in
object,
storage
right
and
then
you
know
once
the
model
is
trained,
then
you
have
the
ML
code,
then
you
know
we
use
Kennedy,
build
source
to
orchestration
to
actually
create
containers
for
robustness,
check
which
is
under
the
covers.
You
know,
using
our
toolbox
for
fairness
check
using
AFN
is
360
and
then
you
know
deploy
the
model.
Finally,
we
will
also
show
that
you
know
once
your
model
is
deployed
and
you
change
something.
You
launch
a
new
version
of
the
model.
How
do
you
actually
do?
B
Hi
I'm,
Tommy
and
now
I'm
go
ahead
to
show
you
our
demos
for
ml
pipeline
using
candidate
so
for
the
demo
flow.
We
first
built
the
air
furnace
360
and
a
ever
serve
purposes
toolbox
using
kinetic,
build
components
then
servers
using
Kennedy's
service,
then
from
there
we
will
blow
out
the
release
of
the
model
using
canary
release.
Then
we
will
apply
some
kinetic
produce
to
split
the
traffic's
and
perform
AP
testing
and
that's
the
august
rate.
B
B
B
B
We
want
to
go
to
our
pipeline.
You
I'll
show
you
how
each
partner
component
is
done
so
over
here
we
can
see
our
training
status
is
complete
and
we
want
to
submit
a
process
check.
Essentially,
we
apply
a
attack
methods
to
add
some
perturbation
to
the
gradients
and
see
is
the
model
able
to
handle
those
attacks.
To
do
this,
we
submit
a
child
which
have
our
credentials
without
data
and
our
trainer
model
located
chris
doumitt
and
from
tentative
serving.
B
If
we
will
see
this
traffic's
want
to
do
some
robustness
checked,
so
we
will
publish
and
a
new
container
the
scale
from
zero
to
one,
and
you
look
back
to
our
key
Rd
graphs.
You
can
see
this
traffic
coming
to
the
low
passes,
checked
on
shade
of
native
service
and
then
later
on.
You
can
see,
since
we
don't
have
any
containers,
people,
buses,
Python,
tentative
service.
We
asked
the
Auto
scale
that
the
scale
from
zero
to
one
and
once
the
process
check
is
done,
we
will
return
a
list
of
metrics.
B
Ladies,
so
to
do
the
unfinished
check,
we
will
do
the
same
things
and
then
you
will
see
Kennedy,
also
publish
and
a
new
container
for
the
finished
component
now
get
to
the
same
thing
for
serving
and
for
serving
will
you
define
our
experiments
with
the
group,
all
Chinese
job,
interesting
experiment
and
track
them
using
the
key
native
of
revisions?
Names.
B
So
so,
once
it
click
on
our
deployments,
if
you
also
create
other
nicotine
controllers,
which
it
will
create
a
models
with
a
model
with
revision
names,
and
we
will
use
that
to
check
our
model
versions
and
usually
I'm
in
the
machine
learning
training,
we
would
use
it
to
multiple
trainees
with
module
parameters
and
then
group
them
together
as
an
experiment
from
here.
We
want
to
deploy
another
models
with
the
different
revision
numbers,
and
then
we
will
perform
some
AP
testing
using
the
kinetic
routes.
B
So
now,
let's
create
another
deployments,
other
new
models
and
this
model
we
will
have
our
vision,
names
model
serving
in
0-2
and
then
to
apply
a
Kennedy
brow.
Rules
is
30
symbols.
All
you
need
to
do
is
define
how
many
sharper
you
want
to
route
to
a
particular
relation
name
in
this
case
routing
90%
of
the
traffic's
to
model
serving
zero
once
and
the
rest
of
the
10%
traffic
to
model
serving
0-2.
B
B
B
So
now,
let's
go
back
to
our
PID
graph
and
see
how
our
traffic
is
routing.
As
you
can
see,
almost
90
percent
of
the
traffic
is
driving
through
the
model
serving
zero
once
the
rest
of
the
tempest
of
the
traffic
is
rolling
through
model
serving
zero,
two
is,
which
is
what
we
expected,
and
we
can
also
monitor
how
the
traffic
is
behaving
using
the
cofounder
graph
to
and
from
the
app
development
perspective.
It
would
be
seamlessly.
B
A
Sothanks,
what
you
just
saw
was
that
you
know
going
through
this
whole
pipeline.
You
know,
step
by
step
the
ideal
way
to
actually
do
it
would
be.
You
know
using
K
native
eventing,
and
then
you
know
wrapping
the
whole
right
hand
side
of
this
into
a
pipeline
right.
So
as
I
say
it
right,
we
were
definitely
evaluating
K
native
pipelines
for
this
I
think
you
know
there
is
still
some
work
which
needs
to
happen
on
this
site
for
it
to
become
a
robust
pipeline
platform.
A
There
are
a
lot
of
bugs
and
issues
right
for
this
particular
demonstration.
You
know
we
actually
created.
You
know
certain
Kennedy,
preventive
sources
channels
or
subscriptions,
which
can
then
trigger
off
beyond
the
training.
You
know
there
hope
to
kick
off
your
pipeline
right.
So
essentially,
once
your
training
is
complete,
you
can
actually,
you
know,
send
messages
to
a
channel
and
then
have
your
pipeline
subscribe
to
that
particular
channel
and
the
pipeline
will
then
start
executing
right.
So
that's
the
in
general.
A
You
know
the
concept
around
it,
and
then
you
know
you
can
actually
define
kale
in
the
pipeline
tasks
and
flow
definitions
so
can
native
actually
gives
you
a
very
you
know:
template
oriented
way
of
defining
what
are
the
tasks
which
you
want
to
run
into
this
pipeline,
and
then
you
know
how
to
tie
them
together
using
the
pipe
blowing
toss
and
the
flow
definition
for
this
demonstration.
We
actually
ended
up
using
alga
workflow
for
pipelining,
because
this
is
said
right
I
mean
we
still
need
to
make
sure
you
know.
A
A
As
you
can
see,
as
I
mentioned
right
so
within
IBM
right
now,
we
use
our
own
demo
stool
chain
pipeline
for
a
lot
of
these
automation,
which
is
built
on
top
of
Jenkins.
Definitely
looking
at
the
world,
which
is
more
oriented
towards
machine
learning
and
data
scientist,
a
a
lot
of
interesting
suit
tools
have
come
up
right.
Apache
airflow
is
a
great
tool
for
a
lot
of
pipelining
tasks
or
go
workflows,
which
we
are
right
now
using
for
this
demonstration.
When
we
actually
ran
that
pipeline
is
a
game
great.
A
A
Essentially,
you
know
you
can
actually
define
the
whole
set
of
tasks
and
what
you
want
to
do
into
this
pipeline
into
the
HTML.
So
add
a
concept
level
very,
very
similar.
What
your
Kennedy
pipeline
defines
right
and
to
the
credit
of
Margo.
This
is
one
of
the
most
mature.
You
know
workflow
orchestrators,
on
top
of
communities.
It
was
built
4
cuban.
It
is
right
from
the
beginning,
and
then
you
know
for
individual
jobs.
Once
you
click
on
them,
you
can
actually
look
at
their
Yama's
right.
A
A
Yeah
so
I
think
some
of
the
lessons
which
we
learned
as
we
went
through
this
whole
experiment
is,
and
a
concept
level
right,
training
to
bring
serverless
actions.
Events,
workflows,
apps,
container
service
mesh
in
one
single
stack
right
and
definitely
most
of
the
modern,
complex
enterprise
platforms.
They
need
more
than
just
container
orchestration.
So
key
native
is
a
great
fit
right.
It
needs
eliminates,
eliminates
the
need
for
stitching
a
lot
of
these
components
individually,
one
by
one
right,
traffic,
routing
and
Canady
testing
on
can
ativ
can
be
done
without
any
htu
knowledge.
A
You
get
a
higher-level
construct
to
handle
that
right.
Built-In
monitoring
tools
are
very
helpful.
Kn
ATM
eventing
shows
a
lot
of
promise.
I
would
say
you
know
it's
a
still
very
early
stage
where
a
lot
of
the
definitions
around
the
channels
and
the
subscription
needs
to
be
worked
out.
More
and
definitely
pipeline
is
something
you
know
which
needs
a
lot
of
work
around.
You
know
creating
a
robust,
DevOps
operator
oriented
pipeline
where
they
can
actually
go.
Look
at
the
logs
look
at
the
execution
times
take
certain
actions
or
from
a
pipeline.
A
You
can
fire
off
certain
events
right.
It
gets
so
that's
an
area,
you
know
which
needs
a
lot
of
work.
So,
as
I
said,
you
know
bringing
it
all
these
open-source
technologies,
that's
how
we
are
building
or
open
source
machine
learning
platform.
Out
of
this
something
like
fiddle,
we
are
actually
using
internally
heavily
with
cube
batch
in
conjunction
for
distributed
deep
learning,
training
job.
We
are
using
AI
fairness
360
in
internally
for
a
lot
of
bias.
Detection
in
our
products,
Jupiter
enterprise
gateway,
is
also
being
used
heavily.
A
A
lot
of
the
kinetic
components
are
in
early
stages
of
evaluation,
which
we
are
trying
out
a
lot
of
the
projects
and
the
github
links
which
we
talked
about
or
on
this
particular
page
right.
So
if
you
do
want
to
go
and
take
a
look
at
some
of
these
projects
around
your
family
for
deep
learning,
the
FN
is
360
Jiali,
which
you
know
he
showed
for
observing
the
traffic.
All
those
links
are
listed
here
with
that
thanks
and
thanks
for
being
here.
A
A
A
So
we
did
a
lot
of
work
on
our
power,
AI
enterprise
site
for
a
large
model,
support
which
we
have
actually
opened
up
a
pull
request
in
both
tensorflow
as
well
as
my
torch
and
the
idea
there
is
that
you
know
the
GPU
memory
is
limited
in
terms
of
you
know,
as
your
model
training
is
going
on,
how
much
waste
in
the
gradients
you
can
keep
in
GPU
it
was
16.
Now
it's
32
gig,
but
a
lot
of
the
models.
A
A
So
scheduling,
how
do
we?
The
another
question,
is
how
do
we
use
scheduling
for
resource
constraints?
So
definitely
that's
where
I
brought
up
the
project
cube
batch.
The
idea
behind
cube
batches,
essentially
scheduling
for
batch
workloads
because
most
of
the
AI
NL
workloads
they
take
a
long
time
and
they
are
very
compute
intensive.
So
that
means
you
know
no
matter
how
many
GPUs
you
put
behind
communities.
When
you
expose
it
to
your
data,
scientist
and
ml
developers,
you
will
have
more
demand
than
you
know
what
you
can
provide.
A
So
what
Q
batch
brings
in
is
the
concept
of
queuing
right,
so
you
can
queue
certain
jobs.
The
concept
of
you
know
lowering
the
priority
of
a
certain
job,
because
a
high-priority
job
has
come
in
the
concept
of
gank
scheduling
right.
So,
for
example,
if
certain
jobs
need
to
go,
if
you
launch
tensorflow,
you
need
the
tensorflow.
You
know.
Parameter
server
along
with
the
world
could
nodes
all
at
once,
or
nothing
at
all
right,
so
making
sure
that
you
know
it
can
do
gang
scheduling.
A
B
A
So
the
question
is:
why
don't
we
use
data
breaks?
Do
we
have
any
discussion
with
data
breaks,
so
data
breaks
is
a
company
which
you
know
is
initially
behind
the
small
kemmel
platform
we
contribute
heavily
to
spark
ml.
Ibm
is
one
of
the
top
five
for
commuters
of
small
kml,
so
we
do
a
lot
of
work
with
them.
In
this
space
they
have
a
certain
project
called
ml
flow,
which
is
exciting
to
us
as
we
are
looking
at
it.
It
just
happens
that
you
know
we
have
ml
flow.
A
Then
you
know,
if
you
look
at
the
open
source
landscape
chip
flow
from
Google
Drake.
A
lot
of
exciting
projects
are
there.
So
as
part
of
this,
you
know
what
we
are
doing
in
the
code.
A
group
is
essentially,
you
know,
figuring
out
exactly
that.
Write
stack
along
which
we
need
to
align
right,
because
a
lot
of
our
products
actually
have
the
same
capabilities.
A
A
A
What
we
were
talking
about
right
initially
when
we
tried
it
with
open,
less,
very
easy
to
execute.
Open
risk
has
been
out
there
for
last
three
years.
Spot
of
apache
has
heavy
contributions
from
Red
Hat
Adobe,
so
it
has
matured
so
the
concept
of
triggers
actions
and
rules,
as
we
call
it
tor
you
have
certain
triggers.
Then
you
have
rules
which
define
based
on
those
triggers.
What
action
actions
to
execute
it's
very
mature
plus
it
can
work
in
a
multi-tenant
environment
chain.
A
Atm
eventing
is
I
would
say,
you
know
maturing
it's
very
right
now
single
tenant,
oriented
plus
there
is
lot
of
issues
right
now
on
the
networking
side,
essentially
to
create
those
creative
channels
being
able
to,
you
know,
have
ingress
and
traffic
going
outside
of
those
channels
right.
So
there
is
still
a
lot
of
work
which
needs
to
happen.
The
good
thing
is,
you
know
we
are
actually
collaborating
with
Red
Hat
and
Google
NSA
PE
to
actually
bring
open
risk
into
the
kn8,
a
venting
mechanism
right.
So
that
means
you
know.
A
It's
a
very
interesting
question
right,
I
mean
and
as
probably
I
can
talk
to
you
offline,
we
have
our
own
views
around
how
it
should
be
done
right
and
if
you
talk
to
quite
a
few
folks
in
the
community,
they
also
think
you
know
there
is
not
much
needs
to
be
done
on
the
que
native
eventing
side.
So
I
can
definitely
take
that
you
know
what
we
are
thinking
would
be
the
right
architecture
and
how
it
should
be
done,
and
that's
a
lot
of
focus
for
the
first
half
of
second
year
yeah.