►
From YouTube: Advanced Model Inferencing Leveraging KNative, Istio & Kubeflow Serving - Animesh Singh & Clive Cox
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Advanced Model Inferencing Leveraging KNative, Istio and Kubeflow Serving - Animesh Singh, IBM & Clive Cox, Seldon
Model Inferencing use cases are becoming a requirement for models moving into the next phase of production deployments. More and more users are now encountering use cases around canary deployments, scale-to-zero or serverless characteristics. And then there are also advanced use cases coming around model explainability, including A/B tests, ensemble models, multi-armed bandits, etc. In this talk, the speakers are going to detail how to handle these use cases using Kubeflow Serving and the native Kubernetes stack which is Istio and Knative. Knative and Istio help with autoscaling, scale-to-zero, canary deployments to be implemented, and scenarios where traffic is optimized to the best performing models. This can be combined with KNative eventing, Istio observability stack, KFServing Transformer to handle pre/post-processing and payload logging which consequentially can enable drift and outlier detection to be deployed. We will demonstrate where currently KFServing is, and where it's heading towards.
https://sched.co/UaVw
A
Myself
is
an
amazing
I'm,
a
chief
architect
in
for
data
in
AI
open-source
platform
in
IBM,
his
Clive
Cox
CG
of
Selden
company
based
out
of
London
doing
a
lot
of
great
work
around
model
inferencing
and
model
serving
so
today
we
are
going
to
talk
about
in
general,
introduction
to
machine
learning
and
the
challenges
specifically
in
the
context
of
moral
serving
and
the
challenges
we
face
there.
And
then
you
know
once
you
have
served
your
models
or
deployed
your
models.
How
do
you
actually
monitor
it?
A
And,
finally,
you
know
wrap
it
up
is
some
summary
in
description.
Well,
this
is
a
port
which
you
probably
would
have
seen.
You
know
many
times
over
right.
It
took
me
three
weeks
to
develop
my
model
more
than
11
months
and
still
not
deployed
a
lot
of
people.
Think
it's
exaggerated,
but
this
is
you
know
it
came
out
of
one
of
our
VPS
and
actually
you
know
talking
to
customers.
That's
the
comment
we
heard
right
and
we
are
surprised
how
much
is
resonated
with
a
lot
of
p48.
A
That
essentially
that's
the
case,
and
why
is
it
the
case?
Because
the
general
perception
is
that
you
know
ml
code
is
the
biggest
part
of
your
machine
learning
infrastructure.
In
reality,
it's
a
very
tiny
part
right.
A
lot
of
efforts
are
spent
around
you
know:
data
collection,
verification
process,
engineering
model,
serving
infrastructure,
suspend
a
lot
more
effort
surrounding
that
ml
code
than
in
the
ML
code
itself.
Right
and
what
is
happening
in
industry
in
general
is
that
you
know
to
handle
this
end-to-end
ml
lifecycle.
A
Starting
from
you
know,
data
preparation,
which
includes
you,
know,
data
cleansing,
data,
ingestion,
data,
analysis
and
transformation,
validation
and
then
going
through
the
whole
model.
Lifecycle,
which
is
you
know,
building
your
models,
training
your
models,
validating
deploying
a
lot
of
technologies
are
emerging
to
actually
you
know
condense
that
right
and
make
them
faster
to
take
you
from
step
A
to
Z
much
faster.
A
A
It's
an
end-to-end
ml
platform
being
on
top
of
kubernetes
to
take
away
the
pains
right
so
that
you're
not
required
to
be
a
community's
admin
or
actually
IDs
expert
to
able
to
be
able
to
do
n2n
machine
learning
on
top
of
cuban
at
ease
it
handles
all
the
phases
of
machine
learning
lifecycle.
You
know
right
from
building
your
models,
training
your
models,
deploying
your
models
on
different,
you
know
or
model
serving
infrastructures,
and
then
you
know
trying
them
all
together
through
a
pipeline.
A
So
if
you're
into
distributed
training,
there
is
a
project
in
chi
flow
umbrella
called
catted
plus,
you
know
there
are
distributed
training
jobs
for
tensorflow,
pi
taj,
actually
boost
chainer.
If
you
like,
Horrible's
approach
of
MPI
base
training
where
weights
and
gradients
on
st.
on
a
more
peer-to-peer
basis
that
support
it
as
well
right
so
distributed.
Training
at
scale
is
supported
through
queue,
flow
q
flow
pipelines.
A
Right,
as
I
mentioned,
there
are
multiple
phases
in
end-to-end
machine
learning
where
you
actually
go
from
the
process
of
you
know,
taking
your
data
cleaning
it
and
then
going
through.
You
know,
creating
your
models,
training
it
deploying
it
managing
it.
Chip
flow
pipeline
is
essentially,
you
know,
allows
you
to
do
all
these
steps
in
a
single
consistent
way.
So
it's
a
sequence
of
ML
tasks
each
backed
by
a
container,
so
it
allows
you
to
go
from
step
A
to
Z,
but
that's
not
the
only
important
part
right
apart
from
growing
from
A
to
Z.
A
You
know
when
you
actually
want
to
trace
back
the
lineage
from
Z
to
a
that.
My
model
was
trained
on
this
particular
data
set
on
this
version
of
tensor
flow.
Using
these
hyper
parameters.
It
was
validated
using
these
libraries
deployed
and
these
are
the
scoring
characteristics
a
lot
of
their
data.
You
can
actually
find
you
know
using
cue
flow
pipelines,
so
whole
metadata,
tracking
and
lineage
is
enabled
IBM
in
Selden.
Both
are
major
contributors
in
queue.
Flow
right.
Ibm
is
the
second
largest
contributor
over
the
course
of
last
year.
A
We
are
contributing
heavily
to
queue
flow,
but
the
community
is
strong
and
very
big
lot
of
great
work
is
coming.
You
know
from
Microsoft
Seldon
Cisco,
electro
plus,
you
know
AWS.
A
lot
of
companies
have
now
started
contributing
and
this
community,
as
you
can
see
from
this
chart,
growing
heavily
and
come
January
2020.
We
are
actually
going
to
launch
Q
flow
100
right,
which
is
focused
on
four
major
co.
A
Why
is
it
you
know
typically
hard
right,
so
some
of
the
problems
are
more
genetic
in
nature
right,
so
things
like
you
know,
is
your
infrastructure
under
scale
or
over
scale
right?
Can
you
monitor
your
deployed
models?
Are
they
healthy?
Can
you
roll
out
a
new
version
of
the
model?
If
things
go
wrong,
can
you
roll
back
right?
Can
you
actually
swap
traffic?
You
know,
without
you
know,
changing
the
infrastructure
and
the
deployed
models.
How
can
you
actually
do
a
genetic
prediction
using
different
protocols,
HTTP
G
RPC
acceptor?
A
Now,
when
you
move,
you
know
more
to
the
model
specific
world
right.
General
problems
like
I
want
to
serve
tensor
flow
models.
Pi
torch
scikit-learn
actually
boost
in
a
consistent
way.
Single
spec
I.
Don't
want
to
write.
You
know
different
sort
of
clients
to
deal
with
different
kinds
of
models
right
and
once
the
models
are
deployed
right
can
I
explain
the
model.
Predictions
can
I
detect
concept.
Drift
can
I
do
bias,
detection
can
I
do
adversarial
detection.
Can
the
model
explain
its
predictions?
A
A
lot
of
these
problems,
which
are
very
specific
to
the
model
deployment
world
right
and
another
thing
which
is
very
common,
is
custom,
pre-processing
and
post-processing
code.
How
do
you
actually
handle,
because
you
know
your
models
are
essentially
dealing
with
tensors
in
tensors
out
right?
What
an
end
user
is
sending
is
probably
images
text
data
right,
so
a
lot
of
times.
In
fact
you
know
most
of
the
times
you
will
end
up.
Writing
you
know.
Custom
free
processes
and
post
processes.
A
Can
that
be
supported,
so
I
think
you
know,
different
parts
of
these
problems
were
being
done
by
different
teams
at
lost
cube
corn
in
Seattle.
We
gave
a
talk.
You
know
how
to
use
server
less
inferencing
using
K
native
bloom
horrible
is
actually
doing
great
work
on
that
side.
You
know,
Microsoft
was
doing
a
lot
of
work
in
terms
of
you
know,
as
your
M
alone,
kubernetes
Selden
was
doing
great
work
around.
A
You
know,
model
graph,
inference
saying
and
then
you
know
Google
obviously
had
been
doing
a
lot
of
work
around
tensorflow
serving
and
optimizing
heavily
around
it.
So
if
you're
all
solving,
you
know
different
parts
of
the
problem
and
what
we
did
was
over
the
course
of
last
year.
We
all
came
together
under
one
umbrella
under
to
flow
and
launch
this
project
right
and
the
goal
was
let's
work
together.
We
are
solving
similar
problems
that
solve
it
consistently
and
drive
towards
the
standard.
A
A
So
definitely
you
know
five
of
us
five
companies,
you
know
got
together
essentially
to
co-found
this
and
over
the
course
of
you
know,
lost
I
would
say
seven
to
eight
months.
This
has
become
very
popular.
If
you
are
following,
you
know,
queue
flow
in
the
last
launch
of
0.7
K
F
serving
was
the
major
release,
feature
highlighted
there,
and
our
goal
initially
was
to
focus
on
80%
of
the
use
cases
initially
right,
which
is
essentially
single
model
rollout
and
the
key
things
which
we
wanted
to
focus.
A
So
that's
how
you
know
we
ended
up
building
k
of
serving
at
the
very
bottom.
We
have
our
layers
right,
the
hardware
layer,
GPU,
CPU,
CPU
and
then
you
know,
kubernetes
sits
on
top
and
then
is
to
you
and
k
native
and
finally,
with
key
of
serving
right.
We
essentially
strive
to
bring
you
know
these
multiple
frameworks,
together,
tensorflow
pi,
touch
on
extensive
RT
and
then
give
a
common
set
of
functions
around
them
being
able
to
predict
right.
A
Get
a
request
and
get
a
response
back,
but
going
beyond
that
being
able
to
explain
model
predictions
being
able
to
inject
custom,
pre-processing
and
post-processing.
Now,
why
actually
did
we
choose
K
native
one
of
the
key
things
was.
Definitely
you
know
this
scale
to
zero
and
the
scale
from
0
to
N
was
an
important
feature.
That
was
one
of
the
core
principles
we
wanted
to
follow.
As
part
of
you
know,
model
serving.
It
was
a
personal
experience
that,
over
a
period
of
time,
these
infrastructures
become
very
huge.
A
Given
that
you
know
model
serving
requirements
are,
very
you
know,
compute
intensive
right,
so
something
that
we
wanted.
Another
thing
is
you
know
the
queue
or
request
queue
based,
auto
scaling
feature
in
K
native
that
lends
itself
nicely
when
you
are
actually
working
on
disparate
hardware
infrastructure
like
GPUs,
CPUs
GPUs,
because
traditionally
CPU
based
auto
scaling
can
also
be
enabled
in
Kenya
today.
But
when
you
have
these
different
sets
of
hardware,
you
know
it's
not
easier
to
actually
be
consistent,
so
I
think
request.
Metal
based,
auto
scaling
works
very
nicely.
A
In
this
case,
history,
oh,
definitely
was
needed.
You
know,
as
I
mentioned,
cannery
was
one
of
the
key
features
which
we
wanted
to
support.
So
being
able
to,
you
know:
do
traffic
splitting
traffic
control,
support,
Canidae
deployments
and
then
also
you
know
another
feature
of
history
which
we
really
wanted
to
enable
was
you
know,
observability
so
being
able
to?
You
know,
have
metrics
logging
tracing,
which
we
can
use
as
part
of
the
model
serving
infrastructure
and
then
enhance
and
customize
it
for
more
serving.
A
A
One
key
distinction
which
you
see
here
from
K
native,
is
essentially
the
concept
of
cannery
right,
so
we
actually
have
our
whole
configuration
for
cannery
so
that
you
can,
you
know
I
curate
multiple
times
over
cannery
until
you
want
to
make
it
a
default,
and
part
of
the
reasoning
here
is
that
you
know
a
it
lends
itself
nicely
to
get
ops
be
machine.
Learning
in
general
is
very,
very
experimental
in
nature
right
the
amount
of
iterations
you
will
have
to
do
in
machine
learning
before
you
get
to
the
you
know.
A
The
final
model
which
you
want
to
deploy
is
very
huge,
so
we
wanted
to
make
sure
that
you
know
not.
We
are
not
impacting
the
main
revision
of
the
default
deployed
model
before
we
actually
activate
multiple
times
on
the
cannery
right.
So
that's
one
key
distinction
which
we
have
here
as
I
mentioned
right.
We
support
multiple
of
these
models:
service,
chance,
flowpile,
oh
Sh,
actually,
boo,
scikit-learn,
onyx
and,
if
you're
falling
in
general,
the
onyx
ecosystem
onyx
is
a
way
to
actually
standardize.
You
know
your
multiple
models
right.
A
So
if
you
have
chain
models
using
tensorflow,
Pytor
scikit-learn,
you
can
convert
it
into
onyx
format
and
deploy
onyx
runtime
one
dot
whole
was
recently
launched.
Plus
you
know.
Just
last
week,
onyx
moved
into
linux
foundation
AI
in
total,
open
governance.
So
that's
an
exciting
news
for
us
at
IBM
and
for
the
community,
and
we
are
actually
you
know,
also
contributing
heavily
to
onyx
as
well.
We
support
you
know
multiple
different
types
of
storage.
You
know,
GC
is
persistent
volume
claims
as
your
blob,
s3,
compatible
storage,
etc.
A
As
you
know,
the
introduction
of
transformer
plus
a
you
know,
it
enables
us
to
do
some
intelligent
routing
in
terms
of
taking
the
request
directly
to
predictor
or
to
explain
it
and
then
from
there
to
predictor
right
once
you
actually
deploy
care
serving
at
your
physical
infrastructure
level.
It
may
look
like
this
right.
We
actually
use
this
to
increase.
A
For
you
know
the
external
traffic
coming
in.
You
can
also
use
your
or
load
balancer.
You
know,
if
you
want
to
configure
like
that,
there
are
certain
deployments
which
we
have
done,
which
you
can
actually
use.
You
know
external
load,
balancer
and
then
k
native.
You
know
the
key
things
which
we
are
definitely
leveraging
is
autoscaler
scale
to
zero
zero
to
one.
That's
one
of
the
key
things
explained
in
a
predictor.
If
you
see
was
deployed,
they
look,
you
know
pretty
much.
The
same.
A
One
thing
which
you
see
here,
which
is
again
a
little
bit
different
from
Cana
demon
from
structure,
is
storage,
initializer
models,
machine
learning
models.
They
are
very
heavy,
you
know.
Typically,
you
know
you
have
a
lot
of
the
weights
and
the
graphs
right
which
can
become
you
know,
gigabytes
in
size
and
if
you
are
actually
downloading
each
of
them
from
a
remote
storage
at
every
time
you
are
initializing
your
model
containers.
A
You
know,
bring
it
up
faster
and
another
key
thing
which
we
have
been
doing
is
not
only
being
able
to
deploy
models
in
a
standard
way,
but
also
being
able
to
interact
with
models
in
a
standard
way,
because
what
was
happening
is
that
you
know
if
you're
using
tensorflow.
You
are
using
a
custom
protocol
for
pre-processing
and
pull,
or
you
know
just
to
do,
request
response
same
with
you
know,
onyx
or
if
you
are
using,
you
know,
Selden
tensorflow,
tensor,
RT
server
right.
They
all
have
their
own
data
plane
protocols.
A
How
do
you
actually
send
a
request?
How
do
you
get
a
response
right?
One
of
the
key
goes
with
respect.
To
you
know.
Kiev
cell
ring
was
also
to
make
sure
that
you
know
beyond
the
standardization
of
the
model
deployment
spec.
We
also
focus
on
the
standardization
of
the
interaction
with
the
model
right.
So
once
you
have
written
a
client
to
actually
talk
to
K
of
serving
get
a
request
response
back,
get
an
explanation
back.
You
shall
be
able
to
use
that
same
client
idly
with
tensorflow
serving
with
onyx
runtime
with
Selden
right.
A
Some
you
know
sample
Yama's
in
terms
of
you
know
the
standardization
of
the
deployment
spec,
as
you
can
see
around
seven
lines
at
the
most.
You
know,
seven
to
eight
lines.
Of
course,
imple
yeah
Mille
to
actually
deploy
scikit-learn
tensorflow
potage
models
in
a
very
consistent
standard
manner.
There
is
no
difference
exposed
to
the
end-user
right
was
deploying
these
models,
and
then
you
can
add.
You
know
three
four
more
lines
into
these
specs
and
get
kennedy
and
pinned
rollouts
right.
So
in
this
case,
for
example,
you
see
that
you
know
we
are
specifying.
A
There
is
a
default.
There
is
a
cannery
and
there
is
a
ninety
ten
percent
split
between
default
and
cannery
right.
So
that's
one
of
the
key
things
which
we
have
enabled
as
part
of
this
I
do
have
a
demo
to
show
at
this
point
you
know
it's
on
my
laptop,
so
I'll
try
and
restart
my
laptop
and
flip,
but
until
then
that
happens
right
at
this
point,
I
will
talk
and
pass
on
to
Clyde
to
talk
about.
You
know
the
next
phase
right.
Okay,
you
were
able
to
deploy
your
models
using
a
standard
spec.
A
Then
you
are
able
to
interact
with
your
models
using
a
standard
right,
so
you're
able
to
get
to
that
point.
Now
that
your
models
are
giving
predictions,
can
they
be
trusted
right?
So
are
they?
You
know,
while
they're
able
to
adversarial
attacks?
Can
you
explain
your
model
predictions?
Is
their
concept
drift
right?
Is
there
an
outlier,
so
Clive
is
going
to
actually
dive
into
some
of
these
production
requirements
which
we
are
hearing
from
our
clients
with
that
clay?
Please
ball
great.
B
Thanks
ash,
so
yeah
I'm
gonna,
take
you
into
those
he's
told
to
me
about
how
to
get
the
core
model
into
deployment
and
how
to
do
a
roll
out
of
that.
But
now
we
want
to
take
it
to
the
next
level.
Whatever
things
you
need
to
add
to
your
core
model
to
really
get
a
robust
production
ml
system,
one
of
those
is
model
explanations
and
we'll
go
why
you
might
need
that
and
how
you
do
it,
then
you
might
want
to
also
record
your
payloads
as
they're
going
through.
B
If
you
can
record
the
payloads
the
requests
and
responses,
then
you
can
do
things
with
them
in
their
time.
After
the
models
made,
the
prediction
you
can
attach
them
to
some
technology
like
a
native
and
then
push
those
payloads
out
to
things
that
will
give
you
understanding,
what's
happening
to
your
model
and
giving
you
a
metrics
as
to
whether
it's
what
is
working
well
in
productions.
Things
like
outlier
detection,
we'll
go
into
that
at
the
soil
detection
and
concept
Jeff.
So
you
can
get
a
really
good.
Understanding.
B
Is
my
model
working
as
expected
in
production,
so
go
through
those
different
stages?
So
why
would
you
want
to
explain
your
models?
Well,
first,
there's
lots
of
regulation
in
Europe
GDP
are
so
people
who
are
submitted
to
automated
automated
decision
processes
need
to
be
able
to
be
given
some
explanations
of
why
they
were
given
the
particular
predictions
that
they
did.
So
that's
obviously
key
for
any
models
you're
putting
in
production
in
those
countries
and
there's
also
regulation
in
the
u.s.
in
that
area.
Secondly,
for
the
people
such
as
data
scientists
DevOps,
they
need
to
understand.
B
Why
is
my
model
doing
what
is
expected
to
be
doing
and
if
you
want
it,
actually
I'm
investigate
the
actual
behavior
of
the
model
on
particular
inputs
which
are
unexpected,
such
as
outliers?
So
all
these
reasons
why
you
might
want
to
have
explanations
to
give
you
a
better
understanding
of
what
your
model
is
doing,
four
bits
for
particular
input.
B
So
what
is
a
goal
of
a
good
explanation?
Well,
one
is
for
it
to
be
human
interpret.
Also,
you
might
have
a
deep
low
on
your
network
with
many
millions
of
weights.
That's
not
going
to
be
very
interpretable
word
to
understand.
You
even
have
some
have
to
have
to
have
a
high
level
of
representation
of
what
is
happening
in
your
model,
but
the
same
time
you
want
it
to
be
not
to
over
simplified
novice.
B
You
can
have
a
very
high
level
explanation,
but
if
it's
too
high
level
you're
not
getting
a
good
idea,
what
the
actual
model
is
doing,
this
is
going
to
be
a
trade
off
always
with
explanations
between
something,
that's
very
intelligible
and
or
something
that's
very
close
to
what
the
model
is
actually
doing.
So,
but
that's
just
one
of
the
facts
of
life
of
explanations.
You
need
to
take
into
account.
So
how
do
you
do
it
once
the
core
thing
we're
going
to
discuss?
B
There
is
black
box
model
explanations,
so
here
you're
treating
the
model
purely
as
a
black
box.
That
could
be
your
new
or
net
word.
It
could
be
a
random
forest.
It
could
be
anything
this
input,
so
here
you've
got
a
set
of
features
and
democratic
features,
etc,
and
the
model
makes
a
prediction
in
this
case
to
actually
say
denial,
own
request
with
a
high
probability,
and
you
want
to
understand
why.
Why
did
that
white?
The
model
make
that
particular
prediction
says
various
ways
to
do
it
and
the
cool
way
to
explain
is:
do
it
essentially?
B
B
Its
prediction,
as
the
as
the
features
that
you're
actually
sending
to
the
model
are
change
very
slightly,
and
that
way
you
can
try
to
judge
how
the
model
is
coming
to
assert
your
opinion
about
it,
social
prediction,
and
then
you
could
turn
all
those
particular
predictions
you
may,
which
could
be
tens.
Tens
of
thousands
to
then
turn
it
into
human,
understandable
explanation.
So
obviously,
as
you'd
see
there
it's
going
to
be
expensive,
so
you
might
have
one
prediction
coming
and
it's
going
to
be
hundreds
or
tens
of
thousands
of
predictions.
B
You
need
to
send
it
the
model.
So
we
are
certain,
have
a
open
source
library
alibi,
explain
which
you
can
see
the
URL
there,
which
is
stated
by
state-of-the-art
implementations
for
different
types
of
explanation,
algorithms,
such
as
anchored
counterfactuals,
contrast
of
explanations.
Now
the
things
connect
to
explanation,
such
as
trust
scores,
so
that
gives
you
can
you
really
put
trust
the
predictions
come
back
from
your
model
if
they
say
how
high
probability
that
they?
Actually
this
is
true?
B
Is
it
actually
the
case
that
you
can
actually
trust
that
prediction
so
in
today,
we'll
just
concentrate
on
one
particular
type
of
estimation
called
anchors.
So
here
you've
got
a
representation
of
a
image
model.
It's
an
image
net.
It's
got
a
picture
of
a
cat,
goes
to
the
moral
and
okay,
that's
a
Persian
cat
with
90%
probability.
So
four
anchors
it's
going
to
find
the
core
features
in
this
case
pixels
or
super
pixels
part
of
the
image
that,
irrespective
of
what's
around
that
it's
always
going
to
give
the
same
prediction.
B
So
here
you
run
it
through
the
explainer.
It's
obviously
picked
out
the
eyes
and
the
whiskers
of
the
cat,
and
irrespective
of
what
happens
in
the
image
around
there,
you
know
this
model
will
always
predict
that
this
is
a
Persian
cat.
So
this
gives
you
a
core
idea
of
what
is
the
model
actually
focusing
on
to
actually
make
that
prediction,
and
so
you
get
some
insight
into
how
it's
doing
it.
So
this
is
a
version.
B
Anchors
has
versions
for
images,
tabular
data
and
textual
data
and
we'll
take
you
through
a
tabular
data
example
in
a
second.
So
how
do
you
add
it
to
care
serving?
This
is
just
a
simple
addition
to
issue
of
your
yeah.
More
that
you
do.
You
either
explain
the
section
next
to
your
predictor,
you
specify
what
type
of
algorithm
you
wants,
in
this
case
anchor
tabular
as
it's
tabular
data
for
this
income
model,
I'm
going
to
show
you
in
a
second
and
some
explain.
B
This
requires
some
analysis
of
the
training
data
to
get
some
understanding
of
what
is
the
likelihood
of
different
features,
so
there's
some
explainin
need
to
be
trained,
but
it's
not
training
on
the
model.
That's
just
training
on
the
training
data
to
say:
okay,
this
feature
appears
system
out
of
x
with
these
values.
So
therefore
you
can
give
you
a
response
back
this
easy
easy
to
understand.
So
that's
what
that's
an
example
on
the
left
for
Anka
tabular,
one
for
anchor
text
on
the
right.
So
just
simply
just
saying
this
is
anchor
text.
B
I
won't
explain
it
for
that
in
this
case,
for
a
movie
sentiment
model.
So
it's
very
easy
to
add.
I'll
go
for
a
quick
demo.
There's
two
demos
in
the
slides:
I,
probably
just
go
through
the
income
prediction
demo.
So
this
is
the
structure
that
we've
added
to
now.
We've
got
the
core
model
and
we've
added
we're
going
to
add
it.
B
Income
explain
that
next
to
it
using
alibi,
explained
or
all
deployed
using
cash
serving
so
we'll
just
go
to
the
actual
model
here,
so
we'll
have
a
look
at
the
actual
llaman
for
the
actual
model.
What
we're
serving
so
we've
got
the
actual
model
at
the
top
there,
a
sky
kittler,
an
income
prediction
model.
B
It's
based
on
this
sort
of
census,
data
from
1996
I've
added
a
few
things
just
to
make
it
not
use
too
much
resources
and
we
add
in
the
anchor
tabular
so
apply
that
what
I've
done
that
already
so
I
just
go
through
some
setup
to
actually
get
the
IP
address
of
the
service
and
the
service
names.
If
you
setup
here
set
up
just
some
of
the
data
so
that
we
can
send
it
to
the
model
and
get
a
explanation.
B
Just
wait
a
second
for
that,
sir,
that's
your
run
could
be
the
internet.
It's
gonna
fail
us
here
at
the
last
second
for
she'd
ever
know
because
I've
got
or
where
do
you
run?
Okay?
This
is
going
a
bit
slowly,
so
obviously
we
can
send
two
examples
to
the
model.
So
here
we've
got
some
examples.
This
is
the
demographic
features
say
age,
marital
status,
except
your
occupation.
Over
here,
it's
predicting
the
person
will
have
lower
than
50k
salary.
B
This
is
from
1996
now
here
we've
got
a
different
set
of
features,
whereas
printing
hire
hire
more
than
50k
salary
for
those
features,
and
we
can
ask
okay.
What
is
an
explanation
for
why
this
first
example
the
one
where
you
had
low
income
predictions?
We
send
that
to
the
explainer?
That's
gonna
have
called
the
model
thousands
of
times,
and
it's
going
to
say:
okay,
so
hits.
What
it's
saying
is
that
of
those
demographic
features.
Here
we
had
at
the
top,
irrespective
of
what
you
check
for
the
other
features.
B
If
it's
person's
mental
status
is
never
married
and
they're
working
less
than
forty
hours
per
week,
you'll
always
get
lower
low
income
prediction
for
that.
So
therefore
you
can
look
in
that.
Give
me
a
lot
of
insight.
Maybe
you
want
to
look
into
that.
Is
your
model
biased
in
some
way?
Is
it
taking
marital
status,
which
it
shouldn't
be
doing?
Maybe
you
want
to
take
that
out
of
your
dataset,
so
you
can
all
these
things,
which
are
obviously
very
important
now
to
understand
how
your
model
is
behaving.
B
You
could
then
get
some
understanding
of
your
confidence
in
that
actual
prediction,
so
you
can
ask
it
for
the
actual
precision,
in
this
case
it's
96
percent
position.
That
means
that
96
percent
of
the
time,
if
those
features
just
appear
the
amount
of
status
and
hours
per
week
you'll,
although
your
model
will
always
give
the
same
prediction
of
low
salary,
and
then
coverage
says
okay,
how
much
of
the
data
set?
Does
this
actually
cover
those
features?
B
So
it's
actually
reasonable
size,
the
data
at
25
percent
or
the
dataset,
and
you
can
dive
into
how
that
goes
down
feature
by
feature.
So
if
we
had
just
them
the
amount
of
status
that
will
cover
32
percent
and
then
you
add
Mausers
an
hours
per
week
that
goes
down,
so
you
give
a
idea
of
how
much
of
a
dataset
is
covered
by
just
these
type
of
features.
In
for
what
you
wanted
to
do,
you
can
do
a
similar
thing
for
high
high
income
example.
B
I'll
skip
over
that
now,
just
in
terms
of
time,
so
it's
very
easy
to
set
up
and
that's
how
you
add:
explanations,
we're
using
alibi,
explained,
office
or
slab.
It
is
also
a
great
light
before
IBM
called
I
explained
ability
360,
which
you
can
also
be
used.
That
would
be
integrated
into
cash
diving
in
the
near
future.
B
So
next
thing,
because
we've
got
explanations
for
our
models,
next
thing
is
payload
logging.
So
why
would
you
want
payload
logging?
Well,
for
obvious
reasons
you
wanted
it?
You
want
to
capture,
what's
going
for
your
model
for
future
retraining
of
your
model
and
you
want
to
perform
near
timer
analysis
all
of
that
model,
one
that
will
request
some
responses.
We've
added
in
2k
of
serving
an
alpha
feature
which
allows
you
to
add
to
any
part
of
the
care
of
serving
description
to
the
predictor.
Explain
all
the
transformer
to
actually
log
request
going
through
the
model.
B
At
that
point,
you
can
choose
to
choose
to
log
both
the
request,
all
the
responses
or
both
and
all
you
all
you
need
to
do
is
specify
a
URL
of
where
to
send
those
payloads.
So
that
means
it's
really
easy
to
connect
to
anything.
You
want
to
do
to
connect
it
into
systems,
and
we
using
cloud
events
for
this
so
cloud
of
instances
standard
for
wrapping,
your
of
event,
type
data
subsystems
which
one
to
work
together,
event
Inc
and
work
together
seamlessly,
but
with
a
standard
set
of
headers.
B
B
This
case
I'm
using
the
K
native
standard
service,
I'm
going
to
set
up
message,
dumper
just
to
print
out,
but
you
could
set
up
to
push
telegram
so
alerting
system
or
whatever
you
want
to
assume
it's
going
to
save
the
data,
and
then
you
specify
a
mode.
Do
you
want
a
low
of
the
requests
or
the
responses,
or
both
this
case
all
so
again,
because
it
all
you
need
is
a
URL.
You
can
tie
it
into
anything.
B
You
could
tie
it
into
K
native
broker
and
tie
that
into
further
systems
like
our
Liza
texture
and
then
push
it
on
to
it.
Alerting
or
you
could
tell
you
into
sake-
Africa
have
a
Kafka
bridge
and
push
it
onto
a
caracal
cluster.
So
it's
up
to
you
and
it's
very
flexible
on
how
you
actually
use
it.
So
now
we've
got
the
payload
logging,
which
is
easy
to
add
in
then
we
can
do
some
interesting
things
and
really
understand
its
model.
B
Working
correctly
in
production,
so
three
things
have
been
a
chat
about
is
out
lies
detection
at
the
server
detection
in
consecutive.
Nobody
come
under
the
concept
of
you
product
shouldn't
trust
predictions
on
instances
which
are
outside
your
training
data.
So,
as
your
model
may
be
trained
a
month
ago
on
data
and
the
data
coming
into,
it
may
be
completely
different
and
you
shouldn't
trust
those
predictions
and
these
type
of
techniques
are
trying
to
detect
when
those
things
are
happening.
So
outlier
detection
is
obviously
when
you
the
data,
it's
not
as
expected
for
the
actual
model.
B
There's
many
different
types
of
that
of
a
lie
detectors
available.
We
won't
go
into
all
the
details,
but
there's
ones
that
are
trained
online,
as
the
data
is
coming
through
the
system,
ones
that
are
trained
on
your
training
data
and
we'll
see
different
types
of
a
lie:
detection
for
the
types
of
data
of
your
model
and
different
types
of
outliers.
B
It
could
be
a
global
outlier,
say
something
if
you've
got
a
model
that
predicts
temperature
and
say
it's
predicting
1,000
degrees,
obviousiy,
that's
a
global
outlier,
but
it's
predicting
say
35
degrees
in
London
in
say,
December,
that's,
probably
a
contextual
out
local.
It's
really
unlikely
at
that
time,
and
then
you
could
have
collective
outliers
which
apply
to
things
like
time
and
time,
series
where
whole
series
of
events
are
together
taken
as
an
outlier.
So
another
thing
you
want
to
do
is
maybe
adversarial
detection.
B
This
is
the
case
where
people
could
be
attacking
your
model
with
very
similar
instances
which
are
producing
radically
different
outputs
from
your
model.
So
here's
some
examples
on
the
left.
You've
got
films,
some
a
chemist,
handwritten
digits,
but
even
though,
for
us
it
looks
like
that's
0
&
6,
the
model
is
going
to
predict
something
completely
different
in
this
case,
2
to
the
0
and
0
for
the
6,
just
by
changing
a
few
pixels
and
similar
funny
example.
B
Here,
where
you're
an
image,
classify
of
a
pig
adds
a
certain
amount
of
very
specified
noise
to
that
still
looks
like
at
the
same
image
to
us,
but
your
models
predicting
something
completely
different
and
you
may
want
to
be
told
if
this
is
happening
to
your
model.
Okay,
so
I'm
just
going
to
go
very
quickly
through
a
couple
of
things.
That
concept
drift
or
says
is
more
similar
to
outlier
detection.
B
But
it's
over
a
whole
series
is
my
training
data
changing
over
time
and
then
we've
got
a
new
library
called
alibi,
detect
which
you
just
come
out
this
week,
which
has
state
of
their
implementations
of
outlines,
detection,
adverse,
elder
detection
and
concept
drift,
and
you
can
find
that
online
and
be
great
to
get
your
feedback
on
that.
So
I'll
just
go
for
a
very
quick
example
on
the
ciphered.
Cyber
10
model
for
out
lie
detection,
so
this
is
a
simple
image
classification
model,
just
the
10
different
types
of
images.
B
Basically,
we've
got
it
running
here
so,
as
we
got
really
close
to
time,
I'll
just
go
down
to
the
actual
example,
so
you've
got
a
prediction
here
of
a
truck
when
the
model
predicts
correctly
it's
a
truck,
and
then
you
can
basically,
as
we
got
the
outlier
detection
tied
in,
that
will
then
go
through
and
check
what
this
is
outlier.
It's
quickly
saying
that's
a
not
it's!
Gonna
just
come
out
to
the
message,
dumper
saying
no,
this
is
not
an
outlier.
We
can
then
create
a
outlet
prediction
by
adding
some
noise
to
the
model.
B
In
this
case,
a
little
bit
of
noise
to
the
left
hand,
corner
the
actual
model,
still
says
it's
a
truck.
You
want
to
know.
This
is
not
actually
part
of
the
training
data
and
should
be
highlighted
again
with
the
outlines.
Detection.
They're
running
you'll
go
through
a
few
seconds
later
and
we'll
get
a
prediction
this
case.
The
second
example
here
in
the
logs.
This
is
an
outlier,
so
you
can
quickly
find
out.
This
is
happening.
This
could
then
go
on
to
the
alerting
systems.
B
B
Adding
these
things
together,
there's
a
list
of
the
libraries
that
were
all
open
source
that
you
can
use
to
achieve
this
there's
various
other
talks,
including
the
one
yesterday
on
care
server.
You
want
to
catch
that
up
online,
giving
an
introduction
and
there's
a
group
in
LFA
I
called
trusted,
AR
community,
which
is
working
on
this.
How
do
you
get
this
trusted
and
our
models
in
production
and
be
great
to
get
your
cooperation
in
that
thanks.
A
Clay,
if
you
guys
have
five
minutes,
I'll,
try
and
see.
You
know
if,
if
the
HDMI
detection
works
and
then
just
do
a
very
quick
demo,
if
it
does
work
or
I,
can
probably
you
know
it
still
nor
detection,
it
was
just
working
20
minutes
ago.
Can
we
just
use
certain
links
from
your
slack?
Maybe
just
show
very
quickly
the
pipeline's
part.
B
A
A
Private
children,
okay,
so
just
in
terms
of
the
scenario
right,
what
we
are
trying
to
do
here
as
affiliate
loads,
so
this
is,
you
know
typical
model.
Is
the
psychic
learn
model
in
this
case?
You
know
this
model.
Essentially
what
it
does
is
you
know
it.
Simulates
shipping,
industry,
shipping
industry,
carries
your
food
items
right,
as
these
food
items
are
being
carried
right
through
air
place
through
sea
right,
they
might
get
spoiled
right
so
because
you
know
they
have
different
temperatures,
carbon
dioxide,
nitrogen,
co2
level
right.
A
So
this
model,
actually
you
know-
is
a
psychic
learn
model
which
is
trained
on
that
particular
data
set.
These
shipping
containers
which
are
coming
in
and
then
there
is
you
know,
pipeline
right,
which
is
in
this
case,
is
a
cue
flow
pipeline,
which
essentially
will
train
that
model
right,
deploy
that
model
using
K
of
serving
launch
or
default
and
a
Canada
version
of
it.
And
then
you
know,
I'll,
probably
you
know,
send
some
load
from
this
side
and
then
be
able
to
show
you.
A
A
A
So
it
just
launches
this
pipeline,
it
will
appear
and
what
you
will
see
is
you
know
this
one
single
pipeline.
You
know
it
will
essentially
go
through
all
the
different
phases
which
we
were
talking
about
into
the
chip
flow
ecosystem
as
it
comes
up
right.
So
essentially,
you
know,
training
the
model
or
deploying
the
model
and
then
launching
you
know
default
in
a
Canada
version
of
that
model.
Right.