►
From YouTube: Owned by Statistics: Using Kubeflow to Defend vs Attacks on Your ML Models - David Aronchick
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Owned by Statistics: Using Kubeflow to Defend vs Attacks on Your ML Models - David Aronchick, Microsoft
Machine learning continues its spread across the tech world and is now in use by more than 80% of enterprises world wide. However, with the increased reliance on this technology, the spectre of additional security attack surface areas rises up. Machine learning attacks are a new area of opportunity for adversaries, and require a new way to approach defense. In this talk, we will cover several of the most common ML attacks today and how to defend against them. We will also show how to use a sophisticated, cloud-native pipeline with Kubeflow will to enable organizations to detect, remediate and defend against future attacks.
https://sched.co/ekBz
A
One
thing
I
want
to
say
is
how
much
I
love
kubecon
every
year
and
it's
such
an
honor
to
be
able
to
present,
even
though
it's
virtual-
and
I
miss
you
all
terribly
so
with
that.
You
know.
One
thing
that
we
talk
a
lot
about
in
machine
learning
is
how
powerful
everything
is
and
how
many
advancements
have
occurred
in
the
recent
past.
A
Specifically,
you
can
see
some
of
those
examples
here,
where
you
have
machine
learning,
models
from
microsoft,
really
setting
the
bar
to
new
and
interesting
advancements
and
and
really
handing
it
back
to
the
community.
You
know,
machine
learning
is
all
about
open
source,
and
it's
not
just
that.
We
built
on
these
open
source
projects
externally,
but
then
gave
back
after
we
were
done
now
with
that.
You
know
you
wonder
how
this
affects
microsoft
and
the
reality
is.
It
affects
us
in
just
about
every
way
possible.
It
touches
our
consumer,
our
enterprise,
our
game,
business.
A
You
know
our
search
business,
they
all
simply
could
not
work
without
the
power
that
machine
learning
gives
us
to
analyze.
Enormous
data
sets
and
when
I
say
enormous
data
sets,
I
mean
stuff,
like
you
see
here,
whether
or
not
it's
people
using
office,
people
asking
questions
of
cortana
or
the
6.5
trillion
signals
that
we
evaluate
on
a
daily
basis
around
security.
A
We
could
not
handle
this
without
the
things
that
machine
learning
provides
with
that
said,
oftentimes
in
the
machine
learning
space,
we
tell
you
these
great
stories
and
then
kind
of
like
leave
it
up
to
you
to
as
an
exercise
to
the
reader
to
go
and
implement
it,
and
the
reality
is
machine.
Learning
is
hard
and
it's
something
that
we
definitely
don't
pay
enough
attention
to,
and
one
of
the
reasons
it
is
so
hard
is
because
of
what
you
see
here.
A
Most
of
the
time
people
focus
on
just
building
a
model,
but
the
reality
is
is
that
building
a
model
is
only
one
step
of
a
very
complicated
workflow
and
thinking
about
the
pipelines
is
how
people
will
help
get
people
to
adopt
things.
More
now,
let's
say
you're
a
data
scientist
and
you
say:
well,
I
don't
really
care
about
that.
I'm
good!
The
answer
is
yes,
you
do,
and
the
reason
is
is
because
of
what
you
see
here:
people
build
these
fantastic
models
and
then
take
forever
to
get
them
out
to
production.
A
One
way
through
that
is
with
ml
ups
ml
ops
is
machine,
learning
operations,
derivative
from
devops
and
git
ops.
It
brings
data
scientists
and
machine
learning
engineers
into
the
development
process.
So
now
they
can
execute
this
inner
loop,
like
you
see
here
on
the
left,
iterating
very,
very
quickly
and
then,
when
the
time
comes,
they're
able
to
expand
and
bring
be
brought
into
the
normal
application
developer
cycle,
where
in
this
case,
they're
able
to
now
integrate
their
sophisticated
models
into
the
development
and
rollout
procedures.
So.
A
Endpoint
now,
in
this
case,
we're
going
to
start
by
collecting
a
whole
bunch
of
examples
of
circles
and
feed
that
into
our
training
data
and
then
out
the
other
side.
When
I
present
a
sample
model
here,
excuse
me
a
sample
object
and
it's
going
to
come
back
and
say
that
is.
It
is
in
fact
a
circle.
So
look
so
far,
so
good.
Looking
great
now,
if
I
present
a
square
to
it,
it
still
says
it's
a
circle
and
I
have
to
wonder
why
what's
going
on
here.
A
Well,
if
I
look
at
these
two
things,
one
is
green.
One
is
blue.
One
is
round,
one
is
square.
How
could
these
two
be
confused
and
the
reality
is
is
because
I
didn't
give
my
exit
model
enough
examples.
In
this
case
I
gave
it
something
which
has
all
the
same
pixels
as
a
circle,
but
in
fact
I
didn't
tell
it
what
to
look
for,
and
so
the
model
says
well,
it
has
the
same
pixels
as
a
circle.
So
it's
a
circle,
but
obviously
that
is
incorrect.
A
Now,
surely
advanced
models
are
better.
We
don't
see
this
in
the
real
world,
but
the
reality
is
that's
wrong
too.
Here's
an
example
of
a
husky
versus
wolf
detector
and
you
know
it
looks
great.
We
were
able
to
do
it
with
only
one
mistake
out
of
the
entire
group
and
you're
looking
pretty
good,
but
when
you
turn
it
over
and
you
look
into
the
explainer.
A
What
you
see
is
that
a
lot
of
the
data
under
the
hood
looks
like
in
the
husky
case.
It
looks
like
it's
picking
up
pictures
of
the
animal,
but
in
the
wolf
case
it
looks
like
it's
just
picking
up
stuff
around
the
edge
and
kind
of
we're
like
well
what's
going
on
here,
as
we
dive
just
a
little
bit
deeper
into
it,
you
see
that
in
fact
lots
of
the
wolf
was
not
included
and
what
we
really
did
was
train
a
snow
detector.
A
Now
you
say
well
this
again,
you
know.
Maybe
this
is
just
for
paper
purposes.
No,
in
fact,
you
see
this
over
and
over
again,
where
in
in
one
case
you
have
a
ship
that
is
actually
looking
for
water.
In
another
case,
you
have
trains
that
are
actually
looking
for
trained
tracks
and
then,
in
the
third
case
and
one
of
my
favorites,
you
see
a
horse
detector
who
is
really
looking
for
copyright
notifications.
Apparently,
a
lot
of
horse
images
have
copyright
notifications
and
that's
just
from
accidental
stuff.
A
A
It
gets
even
worse
when
you
start
thinking
about
military
implications.
In
this
case,
we
have
a
model
that
which
is
looking
for
airplanes
on
a
runway
which
is
easily
able
to
detect
until
you
put
a
sticker
on
the
back
of
the
airplane,
and
it
is
able
to
camouflage
those
airplanes
as
though
they
didn't
exist.
And,
finally,
you
know
thinking
about
what
happens
when
you
put
a
layer
of
glasses
over
these
individuals,
you're
able
to
convince
the
the
model
that
these
are
in
fact
completely
different
people,
and
you
think
that
these
might
just
be
toy
examples.
A
Until
you
realize
that
amazon's
face
recognition,
system,
mismatched,
28
congress,
people
with
their
mug
shots,
indicating
they
might
be
criminals,
and
it's
not
just
that.
It's
also
extremely
hard
to
opt
out.
So
you
can't
even
get
out
of
these
systems.
Even
if
you
wanted
to
so,
are
you
terrified?
Yet?
A
A
You
also
make
sure
that
you
have
a
red
team
in
place
and
attack
your
own
models
first,
along
with
a
rich,
alerting
and
monitoring,
but,
most
importantly,
it's
feeding
the
information
you
get
back
from
your
serving
into
your
model
so
that
you
can
quickly
become
smarter
over
time
and
continually
iterate
and
improve.
This
is
really
only
possible
with
a
rich
ml
ops
pipeline.
A
A
In
this
case,
we
will
be
talking
about
using
something
like
ci
cd,
github
actions
and
jenkins,
loosely
modular
components
that
you
can
use
from
on
cloud
or
on-prem
and
making
sure
that
you're
constantly
measuring
and
updating,
and
the
only
way
to
do
that
is
with
something
declarative
that
you
can
run
over
and
over
again
without
human
intervention
to
drill
in
just
a
little
bit
deeper.
Let's
say
you
start
with
jupiter
in
the
upper
left
hand
corner
there,
where
you're
automatically
iterating
once
you're
done.
A
You
check
that
into
github
and
then
at
that
point,
you're
going
to
connect
it
to
your
pipeline.
In
this
case,
we're
going
to
use
kubeflow
pipelines
as
our
ci
cd
selection
of
choice
and
when
github
receives
that
it's
going
to
begin
that
process,
and
so
first,
it's
going
to
use
kale,
which
is
a
open
source
tool
that
takes
a
jupiter
notebook
and
wires
it
up
with
your
pipeline.
A
And
you
may
kick
off
something
like
your
feature,
engineering
step,
which
may
call
out
to
actually
an
external
pipeline,
such
as
spark
to
run
through
all
your
large
data
processing.
Once
that's
done,
it's
going
to
hand
back
to
kubeflow
pipelines
and
continue
with
the
next
step.
In
this
case,
it'll
kick
off
your
training
step
and
in
this
case
we're
going
to
use
cube
flow
as
a
pipeline
itself,
and
so
it's
going
to
call
out
keep
the
pipelines
will
to
that
training,
step,
execute
and
then
hand
back
when
that's
done.
A
It'll
then
go
to
hyperparameter
sweep
where
again
you're
going
to
reach
out
to
a
q
flow
pipeline,
and
in
this
case
it's
going
to
execute
many
hundreds
of
pipelines
simultaneously
to
search
across
the
entire
grid
of
hyperparameters
and
find
the
one
that
is
best
suited
for
your
model.
Once
that's
done,
it's
also
going
to
hand
back
we're
going
to
package
that
and
hand
it
off
to
serving,
and
then,
from
that
point
from
serving
it's
going
to
be
out
there
for
inference
purposes
and
be
able
to
be
used
by
your
application.
A
B
Now,
let's
make
this
concrete
and
show
what
an
emerald
powered
workflow
actually
looks
like
from
the
perspective
of
a
data
scientist
in
this
demo,
we'll
show
the
steps
a
data
scientist
takes
from
experimenting
locally
in
zoopter
notebook
to
running
a
single
pipeline,
generated
from
that
notebook,
running
hundreds
of
pipelines
for
hyper
parameter
tuning
and
then
choosing
the
best
model
to
serve
in
this
process.
Erector
snapshot,
store
and
ml
metadata
ensure
fully
reproducibility
and
limits
tracking.
So,
let's
dive
right
in.
B
So
let's
pretend
that
we
are
a
data
scientist
and
we
want
to
start
working
on
an
exciting
new
problem.
We
open
the
kepler
dashboard,
go
to
the
notebooks
tab
and
start
a
new
notebook
server,
connect
to
it
and
get
right
to
work.
The
problem
we
are
working
on
today
is
the
open
vaccine
cargo
challenge
and
we're
trying
to
locate
the
weak
spots
of
a
messenger
rna
structure
to
help
create
a
stable
vaccine.
B
B
However,
kl
makes
the
transition
from
notebook
to
pipeline
very
simple,
and
let
me
show
you
how,
in
the
side
of
the
notebook
I
open,
the
kill,
sidebar
and
enable,
as
you
can
see,
certain
colors
appeared
around
in
cell
cells
with
the
same
color
are
part
of
the
same
pipeline
step.
This
information
is
stored
in
the
notebook's
metadata,
which
is
pre-filled
in
this
case.
So
we
just
need
to
annotate
our
notebook
cells
using
kel's,
intuitive
ui,
for
example.
B
It
provides
us
with
a
link
to
the
kubeflow
dashboard,
which
we
can
follow
to
see
the
progress
of
the
pipeline
run
and
every
step
of
this
pipeline
that
you
see
is
snapshotting
by
rock,
and
these
steps,
input
and
output
artifacts
are
tracked
by
ml
metadata.
So
for
every
step
you
can
see
logs
and
also
emblemata
metadata,
which
records
the
input
and
output
artifacts
of
each
step.
In
this
case,
they
are
a
rock
snapshot
which
we
can
actually
follow
and
see
the
snapshot
in
the
rob
ui.
B
Now
we
have
a
reproducible
envelope's
pipeline
generated
automatically
from
our
notebook.
However,
this
pipeline
only
trains
our
model
for
a
specific
set
of
parameters
like
learning,
great
or
bad
size,
tweaking
those
parameters
can
result
in
a
vastly
improved
model.
So
we
want
to
explore
more
configurations.
B
This
procedure
is
called
hyperparameter
tuning
and
kell
provides
an
intuitive
ui
for
enabling
it
to
perform
hyper
parameter
tuning.
We
need
three
things.
First,
we
need
to
define
what
the
hyper
parameters
are.
In
this
case,
our
copper
parameters
are
epochs,
bad
size,
etc,
and
k
provides
us
with
a
special
cell
called
pipeline
parameters
to
declare
a
hyper
parameter.
Our
parameters
for
hyperparameter
tuning.
B
Then
we
need
some
metric
to
guide
our
search.
The
metric
shows
if
our
model
is
actually
performing
better
or
worse,
with
its
new
parameter
configuration.
So
in
this
case
we
use
the
validation
nodes
as
the
metric
guiding
the
hyperparameter
tuning
to
declare
our
metric.
We
simply
print
it
and
declare
the
cell
as
a
pipeline
metric
in
case,
and,
finally,
we
enable,
let's
be
tuning
in
gain
and
specify
our
cells,
algorithm
and
hyperparameter
settings.
So,
for
example,
the
batch
size
can
be
between
32
and
256
in
increments
of
32..
B
In
this
case,
the
settings
are
pre-filled,
so
we
can
start
the
tuning
right
away.
Scale,
as
you
can
see,
provides
a
pretty
intuitive
form
to
do
this
configuration
in
a
ui
driven
way
after
defining
the
parameters
the
metric
and
the
cells
algorithm.
We
are
ready
to
start
our
hyperparameter
tuning
by
pressing
the
button
of
the
kell
sidebar
to
perform
the
hyper
parameter
tuning
kell
uses
cardiff,
a
tube
flow
component
health
will
start
a
cutting
experiment
which
will
create
a
trial
for
each
different
set
of
hyperparameter
values.
B
Now
because
the
hyperparameter
tuning
will
take
a
while,
we
have
prepared
the
results
of
this
tuning
beforehand,
so
we
can
see
the
category
experiment
with
all
the
created
variables
so
far
in
the
graph,
showing
all
the
hyperparameter
configuration
in
their
success
metric
in
an
intuitive
way.
We
can
also
find
the
best
trial
so
far
as
the
ui
will
highlight
it
for
us
and
from
a
catholic
trial
we
can
navigate
to
the
kfp
pipeline
run.
It
corresponds
to
by
clicking
the
small
pipeline
button
next
to
the
trial.
B
So
once
we
click
this
button
and
go
to
the
pipeline
run
notice,
how
many
steps
have
to
click
recycle
icon
on
them?
This
icon
means
that
these
steps
were
cast
to
massively
speed
up
the
execution
time
of
a
pipeline
run.
Caching
is
part
by
rectus
rock
by
taking
snapshots
at
the
end
and
start
of
its
step.
We
can
also
investigate.
We
can
also
navigate
from
a
pipeline
run
to
the
cutting
experiment
it
belongs
to
so
we
started
from
a
notebook
and
created
the
pipeline.
B
Then
we
run
many
pipelines
in
order
to
perform
hyper
parameter
tuning
and
get
the
best
model
after
running
the
hyper
parameter
tuning.
We
have
found
the
best
combination
of
hyper
parameters
for
our
model,
and
now
we
need
to
restore
and
serve
that
model.
We
can
find
the
best
combination.
By
going
to
the
category
it
will
be
highlighted
for
us
from
the
trial
we
can
easily
navigate
to
the
corresponding
pipeline
run,
as
we
saw
earlier
by
clicking
the
little
pipeline
button.
B
Now
we
are
going
to
use
the
snapshotting
power
of
rogue
restore
the
first
version
of
our
model
in
the
notebook
to
do
that,
we
will
go
to
the
last
step
of
our
best
pipeline,
run
and
locate
the
rocksnap.url,
which
is
found
under
the
visualization
tab.
We
copy
the
rock
url
and
use
it
to
restore
a
new
notebook
to
the
state
of
the
pipeline,
run
at
that
specific
step.
B
B
B
B
B
You
can
also
see
logs
from
the
model
servers
pods
like
the
predictor
of
the
transformer
which
performs
the
processing,
and
you
can
also
see
the
configuration
of
the
model
server.
So
all
in
all,
we
saw
an
envelopes,
part
workflow,
where
we
went
from
experimenting
inside
a
notebook
to
generating
a
produce
full
pipeline.
Then
we
performed
hyperparameter
tuning
by
running
the
pipeline.
B
A
So,
thank
you
so
much
giannis.
I
think
you
can
see
how
powerful
an
ml
ops
pipeline
is
and
how
much
flexibility
it
gives
you
not
just
to
move
from
a
notebook
which
is
the
lingua
franca
of
where
people
are
doing
their
coding
around
models
today,
but
bring
that
into
a
very
rich,
very
sophisticated
pipeline.
That
includes
things
like
hyper
parameter,
suites
and
automatic
roll
out
to
production.
A
Now
this
is
a
situation
in
which
a
malicious
attacker
is
going
to
try
and
probe
your
model.
Many
many
times
to
get
an
underlying
example
of
what
that
model
is
actually
doing.
Their
goal
may
not
be
complete
accuracy,
but
as
long
as
they
can
get
close,
they
can
start
to
use
your
model
in
really
malicious
ways,
and
these
are
really
hard
to
defend
against.
So
having
a
great
ml
ops
pipeline
is
critical.
A
The
two
types
of
attacks
I'm
going
to
walk
down
are
a
distillation
attack
where
they're
pulling
things
around
object.
Detection
and
other
things
like
that
and
the
second
is
model
extraction
which
is
more
focused
on
transformers
and
language,
so
a
distillation
attack
the
first.
You
start
with
a
black
box
model
like
you
have
here
on
the
left
and
I'm
just
going
to
begin
to
probe
that
using
sample
examples.
A
I
don't
know
what's
inside
that
model,
but
I
do
have
access
to
the
api.
So
I'm
going
to
start
with
a
heart
and
that
fails.
Then
I
present
a
pentagon
and
it
passes,
and
then
I
decided
to
pick
a
whole
bunch
of
widely
arrayed
examples
to
probe
against,
and
so
now
I
do
a
whole
bunch
of
different
ones,
and
I
begin
to
get
a
better
look
at
it
and
then
I
present
a
lot
of
models.
A
Anyone
want
to
guess
what
kind
of
detector
this
is
it's,
of
course
a
neenah,
simone
detector.
No,
of
course
not.
It
is
in
fact
a
triangle
detector
and
from
this
from
all
these
many
examples,
I'm
able
to
get
a
really
good
understanding
of
what
passes
and
what
doesn't
for
this
particular
black
box
model.
A
Now,
with
that,
I'm
able
to
pull
that
out
and
recreate
the
underlying
model,
using
just
those
examples,
and
now
I
can
present
that
model
to
my
audience,
and
they
never
have
to
go
back
to
that
original
author
model's
author,
which
obviously
is
a
real
pain.
Now
the
issue
here
is
what
how
accurate
can
I
get.
A
It's
really
not
a
lot
so
now,
let's
get
into
a
model
extraction
attack
where
we're
actually
going
to
take
that
nlp
model
based
on
transformers
and
so
on
in
this
case
burnt
came
out
somewhat
recently
from
google.
It
really
transformed
the
way
that
people
do
language
models
in
introducing
the
brand
new
transformer
architecture
at
the
time
and
most
models
that
you'll
see
today,
including
ones
like
on
azure,
are
based
on
some
derivation
of
this
original
transformer
architecture.
So
you
have
a
lot
of
surface
area
for
under
attack
here.
A
The
way
these
work
is,
you
present
a
large
corpus
of
data
to
the
model
to
train
on
and
from
that
point,
then
you
ask
a
question
of
that
underlying
data
and
it
will
do
its
best
to
give
you
a
response
in
this
case,
how
many
instruments
did
prince
play?
That
comes
from
the
corpus,
and
in
this
case
the
answer
is
27
and
you
can
see
how
sophisticated
it
is.
A
In
order
to
attack
these
models,
you
have
these
researchers
from
google
who
have
done
some
really
interesting
stuff
here.
What
they've
done
is
they've
looked
at
the
underlying
model
and
decided
that,
simply
by
presenting
either
random
words
or
words
from
the
corpus
you're
able
to
get
a
really
accurate
representation
of
what
the
model
is
looking
for.
In
this
case,
here's
what
the
random
looks
like
just
obviously
random
words
from
you
know
any
dictionary
in
the
world
or
the
corpus
itself
or
you.
A
You're
able
to
get
to
86,
which
is
really
good
and
certainly
the
basis
for
doing
a
lot
of
work,
and
it's
much
much
less
than
the
millions
of
dollars
that
it's
often
required
to
train
that
model
in
the
first
place,
so
to
use
ml
ops
to
defend
you
can
do
your
best.
You
can
try
and
focus
on
endpoints
and
securing
the
api
watermarking,
but
the
real
value
here
is
going
to
be
in
the
pipeline,
your
ability
to
retrain,
to
train
for
domains
to
fix
things
over
time.
A
That's
going
to
be
where
the
value
is
so
really.
You
should
treat
your
model
security
like
anything
else
where,
if
you're
exposing
it
to
the
world,
it
will
be
attacked
and
what's
most
important,
is
how
quickly
you
can
update
and
iterate
and
detect
those
attacks
and
make
changes
quickly
in
shorthand.
A
I
would
spend
the
majority
of
your
engineering
time
on
the
left,
hand,
side
and
much
less
of
it
on
the
right
hand,
side,
because,
on
the
left
hand,
side,
those
are
the
areas
where
you're
going
to
be
able
to
make
all
your
changes
and,
on
the
right
hand,
side.
You
know
at
best
you
can
stop
people
but
you're,
not
really
adding
a
lot
of
user
value.
So
that's
what,
if
your
attacker
gets,
your
model
is
able
to
take
your
models.
Those
are
two
examples
there.
A
Now,
let's
talk
about
a
third
one
around
data
leakage
where
the
attacker
finds
out
about
hidden
data.
In
this
case,
a
malicious
user
is
going
to
look
for
ways
to
attack
your
model
to
understand
what
it
was
trained
on.
You
probably
are
already
having
this
problem
around
data
leakage
and
the
problem
here
is:
it
just
becomes
more
obfuscated
because
of
ml
to
show
you
some
examples
of
data
leakage
today
that
didn't
require
ml
at
all.
A
Maybe
it
looks
the
network
graph
where
maybe
I'm
a
union
organizer
and
it
detects
who
all
my
friends
are
following,
not
me
and
I'm
able
to
understand
what
they
their
graphs
look
like
and
who
they
may
be
following
or
third,
you
know
when
you
look
at
overall
jogging
and
and
race
recommendations
and
things
like
that,
they're
revealing
what
you
know,
the
actual
layout
of
potentially
confidential
buildings
and
structures.
And
again
this
wasn't
even
me
or
my
friends.
A
This
was
from
the
community
that
was
able
to
develop
this
and
roll
it
out
now,
there's
nothing
so
bad
that
it
can't
be
made
worse,
especially
with
machine
learning.
So
in
this
case,
what
you
see
is
where
the
the
model
is
predicting,
what
I
should
type
next,
and
so,
if
I
type
a
beginning
of
a
sentence,
it
may
move
on
and
reveal
other
things
that
come
from
my
corpus
of
information.
So
it's
revealing
basically
my
private
emails
as
suggestions
for
what
I
should
write
next
now.
A
The
problem
here
is,
of
course,
if
it's
doing
its
job
right,
it
sounds
great.
It
sounds
exactly
like
me.
The
problem
is
that
this
is
going
to
reveal
a
lot
about
what
I
write
in
other
males
and,
of
course
this
gets
quite
bad
here.
You
have
a
number
of
different
examples:
whether
or
not
it's
revealing
the
rest
of
my
address
or
my
phone
number.
Maybe
it
reveals
my
relationship
information
or
when
it
starts
to
get
really
bad.
A
The
problem
here
is
that,
even
in
this
case,
all
you're
really
doing
is
detecting
leakage
that
is
already
occurring
not
preventing
the
leakage
in
the
first
place,
and
it's
up
to
you
to
go
back
and
re-run
your
data
and
re-anonymize
so
that
that
data
isn't
being
leaked
through
inappropriately
now.
There
are
things
like
differential
privacy
which
may
help
over
time,
but
I
can't
stress
enough,
at
the
end
of
the
day,
you're
going
to
reveal
things
that
are
private,
because
that's
the
intent
for
this
thing
to
work
properly.
It
should
sound
like
the
original
user.
A
A
So
in
summary,
ml
ops
gives
you
a
lot
of
goodness
best
practices,
repeatable
workflows,
an
immutable
record
of
what
happened
and
real
acceleration
around
getting
to
user
benefits.
Unfortunately,
it
doesn't
give
you
this
for
free.
There
is
some
work
involved,
but
the
reality
is.
This
is
the
best
we
have
for
making
sure
your
systems
are
working.
A
The
truth
you
can
avoid
here,
your
models
will
be
attacked,
your
pipelines
will
have
issues
and
the
game
is
all
about
mitigation
of
harms
and
quick
recovery,
and
you
can
do
that
using
an
ml
ops
pipeline
and
with
that
here's,
the
final
slide
with
all
the
papers.
I
that
you
may
have
any
questions
and
thank
you
so
much.