►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Security Best Practices for AI on Kubernetes - Guy Salton, Run:AI
Data Scientists and MLOps engineers are embracing containers and Kubernetes for building, debugging, training and deploying deep learning models. There are many advantages for using Kubernetes for AI workloads, but is it secure? In this talk, we will present the security concerns for AI workloads running on Kubernetes and how to mitigate them: Which user is used inside the container? Can the Data Scientist use privileged escalation from his container and access the host filesystem? How to allow Data Scientists to install python packages in a secure manner? Can a Data Scientist have access other researchers code and data from his container? Guy Salton, Solution Engineering Lead at Run:AI, will cover all the concerns above, and provide security best practices to MLOps engineers, to make the everyday work of Data Scientists both secure and productive.
A
Hi
everyone
and
welcome
to
my
talk
today
at
kubernetes
a.I
day
we're
going
to
talk
about
security,
best
practices
for
ai
on
kubernetes.
My
name
is
guy
salton.
I
work
for
a
company
called
run.
Ai
eye
provides
a
platform
for
orchestrating
and
optimizing
ai
workloads
on
top
of
kubernetes
and
we'll
talk
about
some
security
best
practices
today,
a
bit
about
myself.
A
So
I'm
the
solution,
engineering
lead
at
run,
ai,
I'm
a
big
fan
of
kubernetes
been
working
with
it
for
a
few
years
now,
and
I
live
in
tel
aviv
in
israel
and
and
let's
quickly
review
our
agenda
for
today.
So
we'll
talk
about
kubernetes
and
containers
for
data
scientists
and
why
data
scientists
are
adopting
these
technologies.
A
So
you
can
see
a
few
of
the
names
here,
there's
also,
of
course,
the
nvidia
and
the
ngc
library,
which
provides
a
lot
of
pre-trained
models
and
containers
that
researchers
can
work
and
start
getting
started
very
fast
and
with
their
models.
All
of
these
things
were
built
for
containers
and
that's
why
data
scientists
want
to
use
containers
and
when
using
containers,
you
know
that
de
facto,
the
new
standard
for
orchestrating
containers
is
kubernetes
right.
So
you
can
see
a
few
stats
here
this
from
a
recent
study
that
was
done
by
datadog.
A
You
can
see
that
almost
90
of
containers
are
orchestrated
and
the
de
facto
container
orchestration
tool
today
is
kubernetes
used
in
more
than
half
of
containerized
environments.
The
advantages
you
know
with
were
talked
about
many
times.
What
we're
going
to
talk
about
today
in
focus
is
what
data
scientists
actually
need
and
what
kind
of
work
will
they
do
and
how
we
make
sure
it's
secure.
So
what
do
data
scientists
need?
They
need
to
run
containers
that
have
their
deep
learning
framework
inside
it
like
tensorflow
or
pytorch
or
keras.
A
So
this
is
what
they
need,
but
on
the
other
side
you
know
we
have
the
security
team
and
they're
thinking.
Okay,
you
know
data
scientists
want
to
run
their
containers
and
they
want
to
use
some
kubernetes
for
that.
But
how
do
we
make
sure
that
researchers
don't
mess
with
other
containers?
We
don't
want
a
researcher
to
mess
around
with
system,
components
or
other
researchers
containers.
We
don't
want
a
researcher
to
have
access
to
other
researchers.
A
Data
right
data
is
a
big
thing
and
we
don't
want
a
researcher
to
be
able
to
access
somebody
else's
data
change
it
or
even
even
just
read
it,
and-
and
we
also
don't
want
researchers
to
have
access
to
shared
resources,
meaning
they
should
only
be
able
to
work
inside
their
container.
We
don't
want
them
to
access
the
host
running
the
container
and
then
they
can
potentially
change
things
on
the
host
that
can
affect
other
users
and-
and
we
don't
want
them
to
do
that.
A
A
The
second
best
practice
is
for
the
fact
that
we
don't
want
researchers
to
access
other
researchers
data
and
to
make
sure
that
this
is
applied.
We
would
need
to
set
the
right
permissions
on
the
researchers
data
directory
and
so
to
make
sure
that
this
directory
can
only
be
accessible
by
a
certain
user
and
and
then
we
would,
in
our
pod
definition
in
kubernetes,
make
sure
that
we
use
the
right
user
and
we
mount
the
right
directory
and
we
shouldn't
be
allowed
to
mount
somebody
else's
directory
or
impersonate.
A
Somebody
else
and
we'll
see
how
this
is
enforced.
And,
lastly,
we
said
we
don't
want
a
researcher
to
access
the
shared
resources
right.
The
host,
where
the,
where
the
container
is,
is
actually
scheduled
on
to
make
sure
that
this
doesn't
happen.
We
need
to
restrict
the
each
of
the
users,
so
basically
restrict
the
cluster,
the
namespaces
on
the
cluster,
from
running
pods,
with
root
user
and
with
privilege
escalation.
A
We
have
to
enforce
and
make
sure
that
the
researchers
run
with
their
own
user
and
they
don't
try
to
use
privilege
escalation
to
access
the
host.
So
we'll
see
how
this
can
be
enforced
in
kubernetes.
So
let's
go
and
see
the
demo.
I
created
the
public
github
repo
with
all
of
the
files
and
examples
that
I'm
going
to
show
today,
so
you
can
feel
free
to
access
it
and
and
follow
along,
and
so
let's
go
ahead
and
get
started
and
let's
go
and
start
by
looking
at
our
cluster.
A
So
I
have
a
kubernetes
cluster
that
I
created
and
you
can
see
that,
let's
make
sure
I
use
the
right
config
file.
So
if
I
run
the
cube
ctl
get
nodes,
I
can
see.
I
have
a
kubernetes
cluster
with
three
nodes.
I
have
my
master
node,
cpu
node
and
another
node
with
gpus,
and
I
said
that
I
want
different
users
to
have
different
namespaces,
so
they
won't
be
able
to
do
things
on
other
namespaces.
So
let's
say
I
have
two
researchers
in
my
team.
A
I
have
a
researcher
called
bob
and
one
called
alice,
and
so
what
the
first
thing
I
would
need
to
do
is
create
a
namespace
for
bob
in
a
namespace
for
alice.
I
already
did
that.
So
if
I
run
the
cube,
ctl
get
an
s
you'll
see
I
have
a
namespace
called
alice
and
one
called
bob,
and-
and
now
I
want
to
make
sure
that
bob
will
only
be
able
to
do
things
inside
his
namespace
and
alice
will
only
be
able
to
do
things
inside
her
namespace.
A
So
to
do
that,
what
we
did
is
we
have
to
create
a
few
resources
in
kubernetes.
First
of
all,
there
is
one
called
service
account,
then
a
role
and
a
role
binding.
So
we
do
this.
For
each
of
the
user.
We
create
a
service
account
world
and
world
binding
for
bob
and
then
another
one's
for
alice,
and
so
the
service
account
is
basically
a
user
in
kubernetes.
You
see,
we
just
create
a
service
account.
A
We
call
it
bob
user,
we
map
it
to
the
namespace
bob,
then
we
create
a
role
and
for
bob
this
is
also
mapped
to
the
namespace
bob,
and
it
will
basically
allow
us
as
bob
to
do
everything
that
we
want
run.
Pods,
run
jobs,
deployments
everything
on
the
namespace
bob
and
then
a
role
binding
is
what
binds
the
service
account
with
the
role.
So
you
see,
I
create
this
thing
called
world
binding.
It's
also
going
to
be
created
in
the
namespace
bob
and
it
will
bind
this
service
account.
A
We
created
called
bob
user
with
the
role
that
we
created
for
bob,
so
we
did
a
similar
thing
for
alice
and
all
of
these
are
in
the
github
repo.
So
you
can
just
go
and
run
the
cube.
Ctl
apply
dash
f
on
this
yaml
file,
and
this
will
create
all
of
these
three
things
for
bob
and
then
other
three
for
alice.
A
So
if
we
go
and
run
the
cube,
ctl
get
roll.
For
example,
on
the
bob
namespace,
we
will
see
rolls
bob's
roll
and
we
will
also
see
his
roll
binding
and
his
service
account.
So
all
of
these
were
created
and
similarly
for
alice,
and
once
we
have
these
three
three
things
created
for
each
user,
we
can
create
a
kubernetes
config
file.
So
I
won't
go
too
deep
into
this.
We'll
have
an
example
in
the
github
repo,
but
you
this
is
the
file
that
each
user
would
use
to
access
the
cluster.
A
A
So
just
to
see
that
this
works,
we
can
now
impersonate,
for
example,
s-bob
by
using
bob's
kubernetes
config
file,
which
we
just
saw
now
and
you'll
see
that
now.
If
I
try
to
get,
for
example,
pods
on
the
bob
namespace,
I'm
allowed
to
do
that.
Currently,
there's
no
pods
running,
but
if
I
try
s
bob
to
see
pods
running
on
the
namespace
alice,
you
see
I'm
forbidden,
I
don't
have
permissions
to
list
pods
on
the
namespace
alice
as
well
as
in
any
other
namespace
only
on
the
namespace
pop
and
similarly
for
alice.
A
So
this
is
the
first
best
practice
it's
now.
It's
now
implemented
second
best
practice.
We
wanted
to
make
sure
that
each
researcher
only
has
access
to
his
or
her
data,
and
so
to
do
that
I
created
an
nfs
server
and
I
mounted
it
here
to
one
of
my
nodes
in
the
cluster,
the
gpu
node
and
you
can
see.
I
created
a
file,
a
folder
called
alice
and
one
called
bob
in
my
nfs
folder,
and
each
of
these
folders
has
different
permissions.
A
The
only
person
can
access
this
folder
is
alice
and
bob
can
only
be
accessible
by
user
bob
and
to
see
that
what
is
actually
inside
I
can
go
and
show
you
that,
for
example,
in
the
bob
folder
I
have
a
file
called
data
txt
and
it
contains
one
line
and
which
is
bob
data.
And
if
I
look
at
the
directory
of
alice
I'll
see
that
I
have
a
similar
data.txt
file,
but
for
alice
it
will
contain
a
single
line
which
is
alistair.
A
So
I
have
two
folders
one
for
alice,
one
for
bob,
the
bob
folder
has
is
accessible
only
by
user,
bob
alice
is
only
accessible
by
user
alice.
Each
of
them
has
their
own
data
files.
So
this
is
nfs
now
to
be
able
to
mount
these
nfs
directory
into
my
pods
in
kubernetes.
I
have
to
create
two
things.
So
first
one
is
called
pv
persistent
volume
and
then
something
called
persistent
volume
claim,
and
so
persistent
volume
defines
my
volume
that
I'm
going
to
use.
So
this
is
the
persistent
volume
for
bob.
A
You
see,
I
called
it.
Nfs
pv
bob
it's
going
to
be
created
in
the
namespace
bob
and
it's
going
to
use
and
access
my
nfs
server.
This
is
my
nfs
server
ip
and
it's
going
to
get
my
bob
folder
and
similarly
for
alice,
we
created
another
one,
but
this
would
get
the
names
the
the
alice
folder
on
the
namespace
alice.
Then
we
have
to
create
a
pvc
so
to
be
able
to
use
this
volume
in
our
pods.
A
This
is
the
resource
that
we
can
actually
refer
to
in
the
pod,
so
we
created
one
for
alice
again
in
the
alice
namespace,
using
the
same
storage
class
name
as
the
pv
and
then
one
for
bob
in
the
bob
namespace,
and
so
we
have
our
storage,
pv,
pvc
and
nfs
configured
and
now
for
the
last
best
practice.
We
said
we
want
to
make
sure
that
pods
cannot
run
with
a
root
user.
They
can't
run
with
privileged
escalation
so
to
enforce
something
like
that
on
our
cluster.
A
There
is
something
called
port
security
admission.
There
are
a
few
things
that
you
can
do
actually,
but
there
used
to
be
something
called
port
security
policies,
but
you
can
see
that
kubernetes
decided
to
deprecate
it
and
completely
remove
it
from
kubernetes
in
version
1.25.
So
you
should
be
aware
of
that.
The
new
recommended
approach
is
called
pulled
security
admission
using
an
admission
controller.
A
A
Once
these
are
applied,
you
can
enforce
different
security
standards
on
each
namespace
in
your
cluster,
so
there
are
three
basic
standards
and
what
the
standard
that
we
want
to
use
is
called
restricted,
so
the
restricted
standard
has
heavily
restricted
policy
and
this
will
prevent
pods
in
the
namespace
to
do
unsecured
things.
So
if
we
look
at
some
of
the
things
that
it
will
enforce,
you
see
that
once
the
restricted
standard
is
enforced
on
our
namespace,
it
won't
allow
us
to
run
pods
with
privilege
escalation.
It
won't
allow
us
to
run
pods
with
root
user.
A
It
will
require
non-root
user
as
well
as
other
things
here.
So
I
enabled
it
on
my
cluster
and
and
now
to
go
and
actually
do
it
by
the
way.
I
can
see
that,
for
example,
here
in
my
masternode
in
kubernetes,
if
I
go
to
the
etc
kubernetes
manifest,
I
can
go
and
look
at
my
files
here.
So,
for
example,
if
I
look
at
the
cube
api
server.
A
You
should
see
that
under
the
container
commands
I
added
this
feature:
gate
pod
security
equals
true.
So
it's
enabled
on
my
cluster
and
and
now,
if
I
go
and
look
at
my
namespace,
you
can
see
that
I
can
do.
For
example,
cube
ctl
get
ns
pub,
dash,
o
yaml
and
I'll
see
that
I
added
specific
labels
on
this
namespace,
which
are
the
cube
security
in
force,
and
so
I
went
and
actually
enforced
the
restricted
policy
on
the
bob
namespace.
A
I
did
the
same
thing
for
the
alice
namespace,
which
means
that
as
the
the
mode
is
in
force,
when,
if
I
try
to
run
pods
that
don't
comply
with
the
security
standard,
the
admission
would
just
block
them
and
let's
go
and
see
this
in
action.
So
let's
go
and
try
to
run
a
pod.
I
created
a
pod
definition
here
for
bob,
so
I
would
try
to
go
and
run
a
pod
in
the
blob
name
space
and
this
would
run
an
image
called
jupiter
tensorflow
notebook,
so
it
will
contain
jupiter
as
well
as
tensorflow.
A
It
will
go
and
mount
my
bulb
folder
from
nfs.
So,
let's
see
what
happens
if
I
impersonate
s-bob-
and
I
then
try
to
run
this
pod
so
I'll
go
and
run
the
cube.
Ctl
apply
pod
bob,
not
secure
and
you'll,
see
that
once
I
try
that
I
get
an
error
right.
My
pod
security
admission
gives
me
an
error.
It
tells
me
that
I
try
to
run
a
pod
with
privilege
escalation.
I
try
to
run
a
pod
with
a
root
user.
This
is
not
allowed,
so
I'm
blocked.
A
So
this
is
a
good
thing
to
apply.
Now,
let's
look
at
the
proper
product
that
is
his
companies
and
is
where
I
applied
all
the
security
standards.
So
it's
the
same
image,
but
this
time
I
added
security
context,
and
I
said
the
run
is
not
non-rules.
Non-Root
is
true.
I
specify
the
actual
user
id
that
I'm
going
to
run
this
board
with
501
is
the
user
id
of
bob.
I
added
a
few
other
things
here,
like
the
runtime
default
as
the
type
of
my
second
profile.
A
This
is
also
required.
I
also
have
to
drop
all
of
my
linux
capabilities
and
set
my
allow
privilege
escalation
to
false,
so
my
pod
won't
be
able
to
do
privilege
escalation
and
then
again
I'm
going
to
mount
my
bob
folder
from
nfs
to
this
directory.
This
path
in
my
container
and
so
let's
go
and
see
what
happens
when
I
try
to
run
this
mod.
A
I
should
now
be
able
to
exec
into
the
pod,
so
exec
id
the
pod
name
in
the
bob
namespace,
I'm
going
to
use
bash
for
my
session
and
now
I'm
inside
the
board,
and
you
see
that
if
I
run
the
id
command,
I'm
running
as
id
user
id
501,
I'm
not
not
the
root
user
and
if
I
try
to
run
sudusu,
for
example,
see
I'm
not
allowed
to
run
sudo,
as
I
don't
have
privilege
escalation.
So
this
is
good
and
if
I
look
at
my
file
system
here,
I
see
that
I
have
my
workspace.
A
This
is
where
I
mounted
my
nfs
folder
and-
and
I
have
my
data
file
here-
that
was
mounted
from
nfs.
I
can
see
my
data,
it's
bob's
data
and,
if
I
want
to
let's
say,
install
some
dependencies,
I
can
run
the
pip
install
command
to
install
keras,
for
example,
so
I
can
run
it.
It
would
go
and
install
keras
on
my
home
directory,
so
everything
is
good,
so
that
was
the
demo
and
going
back
to
our
slides.
A
A
Then
the
second
best
practice
was
for
the
data.
We
set
the
right
permission
for
the
researchers
data
directory
in
nfs
in
our
case
and
then
make
sure
that
we
use
the
specific
user
id
in
our
pod
definition.
And,
lastly,
we
use
pod
security
admission
to
restrict
pods
from
running
with
things
like
root
user
and
privilege
escalation.