►
From YouTube: Running Machine Learning Workloads on a Service Mesh
Description
Running Machine Learning Workloads on a Service Mesh
Data security is one of the key pillars to ensure successful operationalization of machine learning workloads. A service mesh can help build capabilities around mTLS, authorization checks combined with some other goodies to add security, resilience and observability to existing services and applications. JupyterHub is one of the most popular open source tools of choice for teams running machine learning environments. There has been a lot of demand in the community to add support for running JupyterHub with a service mesh on Kubernetes. This talk would cover the journey of adding Istio ServiceMesh support to JupyterHub, the roadblocks, the troubleshooting journey and how Istio makes operating and securing machine learning workloads easier despite the heterogeneous nature of tools that the data scientists use. This combined with network policies and other security best practices for running workloads on Kubernetes makes for a great operational and usability combo.
A
Hello
and
welcome
today,
we'll
be
talking
about
running
machine
learning
workloads
on
a
service
mesh.
My
name
is
har
simran
singh
man.
I
work
on
the
machine
learning
engineering
team
here
at
splunk,
I'm
based
out
of
vancouver
canada.
My
interests
include
distributed
systems.
I
like
the
scalability
aspect
of
distributed
systems.
I
have
some
experience
working
on
multi-cloud
as
well
as
security
in
the
cloud.
A
A
A
To
give
you
some
background
about
a
year
ago,
we
decided
to
re-envision
our
machine
learning
offering
here
at
splunk.
Splunk
has
grown
massively
over
the
last
few
years,
and
the
customer
demands
for
more
intelligent
systems
have
grown
as
well
to
cater
to
those
demands.
We
we
decided
to
re-envision
our
machine
learning
offering
and
we
had
a
few
clear
goals
for
the
project.
A
A
We
wanted
to
build
a
system
to
allow
the
data
scientist
to
come
and
experiment
with
the
data,
build
their
machine
learning
models,
and
then
we
also
wanted
to
have
them
operationalize
them.
So
we
wanted
to
take
these
models
and
deploy
those
models
in
production
and
and
then
we
also
wanted
to
build
the
system
which
has
a
low
barrier
to
adoption.
A
A
Do
a
different
kind
of
data,
manipulation
and
preparation
for
machine
learning,
and-
and
all
of
that,
so
we
decided
to
take
take
that
as
the
base
and
and
build
our
offering
around
that
project
before
we
jump
into
like
where
we
are
here
at
splunk,
let's
start
off
by
going
over
a
simple
definition
of
machine
learning,
so
I
stole
this
from
wikipedia.
It
says
it's,
the
study
of
computer
algorithms,
that
learn
through
experience
and
and
what
it
really
translates
into
is
is
this.
A
So
we
start
by
defining
a
problem,
and
then
we
source
some
data
for
which
we're
trying
to
solve
this
problem,
we'll
massage
this
data
a
bit
and
then
run
some
algorithms
on
it.
That's
that's
the
modeling
part,
and
once
once
we
get
the
model
as
a
result,
we
would
do
some
evaluations
on
it.
At
this
point
we
might
decide
to
go
back.
A
A
A
A
So
we
have
data
flowing
between
all
of
these
different
components
and
and
we
we
use
our
machine
learning
models
on
top
of
this
data,
but
this
wasn't
always
like
this.
We
started
off
with
a
very
simple
jupyter
hub
deployment,
so
jupiter
hub
is
a
project
that
helps
run
jupyter
notebooks
on
kubernetes.
A
A
Proxy
takes
care
of
routing
traffic
to
the
hub
or
to
the
user
notebooks,
depending
upon
where
the
request
needs
to
be
routed
to
in
order
to
better
understand
jupiter
hub.
Let's
do
a
deployment
on
a
kubernetes
cluster.
I
have
a
kubernetes
cluster
running
that
we
can
use
to
deploy
jupiter
hub.
Let's
look
at
different
components
of
jupiter
hub
that
we'll
be
deploying
today.
A
A
Let's
look
at
all
the
components
that
we're
going
to
deploy
to
the
cluster.
We
have
a
config
map,
a
deployment,
a
secret.
Some
are
back
related
components
for
hub
for
proxy.
We
have
a
couple
of
services,
one
called
the
public
service,
other
one
called
the
api
and
a
deployment
component.
Let's
do
the
deployment.
A
A
A
If
I
go
to
the
browser
and
and
open
up
this
url
now,
I
should
see
a
login
page,
and
I
can
I
can
log
in
here
and
you'll,
see
that
a
new
pod
is
being
spawned
and
we
get
a
jupyter
notebook
in
the
back
end.
Let's
see
what
it
looks
like,
so
let
me
see
all
the
resources
once
again
now,
as
you
see,
there's
an
additional
user
part.
This
is
this
is
what
is
responsible
for
serving
that
notebook
ui
in
the
browser.
A
A
I
hope
that
the
demo
makes
understanding
this
diagram
a
little
easier,
but
you
all
must
be
wondering
by
now.
What
about
service
mesh?
That's
the
exact
question.
The
splunk
security
team
asked
us
when
we
tried
to
move
this
service
to
production.
All
cloud
services
in
splunk
are
required
to
ensure
end-to-end
network
traffic
encryption
and
for
us
this
was
the
foray
into
service
mesh.
A
The
first
thing
that
we
did
was
to
go
out
to
the
community
and
see
what
options
we
have,
and
this
is
what
we
found.
So
there
was
an
open
request
on
the
jupiter
hub,
kubernetes
community,
to
support
securing
connections
between
different
jupiter
hub
components
using
istio
and
that's
exactly
the
kind
of
problem
we
had
on
our
hands.
A
I
did
some
reading
and
I
thought
I
was
ready
to
enable
sto
on
the
namespace,
and
things
should
just
magically
work,
but
our
life
as
engineers
is
not
so
easy
everything
broke
when
I
enabled
sto
on
the
namespace,
so
I
had
to
learn
how
to
troubleshoot
the
service
mesh.
Every
time
I
would
solve
one
problem.
I
would
turn
into
the
next
roadblock,
but
slowly
all
the
pieces
were
coming
together
to
get
a
working
application.
A
A
It
wouldn't
work,
and
the
reason
was
that
hub
was
not
able
to
connect
to
this
pod
to
see
if
it
was
alive
or
not,
and
this
caused
it
to
think
that
the
pods
were
not
being
spawned
and
it
got
stuck
in
an
infinite
loop
to
mitigate
this.
We
introduced
a
kubernetes
service
to
front
all
the
pods.
This
required
patching
the
hub
itself,
so
all
the
traffic
from
the
hub
to
the
user
pods
would
flow
by
kubernetes
services.
A
We
had
a
working
version
and
we
deployed
it
and
gave
access
to
our
users.
Everything
worked
fine
for
the
first
few
days,
but
as
the
number
of
users
grew
on
the
system
and
the
usage
went
up,
users
started
complaining
about
connectivity
issues
to
the
service.
We
were
getting
reports
of
more
and
more
of
issues
like
these.
A
So
all
the
traffic
would
now
come
in
through
an
istio
gateway
that
would
be
routed
to
the
user
pod
or
to
the
hub
itself.
Based
on
the
kind
of
request,
we
introduced
a
new
service
which
was
compliant
to
the
hub
proxy
api
to
configure
istio,
and
all
of
this
was
done
in
a
couple
of
days.
I
don't
think
this
would
have
been
possible
had
we
not
relied
on
what
the
community
has
built
in
terms
of
service
mesh.
All
the
cool
features
that
it
ships
with.
A
So
this
is
what
it
really
came
down
to
in
the
end,
when
hub
starts,
it
registered
itself
as
one
of
the
routes
on
the
istio
gateway
to
handle
all
the
traffic
coming
to
jupiter
hub
once
this
route
is
ready,
users
can
now
come
in
and
and
start
requesting,
their
server
parts.
When
a
new
request
comes
in
to
register
a
server
pod
hub
would
go
create
the
user
part
and
also
create
the
corresponding
kubernetes
service,
and
once
it
knows
that
it
can
talk
to
this
user
part
via
that
service.
A
Once
this
route
is
created
and
is
ready
to
use,
the
route
is
returned
to
hub,
which
then
redirects
the
user.
On
to
the
user
port,
so
the
architecture
is
very
similar
to
what
it
was
before,
except
that
we
replaced
a
few
components
and
we
got
the
system
working
on
on
the
service
mesh,
so
we
got
mutual
tls
so
far
and
we
got
the
scale
that
we
needed.
A
A
Then
we
are
also
updating
hub
with
some
of
the
routing
information
to
tell
it
how
the
traffic
is
being
routed
to
it
and
and
how
it
should
behave
with
respect
to
the
incoming
request
on
the
proxy
itself.
We
are
passing
in
some
additional
configuration.
We
are
passing
in
the
name
of
a
new
istio
gateway,
as
well
as
the
namespace,
where
the
new
virtual
services
would
be
created
and
some
other
configuration
that
we
need.
A
We're
also
deploying
a
peer
authentication
component
to
the
namespace
to
ensure
that
all
components
they
have
mutual
tls
enabled
and
we
are
using
it
in
the
strict
mode,
which
means
that
all
components
must
use
mtls
when
talking
to
each
other.
The
last
change
that
we
are
making
is
to
enable
istio
injection
on
the
namespace.
A
A
A
A
A
We
also
removed
one
of
the
services
and
there
are
no
additional
components
as
of
now.
Let's
also
take
a
look
at
any
virtual
services
that
might
have
been
created
as
a
result.
As
we
see
here,
there
is
one
virtual
service
already
created
and
if
you
remember
from
the
diagram,
when
hub
starts,
it
would
register
itself
as
the
destination
for
all
the
traffic
coming
to
that
route.
So,
let's
see
what
this,
what
this
rule
looks
like
we
describe
this
virtual
service.
A
A
We
didn't
need
to
do
any
port
forwarding
because
we
are
directly
talking
to
the
istio
gateway
which
is
running
at
localhost
id
in
this
case,
and
I
can
log
in
and
you
should
see
the
service
being
spawned,
and
so
the
user
gets
their
notebook
once
again,
let's
go
back
and
take
a
look
at
what's
going
on
under
the
hood.
Let's
take
a
look
at
all
the
services
that
are
now
running
in
the
namespace.
A
We
have
the
new
user
part
running
as
you
can
see.
This
also
has
two
containers.
One
of
them
is
the
hto
sidecar,
the
other
one
being
the
notebook
server
itself,
and
we
also
have
a
new
service
now
to
front
the
user
part.
So
every
time
hub
would
spawn
a
new
user
part.
It
will
also
spawn
a
corresponding
service.
This
allows
this
this
hub
pod,
to
talk
to
this
user
part
and
ensure
that
it's
up
and
running
before
it
starts
configuring.
A
A
A
A
A
A
Here
you
can
see
the
connection
security
policy
is
set
to
mutual
tls
and
we
can
set
up
our
alerts
on
this
to
ensure
that
this
policy
is
enforced
all
the
time
we
can
look
at
the
services
that
are
involved
and
also
where
this
metric
is
being
collected.
It's
important
here
to
know
that
the
connection
security
policy
is
reported
correctly
on
the
reporter
type
destination,
for
every
request.
There
is
a
source
and
a
destination.
A
If
you,
if
you
look
at
the
request
that
originates
from
a
source
where
the
reporter
is
equal
to
source,
you
might
not
see
the
correct
connection
policy,
because
at
that
time
it's
not
known
whether
the
connection
would
be
a
mutual
tls
connection
or
not.
So
it's
always
beneficial
to
look
at
the
reported
destination
had
started
this
project
with
a
very
small
team.
A
We
again
used
features
from
the
service
mesh
to
build
out
these
capabilities.
We
extended
the
features
that
were
built
around
hdr
virtual
services
and
destination
host
to
allow
selective
routing
on
deployment.
So
this
allowed
every
engineer
to
to
deploy
their
component
and
test
them
in
a
production
like
setting
service.
A
So
as
all
the
traffic
is
now
flowing
through
the
service
mesh,
we
can
easily
see
what's
going
on
in
the
system.
So
you
we
can
see
which
service
is
talking
to
which
other
service
we
can
see
how
many
requests
our
service
is,
making.
How
many
requests
are
failing?
How
big
of
a
data
we
are
transferring
and
things
like
that,
while
moving
to
a
service
mesh,
wasn't
our
first
choice,
it
all
turned
out
to
be
a
blessing
in
disguise
in
the
end.
A
In
conclusion,
I
would
like
to
say
that
a
polyglot
environment
needs
a
service
mesh.
The
learnings
that
you
have
from
putting
one
service
on
a
service
mesh
are
transferable
to
all
other
services
that
you're
running
in
the
system,
although
troubleshooting
can
be
a
little
tricky
and
you
can
run
into
some
really
challenging
scenarios
with
the
network
once
you
are
passed
it
operating
your
services
at
scale
in
a
secure
manner
becomes
really
really
easy.