►
From YouTube: Building and Managing a Centralized ML Platform with Kubeflow at... Ricardo Rocha & Dejan Golubovic
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Building and Managing a Centralized ML Platform with Kubeflow at CERN - Ricardo Rocha & Dejan Golubovic, CERN
CERN’s main mission is to expand human knowledge trying to understand the nature of the universe, and machine learning has been growing as a solution for challenges in different areas of development and operations. Areas where ML is being looked at include particle classification using graph neural networks during reconstruction, 3DGANs for much faster generation of simulation data, or reinforced learning for beam calibration. This session presents a recently introduced centralized service covering most use cases, handling data preparation, model training and serving. How it tries to improve resource usage (especially important when handling scarce resources such as accelerators) by offering different resource types (GPU, vGPU, TPU) for each use case. The session will also describe our journey with Kubeflow, the machine learning platform running on top of Kubernetes, and how we integrated on-premises resources and the different possibilities being looked at to extend to public clouds.
A
B
A
A
So
today
we'll
we'll
give
a
talk
about
service
at
cern,
but
just
a
very
quick
overview
of
what
cern
is
about.
So
cern
is
the
european
laboratory
for
particle
physics,
the
largest
particle
physics
laboratory
in
the
world,
and
we
build
like
large
scientific
machines
that
allow
us
to
do
fundamental
research.
The
largest
we
have
is
the
large
hadron
collider.
You
probably
heard
of
it.
A
It's
a
27,
kilometer
perimeter,
particle
accelerator
that
is
100
meters
in
the
ground
and
where
we
accelerate
two
beams
of
protons
to
very
close
to
the
speed
of
light,
and
we
make
them
collide
at
very
specific
points
where
we
build
large
experiments,
and
you
see
here,
cms,
lhcp
atlas
and
alice
to
have
an
idea
of
the
size.
You
can
see
the
geneva
airport
here
on
the
picture.
A
This
is
a
an
image
of
the
accelerator
itself
in
the
tunnel,
and
you
can
see
all
the
magnets
that
help
us
beam
the
bend
the
beam
so
that
it
circulates
in
the
in
the
accelerator-
and
this
is
a
picture
of
of
one
of
the
detectors.
The
cms
detector
compact
moon,
solenoid,
it's
in
a
cavern
40
meters
by
40
meters,
also
100
meters
underground-
and
this
is
where
we
make
the
proton
beams-
collide
this
detector
and
the
others
as
well
act
like
gigantic
cameras,
where
we
take
something
like
40
million
pictures.
A
B
Different
groups
at
cern
work
on
various
machine
learning
projects
in
order
to
achieve
scientific
goals
of
the
large
hadron
collider,
and
we
know
that
setting
up
and
managing
machine
learning
infrastructure
is
not
an
easy
task
and
currently
most
groups
at
cern
manage
their
own
machine
learning
infrastructure.
So
we
have
four
main
experiments
which
all
branch
to
different
groups,
and
that
means
that
a
lot
of
people
use
their
own
machine
learning
infrastructure.
B
One
of
the
main
applications
of
machine
learning
at
cern
is
in
particular
reconstruction,
so
during
proton
proton
collisions
short-lived
particles
are
created
in
the
detectors,
for
example,
higgs
boson,
which
leaves
10
to
the
minus
22
seconds
and
to
capture
the
events
of
the
short-lived
particles.
We
measure
energy
depositions
in
the
detectors
detectors
can
be
considered
as
3d
cameras,
which
leaves
the
opportunity
to
use
convolutional
neural
networks.
B
Another
application
is
in
detector
simulations,
so
large,
hadron
collider
is
getting
upgraded.
There
will
be
more
even
more
data
in
the
future
and
more
sophisticated
and
faster
solutions
are
needed
to
support
the
upgrade
from
various
perspectives,
one
of
them
being
simulations,
so
simulations
are
performed
to
so
that
we
can
accurately
estimate
what
is
going
to
happen
during
the
runs
and
the
the
traditional
methods
are
monte.
B
Carlo
simulations,
but
recently
3d
guns
have
started
to
be
more
commonly
used
and
they
have
proved
to
have
a
similar
performance
to
state
state-of-the-art,
monte,
carlo
and
they
offer
20
000
times
faster
simulation
and
also
with
3d
gun.
Data
can
be
simulated
on
the
fly
which
may
reduce
the
need
for
storing
the
data.
B
So
our
goal
is
to
set
up
a
platform
to
support
the
end-to-end
machine
learning
life
cycles.
We
want
to
be
able
to
extract
data
from
the
detectors
through
spark
or
hdfs
and
operate
on
that
data.
Then
we
want
fast
iteration
services
such
as
notebooks,
because
many
users
use
notebooks
daily.
At
least
the
notebooks
are
a
good
starting
point
for
every
machine
learning
user.
Then
for
more
computationally
extensive
jobs,
we
want
to
be
able
to
perform
distributed
training
with
tensorflow
or
by
torch,
and
even
we
want
to
branch
out
to
public
cloud
when
resources
are
needed.
B
Cool,
so
the
platform
that
supports
all
of
our
goals
is
cube
flow.
Basically
with
kubeflow.
We
are
utilizing
power
of
kubernetes
to
efficiently
manage
resources,
and
we
also
offer
users
there
all
the
desired
features.
The
infrastructure
part
of
cube
flow
is
managed
by
our
cloud
team,
and
our
users
are
physicists
and
scientists
across
the
entire
ser
with
kubeflow.
A
Yeah
so
I'll
pick
up
on
the
nice
description
from
there
and
before
he
does
a
cool
demo
I'll
just
talk
about
the
layout
of
the
infrastructure
we
are
using.
So
this
is
a
very
simplified
overview
of
the
clusters,
the
layout
of
our
clusters,
so
we
rely
on
as
an
entry
point
load
balancer,
and
this
allows
us
to
to
simplify
the
deployment
and,
for
example,
to
do
upgrades
by
just
adding
entry
points
to
the
balancer
new
clusters
on
the
back
end
and
then
there's
a
gateway
that
is
our
ingress
gateway
to
to
the
services.
A
We
do
not
do
today,
any
kind
of
faster
interconnect
or
anti-link
or
anything
like
this
and
the
last
bit
we
have
here
is
cpu,
and
in
this
case
we
have
a
much
larger
amount
of
resources.
It's
not
as
interesting
if
you're
doing
deep
learning,
but
actually
this
platform
ended
up
being
used
for
other
purposes
as
well,
where
workflows
and
pipelines
can
be
useful.
A
So
you
can
see
that
we
have
something
on
the
order
of
hundreds
of
virtual
gpus
or
the
tens
of
full
gpus
offered
to
the
users.
A
number
of
thousands
of
cpus
just
very
quickly,
our
deployment
is
based
on
kubernetes
118
clusters.
Today
we
use
kubeflow,
1
1
still
and
one
difference
from
the
standard
1
1
deployment
is
that
we
upgraded
this
to
one
five
and
k
native
to
zero
fifteen.
A
All
the
clusters
and
the
deployments
are
managed
using
githubs,
and
we
have
a
one
repository
where
we
define
all
the
services
and
all
the
environments
we
support
and
it's
all
managed
by
argo
cd.
There
is
one
very
good
feature
here,
which
is
argo.
Cd
allows
us
to
use
customize
just
for
the
flow
deployment
and
then
for
the
other
components.
A
A
A
So
this
allows
us
to
have
not
only
the
tokens
that
identify
the
user,
but
also
the
mapping
of
the
users
to
the
roles
and
the
groups
they
belong
to
and
what
we
do
in
our
clusters
is
we.
We
have
dedicated
namespaces
per
user,
where
people
have
a
default
quota
that
is
fixed
and
cannot
be
changed,
but
also
we
have
additional
groups
where
people
can
belong
to,
and
this
is
defined
in
the
cern
identity.
A
If
they
belong
to
these
groups
and
in
those
groups,
they
can
request
additional
quota
like
more
gpus,
for
example,
and
then
the
other
very
important
part
is
the
integration
with
our
storage
systems,
as
we
we
mentioned,
like
data,
is
a
key
aspect
of
everything
we
do
so
we
integrate
with
three
the
three
main
storage
systems
that
are
interesting
in
this
case.
The
first
one
is
what
we
call
cvmfs
certain
vmfs
fs,
which
is
a
real,
really
only
distributed
hierarchical
caching
system,
for
that
is
mostly
used
for
software
distribution.
A
The
second
one
is
an
internal
in-house
developed
system
called
eos
that
is
holding
all
the
physics
data.
In
this
case,
the
important
part
is
that
we
need
to.
We
offer
both
kerberos
and
os2
based
access.
Os2
is
very
important
for
things
like
notebooks
and
anything
that
is
like
browser
oriented
and
then
the
last
one
is
hdfs.
A
I
will
just
summarize
a
couple
of
issues
that
we
run
well
into,
while
doing
this.
The
first
one
is
that
the
couple
releases
were
not
always
very
consistent
in
terms
of
what's
supporting
what
so
one
zero
had,
for
example,
multi-user
support
for
for
notebooks,
but
not
pipelines
and
one-on-one
brought
multi-user
pipelines.
But
actually
some
of
the
components
were
not
talking
to
to
this
new
api
properly,
like
kale
from.
A
A
The
last
bit
I
will
mention
before
we
jump
to
the
demo
is
how
we
are
doing
bursting
to
the
public
clouds.
This
is
very
important
for
us
because
we
can
get
access
to
a
much
larger
amount
of
gpus
and
especially
other
types
of
accelerators
like
cpus
or
ipu's.
Tpus
are
very
interesting
for
our
use
cases
also
because
they
are
very
cost
effective.
A
A
We
are
not
still
very
experts
in
istio,
but
we
are
experimenting
with
it,
but
actually
for
coop
flow,
the
most
promising
results
and
the
way
we
are
offering
this
is
actually
to
directly
expose
the
other
clusters
to
the
to
the
users
via
their
jupiter
environment.
So
when
you
get
your
dripter
environment,
your
notebook
environment,
you
actually
get
the
additional
clusters
configured
and
these
clusters
are
configured
with
the
same
cern
sso.
A
So
we
can
do
something
like
using
the
open
policy
agent
to
validate,
who
is
able
to
access
which
which
clusters,
which
groups
can
actually
access
each
cluster,
and
then
we
do
the
quota
management
the
same
way
in
those
clusters
as
well.
This
is
working
quite
well.
This
is
a
simple,
not
so
simple
picture,
but
it's
kind
of
simple
given
what
is
behind.
A
And
then,
when,
once
this
training
is
done,
we
we
write
to
output
artifacts
to
s3,
where
it
can
be
served
from
the
s3
instance,
at
cern
by
exposing
additional
clusters
in
the
environment.
People
can
just
source
those
clusters
and
then
submit
the
tf
jobs
to
external
clusters,
and
this
in
this
case
it
would
be
good
the
google
cloud
where
we
would
have
potentially
thousands
of
gpus
available
or
gpus.
A
B
Okay,
now
we'll
go
back
to
our
demo
example,
so
we
remember
3d
guns.
Basically,
the
the
main
issue
with
3d
guns
is
extensive
training
time.
So
for
our
model,
it
would
take
two
2.5
days
to
to
properly
train
and
if,
for
example,
if
we
want
to
search
hyper
parameters
or
change
the
model
that
iteration
could
last
for
weeks
or
even
months,
so
this
is
with
one
gpu
so
to
create
a
more
scalable
solution,
a
distributed
model
is
created.
B
B
So
tf
job
and
kubeflow
helped
us
automate
this
distributed
training
process.
We
are
able
to
quickly
iterate
over
different
training
configurations.
So,
basically
we
are
encapsulating
tensorflow
distribution
and
we
mean
it's
it's
managing
it
across
kubernetes
spots.
So
we
are
able,
with
the
job,
to
run
distributor,
training
both
locally
and
on
a
public
cloud.
As
ricardo
was
describing
and
yeah,
we
at
gcp,
we
are
using
128
preemptable
machines
for
the
distributed
training,
and
now
we
can
move
to
the
demo.
B
B
B
B
B
So
here
we
can,
in
order
to
run,
we
only
need
to
click
this
button
compile
and
run,
and
we
can
see
our
pipeline
running
so,
while
our
pipeline
is
running,
we
can
check
other
features
which
we
have
in
our
service.
One
of
them
is
eos,
so
eos
is
where
most
users,
where
all
users
at
cern
have
their
personal
directories
and
the
mounting
eos
really
allows
us
to
be
able
to
access
broad
data
from
multiple
users.
B
B
B
We
also
in
this
repository,
have
a
docker
file
to
build
our
image
to
to
run
our
distributed
training,
but
we
are
not
building
it
here.
We
won't
do
that,
but
we'll
show
our
tf
job
yaml
file
so
to
submit
a
tf
job.
Basically,
we
can
define
our
number
of
replicas
here
number
of
gpus
we
are
using
here
and
then
we
also
we
select
the
image
and
we
also
can
select
if
we
want
full
training
and
the
number
of
epochs
and
other
customizable
arguments.
B
Additionally,
we
want
to
run
a
3d
gun
training
on
a
google
cluster,
so
basically
inside
the
cluster
folder
users
would
get
information
about
all
available
clusters
in
our
service.
So
here
we
only
have
a
cern
and
the
gcp
cluster
and
all
the
users
have
to
do
to
access.
The
additional
clusters
is
to
source
these
files,
so
gcp
setup
setup.sh
and
now
they
should
be
able
to
they
should
they
are
in
the
google
cloud.
B
B
B
B
So
we
can
close
this
now
and
here
we
see
that
our
local
job
has
completed,
and
now,
as
ricardo
was
saying
after
the
training,
we
submit
our
model
to
a
bucket.
So
here
we
can
see
the
the
trained
model
stored
on
on
our
buckets.
So
we
have
couple
of
files
for
each
model
and
then
this
is
all
for
for
one
model
and
for
one
ebook.
B
So
we
have
our
discriminator
and
generator
for
3d
gun
stored
in
the
bucket,
and
we
also
have
a
saved
model
in
the
format
so
that
it
can
be
used
for
inference
for
serving
and
also
we
want
to
maybe
want
to
check
this
these
metrics.
Basically,
this
is
how
we
can
store
a
metrics,
how
we
store
metrics
about
our
model
after
each
epoch.
B
B
B
B
We
can
test
it
from
here
and
we
see
that
we
are
getting
results
and
basically,
as
we
were
discussing,
we're
getting
a
3d
output
for
that
represents
the
output
of
the
detector.
So
this
is
what
happens
when
we
do
one
inference,
but
we
want
to
do
10,
curl
requests
at
the
same
time,
so
that
we
can
see
what
happens
to
the
number
of
predictor
pods.
As
we
can
see,
they
are
the
number
of
parts
it
is
increasing.
It's
auto
scaling
so
that
it
can
support.
A
A
Yeah,
so
basically,
I
hope
this
was
a
nice
overview
of
the
service
we
are
offering
and
the
potential
that
it
has
by
offering
like
a
consistent
environment
where
people
can
do
their
development,
but
also
interact
with
the
services
we
handle
all
the
machine,
learning
learning
life
cycle,
steps
from
preparation
all
the
way
to
serving.
We
managed
to
centralize
the
resources
that
are
pre-scarce,
such
as
accelerators
in
this
case
gpus,
and
also,
we
showed
how
we
are
doing
currently
the
integration
with
external
resources
for
gpu's
dpus,
using
public
clouds.
A
The
only
option
is
to
install
the
packages
on
the
notebook,
but
that
doesn't
work
well
when
you're
doing
a
transition
to
pipelines,
for
example,
or
to
distribute
training.
So
we
have
some
experience
using
a
tool
called
binder
for
jupyter
notebooks,
and
we
are
looking
at
integrating
this
with
the
kuflow
jr
web
app
as
well
and
the
last
one
is.
We
are
quite
involved
in
the
work,
ongoing
work
for
flow
improvements
in
metadata
and
artifact
management.
So
it's
something
that
we'll
also
keep
pushing
for
in
the
community.