►
From YouTube: Kubernetes as a Substrate for ATLAS Compute - Fernando Barreiro Megino & Lukas Heinrich
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Kubernetes as a Substrate for ATLAS Compute - Fernando Barreiro Megino, University of Texas at Arlington & Lukas Heinrich, TU München
The ATLAS experiment at CERN is one of the largest scientific machines built to date and will have ever growing computing needs as it explores higher energy and luminosity proton collisions. Recent R&D on the integration of cloud infrastructures with ATLAS' Worldwide LHC Computing Grid resources identified Kubernetes as a commonly available, ideal substrate. While Kubernetes is widely known for its service management capabilities, it also offers powerful batch controllers for containerised workloads. We exploited these capabilities to build ephemeral batch clusters with over 100k vCPU to process tasks that require quick turnaround, make available GPU resources that are not widely available in our own infrastructure, or create interactive facilities, where users can easily spin up private clusters for their distributed analysis from a notebook.
A
All
right,
hello,
everyone
we're
very
excited
to
be
here.
Lucas
has
been
in
previous
cube
cons,
but
for
me
it's
first
kubecon
and
also
first
time
on
official
travel
for
two
years
and
a
half,
so
we're
very
happy
to
talk
a
little
bit
about
the
journey
that
we've
been
going
on
in
the
last
years
to
integrate
kubernetes
into
the
atlas,
distributed
computing
and
help
with
the
search
for
new
phenomena
in
particle
physics.
A
I
am
fernando
barreto,
I'm
actually
from
spain
studied
information,
technology
and
telecommunications
in
madrid
and
after
university.
I
moved
to
geneva
and
have
been
most
of
my
professional
career
working
in
the
atlas,
experiment
covering
different
roles,
and
lately
I'm
running
projects
to
integrate
the
kubernetes
and
the
public
and
cloud
public
and
private
cloud
resources.
And
that's
where
I
know
luca's
from.
B
B
Okay.
So
by
way
of
introduction,
what
is
cern?
So
you
might
have
heard
about
cern
already
in
the
keynote
today.
So
cern
is
one
of
the
major
particle
physics
laboratories
in
the
world,
and
so
I,
as
a
participant
view
it
as
a
particle
physics
lab,
but
the
way
that
cern,
probably
impacted
your
lives.
The
most
is
by
being
the
birthplace
of
the
world
wide
web.
So
here,
on
the
right
hand,
side
you
can
see
tim
berners-lee,
who
was
working
at
cern
when
he
was
drafting
the
first
proposal
for
the
world
wide
web.
B
Okay,
so,
but
in
some
sense
for
us
as
physicists
the
invention
of
the
world
web
web,
which
was
originally
conceived
as
a
way
to
more
efficiently
exchange.
Information
between
scientists
was
just
a
byproduct
of
our
scientific
activity,
and
so
here
you
can
see
cern.
So
it's
in
a
nice
swiss
countryside
and
we
have
many
projects
going
on
at
cern,
but
the
biggest
one
is
the
large
hadron
collider
that
you
might
have
heard
about
in
the
news.
It's
a
27
kilometer
long
tunnel
that
is
100
meters
underground.
B
It
goes
through
france
and
switzerland
and
here
we're
accelerating
particle
beams
to
almost
the
speed
of
light,
and
so
we
have
two
beams.
One
is
going
clockwise,
the
other
one
is
going
counterclockwise
and
at
these
four
specific
points
we
collide
the
beams
head
on
and
then
hopefully
something
interesting
happens
like
the
creation
of
a
new
elementary
particle.
So
fernando-
and
I
we
both
work
at
the
atlas
experiment,
which,
in
my
mind,
is
of
course,
the
best
experiment
at
the
lhc
okay.
B
And
here
you
see
what
happens
at
one
of
these
collisions
points.
So
I
showed
there
are
four
collision
points
and
this
is
the
collision
point
where
the
atlas
experiment
said.
So
this
is
the
atlas
experiment,
it's
one
of
the
biggest
scientific
machines
that
humankind
has
ever
built,
and
so
what
is
it?
It's?
Basically,
a
three
huge
giant
three-dimensional
camera
that
records
what
is
going
on
during
these
particle
collisions
from
all
angles.
B
So
we
want
to
know
what
is
happening
during
these
collisions
so
that
we
can
analyze
it
later
on
offline
while
we're
taking
the
data,
and
so
these
collisions
they're
not
happening.
You
know
once
a
week
or
once
a
day
or
once
a
minute,
they
happen
every
25
nanoseconds
and
they
produce
a
couple
of
megabytes
of
data
per
collision.
So
we
have
40
million
collisions
every
second
and
so
there's
a
lot
of
data
that
is
being
accumulated.
B
B
Okay.
So
if
you're
actually
managing
to
extract
some
interesting
science
out
of
it,
nice
things
can
happen.
So
here,
for
example,
we
can
discover
a
new
elementary
particle.
This
happened
almost
exactly
10
years
ago.
This
was
the
announcement
of
the
discovery
of
the
higgs
boson
and
a
couple
of
years
ago,
at
kubecon
in
barcelona.
We
also
showed
how
we
can
use
kubernetes
to
rediscover
the
higgs,
and
so
because
of
this
discovery
that
we
made
at
cern.
B
B
So
we
need
to
distribute
globally
across
something
like
a
million
cpus
and
fernando
is
going
to
talk
a
bit
about
that
and
it's
a
this
pre-processing
stage
is
a
fairly
well-organized
activity,
where
you
have
only
a
few
teams
that
are
able
or
in
charge
of
pre-process
running
this
pre-processing
campaign,
and
so
we
heard
a
lot
about
batch
processing
also
at
this
conference,
and
there
was
a
co-located
event
about
the
kubernetes
batch
working
group,
and
so
this
is
a
classic
batch
workload.
Where
we're
not
super
interested.
B
When
this
processing
happens,
we're
interested
that
it
happens
at
some
point,
but
it
doesn't
need
to
be
now.
It
doesn't
need
to
be
tomorrow.
It
can
happen,
maybe
in
a
week
from
now-
and
so
this
is
a
kind
of
a
very
large
scale
system
and
then
the
user
analysis,
like
the
data
scientist
view
of
the
system,
is
down
here
and
it's
a
couple
of
orders
of
magnitude
below
that.
B
B
So
ideally,
what
we
want
to
have
is
a
kind
of
interactive
data
analysis
experience
where
you
can
try
something,
even
though
you're
still
kind
of
roughly
at
this
petabyte
scale,
and
so
the
way
that
we're
structuring
this
talk
is
that
fernando
is
going
to
talk
a
bit
about
this
first
production
system
tier
and
then
later
on.
We'll
try
to
do
a
live
demo
and
try
to
actually
do
some
physics.
That
represents
roughly
what
we're
doing
in
this
data
analysis
tier
at
the
bottom.
A
Okay,
so,
while
most
of
the
people
started
hearing
about
cern
and
atlas
around
2008,
which
is
which,
when
atlas
started
to
go
into
production,
and
then
there
was
more
media
attention-
also
books
like
don
brown's
angels
and
demons,
but
these
experiments
are
actually
being
planned
decades
in
advance.
So
atlas
was
being
discussed
in
the
90s
and
the
computing
infrastructure
was
being
discussed.
The
late
90s
and
beginning
of
the
2000s
and
back
in
those
days
there
was
simply.
Industry
was
not
at
the
level
that
it
is
now.
There
was
no
cloud
computing.
A
So
for
this
reason
in
2001
the
worldwide
lhc
computing
rate
was
conceived
and
they
came
up
with
a
plan
of
that
each
of
the
university
and
laboratory
that
was
participating
in
the
atlas
experiment
or
in
the
lhc
experiments.
They
would
also
contribute
a
little
bit
of
for
the
computing
power
of
the
experiment
on
the
computing
storage
so
and
then
the
wlcg
also
developed
all
of
the
middleware
and
storage
elements
and
compute
elements.
A
That
would
be
doing
the
storage
and
processing.
Today
the
atlas
statistics
are
around
165
data
centers
distributed
in
40
countries.
You
can
see
here
in
the
in
this
image
the
center
is
around
is
in
switzerland
around
geneva,
and
since
we
are
in
spain
also
in
spain,
you
see
that
there
are
three
data
centers
one
is
in
madrid.
One
is
in
valencia,
it's
quite
close
actually
here
to
the
to
the
to
this
venue,
and
also
the
tiagoan,
is
in
barcelona.
A
A
You
see
in
this
diagram,
it
was
the
evolution
of
the
last
seven
years,
more
or
less
how
we
have
been
growing
and
to
date
we
are
on
the
seven
hundred
eight
hundred
thousand
virtual
cpus,
for
course,
and
the
resources
are
different.
So
the
main
component
are
the
pledged
resources
which
are
the
traditional
grid
resources,
but
then
we
also
have
opportunistic
or
over
the
pledge
resources
provided
by
cloud
and
by
hpc's
that
we
have
very
successful
collaborations
with
going
a
little
bit
in
a
high-level
diagram.
A
So
you
see
that
we
have
our
users,
they
interact
with
the
data
and
the
workload
management
system.
These
systems
make
the
grid,
look
like
a
unified
resource
and
abstract
all
of
the
distributed
nature
of
the
system
and
then
for
the
workload
management
part.
We
have
this
harvester
component,
which
is
the
one
that
talks
with
all
of
the
batch
systems
and
resources.
A
The
typical
flavor
is
the
are
the
grid
sites
and
the
harvester
talks,
the
hd
condor
or
the
arc
middleware
apis
to
submit
jobs
for
hpc's.
We
have
a
lot
of
collaborations
and
use
different
supercomputers,
and
usually
this
has
to
be
done
on
a
case-by-case
basis,
since
every
hpc
is
different
and
then
the
one
side
that
we
are
actually
focusing
this
present
for
focusing
this
presentation
on
is
the
integration
with
kubernetes.
A
A
Here
we
show
a
little
bit
how
we
integrated
harvest
and
kubernetes.
So
the
the
one
requirement
that
we
had
is
that
we
we
were
not
starting
from
zero.
We
needed
to
integrate
kubernetes
in
a
in
a
structure
that
was
that
is
already
15
years
old
or
so
on,
and
we
needed
to
integrate
the
kubernetes
in
a
way
that
it
offers
all
of
the
services
that
a
traditional
grid
site
is
offering.
A
So,
while
most
of
the
people
know
kubernetes
for
like
application,
management
and
service
management,
kubernetes
also
has
native
job
controllers,
which
allow
you
to
run
budget
applications
directly
on
kubernetes,
and
we
used
just
the
native
job
controllers.
But
if
you
attended
the
batch
working
group,
for
example,
they
are
talking
about
a
lot
of
extensions
that
provide
additional
capabilities
that
we
did
not
use
at
the
moment.
A
So
in
harvester
we
wrote
plugins
that
use
the
kubernetes
python
api.
They
submit
the
jobs,
they
see
the
status
of
the
jobs
and
also
when
something
goes
wrong.
They
clean
up
kubernetes
jobs
and
we
also
extended
it
a
little
bit
with
kubernetes
common
option
options,
so
we
set
limits
so
that
the
jobs
cannot
run
out
of
cpu
or
memory.
A
We
also
use
support,
affinity
and
anti-affinity,
sometimes
when
we
want,
for
example,
the
small
jobs
to
get
packed
together,
not
spread
around
into
around
the
cluster,
and
also
really,
we
use
priority
classes
when
we
want
to
have
some
higher
priority
for
the
larger
jobs,
and
maybe
the
one
thing
that
sticks
out
a
little
bit
from
from
the
pure
kubernetes
world.
Is
this
high
energy
physics
file
system
that
we
need
to
mount
and
that's
the
cvmfs?
A
So
you
need
to
imagine
cvmfs
is
like
a
content,
delivery
network
and
the
atlas
software
and
all
of
the
high-energy
physics
software
is
distributed
through
this
content
delivery
network
and
then
on
the
nodes.
We
have
a
cvmfs
client
that
fuse
mounts
the
file
system
into
the
node,
and
so
we
installed
this
as
a
daemon
set
and
then
cvmfs
shares
the
this
file
system
through
volumes
with
the
with
all
of
the
all
of
the
panda
jobs
that
are
running
in
the
cluster.
A
Following
a
little
bit
the
motto
of
this
kubecon
conference
we've,
I
think,
we've
been
also
going
onward
and
upward.
If
you
see
the
the
plot
we
started,
this
is
in
2020..
We
started
with
a
handful
of
resources.
Each
one
was
contributing
a
couple
hundred
cores
and
we
had
our
mini
kubernetes
grid,
but
then
we've
been
growing
that
significantly
last
year.
The
this
big
part
here,
that's
in
blue,
that's
actually
the
our
first
early
adopter,
which
is
the
university
of
victoria.
Here,
the
the
site
admin
in
victoria
he
at
the
beginning.
A
A
We
use
fairly
big
nodes.
We
try
to
use
the
80
vcpu
nodes
and
also
some
32
vcpu
nodes.
So
the
cluster
overall
is
2000
nodes,
something
like
that
and
from
the
harvester
point
of
view,
it's
a
scaled
harvester
instance
that
is
also
submitting
all
of
the
all
of
the
jobs
to
victorian
to
the
other,
smaller
sites,
and
it's
all
of
this
is
going
to
all
of
this-
is
controlled
by
just
fractions
of
of
our
time
and
then,
if
we
zoom
in
into
the
30th
of
april,
it's
this
un.
A
This
plot
that
I
have
down
here
and
here
you
see
the
contribution
from
all
of
the
different
sites
that
are
contributing
processing
power
to
atlas,
and
you
see
that
we
are
the
on
the
30th
of
april.
Not
always,
we
were
the
second
contributor
just
behind
vega,
which
is
our
eurohbc
and
the
top
500
supercomputer
list
in
the
world.
So
it's
something
that
quite
impressive,
the
amount
of
computers
you
can
come
out
on
with
not
so
much
person,
power
invested.
A
We
are
going
to
be
missing
out
a
lot
of
opportunities
in
the
future.
One
example
that
was
successful
this
year
was,
for
example,
the
atlas
software
team.
They
wanted
to
build
their
software
for
arm
and
want
unneeded
infrastructure
that
from
end
to
end,
they
would
want
to
simulate
atlas
events
on
on
arm
and
while
there
are
great
sites
that
are
interested
in
purchasing
arm,
but
no
one
wants
to
be
really
the
first
one
to
do
so.
A
You
see
how
the
first
ten
thousand
events
ever
processed
on
or
simulated
on
arm
were
being
generated,
and
here
they
are
being
compared
to
events
on
x86
to
see
if
they
align
properly
or
not
and
also
well
one
one
thing
that
I
used
for
for
this
thing
in
particular:
were
these
multi-arc
docker
images
which
really
make
your
life
much
more
easier?
You
just
need
to
build
the
image
once
and
docker
will
automatically.
You
say.
A
Now
we
are
shifting
gears
a
little
bit
into
the
interactive
analysis
that
lukas
described
one
some
technologies,
that
atlas
and
the
high
energy
physics
community
is
interested
in
are
interactive
analysis
facilities
based,
for
example,
on
jupiter
and
on
dusk,
and
so
we
installed
on
our
gke
cluster.
We
installed
also
jupiter
and
task
and
have
been
offering
that
to
the
users
and
also,
for
example,
offering
them
to
to
start
notebooks
or
task
classes
using
gpus
look.
I
will.
A
This
is
how
jupiter
on
the
task
integration
looks
like
lucas
will
show
it
live,
so
I
will
not
get
into
it.
But
the
one
thing
that
is
cool
is
this
plot
here,
and
it
shows
this
was
done
by
lucas.
He's
scaling
up
his
task
cluster
and
running
the
same
task
again
multiple
times,
but
each
time
on
a
larger
cluster,
and
you
can
see
first,
he
run
the
task
with
hundred
workers
and
it
took
40
minutes.
A
Then
he
run
it
with
200
workers,
the
time
that
he
was
waiting
reduced
to
20
minutes
and
like
that
until
he
was
running
it
on
1500
workers.
The
task
is
done
in
five
minutes
and
then
he's
done
with
the
job
and
if
he
likes
the
resources,
the
the
the
results
then
he's
done
for
the
day
and
if
not,
then
he
can.
A
He
can
repeat
the
process
interactively
and
can
really
focus
on
on
his
science
and
with
this
kind
of
system
we
provide
them
that
capability
and
the
cool
thing
is
that
installing
this
setup
there
are
already
hem
charts
available
that
do
most
of
the
of
the
work
for
you
like
the
task
hub,
which
provides
directly
jupiter
and
task
integrated,
helm
chart.
The
thing
that
I
mostly
needed
to
figure
out
was
the
configuration
that
I
needed
to
add
for
scalability
and
also
cost
effectiveness.
A
So
I
wanted
to
have
like
critical
parts
in
in
a
particular
node
pool,
which
is
guaranteed
so
that
the
user
is
not
connected
and
his
notebook
gets
slashed.
But
then,
for
example,
the
workers
where
we
have
thousands
of
workers
that
I
put
into
it
into
cheap
preemptable,
vms
and
like
that,
the
cost
is,
is
much
better.
A
It's
all
of
these
workflow
engines
that
you
could
also
install
on
on
kubernetes
and
like
that
you
have
the
whole
the
whole
computing
capabilities
that
are
needed
in
one
single
kubernetes
cluster
and
also,
I
think
that
the
presentation
in
this
room
after
us
will
actually
be
showing
a
demo
or
or
a
presentation
about
tube
flow
used
for
machine
learning,
at
cern
as
well,
and
with
that,
I
will
pass
it
back
to
lucas.
Okay,.
B
B
So
it's
kind
of
the
history
of
particle
physics,
as
it
goes
through
the
decades
as
we're
able
to
build
more
powerful
and
powerful
particle
accelerators,
we
can
move
from
the
left
hand,
side,
which
is
low
energy
to
the
right-hand
side,
to
high
energy,
and
every
time
we
cross
the
energy
threshold
we're
able
to
create
new
elementary
particles.
So
these
peaks
that
you
see
are
each
elementary
particles.
B
B
And
so
jupiter
is
a
data
science
ide
that
a
lot
of
people
like
so
it's
very
much
used
in
data
science,
machine
learning,
but
also
by
physicists
and
what
we
want
to
do
is
basically
to
grab
some
data
and
analyze
this
data
in
a
nice
way.
But
of
course,
in
the
kind
of
particle
physics
context,
the
data
is
much
too
large
in
order
to
just
be
processing
processing
it
in
memory
inside
of
a
single
node,
even
if
it's
a
large
node,
and
so
we
need
to
have
a
scale
out
system.
B
In
the
background
that
so
the
user
interface
is
just
kind
of
the
front
and
then
we
have
something
that
scales
horizontally.
In
the
background
and
for
this
we
will
basically
take
our
data
lake,
and
so
we
authenticate
to
the
storage
and
we
are
able
to
grab
some
data
from
the
data
lake,
and
here
we
are
basically
scaling
a
cluster.
B
Let
me
see
so
the
scrolling
doesn't
work,
and
so
here
we
see
that
we
scaled
up
the
cluster.
While
I
was
talking,
we
scaled
it
to
500
cores
and
of
course,
500
cores
are
not
a
hundred
thousand
cores,
as
we
talked
about,
but
at
the
same
time
remember
that
this
is
a
multi-tenant
system.
So
I,
as
a
physicist,
want
to
go
to
this
facility
and
then
request
500
cores
to
do
my
interactive
data
analysis,
but
then
there
might
be
100
other
physicists
that
also
want
to
have
each
their
500
cores
and
then
very
quickly.
B
You
scale
up
very
fast,
and
so
here
we
can
use
a
lot
of
the
auto
scaling
capabilities
of
kubernetes
to
scale
the
cluster
to
whatever
size
is
needed.
How
many
physicists
are
actually
trying
to
do
data
analysis
and
then,
if
it's
a
more
quieter
period,
we
can,
then
you
know
scale
the
cluster
down
again
in
order
to
conserve
costs,
especially
if
it's
on
public
cloud
resources
and.
A
B
Once
you
have
a
scaled
up
cluster,
you
can
then
actually
define
your
physics
analysis
inside
of
the
jupyter
notebook,
and
so
I
just
have
a
bit
trouble
with
scrolling,
and
so
once
you
have
the
physics
analysis
defined
in
a
jupyter
notebook
gets
distributed
to
all
the
workers
and
the
workers.
They
are
able
to
do
embarrassingly
parallel
data
processing,
where
each
worker
grabs
a
slice
of
the
data
from
the
data
lake.
They
do
whatever
processing
the
user
requested.
B
So
we
are
trying
to
extract
some
physics,
information
from
the
data
and
then
the
results
get
accumulated
back
into
this
user
interface
until
we
are
and
then
we
can
visualize
it.
So
what
we're
doing
here
in
particular
is
that
we
use
einstein's
famous
like
energy
and
mass
relationship
so
that
we
can
infer
the
energy
of
the
original
particle
by
measuring
the
mass
of
the
original
particle
by
measuring
the
energies
of
the
decay
products
and.
B
I'll
show
desperate
I'll
just
need
to
copy
this
url,
so
this
is
nice.
So,
unlike
like
a
batch
system,
what
we
see
is
that
we
have
basically
a
real-time
view
and
what's
happening
inside
of
the
data,
and
so
here
we
are
actually
visualizing
the
results
and
we
recreated
the
plot
that
we
just
showed
in
the
in
the
original
slide,
and
so
we
can
also
go
to
the
dashboard.
Let
me
see
and.
B
And
so
so,
you
can
see
kind
of
in
a
live
view
of
what's
happening
inside
of
the
dashboard
and
what
is
happening
on
the
data
processing
site,
and
so
this
gives
us
a
very
interactive
feel
of
what's
going
on,
and
so
here
we
basically
process
60
million
events
in
just
two
minutes
and
we
basically
recreated
all
of
this
particle
physics.
History
live
on
kubernetes
in
a
live
demo,
so
I'm
very
happy
that
this
worked,
even
though
the
scrolling
didn't
work
just
as
well
as
we
wanted
to
okay.
B
B
I
log
on
to
the
system,
and
I
can
scale
out
dynamically
to
whatever
many
cores
I
want
and
then
it's
a
multi-tenant
system
where
a
lot
of
people
are
able
to
use
this
interface
and
they
can
use
jupyter
notebooks
in
order
to
do
their
data
analysis.
Okay,
so
for
this
I'll
then
hand
it
back
to
fernando
for
the
summary.
A
Thanks
for
the
demo
lucas
yeah,
so
just
concluding
so
I
think
we've
shown
you
that
kubernetes
goes
far
beyond
just
pure
service
management.
We've
been
using
it
natively
for
bunch
processing.
We
were
even
not
expecting,
we
would
get
to
this
scale.
We
were
thinking.
This
would
be
at
a
few
thousand
course,
but
we
managed
to
scale
up
two
hundred
thousand
cores
in
a
single
cluster
and
I
think
that
we
might
not
be
at
the
limit
yet
and
we
are
already
trying
to
convince
people
to
let
us
try
a
higher
scale.
A
A
Besides
this
batch
processing
processing,
we
also
showed
this
kind
of
next
generation
interactive
services
that
that
users
can
use
for
interactive
analysis
facilities
and
also
kubernetes
provides
a
very
high
elasticity
to
scale
up
and
down
with,
as
the
users
request,
the
workers
and
obviously
there
are
other
functionalities
that
can
be
added.
Other
services.
Functionalities
from
that,
for
example,
are
being
looked
at
in
the
kubernetes
working
group
for
batches
and
so
on
and
well
in
the
next
years.
A
We
will
see
how
this
all
of
this
kubernetes
integration
is
accepted
in
in
our
wlcg
world,
and
maybe
one
dream
is:
can
we
have
the
same
way?
There
is
from
zero
to
two
beta:
can
we
have
a
helm
chart
on
githubs
that
does
from
zero
to
to
a
grid
side
where,
with
one
one
commit
you
will
have
a
grid
site
installed,
so
we
will
see
how
far
we
get
there
and
also
we
need
to
see
how
in
our
university
settings
we
will
also
start
deploying
more
kubernetes
clusters.
A
So
yeah,
it's
it's
not
perfect.
Yet
there
is
still
a
lot
of
work
to
be
done,
but
I
think
that
with
fairly
reasonable
amount
of
effort
that
we
dedicated
to
it,
we
have
been
reaching
very
promising
results
and
well
and
just
to
conclude
some
acknowledgement
acknowledgements
to
people
that
have
worked
with
us
and
also
I
mentioned
that
google
actually
gave
us
the
funding
for
for
lucas
demo.
So
thanks
a
lot
for
that
and
that's
it
from
us.