►
From YouTube: Hunting For New Particles Leveraging Legacy Infrastructure with Kubernetes - Clemens Lange, CERN
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Hunting For New Particles Leveraging Legacy Infrastructure with Kubernetes - Clemens Lange, CERN
In the search for unknown particles in the CERN Large Hadron Collider’s particle collisions, billions of events need to be analysed. Even though large parts of CERN’s computing infrastructure are deployed using Kubernetes, physics analysis jobs are still being run on classical high throughput computing batch systems. While developing a fully cloud native computing approach, one still needs to have access to the ten-thousands of cores available on the legacy batch system to have sufficient resources for the data processing. In this presentation, Clemens will demonstrate how complex physics analysis workflows that are written and scheduled using Kubernetes can make use of classical batch systems. The audience will also learn what complexity a realistic physics analysis can reach, and the important role that software containers and Kubernetes play in the context of open science.
https://sched.co/Zeji
A
My
name
is
clemens
and
I'm
a
particle
physicist
at
cern,
I'm
working
on
the
cms
experiment,
which
is
one
of
the
experiments
at
the
large
hadron
collider
in
the
franco,
swiss
area
very
close
to
geneva
and
switzerland
in
this
photo
on
the
right
hand,
side.
You
can
see
me
standing
right
next
to
the
accelerator,
so
you
can
see
in
blue
the
dipole
magnets.
A
These
are
there
to
actually
bend
the
particle
beams,
so
they
go
around
in
circle
so
and
this
circle
actually
has
a
circumference
of
27
kilometers
and
it's
a
kilometer
underground,
and
you
can
see
here
a
lego
model
of
the
lhc,
obviously
much
smaller,
but
here's
one
of
the
experiments-
that's
particularly
interesting
or
dear
to
me,
which
is
here
in
red.
That's
the
cms
detector,
which
is
recording
the
collisions
by
the
lhc
and
I'm
analyzing.
These
data.
A
So
the
field
that
I'm
working
is
called
high
energy
physics
and
short
tap,
and
it's
all
about
trying
to
understand
the
smallest
building
blocks
of
matter.
So,
as
I
said,
the
particle
detectors
such
as
the
cms
detector,
they
record
these
collisions
and
the
lhc
actually
provides
up
to
40
million
collisions
per
second
and
the
detector.
You
can
imagine
it
as
some
kind
of
digital
camera,
so
that
means
it's
actually
taking
photos
of
of
the
collisions
and
that's
happening.
A
24
7
almost
all
year
long
and
you
can
see
one
of
those
photos
that
is
taken
or
that
has
been
taken
on
the
right
hand
side.
So
the
particle
beams
come
in
from
both
sides
and
then
in
the
very
center
of
the
detector.
The
particle
beams
are
brought
into
collision
and
then
these
sprays
of
particle
develop
here,
which
are
you
know,
lots
of
particles,
they're
very
close
together,
and
these
particles
that
we
are
investigating
actually
thinner
than
a
hair
and
the
camera
is
actually
a
bit
bigger
than
a
standard
digital
camera.
A
A
I
mean
they're
a
bit
better
than
your
digital
camera,
which
might
also
have
100
mega
pixels
because
they
can
take
40
million
seconds.
40
million
photos
per
second
so
because
there's
one
collision,
every
25
nanoseconds-
and
this
is
so
much
data
that
we
can
actually
not
store
this
immediately.
We
have
to
do
something
special.
A
We
have
to
filter
things
as
quickly
as
possible,
largely
based
on
hardware
and
then
later
on,
also
based
on
on
software,
and
we
can
store
up
to
thousands
of
such
photos
per
second
for
later
analysis
and
this
analysis
since
we're
running
all
year.
Long
is
then
a
big
data
analysis,
so
we
are
actually
analyzing
tarot
to
pattern
dates
of
petabytes
of
data
using
c
plus,
plus
python,
and
also
shell
scripts
for
the
plumbing.
A
For
this
kind
of
analysis,
we
really
need
to
have
a
big
infrastructure,
and
this
is
provided
by
the
worldwide
lhc
computing
grid,
which
consists
of
around
170
computer
centers
all
around
the
world.
You
can
see
that
on
the
right
inside
here,
so
the
data
they're
accumulated
at
sun,
so
that's
the
tier
zero
and
then
they
are
distributed
and
reconstructed
and
then
distributed
to
the
different
sites
all
around
the
world.
So
you
can
see
the
tier
one
sites
that
have
particularly
high
bandwidth
links
to
the
tier
zero
there.
A
That
also,
you
know,
have
parts
of
the
data
that's
distributed
so
that
we
can
send
our
analysis
drops
to
the
grid
and
analyze
the
the
the
data
where
they're
actually
located,
and
there
are
even
more
smaller
batch
farms,
so-called
tier
threes
they're,
then
local
and
limited
to
those
working
at
these,
for
instance
research
institutions
and
that,
and
all
these
sites
they're
often
already
managed
using
kubernetes,
and
at
cern
we
have
a
slightly
bigger
local
cluster.
So
we
have
a
batch
form.
A
That's
running
on
ht
condor,
a
high
throughput
computing
software
for
batch
computing,
and
there
we
have
230
000
cores
and
that
allows
up
to
150
000
jobs
to
run
simultaneously
and
that
results
in
about
or
a
maximum
of
1.4
million
jobs
that
are
completed
per
day
now,
the
these
grid
sites
they
are
largely
there
for
the
you,
know
large-scale
processing
and
most
of
these
steps
already
automated.
So
you
know
from
actually
reconstructing
the
data
that
we've
taken
and
also
having
the
associated
simulation
that
we
compare
the
data
to
a
process.
A
This
is
all
automated,
but
the
part
that
is
still
pretty
manual
is
actually
the
so-called
high-level
physics
analysis.
So
this
is
some
an
analysis
that
tries
to
address
a
specific
physics
question.
So,
for
instance,
when
we
were
searching
for
the
higgs
boson,
there
were
several
analysis
trying
to
find
the
higgs
boson.
A
Eventually
it
worked
out,
but
this
is
something
that
is
a
somewhat
manual
process
at
this
point,
because
it
needs
a
lot
of
very
detailed
inputs
that
come
that
are
provided
from
several
groups
and
individuals
from
within
the
collaboration,
so
so,
for
instance,
calibration
constants
and
factors
and
corrections,
a
certain
uncertainties,
etc.
They
are
all
provided
by
the
group
and
one
has
to
collect
them
in
order,
then
to
provide
or
perform
the
final
analysis.
A
So
this
is
pretty
complicated
and
therefore
challenging
task
to
achieve,
and
we
can
actually,
you
know,
keep
track
of
all
this
thanks
to
version
control,
so
we're
using
git,
and
so
we
can
capture
the
implementation
and
to
execute
these
jobs.
We
can
also
make
use
of
software
images
so
effectively
docker
containers,
in
this
case
the
challenging
part,
is
now
once
you've
captured
the
software.
A
So
you
have
the
software
under
version
control
and
you
then
build
a
container,
for
instance,
using
continuous
integration,
we're
using
github,
mostly
for
that,
so
you
have
all
the
dependencies
included
in
your
container.
Then
you
still
need
to
know
how
you
actually
execute
this
container
or
this
image.
So
we
also
need
to
capture
the
commands.
That's
you
know
also
doable,
but
the
big
challenge
lies
in
actually
capturing
the
full
workflow.
A
So
how
do
we
connect
the
individual
analysis,
steps
and
for
those
there
are
several
tools
under
investigation
and
some
of
them
are
already
used
by
smaller
groups.
I
just
want
to
point
out
a
few
here,
so
that's,
for
instance,
a
project
at
cern,
that's
called
rihanna,
that's
a
common
workflow
language
implementation.
It
has
a
focus,
but
it's
also
possible
to
use
it
for
other
things.
A
Then
there's
the
adage
which
really
focused
on
on
only
and
then
but
they're,
also
tools
that
are
very
you
know,
general
purpose,
for
instance
luigi
that
has
originally
been
developed
by
spotify
and
that's
also
being
used
by
a
few
groups,
and
I
asked
myself
the
question:
can
we
actually
do
this
kind
of
high
energy
physics
workflow
processing
in
a
cloud
native
way?
And
for
that
for
workflows
and
kubernetes?
A
I
found
argo
workflows
and
this
seems
to
be
a
really
nice
tool
to
use
so
now
before
I
actually
go
into
implementing
a
workflow
from
my
hand
of
high
energy
physics
in
argo.
I
first
wanted
to
talk
to
you
about
my
cluster,
so
at
cern
we
can
actually
provision
clusters
on-prem
with
openstack,
and
then
we
have
some
additional
plugins
that
are
somewhat
certain
specific,
so
or
high-energy
physics,
specific.
So
there's
a
certain
vm
file
system
so
that
we
can
efficiently
cache
the
software,
which
is,
you
know,
quite
a
lot.
A
So
it's
actually
gigabytes
of
software
that
we
need
to
use
for
and
for
the
block
storage
we
have
file
system.
So
that's
that's
in
particular,
for
the
big
data
sets
there
for
which
we
use
eos.
My
cluster
itself
here
is
actually
pretty
small.
It
has
four
nodes
with
four
cores
each
and
these
have
eight
gigabytes
of
ram.
I
have
some
object,
storage
and
also
some
block
storage.
So
as
three
and
seven
s
and
300
gigabytes
are
sufficient
to
store,
for
instance,
the
output.
A
A
A
Now,
let's
actually
see
how
you
know
a
typical
workflow
would
look
like
when,
when,
when
using
argo
and-
and
here
this
particular
example-
is
about
searching
for
a
new
signal
in
the
data
triggered
the
workflow-
you
can
see
it
here
is
already
running.
There's
a
blue
spinning
wheel.
That
shows
that
this
is
the
currently
active
step
or
the
one
that's
not
finished.
The
yellow
ones
are
where
the
container
is
currently
provisioning.
A
We're
preparing
a
deer
for
the
you
know
to
get
started
for
the
output
and
then
the
workflow
splits
into
several
steps
that
are
related
to
the
different
processes
that
we're
analyzing
here,
and
you
can
see
that
we
immediately
scatter
to
a
large
number
of
jobs
here.
So
that's
a
nice
feature
of
argo
that
you
don't
have
to
determine
how
many
jobs
you
will
have
in
the
end,
but
you
can
dynamically
generate
this
and
also
you
can
directly
observe
what
is
happening
which
part
of
the
workflow
is
currently
running.
A
You
know
what
is
the
status
of
each
of
the
steps
and
you
can
also
directly
observe
the
logs,
so
these
are
streamed
directly
into
the
user
interface.
You
can
see.
There's
lots
of
shell
scripting
here
the
last
command
was
a
python
script
and
this
workflow
is
already
done
and
after
this
highly
parallel
step,
that's
usually
the
one
that's
computational
most
intensive.
A
We
can
gather
our
input,
so
we
have
a
couple
of
merge
steps.
Then
we
can
again
scatter
out
to
steps
where
we,
for
instance,
have
to
take
into
account
systematic
uncertainties
that
use
the
same
inputs
but
run
a
different
variation.
So
that's
what
something
you
can
see
at
the
later
stages
and
then
at
the
very
end,
we
it
all
boils
down
to
a
single
step
that
provides
us
with
a
final
result.
A
That's
the
one
that
we're
looking
at
here
and
you
can
again
see
the
logs
this
time
they're
a
bit
more
interesting
than
before.
So
you
can
see,
there's
a
fit
done
to
the
data
different
processes
here
investigated
and
then
at
the
end
we
actually
get
the
final
result,
which
is
in
this
case
a
plot
called
postfit.pdf.
A
I
cut
it
down
from
four
minutes
to
two
minutes
only,
and
I
told
you
at
the
very
end
that
there's
a
plot
that
is
produced
by
a
fit
in
actually
in
the
end
and
we're
searching
for
new
physics
here
and
what
you
can
see
on
the
left
hand
side
in
in
this
plot
in
the
different
colors,
the
different
processes
that
actually
you
know,
would
end
up
in
this
particular
signal
region.
Where
we're
searching
for
the
signal
and
in
the
very
center
in
dark
green.
A
You
see
something
that's
labeled
as
a
signal,
and
this
is
what
a
potential
new
physics
signal
would
look
like,
and
the
black
markers
are
actually
the
data,
so
you
can
see
already
pre-fit
so
before
any
adjustment.
The
simulation
that
we
produced
matches
nicely
with
the
data
and
now
the
question
that
we're
asking
us
in
a
typical
physics
analysis.
Is
there
any
new
physics
signal
right,
so
we
optimized
so
that
we
can
in
simulation,
actually
see
the
signal.
A
Now
we
have
to
use
the
data
to
test
if
there
is
a
new
physics
signal,
so
we're
running
this
fit
that
allows
the
the
individual
components
to
adjust
to
the
data.
So
the
black
markers
won't
work,
but
everything
else
is
allowed
to
move
and
on
the
right
hand,
side
you
actually
see
that
post
fit
so
after
the
fit.
The
data
do
not
support
the
presence
of
the
signal.
A
Now
let
me
take
a
step
back
and
look
again
at
the
example
workflow
here.
So
that's
a
screenshot
from
the
demo
that
we
just
saw-
and
I
should
point
out
that
you
know
this
is
really
just
a
demo-
a
realistic
physics
analysis
much
more
complex.
Usually
when
I
run
you
know
the
first
step
that
is
here
delightfully
parallel,
as
you
can
see,
so
that
has
the
large
amounts
of
parallel
jobs.
A
What's
what's
really
nice
here
about
argos
that
I
actually
could
scatter
out
dynamically,
so
I
could
create
jobs
I
could
define
at
the
very
beginning.
I
want
you,
know
five
jobs
for
this
particular
data
set.
I
want
a
certain
other
number
for
for
another
step
and
argo
does
this
and
then
at
the
end
I
can
collect
the
information
again
and,
as
I
mentioned
in
the
video
demo,
there
can
be
later
parallel
steps
that
I
can
pick
up
again,
so
this
works
really
nicely
now.
A
What
isn't
so
nice
is
that
life's
a
bit
unfair
and
realistic
physics,
analysis
workflow
can
actually
not
be
run
on
my
cluster
right,
so
I
mean
my
cluster
has
16
cores
and
the
hd
condor
batch
farm.
It
has
230
000,
cores
and
970
terabytes
of
memory.
So
the
question
that
I
asked
myself:
if,
if
I
want
to
be
able
to
run
realistic
cloud
native
workflows
with
argo,
can
I
actually
somehow
make
use
of
these
cars
in
hd
condor?
A
And
I
you
know
because
with
the
16
cores
I
said
I
I
won't
get
very
far,
and
I
found
the
solution
to
that
and
that's
what
I
call
here,
the
hd
condor
operator.
So
the
idea
is
to
introduce
an
htc
job
custom
resource
definition.
You
can
see
the
definition
on
the
left
hand
side
here
so
there's
an
api
version.
A
The
kind
is
the
custom
resource
definition
and
then
you
know
the
name
is
htc
job
and
the
idea
of
this
job
is
that
it
should
mimic
a
classical
kubernetes
job,
so
it
needs
to
somehow
reflect
running,
failed
and
succeeded
status.
I
can
do
that
via
status
properties
of
this
resource
and
in
addition,
I
want
to
be
able
to
understand
what's
happening
inside
the
hd
conda
batch
cluster.
A
So
I
added
additional
fields
such
as
the
job
ids
here
that
reflect
the
actual
job
ids
within
the
hd
conda
cluster,
so
that
I
can
also
you
know,
monitor
and
interact
by
using
these
job
ids,
and
this
is
actually
not
work
only
by
me,
but
this
has
largely
been
done
by
tatters.
Barack
is
a
bioinformatics
student
at
vilnius
university
who
interned
with
me
at
cern
earlier
this
year.
A
Now,
let's
get
back
into
let's
get
down
into
the
implementation.
So
when
you
have
a
custom
resource
definition,
you
need
to
do
something
that
acts
on
it
and
the
operator.
So
in
kubernetes,
that's
done
by
an
operator
so
the
and
we
looked
around
a
bit.
What
kind
of
operators
we
could
use.
We
looked
into
metacontroller
and
then
we
found
the
operator
sdk,
and
that
makes
it
really
easy
to
get
a
kubernetes
operator
implemented
and
running
in
the
first
place.
So
we
went
with
that
now.
A
This
operator
needs
to
know
about
hd
conda
in
one
way
or
the
other,
so
I
built
a
docker
container
that
contains
the
hd
condor
client
and
in
addition,
we
need
to
authenticate
to
this
cluster
so
to
hd.
Condor
and
that's
done
via
kerberos,
and
I
can
create
a
kerberos
token
via
the
secrets
that
I
can
store
in
my
cluster.
A
The
hd
con
operator
itself
is
actually
installed
into
this
container
and
in
addition,
what
this
operator
needs
to
know
it
needs
to
translate
the
information
from
the
drop
spec
into
something
that
can
be
executed
on
hd
condor.
So
here
in
particular,
the
jobs
are
not
executed
using
a
docker
when
running
on
hd
condor,
but
they're
executed,
using
singularity
so
there's
a
translation
step
that
translates
the
image
spec
and
the
comma
the
the
script
into
something
that
is
then
effectively
a
singularity
command
with
a
script
attached
to
it.
A
Then
these
batch
drops
on
on
hd,
condor
side
will
be
executed
and,
in
addition
to
having
the
api
implemented
here,
I
also
have,
at
the
very
end
of
my
job
web
hook
via
cloud
events
that
actually
tell
my
operator
that
a
job
is
done
and
which
one
is
the
good
job.
That
is
done
in
particular,
as
well
as
this
exit
code,
so
that
I
can
reduce
the
number
of
api
queries
from
the
htc
operator
to
the
batch
cluster.
A
A
So
now
what
I
need
to
do,
in
addition
is,
I
need
to
define
some
success,
condition
that
determines
whether
my
jobs
are
all
done
or
if
any
of
them
failed.
So
I
define
a
success,
condition
that
the
number
of
so
the
status
field
in
my
custom
resource
definition
needs
to
be
equal
to
the
number
of
jobs
that
I
started
with
and
if
any
of
the
jobs
failed,
then
also
this
job
is
supposed
to
fail.
So
that
means
now.
I
can
actually
move
the
long
running
steps
into
hd
condor
directly
from
my
argo
workflow.
A
So
what
I'm
going
to
do
now
is
I'm
going
to
run
the
same
workflow
as
before,
but
moving
this
first
step
to
hd
condor
and
I'm
going
to
speed
this
up
a
bit.
But
let's
see
how
this
goes.
Okay,
so
I'm
submitting
my
job
from
the
command
line.
You
can
see
it's
running
the
prepare
deer
step
as
before
in
the
ui
and
now
there's
actually
a
difference
here.
A
You
can
see
two
spinning
wheels
after
each
other,
and
this
step
is
actually
now
running
on
the
batch,
so
we're
creating
inside
the
cluster
the
custom
resource
and
therefore
the
operator
will
submit
drops
to
the
batch
system,
and
you
can
see
that
here
on
the
left-hand
side
is
already
happening
and
also
on
the
right-hand
side.
There
they're
currently
active
zero
drop
id
is
none
and
as
soon
as
the
drops
are
queued,
you
actually
see
active
four
and
you
see
the
individual
drop
ids,
which
you
can
compare
to
the
left
hand
side.
A
Now,
the
inside
argo
I
defined
a
failure,
condition
a
success
condition.
So
all
the
jobs
need
to
be
complete
successfully
so
for
things
to
move
on,
and
I
can
actually
investigate
this
in
more
detail.
If
I
look
at
my
hd
condor
operator
within
argo,
so
here
I'm
looking
at
the
argo
cd
ui,
where
I
also
see
the
logs
of
the
other
parts
and
you
can,
for
instance,
here
see
the
job
definition.
I
submitted
a
job
to
big
bird
10,
so
this
is
the
manifest
for
the
job
submission
you
can
see.
A
A
Again
now
we
just
saw
briefly
the
grafana
monitoring,
but
more
interesting
here
is
now
actually
that
also
the
first
jobs
are
succeeding
in
in
for
the
cr
that
I
was
monitoring
and
again
we
see
that
in
the
logs
and
grafana
in
green
here,
the
drops
that
are
running
in
yellow
is
a
bit
behind
the
ones
that
have
been
scheduled.
You
can
actually
see
that
all
the
jobs
are
running
already
now
we're
already
at
two
succeeded
jobs
and
two
active
jobs,
and
on
the
left-hand
side,
you
can
see
the
total
15
jobs.
A
We
see
that
all
the
data
have
been
transferred
and
also
there's
no
success
statement
for
the
custom
resource,
so
argo
continues
its
job,
but
the
custom
resource
stays
in
place
and
all
the
other
steps
or
as
before,
pretty
fast.
So
there
wasn't
much
to
be
done.
The
images
had
been
pulled
already
from
the
previous
demo.
So
now
we
are
already
at
the
last
step
where
we
make
the
plot,
and
that
then
concludes
the
workflow.
A
Now,
at
the
very
end,
there's
I
said
the
custom
resources
remain.
So
if
I
delete
the
workflow
also,
the
resources
that
were
created
will
be
deleted,
so
you
will
now
see
on
the
top
left
that
this
will
actually
remove
the
drops
from
the
queue
I
left
them
there
in
case
I
wanted
to
transfer
some
data,
but
I've
done
that
automatically.
A
A
All
right,
so
this
worked
really
well.
As
I
said,
I
sped
up
things
a
bit,
so
this
in
total
would
have
taken
something
like
30
minutes
just
because
I'm
not
the
only
one
running
jobs
inside
the
the
cluster
so
and
hd
condor.
So
such
a
cluster
is
not
really
made
for
short
running
jobs,
but
the
short
queue
is
actually
something
like
15
minutes
and
the
default
is
something
like
eight
hours.
So
it's
it's.
Okay,
that
things
take
some
time
to
to
to
be
scared
to
actually
run.
A
But
if
I
run
a
real
physics
analysis,
you
know
that
would
be
perfectly
fine
with
me,
but
just
to
to
close
out.
I
I
think
you
know
what
I
managed
to
show.
You
is
here
that
I
leveraged
legacy
infrastructure,
so
you
know
kubernetes
agnostic,
computing
infrastructure
by
means
of
a
kubernetes
custom
resource
definition
that
is
combined
with
an
operator,
and
the
operator
concept
is
extremely
powerful
for
this
purpose
and
also
I've.
I
found
that
cloud
native
high
energy
physics
workflows
are
possible
using
argos.
A
So
that's
really
nice,
and
now
there
are
a
couple
of
steps
still
to
be
done,
so
the
htc
operator
actually
needs
to
be
made
a
bit
more
flexible,
because
when
you
choose
cues
in
hd
condor,
you
can
also
directly
translate,
for
instance,
compute
requirements.
So
how
long
should
your
job
run
et,
cetera
and
so
you're
going
to
need
to
translate?
A
So
that's
on
the
roadmap,
but
actually,
if
you
want
to
see
something
more
on
this
topic
there,
there
is
a
presentation
by
alessandra,
forte
and
lucas
heinrich
also
at
cubicon
here
who
actually
talk
about
how
they
have
their
own
grid
site
on
the
on
the
laptop.
So
I
could,
in
principle,
run
my
workflow
in
kubernetes,
then
send
it
to
the
grid
where
the
grid
would
be
their
local
laptop.
So
that's
fun.
Isn't
it
all
right?
A
A
A
So
thanks
again
for
for
joining,
I
got
a
lot
of
questions
in
the
chat.
I
already
tried
to
answer
a
couple
of
them,
I'll
just
go
through
them
in
the
following,
probably
the
those
that
are,
you
know
generally
most
interesting
for
other
things.
You
can,
you
know,
chat
with
me
in
the
cncf
slack
afterwards.
You
can
find
me
there
I'll,
be
monitoring
the
the
machine
learning
channel.
A
So
I
mean
that
was
a
very
general
question
on
on.
You
know
general
information
on
cern.
We
actually
have
a
our
own
top
level
domain,
so
you
can
just
go
to
home.cern
cern
and
from
there
you
find
information
in
particular
on
the
the
cern
experiments,
so
the
large
hadron
collider,
and
also
the
experiments
at
the
large
iron
collider,
but
also
smaller
experiments
that
are
taking
place.
A
That's
then,
there
was
another
question
on
how
many
kubernetes
clusters
are
being
used
at
cern
and
how
many
nodes
are
in
the
biggest
cluster.
I
actually
don't
have
a
concrete
answer.
To
that
I
mean
I.
I
know
that
there
are
several
hundred
clusters,
but
I
think
that
most
of
them
actually
rather
small-
and
I
know
that
at
least
one
of
the
clusters
has
around
1
000
cores,
so
you
usually
have
you
know
it
depends
on
how
these
are
set
up
some
some.
A
A
A
So
that's
a
technology
that's
been
developed
at
cern,
I
think
it's
generally
available
and
for
long
term
preservation.
We
rely
on
tapes
a
lot
so
for
data
that
aren't
actively
used.
They
actually
copied
over
to
tape,
and
then
you
know
archived
and
if
you
want
to
run
on
them
this,
what
happened
to
me
and
the
other
day,
then
it
can
easily
take
a
few
days
until
they're
actually
staged
out
to
disc,
and
this
archiving
does
not
only
happen
at
some,
but
it
also
happens
on
the
different
grid
sides.
A
A
So
in
general
I
would
say
that
we
do
not
really
need
dedicated
tools,
so
we
could
actually
work
with
the
existing
tools
but
yeah.
There
are
a
lot
of
there's
a
lot
of
custom,
tooling
we're
using
also
rather
old
systems.
So
you
know
we're
always
a
couple
of
years
behind
with
the
latest
greatest
software
etc.
So
you
know
we
can
work
around
that
with
containers.
But
still
you
know
that's
the
the
the
custom
tooling.
A
Then
you
know
we,
you
have
to
do
authentication,
so
we
we
have
certificates.
Then
we
use
kerberos
in
other
places
and
all
that
all
that-
and
so
I'm
say
one
of
the
few
people
actually
trying
to
you
know
make
these
these
energy
physics
workflows
accessible
for
people
within
the
community,
and
for
that
I
think
it's
really
important
to
give
them
a
really
good
experience,
so
to
make
it
as
easy
as
possible
for
them
to.
A
You
know,
get
started
with,
let's
say
cloud
native
or
automated
workflows
and
also
actually
stay
as
close
as
possible
to
let's
say
some
some
industry
or
open
source
solution
that
can
be
used
because
you
know
otherwise
I
would
have
to
do
all
the
maintenance
and
that's
probably
not
what
I
want
to
do
but
yeah.
So
the
custom
tooling
is
mostly
just
around.
For
instance
the
scatter
gather
paradigm.
A
So
we
you
know
we
have
a
couple
of
data
sets,
but
a
data
set
actually
contains
a
couple
of
a
thousand
files,
and
then
you
want
to
parallelize
this.
In
a
in
a
dynamic
way,
so
and
then
afterwards
you
wanna
collect
this
again,
but
you
don't
want
to
collect
all
the
files
in
one
step.
But
you
want
to
do
this
in
some
kind
of
a
cascade
and
that's
something:
that's
maybe
a
bit
more
have
specific.
A
I
just
went
with
argo
because
I
had
you
know.
I
found
it
a
few
years
back
and-
and
I
was
actually
curious
like-
could
we
use
it?
I
mentioned
in
the
presentation.
There
are
other
tools,
for
instance,
luigi
and
probably
airflow
is
an
equally
good
solution.
It
just
needs
someone
to
actually
give
this
a
try
and
see
if
that
works
for
them.
A
So
there
are
a
couple
of
other
questions,
so
I
mean
at
that.
So
there's
a
question
whether
we
actually
now
completely
switched
to
kubernetes
and
I'm
actually
actually
have
to
say
we
were
only
really
getting
started
with
kubernetes.
I
mean
that
our
id
department
is,
you
know,
heavily
invested
in
in
kubernetes,
but
let's
say
all
the
jobs
that
we're
running
are
still
running
on
hd,
condor
or
slurm
or
other
platforms
where
we
can
execute
containers.
A
Okay,
there,
there
are
a
com,
a
couple
of
other
questions
on,
for
instance,
this
k,
subs
plug-in
for
argo,
cd
and
and
other
links
so
hit
me
up
in
the
chat
I
can
post
them
there.
I
don't
think
I
will
have
time
for
that.
Maybe
I
take
like
one
last
question
so
so
there
was
a
question
like
you
know:
what
is
the?
A
What
needs
to
be
improved
for
kubernetes,
to
make
it
easier
for
for,
for
cern
or
for
high
energy
physics
in
general-
and
I
said
you
know-
probably
it's
a
multi-tenancy
and
I
I
would
say
you
know
the
the
way
that
we
do.
Authentication
and
authorization
is
also
currently
changing.
So
so
in
principle.
Of
course,
each
user
should
be
able
to
create
jobs
in
a
cluster
but,
for
instance,
not
delete
some
other
users
job,
and
so
so
in
principle,
each
user
needs
to
have
their
own
dedicated
namespace.
A
They
need
to
be
authenticated,
for
instance,
via
oauth,
in
one
way
or
the
other,
but
let's
say
most
of
the
works
actually
happening
in
the
terminal.
So
this
needs
to
be
linked
and
that's
currently
giving
me
some
kind
of
headache,
even
though
I
mean
there
are
solutions,
but
this
needs
to
be
figured
out
all
right,
so
I
see
we're
out
of
time.
So
thank
you
again
very
much
for
for
joining
this
presentation.