►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)
Keynote: CERN Experiences with Multi-Cloud Federated Kubernetes - Ricardo Rocha, Staff Member, CERN & Clenimar Filemon, Software Engineer, Federal University of Campina Grande
Using public cloud resources to cover for peak workloads is a practical and economical alternative to over provisioning on-premise resources. This is the case in environments like CERN where its large internal computing infrastructure is usually big enough but where periods prior to big international conferences or large event reconstruction campaigns see a significant spike in the amount of workloads submitted. We will describe early experiences relying on Kubernetes federations to expand the available capacity to external clouds, while still offering a single entry point to our users - using GKE, AKS, Amazon and the Oracle cloud. We will cover some issues we had in some of the clouds (mainly networking) and how we solved them. We will show how they get integrated with our main batch system, and how workloads running on external resources access their corresponding datasets.
"About Ricardo
Ricardo is a software engineer at CERN currently part of the CERN cloud team, focusing primarily on networking and container based deployments. Previously he helped develop and deploy several components of the Worldwide LHC Computing Grid, a network of ~200 collaborating sites around the world helping to analyze the Large Hadron Collider data. Ricardo has presented his and his teams work in different international conferences - Computing for High Energy Physics (CHEP), IEEE NSS/MIC, IEEE MSST, DockerCon and multiple OpenStack summits.
About Clenimar
Clenimar is an MSc student at the Federal University of Campina Grande, Brazil. He is currently researching on distributed, secure workloads in the cloud (using OpenStack and Kubernetes). He spent last summer working at CERN as a Cloud Infrastructure Intern, where he could investigate on how Kubernetes federations could match some CERN use cases, especially regarding the CERN batch service.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
So
today
we'll
talk
about
a
bit
about
CERN
and
what
why
we
have
heavy
needs
and
computing,
but
also
how
we've
been
using
kubernetes
to
try
to
expand
our
resources
in-house
so
a
bit
about
CERN.
So
CERN
is
the
European
Organization
for
Nuclear
Research.
It
was
founded
in
1954
with
its
main
goal
being
fundamental
science
and
we
try.
Over
these
decades,
we've
been
trying
to
answer
some
of
the
big
questions
in
science.
A
Some
are
like
what
is
96%
of
the
universe
made
of
we
only
really
understand
4%
and
we've
been
trying
to
see
what
is
dark,
energy
and
dark
matter.
What
was
the
state
of
the
matter
just
after
the
Big
Bang,
something
called
the
quark-gluon
plasma,
and
why
isn't
there
any
engine
matter
in
the
universe
we
know
in
theoretically
we
should
have
the
same
amounts
of
matter
and
antimatter,
so
we
keep
searching
why
we
don't
see,
and
yet
an
antimatter
around
so
CERN
is
located
in
Geneva
Switzerland,
it's
actually
across
the
French
Swiss
border.
A
So
you
can
see
the
white
dotted
line
there.
It's
the
the
border
between
France
and
Switzerland,
it's
very
close
to
Lake
Geneva.
You
can
see
there
and
very
close
to
the
European
Northern
ops
also,
so
you
can
see
the
back
on
the
background
there,
the
European
organ,
obviously
the
Mon
blob
in
the
in
the
middle,
which
makes
for
a
really
nice
background
in
our
cafeteria,
so
to
have
an
idea
of
the
size.
The
party
in
the
middle.
There
is
the
runway
of
the
Geneva
Airport.
We
have
two
sides,
the
main
one
being
in
may.
A
These
two
beams
in
precise
places
where
we
built
massive
particle
physics
experiments.
You
can
see
their
names
they're,
the
four
main
ones,
CMS
and
Atlas,
which
main
goal
is
to
was
to
find
the
Higgs
boson
and
to
understand
better
its
characteristics.
They
do
a
lot
of
other
physics
too,
and
then
we
have
Alice
and
LHC
B.
So
this
is
these
are
our
main
machines
right
now.
This
is
a
view
of
the
accelerator
itself,
the
Large
Hadron
Collider,
it's
actually
a
100
meters
underground.
You
can
see
the
magnets
here.
A
These
are
superconducting
magnets,
so
that
we
can
accelerate
them
to
these
higher
energies,
which
means
they're.
The
accelerator
is
also
very
cold,
so
it's
a
at
1.9
Kelvin,
which
is
very
close
to
the
absolute
zero,
something
like
minus
271
degrees.
Celsius.
You
can
see
also
that
we
can
walk
around
if
we
need
to
do
interventions.
Of
course,
when
it's
operating
the
access
is
quite
restricted
and
we
can
just
go.
But
if
you
pass
by
CERN
or
close
by
in
a
time
we
have
a
technical
shutdown
or
an
upgrade,
you
can
actually
go
down
and
visit.
A
We
organize
regular
visits
for
it.
Now
the
collisions
themselves
happen
in
caverns,
also
on
the
ground,
where
we
have
instruments
like
this
one.
This
is
the
compact
muon
solenoid,
the
CMS
detector.
It's
not
very
compact.
It's
weighs
14,000
tons
and
it's
20
meters
long
than
50
meters
wide.
So
it's
quite
a
big
machine.
You
can
see
the
people
there
to
have
an
idea
of
the
scale
once
it's
operating.
We
close
it
and
we
do
the
collisions
there.
A
So
you
can
see
all
these
layers
of
detectors
that
will
try
to
track
the
energies
and
the
particles
being
generated,
it's
kind
of
like
a
big
camera
for
very,
very
small
mass.
So
it's
our
modern
cameras
for
for
doing
physics.
We
have
other
experiments
on
sites,
apart
from
the
ones
related
to
the
Large
Hadron
Collider.
So
here
I
have
a
picture
of
the
antimatter
Factory,
so
we
keep
our
own
antimatter
Factory
just
down
the
road
from
the
office,
so
as
you
do
and
it's
actually
across
the
restaurant
also.
A
So
here
we
try
to
create
antimatter
and
to
understand
better
its
characteristics.
The
other
experiment,
I
pull
here,
is
the
AMS
experiment.
So
in
this
case
it's
not
a
reaction.
At
CERN
it's
in
the
International
Space
Station,
so
it's
it
was
sent
there
along
the
penultimate
space
shuttle
mission
and
we
had
the
crew
of
this
mission
coming
to
CERN,
explaining
about
life
in
space
and
how
it
was
to
install
it.
A
A
Now
why
we
are
here:
it's
because
we
do
a
lot
of
computing
and
we
have
a
big
computing
requirements.
You
can
see
here
all
starts
in
the
disaffected.
Where
we
do
the
collisions
in
total,
we
can
do
something
like
a
billion
collisions
per
second.
The
rate
of
data
generation
is
is
huge,
so
we
can
generate
something
like
a
petabyte,
a
second
which
is
not
something
we
can
manage
at
all.
A
So
what
we
do
is
we
have
two
main
filters
directly
after
the
collisions
inside
the
detector
or
very
close
to
the
detector,
where
we
do
the
first
level
filtering,
we
can
reduce
with
these
two
triggers.
We
can
reduce
to
something
like
1
to
10
gigabytes,
a
second,
which
is
something
we
can
manage
right
now.
So
we
store
all
of
this
into
our
data
centers.
Actually,
we
store
it
on
tape
for
for
archival
and
from
then
on.
We
start
all
the
reprocessing
and
analyzing
of
all
the
data.
A
So
for
this
we
have
a
big
computer
center
in
Mara
in
Geneva.
This
is
the
first
floor
has
another
ground
floor.
We
actually
have
a
second
data
center
in
in
close
to
Budapest,
linked
to
this
one.
It's
a
historical
building.
It's
been
there
for
over
or
around
50
years,
and
if
you
visit,
you
can
see
the
pictures
of
the
evolution
of
computing
itself
by
just
looking
at
how
this
data
center
has
changed
from
mainframes
people
walking
around
to
now.
A
A
We
do
mostly
virtualized
resources
right
now,
so
it's
over.
Ten
thousand
hypervisors,
we
store
something
like
250
petabytes
of
data.
Right
now
we
add
60
to
70
bytes.
Every
day.
Every
year,
with
the
accelerator
running,
we
have
over
4,000
projects,
3,000
users
and
to
be
the
main
thing
here.
For
for
this
talk
is
that
we
already
have
more
than
200
kubernetes
clusters.
These
clusters
spent
from
small
clusters
like
10
nodes
to
a
couple
of
hundred
nodes,
the
biggest
ones
we
have
are
around
a
thousand
nodes,
and
we
do
many
things
continuous
integration
and
deployment.
A
A
And
then
we
linked
a
couple
of
hundred
sites
from
small
computer
centers
and
universities
that
help
us
and
provide
us
with
their
computing
power
to
do
the
reconstruction,
the
calibration
of
the
detector
simulation
data
and
the
analysis.
At
any
moment,
we
run
something
like
four
hundred
thousand
jobs
in
parallel
in
this
infrastructure
and
we
provide
over
seven
hundred
thousand
course
now.
The
end
result
of
all
of
this.
All
of
this
infrastructure,
complex,
accelerators
computing,
is
to
reach
these
kind
of
plots.
A
So
on
the
top
left,
we
see
the
Higgs
events
so
when
it
was
first
found
in
our
detectors-
and
this
is
a
shot
from
the
CMS
detector
in
May
2012,
a
reconstruction
of
a
Higgs
event
in
the
CMS
detector
on
the
lower
right
side.
We
have
this
similar
event
from
the
Atlas
experiment.
Now
all
of
this
infrastructure-
all
we
want
to
see
is
this
red
spikes,
which
corresponds
to
a
new
physics
new
particle,
something
that
we
had
never
observed
before
so
did
it.
This
is
what
we
look
for.
B
So
let's
talk
kubernetes
and
Federation,
so
here
you
can
see
a
few
reasons
why
a
kubernetes
Federation
could
would
be
useful
to
us.
So,
first
of
all,
as
you
may
have
noticed,
a
certain
has
a
large
computing
infrastructure,
but
still
at
some
points
of
the
year
we
can
notice
a
load
spikes.
So
even
more
resources
are
needed,
especially
before
big
physics,
conferences
or
during
large
reconstruction
campaigns
from
the
experiments.
B
So
a
federation
would
allow
us
to
easily
add
more
resources
to
to
the
infrastructure
by
creating
new
clusters
that
they
could
be
also
in
external
clouds,
and
then
we
can
add
them
to
the
infrastructure
through
a
single
entry
point
and
then
to
supply
the
this
temporary
need.
Also,
super-nervous
has
well-defined
ways
to
do
monitoring,
lifecycle
and
alarms
it
may
it
makes
life
pretty
easier
and
it
also
got
a
lot
of
tools
that
integrates
seamlessly
with
the
infrastructure
like
primitives,
and
we
like
to
experiment
more
and
more
with
those
tools
and
by
providing
a
uniform
API.
B
It
allows
us
to
define
and
run
applications
inside
certain
and
using
the
same
definitions.
We
can
do
that
outside
CERN
because
we're
talking
about
the
same
API,
so
we're
talking
about
the
same
resources
and
objects.
So
we
prepared
to
use
cases
regarding
Federation
at
CERN.
For
you,
the
first
one
is
the
CERN
batch
system,
which
is
actually
it
actually.
B
It
actually
corresponds
to
80%
approximately
of
the
whole
computing
power
of
CERN,
and
it
actually
runs
the
workloads
of
two
that
runs
the
analysis
from
the
LHC
data
and
also
the
recast
analysis,
which
is
a
new
tool,
it's
as
based
on
native
concepts,
and
it
allows
physicists
to
to
run
reusable
analysis
and
physics
data.
So,
let's
start
with
the
the
batch
system,
it
is
based
on
HT,
Condor
and
CERN
has
many
decades
of
experience
with
that.
So
it
basically
works
like
this:
a
user
when
defining
a
new
workload,
they
advertise
requests.
B
So
they
let
the
system
know
the
amount
of
resources
that
the
specific
workload
needs
in
order
to
to
run
properly
and
then
on
the
other
side,
we
have
the
resources
that
are
actually
the
compute
nodes
that
also
advertise
their
capabilities
to
the
system.
So
they
can
tell
the
system
the
total
amount
of
resource
that
they
have.
Then
there's
a
collector
component
that
collects
those
advertisement.
We
call
them
class
ads
and
then
it
gives
them
to
a
negotiator
component
which
does
the
matchmaking
job.
B
It's
basically
binding
a
request
to
resources
that
are
able
to
supply
those
requests.
It's
basically
binding
workloads
to
a
set
of
nodes
that
are
capable
of
running
those
workloads.
It's
got
also
some
advanced
features
like
it
supports
fair
share
and
preemption,
so
we
can
have
a
priority.
For
example,
you
can
stop
a
low
priority
job
to
free
up
resources
to
a
more
important
one.
It
just
arrived
and
it's
as
I
mentioned
before
it
takes
approximately
80%
of
the
strength,
computing
capacity
and
most
of
it
running
virtualized.
B
So,
for
example,
it
would
be
good
in
the
future
to
move
bare
metal
for
example.
So
we
we
would
gain
some
performance
just
by
skipping
the
virtualization
bits
in
the
middle
of
it.
Also,
it's
important
to
notice
here
that
HT
countered
just
handles
the
computing
part.
So
for
networking
and
storage
it
actually
relies
on
certain
services
that
already
exists.
So
the
computing
part
of
it
is,
they
start
daemon,
which
is
represented
here,
and
it
runs
on
every
node
of
the
bad
system
and
it's
actually
the
responsible
for
running
the
workload
on
the
nodes.
B
We
continue
rised
it
and
once
you
have
it
this
container,
we
can
run
a
net
at
scale.
So
why
not
a
federation
to
try
it
right?
So
we
have
this
experiment.
This
was
actually
my
internship
project
last
year
when,
when
I
was
at
CERN-
and
we
have
this
host
cluster,
which
is
responsible
for
running
the
controller
manager
for
the
Federation
and
also
the
Federated
API
server
for
kubernetes,
then
we
deploy
the
control,
plane,
components
for
HT
Condor.
B
They
scheduler
the
collector
and
the
negotiator,
and
then
we
are
able
to
start
our
Federation
by
by
running
a
simple
command,
which
is
the
cube
fat
in
it.
Once
we
run
it,
we
have
a
federation
set
up
and
it
you
can
start
playing
with
it.
So
here
you
can
see
I'm
really
having
trouble
this
ok
here
you
can
see
at
the
top
right
corner
a
resource
definition.
It's
a
daemon
set
and
what
it
actually
does
it's
described.
I
will
start
the
containers.
B
So,
as
you
know
what
Iman
said,
the
kubernetes
control
plane,
you
make
sure
that
every
single
node
in
the
cluster
runs
the
replicas
fit
so
we're,
basically
turning
the
the
nodes
into
HD,
counter
nodes
that
are
able
to
run
actual
workloads,
actual
production
workloads,
which
is
pretty
cool
because
you
have
a
single
resource
and,
as
you
deploy
that
and
the
Federated
level
the
control
plane
to
make
sure
that
every
single
node
in
all
the
underlying
clusters
will
have
this
replica
running.
So
we
deployed
a
couple
clusters.
B
We
deployed
a
couple
at
cern,
but
we
also
went
to
external
clouds.
We
deployed
a
couple
clusters
and
these
systems,
and
we
also
experimented
with
gke,
AWS
and
azure
so
to
join
those
clusters.
We
just
need
to
run
a
single
command
again,
is
cupid
join
and
once
we
do
that
the
control
plane
to
make
sure
that
those
new
nodes
that
will
just
add
we
will
have
the
the
replica
of
the
daemon
running
too.
So
it's
free,
it's
pretty
cool.
B
You
can
imagine,
for
example,
that
cell
receives
a
new
hardware
and
then
you've
got
those
new
machines.
And
then
you
wonder,
how
can
I
integrate
that
with
the
production
infrastructure?
That's
pretty
tricky,
but
if
you
have
a
federation
you
you
just
need
to
deploy
a
cluster
with
those
new
resources
and
join
them
to
the
federation.
So
in
a
really
short
time
they
will
be
able
to
run
actual
production
workloads
and
help.
Cern
makes
new
physics.
A
Well,
I'll
take
for
the
second
use
case,
so
the
the
this
was
the
first
use
case
we
took,
which
is
taking
a
well-established
application
at
CERN
and
seeing
how
we
could
explore
kubernetes
to
to
allow
us
to
expand
our
data
center
to
the
outside.
So
yesterday
we
got
together
with
the
location
also,
and
we
were
playing
what
we
call
public
cloud
bingo,
just
adding
as
many
public
clouds
as
we
can
to
the
Federation
and
see
how
it
works.
So
we
got
we
got
to
7:00
yesterday,
so
it
was
a
record
now.
A
A
Now
the
cool
thing
here
is
that,
as
its
defined
as
a
workflow,
the
system
takes
each
step
and
defines
it
as
a
kubernetes
job,
and
we
started
by
doing
it
by
running
with
our
colleagues
in
a
single
cluster,
but
actually
we
saw
that
we
could
also
explore
Federation
for
for
this
use
case,
and
this
is
actually
the
the
best
use
case.
We
have
right
now
for
Federation,
so
will
will
play
a
quick
demo
of
how
this
works.
A
So
there
we
go
so
here
we
have
the
how
the
system
works,
and
we
have
some
preset
examples
where
we
can
do
some
Higgs
analysis
dark
matter
supersymmetry.
So
in
this
case
we'll
choose
the
Higgs
analysis.
We
can
parameterize
the
analysis
with
some
input
data
and
how
we
want
to
see
the
output
and
this
registers
the
workflow
in
the
system,
so
the
workflow
will
start
almost
immediately.
The
first
step
of
the
word
flow
is
to
define
how
how
the
different
jobs
should
be
run,
and
this
is
running
on
the
host
cluster.
A
In
this
case,
we
can
see
here
that
we
have
the
Federation
control
plane
on
the
top,
and
then
we
have
three
clusters
that
are
registered
in
the
system
in
this
case
was
two
at
CERN
and
one
in
a
public
cloud
in
two
systems.
So
the
the
three
clusters
are
ready.
The
work
flow
is
starting
to
build
the
jobs,
and
eventually
we
can
see
how
the
work
flow
is.
We
have
the
physics
data
on
the
left
on
the
middle.
We
have
the
computation
of
using
the
standard
model
of
physics
and
on
the
right.
A
We
have
the
new
signal
that
we
want
to
see
if
it,
if
it's
new
physics
or
if
it
corresponds
to
two
existing
known
physics.
So
the
workflow
is
quite
complex.
It
has
a
lot
of
jobs.
We
wanted
initially
to
do
a
live
demo,
but
it
actually
takes
too
long
to
run,
but
one
cool
feature
is
that
we
can
also
explore
things
like
the
collection,
centralized
log
collection
to
give
feedback
to
the
user
immediately
of
how
their
jobs
are
going.
A
We
accelerate
the
analysis
a
bit
here,
many
many
times
so
that
we
can
play
it
live
here
or
not
life,
but
recorded,
and
then,
from
time
to
time
you
will
see
the
feedback
on
how
your
workflow
is
going,
which
is
a
cool
way
that
we
didn't
have
an
easy
way
to
do
this
before,
eventually
we'll
get
to
a
point
where
all
the
jobs
are
executed
and
we
get
our
job
finished
the
output.
In
this
case,
we
have
chosen
to
generate
a
plot,
as
I
mentioned
very
often
the
result
of
physics.