►
Description
Users at CERN, the largest particle physics laboratory in the world, have very diverse computing requirements. CERN's IT department provides two services for running their cloud-native workloads: Kubernetes (based on OpenStack Magnum) and OpenShift (based on OKD). This talk will outline why both services are offered, alongside their benefits and drawbacks. In addition, the deployment model of each service will be described: both services use many open-source CNCF projects (such as OpenPolicyAgent or ArgoCD) as well as custom-built projects for integrating with the rest of CERN's computing environment (which is not always cloud-native) : networking, DNS, storage and authentication. Finally, we will share lessons learned during years of operating these services at large scales with thousands of applications and users.
A
So
here
we
go
again:
I
must
say
when
I,
don't
I
work
on
kubernetes,
like
I,
think
a
couple
of
you
here
and
when
I
when
I
read
the
outline
of
this
talk,
I
must
say:
I,
absolutely
don't
care
about
kubernetes
I,
I,
study
Physics.
So
for
me,
CERN
is
the
Mecca
or
something
close
to
it.
So
when
I
was
particularly
excited
to
to
to
hear
what
Jack
our
friend,
Jack
and
engineer,
sir,
has
to
tell
us.
So
please
give
it
up
for
Jack.
A
B
So
first
a
little
bit
about
me,
so
I
studied
I.T
security
and
cloud
computing
at
Alta
University
in
Finland
and
in
my
free
time,
I
like
cycling
and
also
hiking
in
nature
and
Aya,
then
I'm
also
a
engineer
at
CERN,
which
is
actually
quite
nice,
because
this
is
the
landscape
of
CERN.
So
we
have
the
Lake
Geneva
here
and
at
the
tip
there
is
Geneva
you
can
see
in
the
background
we
have
the
Alps
close
by.
B
B
And
what
that
also
means
is
that
we
have
a
long
Legacy
of
running
web
services
at
CERN,
because
the
web
was
invented
there
and
most
of
these
web
services.
Well,
actually
all
of
them
are
running
in
this
data
center.
So
this
is
the
CERN
Data
Center.
So
basically,
everything
we
do
is
on
premise,
but
actually
80
of
the
workload
that's
running
in
this
data
center
is
just
what
we
call
physics
workload.
So
it's
just
running
simulations
computations,
analytics,
machine
learning,
etc,
etc.
B
But
more
and
more
of
our
users,
of
course,
also
want
modern
ways
of
deploying
applications,
doing
it
in
a
cloud
native
way,
and
we
as
discern
I.T
department,
offer
two
ways
to
do
that.
We
have
a
kubernetes
service
that
is
based
on
openshift,
Magnum
and
or
sorry
an
openstack
Magnum,
and
we
have
a
managed
kubernetes
service
that
is
based
on
okd,
and
we
frequently
get
the
question:
why
do
you
offer
both
like
I
mean
managing
like
doing
a
kubernetes
management
is
already
enough
of
a
hassle.
Why
do
you
duplicate
the
effort?
B
B
So
it's
basically
like
a
VM.
You
create
the
thing
we
give
you
an
SSH
key
and
you
are
root
on
that
thing.
You
can
do
whatever
you
like.
You
have
the
full
control.
If
you
want
to
install
packages,
or
in
the
case
of
kubernetes,
you
want
to
deploy
a
service
mesh
or
you
want
a
different
cni
or
whatever
you
can
go
ahead,
but
you
also
need
to
know
what
you're
doing
so.
It
requires
more
skill,
more
knowledge
from
the
user
and
further
on.
B
If
you
have
very
memory,
intensive
workloads
and
all
of
these
things
are
are
aspects
that
you
can
customize
when
you
have
your
own
cluster
and
finally,
one
of
the
best
benefits
that
we
also
see
is
that
with
a
dedicated
cluster,
you
can
really
decide
which
type
of
dependencies
you
want
to
take
on.
So,
let's
say,
for
example,
you
have
a
stateless
application
and
you
basically
just
want
HTTP
traffic
in
and
then
you're
connecting
to
a
database
somewhere
else.
B
On
the
other
hand,
the
benefit
of
a
managed
kubernetes
service,
where
you
only
have
access
to
the
API
and
not
the
underlying
infrastructure,
is
that
we
can
optimize
the
resource
user
usage
much
better.
This
is
also
what
Robin
mentioned
in
his
talk
earlier
today,
because
by
putting
multiple
different
tenants
on
the
same
cluster,
we
can
utilize
the
the
inefficiencies
between
the
different
workloads
much
better
or
we
can
kind
of
smoothen
out
traffic,
Peaks
and
and
bursts
in,
in
whichever
workload
you
have
in
general.
B
B
B
So
you
you
give
us
a
pod
and
we
just
run
it
for
you
and
that's
all
you
need
to
take
care
of,
and
we
take
care
of
handling
security
upgrades
for
the
for
the
kubernetes
infrastructure
for
the
underlying
VM
infrastructure.
For
anything
else.
That
needs
to
be
done
replacing
nodes
if
they
are
unhealthy.
All
of
these
things,
and
finally,
especially
with
with
okd
or
openshift,
you
also
get
a
nice
user-friendly
web
UI
and
I
really
want
I
want
to
to
to
highlight
how
important
this
is
and
how
many
of
our
users
for
for
this
is.
B
This
is
actually
their
main
entry
point
to
our
platform
because
yeah
we
as
experts,
we
know
how
to
use
cubectl
or
OC
or
Helm
or
whatever
is
our
the
command
line
tool
of
the
day.
But
users
are
really
happy
when
they
can
just
see
their
port.
That
is
running
and
they
can
click
on
it
and
they're
going
to
see
the
logs,
and
this
is
just
really
nicely
prepared
and
it
it's
a
very
smooth
entry
to
the
kubernetes
cloud
native
world.
B
Now,
at
CERN,
CERN
is
an
academic
institution,
a
research
institution,
and
that
means
our
users
also
have
a
lot
of
academic
freedom
to
put
it
lightly.
So
we
don't
have
a
golden
path
for
deploying
containers.
Some
people
are
just
using
the
the
web
dashboard
like
I
just
said,
others
are
using
Cube
CTL,
maybe
from
a
from
a
CI
CD
pipeline.
B
That
kind
of
restrict
how
much
you're
allowed
to
use,
because
we
don't
do
internal
accounting.
So
we
don't
account
for
how
many
resources
you're
using,
but
that
that's
why
we
need
a
a
restriction,
a
limit
on
by
default,
how
many
resources
you're
allowed
to
use.
Of
course,
people
are
allowed
to
request
more
resources
if
they
need
to,
but
by
default
we
have
low
quotas.
B
So,
let's
get
into
the
topic
of
how
people
can
get
a
kubernetes
clusterance
term,
so
this
is
a
service
that
is
managed
by
it
and,
as
I
already
mentioned,
before,
it's
based
on
openstack
Magnum
and
which
completely
automates
the
initial
provisioning
of
the
cluster
and
our
team
has
introduced
feature
toggles
so
that
you
can
enable
or
disable
very
common
integration.
So
this
is
things
like
that.
B
Then
you
run
a
cluster
create
command
and
you
specify
which
flavor
you
want
to
have.
You
just
need
to
specify
also
then
an
SSH
key,
so
that
you
can
access
your
nodes
and
finally,
you
can
specify
additional
labels,
so
in
this
case
I'm
enabling
the
monitoring
component
and
a
couple
of
minutes
later
after
running
this
command,
your
cluster
will
be
ready.
B
B
On
the
other
hand,
we're
also
using
okd,
so
that
is
the
community
edition
of
Red
Hats
openshift
container
platform,
which,
by
now,
since
a
couple
of
years,
is
really
the
foundation
of
the
web
services
infrastructure
at
Stern,
and
that
is
because
it
provides
a
multi-tenant,
highly
available
and
secure
base.
It's
it's
really
amazing.
B
If
you
compare
vanilla,
kubernetes
deployment
to
an
openshift
deployment,
how
much
you
get
out
of
the
box,
we
then
go
ahead
and
enhance
this
kind
of
out
of
the
box
okd
with
additional
features
that
are
either
requested
by
our
users
or
that
are
just
necessary
to
integrate
well
in
the
certain
Computing
environment.
So
these
are
things
like
registering
your
hostname
for
you,
the
Ingress
in
the
DNS
servers,
handling,
automatic
certificate,
renewal
and
requests
very
important
also
for
us
and
for
our
users
is
the
fact
that
we
provide
lots
of
different
storage
Integrations.
B
B
So
our
our
most
generic
and
kind
of
lowest
complexity
use
case
is
the
platform
as
a
service
which
currently
houses
around
1
400
different
projects,
and
it's
actually
our
largest
production
okd
cluster,
and
this
is
really
just
container
as
a
service,
so
users
just
put
pods
inside
or
well
deployments
to
be
precise
and
and
we
run
it
for
them
and
we
take
care
of
handling
all
of
the
infrastructure.
But
basically
the
user
sees
a
completely
normal
kubernetes
API.
B
B
Then
there's
also
the
the
web
us
cluster,
which
was
actually
our
first
okd
cluster,
which
is
at
the
moment
housing
around
4000
projects,
and
this
is
basically
just
a
shared
web
hosting
cluster.
So
it's
serving
files
from
the
EOS
file
system
that
I
that
I
mentioned
before,
but
at
the
same
time
it's
highly
multi-tenant.
B
And
finally,
there's
the
Drupal
use
case
where
our
colleagues
from
the
Drupal
team
have
developed
an
extremely
Advanced
operator
for
running
Drupal
websites,
because
Drupal
is
the
most
widely
used
content
management
system
at
CERN.
So
all
of
the
main
websites
are
currently
using
it,
and
this
Drupal
operator
provides
a
really
simple
interface
for
users
to
say:
I
want
a
Drupal
website
with
that
version
and
they
don't
need
to
take
care
of
anything
else.
And
then
the
operator
goes
to
the
cluster
or
is
running
inside
the
cluster
and
creates
deployments
as
necessary.
B
But
what
happens
behind
the
scenes
is
that
this
portal
actually
does
not
have
any
state
itself.
It
just
takes
the
permissions
that
the
user
has
and
creates
a
resource
in
the
cluster,
in
this
case
a
custom
resource
definition
or
a
custom
resource
with
the
parameters
that
you
put
in
here.
The
same
happens
for
the
Drupal
side.
So
again,
I'm
just
I
just
need
to
say
what
the
hostname
should
be
and
then
kind
of
which
version
of
Drupal
I
want
and
I
click
on,
create
and
behind
the
scenes.
Something
like
this
gets
put
into.
B
The
cluster
and
we
entirely
rely
on
the
role-based
access
control
that
kubernetes
already
implements
and
that
the
multi-tenant
okd
gives
us
out
of
the
box.
So
if
the
user
is
in
principle
to
allow
to
create
such
an
object
against
the
API,
then
our
web
services
portal
will
do
the
same
on
behalf
of
the
user,
and
you
can
see
that
the
same
Fields
kind
of
show
up
here
again.
So
we
can
see
the
site
URL.
B
We
can
see
that
the
the
Drupal
version
has
been
specified
and
then
we
can
see
what
the
the
operator
has
been
doing.
So
the
deployment
has
been
rolled
out
as
expected.
It
checked
if
the
any
database
migrations
need
to
be
applied
and
which
backups
are
available,
and
all
of
this
is
completely
automated.
B
I
also
want
to
mention
how
we
actually
manage
these
clusters.
So
we
have
these
four
clusters
that
I
showed
on
the
previous
slide
in
in
production
plus
additional
ones
in
staging
and
deaf
environments
and
contrary
to
to
the
current
Trend,
we
are
actually
treating
these
clusters
as
pets
because
they
are
stateful.
B
Furthermore,
we
have
we
have
implemented
this
architecture
that
each
cluster
that
we
create
is
completely
self-sufficient
and
isolated.
So
it
has
a
separate
openstack
project.
It
uses
separate
application
registrations.
Everything
is
isolated
from
each
other
and
we
don't,
for
example,
connect
them
with
different
Argo
CD
deployments,
or
something
like
this,
and
this
is
very
important
for
us,
so
that
when
we
make
a
mistake,
we
reduce
the
blast
radius
as
much
as
possible.
B
Finally,
also,
we
don't
really
have
a
need
to
like
other
people,
to
throw
clusters
away
and
create
new
ones
and
move
the
workloads,
because
okd4
makes
In-Place
cluster
upgrades
completely
seamless,
I,
don't
know
who
of
you
ever
tried
to
upgrade
an
openshift
cluster.
But
it's
really
amazing
openshift
by
itself
will
just
go
through
the
entire
cluster
and
update
each
component
that
it
needs
to
in
the
right
order
and
highly
available,
and
as
a
user,
you
don't
even
notice
that
it's
happening.
B
Overall,
the
deployment
then
looks
something
like
this,
so
at
the
at
the
bottom,
we
still
have
our
openstack
cloud.
Then
we
put
okd
on
top
of
it,
and
Argo
CD
is
kind
of
the
brain
of
the
entire
operation.
So
it's
managing
both
the
okd
configuration
because,
if
you're
not
familiar
with
openshift,
everything
in
openshift
is
managed
through
an
operator
itself.
B
So
if
you
want
to
change
the
configuration
of
an
API
server,
you
just
edit
the
API
service
crd
and
you
need
to
change
some
parameters
in
there
and
then
the
operator
will
go
ahead
and
actually
put
those
changes
onto
the
nodes
and
similarly
for
all
other
components,
such
as
image
registry,
English
controllers,
Etc
and,
in
addition
to
managing
okd
Argo
CD,
also
manages
all
of
the
other
things,
and
this
is
actually
just
a
small
sample
of
the
things
that
we
deploy,
such
as
Manila
and
cefs.
Integrations
for
the
storage
I
previously
mentioned
third
manager.
B
B
If
a
resource
is
not
in
the
state
that
we
declared
it
to,
then
argocd
will
just
make
it
that
stage,
and
then
this
is
very
nice,
because
The
Operators
either
our
own
operators
or
The
okd
Operators,
will
then
pick
up
this
change
again.
So.
Eventually,
this
the
cluster
always
converges
into
the
desired
state
that
has
been
described
by
a
git
Repository.
B
And
what
in?
In
addition,
what
we
also
like
about
this
approach
is
that
if
something
cannot
be
converged
into
the
desired
State,
we
automatically
get
an
alert
about
it,
because
the
rocd
application
is
out
of
sync,
then
so
it
might
not
be
that
something
is
necessarily
broken
in
the
cluster,
so
that
the
cluster
is
completely
offline,
in
which
case,
of
course,
we
would
also
notice.
But
maybe
there
is
just
a
small
misconfiguration,
let's
say
in
the
DNS
setup.
B
So
it
combines
really
well
with
this
operator
driven
management
that
is
already
built
into
okd
clusters
and
finally,
where
argocd,
both
the
the
command
line
and
the
web
UI
also
gives
us
a
great
overview
of
the
resources
that
we
have
manually
deployed
into
the
cluster
and
by
now,
they're
really
a
lot
like
we're,
deploying
hundreds
of
resources
into
the
into
a
cluster.
And
of
course
you
cannot
keep
all
of
them
in
in
your
head
or
always
understand
where
each
resource
is
coming
from.
B
B
It's
also
used
for
for
the
Ingress
up
setup
for
volumes
for
networking
visibility,
so
that
an
application,
for
example,
is
only
Exposed
on
our
intranet
or
to
the
internet
or
on
the
technical
Network,
which
is
then
another
internal
Network
that
we
have,
but
also
for
more
tedious
tasks
like
automation
of
Mount
points.
So
for
this,
when,
when
a
user
wants
to
mount
this
EOS
file
system,
we
need
to
make
sure
that
it
has
valid
cables
credentials
and
then
these
credentials
expire
after
24
hours.
B
So
we
help
users
by
actually
injecting
an
init
container
and
a
side
guard
container
automatically
into
their
deployment.
That
already
has
sets
up
everything
as
it
needs
to,
and
the
only
thing
the
user
needs
to
do
is
set
a
small
annotation
in
their
deployment
and
open
policy
agent
will
take
care
of
the
rest.
B
Like
I
mentioned
operators,
okds
operators
were
using
Upstream
operators
such
as
developer,
backup
operator
or
assert
manager
for
requesting
certificates,
but
we
have
also
developed
a
lot
of
operators
ourselves,
and
these
are
what
is
powering
these
other
clusters
that
are
not
the
generic
platform
as
a
service.
So
we
have
the
webios
operator
that
takes
care
of
handling
this
shared
site
hosting.
We
have
a
gitlab
Pages
site
operator
that
integrates
with
our
gitlab
deployment.
We
have
the
authentication
operator
that
takes
care
of
the
entire
Project
Life
Cycle
and
tying
it
back
into
the
CERN
single
sign-on.
B
We
have
a
land
B
operator.
Landib
is
cern's
internal
kind
of
dhp
management
solution,
so
that
needs
to
be
integrated
with
DNS.
If
you
want
to
request
a
new
hostname
and
finally,
also
for
each
of
these
application
templates,
we
are
using
Helm
operators,
which
are
a
very
thin
layer
if
you're
not
familiar
so
it's
you.
Basically,
the
operator
SDK
allows
you
to
build
these
hum
operators
just
by
giving
it
a
Helm
chart
and
then
based
on
the
input
values
of
this
Helm
chart.
B
It
will
automatically
generate
a
crd
for
you,
and
this
is
nice
for
simple
use.
Cases
such
as
grafana,
for
example,
where
the
user
just
needs
to
needs
to
say
I
want
this
version
of
grafana
with
that
and
that
hostname,
then
the
operator
basically
just
runs
a
Helm
template
once
and
installs
the
necessary
resources
in
the
classroom
and
again
it
continuously
reconciles
those.
B
Now.
What
did
we
learn
during
this
time
of
operating
these
services?
Users
are
very
happy
about
the
Internal
Documentation
that
we
have.
This
might
be
something
if
you're.
If
you're,
building
such
a
service,
you
might
be
tempted
to
just
skip
because
well,
there
is
so
much
great
Upstream
documentation
and
that's
true,
but
still
your
users
like
to
have
a
friendly
entry
point
into
your
specific
service
and
then
from
there
you
can
link
to
external
resources.
B
We're
also
very
happy
that
operators
can
help
us
in
so
many
different
tasks,
both
for
us
as
admins,
but
also
for
the
users.
But
we
also
really
need
to
pay
attention
how
we
use
them,
because
they
are
very
sharp
tools
and
we,
just
recently
two
months
ago,
had
a
pretty
big
incident
where
one
of
the
operators
was
actually
misconfigured
and
started
deleting
a
bunch
of
resources
in
the
classroom.
B
At
the
same
time,
not
everything
has
to
be
automated.
We
frequent
relatively
frequently
have
steps
that
we
could
automate,
but
actually
it
would
not
be
worth
the
effort
because
in
the
end
we
just
need
to
run
one
or
two
commands
for
five
minutes
and
honestly
in
five
minutes,
you
cannot
reliably
script
it.
B
If
we
just
did
unmanaged
kubernetes,
because
then
most
of
our
users
would
just
be
completely
overloaded,
but
at
the
same
time,
if
we
only
had
the
managed
service,
we
also
could
not
serve
all
of
our
users,
because
some
of
them
just
have
really
special
requests.
Some
of
them
have
different
compliancy
or
auditing
requirements.
Some
of
them
have
different
compute
requirements.
B
Others
just
want
a
service
mesh
that
maybe
we
cannot
offer
to
all
of
our
users,
because
it's
too
complex
to
implement
so
having
these
two
separate
Services
overall,
is
very
beneficial
for
us
with
that,
I
would
like
to
thank
you
for
your
attention
and
thank
the
organizers
of
this
awesome
Meetup
for
putting
this
all
together.
I
think
it's
really
well
organized
and
you
can
find
the
slides
under
this
link.
Thank
you
very
much.
A
B
B
C
B
That's
a
that's
a
good
point,
so
I
would
say
as
a
starting
point.
Okd
really
helps
with
this
because
well,
you
have
all
of
this
support
from
from
Red
Hat
openshift
that
they
already
make
sure
that
you
get
warnings
early
enough.
Hey
this
thing
is
going
to
be
deprecated
into
kubernetes
versions
like
you
should
really
look
into
fixing
that
now
like
what
we
recently
had
with
kubernetes
chrome
jobs
that
the
one
better
one
was
deprecated.
B
So
that
is
one
thing
and
for
our
internal
self
self-developed
crds
there
we
can
actually
kind
of
collaborate
with
our
users
and
and
do
the
patches
on
the
fly
as
we
need
to.
Let's
say:
if
something
changes
in
the
spec,
we
can
say:
hey,
you
need
to
modify
this
or
that
it's
gonna,
the
behavior
is
gonna
stay.
The
same
just
add
that
field
there.
A
B
B
Then
there's
an
additional
check
that,
unlike
unlike
these
okd
clusters,
you
basically
need,
if
you
want
to
expose
it
to
the
internet.
For
example,
you
need
additional
approval
from
the
security
team
and
in
that
case
they
will
also
check
that
what
you're
doing
in
that
kubernetes
cluster
is
actually
somewhat
secure.
So
those
unmanaged
clusters
by
default
they
are
not
exposed,
at
least
to
the
internet.
A
Thank
you
very
much
Jack,
so
CERN
impresses
as
usual.
We
we
are
going
to
have
the
next
talk
at
five
o'clock
just
before
the
next
talk.
We're
gonna
have
two
very
short
pitches
from
our
very
beloved
sponsors,
and
we
are
so
you
guys
have
10
minutes
to
stretch
your
legs
if
you
like,
if
so,
should
you
want
to?
But
this
is
the
best
room
so
come
back
here.
Please
thank
you.