►
From YouTube: Cloud-Native Computing For Research Users - Jamie Poole, G-Research & Ricardo Rocha, CERN
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Cloud-Native Computing For Research Users - Jamie Poole, G-Research & Ricardo Rocha, CERN
The CNCF Research User Group’s purpose is to function as a focal point for the discussion and advancement of Research Computing using “Cloud Native” technologies. This includes enumerating current practices, identifying gaps, and directing effort to improve the Research Cloud Computing ecosystem. In this session we will introduce the user group, discuss our mission and the particular challenges that research institutions face that we aim to meet with cloud-native technology. We will hear from speakers from public and private research institutions such as CERN (physics) and G-Research (fintech) and discuss various relevant projects in flight.
A
So
welcome
to
our
virtual
kubecon
2021
presentation,
thank
you
for
coming
or
thank
you
for
watching
online.
As
I
should
say,
we're
going
to
talk
to
you
today
about
the
cncf
research
user
group,
who
we
are
what
the
group
goals
are,
who
comprises
the
group
some
highlights
from
recent
sessions.
We've
had
and
also
a
survey
of
our
constituents
next
slide,
please
so
yeah
the
group
is
run
by
myself
and
ricardo,
I'm
jamie
poole,
I'm
a
engineering
manager
at
g
research.
B
Yeah,
so
my
name
is
ricardo
rocha.
I
work
at
cern
as
a
computing
engineer
in
the
cern
cloud
team.
I
focus
mostly
on
containerized
deployments,
networking
and
more
recently,
accelerators
and
machine
learning
as
well,
so
we've
been
using
kubernetes
for
for
quite
a
while.
Now
so
and
we
engaged
in
this
group
when
it
when
it
when
the
idea
came,
I
think
it
was
kubecon
barcelona
in
2019,
I
guess
early
2019.,
so
yeah.
Well,
it's
been.
It's
been
a
pleasure
to
to
to
work
with
jamie
as
well
and
everyone
else.
A
Cool,
so
we'll
talk
quickly
about
the
goals
of
the
group
here,
so,
as
ricardo
says,
I
think
we
put
the
group
together
at
the
beginning
of
2019.
I
feel
it
could
have
even
been
earlier
than
that,
but
it
was
definitely
the
keeper.
In
barcelona
there
was
a
bunch
of
people
who
came
from
different
research
computing
institutes
who
wanted
to
use
cloud-native
tech
and
wanted
to
solve
the
same
kinds
of
problems.
A
So
we
sort
of
came
together,
I
think,
actually
over
a
lunch
and
then
from
there
it
sort
of
became
a
proper
cncf
group
and
yeah.
Basically,
we
want
to
just
bring
together
people
from
different
research
institutions
who
want
to
solve
the
same
common
problems
using
cloud
native
tech
and
really
build
a
sort
of
research
focus
community
who
know
how
to
deal
with
the
same
sorts
of
things
that
we
will
have
to
deal
with,
maybe
slightly
different
from
your
traditional
cloud
native
technologists.
I
suppose.
A
We
ran
a
poll,
in
fact
no,
this
wasn't
the
poll
was
it.
This
is
actually
from
our
weekly
bi-weekly.
Sorry,
meetings
keep
track
record
of
who
attends
and
who
speaks,
and
this
was
the
results
based
on.
I
think,
a
notebook
that
ricardo
put
together,
which
is
going
to
show
you
the
gory
details
of
in
a
minute,
but
you
can
see,
we've
got
quite
widespread
of
institutions
from
around
the
world,
so
I'm
from
g
research.
I
often
get
a
lot
of
us
to
come
along.
B
A
Certain
big
constituent
part
as
well
always
always
present
but
yeah,
quite
a
wide
range
here
of
different
people
from
different
universities.
Private
public
institutions
from
all
around
the
world,
actually
literally
from
the
all
the
way
from
the
uk
to
australia
and
back
again
so
yeah,
really
great
group
of
people
to
get
together.
A
B
To
see
like
not
only
individual
institutions,
but
in
some
cases
we
see
well,
I
I
work
on
the
physics
side,
so
we
see
institutions
like
nfn,
which
are
actually
very
large
groups
of
of
different
places
in
italy
or
egi
foundation,
which
is
driving
the
great
infrastructure
in
europe.
So
yeah
it's
it's
been.
B
It's
been
quite
a
nice
set
of
diverse
groups
and,
as
jamie
said
I
I
guess,
because
this
is
a
research
user
group
and
well,
we
do
a
lot
of
data
analysis
or
at
least
the
users
of
the
services
we
we
provide
do
a
lot
of
data
analysis,
so
I
thought
it
would
be
fun
just
to
collect
all
the
attendees
from
from
the
agenda
page
and
do
a
quick,
a
quick,
quick
analysis
on
on
the
on
the
list.
So
two
silly,
very
silly
ones.
B
In
addition
to
the
to
the
word
cloud
that
we
just
saw,
we,
we
got
65
unique
attendees
over
the
last
10
minutes,
10
meetings.
So
it's
not
like
the
the
whole
lifetime
of
the
group,
but
just
the
last
10
minutes
meetings,
which
is
which
is
quite
good.
We
had
some
some
sessions,
I
think
with.
Like
over
30
people,
so
it's
been
pretty
active
and
then
also
the
the
different
institutions.
We
have
34
different
institutions
participating.
A
Yeah
so
highlights
here
from
the
the
recent
sessions
we've
had
the
structure
of
the
group
at
the
moment
is
we
meet
on
the
first
wednesday
and
the
third
wednesday
of
every
month?
Currently,
four
pm
utc,
you
want
to
get
this
right
with
time
zones
changing
the
structure.
Typically,
is
we
just
do
intros?
Anyone
who
hasn't
come
to
the
group
before
does
a
little
intros
of
themselves
and
says
hello
to
the
group
and
we
sort
of
greet
them.
Then,
ahead
of
time,
we've
decided
on
a
particular
topic.
A
We
want
to
talk
about
that
session.
We'll
have
a
speaker
who
typically
presents
the
chosen
topic
for
15
20
minutes.
Normally,
some
kind
of
presentation
normally
a
little
bit
interactive.
If
possible,
I've
got
a
screenshot
here
of
jeremy
from
oak
ridge
talking
about
their
really
cool
work.
They've
done
recently
on
posix
integrations
yeah,
then
q,
a
for
10
or
15
minutes,
and
then
a
bit
of
wrap
up
talk
about
the
the
topics
we
want
to
do
for
next
time
and
then
move
on.
A
We
record
all
the
sessions
they're
all
available
on
the
github
page.
I
think
actually
shout
out
to
cheryl
hung
as
well,
who
organizes
it
for
us
she's.
In
fact,
gonna
live
stream.
I
think
the
next
one
on
youtube
and
we're
gonna
try
that
out
for
a
bit
see
how
that
goes.
So
that
should
be
really
interesting,
but
yeah
in
terms
of
the
the
recent
topics
they've
been
quite
diverse.
Obviously
all
research
focused
things
that
are
our
members
are
interested
in
mauro
from
g
research.
B
That
we
had
like,
I
think,
three
different
institutions
showing
how
they
are
providing
notebooks
on
on
kubernetes
things
like
the
hub
or
binder
or
in
some
cases,
using
tools
and
frameworks
like
desk
for
python,
so
that
that
one
was
a
pretty
good
one
and
actually
triggered
some
some
follow-ups
between
the
the
people
that
showed
the
interest
in
those
and
we
started
collaborating
on
creating
a
recipe.
We'll
talk
a
bit
more
about
that
one
as
well,
but
but
there
there's
a
couple
of
other
ones.
B
I
would
also
like
do
this
shout
out
to
cheryl
she's,
been
pushing
for
this
group
quite
a
bit
and
helping
us
with
all
the
recordings
and
all
the
setup,
and
also
bob
that
that
actually
founded
the
group
pop
killing
that
founded
by
the
start,
but
in
terms
of
topics.
I
would
I'll
also
highlight
two
that
are
very
close
to
to
to
our
needs
as
well.
B
So
it's
quite
important
that
all
the
components
not
only
kubernetes
but
continuously
and
all
the
the
rest
of
the
stack-
supports
rootless
mode
and
then
the
other
one
is
image
distribution,
because
we
typically
do
very
large
submissions
of
workloads
and
being
able
to
start
these
jobs.
Quite
fast
is
very
important,
and
also
maybe
it's
the
case
for
for
everyone.
B
Researchers
are
not
necessarily
very
good
at
layering
their
images,
so
we
often
end
up
with
very
large
images.
So
we
had
a
really
nice
presentation
from
akihiro
suga,
for
ruthless
and
and
also
from
k,
talk
from
nct
as
well
talking
about
the
continuity
snapshotter
and
the
srgz
support.
So
these
are
really.
A
B
Yeah
and
we
will
keep
the
survey
open,
so
it's
still
time
for
everyone
to
jump
in
and
then
we
can
do
an
update
later
as
well,
but
yeah.
We
can
start
just
with
an
overview
of
the
institutions
that
already
replied,
so
we
can
see
quite
a
span
of
different
areas
already
in
the
names
there's
a
couple
of
cases
where
there's
more
than
one
reply.
I
know,
for
example,
for
cern
this
happens,
because
we
actually
have
different
groups
at
cern
that
focus
on
different
areas
and
yeah.
A
Yeah,
so
no
no
surprises
here,
I
suppose
the
the
bulk
of
it
is
sort
of
non-private
sector.
Anyway.
I
think
private
sector
was
just
represented
by
me
in
the
in
the
feedback,
but
we've
got
a
lot
of
academia,
a
lot
of
government,
a
lot
of
non-profits
when
you
add
it
all
together
so
yeah.
It's
sort
of,
I
guess
not
particularly
surprising,
in
the
sense
that
that's
where
a
lot
of
the
bulk
of
the
research
happens
in
the
world.
A
B
Yeah
this
one
was
just
to
have
an
idea
of
the
size
of
the
institution,
so
we
have,
I
think,
probably
one
answer
over
ten
thousand
people
most
cases
are
a
thousand
between
a
hundred
and
thousand.
So
it's
it's
still
pretty
large
yeah
and
quite
quite
a
few
between
a
thousand
and
ten
thousand
as
well
so
yeah.
A
Here
we've
got
the
different
areas
of
research
which
people
primarily
support,
so
quite
strong,
showing
from
physics,
bioinformatics
and
generic
research
platforms.
I
guess
there's
potentially
a
little
bit
of
murkiness
as
to
what's
going
on
in
there,
but
yeah
the
the
bulk
of
it,
I
would
say,
is
sort
of
science
and
scientific
research.
I
would
say
from.
B
B
Yeah
internet
too
also
for
for
more
education.
Very
it's
quite
interesting
and
the
strong
astronomy
that
there's
the
one
one
community
that
hasn't
been
very,
very
present,
which
is
from
from
the
specifically
the
ska,
but
actually
from
astronomy.
We
have
quite
active
members
from
from
alma
from
the
observatory
in
chile.
So
that's!
I
think
we
should.
We
should
get
them
to
do
a
talk
at
some
point
even
to
to
show
off
their
their
nice
telescopes.
B
A
Okay,
so
this
is
really
good
to
see
very,
very
clear
that
the
vast
majority
of
the
group
are
already
using
kubernetes
in
production.
So
this
is
a
group
of
people
who
do
really
understand
the
platform
and
are
really
using
it
for
value-add
and
getting
something
out
of
it.
This
is
not
just
a
sort
of
collection
of
people
who
are
hobbyists
or
you
know,
trying
it
out
for
the
first
time.
A
This
is
really
sort
of
proper
production
use
at
scale,
certainly
on
a
certain,
a
big
scale,
users,
and
so
we
actually
so
very
large
amounts
of
compute
running
on
kubernetes
in
production.
However,
that
being
said,
there
are
some
groups
here
who
are
sort
of
just
starting
out
and
just
learning.
So
it's
also
a
good
way
for
people
to
get
in
and
learn
more
about
the
platform,
see
what
it
can
do
for
you.
B
Yeah,
it's
even
more
surprising
because
yeah
we'll
see
also
in
the
next
answers,
but
but
there
is
a
kind
of
a
learning
curve
to
get
into
kubernetes,
both
on
the
infrastructure
and
the
usage.
So
it's
quite
nice
to
see
that
so
many
people
are
already
using
it
in
production
for
research
as
well
and
then
yeah,
the
the
other.
The
other
question
we
asked
was:
where
are
the
which
kind
of
infrastructure
is
being
used?
B
A
So
yeah,
I
think,
you're
right,
I
think,
certainly
for
the
sorts
of
institutions
that
we
tend
to
deal
with.
These
have
been
sort
of
large-scale
hpc
research
shots
for
a
long
time.
Long
before
you
know
cloud
native
tech
existed,
I
suppose
so
you
probably
already
own
large
data,
centers
and
amounts
of
hardware,
so
yeah,
certainly
for
us
as
well.
It's
a
case
of
how
we
use
that
in
a
cloud-native
way,
without
necessarily
jumping
straight
into
public
cloud.
B
Yeah
and
the
follow-up
to
this
would
maybe
to
this
survey
would
be
to
to
understand
better
the
requirements
for
hybrid
deployments
as
well,
because
I
know
in
our
case
we
we're
looking
to
extend
the
on-premises
resources
and
try
to
fish
for
things
that
we,
we
don't
have
too
many
like
gpus
and
accelerators,
so
to
be
nice
to
understand.
Not
only
why,
but
also
how
people
are
doing
this.
A
Yeah
yeah
we're
just
looking
at
the
why
people
are
picking
up
kubernetes
and
using
these
cloud-native
tools.
I
think
it's
kind
of
interesting
a
lot
of
this.
I'm
saying
the
latest
slides
as
well,
I
think,
is
really
we're
just
sort
of
highlighting
the
benefits
of
kubernetes,
but
really
showing
that
you
can
actually
make
use
of
those
benefits
in
your
research
setting.
So
you
can
imagine
in
research
institutions
things
like
reproducibility
and
vast
iteration
are
super
important
because
you
need
to
be
able
to
rely
on
your
data.
So
that's.
B
B
And
then
this
one
surprised
me
at
least
that
so
many
so
many
deployments
or
yeah
half
of
deployments
actually
more.
If
we
count
the
next
actually
provide
direct
access
to
the
clusters
to
the
end
users,
but
but
it's
it's
nice
nice
to
see
because
yeah
it
is
a
big,
a
big
move
forward
to
do
this,
but
yeah
traditionally,
at
least
in
our
deployments.
We
already
have
systems
on
top
that
that
will
do
the
higher
level
exposed
services
to
the
users
in
a
higher
level
way,
not
really
necessarily
with
the
coordinates
api.
B
A
So
here
we
just,
I
guess,
threw
out
a
bit
of
a
list
of
sort
of
popular
cloud
native
projects
which
are
relevant
in
the
communities
today
to
see
what
the
sort
of
spread
was
of
people
across
our
group.
Using
these
things,
so
no
surprise,
kubernetes
out
front,
almost
100
percent
a
lot
of
prometheus,
pretty
standard
where
I
suppose
of
collecting
metrics
and
looking
at
your
data,
I
noted
that
service
mesh
was
particularly
low
here,
chatham
tricardi,
just
before
this,
I
think,
probably
actually
in
a
research
setting.
A
B
B
I'm
also
very
interested
who,
from
the
12
responses,
is
not
using
kubernetes,
which
is
kind
of
curious
here.
The
other
two
that
are
interesting
as
well
is
argo.
I
guess
it's
it's
gaining
quite
a
lot
of
popularity
also
for
for
managing
workflows.
So
that's
an
interesting
one.
I
know
tools
like
coopflow
also
rely
on
it
and
there's
others.
A
Yeah
see
icd's
come
up
a
few
times
as
a
topic
to
talk
about
as
well
to
see
how
people
are
managing
their
estates,
because
there's
there's
lots
of
different
models.
I
know
like
even
me,
and
you
do
things
differently.
You
tend
to
sort
of
provision
clusters
for
people
to
then
use
and
they
own
them,
whereas
we
do
the
more
sort
of
managed
approach,
but
both
approaches.
You
need
sort
of
quite
good,
reliable,
ci,
cds,
so
yeah.
It's
very
interesting
stuff.
B
Cool
then,
what
types
of
workloads
are?
I
guess
this
is
it's
pretty
amazing
that
10
out
of
12
are
doing
cheap
to
help
an
interactive
analysis
type.
I
guess
the
popularity
of
these
sessions
in
this
specific
session
in
the
group
kind
of
also
showed
this
and
second
is
batch.
So
I
guess
these
are
clearly
the
two
top
top
things
that
the
members
of
the
group
are
focusing
on
yeah,
it's
kind
of
surprising,
the
the
amount
of
people
doing
databases
and
kubernetes,
but
I
guess
yeah.
Why
not?
But.
B
B
A
Yeah
I
mean,
I
think
the
two
ones
that
jumped
out
for
me
was
the
different
mental
model,
steep
learning
curve,
along
with
overall
complexity.
I
think
we
all
know
it
is
quite
a
complex
system
and
once
you
get
over
that
sort
of
initial
learning
clip,
you
can
really
start
to.
You
know
maximize
the
benefit
from
it,
but
I
can
see
that
for
people
coming
board,
it's
it's
definitely.
You
know
quite
complex
and
there's
a
lot
going
on.
A
There's
other
points
here
which
is
assigned
to
that
around
some
pre-packaged
software
and
integration
with
other
systems,
but
yeah.
It's
it's.
It's
clear
that
there's
it's
not
completely
smooth
sailing
for
people,
but
I
guess
that's
why
we
come
together
in
these
sorts
of
groups
to
work
out
its
ways
of
overcoming
these
problems.
B
Yeah,
the
pre-packaged
software
is
quite
interesting
as
well,
because
for
for
most
generic
tools,
it's
probably
already
there.
It
might
just
be
that
we
are
lacking
something
on
on
this
specific
area
for
research
and
scientific
here
and
then
the
interest.
The
other
interesting
one
is
actually
reproducibility,
because
it
was
one
of
the
like
top
reasons
to
to
rely
on
kubernetes
and
cloud
native.
But
then
it's
also
specified
as
one
of
the
main
points
for
using
kubernetes.
So
it's
clearly
work
to
be
done
here
still.
B
Yeah
so
then
the
last
bit
is
regarding
where
we
can
help
the
most
so
yeah.
I
think
it
was.
The
goals
of
the
group
are
also
shown
here.
So
it's
it's
increasing
the
visibility
of
the
research
use
cases,
but
then
there's
there's
some
specific
things
we
saw
in
the
previous
answer.
Also
this
this
need
to
help
and
package
catalog
and
software,
and
then
the
the
other
one
I
would
highlight.
B
A
B
B
What
is
that-
and
there
was
a
discussion
about
how
to
integrate
with
things
like
hd
condor.
There
are
other
discussions
on
how
to
integrate
with
kubernetes
itself.
There
were
other
solutions,
so
maybe
maybe
this
is
something
where
we
should
put
the
focus
on
yeah
and
yeah.
I
think
that's
that
this
is
the
very
last
one
and
yeah.
A
Yeah
there's
just
some
comments
here.
People
have
the
suggestions,
but
generally
looks
like
people
are
enjoying
the
group
and
it's
good
to
see
that
people
appreciate
the
group
exists,
so
we're
very
happy
to
to
keep
it
going
and
just
want
to
encourage
people's
collaboration.
Really.
I
think
the
next
slide
just
talk
about
our
our
plans
to
build
on
the
momentum
we
have
now.
Collaborate
on
these
recipes
and
yeah
just
really
engage
with
other
groups
as
well
to
try
and
get
people
on
board
and
solve
some
of
these
problems
so
yeah.
A
I
think
the
final
slide.
If
you
move
on
to
how
to
get
involved,
I
think
it's
probably
enough
of
me
and
ricardo
talking,
but
yeah
come
on.
Never
look!
This
is
our
github
page.
It's
got
details
of
the
zoom
meetup,
which
we
do
currently
first
and
third
wednesday.
Every
month,
if
you're
on
cncf
slack,
then
there's
a
public
group.
Ug
research
he's
a
group,
research
and
yeah.
Thank
you,
everyone
for
all
your
participation
over
the
last
couple
of
years
and
let's
keep
it
going.