►
From YouTube: CNCF Research User Group - Eduardo Arango, Red Hat & Bob Killen, University of Michigan
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
CNCF Research User Group - Eduardo Arango, Red Hat & Bob Killen, University of Michigan
Interested in improving the Research experience with Kubernetes, or simply running research workloads on it? The CNCF Research User Group’s purpose is to serve as a focal point for the discussion and advancement of Research Computing using “Cloud Native” technologies. Since the group’s inception 6 months ago, key areas have been identified as gaps within the ecosystem. This session would serve as an opportunity to share with a broader audience some of the key challenges the Research-user-group has identified, and showcase project updates on key tools that the research community is developing to address these challenges. For more information visit: https://github.com/cncf/research-user-group
https://sched.co/ZeuD
A
A
Next
slide,
so
first,
I
sort
of
want
to
get
into
why
well
with
the
the
people
attending
this
talk.
I
don't
think
I
actually
have
to
talk
about
200
of
these
points,
but
in
the
research
ecosystem,
we've
seen
an
increased
use
of
containers
everywhere
and
with
that
has
come
more
complex,
workflows
and
not
just
for
the
research
process
itself,
but
with
enterprises
and
all
sorts
of
groups
sort
of
getting
involved
in
things
like
machine
learning.
A
Additionally,
we're
sort
of
seeing
greater
use
of
interactive
science
gateways
these
days.
People
are
just
expecting
to
get
something
like
a
jupiter
notebook
to
interact
with
their
data
and
sort
of
like
backing
those
science
gateways
and
data
streaming.
Services
is
the
persistent
services
that
are
needed
to
support
them
and
all
these
things
actually
lend
themselves
really
well
to
kubernetes.
A
A
We
also
require
more
advanced
scheduling
than
what
kubernetes
can
provide
out
of
the
box.
The
best
example
of
this
being
mpi
or
recording
the
scheduling
of
multiple
pods
to
launching
around.
At
the
same
time,
workarounds
have
sort
of
been
developed
by
multiple
little
communities
like
there's
the
mpi
operator
in
kubeflow.
A
A
very
awesome
up-and-coming
project
called
volcano,
that
is
a
sandbox
project.
There's
sort
of
like
several
other
little
things
that
out
there
that
sort
of
rely
on
stateful
sets
to
sort
of
do
like
offer
that
similar
behavior
now
mpi
isn't
the
only
thing
many
environments
are
large
and
actually
require
additional
support
for
things
like
fair
share,
cues
and
a
whole
lot
of
other
things
when
you're
running
on
something
like
bare
metal
and
while
these
things
haven't
necessarily
been
forgotten
by
upstream,
they
have
largely
been
underrepresented.
B
And
I
like
that
last
word,
because
also
why
to
form
a
user
group
is
to
represent
us
all.
So
that's
what
I
like
from
the
cloud
native
foundation
is
we're.
Creating
this
user
group
to
share
to
share
pain
points
to
have
a
weekly
or
monthly
meeting
to
say
is
not
working
for
me.
It's
failing
and
I
need
some
help
and,
as
bob
said,
this
is
a
small
user
group
and
we
need
as
much
help
as
we
can.
So
this
is
a
group
for
sharing
success
histories
like
the
project
volcano.
B
That
bob
also
told
us
and
all
the
new
ideas
are
coming
from
these
small
groups,
and
this
is
a
common
place,
so
we
can
share
and
we
can
request
feedback
from
others
in
the
same
field,
trying
to
make
research
a
little
more
cloud-making,
and
when
I
say
these
small
groups
I
mean,
like
we
have
a
small
projects
coming
out
of
cern
like
the
latest
hd
contour
operator,
or
the
common
workflow
language
now
being
able
to
interact
with
kubernetes,
and
we
have
universities
also
joining
gray.
So
it
is
growing.
B
A
A
Now
there
there
have
been
some
various
like
rumblings
and
efforts
in
the
past
boss,
actually
sort
of
came
to
a
head
last
year
at
cube
county.
U,
in
barcelona,
where
a
bunch
of
us
also
like
met
together
over
a
lunch
for
a
few
days,
and
we
decided
we
should
sort
of
make
something
happen,
and
originally
it
was
going
to
be
a
curious
working
group,
but
over
time
it
evolved
into
cncf
into
the
cncf
user
group
that
it
is
today.
A
Now
the
things
like
curious
itself
out
of
the
box
doesn't
support
running
workloads
like
mpi
the
base
batch
capabilities
are
kind
of
lacking
there,
and
while
there
are
so
like
other
things
like
there's,
there's
multiple
multi-cluster
tools
out
there
most
don't
lend
themselves
to
the
distribution
of
jobs.
They
lend
themselves
to
the
distribution
of
like
your
standard
application,
type
deployments,
they're
very
sort
of
focused
on
ensuring
that
you
know
sort
of
like
x.
Number
of
deployments
are
across
multiple
clusters,
not
x,
number
of
jobs
are
spread
evenly
across
the
mall.
A
The
other
thing
is
multi-tenancy
itself
is
also
a
problem.
Kubernetes
isn't
really
like
your
sort
of
standard
hpc
environment,
where
everyone
has
a
posix
identity,
you're
now
dealing
with
application
identity,
and
these
are
just
a
few
of
the
common
themes
we've
seen
and
with
that
we
actually
decided
to
start
a
few
initiatives.
B
So,
let's
talk
about
the
initiatives
that
have
come
from
the
researchers
group
so
right
now
we
want
to
highlight
these
three
main
initiatives:
the
first
one
as
I
was
talking
about
communities.
We
want
to
know
how
big
this
community
can
be
and
we
want
to
know
who
is
using
kubernetes,
who
is
using
kubernetes
and
cloud
native
tools
overall
for
research.
B
B
We
also
have
this
index
and
we
wanna
we're
always
seeing
this
of
awesome
list
as
every
solution.
Right
now
has
awesome
list,
and
I
remember
my
first
awesome
list
was
awesome
containers
I
remember
now,
and
we
want
to
have
this
awesome
list
of
projects
that
can
help
other
researchers
to
get
their
job
done,
because
at
the
end,
that's
the
main
idea
of
this
user
group,
some
random
researcher
in
a
random
research
institute
just
coming
into
this
group,
getting
help
getting
his
research
done.
B
B
So
the
cloud
native
research
workloads
book
really
wants
to
target
system,
admins
operator
research,
software
engineers
and
research
support
staff,
so
kind
of
like
in
where
we
are
right.
Now
that
we
don't
say
system
admins
anymore,
it
evolved
from
system
admin
to
the
devops
guy
to
the
operator.
Guy
now
we
say
sre,
but
it's
kind
of
back
to
the
guy.
B
That
is
dealing
with
all
the
issues
the
system
I
mean
we
want
to
write
this
guideline
with
user
stories
with
pain
points,
as
I
was
saying
in
the
previous
slides
and
really
take
down
from
day
one
to
day
x
what
you
should
be
doing
with
your
qr
netease
deployment
and
how
to
operate
that
in
a
good
way.
So
you
don't
get
into
the
researchers
workloads
right
like
interrupting,
runs
and
executions
and
also
protecting
data,
because
that's
something
that
is
very
hard.
B
As
bob
was
saying,
the
multi-tenancy
and
research
environments,
you
all
the
time-
need
to
be
thinking
that
researcher
a
is
not
deleting
or
reading
information
from
researcher
b,
and
we
want
to
make
research
cloud
native
with
this
book.
When
we
say
this
is
I
I
personally
want-
and
we
want
a
community
to
help
researchers,
teachers
and
students
to
get
things
done
faster,
easier
and
in
a
better
way.
B
So
cluster
operations,
the
topics
we
want
to
touch
in
this
book
is
configuration
management
and
cluster
life
cycle
so
which
things
to
install
from
day
one
how
to
operate
them,
how
to
update
them,
how
to
upgrade
them,
how
to
delete
them?
If
you
don't
need
them
anymore,
so
the
life
cycle
of
an
application
in
top
on
top
of
your
qnets
cluster,
we
want
to
write
that
down
as
a
guideline
management
of
clusters
on
top
of
openstack.
So
a
lot
of
research
institutions
are
right
now
using
openstack
integration
with
parallel
hpc
file
system
and
access
management.
B
So
it
get
very
important
when
you
start
talking
about
like
mpi
and
things
like
that.
How
can
you
make
your
qr
net
disk
cluster
interact
via
our
dnas
and
be
a
specialized
hardware?
So
here
we
are
not
talking
about
software
but
hardware.
How
can
you
enable
that
specialized
hardware
to
your
your
netis
cluster,
exposing
gpu,
so
your
qr
net
is
closer
in
a
good
way
and
manage
large
escape
cluster
right,
like
1000
nodes
in
just
so.
B
What
tooling
work
when
provisioning
notes
is
really
not
that
easy
to
solve
qr
headings
in
a
bare
metal
cluster
is
not
like
when
you
request
to
your
cloud
provider:
hey
give
me
a
kubernetes
cluster,
here's
my
credit
card
and
you
get
a
kubernetes
cluster
running
and
some
dollars
left
of
your
credit
card.
But
how
to
do
that
in
your
own
cluster,
in
your
own
systems,
is
not
that
easy.
So
what
tooling
to
use
and
how
to
use
it,
because
you
can
use
different
tools,
but
is
not
that
standard,
how
to
use
them?
B
How
is
authentication
managed?
And
for
me
this
is
a
very
important
like
authentication
and
also
the
security
policies
for
multi-user
multi-tenant
on
a
research
cluster,
because
it
gets
very
messy
in
multi-tenant
environments.
So
we
need
to
be
able
to
differentiate
users
projects.
A
user
can
be
in
different
projects
like
let's
say
a
professor.
Our
teacher
can
be
in
different
projects,
so
he
has
access
to
different
kind
of
data,
but
also
there
are
some
researchers
that
can
only
read
one
specific
side
of
the
data.
B
Now,
how
can
you
then
use
ironic
to
deploy
bare
metal
provisioning?
How
to
you
can
integrate
with
other
tools
in
the
ecosystem,
to
put
your
qr
nets?
Cluster
on
top
of
openstack,
also
how
to
do
parallel
file
systems
in
the
way
hpc
system
ecosystem.
Does
it
now
in
a
cloud-native
way?
So
we
want
to
write
these
guidelines
in
this
book
gpus.
This
is
one
of
my
favorite
topics.
B
I
I
am
part
of
the
gpu
operator
and
I
maintain
how
to
run
gpus
for
red
hat,
so
I'm
very
passionate
about
this
and
writing
guidelines
and
a
book
about
how
to
do
this
is
very
important
and
then
helping
researchers
to
enable
their
gpus
and
new
and
shiny
nvidia,
dgx
or
whatever,
or
if
you
are
in
the
amd
ecosystem,
how
to
do
that
also.
So
these
guidelines
are
going
to
help
from
the
system
admin
sres
to
the
researchers
to
expose
the
new
and
shiny
gpus.
So
then
they
can
run
tube
flow
or
aiml
research.
B
A
Okay,
that
sort
of
kicks
over
to
the
other
side
of
the
coin,
the
not
the
people
necessarily
running
the
clusters,
but
the
people
consuming
them
and
your
researchers,
your
teachers,
your
students.
This
might
also
eventually
be
like
other
enterprise
type
customers
that
are
looking
at
a
dashboard
for
report
on
something,
but
with
all
that
we
sort
of
identified
a
few
big
key
areas.
A
Next
sort
of
building
on
top
of
jupiter
hub
a
little
bit
is
actually
kubeflow
and
machine.
Learning
this
or
life
cycle
management,
and
just
as
like
jupiter
hub,
has
sort
of
become
the
standard
for
things.
Cube
flow
is
quickly
becoming
sort
of
the
the
go-to
for
things
specifically
related
to
machine
learning
and
machine
learning
life
cycle
management.
It
is
actually
a
sort
of
large
collection
of
tools
that
have
all
been
bundled
together
and
like
integrate,
really
really
well
with
each
other.
A
The
kubeflow
team
has
done
a
incredible
job
with
that,
and
you
know
the
the
one
thing
that's
a
little
bit
lacking
is
like
there's.
A
lot
of
people
are
getting
up
and
going,
but
the
best
practices
around
doing
sweat.
Don't
you
know
quite
exist
quite
yet,
and
there's
sort
of
like
some
other
little
aspects
of
it
like
it
is
possible
to
make
it
multi-tenant,
but
it
you
know
the
the
stories
around
and
people
actually
doing
it.
Multi-Tenant
is,
is
I'll,
say
a
little
bit
lacking.
A
The
other
thing
you
heard
me
sort
of
like
touch
upon
this
earlier
is
like
batch
computing
and
mpi.
These
things
are
not
supported
by
kubernetes
by
default,
but
there
are
tools
out
there
that
do
enable
that
for
kubernetes,
the
the
big
one,
if
you've
happened
to
see
me
talk
about
stuff
before
you,
you've
probably
heard
me
reference
volcano,
a
cncf
sandbox
project
that
is
bringing
these
sort
of
like
classic
batch
computing
style
things
to
kubernetes,
and
it
provides
things
like
cues,
backfill
and
all
things
that
you
would
find
in
a
classic
environment.
A
Okay,
the
next
thing
is
sort
of
the
various
like
workflow
engine
and
get
ups
there.
This
is
a
very
large
ecosystem
with
a
host
of
different
tools,
probably
one
of
the
most
common
ones
out
there.
These
days
is
argo
and
it's
actually
used
by
like
kubeflow
under
the
hood,
but
you
know
it's
not
the
only
one
out
there.
There
are
many
others
out
there
and
some
general
guidance
around
you
know
what
tools
exist
and
how
to
use
them
would
be
very
useful.
A
The
other
thing,
and
after
talking
for
some
time,
we
had
originally
like
lumped
in
git
ops
and
was
sort
of
like
workflow
engines,
since
they
sort
of
you
know,
have
a
very
close
relationship
with
each
other,
but
it's
now
a
large
enough
topic
where
it
will
probably
be
spun
off
into
its
own
chapter
and
actually
started
working
on
a
paper
where
we
will
probably
see
a
lot
more
results
from
that.
I
don't
know
with
the
timeline
for
it,
but
but
sometime
soon.
A
So
how
like?
We
want
to
answer
these
questions
on
how
researchers
can
like
effectively
use
git
ops,
we've
seen
you
know
what
storing
everything
in
git
has
done.
For
you
know
the
management
of
sort
of
enterprise
applications
and
being
able
to
tie.
You
know
what
iteration
of
a
model
to
the
code
to
the
data
that
was
run
to
like
produce
the
result
and
we've
seen
how
useful
that
is.
Now
we
just
need
to
sort
of
like
write
a
little
bit
of
how
more
people
can
do
it.
A
So
in
your
normal
hpc
system,
where
you
have
that
parallel
file
system
and
everything
has
a
posix
identity,
where
everyone's
logging
into
like
a
login
node
that
doesn't
really
exist
in
a
traditional
way
in
sort
of
the
cloud
native
space.
So
this
might
be.
You
know,
shifting
more
people
to
learning
to
use.
A
So
you
can
help
us
by
going
into
like
the
the
scene,
safe
research,
user
group,
repo
and
fill
out
the
site
survey.
If
you
have
a
really
useful
resource
on,
you
know
how
to
do
something,
please
just
like
pr
it
into
the
like
our
awesome
list.
It's
super
helpful.
The
other
thing
is
like
we
have.
Some
people
have
already
sort
of
volunteered
to
help
us
with
the
various
chapters
in
this
book,
but
we
still
have
a
lot
of
sections
that
haven't
been
claimed.
A
A
A
We
have
two
meetings
a
month,
the
first
wednesday
of
the
month
at
11am
cet,
that's
the
more
eu
friendly
meeting
and
then
the
third
wednesday
of
the
month
at
5
pm
cet,
which
is
the
more
you
know
america's
friendly
meeting,
and
with
that
I
think
we
will
call
it
there
and
now
go
over
to
the
q.
A
that's
in
like
a
month.