►
From YouTube: Sponsored Session: Mining Large Data Sets of Biomedical Omics Da... Doug Davis & Theodore Alexandrov
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Sponsored Session: Mining Large Data Sets of Biomedical Omics Data Made Easy with Managed Kubernetes! Doug Davis, IBM & Theodore Alexandrov, EMBL
Sharing of Personal Information with Sponsors In order to facilitate networking and business relationships at the event, you may choose to visit a third partyʼs virtual booth or to access sponsored content. You are never required to visit third party booths or to access sponsored content. When visiting a booth (e.g. by clicking on a third partyʼs logo in the Solutions Showcase or exhibitor directory, and any actions within the booth thereafter including viewing resources), when accessing sponsored sessions in the Sponsor Theater, or by participating in sponsored activities, the third party will receive some of your registration data. This data includes your first name, last name, title, company, address, email, standard demographics questions (i.e. job function, industry), and details about the sponsored content or resources you interacted with. If you choose to interact with a virtual booth or access sponsored content, you are explicitly consenting to receipt and use of such data by the third-party recipients, which will be subject to their own privacy policies.
Wouldn’t it be nice to accelerate your delivery cycle by not focusing on Kubernetes? That’s exactly what developers from the European Molecular Biology Laboratory (EMBL) did. Their mission: reveal the role that metabolites play in biology and medicine by finding the molecules hidden in big spatial metabolomics data. Their challenge: the amount of data they needed to analyze (sometimes larger than 1TB) required significant resources and compute power working at scale to produce timely results with a development team that were not K8s experts.
In this session, you'll learn how EMBL researchers used IBM Cloud Code Engine to speed-up their delivery cycle by allowing them to focus on their research and analytic engine development rather than worrying about the management and technical details of the compute resources needed to perform their data analysis.
A
A
B
And
here
you
see
our
faces,
you
see
who
we
are
we're
a
team
of
scientists,
but
also
software
engineers
and
computer
scientists,
and
what
we
are
working
on.
We
are
trying
to
develop
methods
to
really
improve
the
use
of
this
new
technology
called
spatial
metabolomics
for
life
sciences
and
but
with
applications
in
biology
and
medicine.
B
So
I
will
not
go
much
into
detail
what
spatial
metabolomics
is
about,
but
you
can
see
our
scientific
officer
prasad,
taking
a
very
thin
section
of
a
tissue
and
putting
it
on
a
glass
slide
and
putting
it
into
this
instrumentation,
which
is
called
imaging
mass
spectrometry
and
there
there
will
be
some
magic
happening.
They
are
producing
very
unique
data.
Why
is
it
important?
B
So
let
me
get
you
one
glimpse
into
what
this
data
spatial
metabolomics
data
can
represent
and
what
the
challenges
are
behind.
So,
first
of
all,
it's
very
special
data,
because
you
can
think
of
this
as
an
image,
but
not
with
just
three
rgb
channels,
but
with
over
10
000
channels,
and
you
can
start
thinking
that,
oh
yeah.
So
then,
probably
this
data
is
pretty
big
and
yes,
it
is
just
from
one
tissue
section
or
from
a
sample.
B
We
can
easily
generate
more
than
10
gigabytes,
and
often
we
generate
the
data
which
is
close
to
one
terabyte
coming
from
one
experiment
which
we
are
generating
in
a
few
hours.
Why
is
it
so
big?
Because
every
channel
corresponds
to
a
particular
molecule?
And
here
I'm
showing
you
just
one
example
from
a
public
data
set
from
a
mouse
section
which
is
used
by
gen
and
pack,
a
pharma
company
in
the
usa
for
drug
development
and
what
you
are
looking
at
here?
B
You
are
looking
at
the
localization
of
four
different
molecules
across
this
tissue
section,
and
why
is
it
important?
Because
by
getting
this
information
we
understand?
Where
does
the
drug
go?
What
does
it
do?
Does
it
have
any
side
effects
and
if
there's
side
effects
happening
where
action
with
which
organ
they
are
associating,
and
this
is
just
one
application
of
this
technology,
so
our
team
at
amble
over
the
past
years
we
have
developed,
metaspace
and
metaspace
is
a
very
core
of
this
presentation.
B
So
let
me
take
one
slide
to
explain
it,
so
it
is
a
cloud
engine
and
also
a
knowledge
base
where
a
lot
of
users
from
across
the
globe
they
put
their
data
in
there
is
some
computation
going
on
there
on
high
performance
computing.
They
get
images,
they
look
through
the
images
they
exchange,
they
publish
them
and
so
on
so
forth,
and
they
also
allow
others
to
to
use
them
every
use
as
public
data.
B
So
we
have
a
number
of
submissions
more
than
ten
thousand
submissions
over
the
past
three
years,
many
users,
many
labs
most
of
the
labs
in
our
field,
use
this
essential
resource.
They
publish
results,
they
get
new
biology,
they
get
new
medicine,
so
it's
pretty
pretty
important
platform
in
our
field,
which
enables
data
analysis,
data,
reuse,
data
sharing
and
overall,
open
science
in
biology
and
medicine.
B
So
let
me
show
you
how
it
looks
just
to
get
you
the
feeling
that
about
what's
real
behind
it,
so
you
can,
you
can
go
to
metaspace2020.edu,
you
can
select
one
of
the
projects
or
folders
that
we
have
there
here.
It's
exactly
that
project
with
the
public
data
from
pharma
companies.
You
can
pick
one
data
set
here
and
once
you
pick
it,
then
you
will
see
a
lot
of
molecules
which
were
found
by
our
engine
hidden
in
this
data,
and
now
you
can
flip
go
through
this
data
molecule
by
molecule.
B
B
So
recently
we
got
some
need
an
appetite
for
new
computer
technologies
used
to
be
used
in
metal
space
and
why?
Because
the
platform
is
actually
growing
now,
so
there
is
a
pretty
good
uptake
in
the
field
and
there
is
also
growth
in
the
number
of
submissions
and
in
particular,
of
the
past
half
a
year.
We
experience
super
linear
growth
in
the
number
of
submissions
and
also
what's
very
special
about
our
processing.
Is
that
the
submissions
they
come
irregularly?
B
So
sometimes
there
is
like
hundreds
of
them.
Sometimes
there
is
nothing
so
we
experienced
a
few
pain
points
in
our
software
system
and
one
of
is
deployment
delay
so
scientists
they
want
to
see
results
immediately
and
if
we
need
to
wait
for
the
deployment
of
some
system,
we
use
apache
spark
in
the
past.
It's
actually
not
very
convenient.
The
second
one
is
infrastructure
management
and
queue.
B
If
some
data
set,
we
got
stuck
in
the
queue
we
need
to
go.
We
need
to
fix
it
because,
because
it's
kind
of
painstaking
process
and
also
resource
planning,
because
we
don't
really
know
how
many
data
sets
will
be
coming
because
again,
it's
used
by
the
community-
and
sometimes
there
is
demand.
Sometimes
not
all.
This
brought
us
to
to
motivate
motivated
us
to
actually
search
for
a
new
solution
in
particular
because
our
computing
problem
and
the
process
is
embarrassingly
parallel.
B
And
here
you
see
the
listing
people
who
are
working
on
this
project,
gilvernick
from
ibm
and
lachlan
stewart
from
our
team
and
what
they've
been
working
on
to
apply
the
technology
which
was
developed
by
gil
and
others
called
lithops,
which
is
serverless
computing
framework
and
a
key
result
in
this
project
to
apply
it
for
our
system
metaspace
and
in
the
next.
You
will
see
how
these
three
pieces
are
connected:
our
platform
cloud
engine,
metal,
space
lithops
as
serverless
framework
and
code
engine
from
ibm
over
to
you,
doug.
A
Thank
you,
theo.
Okay,
before
we
actually
start
talking
about
a
little
bit
of
a
deep
dive
into
code
engine
itself,
I
think
it's
important
to
talk
a
little
bit
about
sort
of
what
led
up
to
the
creation
of
code
engine
itself
as
a
platform
in
which
to
run
different
types
of
workloads,
and
you
actually
saw
a
lot
of
this
already
with
what
few
was
saying
and
I'd
like
to
look
at
it,
sort
of
like
lessons,
learned
or
state
of
the
community
kind
of
a
thing
right.
A
We
realize
that
users
or
developers
really
want
to
focus
on
producing
value
either
for
their
customers
or
to
get
their
job
done
whatever.
That
actually
is
right.
They
don't
really
want
to
work
on
managing
and
setting
up
infrastructure
right
to
them.
Infrastructure
is
just
a
means
to
an
end.
It's
just
a
tool
that
they
need
to
use
to
get
their
job
done
for
the
most
part.
A
They
don't
even
want
to
think
about
it
right,
they're
there
to
get
something
finished
and
completed
okay,
so
the
entire
idea
of
being
asked
to
manage,
learn
whatever
you
want
to
call
it.
The
infrastructure
is
really
like.
I
said
on
here
an
undesirable
burden
right,
it's
a
nuisance
to
them
for
the
most
part,
because
it's
a
distraction.
Okay,
now
from
this
k-native
community,
I'm
sorry,
not
canadian.
From
the
cloud
native
community
perspective,
the
the
community
itself
is
also
starting
to
realize
this
as
well
right.
Managing
infrastructure
is
actually
hard
regardless.
A
What
that
infrastructure
is
whether
it's
spark
computing
like
a
lot
of
ai
systems
or
machine
learning,
type
systems,
use
or
kubernetes
itself.
Those
are
wonderful
systems,
but
they're
they're,
actually
kind
of
complex,
especially
for
people
who
aren't
necessarily
developers
right,
they're
they're
people
who
have
other
jobs
to
get
done
like
research,
scientists
and
stuff
like
we're
seeing
right.
So
it's
difficult
for
them
to
to
come
on
board
to
actually
use
these
things
and,
of
course,
there's
this
other
problem.
A
Where
there
are
many
different
platforms
out
there
to
get
your
job
done
right,
you
have
generic
kubernetes,
you
have
things
like
serverless
environments,
stuff
like
that,
and
each
platform
is
actually
going
to
present
a
different
set
of
features
and
with
those
different
features,
also
come
different
sets
of
constraints,
so
oftentimes
people
feel
the
need
to
conform.
Their
design
of
their
workload
to
the
need
to
what
the
platform
sort
of
boxes
them
into
and
that's
a
bit
of
a
problem,
because
that
means
they
don't
necessarily
pick
the
right
solution
for
their
job.
A
Okay,
and
so,
in
our
point
of
view,
really
what
it
means
is
developers
should
code,
not
manage
infrastructure.
If
you
get
down
to
the
brass
tax
of
the
thing,
okay
and
ibm
all
right
from
ibm's
per
side,
we
have
a.
We
have
a
better
solution
to
try
to
solve
this.
Okay
and
that's
what
code
ends
all
about.
A
So
we
looked
at
this
problem
and
said
what,
if,
at
a
high
level,
the
developer
could
basically
focus
on
just
their
workload,
meaning
their
function,
their
application,
their
batch
job,
whatever
the
workload
is
as
long
as
they
can
containerize
it
in
some
fashion.
That's
all!
They
should
really
focus
on
okay
now,
whether
they
give
that
to
us
as
source
code
or
as
a
pre-built
container
image
doesn't
really
matter
there.
They
choose
what
they
want
to
give
it
to
us
right,
but
along
with
that,
if
they
give
us
the
runtime
semantics
that
they
want
right.
A
A
It's
now
the
workloads
job
to
properly
host
and
manage
the
workload,
the
infrastructure
that
goes
around
it
and
everything
else
now.
Obviously,
that's
not
quite
all
you
need
right
as
a
workload.
I'm
sorry
yeah,
you
oftentimes
workloads
need
to
actually
connect
up
to
things
like
managed
services,
so
we
need
to
make
sure
you
can
do
that.
A
A
lot
of
these
things
are
going
to
respond
to
incoming
messages
and
stuff
like
that.
So
obviously
you
have
to
have
networking
that
does
automatic
setup,
for
you
scaling
traffic
splitting
if
you
can
do
blue
green
deployment
type
stuff
or
upgrades
that
kind
of
stuff.
If
you're
connecting
up
to
event
producers
you're
gonna,
have
to
have
some
sort
of
event
orchestration
engine
in
there,
even
in
the
simple
case
of
deliver
the
events
just
once
to
one
particular
destination,
that's
the
simple
case.
A
But
what,
if
you
want
that
one
event
to
get
fanned
out
to
lots
of
different
services
you
have
running
in
there
right?
You
need
some
sort
of
orchestrator
at
some
point,
but
then,
of
course,
as
the
backbone
of
all
this
your
course
needs
security
and
compliance
built
into
the
system.
So
the
end
user
or
the
developer
doesn't
need
to
think
about
it.
It's
just
there
right,
so
this
is
sort
of
our
dream
platform
from
an
abstract
perspective.
But
of
course
the
main
point
here
is
the
stuff
in
sort
of
this
pinkish
box
left-hand
side.
A
That's
all
developer
needs
to
worry
about
right.
They
just
worry
about
their
workload.
From
a
coding
perspective,
a
container
image
perspective
and
what
is
the
runtime
behavior?
They
want
everything
else
on
the
right-hand
side
should
be
basically
hidden
from
them
and
just
manage
under
the
covers
for
them,
and
that's
exactly
what
code
engine
is
doing
for
them
and
that's
what
you
see
theo
basically
talking
about
here.
A
Okay,
so
let
me
just
sort
of
quickly
mention
in
a
little
more
detail
exactly
what
code
engine
does
for
you,
because
code
engine
is
a
managed,
hosted
environment
in
the
cloud
okay
and
with
code
engine
you
give
us
your
source
code,
and
what
do
you
get
for
applications,
meaning
things
that
actually
respond
to
http
request?
You
get
an
internet
exposed
securely
hosted
workload.
A
Now,
if
you
don't
want
to
be
internet
exposed,
you
don't
have
to.
If
you
don't
want
to,
you
can
make
it
private,
which
means
it's
only
available
to
other
things
inside
your
workspace.
Okay,
that
workload
that
will
automatically
scale
up
and
down,
based
upon
the
load
coming
in
even
down
to
zero,
which
means,
if
it's
not
being
used
at
that
exact
moment
scale,
is
down
to
zero.
You
don't
pay
for
it.
A
You
have
zero
down
times
for
upgrades
right,
we'll
slowly
shift
the
traffic
over
from
your
old
version
to
the
new
one,
so
your
users
of
your
application
never
see
at
the
shift
over
or
never
see
a
downtime
between
the
shift.
Okay,
if
you
happen
to
have
a
workload
that
doesn't
respond
to
hp
requests,
this
is
what
typical
batch
jobs
right.
These
are
jobs
where
your
container
comes
up,
does
a
particular
thing
and
then
goes
away
when
it's
done
data
processing
type
stuff,
exactly
what
theo
is
doing
in
his
workloads
right,
those
are
batch
jobs.
A
You
can
scale
those
up
up
as
well.
Right.
Just
tell
how
big
you
want
what
size!
What
is
the
size
of
your
workload,
we'll
scale
it
bring
up
the
resources
when
it's
done
scale
it
back
down?
You
don't
need
to
worry
about
it.
Just
tell
us
what
you
want.
Okay,
obviously,
you
can
then
connect
the
host
services,
but,
of
course,
one
of
the
key
things
here
is
you
only
pay
when
your
code
is
actually
running
right,
your
batch
job
scales
down
to
zero,
because
it's
not
it's
all
done.
You
stop
paying
your
internet-facing
application.
A
Okay,
of
course,
there's
a
there's,
a
simplified
user
experience
around
all
this
which
we,
which
we
obviously
really
like,
and
so
it's
that's
that's
part
of
the
height
of
the
infrastructure
from
you,
but
obviously
you
could
need
to
do
some
customizations,
and
these
are
those
runtime
semantics
that
I
talked
about
right.
How
do
you
want
your
application
to
scale
from
a
min
and
max
perspective?
When
do
you
want
to
scale
right?
What
is
your
resource
usage
memory
cpu,
those
kind
of
stuffs?
A
How
do
you
want
the
traffic
splitting
handled
all
those
kind
of
things
you
still
have
available
to
you
right
through
our
simplified
user
experience?
Okay
and,
of
course,
because
it's
all
in
one
particular
platform,
regardless
of
the
type
of
workload
you
have
applications
functions,
source
code,
batch
jobs,
they
all
live
together,
which
means
you
get
built
in
security
from
a
networking
perspective.
They
can
all
talk
to
one
another
securely
and
you
don't
have
to
figure
out
how
to
make
those
happen
because
they're
not
on
separate
platforms,
anymore.
A
Okay
and
just
to
sort
of
wrap
this
thing
up
here,
because
I
really
like
this
slide.
I
think
it
presents
everything
in
a
nice,
simple
fashion,
right
with
code
engine
as
a
developer,
you
can
deploy
any
type
of
application,
whether
it's
a
container
batch
job,
your
source
code,
your
function
on
a
single
unified
platform
without
provisioning,
managing
securing
configuring
anything
in
the
infrastructure,
meaning
clusters,
networks,
vm
certificates,
you
name
it
it's
all
hidden
from
you,
you
don't
need
to
touch
it
and
best
of
all
you
only
pay
when
your
application
or
workload
is
actually
active.
A
B
Thanks
zach,
so
let
me
wrap
up
this
talk,
so
I
would
like
to
share
our
story
and
how
actually
both
stories
of
us
developing
this
serverless
technology
for
our
needs
and
the
code
engine
which
was
recently
released
converged
into
this
trio.
So
what
we?
What
we
got
and
what
we
learned
out
of
this
is
that
right
now
we
have
implemented
technology
which
provides
scalable
solution
for
science
and
our
users
from
science,
from
academia
from
industry.
B
For
us,
it
was
very
nice
and
also
will
be
very
useful
in
the
future,
because,
right
now
we
have
less
infrastructure
overhead
overhead.
In
terms
of
our
time,
in
terms
of
cost
and
in
terms
of
delays
as
well,
and
also
we
don't
need
to
plan
for
the
resources
ahead,
and
this
is
very
important
for
us,
because
we
can
actually
use
and
provide
the
system
to
our
users
as
they
go.
B
And
I
think
it
was
very
nice
that
we
got
this
opportunity
to
work
with
a
very
talented
team
at
ibm
who
not
only
set
up
code
engine
which,
which
is
great
for
us,
because
it
provided
a
success
to
native
support
for
serverless,
but
also
the
on-demand
scaling.
Definitely
pay,
as
you
go
very,
very
handy
feature
and
easier
for
us
to
code,
because
there
are
no
or
minimal
artificial
constraints
on
memory
and
cpu,
and
this
link
by
technology
framework,
lithops
and
temperature.
B
There
are
many
other
technologies
which
allow
to
use
the
power
of
code
engine,
but
for
us,
lithops
was
instrumental
because
it
kind
of
removed
all
these
bottlenecks
and
simplified.
The
coding
files
allowed
us
to
really
hide
the
complexity
and
focus
our
business
logic,
actual
scientific
process
and
scientific
algorithms,
and
also
hide
the
complexity
of
the
runtime
platform
and
all
the
overhead
connected
to
it.
B
With
this,
I
would
like
to
thank
everyone
at
the
ibm
team
who
contributed
to
this
work,
and
particularly
gil
vernick,
and
was
also
pleasured
here
to
work
on
this
with
the
rest
of
ibm
team
with
a
doc
on
on
this
work,
and
if
you
are
interested,
you
can
check
out
our
joint
blog
post
decoding
dark
molecule
matter
in
spatial
metabolomics
with
ibm
cloud
functions,
which
was
for
the
previous
generation,
and
now
it
will
replace
decoding
the
dark
molecular
meter
with
a
ibm
code
engine.
Thank
you
for
your
attention.
A
All
right
so
sdo
said
we'll
just
jump
into
the
q
a
now.
Just
one
final
note:
if
you
are
interested
in
code
engine,
you
can
stop
by
the
ibm
booth
here
at
kubecon
and
we'll
answer
any
of
your
questions
for
you
and,
of
course,
there's
a
link
here
on
the
screen.
You
can
actually
go
play
with
it
yourself
at
cloud.ibm.com
code,
engine,
okay,
and
with
that
we'll
now
jump
into
the
live
q.
A
portion.
Thank
you.