►
From YouTube: ArgoCon '21 - Processing petabytes in Python w/ Argo Workflows & Dask (JP Zivalich & Severin Ryberg)
Description
Companies with complex computational workloads often use Python packages such as NumPy, Pandas, and Scikit-Learn, but unfortunately, they don't scale up well for especially large data sets. Dask makes it easy to use these tools on large data sets in distributed environments with low latency. Argo Workflows is the best way to run pipelines on Kubernetes but incurs high overhead when tasks are short-lived. This talk demonstrates how to orchestrate these two technologies to achieve the Kubernetes-native scheduling and automation available from Argo Workflows with the low-latency scalability of Dask.
To do so, we'll show how Pipekit provisioned Argo Workflows for ACCURE Battery Intelligence and show how ACCURE uses Dask running on Argo to process petabytes of data with super low latency for their customers.
We hope that listeners of this talk will learn how to orchestrate their Dask workloads with Argo Workflows and additionally gain insights for orchestrating other workloads with Argo such as Spark jobs.
A
A
Severin
is
the
senior
infrastructure
architect
at
acura
battery
intelligence,
working
mainly
in
the
areas
of
cloud
data
and
devops
engineering,
before
that
he
was
an
academic
researcher
in
the
field
of
data-driven
energy
systems,
analytics
in
particular
the
integration
of
renewable
energy
sources
into
large-scale
power
grids.
That
sounds
really
interesting.
Welcome.
Gentlemen.
C
C
B
B
B
So
first
we
want
to
understand
how
to
run
highly
parallelized
frameworks
on
argo
workflows.
Today,
we're
going
to
be
focusing
on
desk
because
that's
what
a
cure
battery
intelligence
is
using,
but
similar
frameworks
that
you
might
be
using
at
your
organization
might
include
spark
hadoop
or
any
one
of
these
sorts
of
types
of
things.
B
Next,
we'll
talk
about
provisioning,
argo
workflows
and
the
needed
cloud
and
kubernetes
resources.
After
that,
severum
will
go
on
a
deep
dive
about
the
desk
setup,
we'll
be
following
that
with
an
interactive
demo,
and
then
several
will
give
his
concluding
remarks
so
I'll.
Kick
it
to
sev
to
talk
about
the
requirements.
C
Yeah
so
before
we
even
thought
about
which
tool
we
would
want
to
use
to
service
our
data
processing
at
a
cure.
We
of
course
had
some
boxes
that
we
knew
that
we
needed
to
check.
So
we
knew
that
we
wanted
to
stay
in
python
and
we
wanted
to
avoid
this
situation
where
we
switched
between
python
and
other
environments,
such
as
how
spark
does,
in
the
back
end,
switching
between
python
and
the
jvm.
C
C
We
also
knew
that
we
wanted
to
have
the
same
if
not
a
similar
development
experience
between
both
testing
and
production
environments
and
what
this
really
comes
down
to
is
we
want
to
write
code
once
and
have
it
work,
whether
we're
not
whether
we're
invoking
it
in
a
sequential
situation
where
we
just
have
one
computer,
where
everything
is
running
sequentially
all
the
way
up
to.
We
have
a
distributed
cluster
of
computers
that
are
all
working
together
on
the.
D
C
D
C
Help
to
solve
all
the
same
time
and,
lastly,
to
you
know,
reduce
friction
within
our
organization.
We
wanted
to
promote
self-service,
especially
for
our
desk
engineers,
or
rather
excuse
me
data
engineers
whenever
possible.
So
we
wanted
to
remove
bottlenecks,
such
as
a
data
engineer
needing
to
talk
with
someone
on
an
infrastructure
team
to
provision
cloud.
C
B
Thank
you
severn
I'll,
be
going
over
the
infrastructure
and
security
requirements
that
a
cure
battery
intelligence
had.
The
first
is
ultra-low
latency
parallelization,
so
it
cures
running
workflows
that
spin
up
a
ton
of
different
like
subtasks
and
sub
jobs
and
within
a
pure
argo
kind
of
mindset
or
cloud
native
mindset.
B
So
specifically,
they
need
to
be
running
different
customers
workflows
on
the
same
kubernetes
cluster,
but
they
need
to
have
good
separation
amongst
the
different
workflows
so
that
one
workflow
doesn't
have
the
ability
to
access
resources
that
belong
to
a
different
customer
and
then,
lastly,
this
all
needs
to
be
production
ready.
So
we
need
things
like
good
logging,
that's
accessible
and
it
cures
elk
stack,
highly
dependent
or
super
dependable
scheduling
and
workflow
run
archiving.
So
if
there's
anything
that
needs
to
be
troubleshooted,
that
a
cure
can
do
that.
B
So
next
I'll
do
a
little
bit
of
a
deep
dive
on
provisioning,
hargo,
workflows
and
the
other
cloud
resources.
So,
first
in
our
provisioning
we're
doing
infrastructure
as
code
a
akure
battery
intelligence
is
using
gitlab
to
store
their
workflow
configuration,
kubernetes
configuration
and
the
pipe
kit
daemon.
That
does
some
of
the
managing
of
argo
workflows
for
the
actual.
B
But
these
individual
workflows
don't
have
the
ability
to
access
other
resources
within
that
kubernetes
cluster.
So
that
brings
us
into
that
customer
specific
namespace
thing.
Additionally,
I
want
to
call
out
that
they're,
using
cron
workflows
to
trigger
workflow
templates
in
each
of
those
cron
workflows
are
sitting
in
each
customer's
namespace,
and
you
can
get
very
custom
about
what
schedule
you
want
them
to
run
on.
B
B
If
we
look
in
the
middle
we've
labeled
this
cluster
scope,
so
some
of
these
are
like
truly
cluster
scope.
Things
like
the
cluster
workflow
template
where
they
really
don't
sit
in
a
namespace.
The
argo
controller
and
argo
server
do
sit
in
a
namespace,
either
the
default
namespace
or
in
the
argo
namespace.
B
But
the
broader
point
is
that
they're
installed
with
the
cluster
scope,
meaning
that
they
have
the
ability
to
orchestrate
different
workflows,
running
in
different
namespaces
and
then.
Lastly,
there
are
several
configuration
namespaces
for
like
logging,
ingress,
metrics
things
like
that,
and
also
we
archive
the
workflow
runs
in
an
aws
hosted
rds
postgres
database,
and
we
do
that
just
because
doing
data
persistence
on
kubernetes
still
is
like
a
little
bit
tricky
and
it's
better
to
just
offload
that
sort
of
thing
to
others
for
convenience.
D
D
D
C
Seems
to
be
so,
that's
obviously
the
the
the
choice
that
a
cure
made
in
the
very
end
in
terms
of
how
we
want
to
scale
out
our
data
pipelines
and
what
desk
is
for
those
of
you
who
aren't
aware
is
it's:
it
allows
for
high
throughput
data
pipelines
completely
in
python,
so
desk
is
developed
in
python
and
it
is
python
the
whole
way
through.
So
it
checks
our
backup
or
a
box
of
staying
in
python,
and
it
comes
out
of
the
box
with
a
lot
of
really
nice
features.
C
So
it
allows
for
this
multi-domain
execution
again
going
from
sequential
execution
on
the
single
machine
all
the
way
up
to
distributed
execution
across
many
machines
in
more
or
less
whatever
kind
of
cluster
you
can
imagine,
and
it
also
allows
for
both
the
the
batch
processing
context,
as
well
as
the
shared
parallel
processing
context,
and
especially
if,
through
the
thing
called
the
futures
interface
of
task.
This
is
what
it
allows
for
the
batch
processing.
This
is
what.
C
B
C
D
C
C
Actually
interested
in
not
having
any
sort
of,
or
rather
dealing
with
any
sort
of
infrastructure
stuff
at
all.
There's
also
services
that
can
provision
clusters
for
you
on
your
behalf
and
akira,
is
also
a
happy
customer
of
a
service
called
coiled,
which
does
this
for
us
there's
a
few
other
options
in
this
space
as
well,
such
as
saturn
cloud,
which
I
don't
particularly
have
experience
with,
but
they're
really
nice
for
the
workers
who
don't
really
know
so
much
about
kubernetes
and
cloud,
and
it
meshes
well
with
allowing.
C
The
things
that
need
to
get
done,
but
of
course
the
desk
doesn't
have
everything
accurate
did
have
to
implement
a
few
things
around
the
desk
tool.
So
one
of
those
things
was
work
avoidance.
So
when
we
execute
the
same
task
with
the
same
data
twice
or
you
know
more
than
twice,
there's
no
point
in
actually
recalculating
that
task,
because
we
already
have
the
result
for
it.
So
we'd
rather
just.
D
C
C
B
C
Pipeline
configuration
which
wraps
around
a
more
generic
standard
desk
scheduler
and
the
pipeline
configuration,
is
something
that
we
would
use
for
one
particular
product
for
one
particular
customer.
So
you
know
one
example
might
be
computing
the
state
of
health
of
all
batteries
for
customer.
You
know
what
whatever
and
next
next
slide
please.
C
D
C
Into
the
cluster
workflow
templates,
it's
a
fairly
simple
template,
at
least
in
its
simplest
sense.
So
actually
it
just
has
four
or
five
pieces
to
it.
It
begins,
of
course,
with
initiating
the
workflow
just
doing
basic
argo
type
stuff,
and
then
the
first
node
in
the
pipeline
would
actually
be
creating
the
das
scheduler.
D
C
D
C
C
Argo
workflows,
it's
a
fairly
straightforward
configuration
again.
It's
also
in
our
example
in
case
you'd
like
to
see
how
to
do
that,
but
basically
for
the
desk
worker
deployment.
This
will
be
a
resource
template
which
will
create
a
deployment
which
spins
up.
However,
many
number
of
task
worker
pods,
you
would
like
to
use
in
your
cluster.
C
Be
a
daemon,
of
course,
so
that
they
stay
alive
until
the
end
of
the
workflow
on
the
right.
You
also
see
the
primary
pipeline.
This
is
just
a
pod
which
executes
a
python
script
that
does
nasty
stuff
inside
of
it,
and
that
will
talk
to
the
scheduler
which
talks
to
the
workers
and
you
know,
distribute
workloads
once
the
primary
pipeline
finishes.
C
Make
sure
there's
no
orphaned
resources
left
over
on
your
cluster,
so
all
the
pieces
in
yellow
are
actually
all
the
same
image.
So
if
we
go
to
the
next
slide,
we
can
look
into
that
a
little
bit,
and
so
this
would
be
an
image
which
is
specific
to
one
particular
product
and
there's
really
not
much
going
on
inside
of
this,
you
know,
for
example,
it
could
be
just
built
on
top
of
the
python
388
buster
base
image.
C
You
just
need
to
pip
install
things
like
the
desk
module
distributed
in
boca
as
core
requirements,
and
then,
whatever
other
python
modules,
your
python
pipeline
or
tasks
need
you
then
can
do
things
like
activating
environments.
If
you
need
to
you
have
to
make
sure
that
the
desk,
scheduler
and
desk
worker
executables
are
available
on
the
path
and
then,
of
course,
you
just
need
to
add
your
your
python
script
and
then
to
make
all
of
this
generic
next
slide.
D
C
D
C
That
is
more
or
less
clear.
We
can
hop
into
the
example
so
for
the
interactive
demo.
We
have
a
fairly
simple
example
for
you
today.
Basically,
what
we'll
be
doing
is
we'll
be.
We
will
look
at
time
series
data
in
spain
and
time
series,
weather
data
specifically
for
the
cities
of
madrid,
valencia,
barcelona,
sevilla
and,
and
basically
we
want
to.
We
want
to
answer
the
question
which
of
these
cities.
C
A
B
Great
I'll
have
my
co-founder
kalyn.
If
you
would
please
post
the
link
to
the
repository
in
the
live
chat,
we've
got
a
link
here,
but
what's
it,
this
will
take
you
to
a
github
repository
that
contains
info
about
this
demo
or
the
sample
code
and
the
workflows
specifically.
It
contains
first,
a
sample,
argo
workflows,
installation,
a
note
of
caution,
this
workload
that
we're
about
to
run
is
too
heavy
to
run
locally
on,
like
docker,
desktop,
mini
minicoop,
etc.
We
tried
several
times
with
pretty
good
computers
and
were
unsuccessfully
able
to
do
so.
B
So
we
recommend
using
your
cloud
kubernetes,
offering
whether
that's
amazon,
google,
azure
digitalocean.
I
don't
think
it
really
matters.
Additionally,
it
has
the
cluster
workflow
template
containing
a
das
pipeline,
the
workflow
template
that
will
invoke
that
cluster
workflow
template
and
then
a
dockerized
python
pipeline
that
will
schedule
tasks
on
the
das
cluster
itself.
B
C
And
then
next
we
will
also
create
the
pipeline
specific
workflow
template,
assuming
we
can
spell
right
and
so,
along
with
the
template,
we'll
also
create
a
config
map
that
limits
the
parallelism
of
this
template
to
only
one
at
a
time
and
we'll
also
create
a
cron
workflow
scheduling,
because
this
is,
in
my
opinion,
one
of
the
nice
aspects
of
mixing
argo
and
ask
together.
So
here.
C
C
C
C
C
All
right
so
here
you
see
the
dashboard.
This
pipeline
currently
has
four
workers
configured
and
you
see
those
running,
and
this
really
highlights
where
desk
and
argo
are
a
very
nice
combination.
In
my
opinion,
yes,
we
have
a
thousand
time
steps
scheduled,
that's
where
that
thousand
comes
from
yes,
but
this
really
highlights
where
desk
and
argo
are
is
a
very
nice
combination,
because
each
of
these
tasks
are
only
several
hundred
milliseconds
long.
So.
D
D
C
Thank
you.
Yes,
I
see
the
dashboard
looks
beautiful.
This
doesn't
come
from
me.
This
comes
from
the
desk
people
and
yeah.
I
could
take
credit
for
it
yeah.
So
back
in
the
pipeline,
you
see
how
the
once
the
pipeline
finishes.
The
teardown
will
begin
on
the
desk
workers
and
if
we
want
to
see
which
is
the
windiest
city,
we
just
got
to
check
out
the
main
logs,
and
you
see
that
the
windiest
city
is
valencia.
I
would
have
guessed
at
least
for
the
first
thousand
time
steps
of
our
data
set.
D
C
D
D
C
C
C
Yes
and
desk
is
really
nice
in
the
sense
that
it
actually
will
respond
dynamically
to
new
workers
coming
online.
So
if
we
actually
take
a
look
right
now
at
the
pods
inside
of
this
namespace,
you
can
see
that
we
have
still
50
nodes,
but
many
of
them
are
in
the
pending
or
container
creating
states
and
they're
not
yet
active
on
our
pipeline.
C
D
C
C
Another
nice
feature
of
mixing
argo
plus
desk,
and
is
that
since
we
have
this
parallelization
limit
set
inside
of
argo,
if
we
were
to
now
submit
the
same
workflow
several
times,
of
course,
these
will
stay
in
the
pending
state
and
they
won't
actually
start
running
anything.
So
if
these
tasks
were,
for
example,
talking
to
a
database
or
hitting
an
api,
something
like
that,
you
would
be
protected
from
you
know,
overwhelming
your
database
or.
C
B
C
This
is,
of
course,
a
very,
very
simple
example,
but
you
know
desk
can
scale
up
easily
to
hundreds
of
workers
and
the
size
of
those
workers
is
just
a
pod
configuration,
so
they
can
be
as
large
or
as
small
as
you
want
really
limited
by
the
cube
right.
This
cluster
that
you're
working
with
and
the
functions
themselves
can
be
total
black
box
functions,
so
you
could
put
whatever
simple
or
complex.
D
C
C
D
C
You
know
well-defined
type
scenarios
versus
exploratory
ad-hoc
scenarios
and
when
we're
in
this
scheduling
automated
workflow
space,
you
know
we
really
feel
like
this.
Argo
plus
task
is
great
because
we
control
all
the
variables
we
control
when
pipelines
run,
we
control
how
big
the
workers
are,
and
we
can
do
you
know
we
can
add
other
things.
Like
noted,
like
slack
notifications,
and
you
know
other
contingencies
to
our
pipeline,
you
know
making
a
more
complex
standard
desk
scalar
workflow,
but
when
we
have
humans
in
the
loop
you
know
such
as,
if
a
data.
C
Then
we
find
that
these
das,
as
a
service
type
tools
that
I
mentioned
previously,
tend
to
be
better,
tend
to
be
better
suited
for
this
sort
of
task.
So
we
found
that
this
tends
to
provide
for
faster
development,
iteration
speeds
and
also
it
empowers
our
beta
engineers
to
completely
service
themselves.
In
terms
of
you
know,
setting
up
large
clusters
and
getting
them
working
for
themselves
and.
D
A
C
And
but
you
know
there
are
some
skeletons
in
the
closet,
so
one
of
the
things
that
we've
noticed
of
that
is
that
there
are
some
instabilities
for
particularly
long
running
tasks
and
in
general,
we've
seen
that
tasks
that
take
hours
are
pretty
much
okay,
whereas
tasks
that
take
days
are
questionable.
You
know.
D
C
C
A
Yeah,
that
was
great
guys.
I
appreciate
that
we've
got
some
questions
that
are
over
in
your
session
chat.
So
if
you
want
to
jump
over
there
and
answer
them
or
there's
questions
that
are
streaming
in
through
the
cncf
slack
channels,
you
can
find
those
questions
there.
That
demo
was
really
really
entertaining.
So
thank
you
both
and
I'm
from
I'm
from
texas,
so
the
texas
tech
reference
I
was
like.
Yes,
that's
amazing
all
right,
gentlemen.
A
Thank
you
both
for
your
time
today,
those
of
you
that
are
so
in
the
audience,
if
you
haven't
filled
out
the
survey,
please
do
so.
We
want
to
know
how
you're
using
argo
right
now
and
in
the
future,
and
then
we
also
want
to
make
argo
con
2022
amazing
for
you.
So
let
your
voice
be
heard
and
then
join
the
chat
thanks
y'all.