►
From YouTube: Large Scale Batch Processing with Argo Work... Rakesh Subramanian Suresh & Saravanan Balasubramanian
Description
Large Scale Batch Processing with Argo Workflows and Events - Rakesh Subramanian Suresh & Saravanan Balasubramanian, Intuit
Argo Workflows is Kubernetes native workflow engine, it is a good fit for batch processing jobs running in Kubernetes. In Intuit, our Batch Processing Platform manages ~40,000 data processing pipelines by using Argo Workflows, 10% of these pipelines are running concurrently each every day. All of these pipelines are scheduled or triggered in an event-driven fashion by using Argo Events, that the workflow executions are based on calendar events, or a logical combination of calendar events, status of upstream pipeline workflows, or events in data like file/database update, REST calls, etc. In this session, we will discuss how we run 40K pipeline workflows in Kubernetes clusters, and share how we use Argo Events to manage ~400,000 different sorts of events for pipeline workflows, and achieve exactly-once triggering.
A
So
welcome
to
our
presentation
hope
you
guys
had
a
very
nice
morning
very
interesting
keynote
there
we're
going
to
talk
about
large-scale
batch
processing
with
Argo
workflow
and
events
myself,
Rakesh
I
work
as
a
back-end
engineer
in
intuits
data
processing
organization
and
I
have
with
me
a
co-presenter
Bala
he's
the
MVP
he's
the
core
contributor
and
developer
of
Argo
workflow.
So
give
him
a
good
shout
out,
so
you
guys
can
bug
him
with
a
lot
of
questions
so
right
after
the
presentation.
A
As
you
guys
are
aware
of
it's
a
global
technology
platform
that
helps
customers
Achieve
Financial
confidence,
we
have
over
100
million
customers
in
the
across
the
world.
14
000
employees
working
really
hard
to
serve
those
customers
and
we
are
a
strong
open
source
contributor,
both
in
promoting
and
also
Building
open
source
projects
and
getting
it
out
there.
A
So,
in
this
slide,
we'll
start
with
the
batch
processing
platform
overview,
starting
with
some
personas
who's.
The
target
audience
that
we
built
the
presentation
for
the
platform
for
and
so
before
we
Deep
dive
I
wanted
to
briefly
touch
upon
some
of
the
personas
that
the
processing
platform
helps
with
data
engineer,
who
wants
to
focus
on
transforming
data
rapidly
so
that
they
can
Aid
their
customers?
A
data
scientist
who
wants
to
build
ml
models
at
a
faster
rate
and
make
their
applications
more
intelligent
and
platform
Engineers?
A
A
So,
let's
talk
about
in
this
slide,
we'll
talk
about
the
intuits
data
processing
flow
from
a
very
high
level,
so
Intuit
has
several
customer
facing
products
with
millions
of
users,
be
it
small
businesses,
tax
filing
and
also
for
verifying
your
financial
health,
so
between
the
products
before
the
data
with
the
data
that
we
get.
There
are
a
lot
of
use
cases
that
we
serve
with
the
data.
We
need
data
storage,
real-time
recommendations
to
the
users,
data
enrichment,
their
accuration
fraud
detection,
so
from
the
product
lineup.
A
Before
the
data
arrives
and
into
its
data
Lake,
we
have
a
real-time
Transformations
done
using
our
stream
processing
platform
and
once
the
data
arrives
in
the
data
Lake,
we
have
batch
platforms
performing
post-processing
and
Analysis,
which,
from
which
we'll
get
reporting
model,
training,
feature
creation
and
other
data
that
can
be
used
for
real
time
and
semi.
Real-Time
feedbacks
back
to
the
products.
A
So
with
that
in
mind,
I
will
talk
about
the
intuits
batch
processing
requirements
from
a
platform
standpoint.
We
wanted
to
build
a
company-wide
solutioning
for
scheduling
and
orchestration.
A
lot
of
the
teams
want
to
have
data
pipelines
or
applications
running
on
a
schedule,
and
data
is
usually
interconnected.
So
we
wanted
to
provide
dependency
management
for
data
pipelines
and
applications.
We
wanted
to
provide
a
standardized
run
to
runtime
compute
layer
for
data
and
other
processing
use
cases
in
into
batch
processing.
We
support,
spark
and
containerized
application
runtimes
in
multiple
kubernetes
clusters
that
we
manage.
A
We
provide
the
seamless
experience
to
the
users
to
build
data
pipelines,
build
contentious
applications,
deploy
them
and
schedule
them
based
on
different
dependencies,
so
Intuit
manager,
60,
000
pipelines
and
10
000,
concurrent
pipelines,
run
at
any
moment,
so
the
platform
is
also
catered
to
solving
a
lot
of
engineering
problems,
so
the
impact
this
has
is
over
1500
Engineers,
including
the
personas
that
we
talked
about.
Besides,
that
there's
also
other
engineering
personnels
that
are
impacted
using
this
platform.
A
All
right
so
we'll
move
on
to
the
next
slide.
So
before
we
talk
about
how
it
integrates
with
Argo
I
wanted
to
quickly
talk
about
the
the
platforms
architecture.
This
is
a
very
high
level
architecture
and
we'll
talk
about
like
how
we
integrate
with
algo
workflow
and
events
for
the
orchestration,
which
will
specifically
Focus
this
presentation
on
so
from
the
the
platform.
The
platform
has
multiple
different
Services
I've
highlighted
a
few
here,
so
we
have
the
pipeline
Management
Service,
which
helps
with
repository
creation,
build
environment,
deployments,
integration
with
artifactory
and
observability.
A
So
from
a
data,
any
engineering
personas.
What
we
expect
is
like
them
to
out
of
the
box,
have
the
ability
to
get
their
environment
up
and
running,
and
then
we
provide
the
capability
through
our
jobs,
orchestration
service,
using
the
the
workflow
and
the
events
layer
for
scheduling
their
jobs
and
creating
dependencies,
and
we
have
namespace
service
to
orchestrate
their
runs
on
the
compute
layer
on
the
multiple
kubernetes
clusters
that
we
manage
and
notification
services
for,
alerting
a
notification
through
various
mediums
that
we
have
on
the
right.
A
So
we
can
see
that
we
have
a
representation
of
what
we
call
as
a
pipeline.
So
a
pipeline
is
an
abstract
which
we
use
for
calling
a
scheduling
on
a
compute
layer.
So
basically,
a
pipeline
has
a
stack
of
process
that,
with
the
data
that
is
getting
ingested
and
the
data
that
is
outputted
each
processor
is
a
code
artifact
and
the
process
does
Transformations
on
the
incoming
data
and
the
pipeline
manages
the
processing
of
this
entire
layer
and
the
dependencies.
So
we'll
get
more
clarity
on
these
abstracts
in
the
coming
slides.
A
So
in
this
slide,
like
I,
put
a
quick
representation
of
how
the
stack
from
a
dag
on
the
left
translates
to
orgo
and
how
we
manage
the
dependencies.
So
basically,
so
in
the
previous
slide
we
saw
the
the
intuit's
processing
flow
with
the
the
real-time
stream
Transformations
and
also
the
batch
Transformations.
So
in
the
batch
transformation
we
can
see
that
there
is
a
dag
which
is
translated
to
pipeline
dependencies
through
workflows
in
Argo.
A
So
the
the
root
nodes
here
are
in
this
example:
cron
workflows
that
are
running
on
a
schedule
and
once
the
workflow
is
fired,
there
is
a
downstream
dependencies,
so
Downstream
dependencies
usually
can
be
between
data.
It
can
be
between
application
layers.
So
it
depends
on
your
use
case,
but,
like
usually,
we
support
a
resource-based
Upstream
dependencies
between
multiple
pipelines,
so
these
are
independent
pipelines
that
are
managed
by
different
organization,
different
teams
in
Intuit,
and
there
are
Downstream
that
are
depending
on
this
data.
A
Maybe
there
is
a
data
enrichment
being
done
in
step
one,
and
then
we
want
to
process
and
write
the
data
to
Hive
in
the
next
step
and
then
do
post
processing
and
build
reporting
and
modern
analysis
in
the
The
Next
Step.
So
so
we
support
two
types
of
pipeline
dependencies,
primarily
a
time-based
dependency,
which
we
support,
using
the
Crone
workflows
and
trigger
condition
based
dependencies,
which
we
support,
using
Argo,
even
source
and
sensor.
A
So
in
this
example,
the
upstream
and
the
downstream
workflow
or
like
we
discussed
earlier,
is
a
pipeline
with
multiple
steps
in
it
and
the
dependency
management
is
done
using
a
Kafka,
Event
Source
that
we
have
built
and
the
sensor
which
takes
care
of
the
downstream
triggers.
So
in
this
example,
it's
a
one-to-one,
but
we
support
one
to
many.
This
is
also
supported,
natively
in
Argo
events,
so
this
just
shows
the
capability
that
you
can
build
a
combining
orgo,
workflow
and
events,
a
complete
orchestration
under
scheduling
infrastructure
for
all
your
data
Pipelines.
A
Specifically,
this
slide
targets,
how
we
scaled
Argo
events,
this
targets,
both
the
stability
aspect
and
also
the
the
scalability
aspect
of
this
infrastructure.
So
from
the
previous
slide,
we
can
see
that
between
two
workflows,
the
internal
dependencies
we
manage
through
an
even
source
and
a
sensor
that
has
an
internal
Nat
stream
bus.
So
the
initial
issue
that
we
ran
into
with
this
architecture
is
that
so
the
Event
Source
like
where
there
are
different
types
of
sources
that
they
Argo
events
support
one,
including
the
workforce.
A
So
the
way
we
were
orchestrating
is
the
even
Source
will
listen
to
the
Upstream
workflow
completion
and
emit
an
event
to
the
NAT
stream,
which
will
then
utilize
the
sensors
for
doing
the
downstream
triggers.
But
what
happens
during
a
critical
failure?
What,
if
like
an
entire
node,
goes
down
or
in
an
even
Source
completely
goes
out
in
those
scenarios
we
Face
some
issues
with
fault
tolerance
where,
like
an
upstream
workflow
completes,
but
an
event
is
not
captured
because
the
Event
Source
went
down.
So
during
the
initial
Journey
we
migrated
all
our
Event
Source
to
Kafka.
A
This
is
a
natively
supported
feature
in
algo
events,
which
meant
that,
like
we
could
turn
down,
I
mean
even
if
during
critical
failures,
if
they
even
saw
spots,
are
unavailable,
the
data
is
still
persisted
in
an
intermediate
queue,
and
so
this
was
the
the
first
step
that
we
took
about
for
stabilizing
this.
But
there
is
still
one
big
scalability
issue
that
we've
had
so
with
each
of
our
pipeline
dependency
management.
We
are
deploying
the
workflows
which
takes
up
pods
from
in
your
kubernetes
clusters.
A
We
are
deploying
the
even
source
and
sensors,
which
takes
up
more
pods,
so
we
are
just
adding
up
more
pods
for
each
pipeline
dependency
management.
So
we
were
looking
into
solving
the
problem
of
creating
independent,
Event
Source
per
workflow.
So
the
good
news
is
that
we
were
able
to
collaborate
with
the
the
Argo
events
team,
and
so
we
came
up
with
a
solution
of
migrating
to
a
jet
stream
bus
and
a
Shad
even
source.
A
So
this
is
supported
in
the
most
recent
versions
of
orgo
events,
starting
from
1.7.1,
where
you
could
utilize
a
single
Kafka,
Event
Source
for
all
the
Upstream
Shad
workflows.
So
we
have
reduced
the
resource
utilization
on
the
even
Source
by
99,
and
we
have
migrated
to
the
new
jet
stream.
Bus
Which
supports
the
capability
for
even
source
to
rely
on
a
single,
even
Source
event,
and
it
can
filter
on
the
data
that's
coming
in
from
the
workflows.
A
So
this
is
our
current
iteration
and
in
our
Target
iteration
we
are
working
on
moving
the
sensors
to
a
Shad
sensor
model.
So
the
the
target
state
is
that
for
each
Argo
events
namespace,
which
we,
which
is
the
orchestration
layer,
we
will
have
a
single
compute
stack
for
the
event
source
and
the
sensor
which
will
manage
all
the
the
pipeline
dependency
management
for
tens
of
thousands
of
pipelines
that
we
manage
every
single
day
and
the
downstream
triggers
are
handled
by
the
the
even
Source
compute
layer.
A
So
there
is
an
effort
going
internally
to
migrate
the
the
events
so
even
sensors
and
decouple
the
the
Manifest
and
the
compute
so
that
we
can
scale
through
the
structure
which
we
will
be
posting
more
about
in
the
upcoming
Argo
release.
Notes
now
I'll
hand
off
to
Bala
to
talk
about
scaling,
Argo,
workflows,.
B
C
C
If
you
want
to
support
this
match,
you
massive
workload,
we
cannot
achieve
with
a
single
cluster,
so
we
need
to
move
into
that
multi-cluster
architecture.
So
when
we
are
thinking
about
the
multi-cluster
architecture,
the
one
main
requirement
we
got
is
like
the
distribution
need
to
be
more
Dynamic
and
it
need
to
scale
in
future.
C
C
If
the
workflow
controller
is
taking
like
a
little
bit
time,
the
Chrome
schedulers
are
missing,
so
we
enabled
that
h
a
using
like
a
kubernetes
leader
leader
apis,
so
that
at
the
given
time
there
will
be
only
one
leader
and
rest
of
the
people.
Rest
of
the
notes,
rest
of
the
part,
will
be
a
follower
whenever
that
any
problem,
the
leader
immediately,
that
follower
will
pick
it
up
and
start
working
on
that.
C
The
next
feature
we
we
enable
like
the
PDP
support
on
the
workflow
Parts.
Whenever
you
have
like
a
frequent
Auto
scaling
clusters,
the
node
the
scale
down
will
trigger
the
node
drying,
which
will
evict
that
the
workflow
parts,
which
will
end
up
to
the
torque
flow
or
step
failure,
or
you
need
to
re-run
that
again,
the
entire
step.
C
To
avoid
this
issue,
we
enable
that
part
disruption
budget
as
a
first
class
citizen
for
that
workflow,
so
that
every
workflow
will
create
a
PDP
for
the
workflow
parts
and
once
the
workflow
is
completed,
the
PDP
object
will
be
get
deleted.
This
is
the
way
we
can
prevent
that
power
eviction
from
the
node
drawing
the
third
feature
we
we
Pro
we
implemented
is
like
a
rate
limiting
that
concurrent
part,
because
the
every
cluster
will
have
like
some
concurrent
power
limitations.
C
So
we
enable
that
the
rate
limiting
the
concurrent
Parts
in
the
multi
levels-
the
user-
can
confirm
the
cluster
level
so
that
the
total
cluster,
if
I,
want
to
restrict
further
10
000
parts
that
they
can
configure
it
in
the
cluster
level
and
each
namespace
level.
Also
they
can
configure
it
to
give
a
multi-tenant
way
of
like
a
some
some
namespace
and
we
need
a
more
level
and
in
some
name
space
we
need
a
restricted
level.
We
can
configure
it.
A
In
the
next
slides,
we'll
quickly,
look
at
the
the
visual
aspects
of
what
we
were
talking
about,
I
think
that
will
paint
a
better
picture.
So
we
talked
about
having
so
two
abstracts
for
the
platforms
that
are
integrating
the
engineering.
So
we
have
internal
workflow
in
our
into
a
developer
portal
where
users
can
build
a
data
processor.
So
a
data
processor
like
I
mentioned
earlier
is
an
Intuit
artifact,
a
code
that
can
be
run
in
the
compute
layer
of
your
choice.
A
These
are
reusable
artifacts
that
can
be
scheduled
in
multiple
different
pipelines.
These
are
shareable
process.
So,
for
example,
in
this
slide,
you
could
see
that,
like
a
process
that
is
being
built,
this
is
of
runtime
Spark
It's,
a
language
that
the
user
has
chosen
is
Python
and
right
out
of
the
box.
The
pipeline
management
layer
builds
up
the
the
repositories
for
the
users
to
get
started,
the
the
CI
CD
and
also
the
artifactory
integration.
So
the
the
platform
users
can
just
focus
on
primarily
writing
code
and
not
the
other
aspects.
A
So
the
next
step,
like
once
a
user
registers
a
processor,
they
could
create
a
data
pipeline
which
is
the
compute
and
the
orchestration
layer
where
they
could
specify
the
artifact.
They
could
stack
multiple
processor
artifacts
and
have
the
pipeline
set
up
and
you
could
specify
the
scheduling
and
so
in
the
scheduling
like
we
discussed
earlier,
using
the
Argo
events,
we
are
able
to
support
resource-based
scheduling
and
time-based
scheduling.
So
in
this
example,
the
pipeline
has
to
run
every
day
at
3,
am
on
completion
of
this
Upstream
data
Pipeline
and
on
completion
of
this
resource.
A
So
on
meeting
this
condition,
the
pipeline
will
automatically
schedule.
So
this
is
how
we
have
decoupled
the
the
dependency
management
between
organizational
organizational
wide
data
pipelines.
So
once
the
pipeline
is
created,
we
provide
the
the
dashboards
for
managing
your
clusters,
your
observability
Splunk
dashboards,
cost
management
and
Spark
history
server
in
this
example,
because
it's
a
spark
pipeline
for
data
processing
and
we
also
provide
the
execution
history.
A
So
all
this
is
a
layer
of
a
layer,
that's
built
on
top
of
the
the
kubernetes
Clusters
we
manage
for
the
compute
and
the
the
Clusters
we
manage
for
the
Argo
orchestration,
so
in
the
next
slide.
I
just
wanted
to
quickly
give
a
Viewpoint
of
like
the
the
capability
and
what
it
enables
the
the
engineering
platforms
to
do
so.
A
So
this
is
the
the
next
scalability
Journey
that
we
are
currently
ongoing
so
where
we
are
decoupling,
the
sensor
compute
and
the
Manifest
management
to
reuse
a
single
sensor
for
managing
all
your
dependencies
so
and
we're
also
working
on
a
next-gen
multi-cluster
workflow
for
supporting
cluster
load
balancing
through
the
workflow
controller,
which
Bala
will
talk.
Give
a
few
words
about.
C
So
based
on
the
hello,
so
this
is
the
one
of
the
top
demand
in
that
open
source
supporting
like
a
multi-cluster
workflow.
Still,
we
are
evaluating
how
we
can
natively
support
the
multi-cluster
load,
balancing
for
our
Gore
flow
controller,
I.
Think
yesterday,
maintenance
notes
I,
mentioned
that
we
are
targeting
for
3.6
feature,
but
still
we
are
looking
for
the
community
to
give
the
their
use
case
and
everything
to
understand
better
for
that
the
common
requirement
for
the
multi-cluster
workflow,
oh.
D
C
C
C
So,
currently,
the
master
orchestration
service
will
have
like
all
the
registered
clusters
and
based
on
those
registration.
It
will
rapidly
schedule
it
in
the
wrong
property
and
you
can
keep
track
of
like
how
many
workflows
are
running
them,
because
that's
the
feedback
loop
through
the
cardboard
flow
for
Hardware.
Even
it
will
tell
you
back,
okay,
this
of
course
done.
If
start
the
downstream,
of
course,
but.
A
So
with
when
it
comes
to
the
kubernetes
Clusters
that
we
manage
so
one
of
the
the
key
challenges
with
running
orgo
workflows
in
scale
is
POD
management.
So
we
run
all
our
kubernetes
clusters
in
AWS
accounts
and
there
is
IP
limitations
on
like
how
many
parts
that
can
come
up
with
low
latency
on
a
specific
cluster.
So
the
need
for
load
balancing
between
multi-clusters
comes
from
that
basis.
So
currently,
the
bpp's
orchestration
service
is
performing
this
load
balancing
through
the
feedback
loop
with
the
Argo
events
in
Kafka.
A
So,
apparently,
per
cluster
we
are
targeting
around
10
000
active
workflows
so
because
our
pod
usage
pervokes
those
around
two
three
parts.
So
that's
around
thirty
thousand
eyepiece
and
the
concurrent
workflows
that
we
support
per
cluster
is
around
three
thousand.
So
we
for
supporting
10
000
concurrent
workflows.
We
are
running
on
three
kubernetes
clusters
that
is
dedicated
for
the
orchestration.
So
we
have
a
separate
compute
clusters
for
running
our
spark,
processing,
jobs
and
other
container
applications,
and
we
are
running
out
of.
B
A
So
in
so
small
number
of
workflows
means
that,
like
there
is
a
high
amount
of
resource
utilization.
So
if
we
are
able
to
optimize
for
a
pod
usage
per
workflow,
you
could
run
as
many
workflows
as
you
can,
but
we
are
usually
optimizing
towards
larger
workflows
so
that,
like
the
resources,
are
shared
more
optimally.
G
A
So
the
compatibility
in
our
case
is
specifically
tied
with
data,
because
we
are
talking
about
data
pipelines,
so
the
workflow
integration
are
done
through
the
events
infrastructure.
So
the
Argo
events
track
the
workflow
State
change
and
using
the
state
change
events
we
emitted
to
Kafka.
From
there
we
do
Downstream
invocations
on
other
data
pipelines,
so
completion
of
a
workflow
which,
in
our
case
a
data
pipeline.
We
assume
the
availability
of
data
so
on
the
downstreams.