►
From YouTube: Building a Batch Processing Platform... - Rakesh Subramanian Suresh & Aroop Maliakkal Padmanabhan
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Building a Batch Processing Platform for Running Data Pipelines Using Argo & Kubernetes - Rakesh Subramanian Suresh & Aroop Maliakkal Padmanabhan, Intuit
Intuit has built a highly scalable batch processing platform with Kubernetes and Argo to enable data engineers to easily deploy, manage, and schedule data pipelines. With hundreds of AI & Data engineering teams managing over 100,000 data pipelines, pipeline deployments have many challenges, including scheduling, orchestration, and managing complex dependencies to eliminate the silos and increase processing effectiveness in the data lake. While there are solutions to these challenges independently, there isn’t one that holistically solves scheduling, pipeline dependency management, and infrastructure deployment and orchestration. In this talk, we will discuss utilizing Argo Events, Argo Workflow, and Kubernetes to build and effectively manage an orchestration and scheduling engine for running various data processing use cases. Besides, we will also cover the learnings and operational challenges of managing this multi-cluster Kubernetes infrastructure and how Argo can be integrated with Kafka for zero downtime scheduling.
A
It's
really
exciting
to
be
here
today
we
are
going
to
talk
about
building
a
batch
processing
platform
on
top
of
multi-cluster
kubernetes
and
Argo
workflows
at
Intuit
scale
right.
We
have
around
40
to
50K
pipelines
so
at
the
metadata
level
itself,
which
has
a
100K
or
two
thousand
K
concurrent
execution.
So
let's
talk
a
little
bit
about
so
here
is
a
quick
introduction
about
ourselves.
A
We
will
be
talking
a
little
bit
about
the
inter
background
and
I
will
be
talking
a
little
bit
about
the
platform
overview
and
Rakesh
will
be
actually
doing
a
deep
dive
into
the
platform
architecture
and
some
of
the
operational
excellence
and
obserability
challenges
we
faced
and
how
did
we
actually
solve
them
and
the
cube
demo
if
we
have
time
so
before
we
actually
Deep
dive
into
the
platform,
a
quick
shout
out
to
Intuit.
So
we
are
a
global
technology
platform
that
helps
you
to
Achieve
Financial
confidence
right.
We
are
a
Purpose
Driven
value,
driven
company.
A
So
next
is
about
batch
posting
platform
over
you.
So
here
is
some
of
the
personals
use
cases
which
you
are
solving
as
a
ml
engineer.
I
want
to
actually
build
and
iterate
on
models
faster
and
more
models
and
features
faster
as
a
data
analyst
I
want
to
actually
provide
insights
to
the
business
as
quickly
as
possible.
As
a
data
engineer
or
as
a
software
engineer,
I
want
to
produce
clean
data
and
I
want
to
manage
and
run
reliable
pipelines,
and
these
are
some
of
the
indirect
customers
who
their
use
cases.
We
are
solving.
A
So
now,
let's
discuss
a
little
bit
about
the
core
platform
capabilities.
There
are.
There
are
main
four
platform
capabilities
we
are
providing.
One
is
the
runtime
scheduling
and
orchestration
capability.
The
second
one
is
the
devops
trolling,
the
third
one
is
governance
and
fourth,
one
is
the
service
level
management,
so
I'll
be
walking
us
through
all
the
four
capabilities
in
detail.
A
So,
in
runtime
scheduling
and
orchestration
capabilities
as
part
of
this
platform,
we
are
providing
four
different
runtime
capabilities.
One
is
the
spark
on
kubernetes
and
the
second
one
is
EMR.
The
third
one
is
data
bricks
and
the
fourth
one
is
the
docker
container.
What
what
is
more
interesting
is
all
these
computers
are
fully
managed.
That
means
the
developers
doesn't
need
to
worry
about
touching
the
hardware
or
doesn't
need
to
worry
about
managing
this
infrastructure
at
all
right
and
we
provide
the
capability
to
switch
between
these
runtimes
any
of
these
run
times
based
on
their
requirements.
A
It
can
be
due
to
the
either
the
performance
reason
or
the
cost
reasons
they
have
the
option
to
switch
between
the
runtimes,
with
a
few
clicks
on
that
pipeline
and
along
with
all
this
spa
runtimes
and
the
docker
run
times,
so
we
provide
Advanced,
intuitive
scheduling
and
orchestration
capabilities.
Also,
that
means
as
a
developer.
You
don't
need
to
learn
any
scheduling
or
orchestration
tool.
It's
a
low
code
solution.
You
can
actually
go
and
actually
build
your
own
data
Pipeline
with
a
few
clicks.
A
The
next
one
is
the
devops
trolling.
All
these
pipelines,
cicd
enabled
which
is
built
on
top
of
Argo
CD,
and
we
provide
a
manage
infrastructure
and,
as
we
are,
we
are
fully
cloud
enabled
company
we
as
a
developer.
They
need
to
actually
work
with
a
lot
of
cloud
resources
right,
so
they
need
to
learn
how
if
they
want
to
do
the
policy
configuration,
they
need
to
learn
about
different
kinds
of
cloud
resources,
but
we
provide
an
intuitive
way
of
configuring,
their
policies
and
automated
provisioning,
because
the
platform
does
the
proofing
for
them.
A
A
A
So
the
next
one
is
the
governance,
so
intrude
is
a
one
of
the
leading
fintech
companies,
so
we
are
part
of
lot
of
compliance
and
security
requirements,
so
compliance
is
very
important
for
us.
So
as
part
of
this
capability,
we
have
tied,
along
with
the
Intuit
authentication
and
authorization
schemas.
Also,
we
have
custom
approved,
workflows
for
compliant
stocks,
compliance
or
other
compliance
pipelines.
A
Now
I
want
to
talk
a
little
bit
about
the
service
level
management.
This
is
a
highly
available
platform.
We
provide
for
nights
of
availability
and
we
run
some
of
the
tight
ASLA
use
cases
right.
Some
of
the
use
cases
which
we
run
on
this
platform
has
two
minutes.
Sla
use
cases
end
to
end
and
we
do
provide
cost
inside
between
different
kinds
of
spark
runtimes
that
enables
the
user
to
actually
choose
what
runtime
is
suitable
for
their
use
cases,
rather
than
just
one
solution,
and
we
provide
pair
path.
What
does
this
mean?
A
Is
it's
a
low
code,
no
code
solution
which
actually
enable
a
developer
to
create
a
data
Pipeline
with
a
few
clicks
earlier,
setting
up
a
production
pipeline
was
taking
probably
a
few
days
or
probably
one
or
two
weeks
now,
a
newly
joined
engineer
at
Intuit
can
actually
productionize
a
pipeline,
probably
in
probably
less
than
15
minutes
right.
So
he
will
be
having
a
production
pipeline
running
with
monitoring,
alerting
capabilities
with
all
the
scheduling
and
orchestration
capabilities.
A
Even
he
can
actually
set
up
a
different
compliance
data
pipeline.
Also,
the
next
one
is
the
shared
Pro.
So
this
is
the
core
capabilities.
I
believe
Rakesh
will
be
talking
a
little
bit
more
about
share
capabilities.
What
this
shared
processor
means
we
actually
onboard
lot
of
platform
and
platform
on
platform
use
cases.
For
example,
ml
platform
can
actually
onboard
to
this
platform
to
actually
trigger
a
workflow
on
the
RMR
platform
right,
so
a
cube
flow
workflow
can
be
triggered
based
on
the
shared
processing
capabilities
on
a
ml
infrastructure.
A
A
A
The
reason
we
are
providing
this
is
because
we
are
most
of
our
services
are
in
kubernetes,
so
we
want
to
actually
provide
this
part
runtime.
Also
in
kubernetes
infrastructure
it
actually
it's
a
similar,
more
homogeneous,
because
everything
is
dockerized
and
it's
very
cost
effective
right.
It
doesn't.
There
is
no
vendors
lock-in
or
there
is
no
surcharge
and
it's
a
shared
resources
and
shared
Opex,
so
very
cost
effective.
Our
developers
love
using
this,
and
it's
very
declarative
in
first
year
with
that
I
will
handle
to
Rakesh
to
do
a
platform.
Architectural
deprive.
D
Thank
you,
I
hope,
I'm
audible.
So
next
we'll
talk
about
the
the
platform
architecture
and
go
over
on
the
high
level
how
we
have
built
the
platform
before
we
get
there.
Let's
talk
a
little
bit
about
the
into
its
data
processing
flow
from
a
very
high
level
on
the
left.
We
can
see
a
high
level
overview
of
a
simple
data
processing
use
case,
so
the
top
layer
has
some
Intuit
customer
products
which
are
emitting
data.
These
could
be
click
stream
data.
This
could
be
data
that
we
would
like
to
post
process.
D
Do
data
enrichment
too.
So
before
the
data
arrives
in
our
data
Lake,
there
are
stream
Transformations
done
through
our
stream
pipelines,
and
so
there
is
a
bunch
of
post-processing
layers
where
we
do
post
processing
and
Analysis.
So
from
this
layer
we
are
able
to
derive
real-time
recommendations,
data
enrichment,
feature
generation,
model,
training,
data,
curation
fraud,
detection
and
a
bunch
of
other
capabilities.
So
we
will
look
a
little
bit
more
into
how
we
are
providing
the
capability
to
build
the
dag
in
the
coming
slides.
D
So
in
this
slide,
we'll
talk
about
the
the
architecture.
So,
as
we
went
over
earlier,
so
batch
processing's
aim
is
to
simplify
the
use
cases
for
a
lot
of
personas
being
it
like
ml
Engineers
data
analysts,
data
engineers,
and
we
do
that
through
providing
a
simplified
development,
deployment,
management
and
orchestration
and
monitoring
as
one
holistic
solution
to
both
engineering
teams
and
also
to
the
platforms
to
integrate.
So
the
high
level
architecture
broadly
has
two
sections,
as
can
be
seen
in
the
top.
D
There
is
a
control,
control,
plane
layer
and
at
the
bottom
there
is
a
compute
plane
layer.
So
in
the
control
plane
we
have
the
service
layer
which
helps
with
the
the
creation
of
the
users
code
repositories,
their
build
environments,
their
artifactory
and
also
hookups
to
observability.
So
this
is
a
fully
automated
self-serve
process
that
we
we
have,
through
our
other
paved
Road
ecosystem
called
modern
SAS,
which
is
fully
running
on
kubernetes,
and
we
have
a
job
orchestration
service
for
helping
with
the
scheduling
and
orchestration
of
your
pipeline.
D
D
So
in
the
previous
slide
we
saw
a
stream
data,
a
stream
data
Pipeline
and
a
batch
data
pipeline.
But
the
abstracts
are
very
similar
between
the
two.
A
data
pipeline
is
a
compute
and
an
orchestration
layer
which
is
a
representation
of
one
end-to-end
data
processing,
job
and
processors
or
code
artifact.
That
runs
in
the
data
pipeline
to
do
data
Transformations
that
can
be
data
enrichment,
curation
training,
your
model
feature
generation.
A
lot
of
that,
like
I,
mean
there
is
a
lot
of
data
process
that
happens
before
the
data
is
usable.
D
So
all
that
can
be
abstracted,
and
these
are
processors
which
are
shareable
so,
for
example,
a
data,
usually
that
needs
an
enrichment
or
curation.
These
can
be
shared
between
multiple
different
data
jobs,
and
so
that's
one
of
the
reasons
why
we
abstract
that
into
a
separate
unit
by
itself,
and
we
make
the
processor
as
a
shareable
entity
that
can
be
added
to
any
of
your
compute
infrastructure
which
being
the
pipeline.
D
So
there
are
varying
use
cases
I'm
running
today
in
the
batch
processing
platform,
be
it
platform
use
cases.
We
have
feature
management
platform
running
to
generate
features
in
a
scheduled
time
frame
using
a
spark
on
kubernetes
layer.
We
have
ml
platform
using
As
for
their
workflow
orchestrations
and
data
scientists
and
other
data
personas
directly,
interacting
with
the
UI
for
running
their
sql's
queries
to
generate
reports.
D
So
the
next
slide
will
we'll
talk
about
the
how
we
are
providing
the
capability
to
build,
dag
on
how
we
provide
dependency
management.
So
the
main
problem
with
the
like.
That's
there
with
like
a
lot
of
data
engineering
teams
and
platforms,
is
how
do
we
make
sure
data
pipelines
are
decoupled
and
also
data
availability
can
be
made
made
sure
from
one
data
pipeline
to
another.
D
So
in
this
example,
the
root
node
or
pipelines
that
have
time
based
dependencies
or
so
that's
starting
up
on
a
specific
calendar
schedule
and
that's
defined
through
the
crown
workflows
and
all
the
downstream
dependencies
that
are
waiting
on
the
root
node
or
waiting
using
the
the
Argo
Event
Source
and
the
Argo
sensors.
So
each
of
these
workflows
are
running
our
orchestration,
compute
plane
that
we
went
over
earlier
and
the
workflows
are
decoupled
from
our
runtime
compute
layer.
So,
like
a
group
went
over
earlier,
so
we
do
support
multiple
different
time.
D
The
runtime
of
your
choice
so
in
the
next
slides,
we'll
talk
about
the
operational
excellence
and
the
observability
capabilities
and
the
journey
that
we
went
through
in
the
past
year
to
make
sure
that
we
are
able
to
get
top
of
the
line,
availability
and
also
the
SLA
number
so
with
the
with
the
help
of
the
paved
Road
ecosystems.
Within
into
it.
D
We
get
a
lot
of
observability
features
for
from
our
paid
path,
kubernetes
platform,
which
is
called
mountains,
which
provides
Advanced,
githubs
and
kubernetes
control,
plane
and
absorbability,
so
deploying
of
a
new
cluster
and
managing
the
cluster,
and
also
the
observability
of
the
cluster,
is
handled
by
different
paid
pass
teams
within
the
team
which
we
interact
with,
which
we
built
a
platform
on
top
of
as
a
platform
Opera
providing
distributed
compute
infrastructure
as
a
service
to
Intuit.
There
are
several
operational
challenges
when
it
comes
to
managing
multiple
kubernetes
clusters.
D
So,
right
now,
in
our
production,
we
run
four
kubernetes
clusters,
two
for
our
orchestrations
and
two
forever,
our
runtime.
So
we
tackled
improving
the
operational
excellence
to
optimizing
on
two
areas.
One
is
platform,
operation,
excellence
and
the
other
is
providing
developer
and
customer
operation
Excellence.
D
What
we
did
is
we
were
able
to
scale
to
thousands
of
concurrent
workflow
runs
and
we
were
able
to
provide
four
nines
of
availability
and
less
than
one
minute
job
deployment
guarantees.
So
let's
talk
about
the
one
minute
job
deployment
guarantees.
So
one
minute
is
the
the
time
that
we
guarantee
for
provisioning
and
deploying
your
entire
data
pipeline
infrastructure
and
get
it
to
a
running
stage.
D
So
once
the
provisioning
happens,
your
data
pipeline
can
run
for
hours,
but
the
time
between
you
fire
the
request
and
the
time
that
we
provision
through
our
platform
is
one
minute.
So
we
achieved
this
by
decoupling,
our
compute
plane
into
orchestration,
compute
plane
and
fine-tuning
the
kubernetes
clusters
for
the
specific
job
that
it's
given
in
our
in
our
orchestration
compute
clusters.
D
So
next
aspect
is
the
developer
operation
Excellence.
So
as
a
platform,
so
we
we
have
to
monitor
our
slas
and
monitor
our
availability,
but
also
to
provide
operational
excellence
capabilities
to
the
users
that
are
interacting
with
the
platform
we
have
pipeline
head
tricks
to
track
the
status
and
health
of
your
pipeline
notification,
integration
for
monitoring
your
data
pipeline
into
SLA,
and
also
failure,
alerting,
Auto
scaling
and
also
pipeline
retries
for
more
more
fault
tournaments.
D
So
there
is
a
more
extensive
demo
coming
up
to
start
with,
so
we
wanted
to
start
showing
like
the
the
capability
internally,
that
we
are
built
for
a
fully
automated
processor
and
a
pipeline
deployment
experience.
So
here
we
can
see
that
like,
for
we
have
an
Intuit
developer
portal
experience
where
users
could
create
a
data
processor
by
providing
their
runtime
in
this
example,
spark
their
language
of
their
choice
and
a
bunch
of
other
default
spark
configuration,
because
this
is
sparked
data
processing
application.
D
So
we
write
out
of
the
box
spin
up
the
pick
it
up
repository
the
build
environment,
the
Argo
CD
for
orchestrating
your
bills
and
also
hookups
to
the
observability.
So
the
developers
can
focus
on
the
code
and
not
on
the
process
of
getting
your
thing
up
and
running.
So
this
has
severely
accelerated
the
amount
of
time
anyone
can
get
to
production
and
then
once
you
also
so
like,
like
we
discussed
earlier,
the
processor
being
the
code
artifact
that
can
be
shared
between
multiple
computes.
D
So
you
can
now
create
your
data
pipeline,
which
is
the
computer
and
orchestration
layer
where
you
will
be
able
to
pick
your
compute
of
your
choice,
for
example
the
the
spark
on
kubernetes
to
pick
the
kubernetes
cluster.
It's
getting
deployed
to
the
auto
provision,
the
namespace
with
the
required
IAM
roles.
We
also
have
a
kubernetes
control
plane
that
is
built
by
Intuit,
so
users
can
configure
their
policies
on
access
to
the
data
and
we
also
provide
abilities
to
add
your
processors.
E
What
we
are
seeing
here
is
into
its
Dell
for
portal
experience
where
data
Engineers
can
configure
their
data
pipelines
and
data
processors.
This
sparkle
kubernetes
data
pipeline
has
been
configured
to
run
on
the
given
kubernetes
namespace
and
the
cluster,
which
are
configured
during
the
initial
pipeline
provisioning
phase.
As
can
be
seen,
the
data
pipeline
has
a
processor,
a
schedule
and
some
additional
configurations.
E
E
Iksl
is
a
control
plane
for
managing
kubernetes
cluster
and
namespaces.
Here
we
are
looking
at
the
developers
configure
namespace
where
they
have
the
ability
to
manage
the
namespace
resources
and
ACLs
besides
developers
also
get
access
to
different
metrics
and
dashboards
for
monitoring
their
pipeline.
This
is
a
sample
dashboard
that
comes
out
of
the
box
that
is
customizable
by
the
developer,
based
on
their
needs
on
their
monitoring.
E
Besides,
blank
dashboards
are
also
provided
out
of
the
box
for
monitoring
pipeline
runs
and
execution.
History
are
available
for
better
traceability
developers
and
teams
also
gather
quick,
costing
View
and
additional
costing
dashboards
for
managing
cost
on
the
pipeline.
Additionally,
spark
history
server
is
provided,
but
the
spark
runs
can
be
looked
at.
E
Each
execution
creates
an
Argo
workflow
which
we'll
just
take
a
look
at
which
will
orchestrate
the
steps
required
for
running
this
data
pipeline.
So
here
there
are
two
running
workflows:
let's
take
a
look
at
one
of
the
workflows,
so
each
workflow
creates
the
spark
application,
so
here
I'm
putting
up
like
all
the
running
spark
application.
E
So,
as
can
be
seen,
there
are
two
Runnings
per
caps
which
are
being
monitored
by
the
workflow,
their
workflow.
Besides
firing,
the
spark
application
on
the
customer's
namespace
also
monitors
the
spark
app
and
then
emits
an
event
to
Kafka,
which
is
then
read
by
the
orchestration
service
for
updating
the
pipeline
state.
E
If
we
go
back
to
the
pipeline
UI,
we
can
see
that
the
state
of
the
pipeline
is
currently
running
and
the
execution
has
been
updated
in
the
execution.
You
can
take
a
look
at
the
configuration
using
with
the
execution
fire
and
also
other
execution
blocks
which
are
readily
available
for
the
users
to
look
at
running
executions.
D
D
F
C
F
C
D
So
we
have
cluster
monitoring
based
on
our
capacity,
so
we
are
able
to
load
balance
both
our
spark
application
and
also
our
workloads
to
multiple
kubernetes
clusters,
so
so,
usually
when
we
are
hitting
close
to
a
capacity,
so
we
provision
new
clusters.
So
the
main
aspect
of
this
is
having
the
ability
to
monitor,
manage
multiple,
connectors
and
load
balances
both
on
the
orchestration
and
also
the
the
spark
job
so
which
we
have.
G
G
G
D
A
B
I'm
kind
of
curious
when
you,
when
you
have
a
one
minute
job,
SLA
I'm
assuming
you
run
an
AWS
key,
said:
I
am
and
some
other
stuff.
Do
you
just
pre-provision
nodes
or
have
some
pre-scheduling
so
that
pods
can
actually
go
onto
nodes
or
is
it
the
pods
are
created
and
the
cluster
Auto
scaler
is
going
to
kick
in.
D
So
the
the
pods
are
not
pre-provisioned,
so
this
the
nodes
are
dedicated
for.
So
we,
since
we
have
a
dedicated
clusters
for
our
provisioning,
I
mean
I'm
scheduling
layer,
so
those
clusters
are
highly
available
and
on
top
of
that,
we
pre-provision
the
IPS
required
for
bringing
up
a
bot.
That
has
been
our
throttling
point,
which
we
noticed
when
we
were
doing
our
load
testing
and
when
we
are
trying
to
provision
thousands
of
PODS,
what
we're
taking
six
minutes
and
12
minutes.