►
From YouTube: SpoK - Running Big Data Applications @ Scale on K8s - Srivathsan Canchi & Nagaraj Janardhana, Intuit
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
SpoK - Running Big Data Applications @ Scale on K8s - Srivathsan Canchi & Nagaraj Janardhana, Intuit
At Intuit, customer data sets are growing exponentially with the growth of the business and the capabilities offered. Processing this data and making it available for downstream applications such as ML, Analytics, Exploration etc. is crucial. Following the trend of running the services workload on Kubernetes, we built a data processing platform with Spark on Kubernetes as the backbone. This allowed us to reap all the benefits of well established processes for CI/CD, security and cluster management. With these we were able to reduce the cost footprint of our data processing jobs by 30%, while simultaneously increasing the speed to production.
https://sched.co/ZepB
A
A
A
A
A
A
When,
while
we
were
extensively
using
emr
and
yarn,
we
were
very
interested
to
integrate
with
the
existing
kubernetes
infrastructure
in
the
organization
we
liked
containers
over
bootstraps
scripts
that
we
were
used
to
in
emr
world
kubernetes
as
a
resource
manager
as
compared
to
yarn.
We
also
wanted
a
modern,
continuous
delivery
process
for
data
processing
jobs.
A
Services
within
intuit
are
deployed
and
managed
on
kubernetes
the
entity
kubernetes
service
shown
some
of
the
ui
elements
are
shown
here,
allows
service
developers
to
manage
their
service
runtime
and
deployment
since
intuit
already
uses
kubernetes
at
a
large
scale
for
running
a
majority
of
our
online
services.
Here
was
an
opportunity
to
have
a
single
resource
manager
to
manage
both
service
and
data
processing
workloads.
A
Containers
for
data
processing
jobs
are
extremely
useful
because
it
helps
data
workers,
write
their
code
once
and
deploy
it
across
environments
in
a
standardized
and
very
predictable
way.
This
becomes
extremely
important
in
the
cloud
where
your
dev
environment
and
test
environment
and
production
environment
are
quite
different.
A
This
approach,
combined
with
the
declarative
management
of
spark
jobs,
means
no
more
complicated
shell
scripts
to
manage.
As
you
can
see
here,
you
can
see
the
containers
that
are
being
deployed
declaratively
and
our
earlier
systems
where
we
used
to
manage
these
deployments
using
numerous
shell
scripts
that
were
orchestrated
by
other
bootstraps
of.
A
Earlier
it
had
taken
us
a
long
time
to
fine
tune,
yarn
scheduler
for
the
for
its
load,
balancing
and
queuing
capabilities
to
achieve
maximum
efficiency
out
of
it.
Given
that
it
is
quite
complicated
and
it
is
hard
to
find
experts
for
who
are
really
good
at
performance
tuning
yarn,
it
becomes
difficult
to
manage
large
installations
based
on
yarn.
A
The
other
really
important
aspect
for
any
software
development
is
the
continuous
deployment
data.
Processing
jobs
are
no
different.
They
need
to
have
really
well-defined,
predictable,
modern
ci
cd
pipelines
in
order
to
power
faster,
build
and
deploy
cycles.
This
improves
the
developer
productivity
and
also
improves
the
quality
of
production
releases.
A
Spock
makes
it
easier
to
use
the
industry
standard,
continuous
delivery,
tooling
for
data
processing
workloads
at
intuit.
We
use
argo
cd
as
an
operator
to
manage
kubernetes
resources
and
as
the
continuous
delivery
platform
for
data
processing
applications
earlier.
This
was
very
cumbersome
due
to
the
complicated
shell
scripts
and
multiple
sets
of
imperative
commands
that
we
had
to
use
in
order
to
deploy
the
applications
and
to
monitor
status.
A
B
B
B
B
B
This
follows
the
gitoff's
deployment
model,
all
the
resources
required
for
the
job,
such
as
the
spark
role,
the
spark
history
server,
the
actual
scheduled
spark
application.
Jobs
are
managed,
as
native
kubernetes
resources
and
argo
cd
make
sure
to
keep
them
up
to
date
with
what's
required
or
what's
what
the
data
scientist
wants
it
to
be.
B
B
One
of
the
critical
challenges
that
we
faced
was
getting
spark
245
and
below
to
work
with
hive
2
and
latest
hadoop.
Aws
jazz
is
a
dependency
nightmare
and
that's
where
the
suggestion
to
move
towards
part
3,
which
has
been
recently
released
as
most
of
the
issues
with
working
with
hadoop
aws
and
hive,
has
been
fixed
in
that
latest
release.
B
B
Below
are
some
of
the
advantages
we
have
observed:
running
big
data
processing,
workloads
on
kubernetes,
the
unified
management
of
clusters
across
different
types
of
workloads,
say,
service
workloads
and
data.
Processing
workloads
is
a
huge
win
container,
simplify
dependency
management
and
the
data
processing
jobs
get
also
get
to
use
all
the
tooling
built
around
the
rest
of
the
organization
like
artifactory,
ci,
cd
tools,
etc.
B
This
also
helps
reduce
cost
as
opex
and
other
resources
are
shared
across
multiple
different
kinds
of
workloads,
declarative
specification
of
data
processing,
jobs
and
infrastructure
make
it
easier
for
data
workers
to
specify
and
manage
them,
especially
for
ml.
This
is
a
huge
win,
as
it
involves
lot
of
configs.
In
some
cases.
B
B
B
The
operational
overhead
of
maintaining
this
has
reduced,
especially
the
security
patches
and
the
cluster
upgrades,
since
it
is
done
as
part
of
the
entire
org,
there
is
no
need
for
managing
them
separately
as
part
of
data
work
and
performance
wise.
We
are
seeing
porting
an
application
running
on
the
previous
stack
to
on
kubernetes.
B
A
B
I
can
see
some
questions
in
spark
versus
emr.
Can
you
elaborate
how
cost
reduction
was
secured
yeah?
So
there
was.
There
were
multiple
aspects
to
the
cost.
One
was,
for
example,
emr
as
a
service
has
a
cost
of
25
additional
surcharge
on
the
underlying
hardware
cost.
Over
and
above
the
underlying
hardware
cost,
which
was
straight
away,
we
could
avoid
and
also
the
operational
expenditure
around
maintaining
this
clusters.
B
Since,
within
the
organization
we
had
operationalized
management
of
these
clusters
for
the
rest
of
the
services
workload,
we
could
build
on
top
of
that.
So,
for
example,
if
there
was
a
critical
security
patch
which
had
to
be
applied
for
our
cluster
in
the
past,
we
have
to
do
it
separately,
but
with
going
to
kubernetes
as
a
single
resource
manager.
B
All
of
that
is
taken
care
of
by
the
rest.
So
there
is
a
shared
operational
expenditure
cost
and
also
the
additional
emr
cost,
but
we
have.
We
also
see
opportunities.
Around
scheduling
are
basically
I
mean
optimizing,
the
scheduler
and
stuff
like
that,
which
we
haven't
done
it
yet,
but
we
think
will
give
us
more
benefits
in
the
future.
A
In
addition
to
what
nagrat
said,
the
other
reason
why
we
are
able
to
leverage
cost-
and
we
realize
cost
advantages-
is
because
we're
able
to
now
join
workloads
together.
So
we
can
run
spark
applications
and
other
ml
workloads
in
the
same
cluster,
utilizing
the
same
compute
as
well.
So
that
gives
us
an
overall
cost
amortization.
A
B
This
another
question:
how
much
effort
was
it
to
move
from
yarn
to
kubernetes
as
a
scheduler
for
your
spark
jobs
from
spark
perspective?
I
think
the
effort
was
not
much
since
spark
supports.
At
least
the
spark
3
supports
kubernetes
as
a
resource
manager.
I
think
it
was
around
the
rest
of
the
infrastructure.
B
So
we
have
to
switch
back
to
the
s3a
file
system,
which
is
the
open
source
equivalent
of
it.
The
other
thing
was
emr
is
a
managed
service
which
provides
you
things
like
hive
and
stuff
rest
of
the
stuff
hype,
server
and
hive
service
to
which,
since
within
the
organization,
we
use
a
centralized
high
metastore
for
managing
the
metadata,
we
have
to
build
support
for
that.
B
A
A
A
Is
to
add
on
to
the
effort
question
I
think
it
took
us
about
six
months
start
to
end
to
actually
create
a
fully
paved
road
approach
around
around
spark
where
we
were
able
to
then
provision
name,
spaces
and
provide
security
controls
and,
and
so
on.
A
lot
of
as
nagraj
said.
A
lot
of
the
work
was
around
making
spark
and
kubernetes
work
with
things
like
name,
space
access
and
roles
specifications,
and
so
on
spark
itself.
We
were
able
to
get
the
spark
application
running
in
a
few
days,
but
the
majority.
B
Yeah,
so
I
think
there
were
some
challenges
with
s3,
a
initially
but
nothing
specific
related
to
bottlenecks.
Obviously
the
I
think
the
emr.
There
is
a
lot
of
work
going
on
in
the
s3a
file
system
related
to
at
least
on
the
commit
side.
There
are
different
commit
strategies,
but
yeah.
We
did
not
face
much
on
the
bottlenecks
on
the
network
bottlenecks.
A
A
A
There's
another
question
on
spark
streaming:
are
we
running
spark
streaming
jobs
such
as
pad
jobs?
We
are
running
both.
We
have
both
spark
streaming
jobs
and
spark
back
jobs.
Running
on
this
infrastructure
majority
are
batch.
There
are
a
few
streaming
jobs
as
well
in
general
for
streaming,
intuit
users
beam
apache
beam
and
we
have
a
beam
based
streaming
platform.
A
A
So
they
from
a
data
scientist
perspective.
They
don't
really
see
the
the
underlying
kubernetes
infrastructure
and
all
the
complexities
that
go
around
with
it.
So,
in
addition
to
just
having
a
spark
application,
we
had
to
build
the
ci
cd
for
the
application
so
that
we
can
provide
our
data
scientists
and
analysts
with
a
interface
that
was
as
close
to
what
they
already
know,
which
is
around
spark
and
so
that
they
can
be
productive
with
that.
B
Yeah,
just
to
add
on
to
that,
I
think
once
one
of
the
big
changes
one
going
from
emr
to
on
kubernetes
was
the
whole
container
journey
so
which
they
were
very
familiar
with,
because
that's
how
they
used
to
deploy
their
machine
learning
models.
So
now
they
could.
They
can
do
the
same
thing
for
their
future
engineering
or
data
processing
jobs,
and
I
think,
since
we
were
using
most
of
the
stuff
which
was
already
built
in
the
organization
like
ci
cd
pipelines
and
build
automation.
B
B
A
B
A
A
A
Hope
this
is
a
question
from
max
on
how
we
manage
the
containers
for
spark.
So
the
question
is
for
spark
jugs.
Are
you
preparing
container
for
every
job,
or
do
we
have
an
universal
container
which
pulls
the
job
jar?
It's
a
little
nuance,
so
we
have
a
spark
based
container,
which
has
all
the
spark
related
stuff
and
then
what
we
do
is
during
the
build
process.
A
B
A
Tests
and
production
environments,
so
the
answer
is
yes,
there
is
a
container
for
every
job,
and
this
is
built.
This
is
a
the
it's
a
it's
a
thin
layer
built
from
the
base
container
of
spark
there
are,
there
is
one
there
is
another
variant
which
we
are
using
for
featurization
feature
processing
in
in
the
machine
learning
workflows
where
we
have.
We
interact
with
the
feature
store,
which
is
in
s3,
and
we
need
to
run
spark
application
to
get
the
features
from
the
features
too.
A
B
Yeah,
just
to
add
into
that,
I
think
this
was
a
very
pleasant
experience.
Also,
for
example,
we
take
the
open
source
base,
spark
image,
and
then
this
one
level
of
overlay,
where
we
add
stuff,
like
hadoop,
aws,
jars
or
specific
jars
required
for
meta,
store,
access
and
stuff
like
that.
So
that's
one
level
of
overlay
and
then
that
is
released
and
on
top
of
that
application
teams
can
take
that
and
just
add
their
stuff.
B
On
top
of
that,
so
I
think
it's
because
of
the
whole
container
thing,
it's
very
easy
to
manage
the
dependency
also
and
whenever
there's
a
new
version
of
a
spark,
we
just
the
there's
a
central
platform
team
or
us
who
are
responsible
for
releasing
the
base
images
and
then
the
application
teams
can
take
that
and
build
their
stuff.
On
top
of.
B
A
I
want
to
go
back
to
stephen's
question
on
what
new
things
they
would
have
to
learn
to
work
in
the
spock
environment.
Coming
from
the
emr
background,
one
of
the
things
that
we
have
done
with
this
is
we've
actually
reduced
the
things
that
a
data
scientist
has
to
learn
because
they
don't
need
to
learn
about
how
to
set
up
an
emr,
how
to
manage
the
infrastructure
and
write
like
terraform
or
cloud
formation
scripts.