►
Description
Modern Big Data Pipelines over Kubernetes [I] - Eliran Bivas, Iguazio
Big data used to be synonymous with Hadoop, but our ecosystem has evolved over time with new database, streaming and machine learning solutions which don’t necessarily benefit from the Hadoop deployment model of Map/Reduce, YARN and HDFS. These solutions require a generic cluster scheduling layer to host multiple workloads such as Kafka, Spark and TensorFlow, alongside databases such as Cassandra, Elasticsearch and cloud-based storage.
Eliran Bivas is a senior big data architect with years of hands-on experience working on both big data and cloud native solutions. Eliran will go over a common solution framework to create cloud native end-to-end analytics applications. It involves using Kubernetes as an alternative to Yarn, running Spark, Presto, machine learning frameworks (TensorFlow, Python and Spark ML kits) and serverless functions coupled with local and cloud-based storage. The session will showcase customer use-cases from IoT, automotive, cloud SaaS and finance. It will also include a live solution demo which demonstrates the benefits of using big data and analytics over a cloud native architecture, eliminating the existing challenges of complexity and moving towards a continuous integration and development architecture for big data.
About Eliran Bivas
Eliran Bivas is a senior big data architect at iguazio and a self-proclaimed tech junkie with a passion for innovation. Eliran is skilled with object-oriented design and development, having worked extensively on cloud native environments. He has broad experience developing with cloud services such as Amazon EC2, Google AppEngine, Heroku and VMware CloudFoundry. Eliran is also savvy with Spark, Presto, Flink and TensorFlow, but his main interests are in JVM languages, Golang, Docker and Kubernetes. Prior to iguazio, Eliran worked at global tech leading companies such as VMware and F5 Networks. He has a B.Sc in Math and Computer Science and an MBA from Bar Ilan University.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
What
I
tend
to
call
myself
a
Joker
engineer
and,
as
you
can
see,
I'm
really
into
Legos,
but
this
is
as
far
as
they
go
down
this
presentation
for
for
the
past
decade
or
so
or
bigger
out
of
all
this
change
romantically,
since
its
first
released
back
in
Apple
to
2006,
Hadoop,
really
defined
the
big
data
era
this
world
today's
network
was
slow,
memory
was
expensive
and
we
needed
something
to
to
process
our
ever-growing
data.
We
believe
that
by
bringing
the
compute
closer
to
the
data,
we
will
solve
all
of
our
problem.
A
A
We
even
started
storing
some
of
our
data
in
on
a
loop,
no
sequel
data
source
like
Redis
or
Cassandra,
or
maybe
elasticsearch,
but
we
needed
a
lot
more.
We
place
Kafka
rabbitmq
right
next
to
our
API
application
flow
was
reading
in
growing
amount
of
data.
We
needed
a
method
to
process
all
those
incoming
events.
We
built
event
process
pipeline
with
spouts
and
bolts.
We
added
fleeing
and
even
toned
back
to
spark
and
got
our
notion
of
swimming
with
micro
batching.
A
Overall,
all
those
frameworks
were
basically
a
way,
a
way
of
bringing
more
and
more
into
our
Big
Data
world.
Now
that
we
were
so
good
at
collecting
data,
we
need
to
figure
out
what
its
mean
Presta
was
added
to
the
mixture
and
with
its
current
solution,
we
turned
to
machine
learning
later
adopted
deep
learning,
and
in
these
days
it
seems
like
many
companies
when
I
attend
sure
floater.
A
The
ever-growing
technology
map
and
deploying
this
infrastructure
became
real
hard
work,
a
lot
of
companies
actually
based
our
offering
and
how
well
they
can
actually
deploy
this
detector
Paula
this
technology
map
into
your
organization,
even
where
I
work
at
Iguazu.
We
provide
Hadoop
compliant
api's,
because
we
deal
with
companies
that
are
required.
Big
Data
solution.
A
A
If
we'll
take
care,
for
example,
we
take
Amazon
the
leading
cloud
provider,
we
have
s3
working
as
a
dupe
compliant
a
file
system
EMR,
which
is
elastic
Amazon,
Elastic
MapReduce,
runs,
of
course,
spark
HBase
and
yarn
and
later
on,
they
added,
of
course,
other
capabilities.
We
see,
and,
of
course
you,
if
you
have
tension
flow
Amazon
has
a
pre-configure.
A
Am
I
running
for
you
on
AWS
and
we
started
streaming
stuff
through
kinases,
but,
as
you
can
see,
nothing
really
change
the
hard
work
of
actually
deploying
all
those
tangled
technology
map
replaced
by
the
actual
weaselly
of
choosing
which
server
less
data
service
you're
going
to
choose
and
trying
to
figure
out
what
your
invoiced
by
the
end
of
the
month
going
to
be,
and
no
one
actually
can
the
cloud
products
to
the
server
lacera.
Our
data
was
in
the
cloud
we
didn't
have
to
worry
where
it
was.
A
It
was
simply
there
or
occasionally
wasn't,
and
there
were
the
real
major
leap
was
the
application
less
code.
This
is
when
lambda
was
introduced.
We
were
promised
a
simpler
to
process
incoming
events.
We
just
code
well,
if
you're
familiar
with
lambda.
This
is
the
promise
we
got.
We
actually
got
something
else.
A
A
Some
of
them
were
using
mezzos
and
application.
Sometimes
is
something
completely
else,
and
now,
with
the
introduction
of
new
architecture,
new
analytic
tools,
we
need
to
rethink
our
ecosystem.
Look
it
back.
There
are
three
engineers
we
need
to
really
understand
their
current
needs.
Application
engineers
want
one
want
what
every
application
engineer
wants.
They
want
agile
development.
They
want
to
release
as
frequent
as
possible
and
get
the
use
of
feedback
as
soon
as
possible
data
engineers,
pretty
simple.
They
simply
want
stuff
to
keep
working.
A
They
don't
care
how
but
keep
it
working
and
I
have
their
data
available
and
the
data
in
a
DevOps
engineers.
One
their
tools
to
keep
work
that
the
the
tools
to
keep
working
but
have
less
maintenance.
They
want
an
easier
way
to
manage
all
those
services
in
application
and
frameworks,
and
they
will
not
manage
multiple
clusters
with
multiple
scheduler.
We
we
must
consolidate
thanks
to
the
container
revolution.
A
So,
let's
first
review
our
data,
we
had
a
guava
look
at
our
data
as
an
structures,
object,
store,
structure,
store
and
streaming.
This
is
basically
how
we
look
at
it
at
a
throughout,
even
if
it's
cloud
or
even
if
you're,
on
Prem.
This
is
your
data.
Decoupling.
The
data
from
our
entire
system
require
a
shift
or,
in
the
current
mind,
sent
aleko
system
shall
grow
from
a
dupe
mindset
of
distributing
the
data
itself
to
distributing
its
access.
A
When
you
run
in
the
cloud,
it
allows
you
to
access
your
data
in
a
distributed
fashion,
think
of
s3
or
dynamo
DB
or
key
Nessus.
You
don't
have
to
worry
whether
the
data
is
you
simply
access
it
from
anywhere
in
any,
and
even
if
you
don't
run
in
the
cloud,
you
can
access
your
data
using
some
services
like
looks
to
the
fests
for
objects.
If
you
keep
your
data
in
Cassandra,
you
can
access
it
without
much
problem
and
your
Kafka
cluster
can
be
accessed
from
anywhere.
A
This
is
what
people
are
actually
referring
to
as
cloud
native.
These
are
resilient
and
always
accessible
data
services
in
between
wave.
The
orchestration,
kubernetes
will
schedule
application,
analytics
tools
of
frameworks
and
manage
our
entire
configuration.
When
we
align
everything
to
a
single
unified
orchestration,
we
need
to
adapt
the
application
layer.
A
We
require
the
utmost
level
to
be
a
great
upgraded.
We
can
simply
run
anything
on
top
of
our
Orchestrator.
This,
furthermore,
can
application
must
be
cloud
native.
They
have
to
use
every
tool
kubernetes
has
to
offer.
Besides
our
application,
big
error
frameworks,
we
can
have
leveraged
several
experiments.
We
can
now
have
our
function
service,
our
very
own,
lambda.
A
Our
big
data
pipeline
massive
Ovilus
as
well,
and
actually
it's
already-
has
it's
no
longer
pipeline.
It's
a
living
system.
Our
system
and
services
are
constantly
processing
in
and
analyzing
data
accessing
the
data
simultaneously
running
function,
micro
services,
analytic
tools,
we
have
daddy
data
coming
in
or
pulled
out
for
my
IT
devices,
external
sources,
dashboard
and
many
more.
The
entire
layer
around
our
data
runs
on
a
unified
orchestration
data
is
being
accessed
from
anywhere
at
any
time.
A
This
is,
of
course,
a
blessed
move.
We
now
know
no
longer
need
to
worry
how
to
build
docker.
We
don't
no
longer
need
to
understand
how
to
deploy
it,
how
to
run
it
on
kubernetes.
We
simply
write
a
code
and
the
function
frameworks
will
simply
deploy
it
for
us,
it
will
compile
us
the
code
and
everything
will
simply
run
in
you
know
in
our
infrastructure.
A
Another
good
benefits
a
function
is
basically,
you
don't
have
to
bind
to
any
specific
language,
your
entire
stack
and
can
be
polyglot,
and
it's
already
has,
but
using
several
as
frameworks
is
the
way
but
comes
with
a
great
price.
Usually
it
it
comes
with
the
slow
performance,
so
slow
development
cycle
and
we're
almost
limited
to
HTTP
endpoints,
not
really
our
very
own.
Lambda.
A
Look
at
all
those
several
as
false
we'd
aguado
decided
to
build
a
real
real
time
service
platform.
Nucleo
nucleus
platform
can
have
any
event
source,
not
just
HTTP.
They
can
be,
of
course,
combined.
You
can
listen.
Some
honestly
Kafka
HTTP
kinases
with
the
same
function
code
that
you
wrote
this
allows
you
a
better
debugging
testing
and
execute
execution
cycle,
since
we
all
also
run
everywhere-
god,
not
just
kubernetes,
you
can
deploy
and
test
everywhere.
A
A
A
So
now,
you're,
probably
saying
okay
I,
listen
to
your
talk
and
everything,
containerized
I'm
using
function,
I'm
using
micro
services,
my
Sparky's
containerized
great,
but
now
I
have
thousands
of
containers
running
on
micros
traitor
and
each
one
of
them
is
gonna
open,
a
connection
to
my
data
service.
It's
gonna
create
a
huge
load
on
our
system
and
you're
actually
right
and
when
I
said
we
need
a
cloud
native
for
amongst
an
application.
It
meant
that
some
frameworks
need
to
evolve
as
well.
A
At
laguardia
we
are
dealing
with
very
large
color
clusters
and
needed
a
way
to
optimize
data
access.
We
build
a
solution
around
shell
memory,
which
brings
the
data
directly
to
our
application
memory.
The
solution
works
closely
with
kubernetes
to
allow
shared
connection
fast
data
access
and
fast
connection,
initialization.
A
A
Now,
if
you
are
not
running
with
something
native
liditz,
we
support
like
new
Cleo
spark,
Hadoop
and
others.
We
can.
We
also
have
you
this
same
solution
available
as
a
fuse
mount
which
you
can
use
a
flex
mount
a
flex
volume
to
using
kubernetes
and
read
directly
video
application
now,
just
like
with
nuclear.
This
entire
walk
was
open-source
and
you
can
check
out
the
solution
and
work
some
ways
how
to
to
leverage
it
with
your
data
services.
A
A
Let's
look
at
a
spark
example:
every
aspect
of
the
system
is
managed
by
kubernetes
deployment
services
and
even
the
configuration
itself
and
config
map
is
not
just
for
flat
configuration
startup
scripts
are
easily
managing
config,
Maps,
I,
know
and
I
will
later
on,
demonstrate
your
IRA
can
leverage
a
lot
of
the
tool
in
kubernetes
has
to
offer
to
manage
your
entire
deployments
and
how
much
simpler
is
using
helm
versus
chef
or
puppet.
You
don't
have
to
use
another
language
Yamma
which
you
have
to
use
already,
because
you're
unique
using
kubernetes
is
being
utilized
by
helm.
A
Another
demo
done
I'll
demonstrate
is
how
our
current
pipeline
looks.
Like
you
know,
plan
like
I
said.
A
current
pipeline
is
a
living
system.
The
data
is
simultaneously
being
accessed
for
multiple
location
at
once
so
I'm
taking
a
real
example
from
one
of
our
customers.
This
is
an
IOT
automobile
company.
They
have
their
cars,
sending
information
constantly
where
the
driver
is
some
metrics
of
the
cars
and
so
on,
and
everything
is
being
processed
in
real
time
and
injected
to
our
data
services
and
simultaneously
there
are
dashboard
showing
where
the
data
where
the
driver
is.
A
A
Now,
let's
create
something
that
allows
that
new
customer
to
access
its
its
new
namespace
I'm
gonna,
do
something
that
it's
not
something
you
should
do.
I'm
gonna
provide
it
with
the
option
to
be
a
cluster
admin.
Usually
we
provide
the
option
with
a
friend
grade
binding,
but
just
for
the
sake
of
the
demo,
we
create
a
new
role
binding
and
now.
A
A
What
do
we
have
here
is
raw
configuration
as
a
JSON
file
being
saved
directly
as
a
config
map.
It's
not
the
usual
way
to
use
a
config
map,
because
usually
people
tend
to
use
it
as
a
map,
a
key
value
or
simply
place
it
as
a
file
and
then
map
it
to
the
container
and
I.
Don't
think
that
I
I
really
love
doing
its.
The
initialization
script
I
also
place
in
a
config
map.
It's
allowed
me
to
better
control
which
which
parameters
I'm
doing
initialization
and
not
by
overriding
all
kind
of
VM
else.
Files.
A
A
A
Can
deploy
any
pre-loaded
function
or,
of
course,
provide
with
me
with
my
own
hit,
deploy
and
we'll
simply
ship
to
the
to
the
cloud,
in
this
case
our
kubernetes
cluster?
But
this
is
not
how
you
usually
want
to
do
stuff.
You
don't
want
to
open
another
IDE
or
another
another
tool.
You
simply
want
to
use
either
cube
CTL
or
other
command-line
tools
to
do
it,
the
automation
for
you.
So
of
course
we
do
have
an
automation
tool
for
you,
which
is
our.
A
A
A
A
A
Real
issue
here
is
that
you
will
count
now
the
function
that
I'd
recently
deployed
start
to
receive
the
events
and
populates
all
the
real-time
data
on
the
map.
This
is,
of
course,
with
a
lot
of
drivers
being
hammered
into
the
system,
but,
as
you
can
see,
everything
is
constantly
live
during
during
the
presentation
now
now,
I
have
two
acts:
2
2
way
of
accessing
the
data,
one
is
the
function.
The
other
is
the
dashboard
and
I
mentioned.
Also
that
data
engineers
might
wanna
use,
let's
say,
Zeppelin,
to
run
small
jobs.
A
A
A
A
Now,
during
the
talk
I
said,
it's
gonna
be
easier
to
run
with
kubernetes.
We
have
much
easier
deployment
and
no
one
actually
stopped
me
and
said
this
is
not
easier.
You
actually
keep
running
cube
city,
l
and
you're,
using
new
CTL
and
you're
still
editing,
yeah,
Mel's
and
init
scripts.
Anything
and
stuff
like
that
and
too
bad,
no
one
interrupted,
but
actually
yeah.
This
is
not
the
way
to
do
it.
This
is
not
the
way
to
deploy.
This
is
mimicking
the
old
way
of
doing
bad
stuff
on
your
deployments.
A
So
when
I
said
that
we
should
leverage
our
tools
in
the
in
our
toolbox,
helm
is
not
just
for
presenting.
How
easy
to
install
with
Elm
is
how
to
install
a
complete
application
with
the
helm.
So
as
a
cluster
is
shutting
down.
What
I'm
about
sure
is
that
nucleo
provide
you
with
the
Yama.
Yama
is
something
that
is
common
to
communities.
A
This
is
a
native
vml
kubernetes,
meaning
you
can
edit
it
and
really
pull
out
the
function
over
and
over
again
through
your
functions
so
taking
some
time,
but
meaning
that
you
can
take
that
yeah
Mel
and
use
it
in
helm,
using
all
the
templates
that
help
provide,
and
now
you
can
launch
everything
with
a
second
a
single
command
instead
of
all
the
stuff
that
I,
just
typed
in
which
is
I,
have
to
stress
out
it.
This
is
the
wrong
way
of
using
kubernetes.
A
A
Okay,
so
we
have
a
clean
cluster
and
now
this
is
the
way
to
do
everything
that
I
just
mentioned,
including
the
the
roll
binding,
including
everything
that
I
just
mentioned,
name
spacing,
and
everything
in
a
single
command.
This
is
the
proper
way.
This
is
something
that,
if
you're
being
around
the
talks
of
helm
and
how
to
use
communities.
A
A
So
few
tips
and
lesson
learned
we
we're
the
guado
had
along
the
way.
The
first
and
foremost
rule
and
I
can't
stress
how
important
is
it
should
really
read
the
menu
and
I'm
neglecting
the
effort.
You
should
definitely
read
the
manual
I've
seen
too
many
hacks
people
are
trying
to
do
with
kubernetes
and,
and
the
manual
is
very
comprehensive,
very
easy
to
follow.
It's
it's
sometimes
how
to
follow,
because
it's
length,
but
it's
not
something
that
you
say.
Oh,
it's
very,
very
difficult,
simply
do
a
copy-paste
of
commands.
A
lot
of
the
times.
A
Second,
is
the
community
community?
Has
a
great
community
is
not
limited
to
just
get
up.
You
can
find
the
slack
channel
you
can
fight.
The
group's
everything
within
the
community
is
helpful,
but
it
come
with
a
special
note
like
when
it
is
the
community
is
young,
so
sometimes
you
might
get
help
that
gonna
contradict
the
manual
try
to
recheck
everything
that
you
are
getting
help
to
know
the
tools
that
kubernetes
has
to
provide
during
the
development
of
many
scenarios
for
our
customers.
A
You
don't
know
how
much
I
want
how
many
times
I
use
port
forwarding
just
to
check
if
the
pods
is
doing
what
I
expect
it
to
do.
The
logging
collect
everything
that
you
have
using
cube,
CTL,
cube
CTL
is
a
great
tool.
You
should
really
understand
every
other
option
that
has
available
also
one
of
the
faults
that
most
people
are
having
in
cube.
A
Ctl
is
when
or
not
to
use
the
oh,
the
output,
because
the
output
of
the
amyl
allows
you
a
lot
of
times
to
understand
what
happened
to
the
service
or
what
happened
to
the
deployment
and
not
all
commands
except
the
all
flag
and,
like
I,
showed
you
always
navigate
with
helm
or
other
solution
that
you
choose
with,
but
stick
to
it.
Helm
has
great
great
options
great
into
that
great
understanding
of
kubernetes,
and,
as
you
can
see
once
you
really
understand
how
to
use
it.
A
A
Another
tip
that
we,
when
you
deal
with
large
cluster
and
many
applications
sharing
the
same
cluster,
is
they're,
not
the
overused
in
out
port,
not
both
it's
great
when
we're
doing
debugging
it
really
it's
great
because
you
now
know
where
to
access.
But
if
you
stick
with
static
node,
a
node
ports,
its
meaning
that
your
start
having
to
manage
all
those
node
ports
use
load
balancers.
There
are
great
options
within
kubernetes
now
the
sixth
item
might
look
trivial,
but
configuration
must
go
into
config
Maps.
Don't
try
to
force
them
in
into
all
kind
of
solution.
A
That
I've
seen
is
loading
files
from
the
House
passed,
but
someone
has
to
populate
that
file,
which
everything
goes
to
a
config
map,
which
very
very
easy
in
a
specific
case
to
a
config
map
is
dealing
with
the
amyl
Yamal
is
very,
very
tricky
syntax.
So
when
you
try
to
override
the
command
is
sometimes
end
up
with
resulted,
you
didn't
expect
so,
like
I've
showed
you
place
init
script
and
call
them.
Instead
of
trying
to
do
some
wizard
deal
with
the
ammo
and
the
last.
A
It's
not
really
related
just
for
kubernetes
for
any
deployment
large
clusters.
You
should
really
collect
operational
data
and
not
just
collect
it.
If
you
just
collect
it,
you're
done
nothing
with
it.
It
doesn't
mean
anything
you
can
simply
shut
it
down.
You
have
to
collect
it.
You
have
to
understand
it.
We
are
big
data
engineers,
so
we
need
to
understand
big
data
means
and
if
the
data
is
meaningless,
shut
it
down.