►
From YouTube: Keynote: Apache Beam Making Big Data Portable With gRPC - Ismaël Mejía, Software Engineer, Talend
Description
Keynote: Apache Beam Making Big Data Portable With gRPC - Ismaël Mejía, Software Engineer, Talend
Apache Beam is a unified programming model designed to create efficient and portable data processing pipelines. Portability in Apache Beam has two meanings: Users of Apache Beam can run their programs in different execution systems e.g. Apache Spark, Apache Flink, etc and they can choose their favorite (supported) language and be able to execute pipelines even if the targeted execution system does not support the language natively. In this talk we introduce Beam, its architecture based on translators (runners) and its portability framework: a set of gRPC services to coordinate the execution of functions isolated in language specific environments. The use of gRPC on Beam provides not only clear and easy to evolve contracts to coordinate the execution of functions but it offers nice isolation properties if combined with containers.
A
Hello:
everyone
thanks
for
joining
this
session,
I'm
going
to
talk
with
you
about
apache
beam
and
how
we
use
grpc
to
make
big
data
processing
portable
portable
between
different
execution
systems
and
portable
between
different
languages.
Also,
so,
first
I
introduced
myself.
My
name
is
ismael
mejia,
I'm
a
software
engineer
and
you
can
follow
me
on
twitter
in
this
handle
mostly
working
in
apache
big
data
projects,
basically
apache
abram,
apache
pmc.
A
I
also
contribute
to
others
and
I'm
a
member
of
the
apache
solar
foundation.
I
work
for
italian,
which
is
a
big
data
and
data
integration
company
that
has
software
opens
to
software
and
supporting
the
cloud
in
case.
You
want
to
take
a
look
about
okay,
so
first
we
are
going
to
talk
about
the
big
data
world,
and
this
is
that
small
introduction,
because
I
know
the
people
who
come
to
this
conference
mostly
are
into
grpc
services
web
services
stuff,
and
this
is
a
little
bit
different
from
a
different
use
case
and
interesting
too.
A
So
in
the
big
data
wars.
What
we
have
is,
basically,
we
have
big
amounts
of
data
that
is
distributed
in
different
machines
through
a
file
distributed
file
system
or
a
distributed
data
store,
and
this
part
this
we
divide
into
partitions
and
with
these
partitions
we
apply
some
functions,
for
example,
maps
or
that
we
use
to
transform
data
from
one
format
to
the
other.
A
This
is
for
the
best
case.
The
streaming
case
is
quite
similar.
The
big
difference
is
that
we
have
continuous
data
that
is
arriving
all
the
time
and
but
the
operations
are
similar,
but
with
the
kind
of
constraint,
and
that
is
the
fact
that
the
data
is
still
continuously
arriving
and
we
have
to
decide
at
which
point
we
want
to
calculate
or
aggregate
the
data.
So
this
is
when
we
stop
and
we
aggregate
so
this
needs
some
extra
functions
to
do
this
and
to
do
this
kind
of
programming.
A
What
we
basically
do
in
big
data
is
that
you
write
your
job,
that
is
a
set
of
steps
of
that
transform
the
data,
and
these
steps
are
coordinated
through
a
coordinator
process
and
assigned
to
different
workers
that
are
going
to
to
execute
the
task
and
the
tasks
are
in
the
end.
Are
you
user-defined
functions
that
the
the
programmer
put
into
into
the
system
so
in
in
in
the
java
cases
is
like
that?
A
A
They
like
the
expressiveness
of
the
language,
also
because
they
can,
they
could
reuse
code
that
they
already
have,
because
they
wanted
to
use
the
libraries
and
the
ecosystems
they
they
have
like,
for
example,
spite
or
you
know
this
kind
of
pandas
libraries
that
people
are
familiar
and
pretty
mature
and
good
and,
of
course,
also
because
of
the
communities.
That's
a
reason
to
prefer
some
language
above
the
others.
A
So,
but
this
is
something
that
we
have
to
give
importance.
The
importance
that
it
has
is
that
the
the
use
of
languages
changes
in
time
and,
as
you
can
see
here
in
the
last
15
years,
we
can
see
the
decline
of
java
versus
the
growing
interest
into
python.
Of
course,
javascript
huge
language,
but
more
people
want
to
have
more
languages
to
be
able
to
do
their
tasks,
not
only
python,
so
we
enter
apache
beam.
Apache
beam
is
a
project.
It's
an
open
source
project
in
the
apache
server
foundation.
This
was
donated
by
google.
A
So
the
idea
is
that
we
use
the
beam
apis
with
the
sdk
and
we
can
run
this
in
every
system
that
we
want
and
every
of
these
existing
systems
that
are
supported.
Of
course,
when
we
talk
about
unified
model,
the
idea
is
that
we
want
to
support
both
batch
and
streaming
computations
with
the
same
model.
The
idea
is
that
we
can
have
somehow
somehow
kind
of
deterministic
streaming
processing
by
assigning
an
even
time
to
every
element
in
in
the
system,
and
with
this
we
can
define
when
to
aggregate
the
data.
A
Just
what
we
want
here
is
to
deal
with
the
trade
of
latency
and
correctness,
so
we
cannot
wait
all
the
time
to
have
correct
results,
but
we
can.
We
can
adjust
this,
so
this
is
what
we
mean
with
unified
models.
So
basically
we
add
some
extra
semantics
with
transforms
to
the
to
the
system
and
when
we
talk
about
portability,
what
we
want
is
to
be
able
to
write
our
pipelines
in
our
favorite
language
and
then
be
able
to
translate
those,
so
they
can
be
executed
in
the
target
system.
A
In
this
case,
for
example,
we
want
to
run
it
in
spark
or
in
flink.
Well
we're
going
to
do
this
trans
transformation
with
the
runners
okay.
So
basically
the
model
force
like
this,
we
define
a
pipeline-
that
is
a
collection
of
that
uses,
a
collection
of
data
that
we're
going
to
process
in
parallel.
We
call
this
collection,
peak
collection,
and
then
we
start
to
do
transforms
to
these
collections
to
produce
new
new
new
collections
and
then
output,
the
the
datas
by
basic
data
flow
model
for
those
who
are
familiar
with
it.
A
It's
a
simple
graph
of
steps,
and
but
each
of
these
transforms
that
we
have
here
are
again
parameterized
by
the
user.
In
with
the
configuration
of
the
transform
the
the
the
user
defined
function,
let's
say
so
how?
How
does
this?
How
about
this
working
beam
until
recently,
for
example,
from
java
to
java
translation?
What
we
do
is
that
users
write
the
program
in
using
the
beam
sdk.
A
This
is
for
java
and
then
we
in
bim
we
translate
this.
We
translate
from
the
bim
functions
into
the
target
functions
if
you
are
familiar
with
the
spar,
for
example,
we
produce
an
rdd
and
we
do
rdd.map
and
we
map
the
function,
and-
and
all
of
this
is
quite
straightforward,
because
the
functions
are
similar
between
the
different
systems,
because
they
almost
all
do
them-
have
to
support
the
basic
same
functions
and
what
we
do
is
mostly
wrapping
and
unwrapping
and
unwrapping
the
the
functions
there.
A
A
Support
python
today,
for
example,
spark
spark
what
it
does
is
that
if
the
user
rises
the
program
in
python,
it
invokes
the
the
java
part
using
py4j
and
then
the
job
is
distributed
and
the
worker
is
going
to
do
the
task,
but
instead
of
doing
the
task,
it's
going
to
instantiate
the
java,
sorry,
the
python
process
and
they're
going
to
talk
with
true
unix
pipes.
This
is
how
it
works.
This,
of
course,
is
a
good
solution.
A
Also
you
have
the
the
you
lose
the
risk,
the
control
of
the
resources,
because
the
python
memory
is
not
inside
of
the
jbm.
So
it's
not
going
to
respect
the
limits
that
you
put
in
the
jvm,
so
it
can
go
out
and
it's
a
common
problem
and
more
important.
The
dependencies
cannot
be
ready
in
the
cluster
when
you
are
running
the
job,
so
it
is.
It
is,
they
should
be
prepared
or
they
won't
be
working.
A
So
this
is
another
issue
that
is
not
solved
by
this
approach,
so
beam
for
to
tackle
this
created,
what
we
call
the
portability
framework
and
the
portability
framework
is,
is
well.
The
main
goal
is
to
tackle
the
problem
with
how
to
execute
any
language
in
every
runner
in
every
existing
system.
Of
course,
we
have
some
constraints.
First,
we
need
we
have
to
support
java
and
python
because
for
historical
reasons,
that's
what
we
have
we
support
now
we
support
us
from
the
beginning.
A
A
A
The
fn
api
is
a
little
bit
more
complex
is
the
set
of
services
that
allows
us
to
control
the
execution
to
transfer
the
data.
I'm
going
to
see
those
in
detail
now,
but
first,
why
we
chose
grpc
what
usbc.
First,
it
has
an
efficient
serialization
format,
and
this
was
important
because
we
wanted
to
pass
the
bus
to
have
a
well-defined
messages
and
efficient,
of
course,
portable
code
offers
as
many
of
the
things
that
we
wanted.
First,
you
post
multiple
languages
and,
and
we
can
use
it
with
java
and
python
pretty
easily.
A
A
We,
it
was
extremely
tight,
so
we
can
have
a
more
representation
that
is
more
consistent
in
time,
we're
going
to
find
errors
more
easily,
not
like
with
json,
and
we
have
a
compact
and
efficient
encoding
and
decoding,
and
we
can
involve
this
like
we
do
with
our
also.
While
we
have
the
support
of
backwards,
compatible
support,
also
protocol
buffers
for
managed
versions,
that's
good
too,
back
to
your
pc.
Well,
other
thing
of
grpc.
A
That
is
really
nice
is
that
we
have
a
really
simple
service
definition,
so
it's
quite
straightforward
to
define
the
services
and
we
gain
the
network
performance
advantages
of
grpc.
We
get
the
the
multiplexing
because
of
http
2,
so
in
the
in
the
networking
side.
So
this
is
pretty
good
and
really
interesting
for
our
use
cases.
We
have
the
rich
communication
models
we
have
subscriptions.
We
have
bi-directional
streaming
which
can
apply
for
different
tasks
as
we
are
going
to
see
in
our
services,
so
the
portability
framework.
This
is
the
full
image.
A
Now
it's
a
little
bit
more
complex
than
than
before,
but
I'm
going
to
explain
it
in
detail,
don't
be
overwhelmed
by
the
image,
it's
quite
straightforward.
Once
jus
it's
explained
so.
First
we
have
now
the
users
write
their
program
and
instead
of
sending
the
code
we're
going
to
transform
all
of
this
into
a
protocol,
buffers
object
that
is
going
to
be
the
input
of
the
execution,
so
this
is
going
to
be
produced
from
python
from
java
in
the
same
kind
of
object
and
and
then
sent
to
the
to
the
job
server
to
be
executed.
A
A
It
allows
us
to
control
the
execution
of
of
a
job
of
a
pipeline,
and
if
you
see
this
is
just
a
submission
and
management
protocol,
we
even
if
you
are
not
familiar
with
your
pc.
This
is
the
first
time
you're
seeing
grpc.
I
think
you
can
immediately
understand
this.
I
mean
like
we
define
a
service
that
has
this
kind
of
methods
and
well
we
can.
We
can
prepare
an
execution.
A
We
can
get
the
state
the
current
state
of
the
job
we
want
to
run
and
we
can
cancel
and
a
running
job,
but
more
important
for
the
for
the
to
prove
the
semantics
of
grpc
is
that
we
can
also
be
interested
in
subscribing
to
some
changes
in
this
job,
and
this
is
common,
for
example.
I
can,
I
can
be
interested
to
know
to
have
an
alert
or
have
a
react
to
diff
when
a
message
is
finished,
so
we
can
do
this.
There
is
this
description
and
then
the
then
also
well.
A
Now
we
have
also
the
the
the
final
set
of
services
is
the
fn
api.
We
call
those-
and
this
fnapi
allows
us
to
invoke
the
function
inside
of
the
container
and
to
control
this
execution.
But
to
do
this
we
need
different
things.
First,
we
need
to
have
the
right
artifacts
as
part
of
the
container,
and
this
is
what
the
artifact
services
do,
and
this
is
basically
to
get
the
dependencies
and
okay.
So
this
is.
A
This
is
what
the
artifact
service
does
and
then
once
we
have
the
dependencies
what
well
we
can
see
the
service.
Sorry,
what
we,
what
you
can
see
this
we
can
put
artifacts
to
store
them
and
we
can
commit
a
manifest
to
save
their
their
as
they
hash
and
and
the
permissions
they
have.
Of
course,
we
can
retrieve
these
artifacts
and
use
them.
A
That's
what
the
what
the
container
does
and
this
container
that
this
we
call
hardness
is
is
doing
then
we
have
to
instantiate
the
hardness
the
container
and
to
do
this,
we
have
this
provision
api.
The
provision
api
allows
us
to
define
a
set
of
resources.
Let's
say
like
we
can
define
memory
for
this
container
limits
against
cpu
limits.
Also
so
this
is.
This
is
what
it
does.
Once
we
have
the
dependencies
and
being
container
resistant
heated,
then
it
comes
the
services
to
pass
the
data
and
the
functions.
A
So
we
have
the
data
plane
services
that
basically
allows
us
to
pass
data
straight
to
the
to
the
system
we
have
below
the
state
api.
That
is
mostly
for
caching
data
in
the
different
workers
and
the
data
apis,
as
you
can
see
here
is
quite
straightforward.
It's
just
bytes,
I
mean
bi-directional
streams
of
bytes
that
we've
other
are
tied
to
at
least
structures
that
are
going
to
be
using.
This
data
is
a
logical
stream
of
data
that
contains
the
elements
and
and
for
the
case
of
protobuf,
we
we
in
protocol.
A
We
have
this
limitation
of
that.
Oh
sorry,
in
protocol
buffers
we
have
this
difference
between
between
languages,
some
languages
support
a
maximum
size
of
two
gigabytes
or
there's
a
maximum
size
of
four
gigabytes.
Well,
in
the
case
of
beam,
we'll
assume
the
lowest
common
size
size.
That
is
two
gigabytes
for
this.
A
Okay,
so
now
the
control
apis,
the
control
apis,
allows
us
to
basically
control
the
execution.
So
we
sent
a
graph
with
the
specific
user
defined
functions.
We
stream
the
elements
and
we
get
the
the
the
elements
out.
How
is
this
done?
This
is
done
with
this
definition.
Again
we
have
a
bi-directional
stream.
A
The
request
here
is
to
process
a
bundle.
A
bundle
is
a
subset
of
data
that
is
going
to
be
tied
to
a
function
also.
Well,
of
course,
we
have.
We
can
request
to
have
progress
of
the
execution
and
on
another,
common
instruction
is
to
split
the
execution,
and
when
this
happens,
this
happens
when
we
want
to
divide
the
execution
and
pass
the
the
missing
part
to
a
different
task
or
a
different,
maybe
in
a
different
work.
This
is
this
allows
us
to
have
more
parallelism
how
this
works.
A
Well,
this
is
a
simple
diagram,
but
basically
we
register
the
functions.
Then
we
request
the
sdk
to
process
the
function.
We
prepare
the
functions
once
we
initialize
the
functions.
We
start
to
stream
the
input
elements
we
process
the
elements,
then
we
signal
this
is
the
end
and
then
we
get
the
outputs.
This
is
basically
how
it
works.
A
A
Well,
you
can
see
this
the
the
different
levels
you
have
and
the
the
timestamp
of
when
things
happen
and
the
message
this
is
pretty
straightforward,
also,
okay,
so
this
is
the
this.
Is
the
world
set
of
services?
This
is
how
we
re-implemented
beam
to
support
multiple
languages,
execution
and
now
we're
going
to
talk
about
the
present.
So
some
of
the
things
that
we
get
with
the
portability
framework
was
isolation
of
user
code.
A
A
These
services,
of
course,
the
the
all
the
erp
services
and
you
have
to
implement
an
api
for
the
language
and
the
integration
is
immediate,
with
all
the
different
systems,
so
we
don't
have
to
re-translate
into
java
or
into
flinks
or
into
spark
no.
Now,
if
this
is
supported,
it
will
be
supported.
We
just
have
to
work
in
the
language
path.
A
Of
course,
then,
the
present
we
have
now
our
implementation,
full
implementation
for
different,
open
source
systems
like
flink
apache
link
and
apache
spark,
and
the
google
data
flow
runner
now
supports
this.
It's
experimental.
This
is
not
open
source,
but
the
users
of
data
flow
in
the
cloud
can
use
it,
and
the
samsung
runner
also
makes
a
work
in
progress,
so
this
is
getting
more
material.
Now
we
have
this
thing.
We
call
the
protagon
the
capability
matrix.
A
A
We
also
are
now
testing
all
of
these
through
what
we
call
the
validus
runner
suite.
The
validate
runner
suite
is
a
set
of
is
a
set
of
they
say
like
a
tck
to
validate
the
completeness
of
the
model,
so
this
is
corner
cases
for
every
kind
of
transform
and
use
case.
We
run
this
for
every
pr
that
is
done
into
beam,
and
now
we
have
all
of
this
passing
for
the
classic
translation
we
did
from
java
to
java,
but
also
for
this
new
translation
with
all
the
grpc
services.
A
Thanks
to
that
now
we
have
support
for
golang.
That
is
the
most
recent
language
that
we
support
now
on
bim
and
go
is
the
first
language
that
is
pure
portable,
so
go
from
the
beginning,
translates
into
protocol
buffers
and
uses
all
these
services,
of
course
go
is
still
a
work
in
progress,
but
this
is
pretty
nice
that
we
can
now
run,
for
example,
go
pipelines
with
a
spark
with
thanks
to
beam
and
finally,
what
is
coming
now
is
just
working
into
performance
tests
to
catch
progressions
and
try
to
make
things
even
faster
okay.
A
So
we
had
nice
outcomes
of
this
work.
First,
one
was
that
since
now
we
can
support
python
like
running
in
different
open
source
systems.
Well,
projects
that
use
bim
python
api
now
can
be
used
also
in
the
open
source
site
and
in
particular
tfx.
That
is
this
tensorflow
project
to
deploy
production
to
deploy,
email,
machine
learning
pipelines
now
is,
can
be
used
in
the
open
source
runners
without
any
modifications,
so
they
can
use
all
these
data
validation
functions
that
are
pre-processing.
Everything
is
this
works
out
of
the
box?
A
A
For
example,
you
can
I
mean
you
can
wonder
how,
when
it's
interesting
to
support
different
languages,
one
case
can
be
when
you
don't
have
an
available
connector,
for
example,
to
a
data
store
in
your
language,
but
it's
in
the
different
language.
So
you
can
use
the
java
api
from
python,
for
example,
just
to
get
the
data
in
or
the
opposite
once
you
want
to
use
one
of
the
libraries,
for
example,
an
inference
function
from
tensorflow
from
java.
What
you
can
do
now
through
the
python
apis.
So
this
is
a
mix
and
match
but
really
useful.
A
Another
nice
outcome,
we
have
is
performance,
and
this
I
have
to
confess
that
I
was
kind
of
negative
when
all
this
architecture
was
proposed,
because
I
mean
I
I
expected
a
big
overhead,
but
the
the
interesting
results
that
are
in
this
graph
that
my
colleague
alex
saint
shared
is
that
with
once
you
have
more
data.
The
overhead
of
this
grpc
communication
and
services
doesn't
matter
that
much,
and
this
is
really
nice
because
it
proves
that
for
our
use
case,
that
is
mostly
big.
A
A
It's
a
good
base,
because
now
apache
flink
is
starting
to
use
it
to
use
the
to
execute
python
code
since
the
last
version,
and
also
they
have
a
project
called
stateful
functions.
That
is
very
interesting
because
you
want
to
take
a
look
at
this.
The
idea
here
is
to
have
lambda-like
functions.
Lambda
in
the
serverless
sense,
and
one
alternative
in
stateful
functions
is
that
you
can
collocate
functions
in
different
languages,
and
this
will
be
executed
with.
Let's
do
the
same
approach
with
this
kind
of
services
that
pass
data,
pass
the
function
and
then
running
it.
A
Finally,
well
and
more
related
to
this
conference,
while
using
grpc
was
really
nice
to
detect
the
changes
and
and
well
first
because
we
have,
we
put
well
first
to
detect
the
changes
and
add
new
features.
For
example,
we
we
have
things
that
we
didn't
conceive
correctly
from
the
beginning,
like
the
addition
of
environments.
A
A
We
can
also
support
new
features
that
were
not
in
the
beam
model
before
like
time,
your
families-
all
of
this
has
been
added
into
the
into
our
messages
contracts,
and
I
want
our
services.
It's
been
pretty
good,
of
course,
having
a
rich
communication
model
to
adapt
to
the
different
use.
Cases
has
been
pretty
useful,
also
for
beam
and,
of
course,
all
the
backwards,
compatibility
and
and
now
soon
versioning
that
we
will
have
has
been
some
problems.
A
Things
of
your
pc,
the
pain
points
that
we
have,
because
both
not
everything
is
perfect,
is
that
we
have
to
vendor
grpc.
But
this
is
so
a
common
problem
in
java,
because
grpc
and
protocol
buffers
leaks
from
others
of
our
systems
apis,
not,
for
example,
our
dependencies,
some
of
those
end
up
leaking
protobuf
and
we
had
a
mess
with
this
liquid
pull
at
above.
So
we
bend
our
grpc,
but
but
we
keep
up
today
with
the
latest
versions
as
much
as
we
can.
A
A
We
have
also
well
we
plan
now
to
have
better
ergonomics
that
running
all
of
this
together
is
more
easy
for
the
end
users
and
have
better
documentation
for
this
also,
we
of
course
will
plan
to
continue
the
work
in
performance
analysis
and
improve
and
improve
also
some
maybe
rough
areas
that
are
still
there
and
we're
going
to
have
deployments
how
to
so.
You
can
run
this
kind
of
architecture
with
the
different
services,
depending
on
your
use
case
in
kubernetes,
or
in
hadoop,
or
or
in
emr,
but
or
with
just
a
spark.
A
A
You
any
correlation
would
be
acceptable
using
it
and
seeing
and
and
using
beam.
Also,
if
you
are
not
just
into
the
erp
simple,
but
those
who
want
to
use
the
big
data
parts
would
be
perfect.
We
are
very
interested
in
having
you
if
you
want
to
go
deeper
into
this
design
want
to
learn
more
about
beam.
I
like
these
links
that
you
can
get
with
slides.