►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)
Making Big Data Processing Portable. The Story of Apache Beam and gRPC - Ismaël Mejía, Talend (Any Skill Level)
Big data applications have been an almost exclusive domain of Java and Scala developers. This not only frustrates engineers who prefer other languages and their ecosystems, but also impedes companies that already have their business logic written on other platforms from achieving the benefits of reuse when they build data-intensive applications. In this talk we introduce Apache Beam. A unified programming model designed to provide efficient and portable data processing pipelines. We will discuss in detail how Beam achieves portability by relying in two concepts: (1) Runners that translate the Beam's model so it can be executed in existing systems like Apache Spark and Apache Flink and (2) the portability APIs, an architecture of gRPC services that coordinate the execution of pipelines in containers to accomplish language portability.
About Ismaël
Software Engineer with more than ten years of experience designing and developing information systems for financial groups, telecom companies and startups. Focused on Big Data and Cloud architectures (aka Distributed Systems). He works at Talend France as an Open Source Software Engineer. He is an Apache Beam committer and PMC member and an enthusiastic contributor to multiple open source projects.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hi,
everyone
I'm
going
to
talk
to
you
about
a
patchy
beam
that
this
big
data
processing
framework
and
the
story
of
how
we
change
it
to
make
it
portable.
This
talk
has
two
aspects.
One
aspect
is
to
introduce
you
into
Apache,
be
in
the
path
shipping
model
and
the
second
part
is
discussing
the
technical
challenges
of
migrating.
A
The
current
infrastructure
we
have
into
a
multi
language
infrastructure
with
G
RPC,
so
I
am
is
mile.
Media
I
am
s4
engineer
that
works
up
talent
in
the
project
active
as
committed
and
as
part
of
the
PMC
in
Apache
beam
and
I.
Also
an
Apache
Software
Foundation
member,
so
I'm
a
little
bit
like
in
the
wrong
place,
because
this
is
cloud
native
confirm
add.
Well,
we
have
to
be
friends
all
of
us.
A
Google
publish
all
these
papers,
like
the
MapReduce
paper,
like
the
BigTable
paper
and
people
in
the
community
from
different
companies,
started
to
create
implementations
like
had
to
patch
base,
etc,
and
also
the
community
bring
new
ideas
like
expert
deep,
for
example,
with
a
nice
API
or
with
in-memory
processing
and
others
ideas.
So
there
was
like
two
parallel
worlds,
where
people
who
were
not
at
Google
were
always
looking
what
Google
did,
but
this
change
once
Google
submitted
this
their
SDK
for
data
flow
into
the
serve
our
foundation.
A
So
this
is
for
me,
excitement
this
is
the
point
of
union
of
these
two
words
and
beam
pretends
to
be
the
unified
programming
model
for
Big,
Data
and
two
aspects
that
are
really
important
is
that
we
are
we
care
about
efficiency,
but
we
can
also
call
portability,
and
when
we
discuss
about
unified
is
in
the
sense
of
that
we
can
do
batch
and
stream
processing
with
the
same
framework.
This
means
that
we
have.
A
We
don't
need
a
special
api's
for
both
words,
and
how
do
we
accomplish
this
with
some
special
elements
that
I'm
going
to
present
immediately
and
that
as
well
Ansari
than
the
runs
for
some
questions?
I
will
see
later
on
so
the
first,
the
first
important
concept
to
see
how
we
achieve
this
is
to
understand
the
separation
of
n
time
and
processing
time.
When
we
talk
about
event.
Time
is
the
time
this
won't
even
happen.
Let's
say
that
you
have
I,
don't
know
playing
AB
game,
your
telephone
or
listening
to
Spotify.
A
So
when
this
user
interacted,
he
created
an
event
and
d7
has
a
time
associated
to
it.
That
is
normally
not
the
same
time
at
which
we
are
processing
the
data.
This
is
the
processing
time.
Normally
it
arrives
later.
So
you
can
see
this
plot
that
we
have
these
events
that
are
arriving
and
in
an
ideal
world
we
will
have
data
arriving
in
time
and
this
diagonal
line,
but
most
reality
is
not
like
that.
We
have
all
these
narrow
delays.
A
Things
are
transmitted
food
and
even
we
can
have
data
and
because,
for
example,
somebody
lost
the
connection
to
the
internet.
So
this
this
event
arrives
once
he
takes,
gets
the
connection
back.
So
this
is
the
case
that
for
normal
data-
but
this
is
the
case
of
death-
arrives
in
return
late,
so
beam
two
to
be
able
to
compute
all
this
data
in
a
precise
way,
but
also
in
a
fast
way,
I
mean
with
low
latency.
A
Ask
for
questions
these
four
questions
that
what
resource
are
calculated
wherein
Aventine?
Are
we
calculating
these
bristles?
When
are
we
in
processing
time?
Are
we
computing
materialize
in
this
resource
and
how
we
refine
the
results?
Okay,
this
sounds
a
little
bit
abstract,
but
with
so
many
missions
is
going
to
be
more
clear.
So,
let's
see
if
the
first
question
is,
what
are
we
doing
so
in
this
case,
we
are
just
going
to
sum
all
these
numbers
that
we
saw
in
the
plot.
A
So
this
happens
with
the
progress
of
time
that
when
we
are
processing-
and
we
are
just
some
in
data-
but
when
we
arrive
to
the
top,
we
compute
the
data.
Let's
not
forget
that
this,
this
computation
happens
when
we
decide
that
it
happens,
and
if
we
have
an
infinite
source
of
data,
we
have
to
say
when
and
what
are
we
going
to
include
in
this
computation?
So
this
is
the
classical
batch
case.
A
This
is
MapReduce
for
all
of
those
of
you
who
know,
but
when
we
have
this
streaming
case,
we
need
to
define
what
the
interval
of
time
we
care
about,
and
we
can
do
this
with
Windows.
We
can
define
a
window
that
allows
us
to,
for
example,
divide
the
events
into
chunks.
Let's
say
what
what
what
was
the
most
listeners
song
in
between
10:00
and
11:00
today,
for
example.
So
this
event
times
get
grouped.
This
does
get
group
in
time
into
Windows,
and
then
we
computed
later
with
this.
Why?
A
This
because
we
need
to
have
answers
faster.
We
cannot
wait
all
the
life
for
all
the
events
to
arrive,
so
we
do
this.
But
again
we
have
a
say
again
the
same
problem
when
we
trigger
the
computation
of
this
data.
When
we
do
the
accumulation
we
have
to
decide
and
the
way
we
decide.
This
is
by
using
triggers
and
triggers,
and
in
this
case
we
were
going
to
discuss
a
trigger.
That
is
the
watermark.
Do
you
remember
the
straight
line
I
mentioned
before
this
straight
line?
A
Is
the
ideal
case,
but
the
watermark
is:
is
it
in
the
real
life
is
not
so
clean,
it
could
be
on,
and
this
deadline
in
in
the
indicates
that
I
won't
receive
more
data.
So
this
in
the
beam
connectors,
send
this
this
line
to
us
this
data
of
the
watermark,
and
then
we
decide
to
cut
and
compute
once
the
water
masses
is
complete,
but,
as
you
can
see
in
this
graph,
the
first
even
arrive
late,
this
red
one
and
we
lost
accuracy
to
compute
faster,
have
low
latency.
A
Well,
we
lost
a
little
bit
of
accuracy,
and
this
is
how
we
refine
the
model.
Then
the
last
question,
so
we
can
tell
things
like
in
this
code:
I
forgot
to
say
that
the
top
level
is
Java
low-level
spite
on
bottom
sorry
in
the
in
the
graph
and
the
way
we
compute.
This
is
with
what
we
can
tell
the
system.
Ok,
wait
one
more
minute
of
processing
time
and
bring
all
the
extra
data
that
arrived
in
this
minute,
or
you
can
also
count
how
many
elements
arrive
afterwards,
that
are
in
this
window
and
also
compute.
A
A
So
all
these
color
box
are
transformations
and-
and
we
call
P
collections-
PP
turns
from
those
and
P
collections
the
lines
that
go
out
of
of
these
boxes.
This
is
what
kind
of
transformations
are
we
talking
about,
what
the
classical
maps
transformations
that
we
call
paru
in
beam.
We
can
transform
pill
element
wise,
so
just
apply
in
one
function
and
that's
it.
A
We
have
group
by
key
and
reduction
like
transformations
that
are
like
counts
and
psalms
and
chorus
that
correspond
to
reduce
in
MapReduce,
and
we
have
all
these
windowing
and
triggering
ones
that
cover
the
the
concepts
that
we
just
quickly
view
to
have
more
precise
computing
computation.
So,
in
the
end,
Apache
beam
is
the
the
has
three
kind
of
users:
the
end
users
who
write
their
programs.
A
We
have
SDK
writers
who
want
to
create
new
SDK
that
implements
the
model
of
Apache
beam,
so
all
these
concepts
of
doing
those
triggers
etc,
and
we
have
runners
who
are
the
system
that
execute
this.
This
graph
is
interesting
because
you
can
see
that
Java,
Java
and
Python
can
be
language
that
are
used
and
and
running
one
in
some
of
these
different
systems
like
Apache
spark
or
flink
or
Google
dataflow,
and
so
today
we
have
all
these
runners
that
are
implemented
or
are
in
progress.
A
At
least
you
can
see
that
we
cover
most
of
the
most
popular
ones.
We
have
in
the
bottom
d2
cloud
native
ones
that
are
the
Google
one,
that
you
don't
care
about
infrastructure
at
all.
You
just
send
your
job
and
it
executes
and
I
and
IV
streams,
one
but
the
world
there
are
also
some
that
are
working
progress,
but
there
is
something
that
maybe
we
use
them
thing
really
well.
We
saw
this
diagram
is
that
all
these
are
Java
based
all
these
executing
the
java
virtual
machine.
A
So
the
normal
question
that,
after
someone
understand
the
the
presentation
a
little
bit
more
is
ok,
so
I
have
a
Python
pipeline
at
pipeline
and
I
want
to
execute
it
with
this
part
owner.
What
do
I
do?
I
translate
this
into
PI
spark
say:
no,
ok
can
I
execute
Python
or
not.
You
have
a
base,
runner
mm-hmm,
and
the
third
question
is
no:
can
I
use
tensorflow?
This
is
a
fan
favorite
of
the
last
year
and
run
it
with
a
Java
pipeline.
No,
and
then
last
one
is
ok.
A
The
Python
SDK
doesn't
have
a
connector
for
Kafka,
can
I
use
the
one
that
is
in
Java
to
connect,
and
the
answer
to
all
of
those
is
definitely
no.
So
let's
say
that
look
this
diagram
that
we
show
before
it
was
nice,
but
in
reality
today
we
are
living
in
this
world,
so
the
only
way
to
execute
Python
in
Apache
beam
is
to
go
straight
to
dataflow.
So
if
you
have
an
on
premise
thing
with
the
spark
or
in
Hadoop,
you
cannot,
you
cannot
run
it.
So
this
is
the.
A
This
is
a
big
problem,
because
this
is
like
half
of
the
model.
That
is
missing
of
the
idea
at
least
of
beam,
and
this
is
what
the
portability
IEP
is
for.
So
the
portability
API
is,
is
a
set
of
services
and
contracts
to
be
able
to
do
this,
but
first,
let's
see
what
we
do
today.
So
this
is
how
we
translate
how
a
runner
translates
a
by
a
pipeline
written
in
beam,
so
the
user
just
takes
the
SDK
it
was.
The
code
sends
the
pipeline
that
is
Java
code.
A
It's
executed
in
it
passes
by
a
runner
that
translates
this
thing
into
a
native
job
that
you
see
inviolate
there
and
this
nut
job
gets
distributed
by
the
arm
system.
Let's
a
spark
or
fling
into
the
different
master
and
workers
and
in
the
end,
the
the
functions
that
we
applied.
This
part
do
functions
as
this
Maps
or
this
combiner,
the
or
reducers
get
executed.
The
user-defined
functions
that
get
executed
in
a
in
a
worker
in
the
clusters
now.
So
this
is
easy
to
do
in
Java,
because
well,
we
can
play
a
lot
with
Java.
A
We
can
play
with
reflection.
We
can
extract
that
compose
it,
but
if
we
think
about
Python,
then
we'd
have
to
think
in
a
different
ways
like
that.
So
what
was
the
real
problem
we
were
achieving
fighting
for.
We
are
trying
to
execute
code
in
any
language,
never
run
it,
but
we
need.
We
have
some
constraints.
We
need
to
still
support
the
code
that
exists.
We
are
not
going
to
change
the
user
part.
We
need
to
provision
an
executed,
respected
environment
I
mean
if
the
person
is
going
to
write
Python.
A
Probably
he
would
like
to
have
all
the
Python
libraries
that
he
has
for
his
project.
We
need
to
be
performin,
have
low
overhead
for
this,
and
we
have
to
support
both
multiple
languages,
of
course,
because
every
SDK
can
be
in
a
different
language
and
we
want
something
easy
to
evolve.
So
an
initial
idea
could
be
just
to
take
this
initial
image.
A
The
of
the
on
top
of
the
of
the
runner
and
add
a
process
in
the
side
to
run
Python
like
this
for
couple
at
operating
system
thread
and
I
roam
there,
but
then
you
have
all
the
isolation
problems,
so
the
natural
ideas
to
expand
this
into
a
docker
container.
So
that's
what
what
this
whole
idea
is
about.
We
had
a
container,
so
we
can
have
now
the
user
code
really
into
this
container.
A
So
then,
what
when
we
had
this
diagram
of
the
transformation?
You
can
see
that
these
two
transforms
now.
If
they
are
written
in
Python,
they
will
become
this.
They
will
become
a
call
of
G
RPC,
but
I
want
to
talk
about
this
later.
That
then
executing
the
worker,
the
Python
code
and
L
get
is
get
this
out.
So
how
can
we
do
this?
We
can
do
this
with
three
elements
that
we
call
the
runner
API.
A
This
is
your
baby
I
at
the
F
and
API,
so
the
runner
API
is
a
way
to
represent
all
those
concepts
I
discussed
at
the
beginning.
All
these
B
transforms
of
these
windowing
triggers
all
of
this.
We
have
struct
it.
Then
then
we
have
an
inner
present
in
a
misrepresentation
way
that
is
going
to
be
protocol
buffers.
Then
we
have
the
job
API.
A
So
this
will
serve
to
use
your
PC
for
this.
Why
well,
the
classical
characteristics
of
CRPC
is
a
it's
an
efficient,
serialization
format,
the
uses
as
an
efficient
written
format.
This
protocol
buffers
is
for
multiple
languages
and
at
least
it
supported
the
two
we
wanted
Java
and
Python.
It
has
a
really
simple
service
definition
and
if
you
haven't
seen
G
RPC
before
I
will
show
you
and
I
am
pretty
sure
that
you
are
going
to
understand
the
ideas
quite
quickly
and
has
pretty
decent
network
performance
uses
multiplexing
for
HTTP
2.
A
So
we
are
going
to
be
able
to
mix
different
things
in
the
main
same
channel
and
it
has
rich
communication
models
like
subscriptions
and
bi-directional
streaming
so
this.
This
is
why
this
is
the
new
architecture
of
how
we
do
things
now
is
slightly
more
complex.
Now
we
added
some
boxes,
so
we
added
the
job
server
that
allows
us
to
control
the
job.
As
I
said,
we
added
this
artifact
station.
That
I
will
explain
in
detail
some
moments
and
we
added
also
the
set
of
services
there
in
the
top
in
the
bottom.
A
Sorry
that
allow
us
to
control
the
execution.
We
are
going
to
detail
all
of
this
okay,
so
first
we
before
we
send
the
pipeline
in
Java
code.
Now
we
transform
this
pipeline
into
this
abstract
representation
and
protobuf.
This
is
the
first
step,
but
the
user
still
sees
this
name
Java
code
or
Python
code.
Then
you
see,
this
is
an
example
of
the
definition
in
protocol
buffers.
A
As
you
can
see,
a
polonaise
is
a
set
of
components,
and
each
component
component
has
some
of
the
things
that
these
functions.
Do
you
can
see
concepts
that
I
mentioned
like
Purdue
would
combine
another
the
lighteners
speak
about,
but
you
get
the
idea,
and
there
are
all
these
hobbies
structures.
Data
structures
inside
there
is
the
job
server
and
the
job
server
in
the
diagram
has
a
service,
and
this
is
a
RPC
service.
A
Also-
and
this
is
a
very
interesting
use
of
ER
PC-
you
can
get
a
stream
of
result,
so
you
can
somehow
somehow
kind
of
subscribe
to
changes
in
the
state
or
messages
that
are
important
for
us
to
track
and
when
we
say
we
say
use
subscribe,
all
is
the
runner
who
is
forking
this
execution.
So
it's
going
to
subscribe
to
see.
A
What's
going
on
all
the
time
we
are
trying
to
solve
this
line
of
between
both
okay,
so
now
this
these
are
this
set
of
services
that
allow
us
to
control
the
execution
and
go
go
with
it.
The
effin
API,
the
goal
of
the
airy-fairy
API
is
just
to
control
the
interaction
between
the
runner
and
a
and
the
container
they
harness
in
this.
In
this
talk,
so
first
element
of
this
is
the
artifact
element,
so
this
is
just
to
stage
things
into
the
docker
container,
then
to
the
harness.
A
Why
we
need
the
staging
if
we
could
have
a
container.
That
has
everything.
That's
a
good
question,
but
well,
if
we
are
going
to
distribute
this
to,
the
general
public
will
probably
will
create
a
general
docker,
and
then
people
can
stash
fing
if
they
want
or
create
their
on
docker
images.
So
this
let's
say
that
this
service
can
be
optional
if
people
provide
their
own
images.
A
Then
we
have
the
provision
service.
The
provision
services
are
about
the
resources
in
the
container,
how
much
CPU
we
need
how
much
ram
we
need
a
memory
I
mean
if
we
need
these
space
and
this
kind
of
stuff.
All
of
these
goes
in
in
the
provision
API,
then,
and
more
interesting.
This
is
how
we
send
data
between
both
of
them.
We,
the
runner,
just,
opens
a
connection
and
just
sends
bytes
yes,
bytes
going
on
and
in
the
in
the
container.
A
There
is
this
thing
we
call
quarters
in
beam
that
allowed
to
transform
from
bytes
into
the
representation
we
have,
so
people
can
still
be
using
Pico,
for
example
in
Python
or
whatever
the
serialization
they
want,
and-
and
this
is
also
decoded
back
into
bytes
and
send
the
coded
back
into
my
story
and
sent
over
the
wire.
We
have
also
an
extra
service
that
this
state
API
that
was
seen
further
okay.
So,
as
I
said,
we
send
bytes
back
and
forth,
and
this
for
to
do
this.
We
use
protocol
buffers,
but
we
found
some
problem.
A
Some
limitation
with
protocol
buffers
is
that,
depending
on
the
language
is
the
biggest
message
in
protocol.
Buffers
can
be
to
bytes
or
4
gigabytes,
so
to
enable
compatibility
between
us
between
all,
we
decided
to
use
the
lower
limit,
and
the
data
is
is
worse
with
logical
streams,
which
means
that
our
add
one
element
can
be
represented
composed
by
different
messages
of
protocol
of
buffers
I
mean.
So
this
is
how
data
works.
This
is
the
state
API.
The
state
API
is
all
around
having
state
in
the
executor.
A
So
sometimes
we
can
compute
things
and
give
this
data
back
to
the
container,
and
this
is
just
mostly
like
a
like
state
store
in
the
kroner,
but
we
encourage
the
container
to
have
a
cache
also
of
this
to
avoid
round
trips
because
well
we're
care
about
performance
in
this.
So
that's
that's
the
idea
here,
and
this
is
fearful
to
useful
to
do
things
like
site
inputs.
That
is,
for
example,
when
I
have
a
collection
of
data
that
I
care
like
I.
Don't
know
that
is
a
small
thing
that
I
can
materialize
like
an
index
table.
A
Yes,
I
can
inject
this
as
a
state
in
their
honor
and
even
cache
it
in
the
in
the
container
okay.
So
this
is
the
more
interesting
one.
The
control
API
allows
us
to
orchestrate
all
the
execution
of
things.
The
this
is
the
one
who
decides.
Okay,
I
want
to
process
a
bundle.
The
design
is
a
set
of
elements.
I
send
you
this
little
part
of
the
graph
and
you
execute
it,
and
you
give
me
the
results
back.
A
This
is
this
is
an
example
of
the
definition
of
the
messages
and
the
service,
so
we
send,
for
example,
a
request
for
a
bone
for
processor
bundle
like
you
see
that,
and
there
are,
there
will
be
some
other
instructions
D
not
here.
So
this
is
how
we
process
a
bundle.
There
is
some
preparation
steps.
First,
we
use
send.
The
offenders
are
no
registered.
A
We
get
all
these
inputs
and
outputs
ready
to
be
connected
back
and
forth,
and
then
we
use
send
the
trigger
to
start
executing
the
functions
the
user-defined
functions,
so
it
is
when
you
execute
the
Python
call
and
this
this
this
process
in
the
end,
since
the
outputs
back.
So
this
is
this
is
more
or
less
how
it
works.
As
you
can
see,
there
was
a
game
between
between
control
and
data.
Api
is
to
do
this,
got
our
services,
let's
say
so.
A
Finally,
we
have
this
login
service
that
allows
us
just
to
get
the
locks
of
container
out,
so
we
just
want
to
know
because
well,
things
are
happening
to
container,
but
the
person
who
runs
the
cluster
probably
is
running
this
on
Hadoop
or
another
different
system
that
he
wants
to
lock
the
locks
there.
So
this
is
how
we
stack
the
locks.
A
So
what?
What
are
the
advantages
of
this
approach
is
that
we
gain
this
isolation
of
code.
We
isolate
the
configuration
of
the
user
environment
and
this
is
nice
because
we
can
provide
all
the
dependencies
in
one
big
chunk
we
or
or
provide
it
with
artifacts.
As
we
said,
maybe
so
we
can
support
multiple
language
execution
and
eventually,
with
this,
we
can
support
mixing
pipelines
in
different
languages,
and
this
is
important
for
the
tensorflow
example
that
we
mentioned.
A
For
example,
if
someone
with
Java
knowledge
just
want
to
add
some
small
part
in
their
position
that
makes
Python
invokes
transfer
flow.
We
can
do
this
and
also
it's
important,
because
in
Java
we
have
support
for
SQL,
so
you
can
send
SQL
queries,
but
this
is
not
in
Python,
so
the
Python
people
can
call
the
SQL
by
invoking
the
Java
harness.
A
A
A
So
what
the
current
status
of
this,
and,
what's
the
future
work
of
this
current
status,
we
have
a
universal
local
runner
that
this
a
local
runner
to
test
these
things.
So
it
uses
to
all
this
machinery
and
allows
to
invoke
Python
from
any
language
there.
We
are.
We
have
an
ongoing
rewrite
of
the
fling
runner
to
support
the
portability
API,
and
the
great
news
is
that,
just
three
or
four
weeks
ago,
for
the
first
time,
we
got
the
word
count
of
Python
being
executed
in
fling
with
all
these
services.
A
Of
course,
workouts
like
the
hello
world,
but
is
is
already
covers
that
most
of
the
things
are
in
the
right
place.
We
have
to
go
ahead
and
also
we
have
a
newcomer
that
that
we
also
be
when
able
to
test
with
the
finger
owner,
and
this
is
the
go
SDK.
So
the
go.
Sdk
is
an
ongoing
workouts
wind
up
in
community
and
the
world
go
SDK,
you
can
see.
The
basic
we
were
counting
in
go
is
the
first
SDK
that
is
natively
written
to
use.
All
of
this
and
disciplines
are
really
interesting
opportunity.
A
Also
because,
as
you
see,
go
produces
static
binaries
and
we
can
invoke
faster
things
with
this
and
particular
we
can
invoke
net
native
things
in
the
container,
so
this
will
be
quite
useful,
for
example,
if
you
want
to
do
manually
some
kind
of
computer,
graphics,
algorithms
in
GPU
or
stuff,
like
that,
I
think
this
is
a
good
use
for
this.
So
what's
it
going
and
what's
the
future,
where
well
still
working
in
completing
the
full
model,
soup
all
have
an
estate
having
Windows
triggers
all
these
concepts
of
the
beginning.
A
We
discuss
in
how
to
do
matrix
correctly
with
the
released
I
mean,
because
we
also
we
need
this
dance
background
for
between
the
worker
and
the
runner,
sorry
and
the
harness
we
have
not
tested
yet
to
invoke
ru
connectors
between
different
languages.
This
example
I
mention
of
calling
the
Kafka
connector
from
Python.
If
it's
not
exist
in
multiple
languages,
pilot
panels
have
not
be
test
but
should
work.
Of
course,
all
of
this
means
that
we
are
not
ready
for
production,
so
this
is
one
of
the
bigger
goals
in
the
rest
of
the
year.
A
This
is
a
lot
of
work,
so
I
I
don't
expect
that
we
have
this
before
at
least
q4,
and
also
we
have.
We
have
to
support
all
everyone
else
like
this
part
and
make
this
more
user
friendly,
for
example,
because
we
would
be
setting
the
abstract
yes,
I
can
invoke
ro
tensorflow
from
Java.
But
how
do
you
do
this?
In
the
programming
level?
We
have
to
refine
some
things,
and
finally,
we
have
this
validation
tests
that
we
already
use
for
the
Java
native
versions.
This
is
like
our
TCK,
that
is
valid
compatibility.
A
Caitli
say
that
is
the
the
ballot:
let's
run
our
tests,
so
we
have
to
validate
that.
All
of
this
is
correct,
semantically.
So
this
presentation
was
in
part
well
to
make
you
the
people
interested
in
these
subjects.
Aware
of
this
ongoing
work,
see
that
is
interesting.
I
find
is
pretty
cool
from
personal
point
of
view
and
just
to
invite
also
people
to
contribute
or
to
come
just
test
and
play
with
these
report
box
and
do
stuff
like
that.
A
Not
only
the
portability
is
decay.
You
can
also
come
with
new
ideas.
For
example,
one
really
exciting
idea
I
found,
is
to
implement
the
dotnet
SDK.
This
will
bring
to
the
community
all
the
drug
network,
with
all
the
good
things
that
it
has.
If
someone
is
interested
also,
you
can
think
about
creating,
maybe
an
active
runner
that
could
be
implemented
in
goal
and
just
replace
flink
or
a
spark
or
whatever
I
mean
there
are
plenty
of
ideas
to
contribute.
A
Not
only
these
know
we
could
be.
Moscow
is
doing
a
lot
of
other
stuff
in
the
sides.
If
you
are
interested
well
already
in
version
2
5,
the
first
stable
version
was
one
year
ago.
So
this
is
a
good
moment
of
stability
in
the
project.
A
part
of
this
internal
changes
and
we
have
a
vibrant
community
of
companies
and
contributors
working
in
the
project
and
some
of
the
new
things
were
working
on
a
part
of
the
portability
and
the
go
SDK.
A
Is
they
move
to
a
new
API
of
connectors
that
is
called
a
splitter
will
do
a
fence?
There
is
a
really
really
cool
subject,
but
well
no
time
to
talk
about
this,
but
it
allows
to
create
more
rich
container
cocoa
connectors.
That
spark
or
flink
all
our
systems
have,
if
you
have
the
time,
take
a
look
at
the
presentation
on
a
strata
that
was
some
months
ago
by
Eugene
and
also
there
is
this
schema
where
the
collections
that
they
are
like
typed
typed
collections
to
be
executed
a
little
bit
like
Sparta's.
A
So
this
is
just
to
greet
some
of
the
people
involved
in
this
work.
Remember
opens
forces
team
game,
so
all
of
these
guys
worked
really
hard
to
put
this
into
place,
and
this
is
some
referencing
in
case
you
want
to
check
later
on
in
this
light.
Thank
you,
I'm.
Sorry,
if
I
went
a
little
bit
fast,
but
my
presentation
was
intended
to
be
for
45
minutes
and
I
realized
today
that
was
35,
so
it
was
like,
but
at
least
you
have
a
quite
complete
deck
to
check
afterwards
so
on.
This
is
all
thank
you.
A
A
A
Well,
this
is
different.
We
have
also
have
it.
Would
we
okay,
I,
just
repeat
the
question:
what
about
groovy
and
all
these
languages
that
are
allowed
to
execute
like
a
scripting
languages
on
top
of
the
Java
Virtual
Machine?
We
have
support
for
this
already.
There
is
an
extension
on
being
that
is
not
married
yet,
but
I'm
working
on
this
pull
request,
so
I
am
aware
of
it
that
allows
to
use
the
scripting
languages,
but
just
in
a
primitive
way.
A
Let's
say
that
we
use
it
to
do
par
to
us
or
do
things
like
that,
but
this
is
not
not
far.
Also,
there
is
an
edge
here
that
is
the
skull,
implementation
of
them
of
the
BIM
model
or
the
SDK
a
list,
so
this
exists
already,
but
this
is
more
to
cover
the
whole
ecosystem,
because
well,
you
can
have,
for
example,
JSON
to
execute
Python
on
Java,
but
you
don't
can
use
all
the
dependencies
of
Python.
So
this
is
what
we
try
to
achieve
any
other
questions.
No.