►
Description
On Sep 27th, 2020, Vijay Kiran, Teodor Heggelund, and Daniel Slutsky interviewed Anthony Khong.
We began with a short presentation about Geni -- a Clojure dataframe library that runs on Apache Spark. The name means "fire" in Javanese.
https://github.com/zero-one-group/geni
Then, the conversation evolved around more general subjects, such as Apache Spark in general, other projects by Anthony, and the Clojure data science ecosystem.
Note that Daniel's comment at 57:41 is wrong.
SICMUtils enables some options for auto differentiation, as commented by Markus Agwin:
https://clojureverse.org/t/video-recording-scicloj-interview-3-anthony-khong-about-geni/6643/3
A
C
A
B
A
B
So
I'm
going
to
be
talking
for
15
minutes
before
we
get
into
just
the
informal
chat.
I
guess-
and
actually
I've
presented
guinea
before
I
think
about
a
month
ago,
in
cyclone
as
well,
but
that
was
in
a
lightning
talk,
so
I
had
only
five
minutes,
so
this
is
going
to
be
an
elongated
version,
where
I
can
elaborate
a
little
bit
more
on
some
of
the
details
and
some
of
the
design
goals
for
guinea.
B
B
Hopefully
so
yeah,
so
the
plan
of
attack
for
today
is
I'm
gonna
briefly
talk
about
what
it
is,
then,
some
of
the
design
goals
that
go
into
developing
guinea
and
some
of
the
future
plans
tentative,
but
we
go
to
it.
So
what
is
guinea
right?
So
I'd
like
to
call
it
a
closure
data
frame
that
runs
on
spark,
and
the
first
thing
is
that
it's
an
idiomatic
closure
data
frame.
B
What
that
means
is
that
it
should
be
nice
to
read
enclosure,
and
it
should
be
nice
to
write
as
well
enclosures
it
doesn't.
It
shouldn't
look
foreign,
and,
apart
from
that,
it's
it's
a
it's
a
data
frame
library
right,
so
you'd
expect
that
it'll
be
able
to
do.
You
know
some
of
the
typical
stuff
like
reading
reading
data
from
file
counting
number
of
rows
and
seeing
what
columns
you
have
right.
B
So
this
is
an
example
of
a
group
by
aggregate
operation
and
guinea.
It
will
look
similar
if
you're
a
spark
user
a
familiar
if
you're
a
spark
user,
because
a
lot
of
this
would
resemble
spark
and
method
chaining,
that's
very
common
in
scala,
but
then
also
it
should
look
like
closure
as
well.
So
we
can
see
that
it
understands
keywords
and
yeah.
It
uses
this
threading
threading
macro
all
right,
so
this
is
grouped
by
aggregate
and
and
sore.
B
B
So
that's
what
training
a
machine
learning
model
looks
like,
and
the
third
thing
is
that
there's
rdd
support
right,
so
rdd
is
a
resilient
distributed.
Data
set,
which
is
just
lower
level
spark.
So
a
lot
of
what
you
do
in
guinea.
Is
you
use
the
the
built-in
function
that's
available
already,
but
then,
when
you
need
to
do
custom
computations,
you
need
to
drop
down
into
lower
level
spark,
and
that's
where
you
do
rdd.
B
And
finally,
it
comes
with
its
own
command
line,
interface
right
and
I
think
I'm
going
to
elaborate
more
on
this
in
a
bit,
but
the
idea
being
that
it
needs
to
be
fast
for
you
to
start
and
start
querying
stuff.
That's
why
you
know
it
comes
with
that,
but
more
on
that
in
a
bit,
but
I
mean
a
fair
question
be
like
why.
B
Why
would
you
be
using
this
right
and
I
think
it's
sort
of
like
an
intersection,
or
rather
the
union
of
y
spark
and
white
closure
and
maybe
in
cyclos?
I
wouldn't
elaborate
much
on
why
closure,
but
then
why
spark
is,
as
I
think,
there's
a
lot
that
we
can
say
about
it.
First,
there's
a
lot
of
developers,
there's
even
a
company
backing
it
right,
so
speed
and
scalability.
B
B
So
so
that's
just
a
very
quick
introduction
of
what
it
is.
The
second
section
that
I
want
to
talk
about
is
just
elaborate.
Some
of
the
design
goals
right
and
for
me,
working
as
a
data
scientist
right
work
with
this
feedback
loop,
a
lot
you
you
have
some
idea
about
the
data
that
you're
working
on
and
then
you
need
to
translate
that
into
query
and
you'll.
Wait
for
your
query
to
start
to
give
you
some
answers,
then
you
build
on
that
idea
to
get
more
ideas
right.
B
And
secondly,
once
you
have
an
idea,
you
want
to
translate
it
into
query.
That
also
needs
to
be
fast,
so
that
goes
into
the
dsl
and
the
conciseness
of
your
queries
and
finally,
with
spark
you
get
this
really
nice
query
speed.
You
get
the
results
quite
a
lot
so
that
you
don't
get
your
train
of
thought
doesn't
get
interrupted,
just
to
elaborate
more
on
that
right
like
so.
B
The
first
thing
is
a
fast
and
accessible
repo,
so
very
often
you'll
be
thinking
about,
like
the
data
set
right
that
you're
working
on,
and
then
you
just
have
that.
One
question
that
you
you
want
to
answer
and
python
is
like
amazing
for
this,
because
it
starts
up
very
well
starts
very
quickly.
Then
you
import
pandas.
B
Do
your
query
and
you're
done.
I
mean
you
get
away
with
this
if
your
data
is
small
enough
and
your
query
is
instantaneous
right,
so
python
is
really
good
and
I
want
to
be
able
to
do
this
as
well
with
with
with
closure
and
guinea
right.
So
if
I
need
to
do
line
new,
that's
that's
too
long
like
that's
too
much
time
if
I
need
to
do
a
require
that
also
takes
in
some
of
the
time
right,
and
I
need
to
pick
my
dependencies.
B
Then
seven
seconds
is
not
bad,
but
apart
from
that,
I
think
you
still
probably
want
to
go
for
r
in
python,
and
the
second
thing
is
translating
that
idea
into
a
query
right
and
so
like
being
able
to
write
queries
very
fast.
So
there's
nothing
stopping
you
from
writing
pure
interrupt
right.
So
this
is
what
you'd
write
if
you're
dealing
with
vanilla
spark
with
the
scala
interrupt
a
couple
of
things.
B
B
Closure
data
data
structures,
and
that's
it's
just
a
little
bit
nicer
and
it's
quicker
to
write
then
also
you
know
you
want
to
be
able
to
maybe
interact
with
closure
directly
right
so
that,
if
whatever
it
is
that
you're
doing
is
returning
you
a
scala
sequence,
a
spark
row
which
is
very
much
like
scala
case
classes.
Then
you
need
to
do
this.
Unpacking.
That's
not
very
nice
right.
So
when
you're
done
with
the
stuff
like
it
should
give,
you
closure
data
structures,
so
we
pay.
B
You
know
particular
particular
attention
on
that
and
finally,
the
the
other
arrow
and
the
loop
is
query,
speed,
right
and-
and
you
get
this
from
spark
for
free
there's,
a
lot
of
people
working
on
on
on
performance
spark
performance.
So
this
is
just
a
nice
group
by
aggregate
example
all
right
and
then
you
you
write
to
to
disc
24
million
rows
with
a
million
groups.
B
Then
guinea
is
is,
is
you
know
very
competitive
here
right
because
it's
just
spark.
I
think
in
the
last
lightning
talk
I
I
said
that
python
was
really
slow,
but
that's
just
because
I
was
writing
my
pandas,
probably
not
so
optimally.
You
know
it
can
be
competitive
as
well,
but
but
ganesh
is
up
there
right,
like
it's
fast.
B
Some
other
goals
right
I'd
like
to
say
that
you
know
gne
to
spark
as
what
closure
is
to
the
jvm
and
close
your
script
to
javascript
right
then,
like
it
really
embraces
the
host.
It's
it's
just
just
using
all
the
facilities
and
like
the
spirit,
is
really
captured
here
by
david
nolan's
quote
and
during
his
talk
parasitic
programming
languages.
Where
he's
talking
about
creating
mobile
apps
for
on
closure
script
right,
saying
react
native
as
an
art
problem.
B
B
So
that's
that's
the
idea
and
also
like
you
get
this
nice
feature
coverage
right
like
which
is
not
possible.
If
you're
not
you're,
writing
everything
from
scratch.
In
the
core
name
space
there,
like
400,
h,
functions
approximately
and
there's
a
lot
of
machine
learning
models
there
already
rdd
support
as
well.
B
So
this
is
what
you
know
being
parasitic,
allows
you
to
do
and
also
being
very
shameless
about
borrowing,
other
idiot,
idioms
right,
so
obviously
closure,
idioms
stuff,
like
remove
stuff,
like
I
don't
know,
ink
deck
right,
it's
not
available
on
spark,
but
you
know
that
we,
we
have
a
pretty
good
idea
as
closure
developers.
What
what
these
things
mean.
So
it
should
be
there
to
operate
on
on
columns
and
then
I
I
use
pandas.
B
I
used
to
use
pandas
a
lot
and
there's
a
lot
of
stuff.
That's
that's
pretty
neat
there
that
doesn't
exist
in
spark,
and
you
know
it's
just
shamelessly
it
and
then
it's
you
know
again.
This
goes
back
to
like
writing.
Fast
queries
right
like
it's
there
and
you
know
whatever
comes
in
your
mind,
like
you
wanna,
you
write
that
then,
and
there
easy
getting
started.
Experience
like
this
is
a
bit
of
a
pet
peeve
of
mine
with
with
closure.
B
Right
like
I
want
the
getting
started
experience
to
be
as
easy
as
possible,
so
starting
from
a
clean
slate.
This
is
very
much
taken
from
inspired
by
borg,
dudes,
babashka
and
clj
condo.
Where
you
know
you,
you
have
an
install
script,
you
make
it
an
executable,
you
move
it
to
your
path
and
you're
good
to
go.
That's
all
you
need
to
do
and
it
should
work.
The
other
dependency
is
java.
That's
it
nothing
else
right,
not
even
line.
B
And
also
beginner-friendly
documentations,
you
know
getting
started.
I
don't
want
people
to
get
lost,
but
you
know
I
haven't
had
much
feedback
on
this
it'll
be
great.
If
we
yeah
some
people
try
it
out
and
we
get
feedback
on
whether
or
not
the
like.
What's
the
what's
missing
in
the
docs
right
and
finally,
like
it's
a
bit
of
a
a
hobby
project
for
me
right,
so
it
should
be
fun
right
and
some
of
the
stuff
that
I
like
doing
is
100
test
coverage.
B
B
Probably
the
last
goal
and
just
very
quickly
some
future
plans
right,
like
there's
four
main
modules
and
spark
there's
spark
sql,
there's
spark
rdd,
there's
sparks
machine
learning
and
there's
spark
streaming,
so
I'm
working
on
spark
streaming
so
that
you
know
we
get
like
this
nice
coverage
of
the
entire
library
better
documentation.
I
think
django
has
an
exceptional.
You
know
a
documentation
style
kind
of.
B
I
think
you
know
probably
could
learn
a
lot
from
them,
so
I
think
that's
that's
next
and
integration
to
other
closure
data
libraries,
I'm
specifically
thinking
about
a
zero
copy
path
to
tech
ml
data
set.
I
think
we've
talked
about
that
a
lot,
but
also,
I
don't
know
note
space
and
oz
right.
I
think
spark
doesn't
come
with
a
nice
visualization
library,
it'd
be
nice
to
to
have
that
as
well.
B
More
borrowed
idioms
experience
and
pandas,
dusk
and
and
spark,
but
that's
that's
about
it
right
so
other
idioms
from
r
and
from
on
julia,
let's
say
would
be,
would
be
awesome
and
also
smoother
experience
when
you're
deploying
it
on
the
cluster.
You
know
because
you
know
that's
that's
what
spark
is
good
at
you
know
we
should
leverage
the
the
free
facilities
which
at
the
moment
is,
is
not
there.
So
those
are
just
some
some
future
plans
that
I
have
and
that's
it
for.
B
A
I
kind
of
would
like
to
take
a
step
back
first
and
just
go
into
the
history,
because
when
I'm
hearing
you
present,
I
I
hear
that
you're
you're
making
making
data
science
work
both
on
technical
and
a
business
sense.
But
I'm
wondering
how
did
you
get
there
and
and
why
did
you
end
up
with
with
the
combination
of
spark
and
closure?
Can
you.
B
B
B
No,
no,
no,
no
complaints,
then,
and
then
I
moved
to
a
an
algorithmic
trading
startup,
where
we
used
a
lot
of
python,
a
lot
of
python,
a
lot
of
pandas
as
well,
and
then
I
started
running
into
a
few
issues
like
the
performance
wise.
It
was,
I
wouldn't
say
bad,
but
it
was
hard
to
predict
and
you
had
to
do
a
lot
of
profiling
to
make
stuff
work
and
a
lot.
B
B
They
use
spark
scala
spa
a
lot,
and
you
know
it's
it's
it's
it's
honest
spark
was
honestly
awesome
like,
and
you
know
you
had
this
luxury
of
working
in
a
big
company
where
computational
resources
is
just
not
an
issue
like
you're
encouraged
to
think
that
storage
is
unlimited
right
and
you
just
work
away
with
that.
But
then
the
problem
I
had
with
that
is
that
the
the
main
way
that
you're
encouraged
to
work
with
that
is
through
a
notebook,
and
this
robbed
me
the
wrong
way
really.
D
B
And-
and
you
know,
notebooks
I
think
is
very
problematic
for
maybe
it's
okay
for
research,
but
then,
when
you
start
to
productionize
it
like
you
need
tasks
you
need,
like
all
the
other
stuff.
You
know
you
need
your
terminal,
I
feel
so.
What
I
did
when
I
was
there
was
that
you
know
this
is
the
notebook.
You
know,
I'm
just
gonna
connect
to
it
via
the
command
line,
interface
right,
but
then
there's
still
some
issues
there.
B
A
B
B
I
don't
know
and
and
then
like
I've
got
this
sort
of
hacky
way
of
connecting
to
via
the
the
command
line
interface-
and
I
don't
know
the
whole
experience
is
just
not
so
nice
and
again,
like
you
know,
I
was
working
on
on
a
on
a
pretty
big
code
base.
So
maybe
it
was
like
me
not
understanding
on
how
to
maneuver
some
some
things
right,
but
I
did
miss
the
fact
that
you
know
you
can
just
type
ipython
and
then
you're
good
to
go.
B
Yeah
yeah
yeah
right
yeah,
so
everybody
was
using
the
like
I'm
guessing
connecting
to
the
same
I
mean
there's
like
the
resource,
scheduler
right,
and
so
you
you
ask
for
for
resources
and
then
they
give
it
to
you,
but
then
getting
like
the
extra
dependencies
there.
It's
just
not.
I
don't
again
that
it
could
be
my
shortcomings
right.
B
So
I
just
get
like
a
very
bloated
jar
and
like
everything
that
I
could
potentially
want
and
then
just
put
it,
which
is
again
not
not
probably
not
so
good
for
when
you,
when
you
try
to
productionize
it
yeah
but
yeah.
So
I
miss
some
things
about
python.
I
really
love
some
things
about
scala
spark
and
then
I've
always
wanted
to
do
closure.
B
I've
been
wanting
to
learn
that
for
like
a
couple
of
years
before,
actually
you
know
jumping
into
the
language
and
actually
executing
a
couple
of
projects
using
it.
Then
I
thought
you
know
one
day
I
you
know
I've
got
to
execute
this
this
data
project
and
then
I'm
using
python,
pandas
and
dusk,
I'm
like
waiting
for
ages
and
then
I
thought
you
know
like
if
we
could
just
do
this
in
spark.
B
It'll
be
okay
but
then
my
other,
my
other
data
analysts
would
would
have
to
learn
scala
right
or
I
don't
know
for
some
reason.
I
didn't
really
consider
pi
smart.
Maybe
I
should
have
but
yeah.
So
I
thought
you
know,
I'm
gonna
wrap
some
things
and
enclosure,
and
then
we
already
know
how
to
do
closure,
because
we've
done
closure
before
and
then
hey.
B
B
A
B
and
then
my
job
in
agoda
was
based
in
bangkok
and
you
know
you
know
I
wanted
to
move
back
home
to
indonesia,
so
that
was
the
motivation,
so
I
thought
you
know
before
starting
you
know
having
a
baby
right
like
if
there's
an
ever
a
time
for
me
to
to
start
my
own
thing
and
and
maybe
potentially
fail.
Who
knows
that
was
the
time.
So
I
thought
you
know
what
I'm
going
to
quit
and
and
do
my
own
thing
and
yeah.
A
C
C
B
B
There
isn't
much
data
really,
but
then
we
work
with
one
of
the
biggest
retailers
in
indonesia
and
they
have
like
millions
and
millions
of
customers
right.
I
would,
I
still
wouldn't
call
it
like
big
data,
because
it's
not
that's
like
a
few
hundred
gigabytes
of
stuff.
It's
it's
fine,
like
it'll,
run
on
a
single
machine,
so
yeah.
So
that's
that's.
What
I'm
working
on
now.
B
C
That,
first
of
all,
I
mean
it's
a
really
really
nice
project.
By
the
way
I
mean
I've
been
in,
I've
been
doing
some
spark
for,
for
some
time,
mostly
mostly
scholar,
driven
things
and-
and
recently
I
think
most
of
the
code
is
driven
by
spark
sql,
rather
than
any
of
other
things,
because
you
know
that's
much
more
sql
has
more
reach
right.
I
mean
there
is
more
people
who
can
write
sql.
C
B
Oh
okay,
so
for
spark
sql
it's
it's
it's
easy
enough
to
call
the
scala
api
and
and
and
that,
but
then
for
the
other
stuff,
oh
for
and
for
machine
learning
as
well,
but
for
rdd
and
for
spark
streaming.
It
gets
quite
tough,
I
think,
to
to
work
with
this
to
work
with
the
scala
interrupt.
So
he
is
using
the
the
java
interrupt
for
spark
streaming.
A
B
B
B
C
Yeah
so
so
you're
using
data
set,
but
I'm
curious
about
the.
How
are
you
mapping
because
datasets
are
statically
typed
right,
you
know
that's
the
main
idea
behind
data
sets
compared
to
data
frames,
at
least
so.
How
is
all
that
the
the
conversion
to
closure
driven
things
like?
If
I,
if
you
get
a
row,
then
how
is
it
converted
into
closure
maps?
C
C
B
C
B
B
Of
local,
but
that's
that's
one
s
part
of
the
project
that
we
really
need
to
work
on,
because
yeah
at
the
moment,
a
lot
of
it
is
just
local
right.
Yeah
we
haven't
had
the
use
case
to
where
we
need
to
have
like
a
big
cluster
right
to
execute
some
of
the
stuff
that
we
need
to
execute.
But
honestly
spark
local
people
like
to
say
that
it's
not
good
for
like
small
to
medium
data.
It's
actually
pretty
good.
C
C
C
It's
it's
it's
much
better,
because
I
don't
need
to
copy
all
my
parquet
files,
which
are,
like
probably
you
know,
with
terabyte
onto
my
computer
and
then
and
then
do
the
work,
and
it's
not
even
a
load
in
in
some
environments
right,
because
you
have
to
work
on
the
cluster
and
get
a
key.
So
do
you
have
any
plans
in
that
direction
or
are
you
looking
for
or
do
you
think
the
design
itself
is?
Is
you
know
compatible
with
that
that
mode
for
for
genie.
B
C
C
C
B
C
It
means
it's,
it's
really,
you
know
hitting
the
right
sweet
spot,
I
would
say,
because
you
know
you're
taking
advantage
of,
as
you
said,
sparks
computational
engine
and
then
giving
a
closure
way
of
it.
So
otherwise
I
need
to
do
maps
and
then
you
know,
but
the
general
closure
data
processing.
I
think
this
this
makes
it
way
better.
C
B
C
C
Yeah
yeah,
I
mean
the
thing.
Is
it's
also
because
you
you?
You
know
the
functions.
You
know
the
you
know
you
know
the
framework,
so
I
don't
need
to
learn
another
new
framework
to
to
to
do
my
work,
which
is
just
the
interesting
part
and
then
with
the
same
code.
You
can
scale
it
up
to
200
machines.
You
know
without
translating
it
into
something
else.
Yeah.
B
Yeah
yeah,
okay,
I
mean
the
first
thing
is
that,
like
I,
don't
have
that
much
experience
and
closure
itself
right,
like
I've,
been
using
it
professionally
since
december
2019,
so
it's
it
hasn't
been
a
year.
So
learning
the
language
itself
is
as
something
then
yeah.
The
scala
interrupt
can
be,
can
really
be
tricky,
but
then
a
lot
of
it
has
been
solved
actually,
like
you,
take
bits
and
pieces
from
other
libraries.
C
B
C
C
B
B
How
do
you
deal
with
that
yeah
and
and
yeah?
I.
I
ran
into
a
lot
of
troubles
with
with
the
rdd,
because
functions
need
to
be
serializable
yeah,
which
is
which
is
tough.
I
like
basically
taking
a
lot
from
spark
plug,
which
is
imperative.
Yeah
I
mean
I
mean
I
think
I
know
that
they're
using
spark,
but
then
I'm
not
sure
they're
they're
pushing
everything
on
on
github,
but
then
whatever
is
there
is
actually
super
useful.
That's
that's
used
in
guinea
as
well,
so.
B
C
D
C
C
C
C
C
C
D
A
B
C
Yeah,
of
course,
yeah
yeah
yeah
I
mean
streaming
is
fairly
reasonably
new
right.
It
starts
with
spot
to
something
2.2,
something
structured
streaming
and
everything
so,
but
but
I'm
curious
how
that
maps
to
idiomatic
closure
as
well-
and
I
think
you're
making
me
explore
that
a
bit.
So
I'm
curious
about
the
streaming
side
of
it,
because
there
is
these
streams
and
types
there
and
all
that
stuff.
So
yeah.
B
C
B
C
B
C
C
But
it
is
it
is.
It
is
impressive,
especially
now
and
when
you
show
that
you
know
the
test
suit
and
also
the
ci
stuff,
that
you
are
making
sure
that
everything
is,
you
know
working
properly.
So
that's
that's,
that's
extremely
disciplined.
I
must
say
I
mean
I
I've
been
I'm
in
the
industry
for
almost
20
years
and
even
even
normal
commercial
prices.
Don't
have
this
kind
of
you.
D
B
C
B
B
C
B
Haskell
from
is
I've.
Actually
you
know
I
still
want
to
dabble
in
it,
but
then
I
I
feel
that,
like
so
I'm
like
I'm
a
huge
fan
of
functional
programming
right,
so
that's
that's
really
one
thing
that
I
really
really
like,
and
I
don't
think
in
indonesia
or
anywhere
starting
a
a
software
team
with
haskell
with
that
carries
a
lot
of
risk,
but
I
still
want
it
to
be
a
functional
programming
language,
so
yeah,
I'm.
B
C
B
B
C
Impressive
anyway,
I
think
that's
yeah,
I
mean
I
I'll,
certainly
give
it
a
try,
because
I
haven't
tried
closure,
driven
things
and
and
because,
as
I
said,
my
day-to-day
work
is
with
a
big
cluster,
with
lots
of
data
and
with
an
enterprise
curtain
code
enterprisey.
You
know
way
of
dealing
with
things
because
I
work
in
a
fintech
and
we
have
a
lot
of
regulations.
C
B
B
Okay,
so
because
that's
that's
what
we
used
because
then
because
you're
you're
sending
texts
either
python
code
or
scala
code
right,
so
so
that
wouldn't
work,
then
I
thought
you
know
you
have
something
that's
very
similar
to
that
which
is
n
rappel
right.
So
if
you
have
a
master,
that's
that's
running
at
the
cluster
and
then
you
can
connect
to
the
n
ripple
of
that
cluster
of
that
muster.
Then
you
should
be
good
to
go.
B
C
Nice,
that's
super
nice,
but
I
mean
this
is
really
fascinating
project
by
the
way.
So
congratulations
on
on
I
mean
putting
it
into
production
and
using
it
and
then
and
also
open
sourcing
it.
You
know,
that's
that's
that
you
know
giving
back
to
the
community
and
that's
that's
really
admirable,
especially
with
therefore
that
you're
putting
it
into
maintaining
it
and
documentation,
and
you
know
setting
up
proper
well
setting
up
an
example.
I
would
say
no
nice,
thank
you,
so
I
think
again,
I'm
speaking
too
much
daniel
dude
or
theodore.
B
Yeah
yeah,
so
I've
got
this
pet
peeve
right
of
data
scientists
not
being
able
to
be
as
being
not
fully
versed
in
software
engineering.
So
I
think
that's
that's
an
you
know.
You
encounter
data
scientists
so
without
software
engineering
background,
but
I'm
adamant
that
that's
not
something
that
I
want
to
have
in
the
company
right
so
that
if
you're
a
researcher
and
my
team,
you
have
to
be
able
to
do
some
back-end
programming
as
well,
and
you
dabble
with
the
software
team
as
well.
B
Sometimes
so
so
everyone
in
the
team
knows
closure.
So
that's
that's
not
really
an
issue,
but
then
we
we
haven't
had
more
data
projects.
If
I'm
honest
with
you,
we've
got
like
mathematical
modeling
projects,
so
yeah,
so
we
haven't
really
had
the
chance
to
do
that.
So,
but
then
the
next
one,
probably
yeah.
B
Yeah,
so
the
what
I'm
referring
to
like
mathematical
modeling
is
that
you
you
go
to
the
client
and
then
you
see
their
processes
and
then
you
make
your
assumptions
and
then
usually
it
boils
down
to
an
optimization,
a
constraint,
optimization
problem.
If
it's
linear,
then
great,
you
use
something
like
or
tools,
linear
programming
and
you're
done.
B
B
But
then
weeds
are
agricultural
products,
so
their
prices
fluctuate
their
protein,
wet
gluten
and
moisture
fluctuate
their
price
that
their
delivery
might
be
late
right,
so
they
have,
they
must
produce
flour
that
has
constant
quality
and
they
have
their
sales
forecast
with
ever
fluctuating
raw
material
right.
So
what
we
made
for
them
is
this
optimizer
that
which
is
just
a
constraint,
optimization
problem,
a
large-scale
one.
It
has
like
100
000
constraints
or
something
like
that.
A
B
B
No
yeah,
it's
it's
just
a
mathematical
modeling.
For
me,
it's
just
like
an
optimization
problem
right,
so
you
have
a
bunch
of
inputs
and
then
you
create
the
the
the
the
problem
and
then
you
solve
it
and
you
give
it
back
to
them.
There
is
no
machine
learning
involved,
there's
no
data
cleaning
involved
because
you
know
exactly
what
you're
going
to
be
given.
So
no
not
really.
A
C
Yeah
yeah,
I
was
wondering
because
you're
talking
about
the
data
science,
data
modeling
sort
of
things,
because
so
what's
your
gen?
What's
your
opinion
about
like
closure
for
the
data
science
stuff,
because
you
know
we're
talking
about?
Obviously
this
cycle
interview
and
then
you
know
we're
we're
interested
in
in
data
science,
work
and
closure,
because
this
is
more
or
less
python
being
the
the
lingua
franca
of
data
science
and
because
you
have
experience
in
python
as
well.
C
B
C
D
B
B
B
What
are
the
things
I
mean
a
lot
of
people
like
working
with
notebooks.
I
don't,
but
I
think
it's
very
important
to
to
gain
that
market
share
to
make
a
complete
data
ecosystem
and
as
as
far
as
I
understand
it,
I
don't
know
I
haven't
played
around
with
that.
You'll
probably
know
more
than
me
daniel,
but
it's
probably
not
there
right
like
there
isn't
a
reason
for
you
to
to
to
jump
ship.
Let's
say
if
you're
working
with
a
jupiter
notebook
to
to
closure.
B
D
D
C
C
It's
actually
older
than
clojure
itself,
so
you
know
it's
a
bit
of
a,
but
I
think
that
the
best
option
is
to
because
closure
has
been
like
the
you
know,
compatible
with
you
know,
host
on
something
you
know
utilize
the
stuff
already
there.
So
I
think
that
is
spirit.
If
I,
if
I
can
comment
on
on
the
project
that
that
you're
building
you
know
that
this
is
the
same
spirit,
right,
utilize
spark
and
then
provide.
You
know
fantastic
experience
using
closure
to
get
advantage
of
that
system.
So
I
I
know
probably
daniel.
C
You
know
more
about
this
as
well.
Right
I
mean
the
the
python
closure
interrupt
interrupt
work.
If
we
can
get
there,
then
suddenly
it
opens
up.
You
know
a
host
of
possibilities
because
that's
what
happened
in
enclosed
script,
ecosystem
right,
you
know
because
it's
on
javascript
and
then
suddenly
you
have
access
to
all
this
all
these
things
and
then
we
just
need
to
make
sure
that
the
the
tools
are
ground.
You
know.
A
C
In
in
a
good
way,
then
it
opens
up
like
a
lot
of
possibilities,
because
you
can't
possibly
say:
okay,
I'm
gonna
start
numpy,
you
know
clone
enclosure
and
I'm
pretty
sure.
After
after
building
two
metrics,
you
know
computation
functions,
then
I'm
like
dumb
done.
You
know,
I'm
not
gonna
build
all
the
you
know
so
that
that's
that's.
That
would
be
a
humongous
task,
so
I
think
the
taking
advantage
of
what
is
there
and
then
making
sure
that
the
interrupt
is
is
awesome.
That's
how
we
got
and
got
away
with
closure.
C
C
So
any
anything
that
you
can
pick
up.
So
I
would
say
I
think
that
would
be
a
nice.
You
know
approach
to
fill
the
gap,
otherwise
we're
such
a
small
community
and
well
compared
to
python
number
of
available
python
programmers
in
general.
It
will
be
a
monumental
task
to
say:
okay,
we're
going
to
redo,
psychic,
learn
or
numpy
or
on,
and
then
you
still
have
to
reach
out
to
the
java
x
system
anyway,
because
if
you
want
to
use
numpy,
then
you
need
a
proper.
C
C
B
C
C
D
About
numpy,
so
I
think
dragon's
work
on
neanderthal
is
a
huge
project
and
it
does
bring
really
high
performance
numerical
programming
with
arrays,
and
so
that
is
not
missing
anymore.
I
guess
all
the
linear,
algebra
parts
are
really
mature,
and
but
I
guess
I
guess,
when
you
go
into
some
specific
applications
like
time
series
analysis.
B
D
C
B
B
You
know,
use
it
very
nicely
and
enclosure,
because
I
mean
dragon's
doing
like
an
amazing
job,
but
there's
no
way
to
compete
with,
like
hundreds
of
developers
doing
numpy
they.
They
just
have
a
lot
more
surface
area
and
if
you
could
just
wrap
it
nicely
and
make
it
so
that
it's
it's
nice
to
interact
with
other
parts
of
closure
code,
and
I
think
that
that's
probably
the
way
to
go.
A
If
you
want
to
use
numpy,
you
have
to
deploy
your
thing
and
you
have
to
control
your
dependencies,
and
you
have
to
do
that
piece
of
work
that
also
it's
kind
of
a
solved
problem
in
the
closure
space
and
in
in
that
same
sense.
I
I
really
like
that
in
jenny,
you
just
you
just
make
this
little
piece
that
can
be
used
in
whatever
way
we
want
to,
and
you
also
made
the
cli
to
make
it
easy
to
get
started
with,
but
it
can
be
used
from
the
repel.
A
C
B
I
guess
yeah
for
me,
like
gunney,
is
kind
of
a
smallish
project
right
because,
like
I'm
not
doing
any
of
the
computations,
it's
just
wrapping
around
so
like
we
can
still
have
numpy
in
enclosure
right.
It's
just
and
it's
not
going
to
be
such
a
huge
project
because
that's
they're
just
wrapping
stuff.
I
I
okay.
I
don't
know
this
is
again
yeah
and
and
also,
but
also
like
as
soon
as
you
do
that,
then
what
numpy
are
you
using?
What
python
are
you
using?
C
But
that's
the
that's!
The
impedance
mismatch
that
you
have
when,
when
you're
bringing
in
you
know
interop
with
a
language
like
python,
which
is
a
bit
of
a
different
model
underlying
model
compared
to
java
stuff,
so
I
think
the
the
main
advantage
being
on
jvm
is
that
you
know
you
can
just
you
can
interrupt
its
color.
You
can
drop
the
jruby
whatever
you
know,
because
they're
all
underlying
tech
is
still
java
and
jvm.
So
it's
much
easier
there
I'm
curious
in
terms
of
the
yeah
and
then
how
lip
python
cld
is
working.
D
B
C
C
B
So
like
even
as
simple
as
like
going
through
the
the
cookbook
or
some
of
the
guides
and
saying
hey,
this
doesn't
work
right
and
yeah
at
the
moment,
like
none
of
the
functions
have
any
dock
strings
by
the
way.
I
really
want
to
get
a
way
of
to
just
import
all
of
the
scala
dot
strings
and
then
just
just
put
it
there
yeah.
B
That's
that's
in
the
works
and
then
yeah
some
some
help
on
like
deploying
stuff
in
the
cluster.
I
think
that
that's
that's
going
to
be
a
big
deal
and
yeah
yeah
feed
feedback.
All
around
I
think
would
be
would
be
awesome
because
at
the
moment
like
I
know
it
works
for
me.
I
don't
know
if
it
works
for
someone
else.
D
B
C
Of
the
of
light,
essentially
the
signal
fast
passing
through
and
at
some
point
there
was
a
router
called
misconfiguration,
so
that
is
the
one
that
is
bouncing
the
emails
back
like
it's
not
getting
a
reply.
So
from
people
point
of
view
it
was
500
miles
and
then,
from
tech
point
of
view
it
is
completely
different.
So
it
was
fascinating
story.
It
was
a
fun
fun
thing.
So
so
it's
something
like
that.
Okay,
you
know
all
the
tests
work,
but
as
long
as
you're
in
my
time
zone.
C
Nice,
okay,
but
it's
it
it's
super
cool
by
the
way,
so
I'll
certainly
give
it
a
try.
I'm
curious
about
the
cluster
side
of
it,
because
that
will
be
my
if
this
works,
then
that
will
be
a
main.
You
know
thing
for
me,
because
you
know
I
don't
run
any
spark
programs
on
locally
unless
you
know
just
for
experimentation.
C
C
A
B
C
C
A
A
B
I
think
that
I
don't
know
the
answer
to
that
at
least
not
not
100.
I've
got
some
ideas,
but
then
making
it
clear
for
people
where
to
go,
I
think
is,
is
very
important
and
also
that
that
that
bridge
we
can
have
a
zero
copy
path
to
tmd,
so
that
people
are
not
forced
to
pick
one
over
the
other
with
making
and
being
locked
in.
That
would
be
great
as
well,
but
then
yeah,
I'm
not
entirely
sure
where
to
to
start
with
that.
B
So
I
need
to
ask
chris
and
also
library
integrations.
I
think
integration
with
with
oz
would
be.
It
would
be
amazing
and
note
space.
So
I'm
thinking
you
know
in
my
head,
like
like
node
space,
being
a
drop
in
replacement
to
the
rebel
and
you'll
just
do
stuff
on
the
rebel
and
that
but
then
you'll
have
nice
image.
C
C
A
B
C
B
C
C
As
I
said,
it's
it's
a
fascinating
project.
I
really
admire
your
your
hard
work
behind
this
one.
I'm
curious
how
where
you're
going
to
take
it
so
I'll
certainly
try
give
that
a
try
and
if
I
have
hiccups
or
something
I'll
reach
out
to
you,
so
you
can
help
me
out
a
bit
and
I'm
curious
about
the
cluster
stuff.
So.
C
C
D
C
Yeah
yeah
I
mean
I
I
don't
want
to
you
know,
cross,
promote
something.
So
I
want
to
keep
things
a
bit
separate.
This
is
not
about
other
work.
I
do
so
it's
mostly
about
you,
know,
psychology
and
genie.
That
is
the
main
thing,
but
yeah,
I'm
I'm
really
happy
to
be
here
and
then
you
know
understand
things
a
bit
better
and
I
I
didn't
know
this
much
about
this
project.