►
Description
Get the slides: https://www.datacouncil.ai/talks/dagster-a-new-programming-model-for-data-processing
ABOUT THE TALK
This talk would introduce Dagster, an open source framework for building and modeling data processing computations. Data processing systems typically span multiple runtime, storage, tooling, and organizational boundaries. But all the stages in a data processing system share a fundamental property: They are directed, acyclic graphs (DAGs) of functional computations that consume and produce data assets. Dagster defines a standard for containerizing, describing and operating these computations, and that standard is opinionated and informed by the best practices in the industry leading to more testable, more reliable, better structured data systems.
By defining a standard one can build these computations in tools that users know and love such as Jupyter Notebooks (via Papermill), Dbt, Spark and leverage that standard in order to build high quality developer- and ops-facing tools to inspect, operate, and monitor those computations. These tools range from our beautiful introspection and execution tool Dagit, to tools that schedule these computations on systems ranging from Airflow to Lambda, among others. Dagster embraces the chaotic reality of the modern data management, and is an abstraction designed for incremental adoption within an increasingly heterogenous ecosystem. We would describe both the technology and the technical and organizational insights gained by production use of Dagster.
ABOUT THE SPEAKER
Nick Schrock is the founder and CEO of Elementl, a company aiming to reshape the data management ecosystem, and the creator of Dagster, a new programming model for data processing. Previously, Nick was a Principal Engineer and Director of Engineering at Facebook. In that time, Nick co-created GraphQL, and led its implementation and adoption across the entire organization and product line. He also formed the Product Infrastructure group, whose engineers, in addition to GraphQL, created React, React Native, and many other broadly-used developer technologies, both inside Facebook and the technology industry at large.
ABOUT DATA COUNCIL:
Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers. Make sure to subscribe to our channel for more videos, including DC_THURS, our series of live online interviews with leading data professionals from top open source projects and startups.
FOLLOW DATA COUNCIL:
Twitter: https://twitter.com/DataCouncilAI
LinkedIn: https://www.linkedin.com/company/datacouncil-ai
Facebook: https://www.facebook.com/datacouncilai
Eventbrite: https://www.eventbrite.com/o/data-council-30357384520
A
A
All
right,
good
morning,
everyone
thank
you
for
coming
I'd
like
to
thank
data,
Council
and
Pete,
for
allowing
me
to
speak
here.
So
my
name
is
Nick
Shrock
I'm,
the
founder
of
a
company
called
elemental,
but
what
I'm
gonna
be
talking
about
today
is
our
first
project
and
it's
an
open
source
project
called
daxter
and
there's
the
first
time
we've
been
able
to
talk
about
it
in
public.
So
it's
a
super
exciting
day
for
us.
A
So
just
a
little
about
me,
I
want
to
step
back
and
talk
about
why
I'm
here
so
I
spent
the
bulk
of
my
career
at
Facebook
and
I
should
worked
at
Facebook
working
on
frameworks
for
our
product
developers,
so
externally,
I'm
best
known
as
the
co-creator
of
graph
QL
I
won't
be
talking
about
graphical
today.
But
it's
kind
of
what
I'm
known
for,
and
you
know
it's
really
been
an
honor
to
watch
that
become
an
open-source
technology.
A
So
I
left
Facebook
in
February,
2017
and
I
started
going
around
talking
to
companies
about
what
their
problems
are
technically
and
I
was
talking
to
people
both
inside
the
valley
and
outside
the
valley.
This
kept
on
they
kept
on
saying
this
over
and
over
again,
our
data
is
totally
broken
and
I
found
that
very
confusing,
so
I'm,
like
I,
don't
even
know
what
that
means
like.
What
do
you
actually
mean
by
that?
When
you
dig
into
it,
you
start
hearing
phrases
like
we
don't
know
where
the
data
comes
from.
A
We
have
no
idea
what
it
means.
We
have
no
idea
how
to
reliably
process
it.
We
don't
know
how
to
test
it.
Our
engineers,
don't
even
wanna,
touch
this
stuff
and
it's
not
fun.
It's
not
sexy
and
my
confusion
got
deeper
because
I
had
been
told
by
reliable
sources
that
data
science
is
the
sexiest
job
of
the
21st
century.
So
then
the
next
step
is
to
go
talk
to
data
scientists,
and
so
what
did
dad?
What
did
your
data
scientists?
A
What
do
you
actually
do
day
to
day
and
the
answer
is
variably,
something
like
this
I
just
convert,
Jason
Python
dictionaries
of
back,
and
so
this
is
an
interesting
journey
to
explore
and
a
more
common
formulation
is
something
like
this.
You
hear
this
that
almost
every
talk
you
ever
go
to,
which
is
I,
spend
20%
of
my
time
doing
my
job
and
80%
of
my
time
doing
it
a
cleaning.
A
So
when
people
talk
about
data
cleaning,
they're
talking
about
all
sorts
of
stuff
right,
they're
talking
about
rolling
their
own
infrastructure
right
common
workflow
for
a
data
scientist
in
today's
world,
is
you
give
them
a
CSV
file
and
a
jupiter
notebook
and
you're
like
do
stuff?
That
is
crazy.
That's
actually
everyone's
used
to
it,
but
it
actually
is
insane
that
that's
what
we
do.
A
Then
you
go
and
talk
to
a
business
leader
right
all
right
so
date
is
broken.
Tell
me
what
that
means,
and
they
say
something
like
this.
Our
beta
projects
fail
all
time
all
right.
It
failure
is
the
norm
which
is
also
kind
of
crazy.
So
summarize
the
state
of
things
you
got
engineers
saying
I,
don't
touch
this
stuff,
you
got
data
scientists
and
they
tell
me
they're
wasting
all
their
time
and
then
the
leaders
saying
that
what
it
just
it
never
works.
This
is
where
we
are.
A
This
is
kind
of
the
state
of
things
the
people
in
this
room
are
kind
of
at
the
upper
upper
echelon
of
Technology.
Here
right,
a
lot
of
people
here,
a
bill
DeJohn
accustomed
infrastructure,
so
they
succeed
more
than
the
norm,
but
this
is
kind
of
the
state
of
everything,
and
so,
as
I
was
kind
of
taking
stock
of
this
landscape.
This
actually
reminded
me
something
you
have
to
bear
with
me
here.
It
actually
reminded
me
of
conversations
I
had
10
years
ago
with
people.
Writing
you
eyes.
A
So
if
you
talked
to
someone
in
2009,
they
would
likely
say
something
like
this.
I
spend
80
percent
of
my
time,
fighting
the
browser
and
20%
of
my
time
building
my
app
now,
if
you
were
around
in
2009
and
within
screaming
distance
of
a
UI
engineer,
you'd
start
hearing
pained
yelling
that
their
screens
about
ie6
right.
This
is
what
they
were
dealing
with,
but
it
was
interesting
well
skipping
here,
so
you
would
hear
things
like
this
all
right.
We
can't
change
our
uy.
We
can't
test
it
breaks
all
the
time
right.
A
All
right,
shooters,
don't
want
to
touch
it
and
when
they
do
all
they
do
is
a
hop
into
the
codebase.
Make
something
fly
around
the
screen
then
get
the
hell
out
and
that
breaks
a
week
later
right.
That
was
the
state
of
things
fast
forward
to
today.
Front-End
engineering
is
incredible:
now
you
might,
you
might
get
a
different
impression
from
some
of
the
chatter
on
Twitter,
but
in
the
reality
of
things
it
is
a
very
healthy
ecosystem
like
working
in
a
full.
A
Modern
JavaScript
stack
is
a
magical
experience
and
yes,
the
browser's
did
get
better,
but
I
would
argue
that
it
wasn't
decisive.
It
was
the
software
abstractions
that
were
built
on
top
of
the
browsers
that
truly
transformed
the
trajectory
and
totally
changed
its
trajectory
of
this
ecosystem.
So
systems
like
angular
systems
like
View,
j/s
and
then
systems
like
react
right,
react
react
was
kind
of
V
in
my
I'm,
probably
biased,
but
it
was
V,
I
think
decisive,
abstraction
that
changed
the
course.
A
This
whole
ecosystem,
both
angular
and
view
now
essentially
defined
themselves
in
opposition
or
in
response
to
react,
and
so
what
was
what
what
made
react?
So
good
and
one
thing
about
it-
is
that
react
acknowledged
the
complexity
of
the
problem.
They
were
solving
you
weren't,
just
scripting
a
webpage.
You
were
building
a
full
application
with
all
the
complexity
they're
in
and
react
provided,
abstractions
and
formalisms
that
matched
that
complexity
and
then
beyond
technology
react
respected.
A
The
discipline
so
front
end
before
react
in
the
late
2000s
was
kind
of
this
software
engineering,
backwater
and
there's
often
kind
of
a
lack
of
respect
and
almost
kind
of
a
self
deprecation
in
that
field,
and
that
is
no
longer
true
anymore.
The
systems
that
are
built
in
front
end
or
real
lots
of
engineers
like
to
engage
in
that
ecosystem.
It
is
truly
a
world
transformed,
so
reacted
us
into
this
world.
A
Of
building
these
fully
fledged
well
abstract
to
front
up
applications
and
I
would
argue
that
we're
on
the
cusp
of
a
new
era
in
data
right,
we're
building
we're
not
just
building
a
bunch
of
disconnected
scripts
that
are
stitched
together
with
a
workflow
engine
right.
These
are
full
applications.
They're
they're
alive,
they're,
extraordinarily
complex,
built
with
a
huge
heterogeneous
array
of
tools
built
by
a
bunch
of
different
personas.
A
They
are
incredibly
complicated
and
I
would
argue
that
they're
they're,
the
abstractions
and
systems
that
are
operating
the
ecosystems
today,
don't
live
up
to
the
underlying
complexity
that
we're
actually
dealing
with,
and
so
what
we're?
What
Dexter
is
trying
to
do
is
being
part
of
this
journey
of
kind
of
moving
this
domain
forward
from
not
just
a
bunch
of
disconnected
pipelines,
but
instead
to
be
building
full
data
applications.
A
So
some
core
principles
here
so
at
Facebook
I
started
a
team
called
product
infrastructure
which
ended
up.
Creating
react.
I
didn't
work
on
react,
but
I
was
you
know
within
shouting
distance
of
them
and
I
worked
on
graph
QL,
but
we
had
kind
of
these
timeless
principles
that
I
think
really
apply
to
what
we're
building
here
in
the
data
domain.
So
one
is
that
this
kind
of
goes
without
saying
is
that
you're
solving
a
real
problem?
A
I'm
gonna,
get
it
we're
gonna
talk
about
the
rest
of
the
time,
see
if
I
can
prove
to
you
that
we
are
solving
a
real
problem
here,
but
in
terms
of
process,
there's
just
a
few
things
I
think
are
utterly
critical.
One
is
an
incremental
adoption
path,
so,
very
typically
in
the
data
domain.
People
want
to
provide
these
fully
vertically
integrated
stacks
that
replace
a
whole
bunch
of
things.
At
the
same
time,
it's
not
a
realistic
approach
may
be
able
to
pick
off
things
one
system,
one
task
at
a
time.
A
You
want
to
preserve
the
tools
that
work
you
have
made
huge
investments,
notebooks
spark
data,
warehouses,
air
flow,
etc,
etc.
You're
not
gonna,
be
asked
to
abandon
those
overnight
and
then
immediate
value
value
in
productivity
gates.
So,
yes,
there's
a
long
term
vision.
We're
like
there's
gonna,
be
some
glorious
end
state
with
data
applications
that
work
perfectly,
but
in
the
short
term,
you
have
to
install
this
thing,
get
immediate
value
all
right,
so
these
are
kind
of
core
principles.
So
I've
been
talking
for
a
long
time.
You're
probably
actually
wondering
what
what
are
you?
A
A
I
found
that
very
confusing,
because
if
you
look
at
a
ml
engineer
and
the
ETL
engineer,
they
are
doing
the
same
thing
just
at
the
end
of
a
supervised
learning
process,
there's
a
model
and
a
ETL
there's
like
a
data,
Harris
table
or
file,
but
regardless
of
what
they
doing
they're
doing
this,
you
are
building
a
graph
of
functions
that
ingest
and
produce
data
assets.
That
is
the
core
problem
that's
trying
to
solve.
So
that
is
the
core
abstraction
on
which
daxter
is
built.
A
So
without
further
ado,
let's
pip
install
this
thing
and
see
what
the
hell
I'm
talking
about.
So
you
open
up
a
code
editor
and
we're
going
to
do
hello
world
here,
so
we
import
a
pipeline
and
a
solid.
So
a
pipeline
is
just
a
dag
of
what
we
call
solids
and
the
solid
you
can
approximate
is
just
a
unit
of
functional
computation.
Fancy
word
for
function
in
this
case.
A
We're
just
gonna
log
hello
world
into
this
context,
object
that
we
flow
throughout
the
computation
and
once
you
kind
of
structure
your
code
like
this,
you
immediately
get
to
lean
out
of
the
box.
So
we're
gonna
launch
this
tool
that
we
called
agate
all
right.
So
we
open
up.
We
have
this
web
tool
boom.
You
can
view
a
pipeline
right
here
is
a
beautiful,
UI,
really
nice.
The
team
has
done
amazing
work
here,
and
so
you
can
start
your
execution
here.
A
We'll
go
through
all
this
slower
later,
but
in
the
end
there
it
is
classic
computer
science
blinking
light
demo.
Okay,
so
Daggett
serves.
Is
this
kind
of
like
development,
environment,
I?
Think
one
of
the
things
we
can
do
in
order
to
provide
a
ton
of
value
is
dramatically
improve.
The
local
development
experience
right
right
now.
The
norm
is
to
develop
in
production
kind
of
sane
and
so
and
as
we
go
through
this
we're
gonna
kind
of
hop
back
and
talk
about
what
we've
introduced
here.
A
So
we've
introduced
the
notion
of
a
pipeline
which
is
a
directed
acyclic
graph
of
solids
and
then
a
solid,
which
is
a
unit
of
functional
computation.
All
right,
well,
hello
world
is
cute
blurs
actually
adapt
a
real
example
with
real
living
breathing
code
into
the
system,
so
we're
gonna.
Take
the
public
example
of
running
running
PageRank
on
pi
spark,
then
adapted
into
our
system.
So
here's
this
existing
code,
it's
a
command-line
utility
takes
a
file
path
and
a
number
of
iterations.
A
It
calls
a
function
with
those
arguments,
but
now
we're
going
to
do
some
stuff
up
here,
so
we
initialize
the
spark
context
we
load
in
a
file.
We
do
some
compute
on
it.
We
actually
do
the
PageRank
algorithm
and
then
we
go
through
and
we
print
out
the
results
of
this
right,
pretty
straightforward,
pice
bar
code.
Have
you
ever
dealt
with
it
now?
We're
gonna
do
is
do
the
best
code
or
use
strategy
of
all
time,
which
is
copy
and
paste
it
in
here
and
then
we're
gonna
make
this
a
daxter
pipeline.
A
A
We
annotate
it
and
we
make
it
a
so-called
solid.
Now
we
mark
with
metadata
and
inputs,
so
we
have
an
input
which
is
a
file
path.
Ok
and
then
we
also
have
a
notion
of
configuration.
So
input
is
equivalent
to
data,
whereas
configuration
more
configures.
How
something
is
comp
computed.
Very
concretely,
inputs
can
come
from
previous
computations
than
the
tag
you'll
see
why
we
did
that
in
a
second.
A
So
we
add
our
context
object
and
instead
of
getting
interations,
that
argument
we're
going
to
get
its
configuration
and
then
we
hop
down
here
instead
of
printing
it
out
we're
gonna
log
it
to
that
context,
object
that
we
provide
and
that's
it
we've
adapted
PageRank
to
our
system
pop
in
here
load.
It
up,
you'll
notice
that
all
the
meta
information
that
we've
encoded
in
the
program
is
surfaced
here.
So,
there's
a
path
right
here,
there's
a
configuration
down
here
and
as
a
result
of
providing
those
things
boom.
A
Now
we
have
this
auto
completing
type-ahead
and
you'll
see
as
we
type
ahead.
We
get
live
errors
about.
What's
going
on
right,
no
more
searching
through
random
Jason
config
files
that
God
knows
what
they
do
when
they
break
right.
You
have
a
very
well-structured
config
system
that
automatically
schematize
is
high
quality
errors,
etc.
Right
you
log
this.
We
can
filter
down
to
debug
yeah
and
we're
printing
out
stuff
to
the
screen.
A
Ok,
this
looking
pretty
cool.
It's
going.
The
next
thing
so
right,
I'm
getting
too
far
ahead
of
myself
here,
so
solids
have
inputs.
They
also
have
outputs,
it's
probably
a
little
bit
of
a
mission
on
this
screen
and
they
also
have
configuration.
Ok,
what's
next
well,
what
we're
gonna
do
you're,
not
supposed
to
read
this
but
effectively.
What
we're
going
to
do
is
take
this
one
function
and
break
it
into
two
functions:
okay,
so
we're
going
to
separate
it
into
loading
the
data
and
execute
it.
A
So
we
have
these
two
functions
to
neuters
here
and
what
we
have
to
do
now
to
make
this
a
career
and
pipeline
is
connect
them
together
with
a
dependency
okay.
So
what
we're
gonna
do
here
is
add
a
thing
to
our
pipeline
definition.
There
we
go.
You
notice
its
dependencies
dictionary
down
here
now,
if
you're
familiar
with
the
system
like
air
flow
air
flow
just
has
a
dependency
graph
between
tasks
or,
as
this
is
a
dependencies
between
the
way
the
data
flows
into
the
system.
A
So
if
you
notice
here
our
dependency
graph
says
like
hey
I,
have
this
input
it's
called
links.
I
want
you
to
get
it
from
the
output
of
that
previous
computation
right
and
stitch
that
together,
and
so
that
actually
gives
us
a
lot
of
power.
If
we
go
back
here
now,
we
zoom
in
to
this,
and
we
see
that
the
inputs
and
outputs
are
connected
right,
go
here
and
execute
it.
Our
config
files
actually
divide
it.
A
Sorry,
those
too
fast
and
now
we're
gonna
show
that
so
we
can
actually
eat
this
thing
when
we
complete
that
execution,
but
then
we
can
actually
go
back
and
select
a
subset
of
the
deck.
So
we
have
this
solid
sub
selection
thing
we're
gonna
be
like
hey
I
only
want
execute
part
of
this
graph.
Now
the
configuration
schema
is
dynamic,
based
on
what
you
selected
and
now
we
can
execute,
and
we
just
excuse
one
piece
of
this
thing
all
right.
A
So
now
we
have
dependencies
we're
able
to
stitch
these
solids
together
into
a
dependency
graph
of
compute.
So
what's
next,
so
you
notice
in
the
code
that
we
nude
up
a
spark
session
directly
and
what
we
think
is
a
really
really
important
concept
is
to
separate
your
business
logic
from
your
environmental
concerns,
and
that
means
what
we
want
to
do
is
have
the
system
manage
a
spark
session
for
you,
okay,
it
makes
it
code
more
testable.
It
also
has
some
other
nice
benefits
as
well.
A
Okay,
so
there's
also
ecosystem
of
libraries
so
from
the
daxter
PI
spark
library
which
we
also
released
today
you
can
import
a
spark
session
resource
and
the
details
here,
don't
matter,
but
the
key
thing
is
that
we're
configuring
it
so
that
hey
there's
a
resource
and
it's
called
spark
and
it's
a
spark
session
resource
all
right
now,
how
many
of
you
have
written
a
spark
job
interact
with
spark
at
all
all
right.
How
many
of
you
have
struggled
with
configuring
spark
right?
That's
a
hundred
percent
hit
rate
well
that
this
combined
with
the
config
editor.
A
So
we
can
scope
this
out
right.
We've
added
some
type
information.
Now
there's
this
config
schema
down
here
right
and
so
what
we've
done
is
we've
imported.
The
entire
spark
configuration
schema
right,
we're
gonna.
We
have
to
click
down
to
this
thing.
Boom
boom
boom
boom
boom
and
then
awesome
bam
right.
The
entire
spark
config
schema
with
imbedded
documentation
all
right
there
right,
you
can
mouse
over
it
stuff
happens.
You
all
see,
there's
an
error
at
the
same
time.
Oh
that's
an
you
know
it's
a
number.
A
We
expected
a
string
and
on
and
on
and
on
right.
This
is
a
highly
generic
configurating
system,
of
which
spark
is
one
example
target
right.
It's
a
very
pluggable
system.
You
can
add
your
own
config
schema.
Now
we're
executing
this
thing.
Look
there's
fubar
right
there
we
configured
it
properly,
okay,
so
now
the
other
thing
this
enables
is
the
ability
to
re,
execute
specific
parts
of
our
pipeline.
We're
actually
going
to
configure
this
to
automatically
persist
the
intermediate
materializations
of
this
pipeline
and
we're
gonna
inject
an
error.
A
So
what
we're
simulating
here
is
like
you're
only
a
pipeline.
It's
been
running
for
a
while,
then
the
last
task
fails,
you're,
like,
oh
god,
now
after
we've
run
this
entire
thing,
it's
a
nightmare,
so
we
just
reran
this
thing.
It's
gonna
pop
up,
an
error
we
have
this
nice
render
in
so
you
can
go,
get
your
stack
trace,
go
to
what
line
of
code
you're
dealing
with
now.
Look
at
this.
We
can
just
we
execute
this
single
thing
right.
A
A
Executing
we
have
it's
green
again
right,
super-powerful,
tooling,
so
we've
added
these
concepts,
and
this
is
not
the
meant
to
be
kind
of
there's
a
lot
of
stuff
to
absorb.
So
you
know-
and
this
is
the
goal
of
this-
is
not
for
you
to
be
able
to
look
at
this
thing
and
immediately
go
and
start
developing
on
it,
but
we're
giving
you
a
sense
of
how
these
abstractions
interleave
in
order
to
provide
real
value
to
our
users.
A
A
We
have
a
Python
implementation
of
it,
a
Python
library
that
you
pip,
install
and
we've
demonstrated
how
we
have
integrations
with
price
mark
underneath
it
by
adopting
the
abstraction
you
get
these
beautiful,
high
quality
tools
right,
it's
kind
of
a
suite
of
tools
called
daggit,
but
they're,
pretty
independent.
We
have
the
catheter,
the
dag
viewer,
our
execution
console
the
way
we've
written
this.
If
we've
written
it
over
an
API
and
the
API
is
really
what
this
system
has
is
in
its
core.
A
So
what
does
this
API
do?
Well,
as
they
described
before,
we
describe
a
data.
Application
is
a
graph
of
functional
compute,
and
what
does
that
allow
you
to
do?
We
can
query
those
graph
of
functions.
We
can
introspect
on
them
figure
out
what
the
shape
of
it
is.
We
can
actually
over
this
API
operate
them,
meaning
that
we
can
both
configure
and
execute
those
computations
and
then
also
over
that
API.
We
can
monitor
them
right.
So
we
we
have.
This
live
subscription
service.
A
This
stuff
happens
to
be
implemented
on
graph
QL
for
obvious
reasons,
but
it's
not
it's
not.
You
know
directly
linked
to
the
system.
It's
actually
it's
an
incidental
fact
about
it,
but
it's
these
core
capabilities
that
are
the
most
important
thing
and
so
Dex
there's
not
just
a
set
of
tools,
because
it's
an
open,
API
using
an
open
standard.
If
you
adopt
this
thing,
you
can
also
build
your
own
internal
tools
on
the
sub
traction,
so
this
ends
up
being
a
platform
for
tool.
A
Building
go
back
here
and
now
all
right
spark
developers
here
how
many
people
use
PI
spark
primarily
and
how
many
people
use
Scala
spark
javis
burg.
Okay,
so
this
is
for
you
is
that
okay,
pi
sparks
cute
and
I'm
a
real
spark
developer.
Nice
I
use
Scala.
This
is
this
pythons
joke?
Well,
just
so
happens.
We
can
also
script
Python.
You
know
spark
computations
written
in
Scala,
and
then
we
can
also,
you
know
by
the
word.
Functional
compute
is
chosen
very
carefully
mean
like
a
coarse-grained
function,
ie
a
spark
job,
a
database.
A
You
know
a
sequel
query
that
say,
creates
a
table
or
some
of
the
artifact.
So
let's
demonstrate
that
so
we
have
out
of
the
box
these
spark
solids
and
snowflake
solids,
just
as
examples
they're,
not
kind
of
ready
to
go
industrial-strength
it's
more
for
the
demo,
but
it
shows
the
promise
of
the
system.
So
now
we
have
a
pipeline
that
is
more
realistic
and
ingest
events,
so
we
download
a
file
from
s3
it
unzips
that
file.
A
You
know,
look
at
all
this
embedded
documentation
and
then
what
it
does
is
actually
ingest
that
event
produces
park'
files
using
spark,
and
we
have
these
things
called
asset
tags
where
we
can
actually
embed
meta
information
be
like
hey.
This
is
the
this.
Is
the
java
class
actually
does
this?
We
also
have
a
auto
complete,
auto,
completing
a
syntax,
highlight
sequel,
editor
right,
so
you
can
kind
of
get
full
context
in
your
pipeline.
A
It
is
available
right
now.
We've
actually
been
developing
out
this
out
in
the
open
the
whole
time
it
has.
These
multilingual
integrations
and
this
out-of-the-box
beautiful
tooling,
but
just
to
go
back
to
what
we
were
talking
about
the
beginning.
What
about
those
data
scientists?
You
really
address
that
yet,
and
you
know
what
is
the
problem
happening
in
data
science
they're,
what
you
know
what
problem
are,
we
think
is
most
important.
Well
essentially
it's
this
is
that
the
current
status
quo
in
most
organizations
is
that
data
science
and
engineering
there's
two
sealed
siloed
world.
A
The
data
scientists
do
some
work
over
here.
They
present
their
model,
throw
it
over
the
proverbial
wall
up
to
the
engineers
to
implement
productionize
it.
The
engineers
don't
understand
the
model,
the
data
science
known
as
the
data
science.
Folks,
don't
understand
the
production
foundation
issues.
It's
a
pretty
pathological
situation,
actually
only
just
a
bunch
of
negative
organizational
dynamics
where
you
want
to
get
to
is
here
where
they,
the
disciplines,
are
respected
and
there's
aspects
of
them
that
overlap,
but
they
also
do
their
own
individual
activities.
A
That
overlap
is
cultural,
but
it's
driven
by
underlying
changes
in
tooling.
So
I,
don't
think
you
talked
about
the
chasm
between
data
science
and
engineering
without
talking
about
Jupiter
right
Jupiter
is
the
de
facto
environment
in
which
data
scientists
do
their
work
and
there's
two
schools
of
thought
about
Jupiter.
There's
this
one
I,
don't
like
them.
It's
actually
a
great
talk.
It's
very
entertaining
I
used
to
be
in
this
camp
and
it
for
the
predictable
reasons
that
a
grumpy
engineer,
it's
like.
Oh,
it's,
not,
reproducible,
I'm,
not
storing
things
in
files.
I
can't
test
this.
A
It's
just
insane
out
of
order
execution
model,
that's
hyper!
Stateful,
like
you,
people
are
all
your
minds,
but
then
you
see
the
way
people
use
them.
It's
clearly
valuable,
so
I
was
trying
to
grapple
with
this
and
then
actually
Matt
who's
over
there
and
his
team
wrote
this
blog
post,
which
I
thought
was
extraordinary,
and
if
you
saw
his
talk
yesterday,
he
gave
a
great
summary
of
the
technology
and
it's
called
paper
mill
simply
put.
It
allows
you
to
execute
a
notebook
as
a
function,
and
then
I
saw
this
and
I'm
like
well.
A
Golly
gee
I'm,
you
know
working
on
a
framework
where
the
fundamental
unit
thing
I'm
building
is
a
coarse-grained
function
of
functional
computation.
So,
like,
let's
make
one
of
these
paper,
notebooks
paper
mill,
notebooks
a
solid,
so
we're
gonna.
We're
gonna
extend
the
example
pipeline.
We
had
that
ended
with
the
sequel
statement.
We're
actually
gonna,
do
a
computation
to
take
that
table
and
make
it
a
data
frame
and
then
we're
gonna
pass
it
to
a
notebook
all
right.
So
this
is
a
notebook
solid
and
you
can
write
in
line
and
daggit
see.
A
What's
going
on,
right,
see
this
computation
that
your
data
science
persons
crafted.
But
how
do
you
actually
integrate
that
in
the
pipeline?
What
does
that
mean-
and
this
is
where
a
paper
mill
kick
kicks
in?
So
if
you
check
out
this
notebook,
looks
like
a
normal
notebook.
You
can
do
all
your
insane
out
of
order
execution
to
your
heart's
content
all
right,
so
we
just
injected
an
error.
We
fix
it,
we
recompute
it.
A
You
have
to
keep
track
of
the
order
of
operations
here,
but
this
is
what
the
paper
magic,
though
there's
a
parameter
self
and
we
can
essentially
say
it's
like
hey
at
runtime.
What
we
want
to
do
is
instead
of
executing
this
notebook,
we're
gonna,
remove
that
cell
and
inject
a
new
cell
with
the
production
data.
A
That's
how
that
works.
So
we're
gonna
go
ahead
and
do
that
so
use
the
same
tools
and
I
was
gonna,
execute
the
thing
blah
blah
blah
blah
loading.
It's
a
snowflake
just
did
a
spark
job
of
some
computation
to
produce
a
data
frame
and
now
we're
executing
a
notebook,
and
then
you'll
see
this
thing.
It
produced
an
output
notebook.
So
what
the
output
notebook
is?
A
Is
it
copy/paste
job
of
the
original
notebook,
but
with
the
production
data
in
it,
and
we
just
clicked
on
it
copied
the
location
where
that
output
note
was
generated,
output
notebook
was
generated,
and
now
we
can
view
this
and
you
in
your
tool
of
choice
right.
So
here
we're
gonna
open
it
up
in
a
notebook,
and
so
this
is
actually
an
output
of
the
pipeline
and
it
serves
as
a
rich
log
of
computation.
A
So
all
of
the
charting
and
graphing
that
your
data
scientists
have
done
can
actually
be
used
in
production
logging
and
it
also
serves
as
the
debugging
environment.
So
whether
or
not
Dexter
succeeds
or
not,
I
think
that
paper
mill,
if
it
you
know,
grows
like
I
think
it
should
should
revolutionize
the
way
that
data
scientists
do
their
work.
I
think
it's
an
extraordinary
fans
and
we
want
to
integrate
with
those
technologies.
A
So
we
add
another
family
of
tech
underneath
within
the
Dexter
family.
So
let's
dig
into
what's
going
on
here.
So
what's
interesting
about
this,
and
if
you're
at
Julian's
talk
just
before
me,
he
set
this
up
perfectly
where
actually,
we
have
many
many
different
personas
interacting
with
our
pipelines,
that
we
have
data
engineers,
analysts,
data
science,
base
scientists,
etc,
collaborating
to
build
a
single
graph,
functional
compute,
a
data
application.
A
Now
you
might
be
thinking
yeah.
This
looks
cute
a
lot
of
fancy-pants
hipstery
web
tools,
I'm
interested
in
using
this,
but
I
have
my
existing
stuff.
I
have
an
airflow
cluster,
how
the
hell
am
I
going
to
use
this
new
thing
with
that.
Well
turns
out
that
what
we
can
do,
since
its
kind
of
structured
on
top
of
an
API,
is
that
we
can
query
that
graph
of
computations
and
then
dynamically
synthesize
an
airflow
dag
right,
so
you
can
encode
your
DAGs
daxter
space
and
then
effectively
view
airflow
is
sort
of
a
compilation
target.
A
A
A
Along
with
all
the
examples
which
everyone
knows
come
with
come
with
the
system
boom
there,
it
is
there's
our
deck
native
in
airflow,
so
we've
demonstrated
how
we
can
go
to
this
kind
of
from
this
compelling
local
development
environment.
Integrating
all
these
different
tools
then
very
easily
completing
the
story
by
making
it
very
straightforward
to
deploy
this
to
air
flow.
A
So
let's
sum
this
back
up
here,
so
EDA
engineering,
you
know
last
year,
this
conference
is
called
data.
Ang
comp-
and
you
know
we-
it's
been
rebranded
to
data
Council
to
include
data
scientists,
but
I
think
also
part
of
the
reason
it
was
rebranded
is
that
eight
engineering
is
still
an
emerging
discipline.
I
would
say-
and
you
kind
of
see
this
because
there's
title
arbitrage
happening
where
people
are
data
engineers
but
they're
like
no.
A
What
we
like
to
think
is
an
elegant
programming
model,
and
then
you
can
leverage
those
that
programming
model
to
use
new,
beautiful
two-lane
that
we
think
really
adds
a
lot,
especially
the
local
developer
experience.
And
then
it's
a
flexible
and
incremental
approach
right.
We
demonstrated
how
it's
integrating
with
all
the
tooling
that
you
depend
on
both
in
terms
of
how
you're
modeling
your
compute,
whether
it's
sparking
data
warehouse
or
a
notebook,
and
also
being
able
to
deploy
to
your
existing
infrastructure
in
a
permissionless
way
right.
You
could
actually
synthesize
an
airflow
dag
using
the
system.
A
You
don't
even
tell
your
ops
team
right.
It's
kind
of
this
bottoms
up.
Adoption
approach
and
I
just
want
to
dig
into
the
flexible
incremental
bit
for
a
second.
What
does
that
mean?
It
means
you
can
use
your
tools
means
you
can
preserve
your
existing
investments
in
your
code
and
it
means
you
can
deploy
your
infrastructure
and
it
means
you
can
adopt
instrumentally
all
right.
A
These
are
critical,
critical
qualities
for
any
successful
technology,
but
there's,
obviously
a
ton
of
work
to
do
you
know
where
we're
at
right
now
is
that
you
know
we
have
one
alpha
user
who's
in
the
audience
today,
the
ape
gong.
You
saw
him
at
the
panel
and
his
company
superconductive,
but
there's
obviously
a
ton
of
work
to
do.
We
have
a
small
team
super
talented
team
they're
over
there
max
Nate
and
Alex
and
they've
just
done.
You
know
extraordinary
work
on
this
project.
A
B
B
B
A
But
I
mean
you
know,
our
kind
of
approach
here
is
not
to
be
overly
restrictive,
with
a
programming
model
being
like
there
are
no
side
effects
ever
that's
not
realistic.
That's
not
the
way
these
systems
work.
Our
request
is
that
the
system's
don't
lie,
meaning
that
you
can.
Yes,
you
can
declare
your
outputs,
but
we
also
have
this
ability
for
the
solids
to
kind
of
indicate
that
we
call
them
materializations.
We
could
be
like
heya
by
the
way
I
emitted.
A
This
thing
you
know,
that's
exactly
actually
how
that
output
notebook
works,
the
the
nope
the
solid
emitted,
the
fact
that,
like
hey
I,
produce
this
materialization,
that's
over
here
like
render
that
and
your
UI
at
a
minimum
but
later
is
a
built.
The
system
we'll
be
able
to
build
formalisms
to
capture
that,
but
we
can't
solve
idempotence
for
you.
You
know
like
if
you
email,
if
in
the
middle
you're,
you
know
it's
a
Python
program
or
whatever
program.
If
you
decide
to
email,
someone
I
mean
he
mail
went
ow,
we
can't
you
know.