►
Description
Noah Kantrowitz, a principal site reliability engineer at Geomagical Labs (part of the IKEA family), discusses how he used Dagster to power a customer-facing applications. In this talk, Noah talks about their data orchestration system, cost management via scale-to-zero, and the Dagster features he uses.
🎞 Slides 🎞
Geomagical Labs & Dagster (Noah Kantrowitz) ➡️ https://drive.google.com/file/d/1sFBrqX8qQEgOKGTKxCqMBwDbdPdPWkHd/view?usp=sharing
🌟 Socials 🌟
Follow us on Twitter ➡️ https://twitter.com/dagsterio
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Check out our Documentation ➡️ https://docs.dagster.io/
A
Our
use
case
isn't
quite
the
normal
one
for
dagster,
so
a
quick
overview
of
our
main
product.
A
user
takes
a
bunch
of
imagery
of
their
room
using
our
mobile
app.
They
upload
those
to
our
cloud
stuff.
We
run
a
bunch
of
fancy
math,
we
build
a
3d
view
of
their
room
and
then
they
can,
in
a
browser,
mobile
app,
drag
virtual
furniture
in
to
see
how
it
would
look
before
buying.
A
So
the
key
points
there
are
that
it's
directly
customer
facing,
so
we
need
it
to
run
quickly
and
there's
very
little
margin
for
error,
because
every
second
counts
when
a
customer's
waiting
on
us.
Also,
the
fancy
map
that
we
run
can
get
very,
very
fancy.
Some
of
the
steps
vary
from
little
python
scripts
to
large
c,
plus
plus
tools
that
use
tens
of
gigabytes
of
ram
and
require
a
gpu
before
we
dive
into
the
dax
setup.
Let's
look
what
we
had
before
for
our
main
product.
We
used
celery
with
rabbit
mq.
A
Each
solid
to
use
the
dr
term,
although
it
wasn't
that
before
runs
its
own
deployment
in
kubernetes,
and
we
have
a
custom
tool
to
compile
from
a
simple
json,
dag
representation
canvas
orchestration
system
that
comes
with
celery.
We
knew
we
wanted
to
keep
most
of
the
structure
of
our
existing
solids
in
place,
both
because
updating
them
would
be
really
costly
and,
honestly
that
part
of
the
system
was
going
just
fine
in
terms
of
workflow.
The
big
problem
was
that
our
data
completely
limited
and
canvas
didn't
really
offer
much
room
for
improvement.
A
We
tried
a
couple
times
and
it
didn't
work
for
inexplicable
reasons,
so
canvas
definitely
had
to
go.
Also.
Our
workloads
are
really
bursty,
so
anything
that
allows
us
to
shut
down
more
capacity
when
it's
not
in
use
helps
us
to
keep
costs
down,
which
is
really
nice
when
you're
allocating
giant
cloud
machines
and
also
if
we
could
get
better
run
tracking
with
more
detailed
information.
That
would
be
really
nice.
I
mean
I
can
write
all
that
instrumentation
myself,
but
if
I
don't
have
to
that's
great,
obviously
we
found
it
liked
daxter.
A
You
wouldn't
be
hearing
from
me
today,
but
in
short,
it
ticked
all
the
boxes
I
was
looking
for
and
anything
that
it
didn't.
We
could
help
to
improve.
So
the
biggest
thing
that
we're
looking
to
improve
on
is
the
thing
I'm
talking
about
today,
which
is
a
getting
a
scalable,
stable
execution
layer
in
place,
because
we
needed
really
high
potential
concurrency
and
much
higher
resiliency
to
failure
the
default
launcher
and
executor
combos
weren't
going
to
cut
it.
A
We
did
look
at
the
extra
salary
since
we
knew
we
wanted
celery
under
the
hood,
no
matter
what,
unfortunately,
the
workflow
for
that
more
or
less
requires
that
the
code
for
all
solids
lives
alongside
the
pipeline
or
at
least
that
all
solids
in
the
pipeline
can
be
runnable
together.
That
wouldn't
really
work
well
for
us
because
of
the
widely
varying
hardware
requirements
for
different
solids,
there's
also
daxter
kubernetes,
which
launches
kubernetes
jobs
on
the
fly.
A
This
can
be
okay
for
infrequently
executed
jobs,
where
failure
delay
doesn't
have
any
customer
facing
consequences,
but
I
really
really
suggest
that
you
keep
cube
api
server
out
of
the
hotpath
of
your
product.
It's
really
just
not
built
for
that
kind
of
thing.
That's
putting
on
my
kubernetes
hat
instead
of
daxter
hat
I'll,
save
some
back
and
forth
and
cut
to
what
we
came
up
with
so
the
normal
stuff.
Daggett
runs
is
a
kubernetes
deployment,
that's
pretty
standard
and
we
wrote
a
custom
launcher
plug-in
that
takes
the
run
request.
A
Serializes
it
into
json
puts
that
onto
a
celery
queue
and
then
each
daxter
workspace
has
a
celery
daemon
that
is
sort
of
the
launcher
daemon.
So
it
gets
those
requests
off
of
rabbit,
mq,
deserializes
it
and
acts
as
the
executor
and
then
we'll
talk
about
it.
A
little
bit
later.
Most
of
our
solids
are
also
running
as
separate
demons
like
they
were
before,
with
some
proxies
to
get
back
and
forth.
A
A
The
minimum
system
state
with
what
I
showed
before
is
just
daggett
and
one
copy
of
the
grpc
daemon
for
each
workspace
running
all
told
that
uses
very
few
resources,
almost
no
cpu
and
just
a
little
bit
of
ram,
because
everything
is
decoupled
via
rabbitmq,
we
use
a
tool
called
kita
to
watch
the
queue
depth
in
rabbitmq
and
drive
our
auto
scaling
and
we've
updated
some
stuff
in
kita
2.1,
with
a
bunch
of
new
features
to
make.
This
use
case
work
better
two
important
things
on
the
celery
side.
A
We
can
tailor
the
execution
environment
things
like
hardware
requirements,
docker
base
images,
sometimes
even
which
operating
systems
to
run
on,
because
we
have
one
thing
that
runs
on
windows
and
it
also
means
that
the
teams
that
are
developing
each
solid
can
be
somewhat
isolated
from
their
usage
and
pipelines,
not
completely
the
use
cases
still
matter,
but
they
can
sort
of
develop
and
release
on
their
own
schedule.
We
can
run
multiple
versions
of
each
solid
in
parallel
and
the
pipelines
can
pick
which
versions
of
things
they
want
to
use.
A
A
This
is
a
simple
use
case
of
remote
solid.
It
looks
an
awful
lot
like
a
celery
task,
because
it
is
a
celery
task.
There
we
have
some
helpers
to
just
sort
of
help,
define
them
more
quickly,
but
overall
it's
just
normal
salary.
Each
of
these
would
be
in
its
own
github
project
have
its
own
ci
its
own
everything.
A
We
could
expose
multiple
tasks
from
a
single
remote
solid
if
the
situation
called
for
it,
but
doesn't
come
up
very
often,
most
of
them
have
one
thing
that
they
do,
but
then
we
need
to
expose
that
remote,
solid
into
dagster's
view
of
the
world,
so
we've
written
a
decorator
that
extends
at
solid
to
create
what
I
call
proxy
solids,
and
this
is
a
short
example
of
one.
So
the
decorator
takes
all
the
same
arguments
as
at
solid,
but
some
additional
ones
like
say
which
rabbit
and
q
this
solid
will
live
on.
A
The
first
yield
is
yielding
what
arguments
should
be
sent
to
the
remote
salary
task,
because
usually,
as
you
can
see
here-
that's
like
extracting
some
data
from
the
dagster
level
arguments
and
then
the
return
from
the
first
yield
will
be
output
from
the
remote
solid.
So
whatever
celery
returned
will
end
up
in
output
there,
and
then,
after
that,
you
could
do
normal
dagster
stuff
data,
manipulation,
yielding
outputs
asset
materializations
all
that
kind
of
stuff.
A
Our
workflow
is
pretty
light
after
that,
each
overall
project
has
its
own
git
repository
for
the
pipelines.
One
of
those
will
be
the
default
pipeline
in
main.pi
that
sort
of
defines
when
a
customer
request
comes
in
that
doesn't
have
an
override
for
which
pipeline
to
use,
since
they
are,
the
pipelines
are
also
versioned.
It'll
use
the
default
one.
We
have
the
solids.pi
for
defining
all
of
those
proxy
solids
and
then
other
stuff.
A
We
use
the
daggett
graphql
api
to
integrate
into
our
backend
apis
for
things
like
launching
new
executions
via
one
of
the
graphql
mutations
or
just
like
discovering,
which
pipelines
exist
for
admin
ui
purposes
as
we
roll
out
this
workflow
to
more
internal
projects.
The
goal
is
for
the
teams
that
currently
own
the
json-based
tag
definitions
to
take
over
writing.
Daxter
pipelines
instead
they're
very
excited
about
that
as
soon
as
I
let
them
they're
ready
for
it.
A
The
current
one
is
not
great
and
just
quickly.
We
have
all
of
this
wrapped
in
a
continuous
deployment
system
via
build
kite,
so
any
commit
that
gets
merged
into
the
main
branch
for
either
a
solid
or
a
pipeline
will
be
built
into
a
container
image
edited
into
the
customized
configuration
pushed
up
into
github
and
then
deployed
via
argo
cd
there's
a
little
bit
more
complexity
in
some
of
the
fancier
remote
solids
because
there's
an
operator
because
I'm
that
kind
of
nerd
but
overall,
fairly
simple,
straightforward
system.
A
Just
as
soon
as
you
merge
it.
It
ends
up
in
the
system,
but
this
whole
setup
isn't
without
issues.
The
biggest
downside
is
the
time
to
cold
start,
a
pipeline
that
hasn't
been
used.
Lately,
we
have
one
pipeline
in
particular
that
it
can
take
10
to
20
minutes
before
things
actually
get
underway,
because
the
machines
it
has
to
launch
are
fairly
large
and
the
container
images
are
many
gigabytes,
because
most
of
the
work
is
in
the
remote
solids.
We
also
don't
yet
have
any
way
to
show
progress
or
status
information
during
a
long
step.
A
A
But
if
you're
doing
weird
stuff,
you're
gonna
hit
weird
bugs,
we
haven't
moved
our
main
product,
yet
as
I've
been
sort
of
hinting
at
throughout
this
we've
been
using
a
lot
of
smaller
projects
to
develop
and
battle
test
this
execution
layer,
and
so
far
things
are
looking
very
positive.
We
can
throw
hundreds
of
simultaneous
runs
into
the
system
and
have
them
come
out
the
other
side,
the
main
blood
blocking
the
main
bug
blocking
us
from
moving
forward
on
the
rollout
is
every
now
and
then
we
do
have
unexplained
task
failures
for
an
obvious
reason.
A
I
think
it's
a
rabbitmq
timeout,
but
I
need
to
add
more
debugging
output
and
I
think,
we'll
be
back
on
track
soon
and
there's
still
also
a
lot
of
room
to
improve
it.
The
biggest
thing
that
would
improve
this
would
be
async
execution
support
right
now,
because
the
execution
process
itself
is
synchronous
and
because
we
want
to
run
very
high
concurrency
most
of
the
time
for
us,
it's
going
to
be
blocked
in
waiting
for
the
remote
celery
task
to
return.
A
So
we
don't
want
to
run
the
overhead
of
a
separate
process
for
every
one
of
those
so
we're
using
threads.
We
run
15
threads
per
workspace
level
worker
which
works
great,
except
that
I
don't
think
anyone
else
using
dagster
is
doing
multi-threaded
execution,
and
so
we've
run
into
a
couple
of
weird
bugs
related
to
that.
But
a
fully
async
executor
would
allow
for
better
time
sharing
for
both
our
weird
use
case
and
everyone
else.
A
We'd
also
like
to
fix
that
incremental
feedback
gap,
or
just
more
generally
have
ways
for
non-dagster
systems
to
yield
events
into
the
dagster
log,
the
dagster
event
framework
and
another
thing
that
we're
looking
at
is
pipeline
level
web
hooks.
There's
been
a
ticket
open
for
this
recently
right
now,
it's
really
easy
to
deal
with
successes.
A
This
is
a
pretty
good
way
to
run
dagster.
I
clearly
think
it
is
our
solids
and
pipelines
aren't
public,
but
the
core
helper
library
is.
You
can
look
it
up
on
github
geomagical
dagstr
gmatchel,
but
it
is
extremely
undocumented
and
I'm
still
really
frequently
making
major
changes
to
it.
So
think
of
this
as
a
place
to
borrow
code
and
ideas
from
please
don't
try
and
import
and
use
it
directly.
I
feel
like
that's,
not
gonna
go
well
and
that's
it.