►
Description
In our fifth Dagster Community Meeting, Dennis Hume from Drizly and Noah Kantrowitz from Geomagical Labs discuss about their production Dagster setups.
Register for the next community meeting ➡️ http://bit.ly/march-cm
👨🏫 Today's Agenda 👩🏫
00:00 - Introduction
01:02 - Dennis Hume demonstrates how Drizly configures its Dagster deployments to support a heterogeneous team.
16:32 - Noah Kantrowitz discusses Geomagical’s approach to building customer-facing applications.
🎞 Slides 🎞
Drizly & Dagster (Dennis Hume) ➡️ https://drive.google.com/file/d/1sF_eIjwzVutPxjUwtayObgYgkUoquvE4/view?usp=sharing
Geomagical Labs & Dagster (Noah Kantrowitz) ➡️ https://drive.google.com/file/d/1sFBrqX8qQEgOKGTKxCqMBwDbdPdPWkHd/view?usp=sharing
🌟 Socials 🌟
Follow us on Twitter ➡️ https://twitter.com/dagsterio
Checkout our Github ➡️ https://github.com/dagster-io/dagster
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Check out our Documentation ➡️ https://docs.dagster.io/
A
Welcome
we
have
these
on
the
first
tuesday
of
every
month,
and
we
use
this
meeting
to
give
the
community
a
chance
to
meet
and
share
ideas.
As
well
as
ask
any
question
to
the
dance
report
team.
Today
we
have
two
really
interesting
presentations
lined
up.
First,
we'll
have
dennis
hume
a
staff
date
engineer
from
grizzly,
we'll
talk
about
how
drizzle
configures
its
stacks
of
deployments
to
support
heterogeneous
teams
and
then
we'll
have
noah
from
geomagical
labs.
Who
will
talk
about
how
they
use
daxter
to
build
customer-facing
applications?
A
If
you
have
any
questions
in
the
presentations
feel
free
to
drop
in
the
zoom
chat
or
stay
until
the
end,
we'll
have
a
q
a
session
to
close
out
the
meeting,
also
a
small
note
for
future
community
meetings.
If
you
have
a
topic
that
you'd
like
to
present
and
share
with
the
community,
please
reach
out
to
us
on
slack,
you
can
reach
out
to
me
I'm
shashank
on
slack
and
we'd
love
to
chat
about.
You
chat
with
you
to
have
a
presentation
at
one
of
these
future
meetings.
B
Okay,
so
hi,
I'm
dennis
I'm
a
data
engineer
at
drizzly
and
just
kind
of
wanted
to
go
over
how
drizzly's
kind
of
adopted,
dagster
and
tried
to
make
this
something
that
works
for
our
entire
team
and
there
we
go
okay,
so
the
drizzly
data
team
is
pretty
similar
to.
I
think
most
data
teams,
we're
kind
of
a
collection
of
analysts,
data
scientists
and
data
engineers,
who
kind
of
all
have
very
different
needs
out
of
our
data
stack
and
really.
B
So
when
we
kind
of
started
rebuilding
our
data
infrastructure
about
two
years
or
a
year
and
a
half
ago,
we
tried
to
build
it
along
this
line
of
like
of
shared
spaces.
We
didn't
really
want
anyone's
workflow
to
be
to
feel
very
different
from
anyone
else's.
We
didn't
want
a
data
scientist
to
feel
like
what
they
do
is
very
different
from
an
analyst,
and
we
didn't
want
data
silos
to
emerge
of
it's
a
very
different
workflow
for
someone
on
marketing
versus
someone
on
strategic
partnerships.
B
B
We
could
also
just
hold
all
of
our
logic
in
one
place
with
dbt,
and
then
it
really
just
never
felt
that
different
if
you're
building
a
a
data,
science,
sql
model
or
a
model
that
would
be
going
into
our
our
visualization
layer
for
marketing
or
something
else,
so
that
worked
really
well.
What
was
a
harder
question
was
how
we
were
going
to
build
that
shared
space
for
non-sql
workflows,
because
here
the
differences
and
needs
kind
of
become
a
lot
more
apparent.
B
So
this
is
where
we've
kind
of
been
hoping.
Dagster
can
fill
that
role
and
what
we've
been
moving
forward
with
dagster
so
for
people
who've
been
using
dagster
for
a
while
you're,
probably
familiar
with
a
lot
of
these
abstractions
of
just
the
differences
between
solids
and
pipelines
and
modes
and
presets.
But
the
main
thing
with
this
is
just
that
kind
of
going
back
to
dagster
appeals
to
different
roles
for
different
reasons.
B
B
This
usage
is
for
kind
of
quick
local
development
and
unit
testing
of
pipelines,
so
as
an
example,
instead
of
actually
pinging
snowflake,
this
could
just
be
some
files
saved
that
just
mimic
what
the
results
of
a
snowflake
query
would
be.
Our
dev
mode
can
be
mocked
or
non-production
versions
of
a
system,
so
this
could
be
for
sticking
with
that
snowflake
example.
B
We
have
a
dexter
compose
setup,
which
again
is
just
for
your
local
machine,
but
this
starts
to
bring
in
more
of
the
extra
dependencies,
such
as
the
postgres
database,
the
daemon
we're
working
with
on
creating
a
broker,
and
so
this
is
for
just
kind
of
more
involved
testing.
So
you
can
get
that
see
that
every
aspect
of
your
pipeline
is
working
correctly.
Then
after
this
we
start
to
actually
like
push
code
into
git
and
based
on
the
branch
you're
going
to
you.
B
The
other
thing
we
do
to
kind
of
limit
some
of
the
confusion
over
this.
The
different
deployments
are,
we
do
filtering
across
our
different
deployments,
so
we
do
filtering
on
pipelines,
modes,
presets
and
schedules,
and
this
is
just
to
make
it
easy
to
know
what
all
you
should
have
available
to
you
within
a
specific
deployment.
B
At
this
point,
the
only
as
I've
mentioned,
the
only
modes
you
have
available
are
local
and
the
dags
for
configuration
is
none
because
there's
no
additional
dependencies.
So
this
is
kind
of
what
the
instance
looks
like
and
then
object
filtering
you
just
have
access
to
local
local
mode
and
and
presets,
and
you
don't
have
avail
any
of
the
schedules
available.
B
B
So
here
we
can
use
pretty
much
what
we're
going
to
be
using
in
production,
except
we
can
use
like
the
docker
launcher
instead
of
a
custom
like
ecs
launcher
that
we'd
use
in
aws-
and
this
again
is
just
allows
us
to
keep
moving
our
code
along
and
get
closer
and
closer
to
what
it
would
look
like
in
actual
production
without
actually
having
to
get
to
that
step.
Quite
yet,
so
here
the
instance
is
different,
because
we
have
a
different
workspace
where
we
have
multiple
repositories.
B
Now
we
have
the
bi
repository
and
the
data
science
repository,
which
is
broken
in
this
screenshot
and
I
think,
is
going
to
be
broken
in
my
demo.
And
then
you
can
see
that
the
the
dexter
daemon
is
running
in
this
environment
and
for
object
filtering.
You
now
have
access
to
both
local
and
dev
for
presets
and
modes,
and
now
have
schedules
present,
so
our
dev
environ
or
our
dev
deployment
is
just
again.
This
is
what
we're
starting
to
get
to
what
it
will.
B
Look
like
in
production,
so
this
is
just
dagster
running
on
aws
resources,
so
we
are
just
kind
of
built.
Our
own
stack
around
ecs.
We
don't
use
ikea
eks
for
our
deployment
of
dagster,
but
this
is
pretty
much
again.
It
looks
pretty
similar
to
docker
compose
but
again
we're
just
using
aws
resources
at
this
point
and
then
our
production
stack
is
the
exact
same
stack,
but
just
on
a
different
account.
B
So
one
thing:
that's
kind
of
holding
this
all
together
are
our
data.
Scientists
have
put
together
some
very
nice
cookie
cutter
templates,
and
this
allows
us
to
just
very
easily
spin
up
new
pipelines
that
adhere
to
our
deployments.
So
this
again
just
makes
it
easier
for
people
to
be
able
to
quickly
get
a
pipeline
off
the
ground
and
not
have
to
worry
about
just
all
the
all
the
infrastructure
in
the
back.
B
B
So
at
this
point
the
resources
being
used
are
snowflake
and
s3,
but
since
we're
in
local
mode,
this
will
all
just
be
mocked.
So
we
can
run
this
pipeline,
it
will
run,
but
it
won't
actually
be
doing
much
just
because
the
s3
it's
not
connected
to
s3.
So
this
is
just
pretty
much
to
make
sure
that
our
pipeline
compiles
and
we
can
see
it
in
the
dagster
ui
and
then
also
that
we
can
write
unit
tests.
So
we
can
write
unit
tests
still
that
ensure
that
that
mapping
is
running
correctly.
B
B
B
B
But
if
you
look
in
the
schedules
for
this
environment,
you
only
see
the
schedule
for
mo
for
development
and
that
again
is
just
to
kind
of
keep
it
a
little
simple
and
you
don't
have
to
make
changes
by
like
testing
something
out
in
one
environment.
And
then,
if
you
forget
to
change
it,
it
accidentally
getting
pushed
to
prod
or
something
like
that.
B
B
If
you
want
to
learn
more
about
kind
of
bridging
the
dagster
dbt
snowflake,
all
together
in
one
place,
our
infrastructure
lead
gave
a
talk
at
dbt
coalesce
in
I
think,
if
that
was
november
or
december
of
last
year,
but
it's
a
snowflake
dbt
talk
primarily,
but
there's
also
dagsters
kind
of
under
the
hood.
For
that
and
that's
all
I've
got.
A
C
Our
use
case
isn't
quite
the
normal
one
for
dagstr,
so
a
quick
overview
of
our
main
product,
a
user
takes
a
bunch
of
imagery
of
their
room
using
our
mobile
app.
They
upload
those
to
our
cloud
stuff.
We
run
a
bunch
of
fancy
maps,
we
build
a
3d
view
of
their
room
and
then
they
can,
in
a
browser,
mobile
app,
drag
virtual
furniture
in
to
see
how
it
would
look
before
buying.
C
So
the
key
points
there
are
that
it's
directly
customer
facing,
so
we
need
it
to
run
quickly
and
there's
very
little
margin
for
error,
because
every
second
counts
when
a
customer
is
waiting
on
us.
Also,
the
fancy
map
that
we
run
can
get
very,
very
fancy.
Some
of
the
steps
vary
from
little
python
scripts
to
large
c,
plus
plus
tools
that
use
tens
of
gigabytes
of
ram
and
require
a
gpu
before
we
dive
into
the
dax
setup.
Let's
look
what
we
had
before
for
our
main
product.
C
We
knew
we
wanted
to
keep
most
of
the
structure
of
our
existing
solids
in
place,
both
because
updating
them
would
be
really
costly
and,
honestly
that
part
of
the
system
was
going
just
fine
in
terms
of
workflow.
The
big
problem
was
that
our
dag
completely
limited,
and
can
this
didn't
really
offer
much
room
for
improvement?
We
tried
a
couple
times
and
it
didn't
work
for
inexplicable
reasons,
so
canvas
definitely
had
to
go.
Also.
C
Our
workloads
are
really
bursty,
so
anything
that
allows
us
to
shut
down
more
capacity
when
it's
not
in
use
helps
us
keep
cost
down,
which
is
really
nice
when
you're
allocating
giant
cloud
machines
and
also
if
we
could
get
better
run
tracking
with
more
detailed
information.
That
would
be
really
nice.
I
mean
I
can
write
all
that
instrumentation
myself,
but
if
I
don't
have
to
that's
great,
obviously
we
found
it
like
dagster.
C
You
wouldn't
be
hearing
from
me
today,
but
in
short,
it
ticked
all
the
boxes
I
was
looking
for
and
anything
that
it
didn't.
We
could
help
to
improve.
So
the
biggest
thing
that
we're
looking
to
improve
on
is
the
thing
I'm
talking
about
today,
which
is
a
getting
a
scalable,
stable
execution
layer
in
place,
because
we
needed
really
high
potential
concurrency
and
much
higher
resiliency
to
failure
the
default
launcher
and
executor
combos
weren't
going
to
cut
it.
C
We
did
look
at
the
extra
celery
since
we
knew
we
wanted
celery
under
the
hood,
no
matter
what,
unfortunately,
the
workflow
for
that
more
or
less
requires
that
the
code
for
all
solids
lives
alongside
the
pipeline
or
at
least
that
all
solids
in
the
pipeline
can
be
runnable
together.
That
wouldn't
really
work
well
for
us
because
of
the
widely
varying
hardware
requirements
for
different
solids,
there's
also
daxter
kubernetes,
which
launches
kubernetes
jobs
on
the
fly.
C
This
can
be
okay
for
infrequently
executed
jobs,
where
failure
delay
doesn't
have
any
customer
facing
consequences,
but
I
really
really
suggest
that
you
keep
cube
api
server
out
of
the
hotpath
of
your
product.
It's
really
just
not
built
for
that
kind
of
thing.
That's
putting
on
my
kubernetes
hat
instead
of
dagster
hat
I'll,
save
some
back
and
forth
and
cut
to
what
we
came
up
with
so
the
normal
stuff.
Dagit
runs
is
a
kubernetes
deployment,
that's
pretty
standard
and
we
wrote
a
custom
launcher
plug-in
that
takes
the
run
request.
C
Serializes
it
into
json
puts
that
onto
a
celery
queue
and
then
each
daxter
workspace
has
a
celery
daemon
that
is
sort
of
the
launcher
daemon.
So
it
gets
those
requests
off
of
rabbit,
mq,
deserializes
it
and
acts
as
the
executor
and
then
we'll
talk
about
it.
A
little
bit
later.
Most
of
our
solids
are
also
running
as
separate
demons
like
they
were
before,
with
some
proxies
to
get
back
and
forth.
C
Dagster
delivers
that
nicely
with
a
minimum
system
state
with
what
I
showed
before
is
just
daggett
and
one
copy
of
the
grpc
daemon
for
each
workspace
running
all
told
that
uses
very
few
resources,
almost
no
cpu
and
just
a
little
bit
of
ram,
because
everything
is
decoupled
via
mq,
we
use
a
tool
called
kita
to
watch
the
q
depth
in
rabbitmq
and
drive
our
auto
scaling
and
we've
updated
some
stuff
in
keto
2.1,
with
a
bunch
of
new
features
to
make.
This
use
case
work
better
two
important
things
on
the
celery
side.
C
We
can
tailor
the
execution
environment
things
like
hardware
requirements,
docker
base
images,
sometimes
even
which
operating
systems
to
run
on,
because
we
have
one
thing
that
runs
on
windows
and
it
also
means
that
the
teams
that
are
developing
each
solid
can
be
somewhat
isolated
from
their
usage
and
pipelines,
not
completely
the
use
cases
still
matter,
but
they
can
sort
of
develop
and
release
on
their
own
schedule.
We
can
run
multiple
versions
of
each
solid
in
parallel
and
the
pipelines
can
pick
which
versions
of
things
they
want
to
use.
C
C
C
Each
of
these
would
be
in
its
own
github
project
have
its
own
ci
its
own
everything.
We
could
expose
multiple
tasks
from
a
single
remote
solid
if
the
situation
called
for
it,
but
doesn't
come
up
very
often,
most
of
them
have
one
thing
that
they
do,
but
then
we
need
to
expose
that
remote,
solid
into
dagster's
view
of
the
world,
so
we've
written
a
decorator
that
extends
at
solid
to
create
what
I
call
proxy
solids,
and
this
is
a
short
example
of
one.
C
So
the
decorator
takes
all
the
same
arguments
as
at
solid,
but
some
additional
ones
like
say
which
rabbit
and
q
this
solid
will
live
on.
The
first
yield
is
yielding
what
arguments
should
be
sent
to
the
rote
celery
task,
because
usually,
as
you
can
see
here-
that's
like
extracting
some
data
from
the
dagster
level
arguments
and
then
the
return
from
the
first
yield
will
be
output
from
the
remote
solid.
C
Our
workflow
is
pretty
light
after
that,
each
overall
project
has
its
own
git
repository
for
the
pipelines.
One
of
those
will
be
the
default
pipeline
in
main.pi
that
sort
of
defines
when
a
customer
request
comes
in
that
doesn't
have
an
override
for
which
pipeline
to
use,
since
they
are,
the
pipelines
are
also
versioned.
It'll
use
the
default
one.
We
have
the
solids.pi
for
defining
all
of
those
proxy
solids
and
then
other
stuff.
C
We
use
the
daggett
graphql
api
to
integrate
into
our
backend
apis
for
things
like
launching
new
executions
via
one
of
the
graphql
mutations
or
just
like
discovering,
which
pipelines
exist
for
admin
ui
purposes
as
we
roll
out
this
workflow
to
more
internal
projects.
The
goal
is
for
the
teams
that
currently
own
the
json-based
tag
definitions
to
take
over
writing.
Daxter
pipelines
instead
they're
very
excited
about
that
as
soon
as
I
let
them
they're
ready
for
it.
C
The
current
one
is
not
great
and
just
quickly.
We
have
all
of
this
wrapped
in
a
continuous
deployment
system
via
build
kite,
so
any
commit
that
gets
merged
into
the
main
branch
for
either
a
solid
or
a
pipeline
will
be
built
into
a
container
image
edited
into
the
customized
configuration
pushed
up
into
github
and
then
deployed
via
argo
cd
there's
a
little
bit
more
complexity
in
some
of
the
fancier
remote
solids
because
there's
an
operator
because
I'm
that
kind
of
nerd
but
overall,
fairly
simple,
straightforward
system.
Just
as
soon
as
you
merge
it.
C
It
ends
up
in
the
system,
but
this
whole
setup
isn't
without
issues.
The
biggest
downside
is
the
time
to
cold
start,
a
pipeline
that
hasn't
been
used.
Lately
we
have
one
pipeline
in
particular
that
it
can
take
10
to
20
minutes
before
things
actually
get
underway,
because
the
machines
it
has
to
launch
are
fairly
large
and
the
container
images
are
many
gigabytes,
because
most
of
the
work
is
in
the
remote
solids.
We
also
don't
yet
have
any
way
to
show
progress
or
status
information
during
a
long
step.
C
C
But
if
you're
doing
weird
stuff,
you're
gonna
hit
weird
bugs,
we
haven't
moved
our
main
product,
yet
as
I've
been
sort
of
hinting
at
throughout
this
we've
been
using
a
lot
of
smaller
projects
to
develop
and
battle
test
this
execution
layer,
and
so
far
things
are
looking
very
positive.
We
can
throw
hundreds
of
simultaneous
runs
into
the
system
and
have
them
come
out
the
other
side,
the
main
blog
blocking
the
main
bug
blocking
us
from
moving
forward
on
the
rollout
is
every
now
and
then
we
do
have
unexplained
task
failures
for
an
obvious
reason.
C
I
think
it's
a
rabbitmq
timeout,
but
I
need
to
add
more
debugging
output
and
I
think,
we'll
be
back
on
track
soon
and
there's
still
also
a
lot
of
room
to
improve
it.
The
biggest
thing
that
would
improve
this
would
be
async.
Execution
supports
right
now,
because
the
execution
process
itself
is
synchronous
and
because
we
want
to
run
very
high
concurrency
most
of
the
time
for
us,
it's
going
to
be
blocked
in
waiting
for
the
remote
celery
task
to
return.
C
So
we
don't
want
to
run
the
overhead
of
a
separate
process
for
every
one
of
those
so
we're
using
threads.
We
run
15
threads
per
workspace
level
worker
which
works
great,
except
that
I
don't
think
anyone
else
using
dagster
is
doing
multi-threaded
execution,
and
so
we've
run
into
a
couple
of
weird
bugs
related
to
that.
But
a
fully
async
executor
would
allow
for
better
time
sharing
for
both
our
weird
use
case
and
everyone
else.
We'd
also
like
to
fix
that
incremental
feedback
gap,
or
just
more
generally
have
ways
for
non-dagster
systems
to
yield.
C
C
This
is
a
pretty
good
way
to
run
daxter.
I
clearly
think
it
is
our
solids
and
pipelines
aren't
public,
but
the
core
helper
library
is.
You
can
look
it
up
on
github
geomagical
extra
jmagil,
but
it
is
extremely
undocumented
and
I'm
still
really
frequently
making
major
changes
to
it.
So
think
of
this
as
a
place
to
borrow
code
and
ideas
from
please
don't
try
and
import
and
use
it
directly.
I
feel
like
that's,
not
gonna
go
well
and
that's
it.