►
Description
In our fourteeth Dagster Community Meeting, the Dagster team showed new updates and assets.
👨🏫 Agenda 👩🏫
Introduction/Announcements: 0:00
Community Spotlight: 4:18
Re-introducing Dagster-Airflow: 7:17
Asset updates - Loading asset values: 18:05
Asset updates - Multi-asset sensors and asset reconciliation: 20:35
Dagit updates: 25:40
Orchestrating Airbyte connections with Dagster: 35:11
Q&A: 40:27
🗞 Related blog posts 🗞
Dagster 0.15.0: Cool for the Summer: https://dagster.io/blog/dagster-0-15-0-cool-for-the-summer
Orchestrating python and dbt with Dagster: https://dagster.io/blog/dagster-0-15-0-dbt-python
🌟 Socials 🌟
Checkout (and star!) our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio
A
Hi
everybody
Welcome
to
daxter's
community
meeting
the
October
2022
Edition
nice
to
see
you
all
after
the
Dexter
day.
For
those
who
don't
know
me,
my
name
is
Johan.
I
am
an
engineer
at
Elemental
and
I'm,
also
Johan
in
the
Dexter
slack.
You
may
have
seen
me
in
there.
So
today
we
forgot
a
pretty
packed
agenda
for
you.
A
Okay,
Joe
is
going
to
talk
about
using
airflow
with
daxer
and
Sandy
will
demo
how
you
can
load
assets,
how
you
can
load
asset
values
outside
of
around
and
Jamie
is
going
to
present
about
multi-asset
sensors
and
asset
acceleration
Josh.
Our
designer
will
talk
about
a
a
few
exciting
updates
about
dagget
and
last
but
not
least,
Ben
will
showcase
the
new
Air
by
integration
that
we
released
last
week.
A
B
A
Oh,
the
other
thing
that
I
would
like
to
call
out
as
most
of
the
features
that
we'll
be
demoing
today
are
experimental
so
and
we're
actively
looking
for
early
feedback.
So
if
you
would
like
to
try
stuff
early
on
and
help
us
improve
any
of
them,
please
reach
out
to
me
on
slack
or,
as
always,
feel
free
to
leave
feedback
publicly
in
the
text
or
feedback
channel
in
stock.
A
Okay,
let's
get
this
started
a
a
few
quick
announcements,
as
always
we
are
hiring
on
here-
are
the
openings
at
Elemental.
Among
these
rules,
I
specifically
want
to
call
out
the
data
engineer,
developer,
Advocate
role.
This
role
will
be
creating
content
representing
Dexter
in
the
developer
community,
engaging
with
our
users
and
growing
a
community
and,
most
importantly,
as
a
data
practitioner,
you
will
be
job
fooding
data
engineering
projects,
indexer
to
understand,
frictions
and
pains
filled
by
our
users.
A
So
if
you
already
enjoy
using
daxer-
and
you
want
to
help-
others
have
amazing
experience
with
that
as
well
reach
out
to
us,
and
you
can
also
find
on
all
other
openings
on
elemento.com
our
next
announcement.
We've
got
the
bye
weekly
live
demos.
We've
had
the
very
first
one
on
last
Thursday
and
this
demo
will
be
held
every
other
Thursday
at
9am,
Pacific
time,
if
you'd
like
to
see
a
live
demo
plus
q
and
A's,
with
Engineers
from
the
core
team
sign
up
in
the
link
here.
A
Next,
we've
got
some
Community
event,
we're
going
to
coalesce
on
I,
believe
some
of
you
or
as
well
and
we'll
be
hosting
a
happy
hour
on
October
19th,
the
Wednesday
in
New
Orleans.
If,
if
you
will
be
around
come
to
say,
hi
me
and
other
folks
from
the
team
will
be
there
and
RSVP
is
in
the
announcement
Channel
in
Dexter
stock.
A
This
is
a
recurring
theme
that
we
started
since
the
June
meeting,
where
we're
showing
off
the
content
that
our
users
have
created
since
the
last
community
meeting,
which
was
the
tax
today
back
in
August.
For
this
time
and
I
have
to
say,
there
is
a
lot
of
exciting
content
happening
and
interesting
discussions
happening
in
the
community.
Lately,
the
first
ones
are
dexterity
Recaps
I'd
like
to
shout
out
to
Hector
Robles
and
our
friend
Simon
at
airbag,
for
creating
this
amazing
summaries
for
the
Dexter
day
and
the
extra
1.0
next
we've
got
lately.
A
There
are
all
a
couple
of
Articles
and
videos
created
in
terms
of
using
daxer
for
among
Ops
use
cases.
Stefan
Samba
talked
about
using
Dexter
with
notebooks
and
paper
mill
adapter
from
doordash
shared
how
they
used
Dexter
to
scale
up
the
simulation
modeling
at
doordash,
and
we
also
have
Seattle
data
guy
interviewed
Evolution
IQ
for
how
and
why
they
chose
Dexter
to
becoming
their
oxygen
layer.
A
And
the
next
theme
in
the
spotlights
is
Dexter
versus
others,
such
as
airflow,
which
is
a
Hot
Topic
lately
and
Davidson
Thomas
discussed
about
daxer
and
it's
software
defined
assets
which
both
includes
on
comparing
contrast
and
pros
and
cons
with
some
of
our
competitors
and
for
folks
who
can
speak
Portuguese
I
cannot
so
I
am
not
going
to
try
to
pronunciate
the
title,
but
you
can
tell
it
from
this
lovely
cover
image.
You've
got
Lucas
writing
about
Dax
versus
airflow
in
Portuguese
and
I'll,
see
chorus
course
for
those
who
attended
the
Lost
community
meeting.
A
We
mentioned
that
Dennis
Hume
started
teaching,
Dexter
Uncle
rice
and
the
last
round
just
finished
congrats
to
Dennis
on
finishing
up
this
very
first
batch
of
the
course
and
the
next
round,
I
believe
will
start
pretty
soon
in
late
November
check
out
the
course
in
the
link
here
so
yeah.
Thank
you.
Everyone
for
supporting
the
community
and
creating
these
awesome
contents.
Lastly,
f
and
did
I
miss
anything
if
you've
created
a
Content
or
if
you
are
creating
a
Content
that
you
want
to
share
and
collaborate
with
us.
A
C
Cool
so
yeah
I'm
gonna
be
giving
a
reintroduction
to
the
dagster
airflow
Library.
If
you
don't
use
airflow,
and
you
don't
need
to
think
about
it,
then
you
can
probably
tune
out
for
this
next
section,
but
hopefully
I'll
be
fairly
quick
and
also
you
should
count
yourself
lucky
so
Dexter
airflow
is
changing
its
focus
kind
of
the
background
of
this
Library.
C
So
before
the
idea
of
like
a
persistent
platform,
that's
based
on
dagster,
but
over
time
it's
informally
become
a
way
for
dags
or
users
who
have
limited
interoperability
with
airflow
and
I'll.
Talk
more
about
this
in
a
second
and
basically
we're
going
to
formalize.
This
focus
on
supporting
users
who
are
incrementally
migrated
migrating
from
airflow
to
dagster,
because
that's
kind
of
how
it's
over
time
actually
been
used,
especially
as
we've
moved
away
from
that
Old
vision
for
how
you
could
use
dagster.
C
So
who
is
dagster
airflow
meant
for,
and
these
are
kind
of,
our
Target
user.
To
personas
so
data
practitioners,
who
want
to
use
the
category
defining
apis
and
debacs
of
dagster
and,
like
the
testing
story,
local
development,
without
infer
or
without
pursuing
in
for
a
platform
change
whether
they
don't
have
the
ability
to
do
that
at
their
place
of
work
or
the
interest
of
like
setting
up
new
infrastructure
specific
to
dagster
and
then
also
platform
Engineers.
C
Who
do
want
to
pursue
platform
or
infrastructure
change
for
a
variety
of
reasons.
Maybe
they
have
Crafty
expensive
or
hard
to
maintain
airflow
instances
or
a
bunch
of
Orphan
dags
that
they
are
de
facto
owners
of
and
want
to
move
to
something
that's
a
little
more
modern
okay.
So
what
are
the
kind
of
three
patterns
that
we're
looking
to
support
and
kind
of
expand
upon
for
those
two?
Two
user
personas?
The
first
is
something
that's
new.
C
You
can
check
it
out
after
this
week's
release,
the
dagster
operator
and
the
dagster
fat
operator,
and
then
the
second
one,
which
is
the
most
common
one
that
people
have
been
using
in
the
past,
is
to
make
airflow
dag
or
make
an
airflow
Jag
containerize
functions.
And
then
the
third
is
make
dags
your
job
from
airflow
dag
a
bit
of
a
words
out,
but
I'll
get
into
each
of
these,
so
the
Daxter
Cloud
operator.
C
This
allows
you
to
orchestrate
dags
for
job
runs
running
on
dags
for
infrastructure
from
airflow
airflow
acts
as
your
data
operations
layer,
so
you'll
still
be
using
the
airflow
scheduler
any
alarm
infrastructure
you
have
configured
on
airflow
and
the
backflow
mechanisms.
So
you,
as
like
an
individual
contributor
could
be
you'd
still
be
like
able
to
deploy
in
quotes
your
new
dagster
jobs
using
the
same
like
data
Ops
patterns
that
you're
using
currently
with
airflow.
C
So
you
still
get
the
obviously
the
programming
and
the
observability
layer,
and
that
means
that
all
your
runs
and
logs
will
be
stored
and
viewable
and
dagged.
So
you
can
imagine
over
time,
potentially
your
team
with
multiple
airflow
instances.
You
might
be
adopting
this
as
a
pattern
to
kind
of
incrementally
migrate
and
start
doing.
C
All
new
development
on
Old
airflow,
with
dagster
All
Those
runs,
will
be
happening
on
like
yours,
like
you
might
be
orchestrating
them
from
a
variety
of
influences,
but
all
the
runs
and
kind
of
the
state
that
you're
storing
like
the
runs
and
the
logs
would
all
be
on
a
centralized
dagster.
So
it'd
be
like
a
nice
way
for
you
over
time
to
move
from
disparate
infrastructure
to
a
centralized
kind
of
platform.
C
It
has
some
downsides,
though,
so
you
have
to
have
the
Dax
your
infrastructure
or
have
Dax
your
Cloud
yeah,
which
might
be
a
bit
of
a
hurdle
for
some
some
some
users,
and
this
is
kind
of
how
it
looks
so
on
the
left.
You
have
an
airflow
tag
that
is
creating
a
Dax
or
Cloud
operator.
You
can
see
the
Run
config
there
and
configuring.
The
serials
app
that
BS
macro,
if
you're
familiar
with
airflow,
that's
like
a
way
you
can
use
the
airflow
kind
of
macro
templating
system.
C
You
can
also
reference
task.
Instance.
There
you
also
yeah,
you
don't
need
to
pass
in
the
whole
config
lab
there.
You
could
also
pass
in
a
file
path
to
a
yaml
or
a
Json
file,
stuff,
that's
pretty
standard
for
for
a
lot
of
airflow
operators.
Then
there's
also
a
connection
that
we
that's
provided
if
you're
using
airflow
2,
where
you
can
capture
like
your
dagster,
Cloud
token
and
organization,
name
and
other
some
other
shared
config
stuff
in
in
the
in
the
airflow
connection.
C
Instead
of
having
to
pass
it
directly
through
the
operator,
then
on
the
right.
This
is
how
it'll
look
in
the
Oracle
UI.
You
get
like
a
direct
link
to
the
Run
UI
with
this
Dax,
your
Cloud
button
at
the
bottom
yeah
pretty
straightforward.
It
relies
on
the
the
graphql
API
for
kind
of
managing
like
kicking
things
off
and
catching
logs
and
canceling.
If
you
would
hit
cancel
cool.
C
So
this
renders
a
Dax
your
job
into
an
error
code,
dag
and
it
uses
a
special
dagster
operator
or
Dax
your
containerized
operator,
which
is
just
really
like
a
fork
of
the
docker
operator
for
each
of
the
Ops.
So
like
the
the
the
the
job
that
you
turn
into
in
an
airflow
dag,
will
an
OP
will
turn
into
this
an
airflow
task
that
uses
one
of
these
custom
operator,
yeah,
dags
or
operators.
C
So
in
this
pattern,
the
airflow
is
acting
as
the
entire
data,
Ops
execution
and
observability
layer.
There's
daxer
is
purely
used
just
for
its
programming
API
and
like
devex,
like
local
development
experience,
there's
no
state
run
dagster
state
that
gets
persisted
anywhere,
so
it's
all
just
dropped
on
the
on
the
floor
and
you
just
have
airflow
for
for
that
type
of
stuff,
and
it
has
some
downsides
So.
Currently,
it
only
works
with
airflow
one.
C
So
this
is
obviously
not
great,
because
most
people
are
in
airflow
2
these
days,
and
it
also
has
no
support
for
custom
operators.
Specifically,
it
doesn't
support
the
kubernetes
Pod
operator
which,
if
it's
fairly
common
for
a
lot
of
airflow
shops,
to
to
kind
of
rely
on
those
as
kind
of
the
the
main
operator
that's
used
in
all
the
dacs
and-
and
this
is
how
kind
of
it
would
look
in
code,
so
the
make
it
airflow
dag
function
down
here.
C
It
returns
a
dag
and
a
collection
of
tasks,
and
then
the
way
you
kind
of
make
it
work
is
the
module
name
here,
which
was
python
module.
That
has
the
airflow
repo
as
part
of
it,
and
you
basically
pass
in
that
airflow
repo,
in
this
case
the
hello
serial.
Sorry,
the
Dax
cerebral
in
this
case
Hello
serial
and
select
the
job
name
that
you're
going
to
create
at
an
airflow
DAC
for
and
then
also
the
airflow
args
you're,
going
to
pass
to
that.
C
C
C
Airflow
is
only
used
for
its
apis
in
this
pattern
and
dagster
is
everything
else.
So
data
operations,
execution
observability
the
downsides
it's
unable
to
so
so
daxer
isn't
going
to
have
access
to
the
airflow
metadb
in
this
pattern.
So
anything
that
relies
on
state
that
is
normally
stored
in
the
airflow
metadb
is
not
going
to
work
out
of
the
box.
It
requires
you
to
there's
some
workarounds
for
some,
some
of
the
some
of
the
some
of
the
ways
that
fails
specifically
with
connections.
C
Directly
to
and
from
the
airflow
metadb
are
not
gonna
are
not
going
to
work
well
out
of
the
box
at
least,
and
so
this
is
how
how
that
can
kind
of
look
with
pseudo
code.
So
you
can
see
here
we
we're
passing
in
the
simple,
dag
airflow
dag
class
and
into
the
make
dagster
drive
from
airflow,
and
it
returns
a
job
that
we
can
then
assign
to
our
or
with
airflow
repository
yeah.
Sorry,
one
other
thing
that
I
forgot
to
mention.
If
I
can
go
back.
C
C
We're
looking
on
adding
we're,
also
going
to
add
some
features
to
address
some
of
the
current
pain
points,
so
adding
full
support
for
airflow,
1
and
2.0,
especially
for
make
airflow
dag,
adding
support
for
custom
operator
types
as
well,
and
if
you're
interested
in
this,
please
also,
if
you're
interested
in
being
a
design
prayer.
Please
just
reach
out
to
someone
on
the
team
directly,
but
also,
if
you're
interested
in
this
topic.
Joining
the
dagster
airflow
Channel
be
great
Next
Step
all
right
and
with
that
I'll
pass
it
off
to
Sandy.
B
Continue
it's
cool
because
you
get
to
see
my
URL
bar
all
right,
so
I'm,
Sandy
I'm
the
lead
engineer
on
the
Dexter
project
and
we
recently
introduced
an
API
that
allows
loading
the
values
of
dagster
assets
into
python
objects.
This
is
already
something
that
happens
behind
the
scenes
in
Dexter,
but
what's
cool
about
this
API
is
you
can
use
it
outside
the
context
of
a
dagster
run?
B
So
if
you
materialize
an
asset
and
then
want
to
pull
up
that
asset
in
your
Jupiter
notebook
to
do
some
exploratory
analysis
now
you
can
do
that
to
show
how
this
works.
This
is
a
basic
dags
or
asset
graph
and
what
it
looks
like
in
dag
it.
So
if
you
hit
that
materialize
button
and
dag
it,
this
pipeline
is
going
to
run
and
you'll
get
a
bunch
of
data
stored
on
your
local
file
system
because
we're
using
the
local
file
system,
I
o
manager.
B
B
The
repository
is
just
a
standard
python
object
after
all,
and
then
you
can
invoke
this
new
method,
which
is
called
load
asset
value,
and
what
that's
going
to
do
is
look
at
the
definition
of
the
asset
find
the
I
o
manager
that
corresponds
to
that
definition
and
then
call
its
load
input
function
to
get
the
object,
so
you'll
notice
that
load
asset
value
accepts
a
python
type
argument.
In
this
case.
That
argument
isn't
actually
strictly
required,
but
in
general,
some
IO
managers
can
load
assets
as
different
types.
B
So,
for
example,
an
Iowa
manager
might
be
able
to
load
a
parquet
file
as
either
a
pi
spark
data
frame
or
a
pandas
data
frame
and
just
lets
you
choose
which
one
of
those
that
you
want
after
you
get
the
python
object.
You
can
do
whatever
you
want
with
it.
So
here
we're
just
rendering
the
first
five
records,
and
one
last
thing
I'll
show
is
that
if
you
want
to
load
multiple
assets,
you
can
use
this
get
asset
value,
loader
API,
and
this
is
useful
if
your
I
o
manager
has
an
expensive
initialization.
B
D
All
right
so
hi,
everyone,
I'm,
Jamie
and
I'm
gonna,
be
blocking
you
through
some
new
work.
We've
done
over
the
past
couple
months
for
adding
new
capabilities
for
asset
sensors.
So
the
new
features
we've
added,
are
kind
of
focused
on
two
different
goals.
The
first
one
is
to
allow
you
to
react
to
asset
materialization
events
in
more
customizable
ways
and
the
second
one
is
focused
on
keeping
your
assets
up
to
date,
kind
of
automatically
and
more
effortlessly.
D
So
with
all
of
this
information
about
you
know,
when
your
assets
were
materialized,
you
can
decide
how
you
want
to
react.
So
we
can
walk
through
a
couple
examples
of
the
types
of
things
you
can
do
so
for
our
first
example.
Let's
say
we
have
two
partitioned
assets
partitioned
on
the
day,
and
what
we
want
to
do
in
our
sensor
is
see
if
those
assets
have
data
available
for
the
same
partition.
So
in
this
case
for
the
same
day,
and
if
they
do,
then
we
want
to
kick
off
a
run.
D
D
Let's
say
we
have
two
other
Assets
in
this
case
these
ones
are
not
partitioned,
and
what
we
want
to
make
sure
is
that
the
clean
version
of
a
data
set
is
materialized
within
20
minutes
of
when
the
like
raw
version
of
that
data
set
materialized.
And
if
not,
we
want
to
send
an
alert
but
like
something
has
gone
wrong
and
we
should
check
on
it.
D
And
so
we
can
also
do
that
by
checking
when
these
assets
were
last
materialized
and
if
the
raw
data
was
materialized
over
20
minutes
ago
and
the
clean
data
hasn't
been
materialized.
Yet
then
we
know
something's
gone
wrong
and
we
can
send
off
this
slack
alert.
Thank
you
all
right.
So
moving
on
to
the
second
kind
of
big
feature,
that's
been
released
recently.
D
D
So
this
is
also
sort
of
best
shown
through
an
example.
So
let's
say
we
have
this
asset
graph
here
on
the
left.
We
have
a
couple
of
data,
sets
stories
and
comments
that
are
in
another
repository
and
then
within
our
repository
we
clean
both
of
those
data
sets
and
then
create
another
data
set
from
that
data.
D
So
we
can
hop
into
dag
it
to
just
kind
of
show
how
this
works.
So
here
in
dag
it
we
have.
You
know
our
repository
that
we're
interested
in
and
then
we
have
this
repository
of
kind
of
our
source
data
and
then
here
we
can
see
our
like
sensor.
Nothing's
happened,
you
know
in
the
past,
however,
many
ticks
because
nothing's
been
materialized,
but
if
we
go
over
here
and
we
materialize
these
assets,
those
will
get
materialized
and
then
on
this
next
sensor,
tick.
D
So
that's
sort
of
the
quick
overview
of
these
two
new
asset
sensor
features.
These
are
both
still
experimental.
We've
got,
you
know
even
some
more
like
documentation
for
them
coming
out
this
week,
and
so,
if
you
start
using
these
early
and
have
any
feedback
or
questions,
please
like
don't
hesitate
to
ask
us
in
the
slack
Channel.
E
Okay,
great
oops,
my
name
is
Josh.
I
am
a
product
designer
on
the
elemental
team
and
I'm,
going
to
be
talking
a
little
bit
more
about
some
updates.
We've
been
working
on
in
the
UI
to
make
it
more
performant
for
large-scale
data
platforms,
as
well
as
some
nice
new
enhancements,
we're
working
on
for
the
instance
overview
page.
E
So
first
up
is
scalability
and
performance,
so
we
want
to
make
sure
that
digit
feels
fast
and
intuitive,
even
at
the
kind
of
largest
of
use
cases
and
for
many
users
their
kind
of
first
experience
with
our
UI
is
in
local
development
when
things
are
really
really
fast
for
them.
But
as
you
kind
of
grow,
the
number
of
jobs
schedules-
sensors,
Etc
we've
heard
a
lot
of
feedback
that
daggit
hasn't
necessarily
scaled
as
well
as
the
rest
of
our
platform.
E
So
if
we
tab
over
to
the
new
workspace,
we
now
get
these
nice
little
cards
that
represent
each
of
our
repositories
in
our
workspace
and
they
give
you
a
little
preview
of
kind
of
what's
in
them.
So
you
can
see
each
of
these
has
roughly
you
know
between
200
and
300
jobs
and
200
to
300
schedules
each
as
well
as
some
assets.
E
E
You
know,
invest
and
make
this
even
you
know,
give
you
more
context
and
make
it
an
even
better
experience.
So,
let's
jump
into
a
demo
of
this
as
well.
The
instance
overview
page
now
will
have
a
new,
except
like
top
level
navigation
item
called
overview
and
it's
the
first
item
in
the
navigation
and
it
it
Remains
the
home
page,
but
now
has
like
a
much
more
like
formal
place
up
in
the
very
first
spot
in
the
navigation.
E
Within
the
overview
you
have
a
couple
tabs.
You
have
a
timeline,
a
list
of
jobs
that
is
spans
across
your
entire
deployment,
and
we've
brought
all
the
same
scalability
improvements
here,
so
here
you're,
actually
looking
at
like
a
few
or
like
over
a
thousand
jobs.
That's
all
very
you
know
searchable
right
from
right
from
this
list.
E
You
know
exactly
where
failures
may
be
happening
across
all
of
the
different
teams
that
are
using
Dexter
if
I
click
into
these
we've
added
a
couple
nice
little
improvements
to
the
way
we
display
things
within
the
timeline
view
as
well.
So
for
starters,
we
we
now
represent
any
ad
hoc
asset
materializations
in
in
a
row
within
the
timeline
view.
E
So
these
are.
These
are
runs
that
are
created
when
you
click
that
materialize
asset
button
that
when
the
asset
doesn't
necessarily
belong
to
any
any
job,
we
now
still
show
it
in
the
timeline
view,
and
you
can
kind
of
dig
in
and
investigate
any
you
know
successful
or
failure
runs
triggered
by
asset
asset
materializations,
and
then
we
also
now
have
better
support
for
batches
of
runs
that
have
mixed
statuses.
E
So
in
the
past,
when
you
know
this
batch
of
three
runs
had
one
failure
in
it,
we
would
highlight
the
whole
thing
in
red
and
kind
of
treat
it
as
like
one
big
failure.
Now
we
proportionally
colorize
These
Bars,
so
you
know
one
failure
doesn't
necessarily
have
this
huge
blotch
of
red
to
the
entire
timeline
view,
and
it's
a
bit
more
accurate
to
you
know,
what's
really
happening,
and
then
the
last
piece
is
that
we're
bringing
pagination
to
the
timeline
view
as
well.
E
So
you
can
now
over
on
the
right
hand,
side.
You
can
actually
like
look
back
in
time
and
see
exactly
what
has
happened
across
all
of
your
jobs
and
asset
materializations
at
any
point
in
time.
So
you
can
navigate
back
by
by
12
or
24
hours,
so
it's
a
great
way
to
kind
of
compare
what
has
happened
between
two
different
days.
E
So
that's
what's
new
in
the
timeline
or
in
instance,
overview
and
timeline
view,
like
the
other
features
demoed
here.
These
are
both
experimental
features
right
now,
so
the
new
workspace
page
is
actually
available.
Now
you
can
go
to
your
user
settings
and
toggle
this
new
workspace
page
and
get
us
a
try
that
out
right
now
and
then
the
new
overview
page
is
going
to
be
available
in
the
release.
E
That's
going
up
this
week
also
under
an
experimental
feature
flag,
we're
going
to
keep
these
both
as
experimental
for
the
time
being.
While
we
get
more
feedback
and
kind
of
smooth
out
any
rough
edges,
but
we're
aiming
to
make
these
the
sort
of
default
experience
by
the
next
major
release
or
or
sooner
and
then
that
leads
me
to
you-
know
I,
encourage
you
all
to
try
these
and
the
best
way
that
we
can
kind
of
continue
to
improve.
E
These
is
just
getting
your
feedback,
and
so
please,
you
know,
share
share
your
thoughts,
whether
good
or
bad.
While
we
continue
to
read
on
these,
you
can
use
the
decks
or
feedback
Channel
or
if
there's
things
you
just
don't
want
to
share
publicly.
You
can
always
DM
me
I'm,
just
at
Josh
in
the
community
Channel
thanks
that
will,
with
that
I'll
turn
it
over
to
Ben.
F
So
hi
everyone-
my
name-
is
Ben
and
I'm.
An
engineer
here
at
Elemental
and
today
I'd
like
to
share
some
of
our
improvements
to
our
air
byte
integration.
So
earbud
is
an
open
source.
Elt
Tool
lets
you
move
data
from
you
know
various
sources,
whether
that's
a
postgres
database,
Salesforce
stripe,
other
tools
to
destinations.
Usually
you
know
a
data
warehouse,
something
like
snowflake
or
bigquery,
and
so
our
data
diagster
earbud
integration,
which
we've
recently
updated,
lets
you
kind
of
manage
the
state
of
those
connections
in
dagster.
F
So
it
lets
you
trigger
six
of
those
connections
and
monitor
their
status
using
diagrams
towards
the
orchestration
layer
and
we've
recently
added
support
for
generating
software
to
find
assets
for
each
of
the
tables
that
are
synchronized
over
by
each
connection.
This
means,
you
know
you
can
set
it
up
to.
You
know,
consume
these
assets
Downstream.
F
If
you
have.
You
know
Transformations
that
you
want
to
perform
on
the
data
after
it's
been
synced
over
by
or
buy
it.
So
one
of
the
recent
changes
that
we've
made
to
the
integration
is
to
make
it
easier
to
load
these
assets
into
your
dagster
instance.
So
before
you'd
have
to
kind
of
manually
construct
each
of
these
assets,
but
now
we
have
the
ability
to
generate
assets
just
by
pointing
diagster
at
your
byte
instance,
so
sort
of
treating
that
instance
as
the
source
of
Truth.
F
You
can
see
here
kind
of
what
that
code
looks
like
under
the
hood
you'll
import,
the
new
load
assets
from
earbud
instance
method,
you'll
configure
an
air
byte
resource
pointing
towards
your
instance.
So
here
it's
just
running
on
localhost
at
this
particular
port
and
then
you
all
you
need
to
do
is
call
load
assets
from
airbind
instance
passing
in
that
instance,
and
that'll
generate
a
set
of
assets.
F
So
let's
take
a
look
at
what
this
looks
like
in
practice.
So
here
we
have
an
errite
instance.
That's
you
know
running
locally
on
my
computer.
We
have
a
a
source
here,
pointing
out
the
dagster
GitHub
repository
that
can
pull
various
pieces
of
metadata.
From
that
repository
we
have
a
snowflake
destination,
set
up
to
write
to
snowflake,
and
then
we've
actually
already
set
up
a
connection
here
that
is
going
to
replicate
some
of
that
GitHub
data.
F
So
if
we
go
over
to
our
dagster
instance,
here
we
can
go
ahead
and
reload
and
we'll
see
that
we
have
software
defined
assets,
kind
of
already
grouped
here
for
that
Stargazer
stream
and
actually
for
one
of
the
nested
pieces
of
data
there.
So
we
actually
get
the
user
data
corresponding
to
the
users
who
have
starred
the
rebound.
We
can
see
the
metadata
column
here.
You
know
we
get
metadata
of
each
of
the.
F
You
know,
columns
that
are
going
to
be
created
by
air
byte
when
we
run
the
sync
and
we
can
actually
go
ahead
and
kick
off
the
materialization
which,
under
the
hood,
is
going
to
trigger
a
sync
in
Airline,
which
we
can
refresh
and
see
right
there.
But
the
nice
thing
about
loading
directly
from
earbud
is
that
updating
configuration
will
reflect
changes
in
into
extra
immediately.
F
There
are
a
couple
other
improvements
that
I
wanted
to
highlight
here.
Everybody
all
also
offers
sort
of
a
config
as
code
solution.
There's
an
Octavia
CLI
that
lets
you
manage
your
airbite
instance
using
yaml
files
kind
of
like
what
you
see
on
the
left
here.
We've
also
shipped
support
for
generating
software-defined
assets
from
dzml
files.
F
You
know,
airbright
configuration-
and
you
know
actually
embed
this
in
your
diagnostal
pipelines-
to
make
it
easier
to
you,
know
dynamically,
create
connections
and
to
version
them
alongside
your
code.
And
lastly,
you
know
these
features
are
experimental
and
we're
definitely
looking
for
feedback
and
for
design
Partners,
so
leave
any
feedback
that
you
have
in
the
slack
feel
free
to
reach
out
to
me
specifically
or
in
the
Dexter
feedback.
Channel
yeah.
That's
all
I
have
I'll
hand
it
back
to
you
on.
A
G
A
G
Would
just
give
a
shout
out
to
Pete
you
mentioned
very
quickly
the
the
crash
course
that
he
created
and
if
there's
anybody
that
people
in
the
community
bump
into
who
you
know,
want
a
very
quick
way
of
getting
up
on
money
on
Dexter
just
to
try
it
out
in
a
sandbox.
So
far,
I
found
that
some
really
good
tool
just
to
send
over
to
people
and
say
look
just
follow
this
guide.
We'll
get
you
up
in
money
and
you'll
get
a
strong
sense
of
it
really
quickly
so
enjoyed
that.