►
From YouTube: Dagster Hot Takes: Debugging
Description
Logging is great, but it isn't the only tool. With dagster, data engineers can also tap into debuggers, unit tests, and local development best practices.
Prefer to read: https://gist.github.com/slopp/7c64973c70b12ffbd506d4e64d80d65e
Resources posted below, or star us on GitHub to get started.
- Star Us: https://gitub.com/dagster-io/dagster
- Example Project: https://gist.github.com/slopp/359ba324a4d6ccfb3bf23d0e9554c080
A
Hey
everyone:
this
is
Sean
with
another
dagster
hot.
Take
today,
I
want
to
talk
about
debugging
a
lot
of
times
in
data
engineering.
We
rely
on
extensive
logging
so
that,
as
things
occur
in
production,
we
have
visibility
into
what's
going
on,
and
while
that's
great
and
I'm
a
big
fan
of
observability
I
think
it
can
be
frustrating
to
only
catch
errors
in
production
through
those
logs
and
historically.
This
is
because
the
code
that
we're
writing
is
heavily
tied
to
those
production
systems.
A
So
what
we
have
here
is
a
pretty
simple
Dexter
project
and
that
project
consists
of
two
assets.
Our
first
asset
is
using
an
API
to
go
grab
forecasts
for
snowfall
at
a
ski
area.
Then
our
second
asset
takes
that
snow
forecast
as
an
input
and
figures
out.
If
you're
excited
about
skiing
fresh
powder,
what
day
would
be
the
the
best
day
to
go
skiing,
and
so,
if
I
wanted
to
test
these
assets,
maybe,
for
example,
I
want
to
see
what
this
snow
forecast
variable
is
all
about.
A
The
way
that
I
can
do
that
inside
of
dagster
is
by
running
these
python
functions,
which
are
all
that
assets
are
decorated,
python
functions
locally
and
so
I
can
create
a
simple
file
test,
stop
hi.
That
is
going
to
invoke
those
asset
functions
now,
there's
one
other
thing
that
that
we
need
to
do
here
these
assets.
If
we
go
back
to
our
project,
rely
on
resource,
and
so
here
we
have
an
API
that
we
want
to
make
pluggable
between
different
environments
and
so
we're
representing
it
as
a
dagster
resource.
A
That
resource
is
in
turn
made
available
to
the
asset
function
through
this
special
object
called
a
context,
and
so
in
dagster,
when
I
want
to
run
things
locally,
I
can
create
that
context.
Using
this
function
build
hop
context,
and
this
is
really
powerful,
because
it
means
that
you
can
plug
and
play
different
resources.
For
example,
if
I
wanted
to
do
a
unit
test,
I
might
have
a
mock
resource
here
instead
of
the
real
production
resource.
A
So
let's
go
ahead
and
run
this
and
I'm
going
to
use
the
debugger
in
vs
code.
So
I've
created
a
break
point
by
this
forecast
object
because
that's
what
I
want
to
have
better
visibility
into
I'll,
go
ahead
and
start
a
debugging
session,
and
now
my
debugger
is
stopped
here
at
this
forecast
line,
and
so,
if
I
really
want
to
get
an
understanding
of
what
this
asset
code
is
and
maybe
do
some
debugging
I
could
step
in
to
this
forecast
step
and
now
I'm.
Looking
at
that
snow
forecast,
asset
I
could
add
an
additional
breakpoint.
A
So
here
I'll
add
a
break
point
after
the
API
is
called
I.
Can
click
play
that'll
hit?
My
new
breakpoint
and
I
can
similarly
step
in
further
and
now
see.
What's
going
on
inside
of
this
API
resource,
I'll,
add
one
more
break
point
here:
click
play
and
now
I
have
the
ability
to
inspect
exactly
what
the
result
of
this
call
is
going
to
be,
and
so,
as
an
example,
I
could
take
this
line
of
code
and
in
my
debug
console
run
it
and
now
I
can
see
exactly
what
this
API
is.
A
A
So
that's
you
know
one
way
that
you
can
run
things
locally.
Using
dagster
can
I
go
ahead
and
stop
the
debugger.
A
benefit
to
this
approach
is
that
I
now
am
very
close
to
having
a
unit
test
written.
So
if
I
go
back
to
my
test
file
here,
maybe
I'll
change.
This
return
statement
to
an
assert
statement.
A
I'll
comment
out
our
invocation
here
and
now.
I
have
something
that's
very
compatible
with
pi
test.
So,
for
example,
I
could
run
a
unit
test
with
in
my
local
environment
or
within
CI
CD,
to
make
sure
in
this
case
that
those
assets
are
runnable
and
again
that's
running
with
whatever
resource
definition.
I
want
here
so
in
a
unit
test.
I
might
use
this
with
a
mock
resource.
In
our
case,
this
is
all
pretty
simple,
so
I'm
just
using
the
the
real
resource
for
testing
in
total.
A
That's
going
to
give
me
a
lot
of
confidence
and
it's
going
to
help
me
iterate
really
quickly.
Now
that
I
am
excited
about
my
code
and
I
understand
what
that
forecast
object,
looks
like
maybe
I'll
go
ahead
and
run
the
project
in
dagster,
so
I'm
going
to
use,
dag
it
to
run
this
locally
and
if
we
come
into
dijster.
I
can
now
take
that
final
step
in
local
development,
where
I
use
dagster
to
materialize
these
assets
by
launching
a
run.