►
Description
If you are looking to get up and running with Dagster in 10 minutes or less, this is a good place to start. Buckle up.
This video is a companion piece to the tutorial found at https://dagster.io/blog/dagster-crash-course-oct-2022
Sourcecode is found on GitHub: https://github.com/petehunt/dagster-github-stars-example
You can launch an instance and follow along on GitPod: https://gitpod.io/#https://github.com/petehunt/dagster-github-stars-example
A
A
If
you
want
to
just
skip
straight
to
the
code,
the
completed
code
is
available
here
at
this
GitHub
repo
and
finally,
as
we
go
along
I'm
going
to
be
using
an
in-browser,
editor
called
gitpod
and
starting
from
scratch
and
then
building
it
up
from
there.
So
you
can
just
go
here
to
get
started
with
that.
It's
free
and
it
makes
getting
spun
up
with
a
python
environment,
very
easy
just
to
set
the
context
here,
we're
going
to
create
this
report
and
store
it
in
GitHub
gist.
A
So
to
get
started,
I've
got
this
Cloud
development
environment.
Here
it's
using
a
service
called
gitpod,
there's
a
link
in
the
in
the
blog
post.
It's
a
free,
Cloud
development
environment
that
guarantees
a
pretty
stable
and
specific
python
version.
You
can
use
whatever
python
environment
you
want.
Dagster
works
on
on.
You
know
Mac
Linux
and
windows,
but
I'm
just
going
to
use
this
for
consistency.
A
So
the
first
thing
we're
going
to
do
is
PIP
install
dagster
this
installs
our
command
line
tools
that
help
you
scaffold
out
a
project
so
we're
just
going
to
use
the
the
default
simple
project,
but
there
are
a
number
of
different
templates
that
you
can
use
by
accessing
the
help
command,
but
I'm
just
going
to
save
Dexter
project
scaffold
name.
My
dagster
project.
A
And
you
can
see
here
that
we've
created
a
project
in
this
in
this
workspace,
so
if
I
CD
my
Dexter
project
I'm
going
to
install
the
dependencies
that
this
example
project
needs.
So
this
is
just
how
you
do
it,
while
it
installs
we'll
give
you
a
little
tour
of
of.
What's
in
here,
we
have
a
little
readme
describing
the
project
linking
to
the
documentation.
We've
got
our
normal
kind
of
python
set
of
dot
Pi,
where
we
list
all
the
dependencies
that
we
need.
A
This
workspace.yaml
tells
dagster
when
we
run
it
locally
where
to
find
the
code.
So
dagster
looks
for
this
workspace.yaml
file
to
figure
out
what
code
load
and
then
we've
got
my
Dexter
project,
which
contains
you
know
the
the
source
code
for
our
project,
we'll
be
doing
most
of
our
work
in
there,
and
then
we
have
the
my
dagster
project
tests
package,
which
contains
all
the
unit
tests
for
our
project,
so
I'm
going
to
start
dagster
using
this
command,
dag
it
there's
actually
two
components
to
dagster.
A
There
is
the
UI
and
then
there's
the
Daemon.
That
runs
the
schedules,
because
we're
not
going
to
be
using
schedules
right
now,
I'm
just
going
to
launch
the
UI,
and
so
you
can
see
here
that
we've
got
this
this
empty
UI,
because
we
haven't
done
anything
yet,
but
you
just
run
that
one
command
dag
it.
It
looks
at
that
workspace.eml
file
and
it
loads
up
your
UI.
A
So,
like
I
said
we
are
building
that
GitHub
Stars
dashboard,
so
we
need
a
number
of
dependencies
in
order
to
do
that.
So
in
order,
so
the
way
we
add
dependencies
is
it's
like
any
other
python
project.
We
just
update.
This
install
requires
part
of
the
setup.pi,
and
then
we
rerun
the
install
steps.
This
pip
install
Dash
e
command,
and
so
this
is
going
to
install
Pi
GitHub.
So
we
can
access
the
GitHub
API
I'm
going
to
install
matplotlib,
which
will
help
us
visualize.
A
So
the
first
thing
we're
going
to
do
is
we're
going
to
want
to
fetch
the
raw
data
from
GitHub,
so
this
is
going
to
create
you
know,
basically,
the
raw
response
from
the
GitHub
API,
so
we're
going
to
use
software-defined
assets
to
do
that,
we're
going
to
use
software-defined
assets
to
begin
to
build
our
application,
so
I
I
just
copied
and
pasted
this
in
before
the
cut.
This
is
our
example.
A
A
Obviously,
inlining
a
token
or
any
sort
of
Secret
In
Your
source
code
is
a
really
bad
idea.
We
will
fix
that
by
the
end
of
this
tutorial,
but
for
now
we're
just
gonna
put
that
in
there
and
we're
going
to
create
what's
called
a
software
defined
asset.
Software
defined
asset
is
a
function
that
returns
some
data.
That
represents
an
asset,
a
data
asset
in
your
graph.
So
this
could
be
a
machine
learning
model,
a
report
or
a
database
table
in
this
case
we're
just
returning
the
response
from
the
GitHub
API.
A
So
we
instantiate
the
pi
GitHub
client.
We
pass
it
the
access
token
we
do
a
little
bit
of
function,
calls
to
get
the
star
gazers
with
the
dates.
This
is
effectively
like
the
username
and
then
the
date
that
they
started
the
repo
and
that
the
exact
time
stamp
that
they
start
the
repo
and
marking
it
with
an
asset.
Decorator
indicates
to
dagster
that
this
is
a
software-defined
asset.
A
So
the
next
thing
we're
going
to
need
to
do
is
now
that
we
have
that
raw
API
response
from
GitHub
we're
going
to
need
to
transform
that
into
week
by
week
counts.
So
we
need
to
go
from
these
pairs
of
timestamps
and
usernames
to
the
number
of
unique
users
that
have
starred
the
repo
in
a
given
week
and
so
I'm
going
to
just
paste
some
of
this
code
in
from
the
blog
post,
you
can
follow,
follow
along
with
a
blog
post.
A
If
you
like,
and
so
right
here,
I'll,
take
you
through
how
this
works.
Again.
We
have
a
second
software-defined
asset,
which
is
called
GitHub
stargazers
by
week.
This
takes
a
parameter
here
called
GitHub,
star
gazers.
What
this
actually
is
is
a
it's
got
a
special
name
because
it
references
this
name
right
here
now,
dagster
via
the
magic
of
software-defined
Assets
in
this
asset.
Decorator
knows
how
to
match
these
two
up,
and
so
this
basically
declares
a
data
dependency
between
the
GitHub
Star
gazers
by
week
and
the
get
up
stargazers
asset.
A
So
what
dagster
will
do
is
it
will
know
to
materialize
the
get
up
stargazer's
asset
before
materializing
Get
Up
stargazers
by
week?
So
now
that
we
have
that
data,
we
iterate
through
it
here
and
we
create
a
new
data
frame.
Where
you
know
we,
we
basically
create
one
row
for
every
user
and
when
they
start
it,
except
we
convert
the
timestamp
from
a
the
exact
time
stamp
to
just
the
the
start
of
the
week,
and
then
we
will
aggregate
by
the
start
of
the
week.
A
So
you
can
see
here
we
call
Group
by
week
which
Aggregates
everything
into
week
by
week
Aggregates.
We
call
count
which
counts
within
the
week
and
then
we
sort
chronologically
by
the
week.
So
then
we
get
an
ordered
a
data
frame
the
start
of
the
week
and
then
the
number
of
users
that
started
during
that
week.
A
If
you
have
questions
about
how
this
works
check
out
the
pandas
documentation,
so
the
next
thing
we
need
to
do
is
go
from
this
data
frame
to
some
sort
of
visualization
Jupiter
notebooks
are
a
really
common
way
to
do
it
normally.
I.
Would
you
know
open
up
Jupiter
to
develop
the
notebook,
but
there's
a
little
library
that
makes
it
easier.
A
We
call
this
execute
pre-processor.preprocess.
This
is
something
that
we've
imported
from
the
Jupiter
Library,
which
will
basically
execute
the
notebook
and
put
the
results
into
it
and
then,
finally,
we
call
mbformat.write
which
will
write
out
the
notebook
as
a
string
that
can
be
read
by
Jupiter
or
any
other
service
that
supports
notebooks.
A
A
So
you
can
see
here:
we've
defined
another
software-defined
asset
with
this
asset,
decorator
GitHub
Stars
notebook
just
it
takes
in
the
GitHub
star's
notebook-
and
this
is
we
don't
need
this
right
now
it
takes
in
the
GitHub
Stars
notebook.
It
calls
the
GitHub
API.
It
tells
it
to
create
a
gist
and
then
uploads
the
contents
of
that
notebook
as
a
file
attached
to
the
gist,
and
then
we
just
log
out
the
the
URL,
and
so
we
we've
basically
created
all
of
our
all
of
our
software-defined
assets.
A
A
So
what
this
is
doing
right
now
is.
This
is
fetching
all
of
the
Star
gazers
from
GitHub.
This
is
actually
quite
a
long
operation
because
it
is
fetching
all
of
the
stargazers
from
the
beginning
of
time,
which
is
a
quite
expensive
operation.
You
have
to
do
multiple
calls
to
the
GitHub
API
in
order
to
do
it
and,
as
you
can
see,
it
takes
a
long
time
one
of
the
advantages
to
modeling
your
computation
as
a
dag
of
assets.
A
The
way
that
that
Daxter
does
is
that
we
can
cache
this
computation
and
reuse
it
in
the
future.
So,
for
example,
if
we
want
to
iterate
on
how
the
notebook
works
or
how
we're
transforming
the
data,
basically
The
Core
Business
logic,
we
don't
have
to
do
that
fetch
again.
Dagster
uses
a
system
called
I
o
managers
and
it
stores
that
in
persistent
storage,
in
this
case
it's
stored
on
on
my
local
disk.
A
But
but
you
know
in
the
future,
you
know,
if
you
kind
of
go
to
production,
you
can
use
S3,
it
will
be
stored
on
you
know
in
a
in
a
more
kind
of
production,
ready
blob
store.
So,
as
you
can
see,
I
made
an
edit
to
my
code
and
I'm
missing
my
context.
Variable
so
I'm
gonna.
Add
that
back
so
context
is
not
the
name
of
an
asset
context
is
actually
a
kind
of
a
special
magic
context.
A
A
So
you
know,
one
of
the
things
to
take
away
from
this
here
is
that
fixing
bugs
can
be
really
fast
when
you
model
your
your
computation
this
way,
and
so,
if
I
take
a
look
here,
I
have
a
brand
new
visualization
uploaded
to
GitHub
and
I
can
go
and
share
this.
With
with
any
of
my
stakeholders,
looks
like
the
stars
are
going
up.
That's
great
one
thing
to
to
kind
of
note
here
that
I
didn't
cover
is
how
this
notebook
is
created.
A
A
The
last
thing
I
want
to
show
you
is
how
to
add
a
schedule,
and
so
you
know
we.
We
basically
have
created
a
one-off
job
at
this
point
so
effectively
you
can
go
into
dagster
click
on
the
launch,
run
button
or
click
on
the
materialize
button
and
do
a
one-off
run,
but
really
when
you
get
to
production,
you
want
to.
You
really
want
to
put
things
on
a
schedule.
A
Let's
all
show
you
how
to
do
that.
We
open
up
this
repository
here
and
I'm
going
to
make
an
edit
here.
So
you
can
see
that
right
now,
our
repository,
which
is
kind
of
dagster's
word
for
project.
In
many
ways
it
just
contains
all
the
assets
from
our
assets
package.
I'm,
going
to
put
some
additional
things
in
here.
A
A
So
we
basically
say
hey:
we
want
to
have
a
job
called
daily
refresh
and
it
will
refresh
all
of
the
Assets
in
the
project.
That's
what
the
star
means
and
we
will
put
it
on
a
daily
schedule
and
then
we
simply
add
the
job
and
the
schedule
to
our
project,
and
we
can
take
a
look
at
that
in
the
dagster
UI.
So
I
go
to
my
workspace.
A
A
A
So
you
know
that's
why
we
have
that
warning
all
right,
and
there
is
a
couple
problems
now
with
this
project,
even
though
it
works,
we
are
able
to
build
a
full
ETL
pipeline
from
GitHub
to
a
visualization
and
run
it
on
a
schedule.
There's
still
some
things
we
need
to
do
before.
We
can
consider
this
project
production
ready.
A
A
A
It's
going
to
contain
our
example
resource,
so
a
resource
is
basically
it's
usually
like
a
client
that
talks
to
an
external
system.
So
in
our
case,
it's
going
to
be
our
GitHub
client
we're
going
to
abstract
that
away
into
something
called
a
resource
and
what
the
resource
lets
us
do
is.
It
can
be
configured
separately
from
the
rest
of
the
application
and
the
assets
will
depend
on
the
GitHub
resource
rather
than
the
GitHub
API
itself.
So
we
can
specify
you
know
different
access
tokens
to
different
assets.
A
A
A
A
You
know:
dagster
object
called
string
source
which
has
some
extra
features,
including
reading
values
from
the
environment
as
or
reading
from
environment
variables,
which,
which
is
very
useful
for
secrets.
So
we're
actually
going
to
use
string
Source
here
now
that
we've
added
the
resource
or
that
we've
created
the
resource.
We
have
to
actually
add
it
to
our
project,
so
I'm
going
to
go
back
into
our
Repository
I'm,
going
to
import
the
resource
and
then
I'm
going
to
add
it
to
our
project
here.
A
So
the
way
that
we
do,
that
is,
we
use
this
function
called
with
resources
which
I
will
import
up
here
and
we
say
with
resources,
and
then
that
takes
in
definitions
and
then
the
resource
definition.
So
the
first
thing
we'll
do
is
just
pass
in
our
assets.
What
this
basically
says
is
it'll
give
the
resources
to
all
of
our
Assets
in
our
project,
and
then
we
provide
a
resource
name.
So
in
this
case,
we're
going
to
call
it
GitHub
API
and
then
it's
going
to
take
in
the
GitHub
API
with
a
configuration.
A
A
You
can
actually
reuse
that
definition
in
multiple
contexts
in
your
application
and
so
there's
a
resource
key,
which
is
kind
of
like
the
instance
of
that
resource,
and
so
the
way
we
create
an
instance.
The
resource
is,
we
call
configured
which
basically
passes
a
configuration
to
it,
and
this
can
also
be
provided
as
an
external
configuration
file.
A
But
for
this
we're
going
to
include
in
the
code-
and
we
can
say
access
token,
which
was
the
the
name
of
the
the
field
in
the
config
schema
that
we
defined
over
here
in
resources,
I'm,
going
to
show
you
this
this
feature
of
string
Source
now
where
we
can
actually
pass
an
object
with
the
key
of
N
and
then
the
name
of
the
environment.
Variable
foreign.
A
A
Okay,
that
looks
good
up
next.
We
have
to
actually
use
this
resource
from
the
the
assets
themselves,
and
so,
if
we
go
over
here,
we
really
use
this
GitHub
client
in
two
places.
The
first
is
in
this
GitHub
star,
gazers
asset,
and
the
second
is
in
this:
get
up:
Stars
notebook
just
asset,
so
one
reads
from
GitHub:
the
other
one
writes
to
GitHub.
A
A
Right
here
we
provide
required
resource
Keys.
It's
basically
defines
a
dependency
on
that
resource
called
GitHub
API,
and
what
this
means
is
that
the
resource
is
now
available
to
this
resource
to
this
asset
and
we
can
get
at
it
by
just
saying
context.resources.github,
API
and
passing
in
or
taking
the
context
as
a
parameter.
I
talked
about
this
before.
A
A
And
it
looks
like
this
is
fetching
all
the
GitHub
stars
from
the
API
now,
because
we've
restarted
the
service
and
I'm
using
the
developer
mode.
We
lose
our
cache
in
between
restarts
of
of
the
dagged
service,
but
you
can
configure
it
with
a
custom.
What
we
call
I
o
manager
in
order
to
persist
that
data
somewhere
in
between
restarts
so,
for
example,
S3.
A
But
as
that
runs,
you
know
it's
taking
a
long
time,
so
I'm
pretty
sure
that
it's
accessing
the
GitHub
API
correctly.
So
this
is
this
is
huge:
we've
basically
just
gotten
rid
of
that
secret
from
our
source
code
and
we're
reading
it
from
the
environment
that
can
be
provided
using
any
any
sort
of
automation
that
you
want.
A
A
I'm,
going
to
open
up
this
test,
assets,
dot,
Pi
file
and
I'm,
going
to
bring
in
some
some
test
code.
Here
first
thing:
I'm
going
to
do
is
bring
in
a
bunch
of
imports,
so
I'm
going
to
import
the
software
defined
assets
from
our
project,
I'm,
going
to
import
a
python
utility
called
Magic
Mock
and
we're
going
to
use
a
lot
of
dagster
test
utility
called
materialize
to
memory,
and
then
some
helper
functions
so
pandas
to
create
data
frames
and
then
date
time
to
help
us
create
some
test
data.
A
A
You
know
three
users
starring
this
repo,
two
of
them
on
the
same
day
in
January
of
2021
and
another
one
in
February,
then
we
are
going
to
create
a
mock,
GitHub
API,
so
using
the
magic
mock
Library
we
instantiate
it
and
then
you
know
I
would
definitely
check
out
the
documentation
for
for
magic
mock
because
You
Know
It.
It
has
some
subtleties.
A
Let
me
just
make
sure
I've
done
this.
Oh
yeah,
okay,
so
that's
correct,
so
so
just
just
to
hand
me
over
this
a
little
bit.
This
mocks
out.
This
call
from
our
asset
that
calls
get
stargazes
with
dates
and
it
mocks
it
out
to
return
effectively
this
this
data
set.
So
we're
simulating
that
the
GitHub
API
is
going
to
return
that
data.
A
Another
thing
we
need
to
Mark
out
is
the
right
path,
so
we've
mocked
out
the
read
path
from
GitHub,
but
let's
also
mock
out
the
right
path
and
the
reason
we're
mocking
this
out
is
so
that
our
test
can
run
and
it's
fast
and
it
doesn't
talk
to
any
external
service
and
it
also
doesn't
trigger
any
effects
in
the
real
world.
So
it
doesn't.
If
you
know
you
can
imagine
these
apis
could
cost
money
or
have
quotas
associated
with
them,
and
you
don't
want
your
tests
burning
through
that.
A
So
this
mocks
out,
you
know
the
create
just
function
to
return
a
fixed
URL.
It's
not
actually,
you
know
upload
a
real
gist
and
then
finally,
we're
going
to
actually
materialize
our
Assets
in
the
test.
So
we
have
this
function,
materialized
to
memory.
We
pass
in
the
assets
that
we
want
to
materialize
and
we
can
also
override
you
know
specific
resources
that
we
would
like
to
use
or
provide
them.
So
this
is
a
resource
key,
and
this
is
the
resource
definition
and
then
the
last
step
is.
A
We
want
to
actually
write
some
assertions
here
and
I
will
paste
these
in
the
first
one:
just
checks
that
there
was
actually
a
successful
run
and
that
the
Run
didn't
throw
any
errors.
The
second
is
we
look
at
the
output
of
the
GitHub
Star
gazers
by
weak
asset.
We
do
a
little
bit
of
pandas
magic
to
compare
it
with
the
expected
data.
A
We've
got
that
here
and
you
can
look
at
the
original
mock
data.
If
you
don't
believe
me,
then
we
also
want
to
assert
that
we
have
actually
called
that
create
just
function
and
that
we
are
returning
the
URL
correctly.
So
we
do
that
here
and
then
finally,
I
added
a
little
smoke
test,
just
to
make
sure
that
the
GitHub
star's
notebook
content
like
the
notebook
file
that
was
written
out
contained.
A
So
if
we
want
to
run
this
test,
we
say
Pi
test
s,
my
dagster
project
tests,
the
dash
s
is
kind
of
running
it
in
verbose
mode.
So
we
see
all
the
log
output
and
any
print
statements
that
we
might
put
in
there,
and
you
can
see
here
that
our
test
passed
in
in
a
pretty
short
amount
of
time.
So
it
wasn't
pulling
all
that
data
from
GitHub.
A
It
wasn't
talking
to
any
external
system,
it's
basically
just
good
old-fashioned,
in-memory
testing
and
it's
always
good
practice
to
try
to
break
your
test
and
make
sure
that
your
test
is
actually
testing
something.
So
this
is
changing
it
to
assert
that
we've
actually
made
a
public
gist
instead
of
a
private
gist,
and
you
should
be
able
to
see
that
this
test
will
break
yep.
It
breaks
and
then
we
can
fix
it
again
so
anyway,
this
was
a
a
crash
course
into
building
an
ETL
Pipeline
with
dagster.