►
Description
Do you want to know if DAGSTER.IO, an open source python data pipeline orchestration library is the right ETL data pipeline orchestration tool for you? Then watch this video.
I decided to demo open source python data orchestration tool for scientists, dagster.io because I wanted to know, if it is worth to migrate my python data flows into DAG based data pipeline tool.
I was after a tool with in-build scheduler, a nice dashboard and a easy to install documentation. So is dagster.io superior to Apache Airflow or prefect.io? We will see. So far my impressions are positive.
Buy me a coffee
○ https://buymeacoffee.com/vietphananh
Also make sure you subscribe to my channel!
MY DEVELOPMENT STACK
○ Linode VPS - https://www.linode.com/?r=cb6b809841dbb52fd72839e010bc42e417b3ff81
VIDEO PRODUCTION GEAR USED:
○ My camera - Sony A6400 - https://amzn.to/3ikLQbc
○ My main lens - Sigma 16mm f1.4 - https://amzn.to/36xHKtZ
○ My B camera - Sony ZV-1 - https://amzn.to/3IoHxpV
○ My C camera - iPhone 12 Pro - https://amzn.to/3KU0l1W
○ My lav mic - Rode SmartLav+ - https://amzn.to/3Jvr05c
○ Joby GripTight tripod - https://amzn.to/3wnxLlx
○ 13’ MacBook Pro - https://amzn.to/3tlZNfr
○ Elgato Green Screen - https://amzn.to/3imGrAB
○ Elgato Cam Link 4K - https://amzn.to/36eOmNV
○ Neewer RGB Lights - https://amzn.to/3IkzSct
COME SAY HI!
Blog:
○ https://vphventures.com/
Instagram:
○ https://www.instagram.com/chillifunk/
Twitter:
○ https://twitter.com/vphventures
MY PRODUCTIVITY APP
Join TheHustle:
○ https://thehustle.app/
MY MUSIC CHANNEL
Good House Music:
○ https://www.youtube.com/channel/UC4Dv6nLAdVPWhmKSthMYMXQ
DISCLAIMER: Links included in this description might be affiliate links. If you purchase a product or service with the links that I provide I may receive a small commission. There is no additional charge to you! Thank you for supporting me so I can continue to provide you with free content each week!
A
A
A
So,
let's
continue
with
our
dexter
demo.
So,
according
to
the
github
dextre
is
a
core
programming
model
in
abstract,
stack,
stateless,
single
node,
single
process
and
multi-processed
execution
engines
and
deget.
On
the
other
hand,
is
a
user
interface
for
developing
and
operating
dexter
pipelines,
including
a
deck
browser
and
a
live
execution
interface,
and
to
try
the
extra.
They
also
include
a
simple
hello,
dextro
tutorial,
so
I
will
open
a
visual
studio
code
and
let's
try
our
first
texture
demo.
A
And
in
this
file
we
will
import,
execute
pipeline
pipeline
and
solid
functions
from
dexter
library.
Then
we
will
define
our
first
dexter
function.
They
call
them
solids.
So
here
we
have
a
simple
function,
get
name
which
returns
a
simple
string
texture
and
we
will
transform
this
simple
function
into
solid
by
the
decorator
solid.
Next
we
will
create
a
simple
function,
hello,
which
will
take
context
and
a
name
with
the
type
string
as
input
parameters.
Of
course,
the
function
is
decorated
with
our
solid
decorator
and
what
functions
do?
A
Is
it
locks
a
simple
hello
name
where
I
guess
the
name
variable
will
be
taken
from
get
name
function?
And
finally,
since
we
have
our
solids,
we
will
create
our
first
pipeline
called
hello
pipeline,
which
is
also
a
simple
python
function.
But
by
using
a
simple
dextro
decorator,
we
transform
the
simple
function
into
the
texture
pipeline
and
in
this
pipeline
we
called
hello
function
and
inside
the
hello
function
we
called
getlang
function.
Then
we
can
execute
our
pipeline
with
the
command
dextra
pipeline,
execute
hellodextro.pi
and
it
seems
like
dextra
rented,
a
lot
of
locks.
A
So
first
it
started
the
pipeline.
It
executed.
Hellopipeline.Pi
it
spawned
its
own
process
for
its
engine
event
and
it
started
the
get
name,
function
and
returned
the
result
of
type
na.
It
stored
the
return
value
into
an
object,
and
it
ended
this
step
with
the
success.
Then
after
it
finished,
get
name
function,
it
started
with
the
execution
of
hello,
solid
and,
according
to
the
logs,
it
retrieved
the
return
value
of
the
name
from
the
object
store
using
pickle
and
it
inputted
the
value
into
hello
function,
as
you
can
see
it.
A
Outputted,
hello,
dexter
and
after
this
step
the
whole
pipeline
finished
the
execution
with
the
status
pipeline
success.
I
am
quite
surprised
that
dexter
provides
a
lot
of
debug
information
in
the
console
and
it
is
very
clear.
What's
going
on
now,
let's
look
at
its
user
interface
digit.
According
to
documentation,
we
can
start
deget
user
interface
with
the
command
digit,
hello,
dexter,
dot,
pi,
and
here
you
can
see
that
digit
server
just
started
and
it
is
accessible
on
this
url
http
localhostport3000.
A
So
let's
copy
paste
the
url
and
check
out
in
the
browser
wow
it
just
works.
I
am
very
impressed
because,
unlike
the
prefect
user
interface,
the
dexter
dashboard
just
works,
I
didn't
have
to
register
the
flow.
I
didn't
have
to
write
multiple
lines
of
code
and
install
docker
with
the
dexter
I
just
executed
one
command
and
the
whole
user
interface
load
without
any
issues.
So
let's
look
closely
what
the
user
interface
offers.
A
So
initially
they
get
loaded
the
definition
window,
where
we
see
our
hello
dexter
pipeline
and
if
I
click
on
the
one
of
the
objects,
I
see
that
it
provides
a
complete
information
about
this
function,
so
the
function
gets
named,
outputs
the
results
and
the
output
goes
into
hello
function,
input
parameter
called
name,
but
because
we
didn't
specify
any
types
for
the
return
variable.
The
results
can
contain
variable
of
any
type.
Okay.
What
about
hello
function?
A
It
takes
the
get
name,
result
in
the
form
of
input,
variable
name,
and
it
transforms
the
type
of
the
input
variable
into
string
and
because
it
doesn't
output,
anything
the
results
can
be
of
any
type
that
will
be
the
pipeline
definition.
What
about
the
pipeline
overview
here
is
just
an
overview
of
the
pipeline
graph
and
it
displays
basic
information
about
pipeline
schedules
and
the
recent
pipeline
runs
now
in
the
pipeline
playgrounds.
I
see
that
I
can
execute
the
function.
So
let's
do
that.
A
And
it
just
executed
our
hello
dexter
pipeline.
It
shows
very
conveniently
in
the
timeline
the
order
of
the
execution.
So
first
the
get
name
function
was
executed.
It
outputted
a
result
variable
the
variable
was
saved
into
the
object
store
in
the
form
of
pickle.
Apparently,
the
pickle
is
saved
into
intermediate:
slash,
get
name,
dot,
compute,
slash
results,
folder
and
the
get
name
function
finished
with
steps
finished
status
and
the
speed
of
execution
was
only
45
milliseconds.
A
Next,
the
hello
compute
was
called
and
the
function
start
execution
by
retrieving
the
saved
object
from
the
results
folder
and
it
passed
into
the
input
parameter.
Then
it's
printed
the
whole
string
into
the
console
because
we
called
context.lock,
so
the
whole
variable
was
printed
into
the
console
and
it
seems
again
that
it
stored
the
value
in
the
results
folder
again,
even
though
the
function
itself
doesn't
return
any
value
whatsoever.
This
debug
window
also
allows
you
to
filter
the
messages
by
logging
types.
A
So
if
some
critical
error
appears,
then
you
can
hide
everything
else
and
just
displace
the
critical
errors
it
seems
like.
It
can
also
show
you
the
slow,
wet
path
which
can
be
helpful
if
you
have
a
very
complex
pipeline
with
mutable
trees
and
multiple
execution
scenarios,
and
I
guess
in
that
case
it
can
show
you
the
execution
path
that
takes
the
most
time.
Okay,
what
else?
If
I
click
on
the
logo,
what
happens
nothing
it
seems
like
this
is
just
an
overview
of
the
dexer
configuration.
A
I
hope
that
most
of
these
settings
can
be
changed,
because
I'm
not
comfortable
storing
too
much
data
in
my
windows,
hard
disk.
Otherwise
it
seems
like
the
dextra
uses,
sk,
light
storage
for
storing
the
objects,
and
it
also
has
its
own
scheduler.
Alright.
So
let's
leave
this
configuration
page
for
now
and
let's
check
out
the
runs
page
and
on
this
page
I
see
this
is
just
a
simple
list
of
our
pipelines
and
on
this
page
we
can
either
view
the
configuration
re-execute
this
pipeline
or
open
the
pipeline
in
the
playground.
A
We
can
even
download
the
file
for
debugging.
It's
very
convenient
right.
What
about
assets?
Page
assets
page
is
an
asset
aware
event,
storage,
for
example,
postgres,
even
log
storage.
The
storage
must
be
configured
in
order
to
use
any
asset
based
features.
You
can
configure
this
on
your
instance
through
dexter.yamo,
apparently
in
dextre,
an
asset
is
the
abstract
representation
of
data
artifact
produced
and
persisted
by
solid.
These
assets
are
generate
processes
to
storage,
to
dexter,
either
database
table
or
analytics
dashboard
and
a
side
effect
of
the
solid
computation.
A
The
act
of
persisting
an
asset
is
called
materialization
and
solid
can
express
materialization
by
yielding
an
asset
materialization
event.
And
if
I
look
at
the
examples
it
seems
like,
I
can
store
pandas
data
frames
into
the
postgres
dexter
storage,
which
might
be
useful
if
you
want
to
use
the
materialized
data
later
in
the
execution
pipeline.
And,
lastly,
let's
check
the
scheduler
page,
and
it
shows
me
the
message
that
dexter,
instant
does
not
have
a
scheduler
configured.
A
schedule
must
be
configured
on
the
distance
to
run
schedules.
Therefore,
the
schedules
below
are
not
currently
running.
A
You
can
configure
a
schedule
on
the
instant
through
dextro
ymo
file
in
dexter
home,
see
the
instant
configuration
documentation
for
more
information.
Well,
I'm
very
curious
about
this
functionality
because
if
I
compare
it
to
prefect-
and
if
I
remember
it
correctly-
prefect.io
has
some
kind
of
a
simple
scheduling
functionality,
but
still
you
would
have
to
create
your
own
chrome.
That
would
run
the
scheduler
itself.
So
I'm
very
curious
if
dextre
has
a
more
fully
featured
scheduler
than
prefect
dot
io.
A
So,
let's
check
the
instance
configuration
documentation.
So
first
I
need
to
figure
out.
Where
is
the
dextre
hall?
Okay,
after
a
bit
of
googling?
I
guess
it
is
some
kind
of
a
environmental
variable
in
the
windows
that
I
have
to
set.
So
I
will
open
the
individual
variable
system
window
and
I
will
create
a
new
variable
called
texture
home
and
I
will
give
it
an
absolute
part
to
a
new
folder
that
I
will
create.
A
I
will
click,
ok,
save
the
variable,
and
now
I
have
to
create
a
yaml
file
inside
the
scheduler
folder
that
I
just
created.
So
I
will
fire
up
the
visual
studio
code
and
I
will
create
a
new
dextr
dot
yaml
file
and
in
this
file
I
will
copy
paste.
The
settings
that
is
recommended
by
the
tutorial
seems
like
it's
just
a
bunch
of
login
informations
for
postgres
database,
not
only
for
run
storage,
but
also
for
even
lock
storage
and
lastly,
for
postgres
schedule
storage.
A
A
Okay,
guys,
I'm
back
after
a
few
hours
of
fiddling
with
the
dexter
home
system
environment.
As
a
last
resort.
I
tried
to
restart
my
computer
and
what
happened
is
that
I
made
the
progress.
I
realized
that
when
I
started
a
virtual
environment
again
it
picked
up
the
last
system
environment
that
I
have
saved
before
and,
as
you
remember,
I
put
the
dollar
sign
dexter
home
as
a
folder
for
my
yaml
configuration
and
after
trying
to
start
dig
it
again,
I
got
the
error
that
dextro.jmo
is
not
directory
or
it
does
not
exist.
A
So
what
does
it
mean?
Is
that
for
one
I
don't
have
to
put
the
dollar
sign
into
dollar?
Sign
they
get
home.
If
I
leave
the
dollar
sign
in
the
folder
name,
dexter
won't
pick
up
the
correct
path,
so
I
removed
the
dollar
sign
and
I
realized
that
it
is
wrong
to
put
the
dexter
dot
gamma
into
the
path
system,
environment
variable
and
when
I
tried
to
restart
the
terminal
to
pick
up
the
environment
variable
again
and
firing
up
the
digit
showed
me
yet
another
error.
A
The
failure
condition
is
following:
could
not
import
module,
dextro
postgres,
run
storage
when
attempt
to
load
the
configurable
class
register,
postgres
run
storage.
So
just
give
me
a
few
minutes
and
I
will
try
to
google
it
up
all
right.
Okay,
after
looking
through
the
dextro
github,
it
seems
like
there
are
plenty
of
python
modules
that
are
not
installed
with
the
dextra
installation
command
and
it
seems
like
most
of
these
item
modules
are
an
integration
to
the
various
databases
and
other
libraries,
for
example,
pandas
salary
by
spark.
A
A
Alright,
it
seems
like
I
got
yet
another
error,
and
the
error
tells
me
that
I
have
not
defined
the
login
variables,
for
example,
dextro
pg,
username,
dexter,
pg,
password
and
dexter
pg
host
and
dexter
pg
database,
and
I
guess
those
configurations
can
be
set
in
our
yml
file
and
by
looking
at
yml
file.
It
is
actually
a
system
environment
variable.
So,
if
I
want
to
make
it
happen,
I
would
have
to
first
create
postgres
database
and
then
create
all
those
system-
environment
variables
that
are
defined
in
the
yaml
file.
A
So
I
will
start
the
beaver
because
it
is
very
easy
to
create
a
new
database
from
scratch
and
in
db,
beaver
under
the
postgres
user.
I
will
create
a
completely
new
database
and
I
will
call
it
dexter
next.
I
will
add
those
system
environment
variables.
As
for
dextropg
username,
I
will
use
postgres
as
a
default
postgres
user.
As
a
dexter,
pg
password,
I
will
type
in
my
password
that
is
tied
to
the
postgres
user.
A
It
doesn't
work
because
could
not
import
module,
dexter,
abs,
compute
log
manager.
Do
I
even
need
amazon
web
services
because
first
I
don't
have
amazon
web
services
account?
Second,
I
guess
I
have
to
turn
it
off
somewhere
here
yeah.
So
I
will
just
comment
out
these
lines
where
the
amazon
web
services
are
defined.
I
will
save
dextro
yaml
file
and
yet
again
I
will
try
to
run
the
digit
in
the
new
terminal
window
and
it
doesn't
have
the
chrome
scheduler,
which
I
guess
I
will
need
for
the
scheduling.
A
So
let's
install
the
chrome
scheduler,
so
I
will
install
dexter,
slash,
chrome,
cool,
let's
try
it
again
all
right.
It
seems
like
after
a
long
long
time
the
server
is
running.
I'm
very
curious.
If
I
get
that
instance
doesn't
exist
at
all,
so
everything
seems
fine.
Let's
look
at
the
scheduler,
yes
and
scheduler
is
finally
set
up,
but
it
seems
like
I
don't
have
any
pipeline
with
the
schedule
set.
So
let's
try
to
create
our
first
schedule.
A
Okay,
it
seems
that
I
have
to
create
a
scheduler.buy
file,
so
in
this
folder
I
will
create
scheduler
dot
by
and
I
will
copy
paste
the
example
from
dexter
documentation,
but
instead
of
hello
serial
pipeline,
I
will
use
hello
dexter
pipeline
and
I
will
set
the
execution
date
today.
So
it's
november
4th-
and
I
want
it
to
execute
in
three
minutes,
but
first
I
have
to
import
time.
I
guess
together
with
date
time,
and
we
will
want
to
call
hello
dexter
pipeline
here,
but
we
don't
want
to
set
any
inputs.
A
I
will
leave
it
blank,
maybe
and
except
for
the
schedule
we
will
have
to
create
a
new
repository.
I
have
to
also
import
the
decorators
and
I
have
to
import
my
pipeline
from
hello
dexter
import,
hello
pipeline.
Now
I
think
everything
is
set,
so
I
will
try
to
fire
up
the
scheduler
with
the
command
dexter
schedule
up
and
now
I
want
to
test
if
the
scheduler
is
working,
so
I
will
try
the
command
dexter
scheduler
up
and
it
throws
an
error.
No
arguments
given
and
workspace
yaml
not
found.
Ok,
what
is
workspace.yaml.
A
It
wants
me
to
create
a
workspace
dot.
Yaml
configuration
file
with
the
configuration
load
from
python
file,
repos
dot
pi,
but
I
guess
I
have
to
create
the
repository
as
well,
but
let's
just
try
the
schedule
up
command
to
see
if
the
error
changes
yeah,
so
the
system
cannot
find
readpost.file,
of
course,
so
in
the
repos.pi
I
should
put
this
function,
hello,
serial
repository
where
I
define
the
hello
pipeline
with
good
morning
schedule:
let's
import
the
important
files,
also
from
scheduler.buy
import
good
morning
schedule
and
from
dexter
import
repository
decorator.
A
Now
I
think
we
have
everything
set
up.
Let's
try
the
dexter
schedule
up
preview
command
again
and
there's
another
error.
Type
error
module
object
is
not
callable.
I
guess
I
should
not
have
included
pi
so
in
the
scheduler
by
there's
a
error
with
the
datetime
definition.
Okay,
so
I
guess
I
should
import
the
date
times
as
from
date
time
import
date,
time.
A
Let's
try
again
all
right,
it
seems
like
I
use
the
wrong
library-
it's
not
time,
but
it's
a
time
function
from
date,
time,
libraries
so
hopefully
now
it
will
work
as
expected.
Yep
good
morning
schedule
was
added
into
scheduler,
then
we
will
run
the
command
dextro
schedule
up
and
then
we
will
start
the
scheduler
with
extra
schedule
start
good
morning
schedule
and
yet
another
error,
please
specify
user
filename
to
right.
A
It
tries
to
stun
the
current
job
and
it
tries
to
write
something
somewhere
and
for
this
issue
there
seems
to
be
a
dead
end
for
me
because
there's
an
issue
on
github
and
issue
states
that
on
windows
there
is
no
way
you
can
run
the
chrome
that
dexter
uses
and
according
to
the
github
activity,
it
will
be
fixed
by
new
scheduler
in
scheduler
runs
ad
groups,
and
this
was
posted
9
days
ago.
So,
ok,
guys.
A
This
is
a
truly
end
of
the
video,
because
there's
no
way
I
can
set
up
a
scheduler
on
dextre
on
windows.
It
is
a
disappointment
because
I
expected
that
I
would
have
a
fully
functional
data
pipeline
orchestration
tool
and
since
the
prefect
did
not
meet
my
expectations
and
dexterous
seems
so
so
promising.
A
It
is
very
disappointing
that
there's
a
bug
that
prevents
me
from
using
the
dexter
scheduler
on
windows-
and
I
guess
if
I
want
to
use
scheduler
on
windows
in
any
form,
I
would
have
to
either
use
it
as
a
manual
data
pipeline
tool,
which
means,
even
if
I
refactor
all
my
data
pipeline
code
based
from
sync
io
to
dextro.
The
only
use
case
I
see
is
that
I
will
have
a
super
nice
dashboard
that
will
show
me
all
the
dependencies
and
I
would
be
able
to
just
run
my
data
flow
with
one
click.
A
There
would
be
no
way
in
the
world
that
I
can
schedule
the
data
pipeline
using
dexter
chrome.
So
let's
wait
few
months
and
see
if
they
will
do
something
with
this
bug,
and
if
this
video
will
get,
I
don't
know,
1000
views,
then
I
will
create
yet
another
video
when
the
bug
will
be
fixed.
So
I
think
in
overall
dexter
is
much
more
promising
than
prefect
just
because
it
works
out
with
the
box.
It
doesn't
have
that
complicated
using
the
face,
and
even
the
coat
itself
is
just
about
putting
the
decorator
at
the
right
place.