►
Description
Group 1001 is a technology-driven financial services company that provides insurance and annuity products.
Group 1001 migrated 73 Airflow DAGs within 2 weeks of evaluating Dagster with just 2 engineers thanks to the dagster-airflow library. In doing so, they achieved much faster development velocity to help meet the organization's data requirements.
Gu Xie, Head of Data Engineering at Group 1001 led the effort.
A
A
If
we
would
explain
it
if
we
do
it
at
airflow,
it
probably
would
take
us,
maybe
two
three
months
to
even
get
the
data
in
place
where
we
could
actually
start
begin
to
figure
out
a
sort
of
insight
to
build
a
visualization
to
build
a
dashboard
and
the
way
I
sell.
It
is
instead
of
two
three
months.
What
about
three
days?
A
That's
the
difference
that
it
would
take
to
move
over
to
Daxter
it's
just
instead
of
three
months.
We
can
do
it
in
three
days,
just
simply:
The
Accelerated
development,
development
cycle
from
that
perspective,
ease
of
development,
ease
of
troubleshooting
and
monitoring,
and
the
fact
that
we
can
actually
reliably
make
changes
without
breaking
any
existing
processes
and
that's
the
way
I
would
sell
it.
That's
the
way.
I've
been
selling
to
the
business,
just
really
how
do
what?
A
If
I
could
take
your
idea
from
three
months
request
down
to
three
days,
then
the
the
there's
no
question
to
ask:
let's
do
it
because
then
we
can
deliver
more
insight,
the
develop
a
faster
value,
because
speed
is
actually
what
we
care
about.
The
speed
is
really
where
we
can
actually
drive
our
competitive
Advantage
within
our
company.
A
Our
entire
architecture
really
centered
around
airflow,
with
with
our
own
python
processes
or
running
via
kubernetes
pod
operator.
We
ran
into
a
load
of
different
issues.
The
biggest
painful
I've
generally
seen
in
in
terms
is
really
number
one
and
from
a
civil
stability
standpoint,
we've
had
numerous
outages
whereby
airflows,
we
simply
wouldn't
run
our
jobs.
A
So
from
a
stability
standpoint
with
a
big
issue
and
sometimes
a
lot
of
times,
our
instances
would
just
completely
be
unresponsive
which
forces
us
to
go
in
and
restart
all
of
the
the
kubernetes
Clusters
as
well
the
entire
airflow
composer
cluster.
So
stability
was
a
very,
very
big
ink
one
for
us
at
airflow.
The
second
bit
of
the
biggest
problem
from
that
I've
seen
number
two
was
really
regarding
our
end-to-end
cycle
time.
A
So
from
a
development
experience
perspective
running
any
sort
of
changes
to
our
dags
and
airflow,
just
due
to
the
complexity
with
using
the
airflow
version,
one
we
actually
we've
never
actually
did
our
changes
locally.
We
would
just
make
a
code
change.
We
didn't
really
test
it
locally.
We
just
figured
it.
We
would
hope
that
it
worked.
Send
it
through
the
cicd
pipeline,
commit
the
code,
push
the
feature
branch
in,
have
it
deploy
into
our
Dev
environment.
A
So
we're
talking
about
the
End
to
End
Cycle
time
of
to
even
do
one
iteration
of
our
of
our
chain,
not
even
an
end
to
one
second,
that
this
even
one
iteration
of
a
change.
It's
10
minutes
was
when
we
actually
spoke
with
the
Daxter
team.
They
mentioned
they
had
this
airflow
migration
capability
and
it
was
phenomenal.
Actually,
we
were
in
when
we
actually
set
it
up
and
configured
it
in
working
with
one
of
our
Engineers
with
Joe
in
about
what
was
it
for
I.
A
Using
a
the
docker
image
that
we
had
from
from
our
Google
Cloud
repository
that
we
pulled
down
locally
we're
able
to
execute
that
kubernetes
pod
operator
locally.
You
know
before
after
four
hours,
just
passing
it
out.
That
was
phenomenal.
We've
never
seen
a
true
local
airflow
execution
of
that
before,
but
in
a
few,
but
with
Daxter
using
the
actual
Daxter
airflow
migration,
we're
able
to
get
that
up
and
running
and
then
in
three
days
later
we
migrated
all
of
our
73
different
nags
over
to
dagster
and
then
in.
A
What
I'd
definitely
recommend
for
a
best
practice
perspective
is
take
a
look
at
Baxter.
Take
a
look
at
its
ability
to
run
the
airflow
migration
and
think
of
it
as
a
way
of
a
stopping
of
a
stepping
stone.
It's
a
way
to
really
be
a
stop
Gap.
You
can
migrate,
lift
and
shift
all
your
existing
dags
onto
the
diagster
platform.
You
get
to
that.
Leverage
diagster's
really
well
thought
out.
A
End-To-End
sdlc
experience
so
from
a
few
people
cover
the
migration
standpoint,
but
you
can
actually
use
the
platform
to
reinvent
and
redesign
those
pipelines
in
a
new
manner
that
will
get
you
better,
better
agility
and
better
scale.
So
that's
the
way,
I
kind
of
frame
it
and
that's
what
our
actual
plan
and
Visions
are
using
Dexter.
We
don't
plan
to
keep
our
airflow
bags
for
the
long
term,
but
nevertheless
it
will
be
short
in
the
middle
medium
term.