►
From YouTube: Incubation Engineering - Exploring Airflow integration
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
Welcome
to
part
three
in
the
series
of
exploring
ideas
for
a
new
scg.
This
video
describes
airflow
and
a
potential
idea
of
to
integrate
it
into
gitlab
tldr
on
airflow.
It's
a
workflow
orchestrator.
It
consists
of
decks
which
are
basically
pipelines
and
those
pipelines
have
steps
in
them.
Airflow
is,
in
my
experience,
the
de
facto
standard
for
workflow
orchestration
for
data
teams
in
general.
Decks
are
written
in
Python.
A
The
python
code
is
written
locally
in
an
IDE,
then,
and-
and
you
would
have
a
merge
request
to
review
the
code
and
actually
deploy
your
deck
to
airflow.
You
would
have
to
then
go
to
airflow
and
see
the
status
of
your
or
of
your
deck
I
think
it's
super
interesting
to
explore
the
concept
of
integrating
airflow
into
gitlab
for
a
number
of
reasons.
One
of
them
is
that
we
have
currently
preview
apps
and
that
will
be
ideal
for
airflow.
Usually
what
would
happen
is
you
would
have
a
DAC?
A
That's
running
fine,
but
there
are
maybe
new
features
that
you
want
to
introduce
into
the
deck
or
as
a
bug
fix.
What
would
happen
is
you
would
probably
have
either
a
local
development
environment,
which
is
is
great.
A
You
can
just
check
everything,
but
for
bigger
data
sets,
it's
not
feasible
to
run
it
on
your
local
machine,
so
you
would
have
to
have
some
kind
of
depth
environment
and
the
problem
with
deaf
environments
in
airflow
is
that
you
get
conflict,
because,
if
you're
all
deploying
to
the
same
environment,
changes
from
another
person
would
override
your
changes.
A
A
I
think
it
would
be
very
interesting
to
integrate
something
into
our
current
ecosystem
of
tools
like
monitoring,
metrics
error,
tracking,
that
kind
of
stuff,
perhaps
even
integrated
into
infrastructure
and
I'm,
not
sure
where
it
would
fit.
Maybe
monitor
just
get
an
overview
of
of
your
decks,
something
like
that
or
in
your
merch
request,
where
it
will
give
some
feedback
on
airflow
I.
A
Don't
know
I'm
not
I'm,
not
sure,
yet
what
it
should
look
like,
but
I
would
like
to
make
it
easier
to
iterate
on
the
development
of
Dax
and
make
it
easier
to
deploy
Monitor
and
operate
airflow
on
a
daily
basis.
If
you
have
any
suggestions
or
questions,
leave
a
comment
down
below
thanks
for
watching.