►
Description
Discussion about the next iterations of the CI/CD Pipeline
Directed Acyclic Graph (DAG) feature
A
So
today
I
have
not
prepared
anything
special
because
I
saw
that
we
do
have
some
leftovers
in
the
agenda,
especially
dog
wants
to
discuss
doug
and
we
also
invited
camille,
so
hello
camille
to
verify
technical
discussions.
It's
great
to
have
you
here
and
yeah
and
ricky's
item
is
the
first
one,
but
I
see
that
he's
not
in
the
call
so
dope.
Would
you
like
to
start
yeah
thanks
just
share
the
issue.
B
Okay,
so
I
understand
that
this
is
this
can
be
like
a
a
long
discussion
and,
if
we'll
need
we'll
do
like
a
follow-up,
but
there
are
like
few
things
that
I
want
us
to
get
out
of
this
discussion
and,
as
I
go
through
this
entire
thread,
I'll
explain
what
I
want
us
to
achieve.
First,
so
hopefully
you
had
the
other
chance
to
to
read
this,
but
the
issue.
B
Actually,
I
want
to
start
with
one
of
one
of
the
most
popular
requests
that
our
users
have,
which
is
the
allow
need
it
to
refer
to
a
job
in
the
same
stage.
This
is
a
very
popular
issue
and
that
our
users
would
like
us
to
to
implement
mainly
because
it's
served,
one
of
the
biggest
pains
that
they
need
to
create
stages,
even
if
they
don't
have
to
create
stages.
B
So
the
first
thing
that
I
want
us
to
to
discuss
is
that
if
we
will
be
able
at
least
today
to
agree
to
implement-
and
maybe
not
maybe
we'll
say
no-
we
cannot
agree
on
that
because
we
need
to
solve
the
whole
thing.
That's
fine
also,
but
I
want
to
emphasize
that,
if
we'll
be
able
to
come
up
with
a
clear
statement,
saying
yes
in
the
first
iteration
we'll
be
able
to
implement
this
proposal
and
we
can
discuss
the
design
and
everything.
But
this
is
something
that
can
come
in
the
first
iteration.
B
It
will
be
a
huge
benefit
for
our
users
and
and
for
for
our
product,
given
like
all
the
discussions
that
I
had,
and
so
this
is
something
that
I
want
us
to
discuss.
First
and
I'll
continue
with
the
discussion,
but
please
let
me
know
what
you
think
about
that
after
we
will
finish
this
entire
discussion.
B
This
also
sparked
the
entire,
at
least
from
what
I
was
reading.
It
sparked
the
entire
discussion.
Should
we
support
stageless
pipelines
and,
as
I
mentioned,
I
need
engineers
here
to
to
help
me
what
is
the
value
of
of
a
of
having
a
stageless
pipeline,
because
if
we
have
like
one
stage
as
a
user,
it's
like
a
stageless
pipeline.
If
I
will
not
create
any
stage,
we'll
create
one
stage
for
the
user
and
then
and
if
we'll
be
able
to
allow
need
to
refer
to
a
job
in
the
same
stage,
then
it's
like.
B
We
have
like
no
stages
now
understand
that
the
back
end
is
a
bit
is
a
bit
different,
but
I
want
to
say
like
from
from
the
user
point
of
view,
I
don't
think
it
made
a
lot
of
sense.
I
mean
we
can
implement
it
from
the
ux
point
of
view.
Like
you
know,
we
can
like
hide
the
stage
the
test
stage,
but
it
doesn't
really
provide
any
any
any
value.
So
the
key
here
is
this
thing.
Allow
me
to
refer
to
a
job
in
the
same
stage.
That's
that's
already.
B
The
whole
thing
revolves
around
this
part,
and
this
is
like
the
first,
the
first
section,
the
second,
the
second,
the
second
section
is:
do
we
want
to
process
all
pipeline
using
the
dag,
algorithm
and
move
on
the
stage
to
be
like
dug
under
the
hood,
because
when
we
do
like
the
stage,
it's
it's,
it's
like
a
need.
It's
like
you're,
using
a
need
to
refer
to,
like
you
you're.
It's
the
same
that
you
use
a
need
to
refer
to
a
stage.
So
that's.
A
So,
in
my
opinion,
this
unification
and
having
just
one
algorithm,
it's
a
prerequisite
for
actually
adding
more
features
to
the
dac
processing.
Because,
right
now
we
have
two
algorithms.
One
algorithm
is
that
we
process
pipeline
based
on
the
semantics
of
a
stage
and
that's
quite
obvious
how
it
works.
It's
how
it
have
been
like
working
all
the
time,
and
then
we
have
doug
and
with
dark
when
there
is
no
legacy
processing
in
the
gitlab
ciao.
A
A
A
Like
I
mean
the
stage
is
processing
like
you,
you
define
a
stage,
and
you
know
that
this
job
is
going
to
be
executed
after
the
preview.
Previous
stage
is
done
right,
so
that's
perhaps
it's
not
legacy
because
it's
still
being
used,
and
we
probably
do
not
plan
to
deprecate
that.
But
I
refer
this
to
as
a
previous
thing
or
legacy
thing.
It's
something
we
had
before.
That's
something
we
still
do
have
in
addition
to
that
right,
which
which
gets
triggered
when.
A
D
D
How
important
is
to
retain
the
hybrid
model
really
because,
like
dove,
like
you
mentioned
about
this
very,
like
very
high
vote
issue
of,
please
refer
to
a
job
in
a
single
stage.
I'm
not
curious
really
like
how
people
really
want
do
they
want
to
switch
like
to
fully
to
the
dark.
You
know
like
a
free
form,
or
do
they
want
to
continue
using
the
hybrid
mode
us
like
that.
You
have
some
jobs
that
are
in
stages,
some
jobs
that
are
in
a
dark,
but
like
with
this
more
free
representation.
B
Yeah,
I
don't
think
like
there
is
a
clear
cut
to
your
to
your
question
and
some
would
prefer
to
use
stages.
Some
would
prefer
to
use
the
hybrid
some
would
prefer
to
use
dag.
I
think,
like
them
most
of
the
users
that
want
to
refer
to
a
job
in
the
same
stage
already
existing
users,
so
they
already
use
our
stage
workflow
stage
pipeline,
and
they
now
want
somehow
to
put
some
dependencies
within
the
same
stage.
B
I
do
think
that
if
they
would
be
like
new
users,
new
users
would
like
only
to
use
that,
because
we've
seen
this
with
with
github
actions
user
really
liked
the
fact
that
they
don't
need
to
define
stages
if
they
don't
have
to.
They
feel
that
the
the
pipeline
is
more
far
is
faster,
that
everything
all
the
jobs
run
in
parallel,
and
then
you
can
just
control
which
job
you
run
first
based
on
the
need,
but
I
don't
think
I
just
feel
like
scenarios
where
we
need
stages.
A
I
just
think
that
it
would
be
much
easier
for
us
to
avoid
the
velocity
laws
as
we
go,
because
this
way
we
can,
we
will
only
only
need
to
maintain
one
mechanism
and
one
algorithm
instead
of
the
two
and
with
the
unification
like
the
stages
are
not
going
anywhere.
We
would
still
allow
users
to
define
stages
and
everything
would
behave
in
the
same
way,
but
under
the
hood
we
wouldn't
have
two
algorithms.
B
Sorry,
I
I
just
want
to-
I
just
want
to
comment
something.
So
we
saw
what
you're
saying
is
is
correct,
but
we
do
have.
One
problem
is
that
we
have
inconsistency
when
you're,
when
you
we're
using
manual
jobs,
which
means
that
if
we
will
switch
our
processing
to
be
dug,
it
means
breaking
changes,
and
this
type
of
breaking
changes,
which
will
be,
which
will
be
a
huge
impact
for
our
users,
really
be
able
to
do
it
only
in
a
year
from
from
today,
because
we
can
introduce
it
only
in
the
next
major
major.
D
Need,
oh,
I
I
I
think,
like
this,
like,
let's
say,
processing
the
processing
status
as
a
doc.
This
manual
job
is
like
one
of
the
things
that
are
like
more
quick
in
the
behavior,
for
example,
in
context
of
the
skipped
jobs
so
really
like.
We
would
have
to
unwind
and
understand
exactly
how
to
allow
to
model
existing
stages
in
a
dark
model
so
like
when
we
would
do
this,
like,
let's
say
internal
conversion,
to
processing
status
as
a
dark.
D
D
E
Think
I'm
not
sure
if
this
would
be
possible.
I
think
we
can
definitely
aim
to
allow
users
to
maintain
their
current
state
of
behavior,
like
maybe
it
means
that
we
need
to
introduce
some
either
keywords
that
allows
users
to
switch
like
to
tune
their
dog
behavior
in
a
specific
way
like
like
today,
for
example,
we
consider
a
job
dependency
to
be
required
when
is
in
the
context
of
a
dag,
so
a
job
doesn't
run
if
the
dag
dependencies
doesn't
run.
E
That
is
in
the
case
of,
for
example,
yeah.
We
were
doing
with
manual
or
skipped
job.
Well,
that's
not
the
the
case
with
the
stage
approach.
So
even
if
we
convert
stage
to
use
dag
on
the
back
under
the
hood,
there
is
this
two
difference
in
behavior
that
either
which
either
we
change
everything
to
to
be
more
like
the
stage
behavior
or
we
change
everything
to
use
the
dac
behavior,
but
we're
going
to
break
one
other
one
or
the
other.
E
But
one
thing
we
could
do
is
to
introduce
some
features
either
in
the
case
of
syntax.
That
can
find
you
in
the
behavior
to
allow
your
user
to
go
back
to
the
behavior
they
had
before.
If
they
just
use
specific
keywords
but
still
means
breaking.
Changes
means
that
the
user
needs
to
upgrade
their
pipeline
configurations.
E
E
F
Front-End
concerns
about
it
as
well
that
we
have
to
take
into
account.
We
can't
just
say
on
the
back
end
they're
being
processed
the
same
and
then
not
be
able
to
reflect
that
on
the
front
end
or
if
there
is,
we
need
to
be
determined
about
it
too
right
when
we
have
needs
and
dependencies
and
we're
showing
that,
like
we
can't
just
convert
it.
We
need
to
be
aware,
if
we're
going,
to
convert
them,
to
use
the
same
processing,
to
be
able
to
have
some
sort
of
tracking
of
that.
F
If
we
don't
want
to
display
them
the
same
on
the
front
end.
Otherwise
we're
going
to
start
having
really
complicated
views
of
things
if
the
needs
become
implicit
right.
If
you
use
a
stage
and
all
of
a
sudden,
that's
just
converted
to
a
bunch
of
jobs
in
your
needs
array,
and
you
can't
distinguish
between
them.
That
also
is
going
to
have
implications
that
we
should
consider.
B
We
are
working.
Definitely
I
agree
with
what
with
what
you
said,
sir,
and
I
think
we
we
are
working
on
that
on
converting
it
and
definitely
like.
There
are
more
things
that
we'll
need
to
figure
out
before
we
will
do
the
conversion,
but,
as
I
mentioned,
I
want
us
to
be
very
focused
on
this
discussion
and
understand
that
there
is
a
breaking
changes
and
a
lot
of
things
that
we
need
to
implement
before
processing
our
algorithm.
Using
that-
and
my
question
is
in
this
discussion-
sorry
for
going
back
to
it.
B
D
D
If
the
current
way,
how
it's
being
done,
it's
gonna,
be
heavily
unoptimized
and,
like
I'm
talking
about
the
processing
optimization
and
like
the
data
optimization,
for
example,
if
you
have
a
stage
that
contains
100
jobs
and
in
the
follow-up
stage,
you
have
10
jobs.
Currently
we
would
store
in
the
database
10
multiplied
by
100
jobs.
D
A
E
D
Kind
of
thinking-
yes,
this
is
exactly
the
same
program,
but
maybe
if
you
would
be
able
like
to
start
with
that
problem,
you
would
actually
learn
and
optimize
why
people
are
using
that
because,
like
in
the
pure
dark
way,
it's,
I
don't
really
kind
of
anticipate
that
a
lot
of
people
will
start
with
very
big
pipelines.
They're
going
to
be
definitely
like
these
examples
of
people,
but
in
most
cases
people
just
want
this
freedom
and
maybe
their
their
freedom
is
connected
like
with
smaller
pipelines
already
than
with
the
stage
one
so
maybe
like.
A
Is
fine
like
yeah?
I
think
it
makes
sense
to
actually
move
forward
with
this
in
some
way
or
another,
but
it
actually
brings
up
the
discussion
about
breaking
changes
or
rather
implementing
new
features
that
might
behave
differently
and
allowing
users
to
opt
in
for
that.
In
this
particular
case,
we
are
like
thinking
about
defining
the
stageless
keyword
with
that
is
either
true
or
implicitly
it's
false
right,
but
is
there
a
way
to
actually
extend
this
somehow
so
that
we
can
actually
experiment
with
other
solutions,
and
we
can
allow
users
to
opt-in?
D
C
Yeah
I
mean
we
can
create
a
topological
tree
or
something
I
don't
know.
I
don't
know
what's
its
name,
but
we
can
just
create
a
tree
with
these
needs
and
we
can
see
if
there
is
a
circular
dependency
or
not.
I
found
an
algorithm
while
searching
on
this,
but
I
don't
remember
it
right
now,
but
actually
we
can
avoid
the
circle
dependency.
C
C
Actually,
we
don't
have
to
change
the
any
algorithm
we
just
can
use
atomic
processing
as
it
is.
I
think
so.
So
that's
why
I
think
we
can
just
implement
this
feature
without
changing
anything
without
introducing
any
breaking
change.
So
that's
why
I
just
want
to
know
if
there
is
any
concern.
Okay,
there
is
a
ui
concern,
but
I
just
want
to
know
if
there
is
a
back-end
concern
of
this.
D
E
I
think
that
if
we
make
this
change
so
before,
actually
speaking
about
this,
I
think
the
problem
we
have
with
the
dag
implementation
is
that
it's
based
on
top
of
the
stage
implementation.
So
it
was
supposed
to
be
something
alternative,
but
in
reality
the
dag
processing
can't
live
without
the
stage
processing
because
it
relies
on
it
for
validations.
It
relies
on
it
for
the
processing
for
optimization
purposes.
I
I
I
think
it's
not
because
it's
necessary,
so
I
and
I
think
that
this
should
be
the
other
way
around.
E
So
it
should
be
that
the
stage
process
processing
should
rely
on
top
of
the
duct
processing
being
the
dark
processing,
a
more
generic
approach
right.
So
the
stage
is
a
more
specialized
way
of
dealing
with
that,
but
in
reality
is
the
other
way
around.
We
have.
The
dog
is
implemented
on
top
of
stage
from
the
point
of
view
of
implementation,
so
making
like
working
on
the
issue
where
we
allow
needs
to
be
referred
on.
Jobs
in
the
same
stage
might
help
us
break
away
that
processing
from
stage
processing.
E
And
and
effectively
kind
of
consider
these
two
approaches
completely
separate,
so
we
don't
have
the
dag,
depending
on
whatever
design
he's
made
on
the
stage
approach,
and
then
we
can
try
to
replace
the
way
around.
So
our
goal
eventually
is
to
have
the
stage
processing
beha
being
processed
by
the
dac
algorithm.
E
Either
we
do
something
in
the
back
end,
where
we
create
stages
like
under
the
wood.
It's
supposed
to
be
something
that
the
user
should
not
really
see
right,
but
either
we
create
stages
under
the
wood
or
we
do
groups
of
jobs
to
simulate
the
stage,
and
I
think
the
ultimate
goal
is
to
build
the
stage
approach.
On
top
of
that,
and
so,
first
of
all,
we
need
to
separate
these
two
things
and
invert.
This
kind
of
relationship.
D
I'm
gonna
add
one
like
observation.
Why,
like
I
made
this
design
choice
of
like
depending
on
the
pure,
it
provides
you
a
sort
of
the
processing
order
of
data.
It
provides
you
a
thought
on
how
you
want
to
visualize
and
it's
going
to
be
always
correct.
Maybe
it's
not
going
to
be
pretty,
but
it's
always
forward.
It's
not
it's
never
backward
or
it's
never.
D
On
like
on
the
same
bucket,
which
is
like
the
stage,
it
actually
makes
a
lot
of
difference
for
the
actual
compute
that
you
have
to
do,
because
you
always
know
that
you
process
the
data
forward.
Only
you
don't
really
have
to
look
data
backward
way,
so
you
can
actually
make
a
lot
of
optimization,
especially
for
very
big
data
sets
based
on
that
assumption.
D
So
actually
the
stages
makes
the
dac
processing
efficient.
That's
just
the
one
of
the
reason
really
because
it's
always
topologically
sorted
by
stages,
and-
and
this
is
like
right
now-
we
would
require
user
to
perform
this
topological
sort.
But
now
we
would
have
to
perform
this
sword
and
somehow
started
information
about
this
topological
sort
to
aid
like
the
the
performance
of
the
processing
somewhere
in
the
system,
which
would
be
disconnected
from
the
stages.
D
I'm
I'm
not
really
sure
if
we
use
that
information
on
the
front
end,
but
this
is
actually
like
the
eight
that
would
can
give
you
like
the
hint
like
what
is
always
that,
if
you,
if
you
are
up
that,
let's
say
stage,
it's
never
gonna
depend
on
something
in
the
future.
On
on
itself.
It's
always
gonna
be
like
referencing
to
something
in
the
past
or
like
to
the
prayer
to
that
one
so
and
it
could.
E
E
D
E
A
Doesn't
have
to
be
a
stage
index
is
legacy.
Think
we
no
longer
need
that.
We
are
capable
of
generating
stages
and
persisting
them
in
a
database,
because
we
do
have
a
separate
table
for
stages.
It's
called
ci
stages.
So
what
before
creating
a
pipeline,
we
can
actually
calculate
the
topological
order
and
hydrate
our.
I
know
that
data
model
with
stages
that
are
auto
generated.
E
E
E
Maybe
the
stage
could
be
used
like
on
the
side
for
us
to
kind
of
migrate
over
time
things,
but
the
primary
thing
should
not
be
the
stage
position
that
we
have
to
use
for
processing,
and
then
we
could
do
the
other
way
around.
Where
stages
become
some
abstraction
built
on
top
of
dag,
where
a
group
of
jobs
might
have
a
position
and
or
or
a
job
can
depend
on
a
group
of
jobs,
and
that
case
will
be
the
equivalent
of
stage,
but
it's
inverting
what
we
have
doing
today.
E
So
I'm
not
saying
that
what
we
have
today
is
wrong,
or
was
like
a
mistake
of
implementation.
I
think
made
a
lot
of
sense
because
our
stage
approach,
it
was
the
by
you
know
the
main
thing
we
had
and
it
made
sense
to
build
dag
on
top
of
stage
for
a
lot
of
reason,
and
we
did
not
anticipate
that
we
actually
wanted
to
invert
things
in
the
long
term.
So
now
that
we
are
talking
about
it,
I'm
wondering
whether
we
should
start
inverting
things
in
our
pipeline
processing.
B
So
if
we
use
the
stageless
tool
and
when
we
say
okay,
when
we
use
the
stages
to
it,
it's
like
dug
and
that's
it
not
based
on
stages.
Is
it
something
that
we
can
maybe
start
experimenting
and
see
like
if
users
are
using
it
more
and
more,
then
we'll
realize?
Okay,
you
probably
need
to
move
forward
with
that.
E
Yeah,
I
think,
before
we
introduce
the
stageless
true,
we
need
to
have
the
infrastructure
in
place
for
actually
processing
something
processing,
a
configuration
where
the
users
specify
stages
to
be
processed
as
dag,
but
maintaining
the
same,
pretty
much
the
same
behavior.
There
might
be
some
differences
that
we
currently
have
today,
where
we
skip
jobs
or
with
manuals
when
these
things
will
not
work
the
exactly
the
same
way
until
we
provide
better
tuning
mechanisms,
but.
E
D
Yes,
I
mean
it
would
break,
but
it
would
break-
and
you
would
know
about
this
being
broken
because
he
would
change
to
completely
different
model
at
that
moment.
So
maybe
it's
like
expected
change
of
the
behavior
because
he's
actually
making
deliberate
choice.
Choice
of
like
saying
I
want
only
dark.
I
don't
want
anything
else.
I
don't
want
stages.
D
I
want
only
dark
exclusively
and
maybe
I
I'm
still
like
thinking
like
of
trying
to
reduce
as
much
as
we
can
today
and
maybe
like
if
we
can
reuse
as
much
as
we
have
today,
maybe
still
generating
status,
but
not
presenting
them
to
the
user,
because
we
know
that
user
deliverability
choose
the
dac,
so
he
doesn't
have
like
this
pipeline
graph
it
and
it
has
a
dark
view
and
it
would
work.
Maybe
the
stage
becomes
something
internal
that
we
don't
present
to
user.
C
A
So
I
think
this
decision
is
up
to
dove
or
cheryl.
Actually,
so
I
I
feel
like,
even
if
we
never
emerged
this
poc,
it
would
actually
shed
some
light
on
the
dark
processing
versus
stages,
processing
and
stages
through,
and
I
expect
to
see
some
you
know
meaningful
benefits
from
from
building
this
poc.
So,
in
my
opinion,
it's
a
good
idea
unless
kanye
thinks
it
it's
not.
So
I
wonder:
what's
your
opinion
about
camille.