►
From YouTube: Cloud Tech Thursdays: Stop merging broken code with Zuul
Description
Cloud Tech Thursday explores the full modern open source cloud stack, from hardware to serverless. Learn about new ideas, projects, and releases around Kubernetes, OpenStack, hybrid cloud enablement, and many other topics.
C
B
B
You
start
acme
gating
to
provide
school
support
and
services,
as
well
as
a
project
that
is
a
gating
system
that
helps
you
stop
or
stop
merging
broken
code,
make
sure
that
every
change
is
fully
tested
and
before
merging
and
tesla's
changes
amongst
multiple
services
or
microservices.
Rather
so
I'm
going
to
let
the
expert
here
jim
go
ahead
and
take
it
away.
E
E
E
And
to
that
I
would
answer
yes,
it
is,
but
I
see
sort
of
the
the
the
the
world
of
ci
ncd
as
a
continuum
and
the
first
ci
systems.
If
you
look
at
them,
if
you
remember
what
ci
stands
for
continuous
integration
right,
the
first
ci
systems
were,
you
would
merge
a
change
to
your
java
project
and
you
would
run
the
build
after
doing
that
and
see
if
the
build
broke.
And
basically
this
was.
This
is
post,
merge,
testing.
This
is
testing
that
was
focused
on
the
past.
E
Things
have
moved
a
bit
since
then,
and
we
have
pre-merged
testing
now.
So
you,
you
write
a
change
and
a
ci
system
tests
that
the
change
that
you
wrote
works
before
you
merge
it,
which
is
great,
because
you
can
find
out
that
you
that
you
can
find
out
that
it's
broken
before
you've
broken
it
for
everybody
else.
E
This
kind
of
testing
is
focused
on
the
present.
So
I
think
gating
is
the
continuation
of
this
and
it's
a
future
looking
system,
it's
about
saying
not
only
you
know,
does
the
change
that
you
just
wrote
work.
It's
does
the
change
that
you
just
wrote
work
when
it's
when
it's
going
to
be
merged
with
the
tree
which
might
have
moved
on
since
you
wrote
it.
Other
people
might
have
merged
changes
since
then,
does
it
work
with
all
of
the
other
changes
that
are
in
flight?
E
You
know
your
colleagues
are
writing
changes
at
the
same
time
you
are,
and
you
may
or
may
not
be
talking
to
each
other
when
those
all
get
sequenced
up
to
merge
is.
Is
the
thing
still
going
to
work
and
does
your
project
have
dependencies
that
you
know?
Is
it
going
to
work
when
this
dependency
changes?
So
that's,
that's.
How
I
see
the
purpose
of
a
project
gating
system
is
it's
ci
and
cd
that
is
focused
on
the
future.
E
E
C
C
E
Okay,
so
how
did
we
end
up
thinking
that
we
needed
a
a
system
like
this
right?
Why?
Why
wasn't?
Why
couldn't
we
just
check
to
see
if
our
build
was
broken
every
morning,
and
so
zuul
came
out
of
the
openstack
project
it
we,
we
started
working
on
zuul
fairly
early
on
in
openstack
development,
and
the
reason
is:
is
you
know
you
if
you
look
at
this
slide
here,
I've
got
a
lot
of
logos
on
here.
E
None
of
those
are
actually
the
openstack
logo,
and
that's
because
when
you
know
we
we
present
openstack
sort
of
to
the
world.
We
talk
about
it
as
as
a
single
product,
but
really
underneath
it's
a
a
bunch
of
independent
projects,
semi-independent
projects
that
are
trying
to
work
collaboratively
to
build
a
cloud
system,
and
so
each
of
these
little
logos
here
is
representing
one
of
those
projects,
and
one
one
logo
in
particular
is,
is,
I
think,
near
and
dear,
to
one
of
our
panelists
today.
E
How
do
we
develop
all
of
these
projects
together
and
make
sure
that
that
we
can,
you
know
every
six
months,
release
something
called
openstack
where
they
all
work
together
and
and
so
that's
kind
of
why
we
ended
up
with
developing
this
system
that
was
focused
on
testing
changes
before
they
merge
and
testing
them
with
other
projects,
dependencies
and
other
projects.
That
sort
of
thing,
so
that's
that's
sort
of
where,
where
zuul
came
from
and
why
we,
we
started
to
go
down
this
road
in
the
first
place,.
E
Something
that
that
zul
does
is
it
it.
It
anticipates
that
changes
are
going
to
merge
to
other
projects
and
incorporates
them
into
its
testing.
So
we've
got
two
little
graphics
here
on
the
on
the
left.
It's
kind
of
illustrating
how
you
can
use
this
with
dependencies,
so
you
could
have
a
a
front-end
ui
change
that
depends
on
a
back-end
server
change.
That
depends
on
changes
underlying
libraries
and
and
you
can
build
up
the
sort
of
speculative
chain
of
what
is
going
to
happen
when
all
those
changes,
merge
and
then
on.
E
The
right
is,
is
a
illustration
of
how
zul
sequences
those
changes
in
its
testing
right.
It's
it
takes
the
latest,
merge
change
and
then
it
it
builds.
On
top
of
that,
it
says
I
need
to
test
this
change
that
somebody
wants
to
merge
and
I
need
to
test
these
three
other
changes
that
depend
on
it
right
and
if
one
of
those
fails,
it
pulls
it
out
of
the
sequence
and
and
keeps
going
so
that's
that's
a
pretty
pretty
central
concept
to
how
zul
does
its
work
and
how
do.
E
Yeah,
so
that's
a
that's
a
good
question.
There's
a
couple
of
ways
that
that
that
we
do
that
one
is
there's
the
there's
of
course
git
dependencies
right,
so
you
actually,
if
you're
in
github,
you
know,
you'll
you'll
see
this
as
a
a
series
of
commits
in
a
pull
request
right.
If
you're
in
garrett,
then
each
commit
gets
its
own
change,
but
it
still
recognizes
the
the
get
commit
hierarchy
and
the
get
dependencies
are
immutable
right
like
there.
E
You
can't
you
can't
pull
one
of
those
out
you
you,
you
have
to
merge
all
of
those
changes,
but
that
is
part
of
the
dependency
sequence.
Another
is
the
implicit
sequence
of
changes
that
happen
when
when
people
approve
them
for
merging
right,
so
I
guess
an
important
part
of
what
I
think
belongs
in
a
gaming
system
that
I
may
have
glossed
over
is
that
the
gating
system
is
doing
the
merging
right.
So
there's
no
there's
no
person
pressing
the
the
the
you
know
merge
this
pr
button
on
github.
E
The
gating
system
is
responsible
for
doing
that,
because
it's
also
responsible
for
for
determining
you
know
at
the
moment
of
merge
if
the.
If
the
change
passes
test,
or
not
so
when,
when
people,
if
they're
in
github
or
garrett
or
whatever
and
they're
they're,
reviewing
proposed
changes-
and
they
say
yes,
I
want
this
thing
to
merge.
They
do
something
to
indicate
to
to
zuul
that
they
want
it
to
merge.
E
E
This
one
needs
more
work,
go
and
make
a
change
to
it,
and
then
maybe
you
find
a
couple
more
changes
that
are
ready
to
merge
and
you
approve
them
zul
is
is-
is
going
to
start
sequencing,
those
as
you
approve
them.
So
there's
this
implicit
ordering
based
on
on
that,
and
that's
actually
kind
of
an
illustration
of
what
might
be
going
on
here
on
this
slide
right.
So
perhaps
somebody
approved
a
nova
change.
Two
nova
changes
in
a
row
that
nova
is
an
openstack
component
right.
E
There's
another
openstack
component
called
sender:
somebody
might
somebody
else
might
have
approved
that,
but
these
projects
are
both
openstack
projects.
They
need
to
be
tested
together.
We
need
to
make
sure
that
that
that
nova
change
isn't
going
to
break
that
sender,
change
and,
and
that
sort
of
thing
so
so
this
implied
dependency
sequence
is,
is
kind
of
the
next
thing.
E
That's
the
that's
how
you
end
up
with
these
sequences
of
changes
looking
at
zuul
here.
The
third
thing
is,
I
think,
the
most
powerful
way
of
sequencing
dependencies
and
and
it's
interesting
because
it
isn't
it's,
it's
actually
not
something
we
started
with
in
zuul
it
was.
It
was
really
an
emergent
behavior,
but
but
now
we
see
it
as
is
absolutely
critical
to
it,
and
that
is
explicit
dependencies
between
changes,
and
so
that's
the
idea
where
I
guess
going
back
to
to
this
thing.
E
Where
you,
you
have
a
front,
end
change,
and
you
actually,
you
say
as
part
of
the
commit
message.
This
change
depends
on
the
back
end
change,
and
so
not
only
do
your
reviewers
see
that
when
they're
looking
at
the
commit
message,
but
zul
sees
that
and
it
incorporates
it
into
its
sequencing
system,
and
if,
if
you
want
to
indulge
me
for
a
minute
or
so,
I
actually
have
a
sort
of
walk
through
for
what
that
looks
like
if,
if
folks
would
like
a
little.
C
C
E
E
E
It's
actually
great
for
bridging
across
projects
that
might
be
in
garrett
or
github
or
peggyer
or
or
gitlab.
All
of
these
we've
got
support
for
all
of
these
in
in
zul
we're
still
working
on
the
gitlab
driver,
but
that's
you
know,
work
in
progress
and
and
as
well.
It
supports
a
lot
of
different
cloud
environments.
So
zul's
execution
is,
can
be
you
know
not
only
in
you
can
run
it
on
a
container
system.
E
A
E
Yeah
so
yeah,
why
don't
I
talk
about
that?
A
little
bit
more,
so
I've
I've
contrived
this
example
and
I'm
I'm
I
I've
intentionally
made
it
not
openstack
specific.
Hopefully
this
is
something
that
that
everybody
here
can
can
relate
to.
So
I've
got
these
three
logos
here,
because
I
apparently
like
slides
with
logos,
but
you
know
with
these
logos,
how
can
you?
How
can
you
not
love
them
so
over
on
the
left?
E
This
is
a
synthetic
example
and,
and
I'm
just
trying
to
create
something
that
sounds
plausible,
whether
or
not
we
could
actually
do
this
or
need
to
do
this.
I
don't
know,
but
it
to
me
this
seems
plausible
right,
so
we
want
to
expose
the
idea
of
of
you
know,
doing
secure,
delete
when
block
devices
are
removed
from
from
sender,
and
we
want
to
make
that
happen
in
kubernetes
when
say,
you've
got
a
persistent
volume
that
gets
deleted.
You
want.
E
You
want
that
pvc
to
to
cause
the
the
underlying
sender
api
to
do
the
secure
delete
right.
So
this
is
a
new
feature
that
we
have
to
move
all
the
way
through
that
that
entire
stack.
So
we
would
start
by
uploading
a
change
or
writing
a
change
to
sender
and
uploading
it
to
opendevs
garrett
openstack
uses
a
garrett
code
review
system,
that's
hosted
by
opendev
for
its
changes.
So
this
is
what
that
looks
like.
This
is
basically
a
change.
That's
in
review.
E
We've
got
the
commit
message
here
on
the
screen
where
we
we,
where
we're
adding
the
api
option
to
to
perform
secure,
deletes
right.
So
the
next
thing
we
need
to
do
is
add
support
for
using
that
api
to
go
for
cloud
right,
so
we
open
a
pr
against
go
for
cloud
and
and
that
lets
the
the
the
world
of
go
access,
this
sender,
api
and
then
finally,
in
kubernetes,
we
update
to
the
latest
gophercloud
and
and
have
it
use
that
secure
delete
option
when
sender
or
when
volumes
are
deleted.
E
So
at
this
point
we've
got
three
changes
out
there.
None
of
them
have
merged
yet,
and
you
know
we
can
test
all
of
them
independently,
but
there's
there's
there's
no
relationship
to
each
other.
We
don't
the
the
people
reviewing
the
kubernetes
change,
they're
kind
of
taking
it
on
faith
at
this
point
that
everything
else
works.
You
know
they
see
this
pr
show
up
and
say:
hey.
We
totally
added
this
api
to
gophercloud,
we're
pretty
sure
it's
going
to
work.
E
You
know,
please
merge
our
change,
so
maybe
maybe
we
want
to
use
zuul
to
to
show
that
this
entire
sequence
can
can
actually
be
tested
together.
So
so
we
go
back
to
our
kubernetes
change
and
we
add
this
depends
on
footer
to
to
the
change
and-
and
you
can
see
that
it
says-
depends
on
and
then
it's
got
the
url
for
the
gopher
cloud
change,
and
so
this
is
how
you
do
that
explicit
dependency
specification
in
zul.
E
We
just
made
it
urls
so
that
so
that
you
can
actually
support
cross.
We
call
them
cross
source
dependencies
where,
where
you
have
changes
in
github,
depending
on
changes
in
garrett
right
so
so
anyway,
this
is
the
kubernetes
change
saying
I
depend
on
the
gopher
cloud
change.
The
gopher
cloud
change
depends
on
this
change
to
sender
and
if
we
go
back
to
so
so
at
that
point,
oh
sorry,
so
I
just
wanna
highlight
again
the
depends
on
here.
That's
it
depends
on
to
a
different
system.
E
So
we're
not
even
saying
we
depend
on
another
github
change,
we're
actually
depending
on
changing
garrett.
That's
okay!
Zuul
knows
about
github,
it
knows
about
garrett.
It
can
stack
those
three
changes
up,
and
this
starts
to
look
like
that
that
that
diagram
earlier
we've
got
these
three
changes
and
the
little
lines
show
the
connection
in
zul.
This
is
by
the
way.
This
is
what
the
zul
status
interface
looks
like.
It
actually
shows
you
these
this
relationship,
so
so
that's
how
you
stack
those
up.
E
We
can
extend
this
one
step
further,
though
we,
this
is
a
brand
new
feature,
and
perhaps
our
zuul
doesn't
actually
have
a
job
that
exercises
it
yet,
and
so
we
can
make.
We
can
extend
this
idea
of
speculative
execution
where
we're
saying
zuul.
I
want
you
to
run
my
tests
with
these
three
other
changes.
We
can
also
say
to
zuul
hey.
I
want
you
to
run
a
job
that
you've
never
seen
before
so
zul's
configuration
is
in
yaml
files,
of
course,
and
so
this
is
what
his
old
job
definition
looks
like.
E
We
can
call
that
job
sender
kubernetes
secure,
delete,
we
can
parent
it
on
a
job
called,
say:
devstack,
kubernetes
devstack
is
openstack's
environment
for
performing
development
right.
So
it's
it's
kind
of
like
mini
cube
for
openstack,
so
you
can
imagine
that
there's
a
job
somewhere
in
the
system
that
sets
up
a
dev
stack
for
for
testing
openstack
and
sets
up
a
mini
cube
for
testing
kubernetes
and
maybe
does
both
of
those
things
for
testing
the
two
systems
together.
E
Zuul
has
a
really
powerful
inheritance
system
for
its
jobs,
so
we
can
actually
take
you.
You
know
you
can
spend
a
lot
of
effort
setting
up
a
job
that
sets
up
your
infrastructure
and
then
make
a
bunch
of
little
jobs
that
make
small
tweaks
to
that.
So
that's
sort
of
what
we're
doing
here,
we're
saying:
let's
take
that
defsat
kubernetes
job
and
just
do
just
add
on
to
that
a
little
bit
to
test
our
secure
delete
system.
E
E
So
if
we
wrap
all
of
that
up
into
a
change
and
upload
it
to
to
garrett,
then
in
zuul
now
we've
got
a
sequence
of
four
changes
and
the
last
one
runs
that
extra
job
that
we
just
defined
and
when
that's
done,
it'll
report
back
to
garrett
and
you
can
actually
see
on
the
bottom
of
the
screen
here.
The
little
cinder
kubernetes
secured
elite
job
succeeded.
F
D
E
On
we,
we
we
were
testing
as
much
as
we
possibly
could
all
the
time
that
kind
of
grows
exponentially
and
it's
it's
ultimately
not
sustainable
to
test
everything,
and
so
we've
gotten
better
about
saying
you
know
these
things.
We
always
need
to
test
this
thing.
We
only
need
to
test
some
of
the
time
you
know
when,
when
this
part
of
the
system
changes
that
sort
of
thing
so
so
yeah,
if
we,
if
we
you
know.
F
E
F
E
We
might
not
always
want
to
run
that
test,
and
so
we
then
might
make
that
contingent
on.
Well,
you
know,
let's
only
run
this
job
if
we're
touching
the
part
of
sender
that's
relevant
to
secure
deletes
and
that
sort
of
thing
so,
and
the
other
thing
to
know
about
the
workflow
here
is
that,
because
of
this
dependency
chain
that
we've
made,
this
change
isn't
going
to
merge
until
the
ones
ahead
of
it
have
merged.
So
since
we
depend
on
that,
kubernetes
pull
request.
E
We're
not
going
to
merge
this
until
that
one
passes
and
that
one's
not
going
to
merge
until
the
gopher
cloud
passes
and
the
go
for
cloud.
One
isn't
going
to
merge
until
the
center
one
passes
so
sorry
not
passes,
but
merges
right.
So
you
we
can
maintain
that
strict
sequencing
to
make
sure
that
not
only
can
we
prove
out
that
this
whole
new
feature
works
in
advance,
but
we
can
make
sure
that
we
don't
merge
any
changes
that
depend
on
it
until
you
know
the
the
dependency
changes
have
merged
as.
E
F
E
Continuous
deployment
of
an
openstack
cluster
always
worked
right,
and
so
we
wanted
you
to
be
able
to
upgrade
your
your
nova
nodes
and
then
upgrade,
or
you
know,
maybe
upgrade
keystone
first
and
then
upgrade
nova
to
use
a
new
keystone
feature
and
so
forth
right.
So
we
kind
of
we
said
that
we
couldn't
require
cloud
operators
to
shut
down
their
entire
cloud
and
then
bring
everything
up
again.
E
E
It
always
has
to
be
a
trick,
strict
tree
structure
and-
and
you
can't
have
the
you-
can't-
have
a
cycle
where
one
thing
depends
on
another
that
worked
out
really
great
for
openstack,
but
this
is
also
kind
of
one
of
those.
It's
an
illustration
of
how
we,
as
a
community,
have
evolved
over
the
years
as
more
people
who
aren't
openstack
started
using
zuul.
They
said
we
don't
have
that
restriction.
In
fact,
our
software
works
a
lot
better.
E
If
we
don't
do
that,
you
know
we're
we're
doing
things
like
so
some
of
the
other
zulu
users
now
are
are
in.
Like
the
in
the
manufacturing
space.
We
have
automobile
manufacturers,
embedded
systems,
manufacturers
stuff
like
that
and
they're
like
we
don't
have
that
upgrade
cycle.
You
know
if
we're
upgrading
the
firmware
on
a
car,
we
kind
of
do
the
whole
thing
all
at
once.
We
don't
we
don't
we
don't
just
upgrade
the
the
entertainment
system
and
then
do
a
stepwise
upgrade
of
the
you
know
of
the
transmission
control
system.
E
Right
like
that,
doesn't
make
any
sense.
So
so
they
said
circular
dependencies
are
really
important
for
us,
because
that's
that
is
how
we
do
our
development
and
so
we've
actually
added
the
the
ability
to
do
circular
dependencies
and
they
work
kind
of
like
how
you'd
expect
or
how
you
would,
as
you
suggested
accidentally
do
right.
Where
you
you,
you
know
you
stick.
This
depends
on
header
and
change
a
that
points
to
b
and
then
do
the
same
thing
in
b
that
points
to
a.
E
If
you
do
that,
zool
will
recognize
that
that
is
a
circular
dependency
and
then
it
actually
changes
its
algorithm
a
little
bit.
It
says:
I'm
I'm
not
going
to
do
a
strict
ordering
between
these
I'm
going
to
recognize
that
all
of
the
the
participants
in
this
cycle
need
to
be
tested
together
and
then
merged
or
not
merged.
F
E
Unit,
so
it's
still
kind
of
the
the
graph.
Those
will
make
still
kind
of
looks
like
it
normally
does,
except
that
there's,
you
know,
potentially
a
a
grouping
of
changes
that
behave,
that
move
together
and
and
all
of
the
changes
in
that
group
have
all
of
the
get
changes
checked
out
for
them
at
the
same
time
as
well.
E
So
I
think
for
me
to
to
kind
of
sum
up
this
line
of
of
thought.
Zul
zul
is
a
tool
that
lets
you
ask
what,
if
right,
it's
about
saying
what,
if
I
merge
this
change,
what
if
what
if
these
two
changes
that
depend
on
each
other,
merge
what
if
this
other
dependency
emerges
like
what
what
is
what's
going
to
happen
in
that
case,.
F
E
I
think
that's
where
the
the
power
comes
from
just
you
know.
I
I
mentioned
some
of
these
other
we've
been
talking
about
openstack
and
kubernetes.
A
lot
right.
So
these
are.
These
are
some
of
the
other
zulu
users
that
are
out
there.
You
can
actually
on
our
website.
A
lot
of
these
users
have
stories
posted
there
about
how
they're
using
zuul.
So
if
you
wanna
read
about
them.
E
An
interesting
one
here,
because
they
they
perform
sort
of
cross-community
testing
they're,
really
at
the
heart
of
what
I've
been
talking
about.
They
do
cross-community
testing
between
openstack
and
and
kubernetes.
So
they're.
You
know
if,
if
you
were
to
to
do
this
whole
sender,
secure
delete
thing
that
I
was
doing.
That's
probably
the
most
likely
place
for
that
job
to
run
because
they're
doing
that
kind
of
testing
today
between
openstack
and
kubernetes.
E
F
E
These
are
some
places
you
can
find
out
some
more
information
about.
Zuul
we've
got
a
website
at
zuluci.org,
where
you
know,
we've
got
docs
linked
to
our
mailing
list,
our
irc
channel
on
on
freenode,
where
you
can
chat
with
us
and
that
sort
of
thing
the
docs
have
instructions
for
getting
started.
So
you
can
get
up,
get
a
zulu
up
and
running
pretty
quickly.
E
E
You
can
see
the
example
jobs
that
we've
got
the
standard
library
of
jobs
so
that
you
know
if
you,
if
you
do
usual,
there's
a
whole
community
of
of
folks
working
to
make
reusable
jobs
so
that
you
don't
have
to
start
from
scratch
and
then
we've
got,
and
you
know
what
I
this
is
one
of
those
things
where
it
for
me.
It
almost
goes
without
saying,
but
I
should
say
it
and
I
can't
believe
we've
gotten
this
far
without
me,
actually
saying
the
words
open
source.
E
Of
course,
zul
is
a
community
developed,
open
source
piece
of
software,
we're
you
know
we're
not
driven
by
one
company.
This
is
actually
you
know
a
bunch
of
a
bunch
of
people
working
for
a
bunch
of
different
companies
as
well
as
some
people
working
on
their
own
time,
contributing
to
to
make
this
thing
better.
E
C
E
E
Yeah,
so
it
is
written
in
in
python
and
it's
actually
kind
of
a
it's
a
sort
of
microservice
architecture,
and-
and
so
it's
a
it's
a
bunch
of
different
python
services
that
that
work
cooperatively.
So
there's
there's
a
web
service,
there's
a
scheduler
that
does
all
of
the
deciding
you
know
what
to
do
with
what
jobs,
to
run
that
sort
of
thing
and
then
the
executors
are
sort
of
they're
our
most
important
scale-out
feature.
E
So
one
of
the
things
one
of
the
pain
points
early
on
actually
was,
as
as
openstack
was,
was
growing
exponentially.
We
had
to
scale
our
ci
system
to
to
match
it.
So
so
we
actually
the
the
execution
component
of
zul
is,
is
a
scalable
component
as
well:
oh
slow,
computer!
There
we
go
so
if
we,
if
we
look
at
this,
is
the
the
grafana
instance
for
opendev,
and
if
we
look
at
it,
we
can
see
down.
E
E
F
E
E
Maybe
eight
nine
hundred
test
nodes
right
now,
so
these
are
about
eight
or
nine
hundred
virtual
machines
that
are
that
are
in
use
right
now
running
running
tests,
so
so
yeah,
the
the
the
stack
internally
is
python
and,
like
I
said
micro
services
for
for
scaling
out.
We
at
this
point
we
so
we're
we're
kind
of
agnostic
about
whether
you
run
zul
itself
in
containers
or
not
you
can
you
can.
E
You
know,
install
it
on
a
bear
server
or
a
vm
or
whatever
in
in
the
way
that
you
would
normally
install
install
a
python
program.
But
I
think
at
this
point
most
of
the
larger
zul
installations
are
running
in
containers.
Some
of
them
are
running
zul
inside
of
kubernetes.
Some
of
them
are
are
just
running
it
inside
of
docker.
E
So
it's
pretty
easy
to
to
just
run
the
exact
container
images
that
we're
running
in
opendev
for
zul's
own
zul.
Let's
see
another
important
thing
to
know
about
the
tech
stack
is
how
the
jobs
are
written.
They
are
written
in
ansible
and
that's
because
zul
is
a
multi-node
aware
system.
So
you
know
with
some
ci
systems
they
they've
they've
got
the
idea
of
one
test
node
down
for
running
a
job
right.
E
But
if
you
wanted
to
say
you
know,
you're
you're
doing
work
with
kubernetes
and
you
wanted
to
actually
stand
up
a
full
three-node
kubernetes
cluster.
With
with
zul.
You
can
do
that
pretty
easily
you
can.
You
can
ask
zul
for
one
node,
you
can
ask
zul
for
three
nodes
for
your
job
and
enzool
will
happily
provide
you.
E
E
So
if
you,
if
you
know
ansible
and
you
like
ansible,
then
that'll
be
really
familiar
to
you.
If
you,
if
you're
like
oh
well,
actually
you
know
I
don't.
I
don't
want
my
ci
system
to
be
opinionated
about
what
you
know.
How
I
do
my
deployment,
that's
fine,
too
ansible,
as
it
turns
out,
is
really
good
at
running
other
stuff.
E
So
if
you
just
have
a
shell
script
to
do
whatever
it
is
that
you
need
to
do
that's
fine,
you
can
write
two
lines
of
ansible
that
say,
run
the
shell
script
on
the
on
the
test.
Node
and
you're
done
you've
escaped
ansible
and
you
know
you
can
you
can
do
whatever
you
need
to
yeah?
I
think
that's,
the
the
the
highlights
of
the
the
tech
stack
there.
F
E
Postgresql
they're
all
supported
and
then
sort
of
behind
the
scenes.
We
have
a
zookeeper
system
for
helping
those
components
communicate
with
each
other,
but
those
are
all
kind
of
behind
the
scenes
details.
You
know.
As
a
zuul
administrator,
you
might
need
to
know
that
they're
there
as
a
little
user,
you
don't
need
to
know
that
they
exist
at
all,
so
so
yeah.
This
is
a
this
is
the
the
current
utilization
of
zul
on
opendev,
and
this
is
the
the
actual
web
interface
for
zul.
E
E
The
the
cryptography
and
the
packaging
folks
have
some
have
some
projects
in
the
system,
because
we're
actually
all
of
the
communities
involved
here-
are
heavily
dependent
on
these
python
packages
and-
and
we
want
to
make
sure
that
this
is
going
to
that-
that
cross
community
testing
that
I
was
getting
at
earlier.
We
want
to
help
the
python
community
make
sure
that
that
they're
able
to
test
on
all
the
platforms
that
they
that
are
important
to
them
and
are
important
to
us
too.
F
E
E
The
check
pipeline
is
what
happens
when
people
upload
changes,
so
this
is
kind
of
before
they've
when
you
write
a
patch
and
you
upload
it
for
the
first
time.
This
is
when
we
run
the
the
initial
test-
I've
even
not
gone
through
code
review,
yet
or
or
that
sort
of
thing.
So
if
we
actually
go
ahead
and
expand
by
default
here.
E
So
if
we
look
at
this
there's
a
bunch
of
in
progress,
some
have
succeeded.
These
jobs
are
paused.
All
of
these
jobs
are
running
for
this
change
right.
So
so,
each
of
these,
each
of
these
little
little
boxes
here
is
a
change
to
a
different
project
in
the
system,
and
then,
if
you
expand
that
you
can
see
jobs
that
have
all
of
the
different
jobs
that
we
run
for
that
change,
so
a
bunch
here
have
succeeded.
Some
have
failed.
E
E
So
these
lines
between
changes
are
our
dependencies
and
then
the
gate
pipeline
is
where
we
changes
that
are
about
to
merge
right.
So
here's
a
queue
of
two
related
changes
to
airship
projects
right.
So
this
one's
going
to
merge.
First,
this
one
is
being
tested
with
it
in
sequence
and-
and
I
actually
don't
know
if
this
is
a
git-
commit
dependency
or
or
just
a
a
dependency.
That's
formed
by
the
you
know
the
approval
sequence
or
it
depends
on
or
what
we'd
actually.
F
E
This
one,
these
two
are
sharing
a
cue,
but
you
can
see
that
this
one,
the
one
at
the
top
here,
has
failed,
which
means
that
zul
has
decided
that
this
one
no
longer
depends
on
it.
So
we're
still
running
the
jobs
on
this
because
we're
gonna,
we
wanna.
You
know
we've
sunk
a
lot
of
time
into
running
these
tests
already,
and
we
want
to
report
back
to
the
user
as
much
information
as
possible,
so
we're
going
to
keep
running
the
jobs
until
until
everything's
done.
E
E
F
C
E
E
E
F
F
E
E
Yeah
we
can,
we
can
access
the
logs
right,
so
we've
we've
got
the
logs
here
in
in
our
log
storage
system.
So
you
know
you
can
see
the
job
output
there's
a
whole
host
like
we
collect
a
whole
bunch
of
logs.
When,
when
we're
doing
this,
I
kind
of
want
to
show
you
the
what
it
looks
like
when
it's
finished
reporting,
because,
frankly,
because
it's
easier
to
use
and
more
there's
some
some
neat
features
there.
E
As
long
as
the
change
is
still
on
the
progress
you
know,
which
is
what
we've
got
here
right-
we're
still
running,
which
one
we're
still
running
this
job,
which
means
zul,
hasn't
like
it's
not
final,
yet
zul,
it's
clearly
not
going
to
merge,
but
but
it
hasn't,
zuul
hasn't
made
a
definitive
determination
because
all
the
the
jobs
aren't
finished
yet
so
it
it.
It
hasn't
yet
reported
back
to
the
code
review
system.
The
final
results
things
could
still
change.
E
E
E
This
is
the
this
is
the
ansible
task
that
failed,
and
this
is
the
last
few
lines
of
it.
So
often
as
a
developer,
you
can
hop
onto
this.
You
know
this
build
summary
page
and
and
actually
just
see
the
error
that
tanked
your
job
right
there.
This
doesn't
mean
anything
to
me.
I
don't
know
if
it
means
anything
to
you.
Amy.
F
E
E
F
E
F
E
And
this
is,
you
know
something
that
we've
we've
learned
after
years
in
the
trenches
in
in
openstack
development
right
this.
This
job
took
an
hour
to
run,
and
my
goodness,
after
the
end
of
that,
if
it's
failed,
you
sure
want
to
figure
out
why
it
failed
right.
So
that's
why
we
collect
so
much
log
information
so
that
when
when
something
goes
wrong,
hopefully
we've
got
everything
that
a
developer
needs
to
figure
out
what
that
was,
and
they
can
go
back
and
see
everything
that
happened.
E
If
the
zul
has
this
artifact
system,
so
you
might
we
don't
this
particular
job
doesn't
have
a
lot.
We've
just
got
this
script
to
download
all
of
the
logs
and
then
the
zul
manifest
itself.
But
examples
of
other
artifacts
that
we
have
are
documentation,
builds
preview
sites
for
documentation.
So
if,
if
your
job
builds
documentation,
you
can
just
add
a
link
here
and
then
developers
and
reviewers
can
click
on
it
and
and
see
the
see
the
see
a
preview
of
the
doc
build
container
images.
E
We
have
a
a
view
of
the
that's
a
big
one,
a
view
of
the
the
tasks
in
the
ansible
playbook
that
make
up
the
job.
So
this
is
the
really
big
thing
that
I'm
gonna
hide.
But
if
you,
if
you
look
at
this,
the
the
job
has
made
up
a
bunch
of
playbooks
and
each
of
those
have
tasks.
And
you
can
you
can
kind
of.
B
A
E
E
This
will
get
you
up,
and
so,
if
you're,
if
you're
a
hands-on
sort
of
person,
this
will
get
you
up
and
running
just
on
your
local,
laptop
or
workstation
or
whatever
using
docker
compose.
So
this
will
bootstrap
an
entire
zool
system
using
docker
compose
and
you
can
you
can
play
with
it
there
we're
actually
in
the
process
of
expanding
this
tutorial
system
now,
so
that
you'd
start
with
this
and
then
build
on
that
for
for
other
tasks
there
are.
E
There
are
some
other
parts
of
the
zulu
documentation
that
kind
of
tackle
this
from
different
perspectives?
So
you
know
if
you
just
want
to
read
more
about
more
about
zuul
concepts
and
things
like
that.
We've
got
discussion
session
sections
there
and
I
think
there
are.
You
know
some
videos
on
youtube
of
conference
presentations
and,
and
things
like
that
and.
F
E
Finally,
there
is
a
zuul
from
scratch
section
in
the
documentation
as
well,
so
it's
kind
of
the
opposite
of
the
quick
start,
whereas
the
quick
start
is
like
just
get
me
a
zul
up
and
running,
so
I
can
play
with
it.
If,
instead,
you
are
more
of
the
no
you
know
actually
I
want
to.
I
want
to
start
with
git
clone
and
and
go
from
there.
You
can
do
this
from
scratch
thing.
So
yeah
I'd,
I'd,
love
to
have
you
know
more
of
the
ebook
style
stuff,
but
we
do
have.
F
A
C
No
we're
I
mean
I
have
other
questions
basically
having
worked
on
the
kubernetes
project
and
prow,
where
once
again,
I'm
looking
at
this
and
saying
we
didn't
learn
nearly
enough
from
zuul
yeah.
Although
our
biggest
challenge
is,
we
have
looked
at
the
sort
of
multi-project
dependencies
and
our
biggest
blocker.
There
is
running
out
of
cloud
resources
to
actually
run
the
tests
yeah
the
because
you
know
given
microservice
architecture
there.
There
are
more
projects
in
kubernetes
than
there
than
there
are
in
openstack.
A
Informative,
you
thank
you
and
without
further
ado,
we'll
see
everybody
soon
coming
up
tomorrow
on
the
channel,
we
will
have
an
open
shift
commons
briefing
about
cloud
native
network
functions.
So
if
you're
curious
about
the
telco
space,
which
I
know
some
of
these
people
on
this
call
are
tune
in
and
we
will
discuss
yeah.
C
D
Yeah
we'll
have
belmaro,
who
is
coming
to
talk
to
us
about
scaling
at
cern
on
openstack,
so
that'll
be
a
really
good
talk
because
they're,
the
largest
user
of
openstack
in
the
world,
so
yeah
we're
kind
of
two
open,
stacky
related
things
in
a
row.
I
I
personally
take
zul
for
granted
because
I
use
it
all
the
time,
so
it
was
really
great
to
get
some
more
of
the
history
and
some
of
the
more
in-depth
usage
which
I'll
go
and
use
daily
now.
So
I
really
appreciated
this
gym.