►
Description
Event-driven Distributed CI for 20000 Projects - Aleksandra Fedorova, Red Hat
Speakers: Aleksandra Fedorova
Linux distributions represent the ultimate challenge for continuous integration, because of the size and variety of the projects involved. For example Fedora Linux contains about 21 thousand independent components. These components can be stand-alone applications written in high-level languages or kernel modules which require a bare-metal hardware for testing. Rather than enforcing one specific implementation of the CI/CD infrastructure to be used across components, Fedora Gating framework allows to aggregate test results of arbitrary CI systems in a common database. The aggregated results are then processed by the decision engine, which ensures that the system-wide gating requirements are satisfied. In the talk the architecture of the framework and the role of the messaging in Fedora infrastructure will be described, covering both the benefits and the challenges of this approach.
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
A
Hello,
everyone
and
welcome
to
a
talk
on
the
event-driven
distributed
ci
for
20
000
projects.
Let
me
start
with
the
introduction.
First,
I'm
alexandre
foderwa.
I
have
phd
in
geometry
and
topology,
but
I
worked
as
a
support
engineer
as
his
admin
devops
engineer
and
I'm
now
a
principal
software
engineer
at
red
hat
and
I've
worked
mostly
on
continuous
integration
for
fedora
center
stream
and
red
hat
enterprise.
A
Linux,
I'm
also
a
member
of
a
fedora
console
and
I'm
one
of
the
organizers
of
the
fedora
ci
special
interest
group,
and,
as
you
can
guess,
we
today's
talk
will
be
about
continuous
integration
in
application
to
the
linux
distribution.
So,
let's
start
with
the
setting
with
setting
the
context
first,
what
is
a
linux
distribution?
A
One
can
give
different
answers
to
this
particular
question,
but
for
the
purpose
of
this
talk
we
will
consider
linux
distribution
to
be
the
integrated
pool
of
packages.
So
we
have
a
repository
which
is
stores
packages
in
a
certain
format,
and
then
users
of
linux
distribution
can
choose
subset
of
s
components,
install
on
their
environment
and
get
a
working
system.
A
So
essentially,
linux
distribution
represents
the
ultimate
integration
channel
challenge,
because
a
user
can
choose
any
subset
of
components
from
the
repository
install
on
basically
any
target
environment
and
expect
that
these
components
will
work
together
and
we
have
wide
range
of
applications.
We
have
personal
computers,
so
we
have
large
scale
data
center
deployments.
We
have
single
board
computers,
so
we
have
containers
or
cloud
images.
A
A
So
this
is
in,
of
course,
we
don't
install
any
component
with
every
other
component
and
we
have
tools
like
package
management
which
restricts
further
the
possible
compatible
groups
of
packages
which
you
can
take
from
the
repository,
but
essentially
it
is
still
like
the
integration
of
everything
with
everything
and
it's
not
only
just
the
integrated
pool
of
packages.
It's
also
a
continuously
integrated
pool
of
packages,
because,
even
though
you
may
have
heard
that
link
distributions
have
longer
release
cycles,
some
have
half
a
years
or
a
cycle,
some
have
longer
waiting
periods,
but
this
release
cycles.
A
This
is
what
targets
the
user
applications
in
user
environments,
but
development
of
a
linux
distribution
is
ultimately
continuous.
So
we
develop
linux
distribution
by
updating
individual
packages
in
that
large
pool
of
packages
which
is
available
and
every
single
package
update
where
we
get
new
sources,
we
put
them
in
the
package.
We
get
this
new
package
with
the
updated
version
and
we
put
it
in
the
same
pool
to
the
rest
of
the
packages
and
we
integrated
with
the
rest
of
the
packages.
A
For
example,
linux
distribution
have
this
interesting
feature
of
being
dependent
on
itself
because
with
tool
chain,
the
tools
which
you
use
build
to
build
linux
distribution
are
part
of
that
linux
distribution
and
are
updated
for
the
same
ci
process,
and
this
creates
additional
challenges,
for
example,
for
reverting
packages,
because
you
don't
do
a
clean
reverb
of
a
package
if
this
package
was
used
to
build
the
some
other
packages
in
in
the
distribution,
so
we
are
not
going
to
talk
about
all
of
that.
We
will
focus
on
this
ui
today,
but
yeah.
A
There
is
a
interesting
topics
in
in
the
linux
release
engineering
world.
Now,
on
top
of
that
interesting
challenges,
we
also
have
just
the
matter
of
scale.
Linux
distributions
are
indeed
huge.
I
have
these
numbers
taken
just
for
with
the
random
examples,
so
for
community
distributions,
for
example
debian
you
do
have
the
more
than
30
000
packages
and
in
fedora
you
have
more
than
20
000
packages.
A
A
A
A
A
So
let's
look
at
the
architecture
of
this
first
and
then
we
will
cover
some
more
technical
details,
so
the
basic
or
concept
of
the
event-driven
distributed
design
we
have
is
the
concept
of
a
ci
system.
Ci
system
is
a
certain
engine
which
knows
how
to
run
a
certain
test.
You
can
think
of
jenkins
instance,
which
knows
how
to
run
specific
pipelines,
but
it
can
be
any
kind
of
implementation
in
any
kind
of
way.
It
just
knows
how
to
run
specific
tests
for
a
certain
for
a
component.
A
A
A
So
every
time
we
have
a
new
package
produced
by
the
build
system,
build
system
sends
a
message
to
our
message:
bus
that
a
new
package
is
new
package.
Update
is
now
available,
our
ci
systems
listen
to
the
message,
bus
and
they
recognize
the
message
of
this
particular
type
and
they
trigger
their
internal
logic.
A
When
we
get
this
message,
some
systems
may
trigger
tests.
Some
systems
may
skip
those
does
some
some
systems
take
trigger
different
tests
based
on
which
component
is
involved
in
which
component
is
mentioned
in
the
message
when
ci
system
triggers
the
test,
it
runs
the
test
in
whichever
way,
and
then
once
done
it
sends
the
test
outcome
as
a
message
to
the
same
message.
Bus
test
outcome
can
be
failed
or
can
be
passed
or
can
be
a
infrastructure
error.
A
Once
message
about
the
test
outcome
appears
on
the
message:
bus
we
have
a
service
which
is
called
resultdb,
which
listens
to
all
kind
of
test,
results,
messages
and
aggregates
them
into
one
database.
This
database
is
just
a
simple
data
store
within
api,
which
allows
us
to
put
all
the
test
results
in
one
place.
A
This
system,
what's
important,
is
that
this
system
doesn't
put
any
requirements
on
the
implementation
of
a
ci
system.
It
can
be
jenkins,
but
it
can
be
your
own
custom
script
or
it
can
be
a
gitlab
ci
or
it
can
be,
whichever
you
use
in
whichever
you
find
comfort
suitable
for
your
specific
test
scenario.
A
For
example,
it
can
also
be
a
manual
ci
system,
which
means
instead
of
having
in
a
system
with
transautomated
tests.
There
will
be
a
person
which
watches
after
the
new
build
messages
on
the
message,
bus
and
once
a
person
recognizes
that
there's
a
new
build
coming,
a
person
can
treat
your
scenario,
perform
the
test
and
then
send
the
manual
test
result
as
a
message
to
the
message,
bus
and
get
it
aggregated
with
the
rest
of
the
testing.
So
it
allows
us
to
put
all
kind
of
testing
into
the
same
common
framework.
A
We
also
have
a
concept
of
group
testing
where,
rather
than
testing
individual
packages
and
sending
them
through
the
gate
or
one
by
one,
we
have
cases
where
packages
have
are
tightly
coupled
and
have
dependencies
on
each
other,
and
so
we
need
to
test
a
group
of
packages
is
one
unit
and
send
it
through
the
gate.
In
this
case,
rather
than
triggering
on
new
build
messages,
we
have
a
possibility
to
initiate
the
mass
of
the
testing
by
sending
the
specific
kind
of
group
trigger
to
the
message.
Bus
and
certain
person
can
comment
say.
A
A
We
also
have
some
extended
variance
of
this
messaging
exchange.
For
example,
we
have
this
ability
to
do
a
chain
testing,
because
our
package
is
not
the
final
artifact
which
we're
building.
We
can
build
containers
using
that
package.
We
can
build
repositories
using
that
package
and
iso
images
and
so
on.
A
Then
I
was
talking
about
how
all
the
test
results
land
in
the
results
db
as
a
storage,
but
it's
not
a
final
step
in
the
gating
process,
because
once
we
have
these
test
results
stored
in
one
place,
we
also
need
a
way
to
make
decisions
on
whether
these
test
results
are
good
enough
for
package
to
be
promoted
to
a
next
step
and
this
promotion
step
of
the
decision
making
step.
We
also
do
over
message,
bus
and
over
events.
A
This
works
as
follows.
So
we
have
a
results:
db,
accommodating
the
test
results
and
once
result
they
be
get
an
update.
It
issues
a
message
itself
that
results
were
updated
and
then
we
have
decision
engine
which
we
call
green
wave,
which
reacts
on
such
type
of
messages
and
it
recalculates
the
decision
for
the
artifact.
A
This
framework
is
quite
flexible
and
it
also
allows
us
to
manage
multiple
gates.
I
was
talking
more
about
the
package
level
gating
in
the
previous
diagrams,
but
we
actually
have
more
than
one
gate
and
more
than
one
type
of
a
gate,
we
can
test
merge,
requests
and
decide
whether
we
merge
requests
can
be
accepted
or
not.
We
can
test
packages
which
are
produced
from
a
merge
code
and
also
decide
whether
package
can
be
promoted
to
the
next
step.
A
We
also
build
repositories,
and
then
we
decide
whether
the
repository
can
be
promoted
to
a
nightly
status
based
on
the
gating
information
in
ci
information
and
then
all
those
gates,
ci
systems,
work
the
same
way.
So
once
you
onboarded
your
ci
system
to
one
of
the
gates,
it's
quite
easy
to
extend
your
reach
and
run
the
same
test
as
a
part
of
emerge,
request,
check
or
as
a
part
of
a
repository
check.
A
So
this
is
the
architecture
overview.
Let's
talk
briefly
about
tools
and
projects.
We
used
to
implement
that
we-
I
have
a
lot
of
links
here.
I
will
not
go
read
all
of
that.
I
will
only
cover
the
basics,
so
we
have
a
message.
Bus
message.
Bus
is
based
on
a
rabbitmq.
It's
publicly
available.
Anyone
can
read,
you
need
authentication
to
write,
but
all
the
data
is
open.
We
have
ci
messages,
schema
it's
a
repository
with
json
schema,
ci
messages,
look
like
large
json
files.
You
cannot
read
this
one
probably,
but
these
are
examples.
A
In
the
repository
of
eci
messaging
schemas,
which
I
mentioned
earlier,
we
also
for
ci
systems.
We
use
jenkins.
We
have
a
jms
messaging
plugin
for
jenkins,
which
allows
us
to
trigger
pipelines
based
on
messaging,
on
repeating
google.
We
have
groovy
library
to
provide
some
high
level
abstractions
on
top
of
raw
messages,
and
our
pipelines
are
already
also
public
and
they
are
available
on
github
as
jenkins
files.
A
A
The
results
they
be
is
a
postgres
database
with
a
custom
api
on
top
of
it
and
the
green
wave
decision
engine
is
also
an
open
source
project
available
on
available
by
those
links,
and
you
can
define
various
gating
policies.
Getting
policies
are
like
list
of
requirements,
and
the
important
point
about
green
wave
is
that
gating
policies
can
be
adjusted
per
component,
so
we
have
generic
tests
which
are
required
for
every
component
in
the
linux
distribution,
but
we
also
can
have
different
sets
of
required
tests
for
different
components.
A
So
kernel
tests
applied
to
kernel,
but
not
to
I
don't
know,
python
update
and,
of
course,
the
talk
wouldn't
be
complete
without
talking
about
challenges
in
future
work
areas,
we
do
have
a
as
a
any
kind
of
large-scale
infrastructure.
We
do
have
problems
with
reliability
on
messages
can
be
lost,
cis
systems
can
feel
tests
themselves
can
be
flaky,
but
these
are
more
of
a
generic
types
of
issues
which
I
I'm
sure
everyone
is
quite
familiar
with.
A
If
we
talk
about
more
cia
related
topics,
then
one
of
the
issues
is
the
lack
of
a
build
built-in
messaging
support
in
ci
engines.
I
mentioned
the
jenkins
plug-in
we
have,
but
in
other
ci
systems
it's
usually
hard
to
connect
the
messaging
to
the
triggering
of
eci
pipelines.
You
need
to
write
your
own
services
and
hooks
and
develop
a
system
which
does
it
in
every
single
case,
there's
also
issue
with
ci
messages.
We
do
evolve.
Our
ci
messaging
schema
over
time.
It's
now
versioned
and
we
add
or
refactor
some
of
the
parts
of
it.
A
There
is,
of
course,
an
issue
with
ci
systems
themselves,
while
from
the
architectural
point
of
view
with
3ci
system
as
a
black
box,
which
only
provides
yes
and
no
answers
on
the
testing.
There's.
Also
a
question
of
the
interaction
between
developer
and
ci
systems,
so
ci
system
needs
to
provide
a
readable
log.
It
needs
to
provide
the
data
about
the
health
about
the
reliability
of
a
test
result.
A
A
A
A
What's
the
interaction
on
messages
here
and
then
how
we
organize,
but
yeah
it
can
be
improved
and
actually
one
of
the
reasons
why
I
want
to
participate
in
events
driven
special
interest
group
that
we
can
actually
work
on
some
more
standardized
approach
to
this,
and
maybe
like
change
our
protocol
to
a
more
common
thing.
But
this
is
a
work
item
to
do,
and
the
next
thing
is,
since
multiple
ci
systems
may
listen
to
the
event
bus.
Do
we
know
that
the
testing
is
complete?
A
Okay
on
spec
is
there
a
plan
to
move
to
cloud
events
in
the
future,
as
I
said
like,
or
I
would
be
interested
to
converge
to
to
convert
to
some
common
format,
but
we
need
to
look
into
differences
and
approaches,
because
we,
currently,
our
messages,
are
very
artifact
focused
rather
than
events
focused.
We
send
messages
like
artifact
is
ready,
and
then
we
send
message
artifacts
tested
and
then
we
send
message.
A
A
So
the
main
reason
why
we
use
it
is
that
we
can
distribute
these
pieces
of
vci
to
different
teams
and
give
them
a
full
control
of
the
testing
of
a
certain
component
of
a
certain
type
of
tests.
We
maintain
generic
tests
for
all
packages
like
basically
package
can
be
installed.
Packaged
plays
well
with
the
rest
of
the
thing,
but
integration
tests
specific
for
a
component.
It's
a
deep
topic
and
every
team
knows
better
how
to
do
it.
A
So
yeah
emil-
I
I
definitely
see
like
we
discussed
these
same
questions
and
same
topics
yesterday
at
birth.
If
we
were
a
meeting
of
the
events
sick,
I
really
like
feel
like
we
are
very,
very
aligned
we
just
used
because
we
originated
from
different
places.
We
used
slightly
different
language,
but
there
is
a
lot
of
room
to
like
converge
on
something
and
work
together
and,
as
I
said
like,
we
are
open
like,
but
of
course
we
have
central
stream
and
rel
ci,
which
is
less
open
because
of
obvious
reasons.
A
But
fedora
ci
is
a
community
driven
ci,
and
this
is
where
we
can
really
work
on
prototyping
and
bringing
things
on
changing
things,
and
this
goes
through
the
community
process
for
the
decision
of
a
fedora
engineering
steering
committee
and
everyone
is
empowered
to
bring
changes
to
fedora
and
to
work
on
setting
it
up.
So
everyone
is
invited.
A
So
the
pipeline
is
distributed
and
run
completely
by
the
local
ci
systems
and
decision
engines,
there's
nothing
like
a
full
which
maintains
the
full
control.
Yes,
indeed,
we
do
have
a
additional
service,
which
we
call
ci
dashboard,
which
tries
to
aggregate
the
data
from
messaging
and
provide.
This
is
a
like
readable
format
for
developers
to
understand
the
overall
state,
but
this
service
is
really
a
read-only
front-end
to
the
whole
pipeline.
There
is
no
written
workflow
anywhere.
A
A
Missing
events
could
cause
a
lot
of
problems.
Yes,
yes,
they
do
cause
a
lot
of
problems.
So
how
do
we
deal
with
this?
We
have
some
checkpoints
where
we
monitor
that
all
packages
actually
passed
through
the
gate
or
not
passed
through
the
gate
during
the
certain
time,
and
then
we
try
to
like
search
what
blocks
we
package
from
going
through
and
we
have
possibility
to
retrigger
pipelines
outside
of
messaging.
A
I
think
what
can
be
related
is
that
when
ci
systems
send
messages
in
the
message
bus,
we
need
some
authority
that
this
is
the
exactly
vci
system,
which
sends
these
type
of
messages
and
no
impersonates
the
ci
system
and
cannot
sound
like
test
is
passing
from
from
somewhere
else.
So
we
have
permissions
on
the
levels
of
the
topics
in
the
message.
Bus,
not
every
one,
can
send
the
message
about
this
type
of
tests
on
the
message.
Bus
this
there
should
be
agreement.
A
A
Yeah,
oh
replying
on
the
comment
in
the
chat.
Definitely
any
kind
of
analytics
anything
you
can
you
can
do.
We
will
be
interested.
As
I
said,
all
our
messages
are
all
on
the
public
message:
bus,
it's
red
button
queue
you
can
just
listen
to
it
and
process
in
any
way,
and
we
have
messaging
schema
and
maybe
worth
mentioning
like
the
entire
life
of
fedor
infrastructure
is
on
that
messaging
bus.
A
I
talked
only
about
the
ci
part
and
the
promotion
and
gating,
but
fedor
infrastructure
actually
uses
message,
but
for
everything
this
means,
even
like
posts
on
the
forum
in
fedora
community
goes
eventually
will
send
a
message
to
a
message.
Bus
and
we
have
a
possibility
like
for
people,
to
read
these
messages
and
get
notified
if
we
want
to
and
so
on.
So
our
fedora
infrastructure
message.
Bus
is
a
very
quite
quite
cool
thing
because
it
it
like
shows
you
everything
about
large-scale
infrastructure
and
it's
fully
publicly
available.
A
So
if
you
have
any
ideas
how
it
can
be
used
and
and
shaped,
I
would
be
very
interested
in
hearing
that
and
then
like
in
any
ways
how
we
can
make
it
make
some
a
research
on
that
how
we
can
make
some
analysis
what
what
kind
of
data
we
can
get
through.
This
I
mean
machine
learning
would
be
interesting
also
to
attach
and
to
get
some
insights
and
how
releases
of
fedora
work,
but
that's
like
magical
topic
which
just
needs
time
and
effort.
Yeah.
A
Is
our
ccd
system
triggered
by
the
message
bus
as
well?
Actually,
yes,
our
cd
system
is
a
bit
more
complicated,
very
different
layers.
In
the
end,
we
we
have.
When
we
talk
about
ci,
we
usually
focus
on
lending
a
package
update
in
the
nightly
build
and
the
nightly
build.
Is
this
nightly
snapshot
of
your
repository
and
the
iso
image
built
from
it
and
which
we
published
to
mirrors
or
whatever
and
the
we
have
this
promotion
step
for
the
iso
image
and
repositories,
for
example?
A
So
we
built
this
snapshot
of
the
current
state
of
europa.
We
test
it
through
the
same
framework
with
the
messaging
and
with
ci
system
posting
a
test,
and
then,
if
testing
is
successful,
gating
system
triggers
the
publication
of
his
image
to
his
place
for
nightly
images
when
this
snapshot
is
good
enough,
but
there
are
like
when
you,
when
you
consider
red
hat
enterprise
linux
release
process,
it's
like
much
more
complicated
later
on
in
the
phases
very
different
steps
and,
of
course,
the
additional
process.
A
A
I
want
to
thank
everyone
for
the
interest
and
for
the
conversation
and
well,
let's
continue
some
discussions.
Whenever
we
can,
I
promise
I
will
try
to
join
the
events
sig
meeting
next
time.
I
recorded
it
in
my
calendar,
we'll
see
each
other
there
and
yeah
fedora
is
open,
so,
let's
meet
whenever
we
can
thanks.
Everyone
see
you
next
time.