►
From YouTube: CI/CD for Autonomous Drive and other complex systems - What we have learned about... Erik Sternerson
Description
CI/CD for Autonomous Drive and other complex systems - What we have learned about using events - Erik Sternerson, doWhile
Complex systems often imply complex builds, complex verification and complex releases, and complex CI/CD pipelines spanning multiple teams and departments. Hardware is often involved as well, adding an additional level of complexity.
Several challenges frequently present themselves in such CI/CD pipelines, mainly related to observability, such as understanding what has been built and packaged, what has been tested where and how, and what is ready to be released to "the next level". These observability issues are often solved in the same manner: using events.
In this talk, we will give an introduction to "industry-style enterprise-scale" CI/CD, present some challenges that are unique to this domain, and describe how the industry is working to overcome them using events.
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
A
Have
you
guys
heard
of
something
called
the
hungry
judge
effect?
Basically,
there
was
a
study.
I
think
it
was
in
the
u.s
a
few
years
ago
saying
that
criminals
who
were
up
for
parole
were
much
more
harshly
judged
just
before
lunch
than
they
were
after
lunch
or
early
in
the
morning
and
since
oleg
needs
to
catch
his
flight,
he
switched
times
with
me,
which
means
I'm
now
going
to
be
harshly
judged.
So
please
be
kind
to
me
and
we
will
get
through
this
just
fine
okay.
A
So
I
asked
sabine
like
who
decides
what
variance
we
should
combine
and
what
variants
we
should
test
and
she
looked
at
me
and
said
eric.
Nobody
knows
how
to
build
a
car,
and
I
thought
well
that
can't
be
true.
We
build
cars
all
the
time.
I
can
see
them
rolling
out
the
factory
down
the
street,
so
I
just
said
probably
she's
exaggerating
for
effect
and
didn't
think
too
much
about
it.
A
But
I
came
to
a
conclusion
quite
a
long
time
later,
a
new
understanding
of
of
this
statement
and
it's
not
it's
impossible
to
build
a
car.
It's
more
like
no
single
person
knows
everything
that
goes
into
a
car,
all
the
components,
all
the
configuration,
all
the
hardware
and
stuff
that
goes
into
it.
So
it's
impossible
for
any
one
person
to
keep
track
of
everything.
A
So
a
car
gets
built
through
collaboration
and
collaboration
requires
communication.
So
my
name
is
eric
stanischan.
I
work
for
dual.
I
also
work
for
volvo
cars
and
volvo
autonomous
systems,
and
I'm
here
today
to
talk
about
continuous
integration,
continuous
delivery
in
autonomous
drive
and
other
complex
projects,
with
a
focus
on
events,
because
that's
why
we're
here
so
my
day-to-day
typically
involves
building
projects
like
this
so
to
the
left.
We
have
a
autonomous
mining
vehicle,
just
runs
around
a
mine
catching
or
bringing
stuff
from
one
place
to
another
and
to
the
right.
A
But
it's
it's
got
to
be
perhaps
slightly
different,
but
what
I'm
going
to
claim
is
that
we
also
do
ci
cd
and
we
also
want
the
seat
at
the
table.
So
I'm
happy
for
any
chance.
I
get
to
yeah
to
talk
about
what
we
do
so
and
we're
going
to
talk.
As
I
said,
a
little
bit
about
ci
cd
for
autonomous
drive
we're
going
to
talk
about
how
we
and
we
I
mean,
like
the
this
kind
of
industry,
are
transitioning
to
events.
A
But
I
want
to
start
by
looking
into
just
what
do
we
mean
by
a
complex
system?
So
a
complex
system
or
a
system
can
be
complex
for
for
many
reasons
it
can
either
be
that
it
has
some
like
really
advanced
algorithms,
maybe,
for
instance,
in
the
autonomous
drive
context.
It's
the
algorithms
that
look
at
a
road
and
try
to
figure
out.
Where
are
the
road
signs
what's
what's
going
on
in
the
world
around
me?
So
those
would
be
pretty
complex
algorithms,
but
there's
another
side
to
complexity.
A
When
you
have
many
parts
like
many
different
computers,
you
have
the
tire
pressure
sensor.
You
have
the
braking
sensor.
You
have
steering
you
have
window
or
windshield
wipers
and
all
that
stuff,
all
those
typically
have
computers
connected
to
them.
So
it's
both
complex
in
a
lot
of
parts
and
complex
in
advanced
algorithms.
A
If
we
were
to
try
to
build
a
car,
it
really
wouldn't
be
feasible
to
just
take
all
the
individual
components,
all
the
software,
all
the
hardware
and
just
dump
them
into
a
car
and
test
it.
First,
some
components
just
will
not
work
together.
The
first
time
you
drive
it
and
the
more
components
you
have
the
harder
it
will
be
to
pinpoint
where
the
actual
problem
is.
A
A
So
these
domains,
typically
not
that
related
to
the
product,
they're
more
related
to
the
organization,
because
you
have
on
the
order
of
many
hundreds
of
teams,
and
you
can't
just
like
yeah,
you
can't
have
them
all
on
one
layer.
So
to
say
it
gets
too
complex,
so
you
break
it
down
into
mains.
You
break
it
down
into
function,
areas
or
break
it
down
into
functions,
and
eventually
you
end
up
with
an
actual
component
somewhere.
A
So
yeah,
what
we
see
in
this
picture,
longitudinal
collision
avoidance,
is
a
fancy
word
for
don't
run
into
the
car
in
front
of
you
and
yeah.
You
can
see
that
it's
you
group.
It
then,
with
more
and
more
components
into
these
functions,
function,
areas
and
domains,
and
the
idea
is
that
these
groups,
which
we
will
call
compositions
as
you
see
it's
not
just
one
component.
There
are
some
some
ellipses
over
here
stating
that
there
are
more
components,
but
each
composition
should
be
meaningful
to
verify
together.
A
A
I
know
that
several
people
in
previous
talks
has
talked
about
integration
as
a
bad
word,
but
in
a
car
it's
really
important.
We
don't
actually
expect
the
people
who
own
the
car
to
some.
They
randomly
replace,
replace
the
computer
that
controls
the
brakes
and
put
in
some
other
thing
there.
So
we're
not
really
looking
for
interoperability.
We're
looking
for
integration.
A
A
Well,
the
previous
answer
was
about
two
or
three
times
per
year,
because
that
was
was
how
often
we
were
able
to
to
bring
components
in
and
test
them,
but
that's
never
going
to
hold
if
you
want
to
well,
if
you
want
to
have
happy
developers
for
one,
it's
really
annoying
as
a
developer,
to
have
to
wait
six
months
for
your
verification
results,
you
will
have
forgotten
what
you're
doing
long
before
then.
So
we
want
to
do
it
continuously,
or
at
least
as
often
as
we
can.
A
A
If
you
go
to
the
cd
foundation
in
their
frequently
asked
questions,
they
will
define
continuous
integration
as
the
practice
of
merging
all
developer
working
copies
to
a
shared
main
line
several
times
per
day.
So
this
is
like
taking
git
branches
or
git
commits
and
merging
them
to
main.
That
is
what
continuous
integration
is
in
the
official
definition.
A
What
we
could
define
for
this
talk
as
sort
of
our
continuous
integration
would
be
frequently
and
iteratively
integrate
components
together
in
increasingly
complex
compositions,
fancy
way
of
saying,
don't
put
the
car
all
together
in
one
go:
do
it
in
smaller
parts
and
gradually
grow
those
compositions
until
you
have
the
full
vehicle,
a
picture,
I'm
really
not
going
to
have
to
have
that
time
to
go
into,
but
can
be
used
to
highlight.
One
thing
is
this,
so
I
would
claim
that
the
sort
of
base
definition
of
ci
or
the
cdf
definition
of
ci
lives
up
here.
A
Let's
talk
a
little
bit
about
before
events
first
so
before
we
started
using
events
in
volvo
cars,
for
instance,
then
releases,
which
is
the
thing
that
you
actually
want
to
put
in
the
car,
so
releases
of
software
components,
hardware,
components
and
configuration
they
were
sort
of
uploaded
to
some
network
drive
or
maybe
a
sharepoint
or
artifactor.
If
you're
lucky
and
then
someone
sent
an
email
saying
there
is
a
new
version
of
this
component.
A
Pipelines
were
typically
built
that
each
team
had
their
own
jenkins
instance.
They
mainly
manually
started
their
pipelines.
It
ran
for
some
time
it
produced
the
release
and
then
back
to
the
step
above
it
take
that
release,
put
it
somewhere
and
send
an
email
and
when
it
came
to
understanding,
quality
or
understanding
the
verification
results.
A
To
be
able
to
use
events,
we
need
some
infrastructure
first,
so
our
infrastructure
contained
or
was
based
on
a
vocabulary.
We
chose
a
really
simple
one:
it
had
two
types
of
events
and
it
was
mostly
object-based.
So
we
had
events
for
saying
I
have
created
a
new
object
and
other
events
for
saying
I
am
updating
an
object.
A
So
now
that
we
have
a
shared
vocabulary,
the
next
we
need
is
a
transport,
so
we
chose
rabbitmq
for
that,
just
to
make
sure
that
everyone
could
send
events
and
everyone
who
wanted
to
listen
for
events
and
finally-
and
this
is
not
strictly
mandatory,
but
it
was
the
approach
that
we
went
for.
We
built
a
triggering
system
and
we
did
it
ourselves.
A
So
the
triggering
system,
you
define
a
set
of
rules,
it's
actually
very
similar
to
what
the
shruti
showed
before
you
define
a
set
of
of
filters.
You
could
say
for
messages
or
events
saying
that
when
an
event
arrives
that
matches
my
filter,
then
I
want
to
take
this
action
and
the
action
was
typically
started.
Jenkins.
Job
somewhere
on
a
server
and
with
this
pretty
simple
starting
point,
we
were
actually
able
to
take
some
pretty
good
steps
from
this
picture
before
where
everything
was
slow
and
cumbersome
and
painful
to
a
much
more
event
driven
setup.
A
Some
important
events
that
we
had
first
was
dealing
with
releases
so
releases
and
we're
talking
again
about
software
releases,
hardware,
releases
and
configuration
releases.
They
are
what
we
actually
want
to
push
through
our
continuous
integration.
We
want
to
bring
them
from
the
bottom
level,
where
we
have
individual
components
to
the
top
level
where
we
have
a
full
vehicle.
A
If
you
remember
this
picture,
then
what
we
have
in
the
bottom
layer
would
be
components
so
built
binaries,
and
what
we
have
above
would
be
composition
of
several
components,
but
we
would
send
release
events
for
all
of
this,
and
just
with
that
first
event,
we
could
see
like
a
shift
in
our
how
people
were
talking.
They
were
more
like
saying,
grab
that
release
from
the
event
bus
rather
than
talking
about.
A
A
Something
called
chain
id,
so
a
chain
id
is,
you
can
see
it
like
a
context
basically
says
that
several
activities
that
share
the
same
chain
id
they
are
related
to
the
same
pipeline
or
the
same
pipeline
run
in
eifel,
which
email
will
talk
about.
They
have
a
concept
called
links
which
in
retrospect,
is
how
we
should
have
done
it,
but
that's
not
how
we
did
it,
and
I
will
talk
about
the
consequences
of
that
later
and
finally
for
activities.
It's
very
interesting
to
understand
the
result,
so
you
want
to
see
if
it
passed
or
failed.
A
The
yeah,
basically
on
a
abstract
level
or
high
level
in
every
transition
to
a
higher
level.
You
do
two
things.
First,
you
either
build
or
compose
a
new
release,
and
then
you
verify
it
so
verification
results
are
actually
or
you
need
to
have
sufficiently
many
good
verification
results
to
want
to
move
to
the
next
level
for
a
verification
result.
We
came
up
with
a
concept
or
introduce
the
concept,
I
would
say
called
confidence
labels.
A
A
We
have
the
confidence
label
itself
and
I
will
talk
more
about
what
that
is,
the
result
of
the
verification
and
links
to
it
so
that
everyone
receiving
the
event
if
they
are
interested
in.
Why
did
something
fail?
They
have
a
link
to
the
yeah
reports
or
blogs
or
whatever
for
the
verification,
so
that
everything
becomes
nicely
connected
together.
A
A
A
A
Typically,
that
means
it's
the
end.
In
that
branch
of
this
composition,
we
don't
want
to
build
a
higher
level
composition
of
something
that
we
know
doesn't
pass
system
safety,
because
if
we
get
up
to
the
car-
and
we
have
compositions
that
don't
have
system
safety,
then
the
car
might
crash
and
that's
not
good.
A
I
would
almost
say
it
forces
the
teams
to
think
about
what
is
the
confidence
that
we
produce.
What
are
we
verifying?
What
is
the
quality
that
we
are
checking
and
then
put
names
on
it,
and
these
names
become
very
powerful
for
communication,
because
if
I
were
to
look
at
your
test
results
and
just
see
okay,
you
have
these
five
test
activities
that
you
have
passed.
A
I
don't
necessarily
know
what
that
means
about
the
quality
of
the
product.
I
don't
know
if
it's
ready
for
me
to
take
to
the
next
level,
but
with
confidence
level
you
could
or
confident
labels.
You
could
say
yeah.
It's
passed
a
system,
safety
test,
it's
past
the
collision
avoidance
tests,
it's
past
various
other
quality
areas,
and
it
could
of
course
be
just
or
it
could
be,
both
manual
tests
and
automated
tests
and
reviews,
and
whatever
you
have
all
of
it,
can
contribute
to
these
confidence.
Labels.
A
If
the
team
continuously
sends
out
the
confidence
labels
that
they
have
reached,
then
I
as
an
observer,
can
just
say:
oh
now,
this
release
has
passed
or
has
the
confidence
labels
that
I
need
for
my
testing
and
then
I
can
pick
it
down.
I
still
have
to
verify
that
it
gets
all
the
confidence
labeled
before
I
push
it
upwards,
but
at
least
I
can
can
start
doing
some
testing
earlier
and
earlier
testing
means
earlier
feedback.
So
that's
good
for
everyone.
A
A
A
Then
anyone
else
can
can
pick
up
those
events
and
use
them
in
whatever
systems
they
feel
is
most
appropriate
for
them,
and
maybe
a
common
use
case
where
that
comes
up
is
like
oleg
said
jenkins,
is
an
automation.
Server,
so
jenkins
would
typically
live
on
our
side,
mainly
in
the
higher
layers,
because
then
we
are
doing
automation
in
the
lower
layers.
We
are
doing,
builds
and
things.
Then
it
might
be
more
stuff
like
zuul
or
gitlab
or
or
go
cd
or
those
kind
of
tools,
because
they
they
have
more
support
for
dealing
with
code.
A
So
not
forcing
everyone
to
use
the
same
cicd
tools,
which
has
been
very
appreciated
by
the
teams.
A
second
thing
we
get
is
observability,
so
there
is
a
concept
that
again
has
been
stolen
from
erickson.
I
think
we
call
it
follow
your
commit
or
follow
your
change
with
the
fact
that
events
are
connected
to
each
other
in
our
setup
is
connected
to
these
chain
ids.
A
We
can
have
a
scenario
where
an
individual
developer
can
follow
from
their
initial
change,
that
they've
done
through
all
the
verification
and
activities
that
has
been
done
through
higher
level,
compositions,
etc,
etc.
All
the
way
up
to
the
end
vehicle.
If
that
is
of
interest-
and
they
can
see
like
how
far
did
my
change
make
it?
When
did
it
stop
and
if,
if
I
go
look
at
the
results
where
it
stopped,
how
does
it
seem
like
it's
my
fault,
or
is
it
something
else.
A
So,
where,
before
again,
a
developer
would
have
to
go
and
contact
the
various
teams
to
understand
if
their
change
made
it
through,
they
can
now
look
at
the
visualization
or
observability
portal
and
see
what
is
going
on,
and
this
is
something
I'm
really
hoping
that
we
will
realize
with
or
implement
with
cd
events
as
well.
It's
really
helpful
in
my
opinion.
A
The
third
thing
we
get
is,
I
call
it
collaboration
without
causation,
it's
related
to
automation
again,
but
basically,
what
I'm
saying
is
a
team
doesn't
tell
the
next
team
that
now
you
should
start.
The
team
sends
the
events
that
they
are
responsible
for
and
the
receiving
team
can
say.
Okay
now
we
have
something
with
the
appropriate
events.
A
So
now
we
can
move
ahead
and
start
our
activities
so
and
in
fact
many
of
these
teams
don't
really
know
each
other
or
about
each
other
that
well,
we
have
one
example
where
we
receive
a
component
from
an
external
supplier.
In
this
case,
we
do
some
testing
in
gothenburg
in
sweden,
then
we
pass
it
off
using
events
to
a
data
cluster
in
stockholm
in
sweden,
where
some
more
stuff
gets
run.
Then
it
another
event
gets
routed
to
shanghai,
where
some
even
further
verification
goes
on,
and
then
it
goes
back
to
gothenburg
again
and
yeah.
A
A
We
chose
a
pretty
naive
approach.
I
would
say
again
when
it
comes
to
using
events.
I
think
I
would
like
to
do
a
little
bit
of
pr
for
cd
events
or
maybe
explain
why
I'm
really
enthusiastic
about
this
project,
because
we
yeah
we
made
some
mistakes
on
the
way
and
I
think
seed
events
is
looking
really
good.
One
thing
that
people
have
mentioned
already
is
with
interoperability.
A
We
don't
have
to
use
that
much
glue
code
anymore
or
write
that
much
glue
code.
We
spend
hundreds
or
thousands
of
hours
just
making
sure
that
we
could
get
our
jenkins
jobs
to
send
events,
that
we
could
send
the
same
events
from
zulu
and
from
go
cd
and
that
we
had
test
results,
published
the
correct
way,
etc,
etc,
and
with
cd
events
and
with
the
adaption
that
we're
already
starting
to
see,
then
it
will
be
much
more
of
a
pick
and
match
solution
to
this
teams
can
select
what
they
want
and
what
they
want.
A
Yes
and
another
really
important
aspect,
as
I
see
it
is,
instead
of
of
going
all
over
the
place,
which
we
did,
I
mean,
I
think,
we're
on
the
third
major
revision
of
our
vocabulary.
Events
vocabulary
inside
internally
involved
cars
based
on
what
I'm,
seeing
with
the
experience
and
the
breadth
of
the
community
surrounding
cd
events.
A
So
it's
really
helpful
for
us.
A
second
takeaway
is,
I
mean
I've
shown
you,
hopefully
that
the
problem
is
quite
complex
and
that
what
we're
building
is
quite
complex.
It's
also
quite
concrete.
I
mean
we
have
real
struggles
that
are
reasonably
easy
to
identify,
but
a
bit
harder
to
solve,
but
also
they
are
shared.
So
I've
been
talking
mainly
about
autonomous
vehicles.
Today,
emil
will
talk
about
more
like
the
telecom
industry,
but
I
would
say
it's
like
80
or
90.
A
As
oleg
said,
there
are
plenty
of
opportunities
to
join
these
communities
if
you
want
to
work
on
them
or
support
them
and
yeah.
Thus
far,
I've
seen
a
lot
of
of
welcoming
a
very
welcoming
attitude
from
these
communities,
so
I
would
encourage
anyone
who's
interested
to
at
least
drop
by
and
say
hello
either
in
this
city
events
con
or
on
slack
afterwards.