►
Description
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
How we Build CI/CD Observability with OpenTelemetry - Michael Friedrich, GitLab
CI/CD Pipelines are blocked, resources are drained, and the cloud invoice for the deployment is going wild - where do you start analysing and troubleshooting? One look leads into the CI/CD pipeline: Next to the job runs, which preparation steps happen, how many context switches are encountered, and which step is actually causing the problems?
Logs and metrics help to see the bigger picture, collecting the job and pipeline execution times, or fetch times for resources. Traces with spans provide more context on executed jobs, enriched with troubleshooting metadata. It is not always DNS, sometimes it’s also the slow IO cache …
Join this talk to learn how we build CI/CD Observability with OpenTelemetry in GitLab. You can learn more about the architecture challenges, implementation approaches, and performance scaling requirements for SaaS platforms.
A
Hello
everyone
today,
I
want
to
talk
with
you,
how
we
build
ci
cd
observability,
with
open
telemetry
and
before
we
dive
in
my
name
is
michael,
I'm
a
senior
developer
of
ventures.
Gitlab,
you
can
find
me
online
as
dns
me
machine,
which
is
dns
m-I-c-h-I
on
twitter
and
linked
in
polywork,
and
also
on
gitlab.com.
A
A
A
One
of
the
challenges
is
so
like
the
number
one
challenge:
we
have
different
tools,
we
have
graphs,
we
have
logs
to
look
at
manual.
Analysis
is
also
like
time
consuming,
so
within
an
active
incident
where
everything
is
like
blocking
and
developers
can't
do
anything.
The
devops
engineers
are
busy
and
everyone
is
on
call.
This
is
stressful
and
this
oftentimes
you're
overlooking
something
which
can
be
solved
rather
easily.
A
Another
challenge
from
the
tools
and
specs
perspective
is
like
what
what
do
we
actually
want
to
measure
in
the
cicd
pipeline?
One
thing
is
to
think
about
metrics,
like
numbers
and
text
strings
as
values,
how
to
store
them,
how
to
do
retention
with
them,
like
deleting
certain
older
metrics,
which
we
don't
need
in
in
a
high
cardinality.
A
It
needs
scalability,
it
needs
high
availability
and
so
on
similar
thing
with
logs.
Sometimes
they
are
two
were
both
hard
to
pass,
which
needs
some
pre-processing
and
post
processing.
We
need
to
think
about
storage
because
it's
not
endless
and
it
costs
money
over
time
when
there's
too
much
logs
and
potentially
they
are
not
helping.
When
they're
like
two
verbose
and
from
a
certain
perspective,
we
also
have
traces
like
there's
a
high
entry
level
to
learn,
but
it's
getting
better
because
open
source
frameworks
are
in
the
making
and
we
get
more
insight.
A
So
there's
potentially
more
insight
available,
and
the
third
challenge
I
found
is
like
either.
Do
it
yourself
or
use
existing
platform
features
like
do
it
yourself
with
an
external
integration
or
it
might
be
upstream
supported?
Is
there
code
instrumentation
needed?
Is
there
anything
I
can?
We
can
contribute
upstream
so
that
everyone
can
use
them
and
everyone
can
contribute
and
like
which
tools
and
platform
do
we
actually
use?
Is
it
like
a
ready-to-use
platform
or
what
is
like
the
evaluation
process
in
that
regard?
A
A
Now,
where
does
exactly
like
what
does
tracing
fit
in
before
we
dive
into
like
observability,
also
is
about
the
knowns
and
the
unknowns,
so
things
we
are
aware
of
and
understand,
and
an
unknown
unknown
is,
for
example,
things
we
are
not
aware
of,
and
do
not
understand
yet.
So
thinking
of
this
with
regards
to
cicd
observability
is
like
we
monitor
the
state
of
our
application.
That's
that's
the
fact
that
something
we
want
we
are
aware
of,
and
we
want
to.
A
We
also
understand
that,
and
on
the
other
side,
the
dns
resolution
latency
actually
delays
the
ci
cd
pipeline
runs.
This
is
something
we
potentially
do
not
know.
Yet
we
are
not
aware
of
it.
We
do
not
understand
it
yet
at
some
point
we
might
discover
that
dns
is
the
problem
as
always,
but
in
that
regard
it's
it's
a
different
way
to
really
like
understand
the
thing.
Now,
how
can
we
like
approach
all
these
challenges
with
observability
within
ci
cd,
with
metrics
logs
and
traces?
A
A
You
can
add
code
to
the
to
your
application,
so
called
instrumentation,
and
the
good
thing
is.
There
is
a
common
specification
now
with
open
telemetry,
which
also
defines
its
specification.
It's
a
framework,
so
you
need
to
bring
your
own
backend
your
own
integration,
for
example,
jager
tracing
as
a
storage,
elasticsearch
click
house
or
something
else.
A
In
2019,
open,
telemetry,
merged
or
kind
of
much
open,
tracing
and
open
sensors,
the
cncf
tag
observability
and
the
open
telemetry
project
have
been
founded
and
later
on.
In
2021,
we
had
also
metrics
and
log
specifications
being
added
next
to
tracing
the
open.
Telemetry
became
a
cncf
incubating
project
and
open,
even
even
though
like
ga
for
tracing,
which
means
open
tracing
was
officially
deprecated,
and
we
can
now
focus
directly
on
open,
telemetry
and
learn
how
to
adopt
it.
A
The
thing
is
with
open
telemetry,
you
could
be
running
a
collector
or
you
are
running
a
sidecar
container
which
which
allows
you
to
send
the
traces
to
I
said,
bring
on
back
end,
for
example,
jaeger
for
traces
promises
for
metrics,
and
there
is
a
huge
ecosystem
being
developed
around
client
libraries
sdks,
allowing
you
to
do
manual
instrumentation
for
compiled
binaries,
like
c
plus,
plus
gold
and
so
on,
or,
for
example,
auto
instrumentation.
So
it's
like
sitting
in
between
the
virtual
machine
in
java
and
tracing
the
calls
and
things
which
are
useful.
A
A
With
pipeline
efficiency
like
how
can
we
identify
long
running
or
blocking
pipeline
chops,
this
can
be
sort
of
matched
or
mapped
into
a
trace
and
spans,
because
sometimes
we
have
the
same
patterns.
Sometimes
they
are
not.
It
allows
us
to
identify
and
see
visually
immediately.
What
what's
going
on
in
there.
It's
also
like
everything
gets
executed
all
the
time,
so
the
timeline
is
full
and
there
is
no
like
break
break
point
or
something
like
something.
We
call
fail
fast
patterns
in
ci,
cd
pipelines.
A
You
don't
need
to
wait
until
everything
gets
executed,
just
fail
as
soon
as
possible,
save
resources
and
tell
the
users
that
something
is
not
working
and
they
need
to
fix
it.
The
other
thing
which
is
interesting
to
analyze
is
like
how
long
does
it
take
to
run
upgrade
install
external
dependencies,
docker,
pull
or
container
image
pulling,
maybe
even
though
virtual
virtual
private
networks,
vpn
low
latency
connections
and
so
on.
A
So
what
needs
to
be
optimized
and
how
to
like
analyze
that
and
also
visualize,
that
for
later
analysis
and
actions,
thinking
again
of
the
problems
to
be
solved,
if
we
are
looking
into
observability
the
knowns
and
unknowns,
we
kind
of
know
that
our
cloud
resources,
the
costs,
are
exploding.
They
are
at
an
all-time
high,
the
cicd
jobs,
actually
the
root
cause
which
has
been
like
identified
already.
A
A
Now
one
of
the
ideas
was
to
keep
it
as
simple,
as
or
as
minimal
as
possible,
with
the
open,
telemetry
receiver
end
point
so
like
the
collector
defined
in
cicd
variables
in
environment
variables,
and
we
want
to
instrument
the
server
parts
which
are
written
in
ruby
and
the
runner
parts
with
dingo
and
start
adding
traces
and
spans
to
that.
The
data
gets
sent
to
the
open,
telemetry
collector
and
the
visualization
is
like
just
using
the
aquarium
ui
to
really
see
something
in
the
beginning
and
later
on.
A
What
else
is
going
on
and
when
I
first
looked
at
the
gitlab
architecture
like
the
server
is
running
the
pipeline
job
queue
and
it
also
receives
the
results
from
from
the
executors,
which
is
actually
the
the
gitlab
runner.
It
does.
Does
the
job
execution
we
have
custom
environments,
different
executors,
where,
where
should
we
be
starting
of
adding
code
instrumentation
with
open
telemetry
to
collect
something?
Now
thinking
about
the
gillab
server
workflow,
it's
a
huge
thing,
and
it's
it's
like
the
the
surrounding
umbrella
to
everything.
A
So
like
the
icd
pipeline
gets
executed,
there
is
a
queue
the
runner
connects
using
via
the
rest
api
collects.
The
jobs
sends
back
the
job
updates
in
an
incremental
stream
later
on
uploads,
the
artifacts
to
the
server
like
a
table
or
a
binary
package,
or
even
a
security
report,
or
something
like
that.
A
The
server
updates
the
data
in
in
the
back
end
stores
the
artifacts
and
updates,
also
the
dashboards
like
vulnerability,
dashboards,
merge,
request,
widgets
and
so
on.
So
there's
a
lot
of
things
going
on,
and
this
didn't
sound
as
right
as
possible
with
like
getting
started
with
a
minimal
change
on
the
other
side.
Looking
at
the
runner,
the
render
is
a
standalone
binary.
It
connects
to
the
server
and
collects
jobs.
This
is
something
we
potentially
can
simulate
with
just
gitlab.com
and
executing
jobs.
When
the
job
is
executed.
A
A
Now
coming
back
to
the
architecture,
server
and
runner
where
to
start
the
runner
is
much
more
approachable
and
you
can
use
standalone
and
the
trip,
duration
and
the
executor
context
is
available
inside.
So
this
was
my
analyzes
and
research
approach
to
say:
hey,
I'm,
starting
with
the
runner
now
and
adding
open
telemetry
india
now
how
to
get
started
with
adding
something
in
there
into
the
gitlab
runner,
which
is
written
in
go.
A
I
knew
I
need
needed
to
read
the
documentation
and
the
example
for
the
open
telemetry
go
sdk,
adding
the
libraries
to
the
go.mod
file
following
the
gitlab
development
guidelines
and
learning
on
the
way,
and
also
making
mistakes.
Of
course
with
adding
everything
together
I
created
a
deaf
environment,
and
this
this
is
all
documented.
A
Then
the
runner
gets
registered
for
gitlab.com
in
a
specific
group
and
only
executes
specific
tags.
So
I
chose
hotel
and
the
way
it's
being
done
is
basically
to
to
define
the
tag
in
the
cicd
jobs
so
that
the
rundown
will
only
execute
jobs
which
are
tagged
specifically
to
that
which
allows
to
really
see
that
the
runner
is
just
executing
these
jobs,
which
are
specifically
triggered
and
not
something
else.
This
can.
A
A
The
other
thing
is
to
really
generate
ci
cd
jobs,
which
do
something
and
sleep
a
little
bit.
To
really
stress
the
runner
in
that
regard
is
we're
using
matrix,
matrix,
builds
in
gitlab
cicd
and
some
templates
with
extends
using
the
hotel
tag
and
then
running
some
jobs.
This
can
be
triggered
manually
on
the
server
or
on
the
gitlab.com
sas
group
and
helps
with
moving
on.
The
other
thing
is
like
connecting
the
runner
starting
the
compiled
runner
binary
after
some
changes
entering
in
the
pipelines
and
just
investigating
standard
output
on
the
console.
A
This
was
the
inter
the
the
most
easiest
way
to
really
see.
What's
going
on,
then
I
thought
of
like
well
adding
a
tracer
to
gitler
to
the
gitlabra
mango.
I
found
like
there
is
context
propagation
used
for
traces
in
the
goes
decade.
This
is
something
you
need
to
think
about
if
the
current
application
doesn't
make
heavy
use
of
that
or
uses
them
in
a
different
way,
it
needs
to
be
considered
and
considered
and
understood,
maybe
just
by
trial
and
error
as
well.
A
A
Certain
to-do's
still
exist
in
the
code.
It's
not
production
ready,
but
still
it
kind
of
worked.
I
also
made
mistakes,
so
I
deferred
the
initialization
of
the
tracer
in
the
same
function.
I
was
calling
and
then
I
wondered
why
nothing
is
happening
because
it
was
immediately
destroyed
afterwards
or
closed
afterwards
and
I
thought
of
well.
How
can
I
trace
or
log
what
they
open
telemetry
is
doing,
and
so
I
found
some
verbosity
functions
from
the
default
logging,
which,
which
is
super
helpful
in
in
specific
vendor-specific
blogs
and
documentation.
A
A
A
The
other
thing
after
this
kind,
this
worked
and
the
first
traces
have
been
locked
to
standard
output,
I've
initialized,
the
external
receiver
endpoint,
so
the
hotel
exporter
for
traces.
There
are
different
transports
available,
grpc
and
http
for
my
tests,
I
kind
of
needed
to
disable
tls,
which
is
still
still
going
on.
So
it's
insecure,
there's,
no
tls
in
future
steps.
The
goal
is
obviously
to
add
tls
as
well
for
the
development
purposes.
A
A
This
is
going
to
be
evaluated
for
the
research
tasks,
and
this
is
something
coming
or
to
be
done
in
the
future
figure
out
which
instrumentation
for
methods
and
context
is
needed,
which
metadata
is
already
available
in
the
source
code,
which
needs
which
other
parts
need
to
needs
to
be
added
to
the
gitlab
runner
and
the
performance
implications
of
enabled
by
default
and
doing
benchmarks
and
scalability.
This
is
still
an
ongoing
project
going
back
from
the
runner,
which
is
now
the
first
trace
and
span
being
sent
to
the
platform.
A
The
gitlab
server
is
written
in
ruby,
so
researching
the
open,
telemetry,
ruby,
sdk,
figuring
out
the
pipeline
execution
backend
and
the
queuing
and
where
to
start
instrumenting
the
code
for
an
mvc,
a
minimal,
reliable
change.
It's
a
good
idea
to
start
with
pipeline
start
and
end
times
and
job
and
start
and
end
times
to
really
build
out
the
pipeline
path
or
the
timeline
path
and
get
going
more
easily
for
the
server
itself
pipeline
execution
start
and
queueing
a
job
measure.
A
The
time
to
execution
like
the
q
chops,
the
qt
time
for
chops
fetch
the
results,
the
time
of
the
execution,
maybe
the
artifact
size
and
number
specific
log
or
trace
job
block
length
and
insights,
storage
updates,
front
and
fetches
and
so
on.
This
is
all
to
be
evaluated
and
not
written
in
stone.
Yet,
but
still
it's
it's
interesting
and
there
is
much
much
more
to
be
challenged
because
the
icd
variables
might
be
protected
or
mask.
A
So
we
need
to
ensure
what
we
can
use
in
there
and
reuse
that
from
the
runner
to
the
server
the
job
lock.
Snippets
might
also
contain
certain
context
of
metadata,
which
is
useful
in
the
traces
so
figuring
out
how
to
kind
of
copy
the
logs
to
to
the
traces,
the
different
executors
for
docker
for
buildings
for
portman,
kubernetes
and
shell
and
the
operating
system
and
architecture.
So
linux,
unix
power,
mac,
os,
starvin
and
windows
platforms
are
different
and
the
problem
could
problem
with
the
crcd
pipeline.
A
Now
one
of
the
challenges
is
that
the
server
generates
a
trace
id
and
the
runner
traces
or
the
runners
bands
which,
which
are
being
added
to
that
they
defined
it
as
a
parent
trace
id-
and
this
is
this-
is
getting
back
sent
back
to
the
server
at
a
certain
point
and
completes
everything.
So
you
can
visualize
the
complete
ci
cd
pipeline
with
the
runner
with
the
server
in
one
global
trace,
and
this
should
be
the
ultimate
goal.
This
is
not
implemented
yet,
but
from
the
research
it
showed
it
showed
to
be
useful.
A
Now
thinking
about
observability
for
devops,
we
also
want
to
customize
it,
and
this
is
like.
Maybe
we
want
to
extend
the
configuration
language
and
it's
some
sort
of
trace
keyword,
maybe
also
exporting
metrics,
allowing
the
users
to
customize
the
ci
cd
configuration
on
the
other
side.
It
might
be
interesting
to
provide
ci
cd
tools,
cli
tools
for
ci
cd,
which
automatically
configured
by
environment
variables,
avoiding
too
much
user
configuration.
A
This
is
also
something
to
be
discussed
in
the
linked
issue,
and
the
other
thing
to
consider
is
the
deployment
when
the
deployment
is
done.
Observability
doesn't
start
yet,
so
we
are
just
getting
started.
Something
gets
deployed
into
kubernetes
our
cloud
virtual
machine
bare
metal,
we're
generating
runtime,
metrics,
logs
traces
events,
error,
tracking
and
so
on,
and
developers
need
visibility
from
the
deployment
tied
into
the
actual
csd
problem.
So
when
we're
debugging
any
certain
requestions,
we
also
need
an
association
with
a
git
tag,
a
commit.
A
A
I've
talked
quite
a
lot.
There
are
certain
other
things
to
really
consider,
and
this
is
something
something
to
be
researched
in
the
future.
Is
it
good
to
use
auto,
instrumentation
or
manual
instrumentation
like
collect
the
traces
automatically?
Is
this
reliable?
Can
we
use,
for
example,
ebpf,
or
something
like
that,
or
should
we
just
define
the
instrumentation
scope
focus
on
the
metadata
in
which,
when
we
want
to
control
as
the
maintainers
of
gitlab
runner
and
gitlab
server,
and
also
limit
the
collected
data,
because
this
can
cause
problems
in
large-scale
environments?
A
A
We
need
to
think
about
scaling
problems,
security
problems
and
also
provide
some
sort
of
visualization,
which
is
which
works
out
of
the
box
on
the
one
devops
platform
or
somewhere
else,
where
like
open,
telemetry
and
is
running
to
observe
the
ci
cd
pipelines.
Now,
the
other
thing
is
scaling
next
to
security.
A
But
on
the
other
side,
the
data
collection
is
still
in
small
and
large
environments,
so,
like
performance
benchmarks
is
cpu
user
usage
memory
crew
increased
in
the
implementation?
Should
it
be
disabled
by
default
on
the
embedded
hardware?
How
does
it
work
with
the
architecture
with
collectors
side,
cars,
deployment
deployments
and
platforms?
Actually,
so
there
were
many
many
things
to
consider
until
this
will
be
available
as
like
as
a
general
available
feature,
and
also
like
when
we're
running
many
many
ci
cd
pipelines
and
jobs.
A
There's
massive
data
collection
going
on
in
the
background,
how
to
scale
that
maybe
the
collector
is
a
single
point
of
failure.
We
need
to
horizontally
use
some
sharing
or
high
availability
or
other
things
which
I
haven't
checked
yet
if
they,
if
they
exist
yet
the
back-end
storage
might
also
be
needed
to
scale
with,
for
example,
click
house
or
elasticsearch
and
again
my
tendency
and
security
comes
into
play
into
all
these
scaling
considerations,
filtering
the
data
away
and
potentially
making
it
slower.
A
Now,
looking
a
little
bit
more
in
the
future,
open
telemetry
provides
metrics
and
logs
or
like
what
is
what
is
coming.
Metrics
are
ga
logs
are
in
draft
and
I
think
they're
becoming
becoming
ga
soon.
We
also
need
to
like
think
about
unifying
a
way
to
instrument
applications
like
creating
libraries
for
ourselves
in
the
lab
kit,
for
go
and
for
ruby
in
gitlab
developers
need
to
learn
and
adopt,
and
we
also
need
to
figure
out
whether
it
makes
sense
to
change
from
slash
metrics
as
an
endpoint
for
parenthesis
to
open
telemetry
into
collector.
A
A
There
are
many
many
ideas,
so
it's
what
I've
been
telling
you
now
is
not
like
set
in
stone.
Everyone
can
contribute,
and
it's
important,
that
we
document
everything
transparently
what
I
have
learned,
what
we,
what
we
are
learning
with
development,
docs,
deaf
environments
and
building
the
foundation
for
so
that
everyone
can
contribute
to
the
ideas.
The
feature
proposal
is
linked
over
here.