►
Description
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
Opentelemetry and Your Tests - Manuel de la Pena, Elastic
JW Grand Ballroom 3
Speakers: Manuel de la Peña
Test data is a gold mine that is rarely explored when discussing software quality and best development practices. Tens of thousands of tests might be executed in your CI pipelines everyday, generating a massive amount of data for you effortlessly. Right now, this data is lying around without producing you any value, but what if you could harness this data to accomplish something? If used correctly, you could be able to, for example, detect test flakiness, notify the right people on failures, and much more. All you have to do is capture this data and turn it into insight using an instrumentation layer. You can add this layer on top of the jUnit de-facto standard for test reports. By doing this, you'll be able to send traces and metrics to a backend using another standard, OpenTelemetry, and explore that information from there in whatever manner it is helpful for your interest.
A
A
It
was
about
me.
I
been
more
than
20
years
around
20
years
in
in
this
industry
with
different
roles,
so
I've
been
moving
from
one
place
of
the
other
in
the
delivery
process
from
I.t
support,
consultancy,
support,
test
automation,
maintaining
some
resource
projects
well,
just
to
give
a
really
beautiful
context.
This
is
my
twitter
account
and
well
everywhere.
You
can
find
me
there
on
the
internet.
A
So
what
I'm
going
to
see
today?
First,
I'm
going
to
start
with
the
journey
in
our
cicd
inner
survivability.
Then
I
will
focus
more
on
testing
and
to
finish
this
presentation,
I
will
share
the
report
that
we
would
like
to
have.
I
would
like
to
share
with
you
that
we
are
in
this
in
a
journey
that
we
are
at
this
moment
here.
We
are
collecting
data
and
we
are
in
a
process
of
thinking.
What
are
what
are
the
the
metrics
and
the
reports
that
we
would
like
to
to
have
so
sharing
here
with
you?
A
What
is
what
we
like,
maybe
provide
us
on
insight,
so
all
the
feedback
that
you
can
provide
us
is
very
useful.
So
in
this
journey
we
are
creating
tools.
We
are
gathering
met,
trying
to
identify
metrics
and
what
reports
that
we
would
like
to
have
all
the
tools
that
we
are
creating
and
open
source
and
well.
I
will,
I
will
share
with
you
there
so
the
first
one
is
what
is
observability,
because
we
are
talking
about
open,
telemetry
austerity.
A
There
are
other
tax
talks
that
we
have
mentioned
it,
but
the
sorry,
the
vast
majority
of
of
definitions
out
there
says
that
opens
obsolete,
is
the
collection
of
data
and
production
systems
right.
So
you
get
data
that
allows
you
to
answer
questions
beforehand.
You
need
them
and
it
usually
comes
with
tools
for
exploration
and
visualization
of
this
data.
These
insights,
so
you
are
able
to
analyze,
react
and
response
unexpected
and
unexpected
behaviors,
because
we
all
know
all
the
expected
things,
but
what
happened
with
the
unexpected
things
that
happen
in
production?
A
Traditionally,
there
are
three
pillars:
application
performance,
monitoring,
logs
and
metrics,
and
this
data
is
the
data
that
we
would
like
to
collect
in
general,
no
survivability.
So
the
title
of
the
talk
is
about
open
telemetry.
So
what
is
open
telemetry
directly
from
the
website?
Open
telemetry
is
a
collection
of
vendor,
neutral
tools,
apis
and
sdks.
That
allows
you
to
instrument,
generate,
collect
and
export
the
telemetry
data.
A
Sorry,
and
it
helps
you
to
analyze
your
software
performance
and
behaviors.
What
are
the
benefits
of
using
open,
telemetry?
Well,
it's
open
source.
So
you
can
contribute.
You
can
open
issues.
You
can
consume
what
what
the
products,
the
projects
that
they
use
and
for
me
the
most
important
thing
is
that
you
are
able
to
instrument
your
application
once
and
depending
on
the
vendor
or
the
backend.
The
call
back
ends
that
you
are
using.
You
can
replace
and
change
whenever
you
need,
you
could
use
elasticsearch.
For
example,
you
could
use
a
ship
king.
A
A
The
first
one
is
about
the
cicd
process
we
own,
or
we
run
production
services
for
cidcicd
in
this
particular
case.
Jenkins
is
our
tool
for
that
and
we're
interested
in
detecting
what
are
the
bottlenecks
in
as
a
owners
of
the
system.
Where
are
the
time
where
the
time
is
spent,
do
we
have
bugs
in
the
plugins
that
we
use?
Are
there
leaks
in
the
in
the
in
the
logs?
A
A
A
The
flagging
detection,
also
about
ownership
of
the
test
failures,
as
I
mentioned,
detecting
who
is
the
or
which
team
is
the
responsible
of
resolving
a
test
failure
and,
finally,
that
improving
the
stability
of
the
test
failures
and
especially
because
we
are
using
jenkins.
Sometimes
the
yankees
ui
is
not
that
easy
to
digest.
A
When
you
are
looking
for
test
results,
test,
logs,
etc,
okay,
so
sorry
in
this
journey,
what
open
source
tools
are
we
using-
and
I
say,
open
source
because
everything
that
we
are
using
is
open
source,
so
we
would
like
the
other
teams
to
to
use
them.
So
you
are
interested
in
please
reach
us
later,
so
we
are
happy
to
to
to
share
with
you
how
to
do
it.
A
Another
project
that
we
have
contributed
in
this
case
to
the
ansible
community
is
the
ansible
open,
telemetry
plugin,
which
is
able
to
add
distributed
traces
for
any
ansible
task.
Well,
it's
in
the
community
there
and
I
recommend
you
this
link,
this
block
entry.
You
know
in
the
last
blogs
about
how
to
use
it
in
your
infrastructure
when
deploying
things
with
ansible
super
easy
to
use.
You
have
an
ansible
configuration
file,
you
add
the
callback,
you
define
your
playbook
and
that's
all
you
have
your
your
distributed.
A
A
Sorry
another
play
that-
and
this
is
one
that
we
have
contributed
to
the
python
ecosystem-
is
the
pi
test
open,
telemetry
plugin.
So
if
you
are
running
your
project
is
written
in
python
and
you
use
pi
test
as
a
test
runner
simply
the
distributed
tests
will
will
be
there
for
your
test,
simply
requirements.txt
file
with
the
dependencies.
A
You
add
it
there.
You
expose
the
environment
variables
where
that
collector
is,
and
you
have
your
test
in
a
nice
view,
with
traces
and
spans
and
finally,
the
jenkins
open,
telemetry
plugin,
our
team.
I
have
created
this
plugin
and
is
in
the
jenkins
in
the
jenkins
organization,
which
allows
you
to
create
traces
from
the
built-in
steps
in
jenkins.
So
you
will
be
able
to
see
all
the
what's
happening
up
in
a
jenkins
pipeline,
see
the
traces
for
that.
A
It
also
collects
metrics
about
the
cpu
work,
allocation
memory,
etc
and,
with
a
simple
pipeline,
you
will
be
able
to
identify
what
are
the
stages
in
that
build.
What
steps
has
been
executed
and
if
you
are
using
distributed
traces,
you
will
be
able
to
see
the
different
races,
although
all
of
them
connected.
A
A
We
mentioned
it
pi
test
and
with
the
plugin
that
we
created
for
this
boot
addresses
in
python,
but
there
are
multiple
teams
with
multiple
languages,
multiple
test,
reporters,
the
runner,
sorry
and
in
especially
in
australia.
We
have
teams
working
with
java
working
with
go,
we
working
with
net
working
with
node.js,
etc.
So
a
lot
of
frameworks,
a
lot
of
test
runners
cucumber,
it
could
be
also
possible
to
do
it.
A
So
we
just
we-
we
found
an
issue
here,
should
we
contribute
the
same
philosophy
the
same
reporter
to
like
the
python
one
to
all
these
communities.
We
don't
have
resources
to
do
that.
At
this
moment
we
are
just.
We
are
doing
up
sf
trying
to
resolve
our
own
projects
or
our
own
problems,
so
hey,
but
we
we
discovered
a
commonality
here
in
all
of
the
those
actors,
there
was
one
actor
that
was
common
to
everybody
to
all
of
them,
and
this
is
xml
xml.
A
This
was
a
de
facto
standard
in
the
past
for
process
to
process
communication.
What
happened?
What
had
to
do
with
the
other
test?
Runner?
The
majority
of
the
test
runners
are
able
to
export
the
xml
to
junit
format
right,
and
this
is
what
junkies,
for
example,
is
able
to
consume
it
in
a
native
manner.
A
So
let's
try
it.
How
can
we
get
with
this
xml?
Well,
yeah,
you
need,
has
a
well
documented,
a
well-known
schema
xsd,
so
we
know
the
the
the
definition
of
the
schema.
The
majority
of
the
test.
Runner
support
it,
and
with
this
simple
file,
which
is
an
xml
file,
we
can
abstract
from
the
process
of
running
the
test,
the
csd
platform,
and
we
can
abstract
from
the
programming
language.
A
A
A
Why
do
I
want
to
highlight
the
git
context,
because
with
when
we
are
submitting,
for
example,
a
change
request,
a
merge
request,
a
pull
request.
We
have
a
lot
of
information
from
that.
From
that
submission
we
can
get.
Who
is
the
author?
The
files
that
were
modified?
How
many
lines
was
added,
were
added.
How
many
lines
were
deleted,
useful
information?
We
can
collect
that
information
from
a
pull
request
and
attach
this
information
to
a
text
execution
and
if
it
fails
boy,
this
fail
this
failure.
Whenever
we
modify
this
file
it
causes
that
this
is
broken.
A
We
can
correlate
things,
so
we
are
trying
to
add
information
about
the
context,
the
git
context
to
to
look
for
this
correlation.
Probably
we
do
we
get
to
close
at
a
alley,
but
we
are
getting
this
information
to
identify
and
get
insights,
and
so
how
can
we
process
this
test?
Remember
the
jenkins
open
telemetry
plugin.
A
We
have
an
internal
version
of
the
project
that
is
not
released
yet
is
not
contributed
but
leverages
the
junit
plugin
in
jenkins,
because
there
are
extension
points
in
that
plugin
that
allows
you
to
execute
code
after
each
test
is
executed.
So
we
created
a
open
telemetry
data
publisher,
which
is
the
number
the
name
of
the
extension
point.
So
whenever
jenkins
is
parsing,
the
junit
file,
it
will
send
traces,
metrics,
etc.
For
that
test
execution,
you
have
the
link
for
the
documentation.
A
Sorry-
and
this
will
be
how
it
goes
in
in
white.
It
will
be
a
traditional
unit
step
in
a
jenkins
pipeline.
If
you
add
the
data
publisher,
you
will
have
this
for
free,
so
you
just
change
your
pipelines
with
one
line
of
code
and
you
have
it.
This
is
not
merged.
Yet
we
want
to
go
very
carefully
with
that.
We
need
to
explore
it
internally
first,
but
it
will
be.
This
would
would
be
what
we
like
to
do
it
and
that
would
be
the
result.
A
A
A
In
this
process
we
read
the
scm
context
of
where
this
code
is
executed,
a
pull
request,
a
merge,
commit
a
release,
etc
get
the
information,
and
we
would
like
to
run
this
in
multiple
platforms.
That's
why
we
chose
go
so
we
are
able
to
run
this
in
linux,
darwin
or
windows,
so
we
are
abstracting
also
from
the
cicd
platform.
This
is
the
url
of
the
of
the
of
the
price,
so
contributions
are
accepted.
A
A
A
This
is
the
content
of
the
xml
file
representing
a
unit
execution.
There
are
no
services
in
jagger
right
now,
because
it's
completely
empty
yeah.
We
are
gonna,
execute
the
code
in
the
unit,
sorry
for
the
size
of
the
of
the
of
the
terminal,
but
it
was
recorded
a
few
days
ago.
I
didn't
consider
this
was
the
big
is
passing
the
flux
to
identify
that
this
is
a
go
project
and
the
second
one
is
a
javascript
project.
A
So
we
will
see
two
services
now
one
for
the
golan
project
and
one
for
the
javascript
project
and
this
golan
project.
If
we
find
the
traces,
we
will
see
the
distributed
traces
representing
the
xml
execution
with
all
the
context.
Sorry,
the
the
scm
context
that
we
added
in
these
stacks.
We
are
xml
repository
scm,
the
type
of
the
repository,
the
test,
suite
name,
etc,
the
same
for
the
javascript
project.
This
is
what
I
copied
from
from
the
internet,
and
this
casper
gis
played
with
a
huge
number
of
of
tests.
A
The
person
that
we
passed
and
all
the
metadata
the
attributes
to
the
to
the
spans
and
the
traces
are
there,
and
I
think
the
rhythm
was
done
so
we're
talking
about
attributes
that
we're
using
what
name
conversions
are.
We
are
we're
following
first
one.
We
would
like
to
define
two
types
of
attributes:
the
first
one
is
related
to
the
sem
specifically
for
git,
and
the
second
part
is
about
the
test
reports
themselves
for
git.
A
A
This
could
be
extended,
of
course,
because
it's
in
the
library,
if
you
contribute
that
you
need
more,
we
can.
We
can
get
to
an
agreement,
forget
something
that
is
specific
for
git
the
clone
death.
It
is
a
shallow
clone,
it's
not
a
shallow
clone.
How
many
files
were
modified?
How
many
lines
were
added,
things
like
that
can
be
can
be
added.
Can
this
is
the
information
that
is
extracted
from
the
repository
when
the
the
code
is
executed
and
about
the
test
reports
or
the
attributes
for
the
test?
The
test
themselves?
A
A
Well,
we
are
getting
the
attributes
and
we
are
passing
downstream.
All
of
the
attributes
for
test
suite,
we
collect
metrics,
like
the
number
of
skip,
pass,
fail,
test,
etc.
For
each
test
suite
we
got
two
different
attributes
and
for
each
test
case
we
capture
different,
attribute,
sorry
attributes
for
the
for
the
test
case,
and
we
are
passing
down
the
attributes
from
top
to
the
bottom,
and
later
I
will
go
back
to
this
example
of
the
test.
Results
is
basically
the
xml
definition,
the
schema
definition
for
4g
unit.
A
And
we
we
are
here
now
this
is
at
the
mo
at
this
moment.
We
are
at
this
point
of
the
journey
at
this
point
of
the
journey
where
we
are
defining.
How
do
we
want
to
present
this
information?
We
are
collecting
data,
collecting
data
and
well,
let's
figure
it
out
with
this
amount
of
data.
What
do
we
want
for
sure?
We
have
three
two
reports
that
we
are
interested
in
related
to
the
ownership,
which
is
the
first
one
well
everybody's
interested
in
this
fluckiness
analyzer.
A
A
And
finally,
this
is
the
next
steps
that
we
would
like
well
that
we
would
like
to
to
to
continue
with.
I
talk
about
attributes
that
are
passed
downstream.
We
need
to
check
with
the
with
the
with
the
functional
teams,
in
inelastic,
for
example,
that
are
writing
the
the
observability
solution
for
apm,
for
example,
about
what
is
the
the
best
strategy
to
store
the
attributes?
It
should
be
a
store
at
the
top
level
should
be
a
store
everywhere,
because
sometimes
the
query
could
be
slower,
slower,
sorry
and
well.
A
What
is
the
the
most
efficient
strategy
to
store
the
attributes?
The
second
concern
that
we
have
is
about
semantic
conventions.
We
are
inventing
inventing
the
con.
We
selected
those
attributes,
because
we
thought
that
were
that
make
sense,
but
there
are
open
telemetry
schemas.
Should
we
stand
for
them
in
this
talk
in
this
conference
and
the
the
cd
events
forces
are,
is
also
a
thing.
So
should
we
stand
for
the
cd
events?
Elastic
has
its
own
elastic
common
schema.
Should
we
adhere
to
that
schema?
A
The
second
or
another
concern
is
merging
the
that
that's
data
polisher
with
the
genki
sultan
plugin,
so
everybody
with
a
plug-in
configure
will
be
able
to
get
the
benefit
of
this
open
telemetry
in
the
for
the
test
reports
again
defining
better
reports.
So
we
are
not
a
testing
team.
So
probably,
with
the
help
of
other
more
close
to
qa
teams,
we
will
be
able
to
understand
what
they
need
or
what
other
things
need.
A
And
finally,
we
would
like
to
have
a
full
integration
with
github.
Remember
the
cli
dj
you
need
to
otlp
developers
are
in
in
github,
so
they
work
there.
They
don't
want
to
switch
context
to
another
tool.
So
probably
we
need
more
integration
with
github,
probably
creating
a
github
action
that
allows
to
to
run
this
code.
There,
I'm
probably
submitting
back
the
results
of
the
flag
analyzer
as
an
example
to
to
give
her
back
so
in
the
in
the
in
the
pull
request.
Summary
share
this
information
and
I
think
that's
that's.
Oh
sorry.
A
I
think
it's
covered,
so
we
have
an
entity
which
is
the
apm
server
that
is
able
to
massage
the
information
and
well
this
rating
you
can
configure
and
and
it
will
send
it
in
batches,
not
to
lose
information,
etc.
So
if
that's
your
concern
in
the
elastic
stack
that
will
be
covered,
but
the
pieces
that
the
elastic
stats
are,
but
I'm
pretty
sure
that
other
vendors
will
resolve
this
problems.
A
Yes,
yes,
for,
for
the
collection
side,
the
the
tool,
the
plug-in
or
the
cli,
we
need
to
stress
test
the
the
process,
so
we
are
not
there
yet,
but
I
understand
that
and
that
I
think
it
could
be
really
really
important
to
verify
that
huge
amounts
of
data
be
in
the
xml
file
or
the
number
of
text
results
should
be
well
budget
or
parallel
parallel,
but
sorry
to
concurrently,
execute
it
so
try
to
optimize
the
process
right
now.
It's
serial.
A
Is
part
of
our
journey
we
would
like
to
under
to
get
to
that
point
to
identify
which
files
are
causing,
which
failures?
I
think
we
can.
We
can
get
there
if
we
run
a
lot
of
bills
and
we
collect
information
from
those
who
build
and
we
are
able
to
identify
that
one
file
is
always
in
the
chain
set.
When
this
test
fails,
we
can
create
correlation,
but
we
need
to
measure
we
need
to
to
to
get
that
information.
First.