►
Description
DevOps is not just about operating the software in production, but also about releasing that software to production. Well-functioning CI/CD pipelines are critical for the business, and this calls for quality observability, to handle broken and flaky pipeline runs effectively.
On this episode I hosted Oleg Nenashev, a core maintainer and board member in the Jenkins project, as well as a TOC member in the Continuous Delivery Foundation (CDF). Oleg is a community builder, open source advocate and consultant, now at WireMock Inc. He is also a CDF and CNCF ambassador.
We discussed CI/CD, observability, the prominent open source projects and foundations, as well as a new proposal for extending OpenTelemetry to natively support CI/CD observability use cases.
The podcast episodes are available for listening on your favorite podcast app and on this YouTube channel.
We live-stream the episodes, and you’re welcome to join the stream here on YouTube Live or at https://www.twitch.tv/openobservability.
Follow us on Twitter @openobserv to get the live stream times and other updates, and to pitch in with your thoughts and comments.
You can read the recap post: TBD
Resources
=========
Observability Has a Complexity Problem (APM Digest): https://www.apmdigest.com/observability-devops-pulse-2023
OTLP 1.0 is out https://twitter.com/horovits/status/1675946183032729622
Open Standards in observability: updates from KubeCon: https://logz.io/blog/observability-open-standards-kubecon-open-source/?utm_source=devrel&utm_medium=devrel
CI/CD observability support in OpenTelemetry proposal:
https://github.com/open-telemetry/oteps/pull/223
State of Continuous Delivery report https://cd.foundation/state-of-cd-2023/
Chapters
========
00:00 - show intro
01:00 - episode and guest intro
10:08 - what’s new in Jenkins
15:46 - is Jenkins cloud-native?
16:52 - understanding the CI/CD landscape
21:54 - updates from the Continuous Delivery Foundation
27:00 - CI/CD observability support in OpenTelemetry
40:31 - observability with Backstage IDP open source
47:47 - how to contact Oleg
48:51 - State of Continuous Delivery report
52:32 - OTLP 1.0 release and other open standards updates
54:32 - KubeCon will hold a dev-centric event for the first time
55:55 - Jaeger 1.47 is out
57:30 - DevOps Pulse survey insights
58:55 - outro
A
Hello,
everyone
and
welcome
to
open
observability
talks,
I'm
your
host
Orton
horvitz
and
here
at
open,
observability
talks.
We
talk
about
anything,
devops,
observability
and
open
source.
So
may
the
open
source
be
with
you
I'd
like
to
thank
our
sponsors
logs.io.
The
cloud
native
observability
platform.
Logdio
takes
the
best
of
breed
open
source
projects
such
as
Prometheus,
open
search,
Jaeger,
open
television,
others
and
offers
them
as
a
unified
observability
platform
built
for
scale
for
those
joining
the
live
stream
on
YouTube
or
twitch
feel
free
to
share
questions
and
comments
on
the
live
chat.
A
It
will
make
things
much
more
interesting
for
for
us
here
on
the
show
and
with
that,
let's
move
on
to
today's
episode.
We
all
know
that
observability
is
a
must
have
for
operating.
You
know
operating
systems
in
production,
as
we
all
do
that's
established,
but
we
often
neglect
our
own
backyard.
Our
software
release
process
essentially
and
I've
seen
that
in
companies
I've
worked
with
as
well,
and
this
leads
teams
to
wasting
time
and
energy
and
handling
failures
in
the
cicd
pipeline
and
makes
the
developer
on
duty
shift.
A
Olegninashev
is
also
a
member
of
the
technical
oversight
Committee
in
the
continuous
delivery
Foundation
the
CDF
and
there's
also
a
CDF
Ambassador
amongst
many
other
titles
is
also
an
observability
veteran
or
just
a
cicd.
He
worked
at
the
dinotrace
and
other
things
we'll
hear
about
very
soon.
So
definitely
the
right
person
for
this
discussion.
A
B
A
A
B
B
Actually
I
started
as
a
hardware
and
embedded
engineer,
and
my
first
project
was
actually
about
observability
too,
but
it
was
a
true
Russian
style
of
observability
oil
flow
monitoring
based
on
x-ray,
Spectrum
analysis
yeah,
so
it
basically
it
involves
radioactivity
and
oil.
What
else
would
you
expect,
but
yeah
ever
since?
For
me
it
was
every
project
I
worked
on.
That
was
always
some
bits
of
observability,
whether
it
was
a
hardware
project,
a
software
project
or
automation,
infrastructure.
B
All
the
time
you
basically
end
up
with
a
question
how
to
actually
troubleshoot
your
system,
how
to
expose
other
data
that
helps
you
to
troubleshoot.
So
for
me,
definitely
I'm
nothing
newcomer
to
the
observability
space
and
that's
why
I
was
so
excited
when
opened
Telemetry
project
started
I
believe
we
will
talk
about
open,
Telemetry
a
bit
later
but
yeah.
For
me,
it's
one
of
the
key
highlights
on
the
open,
observability
space.
Of
course,.
B
Yeah,
so
how
go
to
the
cicd
space
actually
well,
pretty
much.
Has
everyone
else,
yeah,
Hardware
embedded.
You
have
a
lot
of
tests
test,
automation,
So.
Eventually
you
need
automation
systems
for,
for
me,
I
started
with
Hudson
I
believe
in
2009,
so
basically
one
Hudson
was
gaining
popularity
and
then
it
happened.
Basically,
as
usual,
there
were
issues
with
our
builds
I
fixed
these
bills.
So
then
I
started
owning
car.
It
wasn't
even
pipeline
at
that
stage
it
was
freestyle
jobs,
so
all
classic
stuff
and
yeah
later.
B
Actually
it
was
a
good,
switching,
more
and
more
towards
developers
to
link
so
in
Hardware
the
restroom
electronic
design
Automation
in
software
just
developer
tools,
so
I
gradually
switched
more
towards
that
area
and
I
started
maintaining
more
and
more
automation
infrastructure
in
my
companies.
So
when
I
joined
synopsis,
it
was
2012
by
the
time
I
was
basically
leading
infrastructure
for
a
small
Department
of
300
engineers
and
many
sites
and
yeah.
All
the
automation
infrastructure
was
built
on
the
top
of
Jenkins.
It
was
still
before.
B
Docker
was
a
thing,
but
it
was
already
solid
based
on
infrastructures,
code
configurations,
code
and
of
course,
observability
for
us
has
been
a
really
big
problem,
because
yeah
the
scale
was
10
sites,
a
lot
of
computing
grids.
So
maybe
you
heard
about
things
like
this
sound
great
engine,
so
it
again
it
was
before
the
kubernetes
Clusters,
but
it
was
basically
the
same
in
terms
of
resource
management
and
Scalia
scaling
problems
and
when
you
run
Services
there,
of
course
you
need
to
understand
how
the
infrastructure
behaves.
B
B
So
agent,
the
Jenkins
community
in
2012,
shortly
after
Hudson,
was
renamed
to
Jenkins.
It's
not
a
fork,
it's
renaming
and
what
happened
the
next
so
initially
I
got
permission
to
contribute
because
we
needed
to
improve
security
to
basically
align
drinking
security
and
special
identification.
Authorization
model
with
company
requirements
and
I
was
lucky
to
finally
get
permission
to
contribute
to
open
source
because
in
Hardware
companies
it's
not
easy.
B
I,
probably
created
three
private
Source
versions
of
a
VCS
plugin,
so
it's
a
simulation
tool
for
Hardware
and
yeah
when
I
got
permission
to
contribute
to
open
source,
I
started
from
role
strategy
and
a
few
other
plugins
so
create,
for
example,
job
restrictions,
plugin
for
Jenkins
and
created
ownership
based
security
model
there.
So
our
key
goal
was
to
empower
developers
to
actually
maintain
the
jobs,
maintain
their
projects
and
their
products
as
a
part
of
okay
system.
B
These
folders
Etc-
and
there
was
a
big
ecosystem
around
that
built
or
or
internal
users
and
then
well
how
I
became
my
team
is
basically
the
same
as
maintaining
of
automation,
infrastructure.
There
were,
of
course,
issues
with
Jenkins.
Nothing
always
works
as
expected
in
open
source.
So
in
private
source,
so
I
started
submitting
in
fixes
here
and
there
and
finally,
I
was
annoying
enough
that
somebody
gave
me
right
permissions
to
Jenkins
score.
So
this
is
a
real
story
of
how
I
became
a
Jenkins
community.
B
It
wasn't
young
at
that
point:
Jenkins
was
already
almost
10
years
old,
actually
and
80s
is
open
source
project,
but
Jenkins
has
a
different
Philosophy
from
many
other
projects,
especially
in
the
cncf
landscape,
because
in
Jenkins
actually,
the
idea
is
to
empower
contributors
as
much
as
possible,
so,
for
example,
maintenance
of
plugins
maintenance
of
a
system
that
is
really
low
entry
bar
and
for
the
Jenkins
core.
At
that
point
it
was
also
quite
a
low
entry
bar
because
I
definitely
like
to
maintainers.
B
So
the
idea
was
that
if
we
could
unbolt
more
people,
we
could
introduce
but
a
contributor
experience
and
then
over
time
we
could
basically
introduce
additional
processes,
assistance
tools,
and
this
is
what
happened
so
by
2000
20
20.
We
actually
improved
the
maintenance
process
for
the
Jenkins
score.
Quite
a
lot
from
many
things,
including
curious
automation.
You
won't
believe,
but
until
2019
Jenkins
was
released
from
a
concert,
cares
he's
a
creator
of
Jenkins.
B
B
Yeah,
so
it's
actually
a
quite
a
good
question
in
my
case,
I've
been
on
a
kind
of
sabbatical
in
Jenkins,
since
the
war
in
Ukraine
started
because
yeah
I
changed
my
priorities.
I
dedicate
quite
a
lot
of
time
to
a
volunteering
and
some
of
other
personal
events
and
yeah
I
still
participate
in
the
Jenkins
governance
board.
I
help
people
but
I,
don't
participate
actively
in
maintenance
now.
B
But
what
happens
in
the
project
in
recent
years?
There
are,
there
have
been
quite
a
lot
of
improvements
and
actually
I
tried
to
position
most
of
these
improvements,
either
as
Cloud
native
Focus,
so
basically
enabling
Jenkins
utilization
in
modern
deployments,
including
kubernetes
public
clouds
with
kubernetes
or
not,
and
the
second
part
was
actually
developer
experience
because
there's
a
lot
of
things
that
improved
experience
with
rankings.
For
example,
you
may
hate
changes
web
interface,
but
actually
even
the
interface
of
Jenkins
controllers
changed
a
lot.
B
But
in
fact
you
do
not
need
this
interface
much
these
days
because,
for
example,
you
have
quite
a
good
GitHub
integration
through
GitHub
apps.
So
for
the
majority
of
use
cases,
developers
can
see
the
results
right
inside
the
Hub
requests,
and
there
were
a
lot
of
changes
like
that
which
actually
made
Jenkins
usage
quite
useful,
one
quite
easy
in
the
modern
environment
for
cloud
deployments.
Yes,
there
is
a
lot
of
capabilities
that
these
days
there
is
an
official
Hound
chart
that
is
official
operator
for
Jenkins.
B
That
allows
scaling
instances
to
zero
if
needed.
Jenkins
itself
is
not
Cloud
native,
but
Jenkins
build
a
case
system
at
the
moment
is
cloud
native,
so
basically
a
built
infrastructure
with
kubernetes
plugin.
These
other
plugins
can
scale
across
multiple
instances
quite
easily
and
you
just
need
a
server
which
basically
operates
as
a
control
plane.
So
drink
is
controller
by
now.
Well,
it
has
always
been
a
kind
of
control
plane
with
web
UI.
We
haven't
decoupled.
This
part
I
would
be
super
happy
to
decouple
them,
but
still
it
remains
of
the
same
architecture.
B
It's
still
the
deployment
to
kubernetes
of
Jenkins
became
quite
easy.
There
are
distributions
allowing
that,
of
course,
the
Jenkins
became
fully
integrated
with
configurations
code,
so
I
was
happy
to
participate
in
the
configuration
is
called
plugin
development
and
in
other
stages
of
Docker
changes,
pipelines
code
changes
so
now
yeah
Jenkins
has
you
know
it
now,
it's
completely
different
from
Jenkins
10
years
ago
in
terms
of
operations
experience.
This
is
why
some
of
my
experience
is
not
quite
relevant.
A
B
Go
on
back
to
cicd
space
for
Jenkins
itself.
There
were
a
lot
of
investments
in
observability
a
long
time
because
drinks
has
distributed.
The
system
is
complex
to
manage
and
since
2018
I
believe
there
has
been
continuous
investment
in
but
reporting
about
analytics.
We
integrated
additional
monitoring
Frameworks
in
Jenkins,
then,
when
open
Telemetry
became
available
shortly
after
there
was
open
to
them
to
plug
in
for
Jenkins
that
they
did
some
initial
integration.
B
Then
later,
for
example,
I
was
mentoring,
a
student
who
worked
on
open,
Telemetry
integration
for
Jenkins
agents.
So
this
remote
built
engines
that
connect
to
the
system,
and
now
we
can
actually
get
insights
about
Jenkins
being
used
as
your
pipelines.
So,
even
if
you
trigger
the
build
outside
of
Jenkins,
then
all
the
spawns
reach
the
Jenkins
side
and
then,
when
Jenkins
invokes
build
tools
because
Jenkins
itself,
it
doesn't
build
your
project.
B
It
invokes
the
tool
for
that,
so,
whether
it's
maybe
in
Gretel
or
something
else,
all
these
tools,
or
by
now
also
have
some
integration
with
open
energy,
and
it
has
been
super
exciting
to
use,
let's
say
Jaeger
or
parameters
with
some
additional
grafana
dashboards
and
see
all
breakdown
of
distributed
bills.
So
you
were
able
to
navigate
across
multiple
instances,
multiple
containers
because
now
we're,
for
example,
on
jkci
WE
package,
most
of
build
steps
in
separate
containers.
So
similarly,
like
techtone
and
argado
and
I
believe
this
is
a
good
approach.
B
A
B
B
I
well,
I'm
not
alluded
about
Cloud
native,
but
what
I
keep
saying
about
Jenkins
that
it
doesn't
have
to
be
Cloud
native.
What
it
has
to
be
is
being
Cloud
friendly.
So
it
means
that
for
users
it's
easy
to
deploy
your
Jenkins
in
the
cloud
or
in
kubernetes
yeah.
You
definitely
get
all
this
scalability
capabilities.
B
You
definitely
get
pluggable
storage
and
many
other
things
that
are
enable
you
to
deploy
Jenkins
in
your
custom
environments,
but
at
the
same
time
Jenkins
as
a
controller
is
not
Cloud
native
and
it's
2023
and,
to
be
honest,
I
do
not
believe
that
Jenkins
should
become
Cloud
native
at
the
moment,
because
it's
a
massive
investment.
There
would
be
a
lot
of
breaking
changes
here
and
there
and,
in
my
opinion,
it's
not
no
longer
Justified,
because
there
is
a
lot
of
other
Cloud
native
CI
and
CD
Solutions.
A
B
A
B
Yeah
I'm
definitely
well,
even
if
I'm
Jenkins
governance,
Sports
member
I'm,
not
a
drinking
zebot,
so,
for
example,
now
to
be
honest,
I
use
more
GitHub
actions
or
ticked
on
than
Jenkins
and
well.
Why
not?
Because
Jenkins
is
just
a
stool
and
everything
else
is
just
two,
so
for
me
it's
good
to
actually
provide
interoperability
and
when
I
joined
the
CDF
QC.
B
My
key
objective
was
to
actually
facilitate
observably
interoperability
between
the
projects
and
after
then,
observability
of
the
projects
so
go
into
the
cloud
native
space.
As
you
mentioned,
there
is
a
lot
of
tools.
Actually,
there
is
insane
number
of
different
tools
that
address
many
bits
of
continuous
delivery.
B
Fortunately,
or
not,
with
adoption
of
devops
practices
Etc,
actually
a
CD
became
a
priority
to
CI.
This
is
something
that,
to
be
honest,
makes
me
always
sad,
and
this
is
why,
for
example,
in
the
Ambassador
chart,
I
raised
the
question,
so
would
be
anyone
be
interested
to
have
a
cognitive,
CI
webinars,
but
yeah?
Actually,
what
happens
these
days
is
that
yeah,
that
is
continuous
integration,
part
that
to
some
extent
got
really
simplified
for
cloud
native
applications,
so
I
mean
not
so
many
developers
do
coverages
different
integration
testing.
B
Yes,
there
is
a
lot
of
focus
in
security,
but
still
compared
to
how
we
used
to
do
CI
for
embedded
projects.
Hardware
projects
I
think
there
is
a
lot
of
things
that
could
be
improved
and
there
is
delivery
tools
that
actually
got
massive
boost
to
this
introduction
of
kubernetes,
because
there
is
a
whole
landscape
of
tools
and
I
would
say
that
half
of
the
tools
are
somehow
related
to
deliver
and
deployment.
So
you
take
Helm
and
cncf,
for
example,
on
its
quadrant
says
that
Helm
is
the
only
career
grade,
delivery,
Tool!
B
Well,
it
was
one
year
ago,
so
maybe
it
changed
by
now,
but
yeah
it
might
feel
a
bit
old
but
yeah
if
you're,
just
looking
natural,
of
course,
yeah
Helm
is
a
delivery
tool
and
basically,
when
we
talk
about
more
classic
tools
with
spy
plans,
yeah,
of
course
we
have
Argo.
Well,
we
have
multiple
organs.
Actually,
so
we
have
our
rollouts.
We
have
Argo
CD
that
have
slightly
different
use
cases
and
implementations.
B
So,
whether
you
like
devops,
sorry,
githubs
or
not,
you
can
find
a
tool
for
you,
then
of
course
Vlogs
and
then
really
on
the
cloud
native
space.
We
also
have
tikton.
So
tikton
is
a
graduated
project
in
the
continuous
delivery
Foundation.
But
it's
basically
a
part
of
this
ecosystem.
I
would
say
that
Foundation
placement
is
a
more
question
of
politics
than
a
question
of
Technology
but
yeah.
This
is
a
separate
topic
and
there
is
also
a
lot
of
tools
that
actually
enable
particular
bits.
B
So,
for
example,
if
you
need
to
manage
environments,
you
have
cross
plane,
so
you
use
cross
plane.
You
can
easily
deploy
environments
for
your
integration,
testing
purposes,
let's
say
or
for
production.
If
needed,
then
there
is,
for
example,
istio.
So
if
you
do
Progressive
delivery,
if
you
do
a
b
testing
most
likely,
you
go
resistor
to
manage
traffic
for
a
system.
Is
it
a
delivery
at
all?
Then
I
guess
so,
and
this
is
actually
where
liter
is,
that
continues,
delivery
and
continuous
deployment
and
then
continuous
integration
is
not
a
single
tool.
B
It's
a
holocaust
system
of
tools
and
actually
for
years
of
organization,
it's
rather
policy
culture
and
process.
It's
not
and
tools
just
enable
it
and
automate
it.
So
you
can
take
a
lot
of
tools.
That
is
a
huge
inventory
for
different
use
cases,
but
all
these
activities
actually
start
from
your
organization
and
organizational
approach
to
the
delivery
of
your
software.
A
Restating
that
in
this
show
in
other
places
so
many
times
and
still
you're
right,
that
people
tend
to
focus
on
the
tools,
maybe
because
it's
the
easiest
to
comprehend,
because
the
messiest
are
the
people,
the
people
in
processes.
So
people
go
to
where
it's
easiest
and
it's
a
tool,
but
definitely
the
tools
is
not
the
sum
total
of
that,
and
you
mentioned
the
CDF
a
few
times.
A
We
actually
cover
the
the
CDF
The
Continuous
delivery
foundation
in
a
a
past
episode
with
Andy
grabner
from
dinotrace
and
from
the
captain
project
about
a
year
ago
about
a
year
passed
and-
and
you
also
a
CDF
Toc
member
until
recently.
So
maybe
you
can
give
us
the
latest
updates
from
the
cncf
this
past
year.
B
At
the
same
time
now
I,
my
only
public
title
is
in
CF
Ambassador
CDF,
ambassador
okay,
so
this
Foundation
yeah
there
were
quite
a
lot
of
changes
over
time
in
the
foundation.
But
what
happened
in
the
last
year
that
Foundation
actually
grew
in
terms
of
projects.
So
one
of
big
highlights
last
year
that
project
Pierce
joined
so
project
purposes
distributed
well
delivery
Network
for
artifacts.
So
this
is
distributed
package
delivery
system.
Whatever
you
choose
so
again,
it
has
delivery
in
equation.
B
Again,
it's
not
a
classic
pipeline
tool,
but
the
reason
of
the
continuous
delivery
foundation
and
that
actually
it
wants
to
enable
all
stages
of
software
delivery
and
not
just
have
a
bunch
of
competing
continuous
integration.
Continuous
delivery
tools
that
they
Implement
Pipelines
was
one
of
big
highlights.
Another
big
highlight
was
that
actually
tickton
project
got
graduated,
so
it's
a
second
project
to
do
it
in
the
foundation
after
Jenkins,
there
were
quite
a
lot
of
adjustments
there
and
we
worked
closely
with
contributors
to
enable
that
and
another
thing,
that's
rather
more
important.
B
Focus
system
is
CD
events
because
City
events
again
for
me
see
the
events
falls
into
the
observability
space
because,
for
me,
Eventing
is
actually
one
of
the
key
ways
to
do
observability
and
actually
what
is
still
missing
from
open
Telemetry.
In
my
opinion,
because
for
me
yeah
in
open
Telemetry,
there
was
traceability
spans,
but
spans
are
not
always
like
that.
You
well
basically
everyone
who
does
wrap
events
in
response,
but
still
for
me
it's
another
type
of
data,
and
you
know
the
cloud
native
space.
B
We
have
Cloud
events,
so
Cloud
events
is
basically
a
project
that
allows
to
send
and
receive
events
during
the
system.
So
basically
it's
a
way
of
asynchronous
communication,
and
this
is
heavily
used
for
various
kinds
of
observability
within
the
system.
So,
for
example,
there
are
projects
like
tecton
pipelines,
Jenkins
that
integrate
with
CD
events
to
Center
data.
B
There
is
also
key
native
that
actually
uses
Cloud
events
quite
a
lot
inherent
internally
for
various
kinds
of
communications
and
reactions
and
I
think
that
it's
actually
a
super
important
part
for
observability
purposes
and
what
the
continuous
delivery
Foundation
invest
in
is
interoperability
of
cicd
systems
and
basically
unifying
way
to
communicate
the
events
through
Cloud
events.
So
there
is
a
specification
built
on
the
top
of
that
which
is
called
CD
events.
B
A
A
It
made
perfect
sense
to
me
to
look
into
the
City
events
specification
as
a
as
a
good
starting
point
and
maybe
a
good
point
for
collaboration
between
the
two
foundations.
It
gets
confusing,
as
you
said.
Sometimes
it's
more
politics
than
than
technology
like
why
Argo
and
flux
are
in
cncf,
and
then
you
see
non-cicd
projects
under
the
CDF
like
Piercy
and
others.
So
I
I
don't
really
care
about
politics,
I'm,
not
a
politician.
A
What
I
do
care
is
that
if
there
is
a
value,
a
concrete
value
and
a
lot
of
thought
put
into
formalizing
what
a
continuous
delivery
event
is.
What's
the
types
of
events,
what's
the
semantic
conventions
and
so
on,
there
is
definitely
good
base
point
to
start
off
of
that
and
create
a
native
support
in
open
Telemetry.
A
Essentially-
and
it's
very
early
on
so
I'll
post
the
link
here,
and
also
on
the
show
notes
for
the
podcast
listeners
but
happy
to
get
more
people
involved
in
any
any
thought
processing
in
formulating
what
we
need
there
and
you
being
an
authority.
Obviously
you're
one
of
the
people
I
turn
to
very
early
on
to
get
your
take
on
that
so
happy
to
to
get
your
your
two
cents
on
that.
B
Yeah,
so
for
me
it's
the
difference
is
definitely
an
interesting
project.
Even
before
CD
events
started
officially,
there
were
actually
some
initiatives,
so,
for
example,
in
Jenkins
we
created
the
first
version
of
cloud
events.
Plugin
again
it
was
a
Google
summer
of
code
project.
I
was
rather
technical
advisor.
There
I
wasn't
mentoring
it,
but
the
idea
was
to
actually
enable
observability
for
the
systems
and
also
connect
bits
of
the
system,
because,
as
you
said,
that
on
yeah
you
may
have
multiple
tools
and
you're
most
likely
to
have
multiple
pools.
B
Another
reason
for
that
that
actually
it
allows
you
to
create
developers
to
link
for
that
because
all
such
tools,
they
also
need
some
capabilities
for
debugging
troubleshooting
analytics.
So
it's
not
like
putting
all
this
data
into
grafana
dashboard
and
having
called
the
insights
special
tools,
I
needed
most
of
the
time.
For
example,
Stevens
has
a
CLI
that
actually
provides
some
generational
analytics
capability
that
are
floating
ideas
about
having
commod
Tools
around
the
space
and
with
strong
adoption.
B
It
wasn't
used
plugin
due
to
some
additional
data
requirements,
so
it
was
collecting
more
data
from
the
system
from
observability
and
basically
providing
insights
for
users,
and
it
was
using
all
these
open
protocols
to
extract
the
data
on
the
second
site
yeah.
When
we
talk
about
these
tools
yeah,
it's
also
a
good
opportunity
for
producers,
because
you
can
trigger
different
events
and
Trigger
different
continuous
delivery
and
deployment
operations
through
the
standard
protocol.
So,
for
example,
if
you
use
Progressive
delivery,
if
you
use
EB
testing
integration,
you
can
increase.
B
Cloud
events
is
one
of
the
ways
to
actually
orchestrate
your
pipeline.
So
for
me,
for
example,
when
we
talk
about
So,
currently
working
from
ymog
or
one
is
quite
a
popular
API,
more
can
solution
for
many
Technologies
must
start
with
Java.
It
not
supports
many
other
languages
and
what
we
can
do
with
Cloud
events.
We
can
firstly
receive
these
events
and
based
on
these
events,
we
can
actually
initialize
some
tests
or.
A
B
Api
providers
for
your
test
environments
and
we
we
can
make
it
fast
so
for
the
feedback
loop
for
your
developers
is
much
quicker
and,
of
course,
we
can
also
connect
to
Cloud
events
and
actually
emulate
your
system
for
different
kind
of
purposes.
So
this
is
example
of
basically
a
tool
that
is
quite
external
to
the
observability
space,
but
which
again
can
leverage
all
these
capabilities
and
many
other
tools
we
have
in
the
ecosystem
can
find
some
additional
use
cases
for
security
purposes,
for
deployment
purposes,
etc,
etc.
B
A
Yeah,
actually
it's
it's
interesting
and
going
back
to
the
to
the
pain,
I
think
even
within
my
current
company
logs
IO-
and
you
know,
we've
been
doing
as
log
I
always
is
all
about
observability
and
we've
done
that
so
well
on
production.
Still,
when
I
looked
into
the
cicd
pipelines,
I
saw
the
Gap.
I
saw
that
there's
like
we.
We
give
ourselves
discount
for
some
reason
and
not
not
investing
as
much
in
the
on
the
release
cycle
and
we've
been
through
a
process
in
which
we
realized
with
heavy
users.
A
A
A
Looking
at
the
run,
a
pipeline
run
as
a
trace
with
the
various
steps
as
it
spans
and
then
analyzing
it
in
this
View
and
and
actually
you
can
you
can,
then
it
goes
beyoncicd,
I
guess
into
the
the
build
process,
but
you
can
actually
get,
for
example,
integration
with
Maven,
and
then
you
can,
what
used
to
be
one
OPAC
long
span?
Suddenly
you
can
open
it
up
and
break
down.
A
B
B
For
me,
it's
not
only
about
going
deep,
but
also
about
going
to
the
high
level,
because
for
us
historically,
it
was
a
problem
that
each
tool.
So
there
is
a
lot
of
CI
NCD
tools.
Each
of
them
had
some
notion
of
observability
troubleshooting
analytics
embedded
and
we
as
users
as
the
maintenance
of
the
systems,
had
to
use
multiple
tools.
B
But
now
we
having
a
unified
standards
based
on
open
Telemetry,
CD
events
and
other
Technologies,
we
can
actually
rebuild
the
whole
delivery
Trace
from
commit,
or
if
you
want,
even
from
merge,
train
from
group
of
commits,
say
even
you
can
just
hook
on
value
stream
and
build
observability
from
very
beginning
when
before
the
court
appears,
and
then
you
get
observability
to
all
stages
and
all
tools
used
for
the
stage
service,
all
the
information.
So
this
is
invaluable
information
for
all
kinds
of
development
and
well.
B
Basically,
this
is
what
we
already
do
in
production,
and
this
is
what
is
important
part
because
yeah,
if
for
me,
one
of
Kitty
takeaways
from
devops
culture,
Etc,
that
everything
is
a
product
and
basically
your
operations
infrastructure.
Your
delivery
process
is
integral
part
of
the
product.
If
you
want
to
actually
do
continuous
delivery,
continuous
deployment
and
well
yeah
speaking
marketing
customers,
competitors
to
the
market-
hopefully
not
customers.
A
Yeah,
that's
for
sure,
and
I
I
I
really
invite
people
to
join
chime
in
that's
the
beauty
in
open
sources,
so
that
someone
like
me
or
Oleg
is
our
authorities
to
decide
it's
actually
a
group
discussion
by
the
community
and
and
do
look
us
up
in
the
cncf
slack
on
the
there's,
a
channel
cicd
Dash
observability.
But
in
a
short,
so
oh
one,
one.
Why
so
cicd-011i?
Why
sorry?
A
This
is
the
slack
channel
for
for
this
discussion,
chime
in
obviously
on
the
on
the
GitHub
I
I
shared
the
link
for
the
proposals
on
the
GitHub
itself
and
and
hopefully
to
make
it
more
formalized
and
also
good
to
see
the
discussion
in
the
on
the
CDF
side
on
the
continuous
delivery
side
in
in
multiple
six,
so,
obviously
to
see
the
events
that
was
mentioned,
but
then
there's
also
the
interoperability
Sig
and
the
the
best
practices
thing
right.
Lots
of
discussion,
Roundup,
observability,
yeah.
A
It
should
be
interesting
also
to
read
the
the
notes
from
that
discussion.
There's
another
point
that
I
wanted
to
discuss
with
you.
The
actually
the
the
previous
episode
we
had
here
on
the
on
the
show
I
I
was
I
did
that
on
Backstage
project,
where
I
hosted
the
director
of
engineering
from
Spotify
who
leads
to
project
development,
and
essentially
you
know
backstage
is
an
internal
developer
platform
that
Spotify
developed
for
for
its
engineering
and
then
open
sourced
and
donated
it
to
the
cncf.
A
B
Well,
yeah
so
backstage
itself,
it's
actually
a
developer
portal.
So
the
idea
of
developer
portal
is
quite
simple:
you
bring
in
all
metrics
all
information
that
is
useful
for
your
development
in
the
single
place
and,
of
course,
when
we
talk
about
operations,
when
we
talk
about
delivery,
all
kinds
of
observability
data
should
be
there
too.
B
Controller
point
and
the
position
center
for
all
kinds
of
operations
and
yeah
when
it
comes
to
CD,
cicd
I
believe
that
for
every
developer
there
should
be
something
on
this
portal,
so
whenever
they
use
Argo
whether
you
use
Jenkins,
sticktone
GitHub
actions,
you
need
to
have
quick
access
so
easily
on
the
developer
portal.
When
it's
the
pipeline
starts
failing,
you
get
a
big
notice
with
some
additional
observability
tips
to
access
things
quicker,
and
the
question
is
how
you
actually
get
it
so
for
backstage.
Yeah
backstage
has
some
notion
of
plugin
system.
B
Well,
I
would
say
that
a
lot
to
be
improved
today,
so
one
of
my
friends
Anton
wise,
he's
actually
working
on
some
experiments
about
developer
portal
and
plugin
ecosystem,
but
yeah
in
principle.
If
you
take
a
standard
tool,
let's
say:
Jenkins
Argo
CD,
whatever
you
install
a
plugin
for
backstage
and
get
it
working
and
there
are
plugins
for
these
tools.
The
problem
is
with
these
plugins
that
all
of
them
are
different
or,
and
none
of
them
actually
includes
much
of
observability
at
the
moment,
and
for
that
it
would
again.
B
This
is
another
opportunity
for
having
open,
standard
and
open
Integrations,
because
you
could
create
a
backstage
plugin,
let's
say
for
CD
events
or
for
cicd
observability
with
open
Telemetry.
So
basically
the
proposal
data
and
many
others
are
working
on
and
then
once
you
have
this
Plug-In
or
integration,
you
just
connect
it
and
regardless
of
your
CI
CD
system.
B
Basically
you
get
the
information,
so
whatever
you
use
under
the
hood,
when
you,
maybe
you
have
multiple
tools,
but
still,
if
there
is
common
notion,
let's
say
yeah
deliver
a
failure,
success
with
some
metadata
that
you
can
reverse
engineer.
So
this
is
already
a
part
of
the
standard
proposal.
Then
you
can
also
show
these,
regardless
of
how
the
system
is
implemented
and
for.
A
B
Yeah,
so
there
is
existing
plugin,
this
existing
plugin.
Well,
actually,
there
are
multiple
existing
plugins
like
it
happens
in
open
source.
The
ID
is
that
yeah,
since
open
Telemetry
has
a
standard
interface.
If
you
use
a
tool
like
parameters,
you
can
also
use
promptql
to
extract
the
data
which
is
so
much
more
capable
than
just
open
Telemetry,
so
you're
unlikely
to
use
backstitches
and
points
for
receiving
component
limited
data
right
most
likely.
B
You
will
query
a
database,
so
there
is
a
two
behind
open
Telemetry
that
stores
and
exposes
this
data,
so
it
could
be
yeah.
It
could
be
parameters,
it
can
be
data
dog,
dynatrace
whatever,
and
if
you
can
query
this
data,
you
can
actually
display
this
data
and
what
open
Telemetry
integration
does
the
basic
and
grafana
there
is
also
grafana
plugin.
So
basically
they
provide
you
with
a
set
of
standard
dashboards
for
common
operations.
So
we
believe
there
is
a
template
for
dorm
metrics.
That
makes
some
assumptions
based
on
existing
tools
and
actually
can
get.
B
You
started
quickly,
so
personally,
I'm
not
a
fan
of
Dora
metrics
at
all,
but
well
for
management
for
slides,
it's
okay
and
yeah.
It's
something
that
gets
you
started.
If
you
need
something
more
complicated.
Yes,
you
can
create
dashboards
in
your
tool.
You
can
embed
this
dashboards
into
your
developer
portal.
So,
for
example,
if
you
build
rafana,
you
will
just
embed
them
into
the
backstage.
You
don't
need
to
rebuild
them
again
inside
backstage
and.
B
B
So,
for
example,
have
an
SDK
that
could
be
embedded
into
whatever
Project
without
actually
bring
it
in
a
lot
of
vendors
dependent
code,
not
because
I,
don't
like
vendors,
but
because
I
want
my
users
to
have
freedom
of
choosing
vendors
and
basically
the
same.
It
would
be
interesting
for
tools
I
use
in
my
daily
job,
including
backstage.
So
again,
it's
a
question
of
unifying
things,
and
this
is
another
open
standard.
A
B
B
So
you
can
just
reach
out
there
and
ask
any
kinds
of
questions
you
have
and
if
you
also
want
to
chat
about
the
adoption
perspectives,
because
currently
the
adoption
is
probably
one
of
the
biggest
problems
for
all
these
tools,
because
Vaca
system
is
still
a
bit
scattered.
So
there
is
a
lot
of
implementations
yet
to
be
created,
though
there
are
some
hugs
and
tweaks
and
Standalone
tools
that
allow
to
somehow
integrate
if
needed,
and
if
you
want
to
chat
about
that,
I
would
be
happy
to
do
so.
A
The
first
one
that
I
chose
is
actually
especially
for
you.
I
I
found
it
very
interesting.
The
continuous
delivery,
the
state
of
continuous
delivery
report
that
the
CDF
produced
fairly
recently
so
first
of
all,
I
wanted
to
share
with
the
audience
and
obviously
a
worthwhile
having
a
look
at
the
full
report:
I'll
I'll
post
a
link,
as
always
on
the
on
on
the
chat
here
and
on
the
on
the
show
notes
for
the
listeners
at
the
note
on
the
live
stream.
A
One
thing
that
I
did
want
to
maybe
get
your
take.
It
said
that
it's
very
clear
that
that
there's
no
there's
no
clear
indication
that
the
velocity
for
code
changes
has
increased
over
the
last
two
and
a
half
years
among
the
general
developer
population
and
I'm
curious
as
a
cicd,
veteran
and
they've,
seen
all
the
migration
and
all
the
evolution
and
introducing
devops
and
the
process,
services
and
practices
and
tools
and
so
on.
And
then
we
see
this.
How
do
you
interpret
this
interesting
data.
B
B
Of
course,
if
you
approach
the
infrastructure
in
my
service
way,
if
every
your
container
is
a
cattle
of
course,
you
can
kill
things
and
restart
the
thing,
but
in
practice
not
many
of
us
actually
do.
That
and
I
believe
that
some
investment
in
the
quality
of
software
would
be
justified,
especially
everything
that
all
of
these
are
kind
of
solved
problem
in
all
the
architectures.
So
performance,
testing,
stress
testing
and
many
other
bits
so
I
believe
that
more
investment
should
be
put
in
there
for
cloud
native
development
and.
A
Cloudnet
fabrications
interesting
yeah.
It's
interesting
to
look
as
also
the
community
how
we
are
where
we're
we
need
to
improve,
because
you
know
seeing
no,
no
progress
in
in
two
and
a
half
years
for
me
was
striking
with
all
the
discussions
we've
been
going
on
in
the
community
and
the
tooling
and
everything
yeah.
B
So
the
problem
is
tolerance
is
that
yeah
there
is
a
lot
of
best
practices.
There
is
a
lot
of
best
practices
documented.
Some
best
practices
get
enforced,
but
not
enough.
So
I
believe
I
still
can
maintain
my
career
just
by
recommending
to
it
minus
ex
flux
to
your
bash
scripts,
because
even
if
it's
a
common
sense
for
last
40
years,
maybe
by
now
well
at
least
30..
It's
still
a
lot
of
people
forget
about
that
and
then
their
pipelines
fail
in
Reckless
ways
or
more
words.
A
Yeah
and
to
do
the
next
big
news,
at
least
for
me,
this
is
a
major,
and
actually
it
was
just
just
a
few.
A
few
days
ago,
last
week
that
open
Telemetry
protocol
otlp
reached
version
1.0,
so
1.0
otlp
is
out.
For
those
who
don't
know,
the
otlp
specification
defines
the
encoding
transport,
the
delivery
mechanism
for
Telemetry
data
across
the
different
signals,
traces,
metrics
and
logs
at
the
moment.
So
that's
a
very,
very
important
Milestone.
A
That
I
was
happy
to
we
escorted
throughout
the
show,
obviously
but
happy
to
share
this
reaching
maturity
and
reaching
stability
within
the
open,
Telemetry
project
and
actually
I
just
wrote
an
article
on
Open
Standards
in
observability
summarizing
the
important
updates
from
kubecon
and
later
so
for
the
past.
Let's
say
a
couple
of
months
or
so,
including
this
one,
you
know
otlp
and
the
sun
setting
of
open
senses,
the
Prometheus
standardization
and
more
highlights
that
you
don't
want
to
miss
so
obviously
highlight
this.
A
This
could
be
very
interesting
if
you're
looking
for
more
details
on
relevant
Open,
Standards,
open
specifications
and
things
around
that
and
recent
updates.
Another
very
interesting
thing
talking
about
the
open,
Telemetry
becoming
the
de
facto
standard.
It
was
interesting
to
see
even
Microsoft
also
realized
that
and
launched
Azure
monitor,
open,
Telemetry
distro.
They
call
it
and
saying
we
are
refactoring
azure's
native
observability
platforms
to
be
able
to
be
based
on
on
open
Telemetry.
A
Another
thing,
maybe,
on
the
on
the
event
side,
very
important,
as
we
all
are
gearing
up
towards
the
cubecon
North
America
and
Chicago
in
October
I
wanted
to
for
developers
in
the
audience
make
sure
that
you
see
the
upcoming
kubecon
will
hold
for
the
first
time
at
developer-centric.
Co-Located
event
app
developer
con,
so
it
will
cover
cncf
projects
such
as
k-native
and
Dapper.
A
So
if
you
are
developers-
and
you
felt
a
bit
out
of
context
or
or
not
properly
treated
on
previous
coupons,
this
is
about
to
change
and
do
check
it
out
I'll
post
the
link
also
here
for
the
app
developer
con.
What
do
you
think
about
that
you're
also
from
a
developer
background?
So
how
do
you
feel
about
that
or
like.
B
I'm
definitely
interested,
unfortunately,
I
won't
be
able
to
participate
in
person
because
you're
getting
a
US
Visa
right
now.
It's
mission
impossible
for
me,
but
I
will
be
participating
remotely
as
a
program,
Committee,
Member
and
many
other
ones.
So
hopefully
you
will
have
a
live
stream
from
the
conference.
B
A
Ambassador
I'm
sure
that
you're
happy
to
spread
the
word,
even
if
you'll
know
there,
because
this
is
a
major
thing
for
developers
and
I
know.
At
least
in
my
in
my
circles,
that's
got
a
lot
of
attention.
Another
important
update
just
from
the
recent
couple
of
days
that
Jaeger
tracing
1.47
is
out
for
those
who
are
on
the
on
the
open
source
side
of
the
distributed
tracing.
A
This
is
the
the
open
source
under
the
cncf
stack
for
distributed
tracing
lots
of
things
there
Cassandra
configuration
feature
for
TLS,
disabling
Kerberos
for
a
fast
negotiation
from
Kafka
consumers,
support
Prometheus
normalization
for
open
Telemetry,
metrics,
a
proposal
and
Telemetry
and,
in
general
other
updates
around
open,
Telemetry
compatibility
and
another
updates
so
do
check
it
out.
I
think
definitely
an
interesting,
interesting
one
to
see
and
showing
the
both
the
interoperability
and
the
standardization
that
that
you
and
I
O
like
just
talked
about
also
on
this
on
this
chat.
Yeah.
B
I
think
it's
a
really
good
progress
on
Holocaust
system,
so
open
Telemetry
definitely
becomes
a
commodity
for
all
kinds
of
Eventing
within
the
scope
of
what
is
supported
for
by
open
Telemetry,
though
I
still
hope
to
see
a
more
standardization
and
unification
happening
in
areas
that
are
not
yet
supportable
open,
Telemetry
or
Cloud
events.
There
is
still
a
lot
of
gaps
for
for
observability
that
we
could
address
by
introducing
more
Open
Standards.
A
Yeah,
that's
for
sure.
Maybe
we
have
time
for
one
last
note,
there
was
a
the
recent
devops
pulse
survey,
the
yearly
server
about
the
state
of
devops
and
in
particular,
obviously
things
related
to
observability.
It
was
very
interesting,
I
lots
of
data
there
and
it's
it's
publicly
available,
but
I
I
did
compile
a
few
interesting
bits
and
and
posted
them
as
a
as
an
article
under
APM,
digest,
I
call
it
observability
as
a
complexity
problem,
which
is
something
that
we've
discussed
also
on
this
on
this
show.
A
A
Yeah
pleasure
and
definitely
going
to
follow
up,
as
I
said,
also
on
the
on
the
City
events
call
next
week
or
in
a
couple
weeks
and
in
this
month
and
more
so.
Thank
you
very
much
and
also
thank
you
for
all
our
guests
for
joining
me
and
for
chiming
in
on
this
on
this
episode.
As
always,
all
the
episodes
are
available
on
your
favorite
podcast
app
or
on
YouTube
on
our
YouTube
channel.
A
If
you
are
listening
now
on
the
podcast
app
then
do
know
that
we
stream
the
episodes
live
on
Twitch
and
YouTube
live.
So
just
you
can
find
all
the
details
about
that
on
our
Twitter
handle
at
open,
observe
where
about
the
times
of
the
upcoming
live
streams,
and
also
to
share
your
comments,
suggestions,
news
bits.
If
you
think
there
are
important
things
that
I
missed
happy
to
get
more
news
bits
from
you,
I'm
Dothan
Orbitz.