►
Description
Presented at Open Source 101 2023 - Charlotte
Presented by Eric D Schabell - Chronosphere
Title: Observability For You and Me with openTelemetry
Abstract: Are you interested in dipping your toes in the cloud native observability waters, but as an engineer you are not sure where to get started with tracing problems through your microservices and application landscapes on Kubernetes? Then this is the session for you, where we take you on your first steps in an active open-source project that offers a buffet of languages, challenges, and opportunities for getting started with telemetry data.
The project is called openTelemetry, but before diving into the specifics, we’ll start with de-mystifying key concepts and terms such as observability, telemetry, instrumentation, cardinality, percentile to lay a foundation. After understanding the nuts and bolts of observability and distributed traces, we’ll explore the openTelemetry community; its Special Interest Groups (SIGs), repositories, and how to become not only an end-user, but possibly a contributor.We will wrap up with an overview of the components in this project, such as the Collector, the OpenTelemetry protocol (OTLP), its APIs, and its SDKs.
Attendees will leave with an understanding of key observability concepts, become grounded in distributed tracing terminology, be aware of the components of openTelemetry, and know how to take their first steps to an open-source contribution!
You can find slides here: https://www.slideshare.net/AllThingsOpen/observability-for-you-and-me-with-opentelemetry
Find more info about Open Source 101:
On the web: https://opensource101.com/
Twitter: https://twitter.com/os101series
LinkedIn: https://www.linkedin.com/company/open-source-101/
Instagram: https://www.instagram.com/os101series/
Facebook: https://www.facebook.com/os101series/
A
Welcome
are
you
in
the
right
room,
we're
going
to
talk
about
some
open,
Telemetry,
some
open
source
stuff?
All
of
this
is
going
to
be
introductory
level.
Welcome,
come
on
in
I
kind
of
wanted
to
get
an
idea
who
I'm
talking
to
so
how
many
of
you
are
students
here,
none
of
you,
I
was
told
students
were
coming,
I
guess,
I
scared
them
away;
Developers
hold
it
hold
them
up
a
second
okay:
architects
to
managers,
three,
a
couple
of
managers:
what
are
you
doing
here?
A
A
the
rest
of
you,
30.
Jesus,
my
gosh
man,
okay,
so
you're
at
an
introductory
open
source.
That's
kind
of
interesting
to
me
because
usually
this
is
something
you
get
into
when
you're
a
little
bit
younger
and
stuff,
how
many
people
been
working
in
open
source
at
all
for
your
whole
careers
or
all
right.
Anybody
working
with
open,
Telemetry,
anything
like
that
in
observability
space.
So
you
know
all
about
open
Telemetry.
Why
are
you
here
just
making
sure
I'm
telling
the
truth
or
you
know
all
right?
What
are
we
going
to
talk
about
today?
A
It's
pretty
pretty
straightforward,
pretty
simple,
I'm
not
going
to
try
to
go
too
deep.
Technically,
it's
an
introductory
course
you're
going
to
get
a
little
bit
of
background
on
cloud
native
observability.
You
notice
that
weird
term
does
everybody
know
what
Ali
is
everybody
know
what
kubernetes
is
you
know?
Sometimes
they
talk
about
k8s.
You
know
what
the
8
stands
for
number
of
letters
between
the
first
and
the
last
right
same
thing,
with
observability
now
I
used
to
work
at
a
in
the
app
Dev
space
and
now
I
work
in
the
observability
space.
A
I
get
really
tired
of
everybody
gets
tired
of
typing
that
into
stuff
right.
It's
really
long!
It
takes
up
too
many
things
on
the
social
media.
It's
pain
in
the
butt,
so
everybody
calls
it
Ollie,
don't
get
panic.
That's
all
it
is.
The
big
deal
about
observability
is
not
just
doing
observability
but
doing
it
at
scale
and
what
that
means
right.
So
we
start
going
into
the
cloud
and
Cloud
native
and
get
do
the
stuff
at
scale.
A
Just
so.
You
have
an
understanding
of
what
you're
dealing
with.
Is
this
thing
and
don't
try
to
position
it
in
the
wrong
way
inside
your
architecture?
If
you
want
to
explore
this
stuff
and
then
getting
your
feet,
wet
is
not
so
much
about
how
to
contribute
to
open
source,
because
I
think
you
can
figure
that
out
and
there's
enough
talks
about
that
stuff.
But
in
this
specific
case,
how
you
can
engage
is
I.
A
Think
a
little
bit
more
important
a
little
bit
how
it
works
around
the
cncf
which
they're
a
part
of
also,
if
you
have
any
questions
or
anything,
feel
free
to
stick
up
a
hand.
I
got
some
pretty
bright
lights
in
my
face,
but
if
you
hold
it
up
long
enough,
I
will
find
you
see
you
unless
you're
sitting
right
there
or
right
there,
because
you're
really
in
the
lights,
you
might
have
to
wave
at
me
a
little
bit
yeah.
A
So
that
would
be
a
good
idea,
so
Cloud
native
observability,
slash
at
scale
so
I
think
this
ice
cream.
Pretty
much
says
it.
How
many
people
here
work
with
cloud
in
their
work
every
day?
So,
like
you,
know,
actual
Cloud
providers
log
into
that
stuff?
It
feels
like
you
dropped
your
ice
cream
right.
They
promised
a
lot
of
stuff
everything's
going
to
be
fantastic.
I've
done
some
maintained
stuff,
since
we
have
such
a
mature
audience
here
in
the
in
the
I.T
space.
A
We
all
remember
the
days
where
we
did
everything
on
our
laptop
right
when
you
first
started
developing
everything.
Was
here
locally
your
servers,
your
your
your,
your,
your
your!
Your
Java
apps,
whatever
it
was,
you
were
doing,
it
was
fantastic
right.
You
had
control
over
everything
when
you
deploy,
okay,
you
throw
it
over
the
wall
and
the
operations
on
whatever
floor
in
the
building
was
at.
They
would
get
this
stuff
and
move
it
to
the
next
VM
or
the
next.
You
know
data
center
piece
or
whatever
it
was
when
everything
goes
up
into
the
cloud.
A
It's
supposed
to
be
really
easy.
It's
just
entries
in
my
IDE
and
everything's
supposed
to
be
fantastic,
but
I
feel
like
this
too.
It's
like
my
ice
cream
fell
on
the
floor.
They
didn't
they
didn't
really
come
out
with
the
promises
all
the
time
right
initially,
when
you
get
in
there.
It's
super
easy
right.
Everybody
know
what
a
container
is.
A
It's
not
these,
but
just
for
the
image
right,
building
a
couple
of
containers
and
getting
started
on
your
laptop
and
and
moving
that
in
a
small
organization
over
onto
a
platform
like
kubernetes,
whatever
Cloud
you're,
using
not
that
hard
right.
We
all
get
our
heads
around
that
really
quickly.
We
start
pushing
this
stuff
out.
It
seems
like
it's.
A
Okay,
right,
not
a
big
deal
when
you
start
looking
at
what
is
happening
in
the
cloud
and
I'm
easing
a
little
bit
of
the
the
ollie
stuff
in
here,
I
Googled
around
a
little
bit
to
find,
like
a
just
an
example,
application.
Something
really
simple
in
the
speaker
notes.
Also,
this
slide
stuff
is
online
right
now,
my
website,
it's
my
last
name.org,
so
you
can
see
the
speaker
notes
of
this
stuff.
A
link
to
this
example
itself.
It's
pretty
representative
for
what
I
could
find
I
found
several
examples.
A
This
is
roughly
some
well
documented
anyway,
it's
just
a
hello
world
out
very
simple.
Like
a
front-end
website
thing,
he
generated
some
hits
on
the
front
end
like
some
users
might
be
using
nothing
dramatic,
just
really
light
loads,
and
then
he
put
all
the
metrics.
You
do
around
that
stuff
to
collect
and
look
at
it.
So,
like
end
user
metrics,
the
actual
app
metrics
the
nodes,
it's
a
four
node
cluster
and
kubernetes.
That's
it!
What
do
you
think
happened?
A
How
much
data
got
generated
around
this
I'm?
Not
talking
about
the
data
that
you're
doing
as
a
business
I'm
talking
about
the
data
around
the
infrastructure
that
you're
collecting
half
a
terabyte,
half
a
terabyte,
think
about
that!
That's
that's,
not
a
scale!
That's
one!
How
many
apps
are
you
running
in
your
company?
You
know
what
I'm
saying
so
just
keep
this
in
the
back
of
your
mind
a
little
bit
right.
This
is
this
is
why
the
ice
cream
ends
up
on
the
floor.
A
On
top
of
that,
when
you're
talking
about
observability
data,
what
most
people
as
developers
don't
think
too
much
about,
we
don't
care!
You
know
you
want
me
to
put
some
tags
and
labels
and
instrumentation
in
my
code
no
problem
on
average
companies
have
to
keep
this
or
are
keeping
it
for
13
months,
so
the
half
a
terabyte
is
being
kept
13
months
for
each
one.
That's
just
30
days
of
collection.
That's
all!
It
is
a
lot
of
them.
A
Don't
really
think
about
this
until
they
start
seeing
the
bills
and
they're
like
what
can
I
do
about
this.
One
of
the
things
you
can
do
is
think
about
instead
of
having
and
we're
all
running,
really
fast
like
when
the
pandemic
hit
places
like
doordash
and
things
like
that,
exploded
right
and
so
their
infrastructure
exploded
underneath
them
because
everybody
was
rushing
at
that
stuff
and
so
it
auto
scales
up.
You
start
storing
all
this
data.
You
start
getting
the
bills
in
you're
like
hey,
wait
a
minute,
but
what
about
those
training?
A
Environments
that
only
come
out
for
two
weeks
right?
Somebody
spins
up
a
course
and
is
working
on
it
for
two
weeks,
and
it
goes
away
if
you're
just
running
around
doing
all
this
stuff.
You
don't
have
all
these
fine-grained
plans
in
place.
It's
just
a
policy
across
the
organization
for
13
months
right
instead
of
saying
hey.
Maybe
this
environment
only
needs
to
be
for
two
weeks.
We
could
throw
that
data
away
right
or
not,
keep
it
at
all,
and
we
really
don't
care
about
the
training
environments,
something
to
think
about.
A
So
those
two
containers
as
you
grow
start
turning
into
something
like
this:
a
lot
of
containers,
a
lot
of
areas
of
your
infrastructure
getting
filled
up,
maybe
you're,
using
multiple
clouds,
maybe
using
some
local
stuff
each
one
of
those
cranes.
Think
of
that
as
a
piece
of
tooling
you've
added
around
this,
so
it
starts
to
get
really
kind
of
big
right,
cumbersome.
You
have
devops
teams,
you
started
scaling
up.
Is
anybody
here
doing
devops?
A
Only
three
four
are
y'all
lying
you're
trying
to
do
devops,
maybe
maybe
you're
doing
devops,
maybe
you're
not
whatever,
but
the
idea
is
that
you
start
melting
these
teams
together,
you
have
processes
in
place
where
everybody's
kind
of
helping
out
across
all
of
it
right.
The
thing
is
to
get
your
head
around
this
stuff
and
start
growing,
your
your
observability
infrastructure
around
it.
Now
you
have
on-call
stuff
right.
Somebody's
anybody
here
carry
a
beeper
for
their
work
or
do
on-call
stuff.
One
two,
three
four
ish!
A
A
A
loaded
question:
it's
a
loaded
question,
a
really
a
fun
story.
I
hope
she
doesn't
mind.
I
work
with
a
lady
that
I
hired
here
that
spent
six
jobs
as
an
SRE.
That
was
their
first.
She
transitioned
out
of
something
completely
different
into
it
and
thought
SRE
would
be
fun.
She
burned
out
six
times
right.
The
last
time
when
I
finally
met
her
I
said:
what
are
you
looking
to
do?
A
Go
to
get
the
seventh
burnout
or
how
about
you
come
over
here
and
start
taking
all
these
great
things,
you've
learned
and
all
these
things
and
share
back
right.
Do
what
I'm
doing
teach
people
and
she's
now
on
board
doing
that,
and
she
has
amazing
Stories,
ask
him
I
bet
you
have
all
kinds
of
cool
stories
about
when
it
didn't
go
well
or
what
happened
and
how
you
fixed
it,
and
these
stories
are
are
golden.
It's
really
really
really:
educational.
To
talk
to
people
like
that.
A
One
of
the
funniest
answers
when
I
I
ask
this
everywhere.
I
go
that
question.
One
of
the
funniest
answers
I
ever
got
was
the
guy
said:
I
do
an
SRE
for
a
while
and
then
I
go
back
to
development
and
I'm
tired
of
that
I
go
back
to
SRE,
so
he
wears
it
off
before
he'll
go
back,
so
you
get
it's
sort
of
like
when
the
I
think
when
women
have
babies
like
they
forget
all
that
pain,
and
then
they
have
another
one.
You
know
what
I
mean.
A
It's
like
I
hear
this
from
my
wife
all
the
time.
It's
not
my
story,
but
it's
it's
it's
how
we
forget
the
pain
to
go
and
have
a
second
kid
right,
I
think
it's
kind
of
what
he
was
doing,
it's
kind
of
funny,
but
again
back
to
this
Cloud
native
at
scale.
It's
hard
right.
You
start
finding
out
how
hard
this
is
and
yeah.
A
So
we
look
back
at
the
simple
containers.
The
Ali,
the
observability
you're
setting
up
the
monitoring
is
pretty
easy
right.
The
view
is
wonderful.
Everything
is
easy
to
understand.
You
might
have
a
single
pane
of
glass.
You
know
the
the
big
dream
that
everybody
had
one
place
to
see
everything
that's
going
on.
You'll
have
a
Dev,
devops,
rolling
everybody's,
pretty
happy.
Everybody
understands
the
infrastructure.
The
on-call
things
aren't
too
bad.
You
know
something
goes
down.
A
Everybody
knows
how
to
find
it
and
fix
it,
but
when
we
get
to
that
large-scale
stuff-
and
it
doesn't
take
much
to
get
to
this
I
know
this
is
a
super
extreme
image.
But
it's
not
hard.
It
really
isn't
she
didn't
burn
out
six
times
for
nothing.
You
know,
there's
lots
of
reasons
for
that.
I
won't
go
into
that,
but
what
you're
seeing
here
is
people
are,
are
going
on
call
and
getting
burned
out.
A
You
see
that
if
all
my
devops
teams
are
over
here
working
and
doing
the
dev
and
the
Ops
like
they're
supposed
to
and
the
new
new
business
and
all
that,
and
then
you
start
developing
your
own
Ali
infrastructure,
your
own
observability
stuff,
more
and
more
people
from
this
side
are
being
drug
over
to
this
side.
To
maintain
this,
and
that's
not
what
you
signed
up
for
you
know,
I,
don't
know
any
developers
that
really
love
triaging
issues
really
funny
thing:
I
did
for
a
different
talk
was
looked
at
a
developer's
time.
A
A
Around
18
of
their
time
is
spent
on
coffee
machine
lunch
meetings,
whatever
you're
doing
and
32
percent
of
your
time
is
spent
on
actual
developing
new
code.
You
know
doing
you
want
to
do
being
Innovative
what
you
signed
up
for
what
you
study
for
the
reason,
you
love
what
you
do
and
that's
super
painful
that
50
of
your
time
is
doing
stuff.
You
don't
want
to
do.
We
actually
have
an
observability
report
that
came
out
that
we
get.
A
We
had
independently
researched
it
at
chronosphere
where
I
work
and
they
had
they
had
32
percent.
We
had
500
developers
and
Engineers
answer
this
thing
that
are
actually
working
at
large
and
born
in
the
cloud
organizations
and
32
percent
of
them
said
that
this
is
affecting
their
personal
life.
That
they're,
you
know
not
happy
with.
What's
going
on
and
29
wanted
to
quit,
Jesus
I'm,
sorry,
but
that's
isn't
that
nuts,
you
know
I
mean
we're,
not
really
surprised.
If
you
look
around
you,
you
know
and
you've
been
in
one
of
these
organizations.
A
I
mean
they're,
not
it's
not
not
everything's
on
fire,
that's
not!
What's
going
on!
It's
just
super
complicated
right
and
things
are
very
fast
paced
in
this
world,
they're
scaling
out
of
control
really
quickly.
So
you
can't
get
your
head
around
all
this
stuff,
really
interesting.
That
I
watched
a
I
believe
it
was
an
IDC
talk
on
cloud
in
Europe.
It
was
Europe
report
and
it
was
over.
A
80
percent
of
companies
last
year
were
going
to
have
a
fin
Ops
role
by
the
end
of
this
year,
which
is
a
huge
number
for
an
analyst
to
put
on
a
piece
of
paper.
That's
huge,
usually
40
dish
is
already
a
trend
right.
Eighty
percent
like
within
that
they
can't
control
these
costs.
They
can't
figure
out.
What's
going
on.
A
The
bottom
line
is
water
through
a
pipe
is
where
it's
coming
from,
but
it
gets
a
little
scary
and,
what's
really
remarkable,
is
they're
seeing
really
directly
that
spend
so
when
you,
when
you're
here,
at
what
I'm
doing
as
a
business
and
my
data
and
my
spend
here
to
make
whatever
I'm
doing
with
my
customers
and
you're
watching
this
go
like
this,
and
and
we
see
it
time
and
again
at
customers.
It's
crazy
question.
B
Yeah
in
Europe,
yes,
most
of
the
time
what
the
company
I'm
in,
where
I'm
an
SRE,
we
they
not
only
is
it
just
very
chaotic
and
I'm
in
that
kind
of
burnout
phase
that
you're
talking
about,
but
then
we're
also
fully
remote,
which
seems
to
compound
it
because
it
sort
of
feels,
like
you
know,
I
feel
very
untethered.
Not
seeing
people
not
getting
coffee
having
the
banter.
A
A
A
I
was
working
out
where
you
you
do
your
code,
you
throw
it
over
the
fence
and
they
come
back
and
say:
I
see
all
these
Java
exceptions,
I,
don't
know
what
the
hell
this
means
you
have
to
run
down
to
the
third
floor
and
go
look
at
the
stuff
over
their
shoulder
and
trying
to
figure
out
what's
going
wrong
to
get
it
to
work
in
the
test
environment.
So
we
can
push
it
further
in
the
in
the
production
process.
A
Maybe
face
to
face
would
help
a
little,
but
I
think
that
that
you
know
not
listening
and
being
overwhelmed
with.
What
we're
talking
about
is
is
is
is
a
problem
that
that's
my
personal
take
on
what
I'm
seeing
I
don't
know
if
that
resonates
with
what
you're
you're
feeling
but
yeah
it's
it's
it's
hard
right.
It's
it's!
What
everybody's
looking
to
fix
and
and
when
you
look
at
that
the
problems
that
are
going
on,
let
me
jump
to
the
next
one.
This
is
possible.
Right
There
are
places
that
have
done
this.
A
So
this
is
a
picture
of
a
well-organized
infrastructure
right.
You
even
have
some
special
project
that
are
really
nice
that
are
run
really
well.
When
you
do
this
stuff
at
scale,
it's
more
about
taking
the
data
and
all
this
information
you
got
coming
in
and
and
quantifying
what
you
need
and
aggregating
away
what
you
don't
so,
instead
of
having
that
big
flood
of
stuff
coming
in
and
paying
for
all
that
and
having
this
problem
and
now
reducing
the
thing
down
per
section
of
your
organization
per
team.
Has
a
dashboard
has
the
information
they
need.
A
You
need
tools
in
place
that
can
do
this
and
you
need
processes
in
place.
You're,
seeing
things
like
platform,
engineering
being
defined
and
expanded
you're,
seeing
SRE
teams
being
defined
and
expanded
you're
seeing
centralized
observability
teams,
you
see
layers
of
observability
and
control
coming
over
the
top
of
this.
You
see
fin
Ops
being
put
in
place,
I'm,
not
saying
it's
easy
to
do
this
and
everybody
always
has
it
under
control,
because
you
don't
know
what's
going
to
happen
tomorrow
at
that
scale,
all
of
a
sudden
something
explodes
like
GameStop.
A
B
A
Fair
you've
been
in
it,
I
was
going
to
ask
all
the
Americans
and
see
what
they
say
yeah.
This
is
Barcelona,
it's
pretty
crazy
when
you're
walking
around
you
don't
feel
like
it's
that
organized.
But
when
you
come
in
on
an
airplane
pay
attention,
it's
it's
a
beautiful
city,
so
we're
going
to
slowly
start
diving
down
into
into
the
to
the
information
we
want
to
get
into
around
the
Telemetry
and
observability
in
the
in
the
open,
Telemetry
space.
A
So
I
want
to
give
you
a
few
definitions,
so
we're
all
speaking
the
same
language
and
kind
of
know
the
right
right
area
we're
doing
here
so
basically,
Telemetry
is
collecting.
You
know
remote
systems,
information
really
broad
right.
Anything
it'll
give
me
I'll,
take
it
right.
You
have
different
areas
such
as
you
know,
metrics,
and
things
like
that
that
are
that
are
quantifying
this
space,
but
that's
all
Telemetry
in
theory,
all
good,
let's
get
down
into
some
of
the
nuts
and
bolts
of
what
the
open
Telemetry
is
doing.
A
A
That
was
the
first
kind
of
complete
platform
collecting.
You
know:
Telemetry
information
in
the
open
source
world
in
the
cncf.
It's
a
graduated
project.
It's
fully,
you
know
functional
behind
that
came
things
like
open
tracing
and
a
couple.
Others
attempts
to
kind
of
quantify
this
specific
space
where
we're
talking
about
microservices,
where
you
have
a
really
disparate
bunch
of
things
going
on
in
your
infrastructure,
and
you
want
to
figure
out
which
call
when
to
where
and
how
long
and
what
what
happened
on
the
way
you're.
A
Basically,
it's
a
complete
complex
distributed
system
and
collecting
metrics
and
and
Telemetry
information
around
that
was
kind
of
a
new
thing.
It's
really
hard
I
had
friends
who
used
to
do
this
for
companies
and
they
spent
a
lot
of
time
trying
to
figure
out
what
was
going
on
in
the
testing
environment,
trying
to
find
the
good
with
good
practices
around
how
to
do
this
I
think
it
was
2019
they
merged
these
all
into
open,
Telemetry.
Just
one
thing:
one
one
project
one
push
forward:
it
is
not
a
graduated
project,
yet
it's
on
the
way.
A
It's
in
the
I
think
the
second
phase
of
the
incubation
phase
go
check.
It
out
beautiful
documentation,
really
good,
getting
started.
It's
really
nice
to
have
developer
and
Ops
divided,
so
you
can
take
paths
through
the
learning
thing
in
the
beginning.
It's
pretty
neat
what
it
is.
Not
anybody
here
heard
of
Prometheus
and
Jaeger,
and
things
like
that.
Those
are
actual
platforms,
so
they
collect,
you
know
all
of
it.
They
store
it
they're
the
whole
back-end
stuff.
A
This
does
not
have
a
back
end,
so
what
it's
doing
is
generating
stuff
on
the
front
end
and
pushing
it
to
any
back
end.
You
want
in
theory,
that's
the
idea.
Right
just
want
to
be
real
clear
on
that,
because
a
lot
of
people
jump
out
and
they're
like.
Oh,
we
got
all
this
nice
stuff,
but
you
have
to
have
a
back
end
already
have
some
kind
of
metrics
collection
on
the
back
end
Telemetry
collection
now
getting
a
little
farther
into
the
Telemetry
in
this
space.
A
These
are
basically
the
two
things
you're
going
to
hear
about.
You're
going
to
hear
about
traces
and
you're.
Going
to
hear
about
spans
so
span
is
a
single
unit
to
work
it
gets
defined.
It
doesn't
really
matter
what
the
details
around
that
are,
but
think
of
like
a
name
and
address
that
kind
of
thing,
and
then
a
trace
consists
of
multiple
spans.
A
So
a
really
simple
analogy
for
you
all
those
microservices
out
there
a
span
would
be
a
hop
from
one
to
the
other,
and
a
trace
would
be
the
complete
path
it
took
to
get
the
answer
back
to
the
customer,
we'll
look
at
what
that
looks
like
a
little
bit
later.
I
owe
you
that
one!
Yet
what's
inside
of
open
Telemetry
there's
a
specification,
it's
pretty
straightforward!
You
have
an
API
that
it
works
against.
Everybody
loves
apis
right.
We
have
a
contract,
it's
very
simple.
We
understand
what
we
could
use
and
what
we
can't
do.
A
You
have
sdks.
These
are
the
things
to
use
for
language
specific
tasks.
A
lot
of
the
auto
instrumentation
has
approached
this
through
sdks.
If
you
have
some
special
you
want
to
do,
you
can
go,
get
one
for
Java
and
do
something
special
right.
You
can
add
more
stuff
to
it
and
then
there's
a
special
data
specification.
A
It's
called
the
open,
Telemetry
I,
believe
language
protocol
o
TLP
kind
of
funny,
because
I'll
come
back
to
it
at
the
end,
when
we
talk
about
how
to
engage,
there's
a
special
interest
group
right
now,
that's
working
on
a
specification
across
all
this
landscape
about
how
to
how
to
how
to
describe
this
protocol
that
we're
doing
with
with
Telemetry
in
general,
a
little
bit
more
broad
than
just
this.
This
is
very
specific
to
tracing
Telemetry
data.
A
So
what
does
this
look
like?
Everybody
loves
an
architecture
right,
very
simple,
without
getting
too
detailed
into
what's
going
on,
the
three
things
I
just
described
you
see
are
inside
I'm
a
Java
guy
from
the
old
day,
so
I
do
job.
In
my
examples,
anybody
here,
Java
developer
you'll,
have
a
job
forever
right.
A
That's
my
plan
when
I
retire,
if
I
get
bored
I'm
going
to
do
some
Java
stuff,
like
the
old
guys
used
to
do
Cobalt
at
the
bank
right,
there's
so
much
job
out
there
anyway,
Auto
instrumentation
is
the
one.
That's
really
great
right.
It's
just
you.
You
load
that
Library
dependency
or
whatever
into
your
language
and
it'll
self-detect
in
your
apps,
certain
basic
things
around
Telemetry
data.
A
You
also
have
the
ability
to
do
that
by
hand.
You
don't
have
to
do
it
with
automation.
So
if
you
have
specific
tags
and
labels,
we
talked
to
I
mentioned
multiple
times:
I,
don't
care
as
a
developer
when
my
specification
comes
in
and
they
want
me
to
put
labels
and
tags
and
stuff
on
I'll.
Do
that
you
know
I,
don't
care
if
I'm
blowing
up
at
the
other
end
my
bill
for
the
month.
It's
not
my
fault,
somebody
else
asked
for
this.
A
You
see
the
the
API
and
SDK
might
be
involved
or
not,
and
then
you
have
a
collector,
it's
basically
running
on
the
same
node,
and
it's
just
collecting
this
stuff
right.
We'll
we'll
talk
a
little
bit
about
the
magic
inside
of
that
collector.
Choosing
the
protocol
to
push
this
stuff
as
it
does
for
each
one
of
these
traces.
Each
one
of
these
spans
is
collecting
all
this
stuff
and
pushing
it
out
of
some
form
or
another.
A
Now
we
didn't
have
a
back
end
right.
So
now
we
need
to
add
a
back
end
of
some
sort
to
actually
collect
this
stuff.
Where's
the
collector
going
to
push
it
right
and
you
still
see
we're
using
the
really
basic
agent
there's
two
ways
you
can
do
it:
you
can
do
an
agent
or
a
Gateway,
we'll
look
at
the
Gateway
in
a
second.
The
agent
is,
on
the
actual
same
place
that
your
apps
are
at
right,
sidecar,
that
kind
of
thing,
if
you're
doing
kubernetes
I
think
it's
called
a
demon
set.
A
Whatever
you
know
details
you
can
find
that
in
the
documentation
using
the
the
protocols
pushing
all
that
stuff
back
out
to
whatever
back
end,
you
want
the
back
end
is
needed
because
that's
the
time,
series,
storage
right.
We
need
the
storage
special
kind
of
database.
So
we
can
query
that
stuff
and
produce
dashboards
and
all
this
fun
stuff
right
your
day,
job
right,
very
much
so
I'm
not
lying
yet
am
I.
Anybody
checking
up
on
me
all
right.
A
So
if
we
get
to
the
point
where
you're
going
to
use
a
Gateway,
this
might
happen
when
you
start
getting
really
big
right,
you're
going
into
the
clouds
you
do
it
per
cluster
per
data
center
per
region.
You
add
a
collector,
it's
basically
the
same
thing
as
the
agent.
It's
just
collecting
a
bunch
of
agent
stuff
together,
it's
another
layer
of
abstraction,
think
of
a
Gateway
of
some
sort,
and
it
says
the
same
thing
pushes
it
to
the
back
ends
as
you
wish
right,
everybody
on
board
so
far
good.
A
A
So,
if
you
remember,
I
was
talking
about
all
this
flood
of
data
you
have
and
stuff.
In
this
process,
you
can
Define
what
you
want
and
what
you
don't
want
so
by
hand.
Unfortunately,
you
know
not
a
lot
of
help
there.
You
can
go
and
decide
what
you
want
to
keep
and
what
you
don't
want
to
keep.
You
can
decide
what
you
want
to
aggregate.
A
So
after
we've
manipulated
everything,
we
say
export
or
send
it
back.
So
the
exporter
is
basically
designing
how
are
used
to
say
what
back
end
I'm
using
put
it
in
the
format
it
wants
and
sends
it
to
it.
I'm
pretty
sure,
there's
exporters.
There
was
actually
a
bit
of
a
hoo-ha
about
one
of
the
the
closed
Source
vendors
manipulating
one
of
those
exporters,
one
of
the
maintainers
of
the
exporter
about
making
it
so
easy
to
push
open
store
stuff
into
their
back
end.
A
So
go
look
that
one
up
or
ask
me
in
the
in
the
break:
I'm,
not
gonna,
because
we're
live
streaming.
I'm
not
gonna
hit
on
anybody,
but
it's
a
good
example
of
Open
Source
and
what
happens
when
you
don't
play
nice
right,
but
these
exporters
are
there's
a
there's,
a
huge
collection
of
these
things,
so
you
can
use
almost
anything
you
want.
Most
of
the
examples
I
put
in.
There
are
just
the
open
source
stuff
you'll
find
at
the
cncf
other
projects,
some
big,
some
small.
B
A
Yeah,
so
the
question
is:
is
my
drawing
lying?
If
I
can
summarize
that,
if
you're
using
Prometheus,
for
example
as
a
back
end,
the
the
question,
the
the
guy
that
asked
the
question
thinks
that
they
you
talk
about,
scraping
it
scrapes
yeah,
there's
push
and
pull
yeah,
it
depends
if
I
had
to
find
it.
So,
for
example,
if
you're
using
a
Prometheus
exporter,
it
would
be
scraping
that
end
point.
A
Basically,
exposing
a
net
is
kind
of
technically
exposing
an
end
point
and
you're
scraping
that
yeah
anybody
here
use
Prometheus
or
all
of
you
use
Prometheus,
not
too
many
Prometheus,
basically
exposing
HTTP
HTTP
endpoint
and
it's
called
something
slash
metrics
and
you
can
actually
open
in
the
browser
and
it
dumps
everything.
That's
basically
the
apis.
You
can
query
all
right,
so
you
can
see
all
the
the
different
calls
and
metrics
you
can
see.
So
that's
what
it
would
do
here.
In
that
case
it
would
expose
an
endpoint
like
that
answer.
Your
question.
A
Think
your
back
end,
so
the
question
again,
the
question
summarized
is
again
about
back
backhands,
pushing
or
pulling.
How
would
you
decide
which
one
to
use
I
think
your
back
end?
Pretty
much
decides
that,
for
you,
Prometheus
is
scraping
targets,
that's
just
the
way
it
goes,
I
don't
know
if
you
have
an
advantage
either
way,
it's
just
what
it
does.
A
The
reason
I
do
know
that
the
reason
that
Prometheus
does
the
scraping
and
does
to
go,
get
it
and
bring
it
in
is
that
it
has
the
least
amount
of
impact
on
your
existing
applications
right.
It's.
The
idea
is
not
to
put
the
load
there
to
put
the
load
somewhere
else.
Answer
your
question
yeah
another
one.
A
The
question
is:
what
is
the
the
messaging
format
for
the
hotel
protocol?
I'm,
assuming
to
be
really
honest,
I
haven't
dug
that
deep
to
check
all
the
different
things
you
can
do,
but
if
it's
like
Prometheus
and
all
the
other
ones,
I'm
assuming
it's
in
most
likely,
yaml
and
Json
for
sure,
if
not
others
I
think
the
The
Only
Exception
will
be
pushing
towards
the
back
end
with
exporters,
for
example,
if
you're
talking
to
something
like
a
proprietary,
whatever
that
has
their
protocol,
that's
been
implemented
there.
So
there
will
be
some
translation.
A
I
was
waiting
for
that
question.
So
that's
the
question
of
the
day.
How
do
you
mask
sensitive
data?
That's
exactly
what
those
processes
are
for.
So
one
of
the
things
you
do
definitely
want
to
optificate
is
you
know
any
kind
of
client
sensitive
stuff
before
you
export
it
across
regions?
You
know
you
don't
want
to
have
and
there's
countries
in
Europe,
for
example,
where
you're
not
allowed
to
export
data
across
the
border,
private
information
and
stuff
like
that
gpdr
stuff.
A
A
Across
spans,
how
do
I
correlate
data?
Oh
that's
a
good
one
and
a
very
long
answer
right
using
your
preprocessor
is
the
short
answer.
If
you
want
to
talk
to
me,
offline
I'm
happy
to
discuss
this
after
this
thing,
just
because
it's
a
more
deep
than
this,
you
know
in
a
sandbag
of
me.
Are
you
now?
That's
that's
a
valid
question,
but
I
mean
it
gets
a
very
technical
right.
So
we
can.
We
can
chat
about
that
in
a
bit.
Yeah
I
don't
mean
to
Waffle
them
off,
but
it's
that's
just
time
constraints
right!
A
Okay,
here
we
go
so
his
question
is
kind
of
valid
for
this
slide
right.
So
how
do
we
correlate
this
stuff
across
band
space?
There's
two
views
on
on
spans
and
traces
here.
The
one
with
the
tree
is
basically
saying
each
one
of
the
nodes
is
one
of
those
spans
one
of
those
individual
piles
of
information,
we're
collecting
and
when
I
look
at
a
complete
Trace.
That's
what
the
trace
looks
like
I
think
for
human
consumption.
The
the
time
period.
One
is
a
little
bit
more
easy
to
consume.
A
But
you
could
see
the
question
he's
asking
where
the
technology
is
going
to
get
a
little
bit
more
complicated.
We
want
to
correlate
across
these
things
right
and
a
really
short
answer,
for
you
would
be
you're
going
to
start
poking
at
the
spans,
more
so
than
you're
going
to
build
a
new
Trace
with
spans
right
across
traces,
there's
also
kubernetes
operator.
A
A
So,
as
you
see
in
this
picture,
I
can
spin
up
a
replicas
right,
just
they're
all
the
same
or
as
you
also
notice.
What's
inside,
there
is
an
auto
implementation
aspect,
which
means
language
specific
right,
so
one
of
these
might
be
Java,
one
might
be
go,
one
might
be
rust,
one
might
be
whatever
the
hell
you're
doing.
Python
I
don't
care,
but
they
all
contain
these
tools
inside
there
and
all
you're
doing
is
taking
whatever
code
you're
going
to
run
and
pointing
it
at
that.
A
Hey
put
this
inside
this
container
for
me
and
spin
it
up
on
one
of
my
things.
So
I
can
collect
half
a
terabyte
of
data
in
30
days
right,
that's
the
same
idea,
basically
Limitless
when
something
happens
to
you,
for
example,
when
you're
doing
the
the
doordash
explosion
during
the
pandemic,
you're
going
to
see
this
endless
pile
of
containers,
spinning
up
operators
just
makes
it
really
easy
to
use
predefined
stuff.
So
your
developers
don't
have
to
do
a
lot
of
leg
work
themselves
right.
A
A
If
you
want
to
get
that
far
down
the
road,
but
what's
a
little
bit
more
interesting
is
is
engaging
within
the
cncf
or
inside
individual
projects.
Right.
What
you
see
here
is
a
slack
Channel
they're,
pretty
big
on
the
on
the
slack
stuff.
All
I
did
was
search,
I
think
here
on
open
Telemetry.
If
you
can
read
that
and
there's
just
all
kinds
of
groups
and
stuff
going
on
so
the
minute,
they
start
digging
into
something
new
and
they
have
a
sub
project
or
whatever
going
on
or
a
specification
they're
working
on.
A
You
can
find
a
group
about
it.
They
have
calendars
where
they
list
meetings
you
can
attend.
You
can
see
recordings
of
those
meetings.
If
you
can
attend,
it
happens
to
me
a
lot
because
I'm
in
the
wrong
time
zones
very
interesting
to
see.
What's
going
on,
you
have
very,
very
simple
stuff,
like
the
hallway,
where
people
just
talk
about
all
kinds
of
stuff.
They're
doing
you
have
a
social
media
channel
where
people
post
what
they're
posting
that's
relevant
to
this
specific
project.
A
These
These
are
really
easy
ways
to
start
having
chats
and
you
can
even
learn
without
saying
anything
even
have
channels
where
they
drop
in
like
I'm
having
a
problem
with
this
piece
of
code.
Why
doesn't
it
work?
I'm?
Not
you
know
who
doesn't
love
the
guy
that
goes
on
slack
on
the
channel
and
dumps
a
you
know:
exception.
Trace
I
mean
you
can
live
and
learn.
You
know
it's
it's
fun
to
watch
another
big
one
is
kubecon
cloud
native
con.
That's
like
the
premiere
event
that
they
do
for
cncf.
A
They
do
one
every
year
in
the
North
America
and
one
every
year
in
Europe.
If
you're
coming
to
the
Europe
one,
it's
in
my
my
home
country
in
the
Netherlands,
it's
going
to
be
in
Amsterdam
you're.
All
welcome
to
come.
I'll
give
you
a
tour
of
the
city.
I
spent
nine
years
living
there
and
I
went
to
college
there.
So
even
did
security
there
for
a
while,
so
I
know
some
places.
A
You've
probably
never
been
happy
to
show
you
around
just
hit
me
up
they're
a
lot
of
fun:
they're,
big
they're,
they're,
quite
big
they're,
also
very
developer
orientated
a
lot
of
in-depth
stuff
around
the
open
source,
like
what
you're
feeling
here
like
what
you're,
seeing
with
all
things
open,
pretty
comparable.
We
have
a
front
row
here.
We
have
a
cncf
Ambassador,
so
she
can
tell
you
all
kinds
of
stuff
they
haven't.
Let
me
in
that
club.
Yet
maybe
I'll
stop
talking
about
this
stuff.
A
If
they
don't,
you
have
special
interest
groups
as
an
example,
one
of
the
ones
that's
kind
of
interesting
that
I'm
following
right.
There
is
called
tag
observability,
it's
in
a
technically
a
technical,
Advisory,
Group
I,
believe
they
call
it
it's
just
a
bunch
of
people
getting
together
and
say
Hey
what,
if
we
defined
a
standard
protocol
around
all
this
metric
stuff
and
not
just
specific
to
metrics,
but
also
trans,
you
know:
how
can
we
do
something
specific,
not
Hotel
protocol?
A
Not
you
know,
Prometheus
stuff,
not
you
know
just
one
and
it's
kind
of
fun
to
get
in
at
the
beginning
and
watch
something
like
that
evolve
and
and
the
documents
grow
and
the
ideas
go
back
and
forth.
So
maybe
you
find
something
interesting
like
that,
and
then
you
have
the
ability
to
get
into
some
of
the
end
user
communities.
Some
of
this
is
sponsorship
based.
Some
of
this
is
just
people
that
are
that
are
hosting
events.
A
So
it's
it's
well
worth
kind
of
poking
around
and
seeing
you
know,
maybe
somebody
you
know
works
at
one
of
these
places
or
somebody
you
you
know
actually
sets
up
one
of
these
events.
It's
pretty
surprising
how
big
it
is.
I,
don't
know
if
you've
ever
seen
the
cncf
like
complete
overview
picture
people
like
to
put
that
up
there
and
call
it
the
open
source
overview.
It's
got
like
a
gazillion
things
about
this
small
on
it
right,
it's
so
much
stuff.
So
this
is.
A
It
was
hard
just
to
do
this
to
get
it
viewable
for
you,
and
this
is
the
the
community
of
sponsors
and
stuff.
It's
it's
a
pretty
pretty
big
organization.
They
have
a
lot
of
structure
in
place
of
the
stuff
you
heard
in
the
keynote
is
definitely
in
full
force.
Here
you
know,
contributor
guides
are
very,
very
structured
in
how
they
do
stuff,
so
you
should
be
able
to
find
a
path
to
help
out
and
to
get
engaged.