►
From YouTube: Log Support in OpenTelemetry - Steve Flanders, Splunk
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Log Support in OpenTelemetry - Steve Flanders, Splunk
OpenTelemetry’s charter is to provide an observability framework for cloud-native software. Its goal is to provide full support for traces, metrics, and logs and provide a single implementation that can be leveraged. Traces have already reached a stable maturity level and metrics are well on their way. While logs were outside the GA scope of the project, they are an important data source and initial work has already kicked off to support them. - An initial data model has been accepted by the project - An initial implementation exists in the Java client library - Initial support has been added to the collector End users are interested in the plans for logs in OpenTelemetry. Will logging client libraries be introduced? Will native logging capabilities be added to the collector? How do context and resources work with logs? This session will dive into these topics to explain the current state and future direction. A demo of the current capabilities will be provided.
A
A
That
means
the
goal
is
to
provide
a
set
of
apis
libraries
agents
and
collector
technology.
That's
bundled
automatically
and
available
so
that
you
can
easily
generate
emit
and
collect
the
telemetry
data
that
you
need
in
order
to
observe
your
systems.
Open
telemetry
is
a
very
active
project.
In
fact,
it's
the
second
most
active
project
in
cncf
behind
only
kubernetes,
and
this
is
according
to
cncf
dev
sets.
A
You
can
learn
all
about
it
and
the
links
provided
below
telemetry
is
really
focused
today
around
three
primary
signals:
traces,
metrics
and
logs
and
each
are
in
a
different
phase
of
maturity.
Traces
recently
reached
the
stable
milestone
with
three
major
languages,
providing
support
two
more
right
on
the
way
and
the
rest
planned
later
this
year.
A
The
goal
is
that
traces
and
metrics
will
be
stable
across
all
major
languages
later
this
year
and
logs
will
at
least
reach
the
beta
milestone
hi.
My
name
is
steve.
Flanders,
I'm
a
director
of
engineering
at
splunk
and
actively
involved
in
the
open
telemetry
project,
including
the
collector
and
website,
I'm
also
a
member
of
the
cncf
sig
observability.
A
I've
been
working
in
the
open,
telemetry
space
since
its
inception
and
previously
on
the
open
census
project.
I've
also
been
in
the
observability
space.
Now
for
over
a
decade
prior
to
splunk,
I
was
at
an
omniscient
that
was
acquired
by
splunk
about
a
year
and
a
half
ago,
working
in
the
distributed
tracing
space
and
prior
to
that,
I
was
working
in
the
logging
space
at
vmware.
A
Open
telemetry
is
about
cloud
native
telemetry
if
you've
heard
of
observability
you've,
probably
heard
of
the
three
pillars
of
observability,
which
are
represented
as
verticals
here:
traces,
metrics
and
logs
these
signals
or
data
sources
are
full
of
rich
information,
but
the
way
in
which
they're
implemented
is
actually
many
layers.
Deep
for
each
of
these,
you
need
some
sort
of
instrumentation
api
that
is
used
to
actually
generate
and
emit
the
telemetry
data
that
you
care
about.
You
need
an
implementation
of
this
api,
typically
known
as
instrumentation
libraries
or
client
libraries.
A
All
of
this
is
what
the
open
telemetry
project
is
hoping
to
standardize
and
allow
for
broad
adoption
and
popular
open
source
libraries
and
frameworks.
The
focus
has
been
primarily
on
traces,
metrics,
very
close
behind
and
logs
are
being
planned
at
this
time
where
open
symmetry
draws.
The
line
is
that
it
doesn't
actually
provide
a
back
end.
Instead,
it
is
a
open
source
and
vendor
agnostic
technology,
so
it
plugs
into
a
variety
of
different
backends.
Thus,
the
end
user
can
decide
where
they
want
to
send
their
data,
giving
them
full
control
of
it.
A
A
A
The
collector
is
a
single
binary
that
can
be
deployed
in
a
variety
of
different
form
factors,
including
as
an
agent
or
as
a
gateway.
It
serves
to
receive
process
and
export
data
in
a
vendor
agnostic
way.
Let's
drill
into
each
of
these
components,
the
specification
is
organized
into
what
open
telemetry
calls
signals.
These
are
things
like
traces,
metrics
and
logs.
There
are
other
signals
too,
but
for
the
purposes
of
this
talk,
we'll
just
focus
on
the
different
data
sources
themselves.
A
A
There
are
core
packages
which
are
part
of
the
open
source
community,
with
popular
things
being
provided
out
of
the
box
like
jaeger,
zipkin
and
prometheus,
and
then
there
are
contrib
packages
which
may
not
be
as
applicable
to
all
end
users,
but
do
also
provide
support.
This
is
where
things
like
vendor
exporters
can
also
exist.
A
Resources
is
another
foundational
piece
of
open
telemetry,
with
the
idea
being
that
these
different
signals
come
from
unique
objects
in
your
environment,
but
being
able
to
actually
understand
what
that
is
like
what
pod.
Maybe
what
container,
what
cloud
provider
these
things
run
in?
That's
where
resources
come
in
the
idea
is
regardless
of
the
signal.
A
A
As
you
look
at
these
fields,
some
of
them
may
be
pretty
common
to
you
like
if
you're
familiar
with
logging
or
syslog,
for
example,
things
like
a
timestamp
pretty
common
severity
is
pretty
well
known,
whether
it's
like
an
info
warning
or
error
message,
and
then
body.
Sometimes
it's
referred
to
as
message
in
the
case
of
syslog.
These
are
pretty
common
fields
that
you
might
see,
or
at
least
understand
the
concepts
in
open
telemetry.
We
can
reset
represent
this
in
a
structured,
json
payload
like
the
following.
A
In
this
example,
we
have
a
timestamp
and
then
a
body
of
a
message
with
nested
key
value
pairs.
We
could
even
extend
this
slightly
by
adding
severity
information.
Again,
each
of
these
fields
are
optional,
so
what
you
specify
is
up
to
the
implementation,
that's
generating
this
particular
message
beyond
the
standard
fields
that
you
may
recognize.
There
are
also
some
additional
fields,
many
of
which
are
related
to
open
symmetry
in
some
way.
For
example,
you
have
resources
and
attributes.
These
are
core
concepts
in
open
telemetry.
I
already
talked
a
little
bit
about
resources.
A
Attributes
are
very
similar.
You
can
think
of
them
also
as
key
value
pairs,
but
instead
of
being
applied
generically
to
an
object.
That's
the
limit,
that's
generating
the
signal.
You
would
instead
add
attributes
to
specific
signal
events,
so
each
individual
log
message,
for
example,
could
have
different
attributes
attached
to
it.
A
A
A
A
Here's
an
example
of
a
resource
where
we
have
this
service
name
and
version
of
a
particular
resource,
but
also
the
kubernetes
pod
uuid.
This
could
have
additional
metadata
as
well,
for
example,
maybe
the
cloud
provider.
If
this
was
deployed
into
a
public
cloud,
and
then
we
have
attributes
the
key
value
pairs
that
are
specific
to
this
particular
signal
message
that
is
being
sent
here
is
where
you
could
define
things
like
semantic
conventions.
For
example,
you
can
see
the
http
status
code
key
and
a
value
of
500..
A
One
final
thing
to
note
about
this:
payload
is
you
may
notice
that
it
has
a
timestamp
and
it
has
a
severity
field,
but
the
body
also
contains
the
timestamp
and
the
severity
now
this
could
be
parsed
out
and
added
as
metadata
to
the
structure,
json
payload
or
it
could
be
added
supplementally
being
passed
from
one
collector
to
another.
So
there's
no
requirement
that
this
data
actually
match.
What's
in
the
body,
you
don't
even
have
to
parse
that
data.
If
you
do
not
want
to.
A
A
As
a
result,
it
needs
to
have
the
ability
to
convert
from
and
to
different
formats,
with
the
log
data
model
and
its
current
incarnation.
It
supports
a
variety
of
different
open
standards,
as
well
as
vendor
standards.
For
example,
you
may
be
familiar
with
the
apache
http
server
log
model
or
the
elastic
common
schema.
A
A
So,
in
summary,
an
open,
telemetry
log
record
is
what
you
might
refer
to
as
a
log
or
an
event.
An
important
thing
to
note
here
is
that
open
symmetry
does
not
actually
distinguish
between
logs
and
events.
Today,
though,
it
could
in
the
future.
Now
this
log
record
is
really
just
a
structured,
json
payload.
A
There
is
a
notion
of
error,
semantics
that
are
that's
provided,
so
you
can
denote
errors
within
your
structured
log
messages
and
you
can
convert
from
and
to
a
variety
of
popular
formats.
Now
this
log
message,
that's
being
generated.
Log
record
can
be
actually
be
embedded
in
spans,
for
example,
there's
a
notion
of
span
events
for
which
the
log
record
could
actually
be
attached,
or
it
could
be
standalone,
which
is
a
little
bit
more
traditional,
especially
in
non-tracing
environments.
A
Next,
let's
talk
a
little
bit
about
instrumentation
libraries.
As
I
mentioned,
this
is
a
language
agnostic
way
to
instrument
your
application,
so
there's
an
instrumentation
library
per
language
provided
within
that
instrumentation
library
there's
an
api
that
is
used
to
actually
generate
the
telemetry
data
that
you
care
about
an
sdk
that
is
used
to
process
and
export
that
data
and
then
all
the
other
core
specification
concepts
that
I
discussed
earlier,
including
resources,
semantic
conventions
and
the
like
are
all
also
supported
in
the
instrumentation
libraries.
A
So
how
does
logging
play
in
logging
is
still
quite
early
when
it
comes
to
instrumentation
libraries
in
open
telemetry,
but
there's
some
amount
of
reference
architecture
or
reference
implementation
provided
with
java.
Today,
first,
you
can
take
a
look
at
how
you
could
potentially
manually
instrument
your
app
for
logging
using
the
sdk
extension
provided
in
the
open,
telemetry
java
project.
A
If
you're
looking
for
automatic
instrumentation,
then
the
java
instrumentation
repository
offers
a
logger
mdc
that
logger
mdc
offers
the
ability
of
entering
in
things
like
the
trace
id
span
id
and
trace
flags.
You
may
remember:
these
were
fields
that
were
part
of
the
trace
context
defined
in
the
log
data
model.
A
A
It's
still
being
considered
whether
or
not
manual
log,
instrumentation
and
instrumentation
libraries
for
logs
will
be
provided
in
open
telemetry,
but
first,
a
stable
data
model
needs
to
be
agreed
upon,
because
the
instrumentation
libraries
need
to
depend
on
that
data
model
and
until
the
data
model
is
actually
stable,
taking
a
dependency
at
least
on
the
manual
log.
Instrumentation
is
not
advised
at
this
time,
though,
using
automatic
trace
ingestion
injection
is
perfectly
fine,
because
changing
those
conventions
is
pretty
easy
going
forward.
A
Now
you
might
be
wondering
why
logs
are
not
ready
in
open
symmetry,
yet
you
might
recall
earlier
that
there
are
different
signals.
Traces,
metrics
and
logs
to
date,
traces
and
metrics
have
been
the
primary
focus,
but
logs
is
quickly
starting
to
pick
up
momentum
in
the
open
telemetry
project,
especially
with
the
announcement
of
the
stable
release
for
traces.
A
Of
course,
prs
are
welcome,
so,
if
you're
interested
in
getting
involved,
please
do
now.
Let's
talk
about
the
collector,
the
collector
is
a
component
that's
configured
via
pipelines.
Each
of
these
pipelines
is
made
up
of
one
or
more
receivers
a
way
to
get
data
into
the
collector.
These
can
be
push
or
pull-based
processors.
What
you
want
to
do
with
the
data
as
it's
passing
through
the
collector
and
then
exporters
again,
these
can
be
push
or
pull
base
in
order
to
send
the
data
to
the
destination
or
destinations
of
your
choice.
A
A
The
collector
can
also
do
things
like
crud
operations
on
metadata.
This
is
powerful
for
things
like
pii
reduction
or
maybe
even
enhancing
the
telemetry
data
that
is
passing
through
the
collector
and
again.
The
core
concepts,
like
you
saw
in
the
specification
like
resources,
are
natively
supported
in
the
collector
at
a
very
high
level.
Here's
a
reference
architecture
of
how
open
telemetry
could
be
deployed.
A
It's
pretty
common
to
deploy
at
least
the
open,
telemetry
collector
running
as
an
agent
on
each
of
the
hosts
within
an
environment,
and
then
the
open,
telemetry,
instrumentation
library.
These
are
language
specific
on
the
associate
applications
in
this
configuration,
the
instrumentation
library
out
of
the
box,
is
configured
to
send
data
to
the
open,
telemetry
collector
running
as
an
agent,
and
that
agent
can
be
configured
via
yaml
to
send
data
to
the
backend
or
back
ends
of
your
choice.
A
A
If
we
drill
a
little
bit
into
the
architecture
of
the
open
symmetry
collector
itself,
you
might
remember
the
concepts
of
receivers,
processors
and
exporters.
Those
are
pretty
foundational
to
the
open,
telemetry
collector.
Now
you
actually
stitch
these
different
components
together
through
what
are
called
pipelines
here,
they're
represented
in
different
colors,
for
example.
You
can
see
an
orange
pipeline
defined
where
otop
otop
is
actually
the
protocol
that
open
telemetry
supports
out
of
the
box,
so
open
symmetry
is
being
received
in
so
maybe
this
is
coming
in
from
an
instrumentation
library.
A
For
example,
it's
going
through
a
batch
processor,
an
attributes,
processor
and
it's
being
exported
in
jager.
As
you
may
remember,
the
collector
can
receive
in
one
format
but
export
in
another,
making
it
a
vendor
agnostic
solution.
The
green
line
represents
a
second
pipeline
here
again.
Otop
data
is
being
received
this
time,
it's
going
through
a
batch
and
a
filter
processor
and
it's
exporting
to
two
different
destinations,
otlp
and
prometheus
and
parallel.
A
As
we
look
at
log
collection
in
open
telemetry,
this
is
probably
the
most
mature
aspect
today.
The
open,
telemetry
collector
supports
a
variety
of
different
components
that
are
logging
specific,
for
example,
from
a
receiving
perspective.
There
is
a
file
log
receiver
which
supports
tail
logging.
So,
for
example,
you
can
point
it
at
a
log
file
and
it's
able
to
pick
up
changes
that
are
written
to
that
log
file.
A
I
would
be
remiss
if
I
didn't
point
out
that
recently
there
was
a
donation
by
observe
iq
of
the
stanza
logging
agent
to
open
telemetry.
You
can
read
more
about
it
on
the
github
issue
linked
above
now,
stanza
is
being
fully
integrated
into
the
open
symmetry
collector
and
things
like
the
file
log
receiver,
which
support
tailing
capabilities.
A
That
was
part
of
stanza,
but
it's
one
of
many
parts
of
stanza.
For
example,
there's
journal
d,
syslog,
tcp,
udp
windows,
event,
log
support,
filtering
parsing
and
more
all
of
these
capabilities
are
natively,
being
added
to
the
open,
telemetry
collector,
so
it'll
have
more
logging
support
in
the
future.
In
addition,
there
are
a
variety
of
additional
destinations
which
will
be
supporting
logs
in
the
future.
A
A
I
will
deploy
fluent
bit,
which
will
be
configured
to
collect
docker
events
and
send
it
to
the
open,
telemetry,
collector
and
I'll.
Have
the
open,
telemetry
collector
send
that
data
to
two
different
destinations,
one
being
a
logging
backend
and
the
other
one
being
to
standard
out?
The
collector
will
also
be
configured
to
add,
attribute
and
resource
information,
basically
enriching
the
fluent
bit
docker
events
that
are
being
passed
through
the
open,
telemetry
collector.
A
Let's
go
ahead
and
try
this
out
so
first
I
will
start
a
docker
container
that
is
running
fluent
bit.
So
this
is
just
stock
fluent
bits.
The
one
of
the
latest
versions
it's
configured
with
an
input
of
the
docker
events,
input
module
and
for
an
output
perspective.
It
will
send
data
to
a
port
8006
on
a
different
docker
container.
A
So
we'll
go
ahead
and
start
that
container
now
I
haven't
actually
started
the
open
symmetry
collector.
Yet
so,
as
this
starts
analyzing
data,
it
will
not
be
able
to
send
the
data
and
it
will
start
throwing
errors
while
that
gets
fired
up
on
the
other
end.
Here
I
have
an
open,
telemetry,
collector
yaml
file,
so
in
this
yml
file
we
will
enable
the
fluent
forward
receiver.
So
that's
the
port
8006
that
you
saw
with
fluent
bit
so
we'll
be
receiving
that
data
in
natively
we'll
go
ahead
and
configure
an
attributes
processor.
A
So
we're
just
going
to
insert
a
single
key
value
pair
here
that
key
is
called
foo
with
a
value
of
bar.
We
will
also
configure
resource
detection,
so
we
can
add
a
system
information
to
the
data
that's
passing
through
and
we're
going
to
send
that
data
out
to
logging
locally,
as
well
as
a
splunk
heck
exporter.
A
I
was
hoping
to
show
off
elasticsearch,
but
unfortunately,
that
exporter
does
not
fully
support
logs,
yet
it'll
be
added
here
soon
and
then
it'll
be
easy
enough
to
demo.
Here,
then
we
just
define
a
pipeline
that
says,
receive
fluid
forward
in
go
ahead
and
add
attributes
and
resource
detection
detection
processors
and
then
export
that
data
out
here
to
two
destinations.
In
parallel
with
that,
I
will
go
ahead
and
start
the
container.
This
is
the
container
of
the
open,
telemetry
contrib
repository
and
it's
using
the
yaml
file
that
I
just
built
and
exposing
port
8006.
A
A
So,
for
example,
we
can
see
there's
a
log
event
coming
in
where
resource
information
is
being
added,
the
host
name,
here's
my
hostname
and
then
the
os
type,
which
is
linux.
Then
we
get
a
timestamp
for
that
event.
The
severity
is
not
set,
nor
is
the
short
name,
the
body
this
body
is
coming
from
fluent
bit
itself
and
then
beyond
that
there
are
attributes
like,
for
example,
the
fluent
tag
is
being
set,
so
you
can
see
it's
collecting,
docker
events
and
you
may
recall
the
foo
bar
attribute
in
the
open
telemetry
collector.
A
Now
that
data
is
being
passed
into
a
different
destination
as
well
being
the
splunk
heck
destination
and
if
we
go
ahead
and
look
at
the
data
coming
in,
we
can
see.
For
example,
here
is
a
payload
that
payload
has,
for
example,
metadata,
including
the
foo
bar
attributes
that
we
saw
earlier.
You
can
see
the
fluent
tag,
the
docker
events
and
a
bunch
of
other
rich
metadata.
A
That's
extracted
from
that
payload
again,
this
same
behavior
would
be
applicable,
no
matter
which
destination
you
send
the
data
to,
and
it's
very
easy
to
change
the
configuration
of
the
collector
just
by
editing
the
yaml
file,
so
very
quick
and
easy
to
get
started.
This
supports
popular
open
source
things,
including
fluent
bit,
and
fluentd
native
log
support
is
being
added
to
the
collector
and
from
an
instrumentation
perspective.
There
are
options
available
today.
A
With
that
I'd
say:
please
do
check
out
our
special
interest
groups
there's
a
variety
of
these
that
are
defined.
Some
are
language
specific
for
instrumentation.
Some
are
specification
specific.
The
collector
has
a
special
interest
group
as
well.
It's
a
great
way
to
get
involved
in
the
open,
telemetry
community
definitely
join
the
conversation.
A
Each
of
these
special
interest
groups
has
their
own
github
project
and
github
discussions
is
leveraged.
In
addition,
the
open
telemetry
group
is
on
the
cncf
slack,
which
is
cloud
hyphen
native.slack.com
and,
of
course
pr
is
welcome
and
everyone
is
welcome
to
join
the
community.
So
take
a
look
at
the
github
issues.
Many
are
labeled
with
either
good
first
issue
or
help
wanted
labels,
which
makes
it
pretty
easy
to
get
started
or,
of
course
you
can
join
a
special
interest
group
and
ask
how
you
can
get
involved
and
with
that.
Thank
you.
A
So
much
as
you
can
see,
open
telemetry
is
moving
quickly
to
provide
cloud
native
observability.
The
goal
is
to
really
help
standardize
and
make
it
really
easy
to
get
instrumentation,
regardless
of
the
different
signals
and
collect
and
process
that
data
in
an
efficient
vendor
agnostic
way.
Today
there
is
rich
support
for
traces
that
is
already
stable
in
languages,
including
java
and
python.net
other
languages,
including
javascript,
and
go,
are
planning
to
be
stable
very
soon.
Here.
A
With
addition,
a
decision
around
what
will
happen
from
an
instrumentation
library
perspective
at
a
minimum,
you
can
expect
to
see
more
trace
injection
into
logs
across
languages,
and
the
collector
will
offer
native
support
for
logging.
In
fact,
it
already
offers
some
pretty
broad
support
for
it
today
and
has
already
been
tested
at
scale
for
both
tracing
and
metrics.