►
From YouTube: Why Do My Logs Suck?! A Practical Guide to Improving Your Logs- Patrick Stephens, Couchbase
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Why Do My Logs Suck?! A Practical Guide to Improving Your Logs- Patrick Stephens, Couchbase
For the cloud-native transformation of Couchbase, we had to do something about our logs. We had various logs scattered around from different language binaries (Java, Erlang, C, etc.) and in different formats with none fit for “human consumption”.
If you’ve ever started tearing your hair out over differing log formats then this is the talk for you! If you have ever tried to understand what a developer was thinking when adding a log over multiple lines then this is also the talk for you! We include some specific examples of difficult logs and how we solved them with no change to the log itself.
Pragmatism was essential:
We did not have the resources to refactor every log statement across all these teams/codebases.
We needed to support an approach that allowed us to provide an out-of-the box solution but also easy integration with anything already deployed.
We had a desire to focus on CNCF or at least open source solutions that are loosely coupled to lower the limits for adoption.
A blog post covering more in-depth examples to support this presentation can be found here.
A
Hi
there
I'm
patrick
stevens,
I'm
a
software
engineer
at
couchbase,
and
I
started
back
in
february
with
a
mandate
to
try
and
improve
the
observability
of
couch
base.
Prior
to
that,
I
was
working
in
primarily
a
defense
domain
for
almost
two
decades,
so
getting
logs
and
and
monitoring
there
is,
is
quite
important
and
can
be
quite
difficult.
Sometimes
I'm
going
to
cover
some
of
the
issues
we
came
across.
I've
got
quite
a
longer
blog
post
that
covers
them
in
much
more
detail
with
nice
work.
A
Examples
which
I'll
pop
the
links
up
to
I'm
just
going
to
kind
of
give
you
a
little
bit
of
a
flavor
of
what
what
we've
been
doing,
and
maybe
some
some
simple
examples
to
to
lead
you
in
so
going
to
go
through
a
quick
summary
of
what
we
what
problems
we
had.
You
know
what
what
kind
of
stuff
we
had
to
solve,
what
we
did
and
then
a
little
overview
of
like
some
some
tips
and
tricks
I've
got.
So
how
could
bad?
Could
it
be?
A
I'm
not
going
to
go
over
it
in
a
huge
amount
of
detail,
but
essentially
that
the
main
two
things
we
need
to
cover
are
we
don't
want
to
change
any
of
our
logs
and
there's
loads
of
stuff
already
out
there.
We
can't
change
it.
We've
got
loads
of
support
stuff,
we
can't
change,
we
have
to
be
pragmatic.
We
don't
have
the
resources
to
refactor
all
our
log
statements
to
standardize
them
and
make
it
all.
You
know
wonderful
and
we've
got
all
this
existing
tooling.
A
A
Rather
than
having
your
own
bespoke
solution,
which
you
don't
have
to
completely
bulge
into
a
completely
different
thing,
just
on
the
couch-based
side,
so
the
container
that
runs
is
is
a
single
container,
but
it's
multi-processed
that
writes
logs
into
different
files
in
different
formats
and
none
of
these
logs
get
output
by
default
to
standard
output
and
there's
additional
tooling.
That
goes
in
and
extracts
the
logs,
and
this
also
has
to
run
on
premise
and
in
kubernetes,
which
is
one
of
the
other
reasons
as
well.
We
went
with
fluent
bit
starting
from
a
nice
embedded
background.
A
You
know
financial
or
healthcare,
and
we
also
need
to
support
different
endpoints,
so
something
that
can
send
to
s3
to
cloud
analytics
to
data
dog,
whatever
you
want
without
this
without
us
having
to
write
that
code,
and
so
if
we
can
target
something
that
that
then
lets
us
say,
oh
to
a
customer,
if
you
want
to
use
it
to
send
it
to
us
bespoke
endpoint,
then
this
is
how
you
do
it
and
you
can
configure
it
as
co.
As
as
just
a
configuration
option,
so
I'm
going
to
cover
what
our
logs
look
like.
A
It
all
comes
from
the
open
source
repo
which
is
linked
there.
I
use
these
examples
from
for
continuous
integration,
so
this
is
the
repo
just
showing
just
a
quick
quick
overview
of
it.
There's
a
there's,
a
test
test
directory
with
all
the
log
stuff
in,
and
you
can
see,
there's
kind
of
expected
known
input
and
expected
output
there.
So
this
is
our
audit
log
or
an
example
of
some
of
the
messages.
It's
just
a
simple
json
format,
so
one
of
the
first
ones
we
did
because
it
worked
well
and
here's.
A
The
kind
of
this
is
the
expected
output.
So
during
continuous
integration
I
can
run
this
and
I
can
just
diff
against
it
and
it
makes
a
very
simple,
quick
check.
We've
got
some
more
logs.
This
is
a
java
one,
but,
as
you
can
see,
someone
decided
four
characters
was
enough
for
the
the
log
level.
We've
got
some
standard
java
stack
traces,
not
too
bad,
and
then
you've
got
a
humongous
java
thread
dump.
So
this
is
one
log
statement
and
that's
what
they
want.
A
This
is
one
of
the
problematic
ones.
This
is
eventing,
so
it's
got
multiple
formats
in
the
same
file.
You
can
kind
of
see
there
there's
different
time
stamps
the
log
levels
different
cases.
One
of
the
lines
doesn't
have
a
log
level
and
there's
some
you
know
multi-line
stuff
as
well.
So
those
three
lines
there
cover
some
of
the
problems
we
have
to
solve
in
a
single
kind
of
log,
multi-format
kind
of
approach
and
in
fact
the
timestamps.
In
the
end
I
just
ignored
and
just
used.
A
The
time
of
parsing
is
a
similar
log,
again
timestamps
different
format,
log
levels
or
all
in
uppercase
in
this
one,
but
that
one
doesn't
have
a
level
and
you
can
see
it's
kind
of
an
example.
There's
some
bigger,
bigger
multi
line
here
as
well,
but
it's
very
similar
to
the
previous
log
and
actually
those
two
are
parsed
together,
while
using
the
same
the
same
regex,
they
actually
go
to
different
streams.
A
Here's
one
of
our
large
erlang
codebase
logs.
So
this
is
where
you
generally
get
a
lot
of
multi-line,
but
they
one
good
thing
here
is:
there's
a
nice
good
regex
for
the
for
the
start
of
each
line.
But,
as
you
can
see
the
multi
lines,
I
mean
this
is
quite
a
small
example,
but
they
can
cover
quite
a
few
lines
and
have
embedded
data
and
stuff
like
that.
A
So
I
had
some
problems
with
some
of
these
erlang
ones,
mostly
to
do
with
the
regex
side
of
things,
and
I
think,
on
on
the
next
one
I
show,
which
is
another
erlang
one,
but
with
more
more
multi-line
in
it.
This
one
had
a
problem.
The
the
first
line
parser
couldn't
handle
the
new.
You
know
the
close
of
of
the
the
closing
bracket
and
then
a
new
line
as
the
part
of
the
first
line
parser.
So
I
had
to
tweak
it
slightly.
A
So
there
were
little
problems
like
that
which,
which
I
came
across
during
during
the
during
the
work.
I
was
doing.
The
other
thing
we
have
is
rebalance
reports,
and
these
are
big,
big
json.
You
can
see
there.
You
know
447
kilobytes,
but
it's
all
one
line
and
one
big
problem
with
it
was
there's
no
new
line
at
the
end
of
it,
and
the
tail
plug-in
doesn't
flush
until
it
hits
a
new
line.
A
A
A
I
wanted
to
test
them
and
make
sure
that
as
we
iterated
it,
we
we
we
weren't
breaking
any
previous
stuff,
and
one
of
my
key
points
here
is:
if
you
can
avoid
writing
brittle
regexes,
it's
much
easier
to
standardize.
Your
logging
on
the
logging
log
producing
side
than
it
is
to
write
some
horrible
regex
that
will
probably
break
next
time.
A
Someone
changes
the
logging
and
I
link
out
to
my
blog
post
there,
but
if
you
make
sure
your
egg
x
doesn't
have
the
log
the
default
log
key,
which
is
what
the
tail
plug-in
uses
by
default.
If
it
can't
parse
things,
you
can
easily
tell
if
your
regex
is
failing
to
pass
fluent
bit,
doesn't
really
show
you
if
the
regex
fails
to
pass
without
starting
to
look
at
metrics
and
stuff
like
that.
But
if
you
so
for
my
reg
x's,
I
never
have
the
log
key
in
it.
A
So
if
I
see
that
in
the
output
I
know
the
regex
has
failed
and
yeah.
So
I
started
with
the
audit
log.
You
saw
that
that
was
a
simple
json
one,
so
no
reg
regex's,
so
good
good
first
starting
point.
It
was
also
what
some
of
the
security
customers
wanted.
They
wanted
to
afford
it
all
their
audit
logs
and
stuff
to
to
different
endpoints
for
their
security
accreditors.
It
kind
of
put
the
whole
framework
in
place,
and
then
I
could
iterate
it.
I
added
a
load
of
regression
testing.
A
As
you
add
more
logs,
you
find
more
problems,
so
there's
corner
cases
in
all
those
logs
they're,
not
real
logs
they're
kind
of
examples
of
the
worst
bits
of
each
log,
and
so
we
can
iterate
it
and
also,
as
we
upgrade
through
a
bit.
We
want
to
make
sure
we're
still
parsing
stuff,
because
there
were
a
few
issues
with
parsing
discovered
from
regression:
testing
user
regex
testers,
there's
a
lot
of
them.
Calipture
have
just
added
one
which
is
really
good,
but
I
it
wasn't
available
when
I
was
doing
this.
A
Rubler
is
the
one
that's
recommended,
but
again
they
they
work
well
to
validate
regex,
but
then
transferring
that
and
translating
that
into
the
bit
configuration
sometimes
there's
extra
escaping
and
sometimes
stuff
just
doesn't
work
and
also
take
examples
of
those
logs
and
do
them
locally.
Do
everything
locally,
don't
run
it
in
a
massive
kubernetes
stack,
so
we
added
a
watcher
process
as
well.
A
So
I
I
reuse
the
cube
sphere
approach,
which
is
now
the
flight
operator
approach
which
just
has
a
simple
watcher
process
in
there
that
when
the
configuration
changes,
it
restarts
fluid
bit,
and
that
was
just
really
quite
simple,
straightforward
and
brought
other
benefits
as
well
later
on,
where
we
can
start
adding
in
extra
metadata
as
variables,
and
I
could
do
that
rebalance
report
processing.
I
touched
on
before
and
yeah
one
other
key
aspect.
We
need
to
make
sure
for
our
deployment.
A
Is
we
never
restart
the
pods
unless
we
absolutely
have
to,
because
obviously
the
databases
don't
really
like
that?
So
if
we're
saying
oh
you've
changed
the
log
configuration
just
to
do
some
minor,
you
know
update,
you
want
to
send
it
somewhere
else
change
credentials.
Now
you
need
to
wait
for
your
database
to
restart
that's
not
going
to
go
down
very
well.
So
again,
it's
all
about
localizing
those
changes
and
making
sure
it
all
works,
and
this
koozier
approach
was
was
pretty
pretty
straightforward.
A
You
know
I
took
took
their
code
extended
it
added
couch-based,
specifics
and
yeah.
It's
it's
working
pretty!
Well,
we've
got
it
deployed
now
for
for
about
six
months,
so
it's
yeah.
It's
all
going
pretty
well
right
and
then
a
general
overview.
So
what
else
did
we
do
so
all
the
logs
kind
of
they're
wildly
inconsistent,
but
they
follow
a
similar
pattern.
Every
log
has
a
timestamp
every
log.
Pretty
much
has
a
log
level.
A
You
know
info
warning
whatever
some
didn't
like
the
audit
level,
but
I
decided
they
were
a
specific
level
for
all
of
the
statements
and
then
message
is
just
everything:
that's
not
the
right,
not
those
those
other
bits,
so
keep
it
simple
generic.
So
we
can
parse
everything
with
it.
We
then
tag
by
file
names,
so
we
can
root
the
different
yeah
we
can
do
processing.
By
far
now,
we
can
do
routing
to
different
endpoints
by
filename,
which
is
which
was
quite
key
for
particularly
for
the
audit
log.
A
You
know
a
lot
of
times
people
I
just
want
to
see
the
logs,
but
actually
audit
logs
have
to
go
to
maybe
a
security
monitoring
tool
or
something
like
that
added
common
processing
as
well.
So
you
saw
the
levels
were
wildly
disparate
there.
You
know
you've
got
info
written
in
different
cases
and
stuff
like
that.
Two,
you
know
four
character:
debug
rather
debug,
so
I
added
processing
to
standardize
that
to
constrained
enums,
which
then
makes
it
work
really
well
things
like
grafana
and
stuff.
A
You
can
filter
on
one
value
rather
than
all
the
variants
of
it
same
with
filename,
just
showing
you
here.
You
know
the
filename's
really
long.
It
includes
the
whole
path
and
that's
a
bit
of
a
pain
when
you,
when
you're,
using
it
in
grafana,
and
so
I
added
extra
extra
stuff
to
to
chop
it
up
into
smaller
bits.
So
we
could
then
filter
on
that
and
added
some
extra
metadata.
So
our
pods
include
some
configuration
information.
A
A
So
what
went
well
community
support
was
a
great
one.
For
me,
I've
got
it
at
the
end,
but
a
lot
of
the
stuff
I've
shown
you
and
how
I
did
stuff
came
from
examples
from
other
people
fluent
bit
just
worked.
We
only
had
one
issue
I
found
doing
regression
testing,
while
upgrading
with
with
that
very
humongous
json
stuff
it.
It
was
failing
on
the
really
humongous
ones
after
pass,
but
that
was
fixed
very
quickly,
because
I
had
a
nice
regression
check
straight
away.
I've
contributed
all
these
parsers
back
as
well.
A
The
stuff
we
took
from
cubesphere,
which
is
in
the
fluent
operator,
now
great
way
to
handle
dynamic
configuration,
reload
and
and
a
lot
of
other
stuff
as
well
the
standardization
stuff
for
the
log
levels.
Someone
on
slack
pointed
me
at
it.
I
was
like
oh
wow,
it's
quite
a
long
set
of
filters,
but
they're
very
simple,
and
they
just
sort
of
apply
sequentially
and
another
one
counter
space
does
sort
of
reduction
in
of
its
logging
for,
like,
basically
it
hashes
some
of
the
strings
that
are
could
be
sensitive.
A
So
I
I
demonstrate
how
you
could
do
this
with.
Whilst
you
know
in
flight
using
lower
filters
and
lower
filters
seem
to
be
like.
Basically,
if
you
can't
do
anything
anywhere
any
other
way,
you
can
probably
do
it
with
la,
but
you
know
it
might
be
better
to
do
it.
A
simpler
way
to
start
with
yeah,
so
it
was
it
was
you
know
everything
just
worked:
it
went
pretty
well
pretty
straightforward,
it
was
quite
a
good
experience
and
actually
one
of
our
customers
requested
this
kind
of
capability.
A
Whilst
I
was
developing
it
and
they
were
already
using
fluent
bits,
so
you
know
it
just
fed
in
seamlessly.
While
we
were
doing
it
and
I'd
definitely
say,
contribute
and
sort
of
engage
with
the
community.
I've.
I've
had
a
really
good
time,
and
people
have
really
helped
me
out
and
and
stuff
like
that.
Things
like
the
levels
but
also
one
came
up
recently.
A
I've
not
not
added
yet
but
making
numeric
versions
of
the
levels
makes
it
much
easier
to
do,
alerting
and
and
querying
in
some
of
the
some
of
the
grafana
or
or
prometheus
or
whatever
other
stacks
you're
doing
what
what
went
wrong
or
what
to
do
when
stuff
goes
wrong.
I
do
touch
on
it
on
my
blog
post
down
the
bottom
there,
but
regexes
are
a
nightmare.
A
Know
you've
got
one
problem.
You
had
a
regex
to
solve
it
now.
You've
got
like
100
problems,
so
try
not
to
do
them.
There's
lots
of
tools
to
try
and
help
you
out,
but
none
of
them
are
perfect.
The
calypso
tool
is
probably
the
closest
because
that
kind
of
runs
the
same
regex
engine
under
under
the
hood.
I
think
it
runs
fluentd
as
a
as
a
container
to
do
it
simplify
stuff
people,
keep
I
see
in
the
community
channels,
people
try
and
oh
here's
my
output
from
kibana,
and
it's
not
working.
A
It's
like.
No.
No,
let's
start,
you
know
start
right
at
the
beginning
test
locally
start
with
known
input
to
standard
out,
see
what
it's
doing
there
before
you
start,
adding
in
the
whole
downstream
stack,
which
is
can
be
pretty
difficult
to
manage.
Is
there
a
problem
with
that?
Is
it
not
being
sent
there?
You
know
it
might
be
unrelated
to
fluent
bit
so
yeah
just
confirm
what
you
think
is
true.
A
First
at
you
know
the
earliest
point,
you
can
add
more
logging
output
to
standard
output,
don't
use
like
oh
I'm,
sending
somewhere
else
and
then
I'll
look
at
it
there.
Well
just
look
at
the
logs
straight
out
from
a
bit.
It
will
tell
you
things
like:
maybe
you've
got
the
wrong
path
and
you're.
You
know
your
environment
variable
or
something
like
that.
Your
wild
card
doesn't
work.
You
haven't
got
permissions.
A
These
are
all
fun
things
to
discover
that
you
won't
find
until
you
look
at
those
logs
and
to
be
honest,
I
don't
think
I
had
any
real
problems
with
fluent
bit.
It
was
mostly
what
I
was
doing
or
just
my
misunderstanding
of
configuration,
which
I
guess
you
know
I've
tried
to
improve
the
documentation
a
little
bit
as
well,
but
it's
those
kind
of
things
and,
generally
speaking,
it's
probably
working
it's
the
same
as
like,
never
suspect
a
compiler
bug
when
it's
probably
your
code,
that's
wrong!
It's
it's!
You
know
it's
well.
It
works.
A
Well,
usually,
there's
very
specific
problems,
sometimes
with
specific
releases,
but
generally
I've.
I've
not
had
a
problem,
stuff's
been
working,
and
then
you
know
it's
just
me
that
had
it
wrong,
bodges,
so
rebalance
reports.
This
is
a
massive
bodge,
hopefully
be
fixed
in
cash
based
server
at
some
point
in
future,
but
we
still
have
to
support
the
existing
ones.
Essentially,
it's
a
huge
amount
of
complexity
to
manage,
adding
a
new
line
to
the
file
we
mount.
A
The
logs
volume
read
only
to
make
sure
we
can't
change
anything
and-
and
you
know,
protect
it
from
that
side.
So
I
have
to
create
temporary
copy.
Add
a
new
line
manage
that
make
sure
I
don't
fill
the
temporary
copies
up
and
clean
them
up
when,
when
I
finish
with
them,
it's
just
a
huge
amount
of
complexity.
A
A
So
in
the
end,
what
I
do,
because
I
don't
really
care
too
much
about
the
difference
between
when
it
was
logged
and
when
I
parsed
it,
I
just
ignore
the
timestamps
and
use
the
time
of
parsing
by
default,
and
so
there
is
a
bit
of
a
difference
and
it
affects
my
ci
cd,
because
I
have
to
ignore
the
value
of
the
time
stamp.
When
I
do
a
diff.
A
But
if
you
use
the
wrong
filter,
it
just
exits
if
they're
not
set,
so
I
you
know,
I
I
had
a
bit
of
a
I
switched
from
modify
to
the
record
modifier
filter
because
it
meant
that
I
could
just
have
if
any
of
these
variables
are
here,
put
them
in
the
output
and
just
give
me
you
know
just
ignore
the
warning
saying:
oh
you
haven't
set
this.
It
works
really!
A
Well,
you
know
those
variables
were
set
as
environment
variables
by
my
watcher
as
part
of
the
downward
api,
and
so
only
reason
once
at
startup
just
keeps
injecting
them
in
it's.
It
works
pretty
well
and
helps
me
out
quite
a
bit
actually
with
some
debugging
and
stuff
like
that.
If
you
want
to
see
more
stuff,
so
my
time's
up
or
more
specifically,
let's
move
on
to
the
kind
of
the
future
work,
what
we're
doing
in
the
future
or
now
but
yeah.
A
So
one
thing
I
wanted
to
touch
on
yeah
fluke
on
eu
I
went
to
and
there
was
loads
of
suggestions
from
that
and
like
stuff
feedback.
That
was
really
good.
I
didn't
think
I
was
a
bit
concerned
about
performance
of
it,
but
loads
of
people
using
it.
So
I've
adopted
it
more
heavily
on-premise
deployments
we're
rolling
out
now,
basically
using
my
container
and
configuration
and
deploying
it
on
premise.
A
I
want
to
do
monitoring
ci
cd,
so
I
now
make
a
change.
I've
got
known,
input
see
what
the
impact
of
that
changes
on
processing,
performance
and
stuff
like
that,
which
is
a
bit
tricky,
but
calypso
have
also
introduced.
This
whole
custom
plug-in
approach
where
you
can
feed
it
back
and-
and
it
looks
quite
useful
for
that,
so
I
can
do
ci
cd
and
there's
new
multi-line
multi-format
options
now
1.8,
but
I'm
still
using
the
1.7
stuff,
but
there's
loads
of
good
ideas
and
customer
requests.
A
Documentation
from
catch
base
operator
goes
into
a
lot
more
detail
and
that
blog
post
on
tips
and
tricks,
I
think
is,
would
be
quite
useful.
It's
basically
this
presentation
but
expanded
for
you
know
a
lot
more
detail
and
you
can
kind
of
work
through
it
and
see
the
specifics
rather
me
just
talking
through
it.
So
thanks,
I
think,
we'll
probably
be
moving
into
q
a
now,
although
I'll
I'm
presenting
virtually
but
I'll
be
joining
shortly.