►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Bring Your Own Bytecode to the Logging Party - Guba Sándor & Dudas Adam, Cisco
Logs have become a huge burden on modern architecture. As there are more and more production distributed systems, properly handling logs, searching for failures, transforming, and extracting data is a crucial task. As the creators of the Logging Operator, Sandor and Adam know the complexity of these pipelines and the amount of work needed to set them up properly. All logging pipelines require extensibility. There is always a “thing” that makes your case a very specialized one. The common solution is to use a plugin system within your logging pipeline. WASM opened a lot of possibilities for portable code. WASM on the data plane is not a new concept. Envoy proxy uses WASM plugins to filter, transform, and authenticate HTTP content. This talk describes a plugin system specialized for logging purposes. It enables you to run the exact same plugins in the browser and the backend system, drastically reducing the turnaround time from planning to production. Moreover, you can test your setup on live streamed data.
A
Yeah
so
hello,
everyone
thank
you
for
joining
us
to
our
talk,
bring
your
own
Biko
to
the
logging
party,
I'm
Chandra
guba,
an
engineering
technical
leader
from
Cisco,
formerly
co-founder
of
bonsai
cloud
and
I,
have
like
more
than
five
years
experience
with
kubernetes
and
fluent
logging
ecosystem.
Please
welcome
my
colleague
Adam
as
well.
Hi.
B
A
So,
for
start,
let's
refresh
our
memories
about
kubernetes
logging.
It's
not
too
difficult,
because
you
can't
configure
anything.
The
container
runtime
writes
the
container
standard
output
on
the
hot
file
system
and
the
equivalent
service
can
read
those
files
and
the
user
can
access
them
via
the
kubernetes
API.
A
And
if
you
want
to
do
anything
more
with
your
logs,
you
have
to
deploy
some
subsystem
to
to
handle
these
things
and
logging
operator
is
one
of
I
think
the
most
interesting
tool
to
handle
that
under
the
hood
it
deploys
a
fluent
bit
demon
set,
which
tells
those
files
from
your
host
attach
kubernetes
metadata
and
send
all
the
logs
to
affluentes
TS
tool
set
in
fluently.
We
add
a
special
plugin
called
the
router,
which
makes
you
able
to
root
the
different
kind
of
logs
based
on
the
kubernetes
metadata.
A
What
it
means
is
is,
if
you
have
a
label
course,
for
example,
app
engine
X.
You
will
know
that
those
logs
will
be
nginx
access
logs
and
you
can
configure
your
log
pipelines
accordingly.
So
you
can
extract
the
metrics
from
them
parse
those
files
and
at
the
end
of
the
line
you
can
send
a
wherever
you
want
to
do
it
and
I
could
go
on
and
on
for
hours
about
logging
operator.
But
it's
not
a
deep
dive
on
that
matter.
A
I
just
want
to
highlight
that
it
has
a
more
than
a
thousand
GitHub
Stars
thriving
community
on
slack
with
more
than
500
members,
and
if
you
are
using
Venture
distribution
of
kubernetes
under
the
hood,
you
are
using
a
logging
operator
as
well
and
if
I
am
starting
with
this
out
of
a
vast
conference,
so
logging
is
hard.
For
example,
log
formats
are
so
different
that
you
can't
even
like
count
them.
A
If
you
just
check
access
logs
like
nginx,
Apache,
AJ
proxy,
they
have
all
the
same
information
but
in
different
formats,
not
to
mention
about
custom
applications
that
you
are
operating
in
different
companies.
They
are
like
a
whole
lot
of
different
formats
and
there
are
routers.
There
are
appliances
there
are
iot,
so
it's
hard
to
make
a
common
sense
of
logs,
and
if
you
are
go
one
step
further,
you
not
just
want
to
parse
those
locks
and
pass
along
the
line.
You
want
to
abstract
value
from
it.
A
You
want
to
maybe
want
to
create
metrics
from
an
access
log
or
you
want
to
unify,
unify
different
types
of
logs
like
HTTP
or
jrpc
success
rates,
and
if
you
prepared
everything
and
everything
works,
fine,
there
are
still
exceptions
like
go
panic
Java
exception.
So
there
is
another
thing
that
you
need
to
be
prepared
and
if
it's
not
enough,
you
have
always
something
special
in
your
infrastructure.
A
There
can
be
like
a
special
time
formats
that
you
need
to
unify,
or
you
want
to
inject
a
third
party
information
into
your
log
lines,
for
example
a
typical
example
when
you
want
to
have
kubernetes
method
that
are
on
the
log
lines,
you
need
to
ask
the
kubernetes
API
get
the
information
attached
to
the
log
lines
and
it's
becoming
a
common
practice
as
well
to
use
AI
to
process
your
logs.
So
there
are
so
many
logs
that
it's.
A
So,
just
to
summarize,
what
is
the
requirements
of
such
system?
It
needs
to
be
performant,
we
are
doing
work
on
the
data
plane
and
there
is
a
lot
of
data
when
we
are
talking
about
logs.
It
should
be
pluggable,
because
if
it's
not,
you
will
have
a
big
monolithic
code
base.
You
have
to
manage
a
lot
of
third-party
contributors.
A
You
need
to
watch
for
code
Clarity,
the
compile
times
could
go
to
the
sky
and
with
pluggable
infrastructure
you
can
simplify
the
complexity
of
The
Code
by
the
making
it
a
configuration
in
instead
of
code
and
safety
is
also
a
really
important
part
of
logging.
Remember
logging
usually
handles
sensitive
data
as
well,
and
why
I'm
talking
about
logging
and
now
I
will
start
to
talk
about
events.
I
think
you
know
where
I'm
heading
so
performance
is
crucial,
invest
and
the
the
one
of
the
main
purpose
of
vasm
is
real
life
native
Black
Performance.
A
It's
plug-in
bill.
I!
Don't
need
to
describe
that.
We
are
talking
about
this
all
day.
Safety.
I
think
this
is
the
maybe
more
important
than
performance
itself
and
we
have
private
how
to
use
vasma
as
plugins.
For
example,
in
Android
proxy
you
can
use
Advanced
plugins
to
filter
or
authenticate
HTTP
traffic.
Vasm
is
language
Independence,
so
you
don't
have
to
learn
yet
another
language
or
configuration
language
whatever
to
achieve
what
you
want
and
a
plus
one
that
it
runs
into
browser
as
well.
A
So
we
have
a
theoretical
architecture
of
SAS
system
where
you
have
the
log
Source
on
the
left
side
and
the
logs
are
coming
into
the
framework.
It's
a
awesome
runtime
and
you
have
everything,
invest,
plugins
and
transporting
routing
filtering
all
the
way
to
the
end.
So
from
a
complex
system,
we
created
a
distributed,
complex
system
which
doesn't
sound
a
really
good
thing
to
do.
Unless
you
have
the
tools
to
operate
and
observe
what
happens
in
your
system
and
we
will
concentrate
on
that
part,
we
will
introduce
a
tool
that
will
receive
log
lines
through
websocket.
B
B
So
demo
time.
Let's
see
this
in
action,
so
our
basic
interface
in
the
browser
is
this:
in
the
middle,
you
have
an
input
field
where
I
can
input
anything
press
run
and
you
get
the
same
thing.
We
don't
have
any
pipeline,
yet
it's
empty.
So
if
I
type
something
run,
it's
the
same
thing
now,
I'll
activate
Auto
runs,
so
we
don't
have
to
run,
but
press
run
all
the
time.
So,
let's
create
a
plugin
here
is
a
basic
plugin
written
in
go.
This
is
like
the
hello
world
of
such
plugins
in
the
receive
method.
B
B
B
B
A
B
B
We
also
have
an
engine
exposure
plugin,
which
is
parsing
nginx
logs.
So,
as
you
can
see,
we
have
a
few
errors.
This
is
because
the
code
wants
to
have
a
structured,
Json
message
as
input.
So
if
I'll
do
that,
now
we
get
something.
So,
as
you
can
see,
the
engine
exposure
is
parsing.
The
message
field
of
the
log
and
extracting
key
value
pairs
such
as
the
agent
the
status
code
or
the
remote
such
things.
A
Yeah,
so
it's
really
nice
to
have
this
kind
of
tool
set.
You
have
the
text
box.
You
can
try
out
each
line
of
code,
you
want,
but
it
would
be
much
better
to
grab
those
logs
from
a
production
or
production-like
system
and
go
through
them,
because
we
want
to
simulate
the
the
flows
of
a
backend
system
in
your
browser
and
back
in
Cuba.
A
Con
Valencia
I
had
a
talk
about
the
lock
socket,
which
is
a
a
tool
to
tap
in
the
production
logs
of
the
kubernetes
cluster,
so
it
integrates
with
logging
operator
tapping
the
logs
and
fortunately
push
out
on
a
websocket.
So
now
we
have
this
architecture,
we
have
a
kubernetes
cluster.
We
have
logging
operator
installed,
we
have
log
socket
installed
and
we
utilize
the
port
forward
to
connect
the
cluster
with
this
local
machine,
and
we
can
now
connect
and
tap
into
the
logs
of
a
kubernetes
cluster.
B
So
get
back
to
it,
I'll
just
remove
these
reverse
plugins
because
they
are
not
doing
any
good
so
to
connect
to
a
websocket.
You
need
a
URL.
Let's
generate
that
URL
and
also
I'd
like
to
add
this
is
not
specific
to
lock
socket,
you
can
use
any
websocket
URL
here.
That
is
sending
data
we'll
use
that
as
log
data.
So
if
I
select
any
of
these
messages,
the
input
updates
and
the
auto
run
runs
our
Pipeline
on
the
log
line.
A
Yeah,
so
we
have
now
production
access
logs,
but
what
is
the
common
property
of
excess
loss?
They
are
boring,
especially
the
status
because
they
are
just
numbers,
but
fortunately
we
had
just
the
right
plugin
to
make
it
a
bit
more
fun.
We
wrote
a
plugin
in
assembly
script
which
will
change
the
sax
codes
numbers
into
emojis,
so
the
200s
will
be
a
smiling
face.
The
300s
will
be
pointing
finger,
the
forehands
of
it
will
be
a
set
phase
and
the
500s
will
be
seek
phase.
So
let's
try
that
out.
A
So,
as
you
can
see,
we
have
an
input
and
we
have
the
plugin,
the
parser
plugin
and
after
that,
the
mog5
plugin
and
we
have
the
status
codes
changed.
But
one
of
the
most
benefits
of
the
other
assembly
is
to
have
more
languages.
So
this
was
written
in
assembly
script
and
we
have
the
another
one
written
in,
go
I!
Think
yes,
and
it
will
help
us
change
the
borrowing
agent
long
strings
into
a
more
fun
and
modified
agent
strings.
A
And,
as
you
can
see,
we
changed
the
status
code.
We
change
the
Asian
screen
agent
text
as
well
and
we
can
click
through
the
plugins
so
check
at
which
plugin
what
will
be
the
output.
So
this
is
an
important
thing
if
you
are
building
a
long
Pipeline
and
you
have
an
arrow
in
the
middle
because
you
transfer
something
badly,
you
can
just
click
through
and
and
get
where
the
error
happens.
But
why
the
stop
the
madness
here
we
have
another
rust
plugin
that
will
change
anything
that
it
finds
you
emojis.
A
Yes,
so
we
have
everything
in
place,
so
we
created
the
boring,
assess
logs
into
good,
modified
access.
Logs
and
I
know
that
this
is
not
the
most
production
ready
thing
to
do
and
I
don't
advise
to
write
it
in
production.
But
you
can
get
the
idea
where
we
are
heading
so
you
can
do
and
inspect
every
logs
in
your
production.
A
You
can
create
your
own
velocity
plugins,
you
can
try
them
out
and
you
can
build
this
whole
pipeline
in
your
browser
and,
if
you're
done
with
it
in
your
browser,
you
can
push
these
plugins
to
whatever
backend
system
then
run
around
those
trees.
And
if
you
don't
like
working
in
the
UI,
because
you
are
a
CLI
to
Maniac,
then
you
have
an
option
to
use
our
log
test
CLI
tool
as
well
and
just
grab
those
emojis
from
the
excess
looks.
B
So
a
few
lessons
we've
learned:
while
we
were
developing
the
tool
and
the
plugins
for
it,
there
are
still
few
languages
that
have
full
vasm
support.
There
are
many
awesome
languages
that
have
some
Wes
and
support,
but
a
few
have
the
full
support.
Rust
seem
to
be
the
best
option
for
us,
as
we
were,
trying
to
control
the
modules
interface
quite
strictly,
maybe
C,
C
plus
plus,
has
the
same
with
golang.
B
We
ran
into
some
problems,
for
example
the
built-in
Json
package
of
go
uses
Reflection
by
default,
which
is
not
really
supported
by
tiny
go
at
the
moment.
So
we
couldn't
use
that
we
had
to
use
a
third-party
plugin
assembly
script,
which
is
trying
to
mimic
typescript
and
JavaScript
is
missing
Json
and
the
regex
support
it
needs
also
third-party
libraries
for
this
python,
Ruby
JavaScript
and
as
I've
heard
c-sharp
and
the
CLR
has
the
mostly
the
same
thing
it.
B
It
has
kind
of
an
interpreter
which
is
compiled
to
wesm,
and
this
really
restricts
the
exports
and
imports
of
the
module
mainly
to
the
interpreter's
functions.
So
it's
not
really
easy
to
export
or
import
your
codes
functions.
Vs
support
is
a
hit
and
miss
it.
It's
not
available
in
the
browser
for
obvious
reasons
and
different
runtimes
implemented
differently.
It's
changing
version
models
without
generated
JS
bindings,
which
is
the
default.
As
we've
experienced
for
most
languages.
It's
try.
The
compiler
tries
to
generate
these
bindings
untrivial
to
create
and
use.
A
Yeah,
so
with
this
project,
our
future
plans
other
than
implementing
the
backend
system,
for
it
as
well,
is
like
adding
monitoring
information
like
memory
usage
and
execute
execution
time
to
the
plugins.
So
you
can
measure
properly
what
input
gives
you
the
best
output,
the
configuration?
So
imagine
if
you
want
to
build
a
production
grade
system
from
that,
you
want
to
have
a
version
marketplace
with
plugins
with
different
configuration,
so
that
should
be
included
in
this
testing
UI
as
well.
A
Creating
stateful
plugins
is
a
bit
harder,
for
example,
if
you
think
about
batching
deduping
handling,
multi-line
logs,
you
need
to
have
some
states
in
your
web
to
to
process
that
and
I
think
the
lowest
hanging
fluid
is
stream
processing
in
the
browser.
It's
totally
the
same
that
we
introduced
just
automatically
grab
the
latest
logs
and
do
the
transformation
on
the
pipeline
and
last
slide.
We
have
some
useful
links
that
helped
us
to
put
together
this
presentation.
A
The
logging
operator,
which
is
the
back
end
to
grabbing
the
logs
the
log
socket,
which
provides
website
output
from
the
logging
operator
flows.
You
have
the
whole
demo
published
on
GitHub,
so
you
can
just
access
it
and
try
it
at
home.
Everything
that
you
saw
here
is
open
source
and
there
is
the
link
for
the
cubecon
wall
and
see
a
presentation
as
well.
So
you
can
get
a
bit
more
understanding
about
how
logging
operator
works
yeah.
So
thank
you
for
your
attention.
C
A
I
would
say
that,
as
we
were
working
with
a
lot
of
stuff
and
used
a
lot
of
tools
to
manage
process
logs,
the
way
how
events
could
fill
in
gaps
with
these
plugins
and
performance
system
is
is
a
really
good
opportunity
to
make
one
one
good
tooth
for
all
I
I
think
it
sounds
too
good.
But
I
have
a
feeling
that
in
a
couple
of
years,
that
will
be
the
reality
enriching.