►
From YouTube: Fluent Bit: Extending Your Logging Pipeline with Go - Warren Fernandes & Jason Keene, Pivotal
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Fluent Bit: Extending Your Logging Pipeline with Go - Warren Fernandes & Jason Keene, Pivotal
Fluent Bit is a high-performance and lightweight logging agent implemented in C. Unlike its better known sibling Fluentd, it has fewer plugins for processing and forwarding logs. To facilitate the need for plugins, Fluent Bit has exposed interfaces for writing output plugins in Go. In this talk Warren and Jason describe how they implemented an output plugin for the syslog protocol whilst explaining the challenges they faced. Topics Include: - Why Fluent Bit - An overview of Fluent Bit’s architecture - How Fluent Bit interfaces between C and Go - How to write a Go output plugin - What are the current limitations of Go plugins - Recent and future improvements to Fluent Bit’s Go interface This talk would interest those evaluating Fluent Bit to be a part of their logging solution. You’ll learn how extensible it can be by using third-party plugins or by building one of your own!
https://sched.co/MPby
A
Okay,
I
think
we're
gonna
get
started
here
thanks
everyone
for
joining
us
here,
I
know
it's
the
end
of
the
day
and
probably
pretty
exhausted
from
two
days
of
cou
calm.
But
we're
gonna
be
talking
to
you
to
you
today
about
fluent
Pitt,
which
is
a
vlogging
agent
that
you
might
be
familiar
with,
and
we're
going
to
talk
about
how
you
can
use,
go
to
integrate
with
fluent
bits
and
extend
your
logging
capabilities
using
go.
So
that's
the
main
topic,
just
a
quick
show
of
hands.
A
How
many
people
here
have
or
are
regular
users
of
go
okay,
so
good
good
amount
of
you
say
yeah
it's!
This
will
be
useful
for
you
all,
so
my
name
is
Jason
Keene
and
my
colleague
here
is
Warren
Fernandez.
We
work
for
pivotal
and
we
work
mainly
on
observability
tooling
for
kubernetes
and
we're
both
really
avid
go
users.
A
We
love
using
go
and
pivotal
is,
as
a
company
is
embrace,
go
pretty
heavily
so
pretty
much
any
problem
we
encounter
II
see
if
it's
possible
to
solve
of
using
go
and
go
is
a
language
that
is
increasingly
important
in
the
cloud
infrastructure.
Space
kubernetes
as
I'm
sure
everyone's
aware
is
appointed
ago.
You
know
pretty
much
all
the
tooling
that
we
use
is
in
somehow
related
to
go
or
implement
it
directly
and
go
so
it's
incredibly
important
for
us
as
a
company
and
as
a
community.
A
We're
then
going
to
kind
of
go
over
the
architecture
of
kind
of
do
a
little
dive
into
how
fluent
bits
works
internally
and
then
we'll
cover
the
NGO
interface,
so
how
a
little
bit
interacts
with
go.
And
finally,
we're
gonna
cover
some
of
our
ideas
and
plans
for
moving
the
ball
forward
in
regards
to
using
go
with
flip
that.
A
So,
let's
start
with,
why
why
we
choose
Olympic,
so
the
product
that
we
are
working
on
is
observability
capability
for
kubernetes,
and
one
of
the
aspects
of
that
is
collecting
logs
and
a
lot
of
our
customers
use
syslog
as
a
protocol.
So
we
basically
needed
to
get
all
the
logs
off
these
clusters
and
ship
them
to
syslog
destinations
message.
Reliability
was
a
primary
consideration
for
us,
so
we
wanted
to
have
a
system
that
would
not
just
forget
about
messages.
A
We
wanted
something
that,
if
the
log
messages
written
out
by
the
workload
we
wanted
to
eventually
get
delivered
to
the
logging
destination.
So
that
was
our
primary
consideration.
We
wanted
to
have
really
good
support
in
the
community.
The
communities
community
I,
preferably
a
project-
that's
part
of
the
CNC
F
go
again-
is
like
first-class
thing
that
we
we
want
a
project
as
first-class
support
for
go
RFC
5424,
so
the
syslog
specification
is
standardized
in
the
IETF
RFC
5424.
A
A
lot
of
this
law
implementations,
don't
necessarily
follow
this
standard,
so
we
wanted
to
make
sure
that
we
had
a
standards,
compliant
implementation
and
finally
I'm.
Looking
for
a
agents,
we
were
looking
for
something
that
wouldn't
consume
a
lot
of
resources
and
would
be
somewhat
performance.
So
those
are
all
our
criteria
in
order.
So
let's
take
a
look
at
flow
and
D
and
Flint
bit
and
kind
of
how
they
differ
in
these
criteria,
so
just
kind
of
overview
both
of
these
and
how
they
differ.
A
So
fluent
D
is
implemented
in
Ruby
and
has
some
gems
that
use
like
C
modules
for
performance,
critical
code
so
stuff
like
the
message
pack
capability
is
implemented
in
like
a
C
extension
for
Ruby,
but
most
of
its
implemented
in
Ruby
fluent
bit
itself
is
implemented
in
c
fluent
d.
Its
might
have
heard
you
know
it
has
a
really
vast
ecosystem
of
plugins.
Last
number
I
heard
was
over
a
thousand
different
plugins
available
for
fluent
e,
which
is
incredible.
A
Thump.
It,
on
the
other
hand,
is
a
relatively
new
project,
so
it
has
less.
You
know.
Plugins
less
capability
in
that
regard,
and
the
plugins
for
film
bit
are
mostly
statically
built
with
fluent
bit
so
they're,
not
things
that
you
kind
of
can
just
like
pull
off
the
internet
and
start
running.
So
there's
less
of
those
so
there's
you
know
Neron
54,
last
time,
I
checked
so
yeah
fluent
D.
A
There
are
options
for
go
and
that's
what
we're
gonna
be
going
in
today
into
today,
and
then
there's
also
some
Lua
capability
as
well
fluent
D
when
it
comes
to
memory
and
CPU,
just
an
idle
state,
fluent
e
will
consume
about
40
Meg's
of
RAM
and
fluent
Belle
will
consume
much
less
around
500
kilobytes
and
as
well
as
memory
CPU
fluent
e
consumes
a
little
bit
more
CPU
than
flip
it.
Just
an
idle
state.
A
A
Whereas
swingbed
has
some
of
those
capabilities
but
doesn't
have
the
large
amount
of
plugins.
So
it's
not
really
well-suited
for
that
role,
but
it
is
better
suited
for
kind
of
like
the
edge
collection
of
data.
So
has
really
good
input
plugins
and
it's
really
performant.
It
doesn't
consume
a
lot
of
resources,
so
it's
better
as
like
a
daemon
set,
so
an
agent
that
you're
running
your
cluster
on
every
single
node
and
finally,
both
fluent
D
and
fluent
bit,
have
support
for
something
called
the
forward
protocol
which
just
allows
them
to
interoperate.
A
So
it's
not
really
a
question
of
flowing
D
versus
for
a
bit.
It's
really
which
technology
is
appropriate
in
the
right
location.
So
if
you
want
to
use
one
D
where
it
makes
sense,
you
can
do
that
and
you
can
interrupt
with
one
bit
using
this
forward
protocol
so
going
back
to
our
national
criteria.
Message.
Reliability,
so
both
pulling
the
influent
bit
do
a
really
good
job
at
message:
reliability.
They
both
support,
buffering
in
memory
and
to
disk,
that's
something
that
was
introduced
recently
and
flip
it
a
few
months
ago.
A
So
these
support
buffering
and
they
support
retracted
messages.
So
if
a
message
fails
to
send,
you
can
have
the
engine
retry.
The
message
for
you
kubernetes
support
both
have
really
good
first-class
support
for
reading
log
files
off
a
disk
and
then
kubernetes
metadata
filter
for
taking
the
log
message
figuring
out.
You
know
what
pod,
when
container,
it's
associated
with
and
annotating
the
message
with
that
information,
CNS
CN
CF
status
so
flowing
the
as
a
recent
graduate
from
the
CN
CF,
so
I
think
it's
a
six
project
that
was
graduated
from
CNCs.
A
So
it's
really
mature
for
a
bit,
on
the
other
hand,
is
kind
of
a
newer
project,
it's
technically
a
sub-project
of
fluency,
so
it's
still
kind
of
part
of
that
CN
CF
status,
but
it's
still
relatively
new
go
support.
Fluent
D
doesn't
really
have
any
support
for
go
fluent
bid.
Support
supported
at
the
time
of
us
going
through
this
criteria,
only
supported
output,
buggins
RFC
5424.
So
there
was
an
implementation
of
a
systolic
gem
for
fluently.
A
When
we
investigated
it,
it
turned
out
to
not
really
be
five
for
two
for
compliance
and,
if
you
use
it
with
a
kubernetes
cluster,
since
the
tag
is
the
file
path
of
a
the
log
and
that
actually
gets
written
out
in
the
systolic
message
as
the
tag
it
actually
caused
the
Ruby
interpreter
to
throw
an
exception.
So
pretty
much
if
you
use
it
with
kubernetes
that
doesn't
really
work
too
well,
so
that
wasn't
really
a
good
option
for
us
and
then
flump.
A
It
didn't
really
have
support
for
syslog
output
plugins
at
the
time
and
I
think
it
has
like
input
plugins
support,
but
it
doesn't
have
support
for
sending
out
messages
via
Susilo
resource
usage,
so,
like
I
mentioned
before,
both
with
resource
usage
and
performance
flump,
it
kind
of
wins
out
in
that
regard.
So,
given
the
kind
of
all
these
criteria
we
decided
fluent
pet
would
probably
be
the
best
choice
for
us,
and
you
know
the
main
thing
that
came
down
to
is
ease
of
implementing
the
plug-in
using
go
so
yeah.
That's
it.
B
So
I'm
just
gonna
cover
a
high-level
overview
of
the
architecture
of
phone
bed,
just
FYI
most
of
the
stuff
that
we're
talking
about
it's
already
being
covered
by
Eduardo
Silva
who's,
one
of
the
core
maintainer
Assaf
fun
bit,
and
this
is
a
screen
shot
of
his
talk
last
year
and
Cuba
on
Seattle.
How
many
of
you
guys
went
to
the
talk
like
a
couple
hours
ago?
He
had
his
yogi
cope.
We
have
one
all
right
new
stuff,
for
you
guys
yeah.
B
He
gave
a
deep
dive
of
the
newer
stream
processing
support
fulfillment,
but
I
won't
be
covering
that
I'm
all
right.
So
this
is
essentially
the
logging
pipeline
of
who
embed
you
might
have
seen
this
on
day.
One
keynote
staged
very
similar
diagram,
we're
trying
to
cover
this
pretty
quickly.
So
one
of
the
things
that
you
have
to
understand
look
fluent
bit.
It
was
designed
to
be
a
data
analysis
tool
and
essentially
was
designed
to
be
a
ubiquitous
across
a
lot
of
platforms
right.
B
It
was
actually
designed
for
embedded
world
and
not
really
with
any
sort
of
so
structure
in
mind
was
trying
to
be
as
flexible.
You
know
ubiquitous
across
the
whole
platform,
so
they
try
to
build
the
pipeline
with
a
lot
of
flexibility.
So
the
let's
start
with
stage
one
to
make
first
thing
that
fluid
does
is
it
has
an
input
stage?
B
This
is
where
it
actually
collects
the
logs
right,
and
this
could
be
from
tailing
files
to
reading
off
the
network,
or
you
know
various
myriad
of
ways,
one
of
the
T
key
things
that
the
input
stage
does.
Is
it
actually
tags?
The
tag
is
fighting
the
configuration
for
that
instance,
and
it
says
okay
I
want
to
tag
these
records
with
a
specific
thing
like
a
hostname
or
cluster
name
whatever
from
there
on
the
data,
the
chunk
of
data
gets
sent
over
to
a
parser.
B
So
you
can
literally
get
a
string
of
texts
that
are
the
tokens
are
two
spaces
away,
so
this
parsing
stage
does
the
important
job
of
normalizing
that
data
and
adding
structure
to
those
log
messages
right
from
then
on
and
goes
to
the
filtering
stage,
and
this
is
actually
my
opinion
like
one
of
the
most
important
things
in
flume
bed.
So
this
filtering
stage
allows
you
to
one
thing:
it
does
allows
you
to
enrich
your
data.
B
What
does
that
mean
so
in
our
world
in
a
kubernetes
will
write
the
log
that
comes
in
from
the
docker
file
on
the
dock?
A
log
file?
Is
you
have
a
JSON
with
a
key
of
log
stream,
which
is
standard
out
a
standard
error
and
then
I
think
a
timestamp
and
that's
very
basic?
But
if
you
use
the
kubernetes
metadata
filter,
it
enriches
your
log
with
all
this
metadata
of.
Where
did
this
log
come
from
the
pod
name?
B
The
container
ID
labels
annotations
again
they'd
built
for
data
analysis,
so
you
have
no
all
this
rich
metadata
that
you
can
use
later
downstream.
So
that's
one
important
job
at
the
filtering
stages.
The
next
important
job
is
removal
of
data.
If
some
of
you've
used
tools
like
Splunk
y-you
can
understand
how
expensive
a
log
can
be
and
storing
every
single
log
and
some
sometimes
you
want
to
do
that.
Sometimes
you
don't
need
that.
So
this
is
the
stage
where
you
can
say
how,
if
it,
you
know,
doesn't
match
the
tag
or
it
doesn't
have
this.
B
You
know
word
like
the
word
error
in
it:
I
don't
care
about
it,
get
rid
of
it,
remove
that
data,
and
so
it
kind
of
lightens
that
load
after
that
it
goes
to
the
buffer
Jason
I
mentioned
before
that
one
of
our
cry
tears
was
reliability
and
we've
been
in
the
observing
space
working
foundry.
One
of
the
things
we
kept
on
seeing
was
a
back
pressure
and
the
outputs
are
failed
to
send
messages
fast
enough,
so
really
having
this
buffer
by
default.
B
It's
like
an
in-memory
buffer
but
I
think
as
a
version
wondered
how
you
can
configure
from
bit
to
have
alarms
on
disk
like
a
storage
mechanism
for
the
buffer,
and
this
buffer
then
allows
fun
bit
to
do
message.
Retries
then
it
comes
the
routing
stage,
which
literally
remember
they're
tagging
on
the
inputs,
the
right
in
the
beginning.
B
It
does
a
wild
card
or
reg
x'q
match
on
the
tag
and
says:
okay,
you
know
plus
tag
things
that
are
tagged
with
cluster
eggs
and
to
output,
a
cluster
B's
and
up
a
B,
so
yeah,
and
then
you
have
outputs
and
the
stuff
that
we're
talking
about
is
up
the
output
side
of
things
and
did
I
mention
that
54
plugins
right
now
or
50,
ish,
plugins
and
foam
bed.
These
are
the
stages
that
can
be
pluggable
right
and
again
they
did
this
so
that
it
you're
empowered
to
be
as
flexible
as
you
need.
B
So
things
don't
work
out,
hey
write,
a
plug-in
right
and
just
in
our
case,
we
just
didn't
want
to
write
a
C++
have
that
statically
linked.
So
we
saw
that
fluid.
It
was
the
ability
to
dynamically
lingo
plugins,
which
is
why
we're
here
alright.
So
let's
go
on
a
little
bit
of
the
internals,
so
Foom
bit
his
runs
as
a
single
thread.
B
Remember
I
said
the
initial
intent
was
to
run
in
an
embedded
system,
so
it
had
to
be
very
lightweight
right
and
everything
in
fluent
bit
is
configured
with
this
config
file,
everything
so
things
like
flush
into
worlds.
If
service
level,
config
to
the
inputs
and
outputs
and
all
of
the
instances
that
you
need
to
configure
and
the
first
thing
it
does
and
it's
the
reason
it
allows
this
config
file,
it
looks
and
sees
oh
it's.
What
are
the?
Are
there
any
go?
B
Plugins
or
there
are
you
have
configured
perfect,
I'm
going
to
load
those
shared
objects
in
and
load
these
plugins
in
and
register
the
plugins?
The
first
thing
it
does.
It
says:
okay,
now
this
go
plug-in.
Is
you
know
in
my
list
of
output,
plugins
right
and
then
it
goes
on
down
the
config
file
and
says:
ok,
let's
initialize
each
of
these
input,
plugins
filters
and
output
plugins.
B
So
when
it's
part
of
this
initialization
process,
what
happens
is
so
think
of
like
a
it's
saying,
okay,
I'm
gonna
go
to
my
list
of
input
plugins
and
there's
an
input
plug-in
that
says:
ok
I
need
to
read
data
from
some
Network
server,
so
I
need
to
set
up
a
TCP
connection.
So
as
part
of
that
initialization
list
input
plug
and
says:
ok,
I'm
going
to
set
up
the
CCP
socket
and
we're
start
reading
data
and
then
what
happens.
Remember.
B
Flume
bid
is
running
as
a
single
thread
right
and
did
I
can't
mention
the
word
scheduler
and
event
loop
there.
So
it
kind
of
gives
you
a
hint.
So
everything
just
just
stops
right
in
fluent.
It's
like
hey
I
got
other
stuff
to
do.
I
got
other
plugins
to
initialize,
what's
what's
the
process,
so
what
Foom
bid
does
is?
Actually
it
invokes?
This
age-old
flow
control
of
co-routines
right
and
this
co-routines
essentially
allows
for
a
cooperative
passing
of
execution
between
the
plugin
and
the
engine
itself.
So
what
the
input
plug-in
does
it
says?
B
B
I
got
some
data,
that's
coming
through
the
socket
the
sockets
ready
to
be
read,
so
it
then
resumes
control
back
to
the
importer
plug-in
and
says
you
are
ready
to
read
and
it
then
import
plugin
says
thank
you
and
it
says:
okay,
I'm
gonna
start
reading
record.
So
there's
this
really
nice
cooperative
flow
between
the
engine
on
the
plugins.
B
It
says:
okay,
I
got
this
chunk
I'm
going
to
tag
this
data,
send
it
on
to
the
next
stage.
The
filtering
does
the
matching
on
this
tag.
It
says
it
goes
through
the
list
of
filters.
It
says:
okay,
this
record
I'm
gonna
append
some
data
and
enrich
it,
and
the
other
records
is
like
a
door
care
watching
get
rid
of
you
from
there
on
and
goes
to
the
buffer,
and
you
know
it's
stored
in
the
buffer.
B
While
this
is
all
happening,
the
event
loop
kicks
in
and
they
have
this
flush
interval
which
is
configured
the
flesh,
becomes
answers
well,
all
right
time
to
send
the
data
on
through.
We
got
a
sense
and
the
data
out
so
form
bit
eternally
what
it
does
is
it
creates
these
tasks
for
each
of
the
input,
instances
and
says:
ok,
I'm,
going
to
create
these
tasks.
In
my
event,
loop
and
we
start
flushing
data
out
and
again
that
same
dance
happens
right
for
the
output
plugins
with
the
output
plugins.
B
Most
of
the
time
is
network
I/o
out.
So
it
says:
ok,
it's
time
to
time
to
send
the
data
out.
So
again
we
have
this
cooperative
dance
between
the
engine
and
the
plug
and
once
the
output,
the
last
thing
that
the
output
plug-in
does
is
when
it
sends
the
data
it
can
actually
respond
back
to
the
fluid
engine.
Saying
hey
everything
went
well
or
something
went
wrong.
Can
you
retry
sending
that
pillar?
A
Right,
so
this
is
the
fun
part
for
me.
So
how
does
a
bit
interface
with
the
go
so
flip?
It
is
a
little
bit
as
C
and
we
have
some
code.
Where
didn't
go,
that
we
want
to
interact
with
fluent
bit.
You
may
be
a
winner
of
something
known
as
C
Co,
which
is
like
the
Interop
capability
between
go
and
see,
so
we're
gonna
be
using
that.
A
But
how
exactly
do
we
want
to
kind
of
call
from
C
in
to
go
so
that's
gonna
be
handled
by
just
you
know,
regular
dynamic
linking
so
the
C
program
is
gonna.
Call
DL,
open,
DL
sim
in
order
to
you
know,
call
into
the
shared
object
that
we
provide,
but
you
might
be
wondering
like
how
does
this
dot
Esso
like
get
created?
Normally,
when
I
compile
things
with
go
I
get
a
static
binary
so
that
one
of
the
major
selling
points
would
go
is
static
binaries.
So
how
do
we
generate
this
dot
s?
A
Oh
so
there's
this
flag
that
you
can
give
the
go
compiler
when
you
say
go
pill
can
say
built
mode
and
there's
various
options
that
you
can
give
that
flag
and
the
ones
that
you're,
probably
familiar
with,
or
at
least
most
people
would
be
familiar
with,
are
archived
and
exe.
So
archive
and
exe
archive
will
generate
those
a
files
that
you
have
in
your
F
undergo
path,
pkg
and
then
exe
is
just
kind
like
your
normal
belt
boat
for
building
a
set
executive.
A
All
so
an
archive
will
get
built
when
you're
building
a
package
and
exe
will
get
built
when
you're
building
a
main
package.
So
those
like
that
the
regular
modes,
some
of
the
more
esoteric
ones,
are
plugin.
So
you
might
be
thinking
plugin.
That
sounds
like
what
we
want
to
use,
so
plugin
actually
is
used
to
generate
a
dynamic
linkable
shared
object
of
go
code
that
is
intended
to
be
called
from
another
go
process.
So
it's
not
exactly
what
we're
looking
for
we're
looking
to
call
into
the
go
code
from
C.
A
So
one
thing
that
we
could
choose
here
is
a
C
archive.
What
that
will
do
is
it'll,
allow
us
to
call
from
C
into
the
go
code,
but
it
will
generate
as
a
a
a
static
archive.
So
it's
not
exactly
what
we
want,
so
the
one
as
you
can
tell
here
that
we
want
is
C
shared
so
that
allow
the
C
program
to
dynamically
link
against
a
shared
object
that
we
compile.
A
So
what
does
that
look
like?
So,
if
you're
in
go,
and
you
want
to
create
one
of
these
shared
objects,
you
just
create
a
main
package.
You
say:
I
want
to
import
C,
which
is
like
a
pseudo
package
to
signify
to
the
compiler.
You
want
to
C
go
and
then
you
just
kind
of
write
your
functions
as
you
would
normally,
and
you
just
put
this
little
comment
above
the
function
that
just
says,
I
want
to
export
this
into
C.
A
So
this
is
like
the
extern
capability,
so
it
just
makes
it
a
global
symbol
in
your
symbol
table
and
then
you
have
to
have
a
maned
at
the
bottom,
just
kind
of
as
boilerplate.
The
compiler
requires
that
so
you
just
have
to
include
that,
but
that's
about
it.
So
you
just
say:
go
build
output
plug-in,
do
so
and
then
build
mode,
C
shared
and
that
will
generate
a
plugin
so
and
if
you
use
something
like
read
off
to
just
dump
out
the
symbol
table,
you
can
see.
A
There's
the
dynamic
symbol
table
has
a
symbol
for
my
awesome
function,
which
is
what
we
want.
We
want
to
a
symbol
that
we
can
link
against
from
our
C
program.
So
that's
what
we
wanted,
so
that
takes
care
of
generating
this
plugin
SSO.
But
you
might
also
be
wondering
well
C
and
go
are
entirely
different
beasts
like
they
have
different
memory
layout
for
different
types.
They
have
different
calling
conventions
like
how
do
we
like
manage
the
differences
between
these
two
different
languages?
A
So
the
first
thing
is
like
calling
conventions,
so
C's
callin
convention
is
entirely
different
from
gogo
has
its
own
special
calling
convention.
So
how
do
we
minimize
that
different?
So
the
compiler
actually
just
takes
care
of
that
for
you
when
you
compile
that
dot
SOF
file?
In
addition
to
the
my
awesome
function
symbol,
the
original
go
function
is,
you
know,
generated
as
well,
just
not
as
an
extern
symbol
and
it's
just
my
main
dot.
A
My
awesome
functions
so
that
that's
like
your
go
function
and
then
the
function
that
it's
exposing
to
C
is
a
function
that
follows
the
C
calling
convention
and
then
knows
how
to
follow
the
go
calling
convention
and
just
calls
into
your
go
code
and
handles
kind
of
the
mapping
between
those
two
worlds
for
you.
So
you
don't
have
to
worry
about
that.
What
about
memory
layout
for
different
types
so
and
see
you
know
like
a
char
star,
is
different
from
a
go
string.
So
how
do
we
manage
these
differences?
A
So,
on
the
left
hand
side
here,
you
can
see
all
the
types
that
you'd
normally
have
in
C
and
on
the
right
hand,
side
is
how
you
represent
those
types
and
go.
So
if
you
want
to
refer
to
those
types
from
go
code
void,
stars,
unsafe,
pointer,
char
star
is
a
pointer
to
C
char,
so
on
and
so
forth.
Some
of
the
interesting
ones
are
like
structs,
enums
and
unions.
A
You
just
say
C
struct
and
then
an
underscore
or
followed
by
the
name
of
the
struct
to
kind
of
get
a
reference
to
that
type
and
go
and
then
the
inverse
going
back
into
C.
So
if
you
want
to
refer
to
go
types
in
C,
be
when
you
compile
your
des
Eaux
it'll
actually
give
you
a
header
file
as
well,
and
that
header
file
will
contain
definitions
for
all
these
types.
And
so
you
just
kind
of
pound
include
that
header
file
in
your
C
code
and
then
get
all
these
types
defined
for
you.
A
So
things
like
strings
or
representative
juices
go
string,
goes
slice,
go
in
so
go.
Slice
is
just
kind
of
like
this
slice
header,
so
it
doesn't
actually
have
the
type
information
for
your
slice
of
bite.
So
it
all
know
it's
a
slice,
but
not
that
it's
a
slice
of
bite
but
yeah.
Most
of
these
things
are
pretty
straightforward
and
then
seco
provides
capability
for
conversion
between
different
types,
so
different
memory,
layouts,
so
the
important
ones
are
like
strings
and
bytes.
A
A
A
So
how
do
you
write
a
fluent
bit?
Go
plug-in
itself,
so
I,
just
kind
of
to
explain
the
generic
interface
between
C
and
go
for
what
we
want
to
do.
So
what
are
the
specifics
about
like
using
flip
it
using
this
interface?
So
there's
only
four
functions
that
you
have
to
implement
and
these
four
functions
are
register
and
it's
flush,
CTX
and
exit
and
we'll
go
into
kind
of
the
details
of
these
register.
A
There's
some
helper
functions,
an
output
package
that
you
can
use
and
basically
the
job
of
the
register
function,
is
to
allow
the
go
code
to
communicate
back
to
the
C
code.
Information
about
the
plugins,
so
in
this
case
we're
setting
the
plugins
name
and
a
description
for
the
plugin,
the
in
its
said
that
so
the
register
gets
called
once
for
per
shared
objects.
So
you
kind
of
think
of
it
as
global
for
your
plugin.
The
in
it,
however,
is
gets
called
once
per
instance.
A
So
if
you
can
figure
it
like,
say
five
different
instances
of
syslog
output
plugins,
then
this
will
get
called
five
different
times
and
in
this
function
you
typically
read
from
the
config.
So
there's
this
a
fop
plugin
config
key.
So
you
can
just
read
whatever
key
you
want
out
of
the
config
and
then
the
important
bit
here
is
to
set
the
context.
A
So
you
call
FL,
be
plug-in,
set
context
and
you
can
set
any
arbitrary
like
go
value
as
the
context
for
that
instance,
and
that's
important
later
on
when
we're
flushing
to
be
able
to
disambiguate
implications
of
the
flush.
So
this
is
what
the
flush
looks
like.
So
the
first
thing
we
do
is
kind
of
reconstitute
that
context,
and
we
just
do
that
by
making
call
int.
This
helper
function
called
get
context,
and
then
we
have
to
type
a
certain
because
it
loses
its
static
type
information.
A
It
still
has
its
runtime
type
information,
but
we
need
to
reconstitute
it
as
like
a
original
type.
So
in
this
case
it
was
a
string,
so
we
type
assert
it
to
be
a
string
and
then
there's
other
helper
functions
that
allow
us
to
then
decode
the
data
from
fluent
bits.
All
the
messages
that
are
in
code
would
be
a
message
back.
A
We
can
DK
decode
those
and
handle
them
kind
of
one
by
one
in
this
case,
we're
just
kind
of
logging
out,
but
the
data,
and
then
at
the
end
here,
as
Lauren
mentioned
before,
you
can
kind
of
return
and
one
of
three
values.
Either
everything
was
okay
or
you
need
to
retry
or
there's
an
error
and
I.
Finally,
just
kind
of
like
an
exit
tear
down
clean
up
function
that
you
can
implement,
so
you
can
put
whatever
you
need
to
do
in
there.
So
yeah,
that's
pretty
much
it
for
the
go
interface
overflowing
bit.
B
Oh
all
right
so
moving
forward.
So
what
we
kind
of
explained
to
you
was
our
use
case
where
we
needed
to
output
stuff
in
RC
502
for
syslog
format
right
and
we
worked
working
with
the
flume
bit.
Teams
like
eduardo
and
we
were
able
to
add
some
contribute,
some
value
in
the
in
the
sense
that
right
now,
Jason
I
mention
about
multiple
instances.
So
you
can
have
now
multiple
instances
of
your
syslog
plug-in
in
the
configuration
file
and
you
can
and
actually
like
sense
to
different
destinations,
for
example
right.
B
But
what
else
can
we
do
to
better
support
this
goal?
Plugins
right,
let's
say,
say
you
guys,
love
this
talk.
You
know
you
tweet
about
it
and
then
you
like
leave
this
room
and
say
I'm
gonna
build
some
gold
plugins
for
four
flown
bit.
I
want
to
build
multiple
different,
go
plugins,
because
that's
what
I
need
I
need
to
send
600
552
for
or
my
awesome
protocol
or
you
know
whatever
she
have
multiple
different
go
plugins.
B
You
follow
this
process
and
then
you
will
kind
of
realize
that
hey
this
thing
was
supposed
to
be
lightweight
is
no
longer
as
lightweight
as
it
seems
right
because,
as
you
can
see
each
time
you
build
this
go
plug
and
as
the
see
more
you
have
the
entire
go
runtime
kind
of
packaged
with
it
right.
So
we
had
an
idea
of
like
improving
this
in
the
future.
So
you
just
to
go
back
to
the
go
build
modes.
You
have
this
plug-in
and
the
C
archive.
So
let's,
let's
take
a
look
at
that.
B
How
we
can
achieve
that.
So
one
of
the
ideas
we
had
was
having
a
go
plug-in
manager,
that's
built
with
the
C
archive,
build
mode.
That
way
you
can
statically
link
it
with
foam
bit
and
that
contains
to
go
wrong
time
and
then
that
plug-in
manager,
then
loads,
go
plugins
that
are
literally
built
with
the
plugin
build
mode
and
that's
as
simple
and
clean
you
don't
have
to
worry
too
much
about
the
C
go
set
of
things.
That's
just
one
idea
we
had
for
moving
forward
and
improving
plug-in
management.
B
Another
idea
we
had
was
establishing
a
version.
A
bi
or
application
binary
and
the
context,
for
example,
that
if
the
pass
in
requires
a
certain
way,
the
struct
is,
you
know
defined.
So
if
we
update
fluent
bit
to
have
anything
more
than
your
plugin
cannot
goes
off
scope
so
establishing
a
version
API
allows
you,
maybe
the
plugins
during
registration
time
would
say:
okay,
I
conform
with
a
bi
version,
1.2
and
flume
is
like
alright
you're
good
to
go.
B
So
that
was
another
thing,
establishing
a
version
API
so
that
nobody
breaks
in
the
future
is
flowing
bit
revs
versions.
The
next
thing
is
again:
this
is
only
for
output
plugins,
so
we
have
input
and
filter
plugins
that
can
be
also
implemented
in
go.
In
fact,
there's
a
PR
opened
for
supporting
filter
go
p--
plugins.
So
thank
you
for
that.
Whoever
you
are,
then
nothing
well.
B
We
have
metrics
and
logging,
so
go
so
for
bit
actually
exposes
a
metric
standpoint,
and
that
allows
you
to
get
metrics
of
the
records
flowing
through
your
system
while
JSON
or
you
know,
Prometheus
or
form
right,
and
it
tells
you
like
all
the
records
that
goes
in
your
input
and
they
have
been
filtered
out
and
then
have
been
successfully
sent.
This
is
what's
great
for
your
for
the
C
plugins
that
are
statically
linked
in
to
embed.
So
one
of
the
ideas
we
had
to
is
to
support
that
for
go
plugins
as
well.
B
Right
and
all
of
this
right
now
is
kind
of
work
done
in
this
form
bid
go
repo
again
we're
at
a
very
early
stage
of
us
doing.
This
go
support
for
fluent
bit.
So
if
you
are
interested
in
this
space,
we
want
you
to
join
the
flung
bit.
Slack
channel,
aren't
fluent
all
I,
think
it's
the
slack
domain
and
make
issues
and
PRS
and
contribute
to
fluent
bit
go
Eduardo
has
been
very
open
and
getting
these
PRS
through
and
improving
the
ecosystem.
So
yeah,
that's
about
it.
Thank
you
very
much.
D
Thanks
for
the
talk
very
interesting,
I
have
a
question
about
what
go
gives
you,
because
I
would
kind
of
a
cue
that
what
you
showed,
while
it's
very,
very
interesting
I,
wouldn't
call.
It
really
go
code
because
use
the
left
to
deal
with
these
types
and
the
conversion,
and
so
on
so
I'm.
Assuming
that
you,
maybe
you
could
talk
a
little
bit
about
what
you
Waco,
actually
helps
you
where
it
makes
it
simpler
to
to
use,
go
instead
of
writing
the
pregnancy
right.
A
Yeah,
so
we
try
to
hit
handle
most
of
the
complexity
of
like
dealing
with
see
types
in
the
interface
and
these
kind
of
like
helper
functions
that
we
provide
so
ideally
like,
as
we
advance
like
the
the
package
will
provide
most
functionality
that
a
normal
go
programmer
would
want
just
using
these
helper
functions.
So
you
just
call
it
those
helper
functions
and
they
kind
of
hide
all
that
complexity
behind
them.
A
As
far
as
like
what
like
it
gives
you
like
from
our
perspective,
like
we
have
really
robust
implementations
of
network
protocols
and
different
libraries
that
we
developed
over
the
years
and
go
and
be
just
being
able
to
leverage
that
and
the
huge
ecosystem
of
go.
You
know
third-party
packages
and
the
standard
library
like
it's
just
a
wealth
of
treasure.
B
I
think
one
of
the
requirements
for
product
was
also
time
to
market,
and
since
we
all
very
familiar
with
go,
I
mean
we
start
like
we
were
a
favorite
doing.
The
C
thing
is
just
in
order
to
get
the
stuff
out.
You
know
get
it
out,
we
iterate
iterate.
We
were
able
to
move
faster
with
some
bit
of
writing
go
plugins
and
we
did
that
and
then
we
were
like.
Oh,
this
is
great.
How
can
we
improve
on
it
and
it's
kinda?
What
kind
of
that
also.
E
A
So
when,
when
the
garbage
collector
is
invoked,
was
your
first
party
question
so
like
if
you
wanted
to
invoke
that
you
can
obviously
do
that
manually,
it
just
kind
of
whenever
the
run
time
decides
to
run.
So
whenever
you
call
in
to
go
code,
if
you
ever
look
at
the
output
of
the
compiler,
it
has
kind
of
hooks
into
the
runtime
where
it
has
an
opportunity
to
run
the
garbage
collector.
A
A
So
the
major
overhead
that
we've
ran
into
is
when
you
have
multiple
types
of
go:
plugins
dynamically
linked
at
the
same
time,
you
have
a
duplication
of
they
go
runtime
in
every
single
plug-in,
which
is
far
from
optimal.
So
that's
kind
of
like
what
we're
working
on
now
is
improving
that
to
where
you
don't
have
any
of
that
overhead
and
in
the
experiments
of
Iran,
which
is
having
a
single
go
runtime
and
calling
into
the
plugins
directly.
It's
the
same
function
call
overhead.
They
have
normally
and
go
so
it's
no
additional.