►
From YouTube: A New Approach to Logging as a Stack: Fluent Bit + PostgreSQL (FPS) - Jonathan Gonzalez, EDB
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
A New Approach to Logging as a Stack: Fluent Bit + PostgreSQL (FPS) - Jonathan Gonzalez, EDB
Logging at scale is a very interesting challenge, and having the right open source stack is mandatory. There are many ways to solve the log collection, processing and aggregation problem, but when it comes to perform data analysis once the data has been centralized, it's really easy to face latencies and get general problems when indexing our data that don't comes with a fixed structure. In this session, we will share our experience of implementing a new stack that we called FPS: Fluent Bit + PostgreSQL. The open source database PostgreSQL offers a very interesting storage engine for non-columnar structured documents: JSONB (binary JSON) which have shown to be a very efficient and scalable implementation to store trillions of record logs and provide fast results when performing queries.
https://sched.co/ekD0
A
Hello-
everyone,
I'm
jonathan
gonzalez
from
chile
and
I'm
currently
the
maintainer
of
the
post,
resql
output
plugin
for
flow
and
beat
in
the
last
ngo
for
my
life,
I've
been
working
with
data
processing
and
log
processing
for
reports,
stats
analysis
and
everything,
infosec
reporting.
Even
so
this
time,
I'm
going
to
talk
about
the
output
plane
for
fluent
beat,
but
what
does
it
mean?
This
means
that
I'm
going
to
explain
you
a
lot
of
stuff
related
to
how
to
process
the
locks
on
the
flow
and
bit
size
and
push
the
locks
inside
pull
the
sql.
A
So,
let's
start
the
talk,
one
of
the
motivation
to
pay
start
this
plugin
and
a
year
and
a
half
ago
it
was
to
play
with
the
post
sql
type
json
b,
which
is
apparently
representation
of
javascript
object.
Notation
we
needed
to
create
basically
to
process
years
of
logs
and
find
one
line
between
millions
of
files.
Wasn't
that
easy.
A
Like
I
don't
know
in
summer,
we
have
a
lot
of
visits
on
this
site,
or
this
summer
users
started
to
log
in
more
stuff
like
that
people
can
ask
questions
in
the
beginning.
The
initial
idea
was
to
follow
this
simple
process:
fluid
bit,
collect
data
collect
logs
from
different
sources,
using
different
input
plugins
and
send
them
directly
into
poster
sql
into
the
table
and
from
there
you
will
be
installing
logs.
A
But
with
time
and
talking
with
my
colleagues,
we
finally
decided
that
a
best
idea
would
be
keep
fluent
bit
sending
logs
into
post
sql,
but
first
process
them
using
pl's
sql
language.
This
will
be
allow
us
to
separate
files
in
between
different
tables
or
maybe
purposes
in
something
or
even
just
create
some
rules.
A
We
have
the
logs
inside
our
sql
or
in
our
case
inside
our
pl
sql
from
it
will
send
the
data
as
the
data
logs
as
json
objects,
which
mean
that
they
will
have
a
big
json,
plus
some
data.
In
our
case,
it
will
contain
times
a
timestamp
and
attack,
plus
the
json
object.
With
everything
in
the
main
table.
A
We
will
sort
the
raw
data
into
one
main
table,
meaning
it's
the
table
option
in
the
in
the
plugin
and
we
decided
to
process
with
using
pl
sql
to
process
and
split
the
data
into
many
tables,
one
table
or
just
split
the
data
for
the
table
or
using
store
them
in
the
same
table.
That's
not
an
issue,
so
we
can
use
any
object
inside
any
field
inside
the
json
object
to
decide
when
to
store
the
data
where
to
store
it
split
it
by
using
some
special
handle
or
anything
like
that.
A
So
you
will
take
the
decision
based
on
the
json
object,
but
what
is
json
b
or
json
on
pos
sql?
Well,
json
is
just
javascript
object,
notation
in
in
in
our
pos
sql.
There's
a
secure
technical
report
that
you
can
check
out
on
the
slide
right
here:
template
which
is
the
implementation
of
json
on
post
sql.
A
The
idea
is,
you
can
have
json
or
json
b,
but
we
use
json
b
because
you
support
indexing,
which
means
that
it's
easy
to
create
the
data
and
keep
it
indexed,
because
we
know
that
index
create
faster
queries
right
and
it's
easy
to
query.
Json
is
just
you
can
select
data
and
a
field
inside
in
this
case
date
from
the
table
and
you
will
get
the
data.
A
A
You
can
put
it
on
the
configuration,
but
we
strongly
recommend
use
pgpas
file
because
it's
the
best
option
and
secure
for
in
this
case
the
database
and
a
table
the
table
will
be
created
if
it's
not
exist
on
the
plugin,
when
it's
a
startup
and
connects
to
pulse
sql,
so
you
don't
need
to
create
it
and
we
support
codextv
using
codecrossdb
equal
true.
This
is
a
codecrdb
support,
postsql
protocol,
but
some
queries:
don't
they
don't
support
some
functions
and
some
queries
may
change
so
we
created
some
special
option
for
it.
A
A
A
This
is
what's
using
the
tail
plugin
into
some
processing,
apache
locks,
but
gears
of
logs
we
needed
to
analyze
three
years
of
apache
logs,
which
is
a
really
really
big
tax,
because
we
have
around
one
terabyte
of
data
and
we
needed
to
process
it
in
just
two
weeks.
So
for
this
we
decided
to
use
the
tail
plugin,
which
needed
some
updated,
and
we
added
some
options
into
the
documentation
to
process
the
pos
sql,
the
apache
locks
into
post
sql.
A
We
use
the
pl
sql
language,
the
processing,
speed
data
into
different
tables
which,
because
otherwise,
we
will
have
billions
of
rows
inside
just
one
table,
but
in
our
case
we
use
partition
tables
per
month
which
allow
us
to
create
proper
indexes
and
query.
Just
one
amount-
and
this
query
for
month
took
less
than
one
second,
which
is
really
really
amazing,
because
in
other
databases
store
logs,
this
will
obviously
create
more
than
one
second,
because
we
have
a
lot
of
data.
A
So,
let's
deploy
inside
kubernetes
our
fluent
bit
plus
sql
output
plane,
we
will
provide
some
url
on
github,
which
is
open
source
and
you
will
be
able
to
use
customize
to
deploy
inside.
So
let's
have
some
fun
in
the
final
configuration
you
will
be
able
to
see
the
simple
fluid
bit
configuration
with
the
include
statement
to
add
the
conf
files
into
the
front
bit
configuration
and
in
the
file
pause.
A
Sql
customization
is
just
a
conflict
map
generator
that
will
use
the
merge
behavior
to
add
output,
postgresql.com
file
into
the
fluent
bit
config
map
and
in
the
output
postsql.conf
file.
You
will
be
able
to
see
the
host,
which
is
out
of
kubernetes
deployment
using
a
dummy
password
and
the
user
fluent
bit
for
the
database
fluent
bit
and
tablespoon
bit.
That
will
match
everything
to
the
output
plugin.
A
Okay,
that
was
easy,
but
let's
use
some
pl
sql
language,
because
we
want
all
the
data
in
a
separate
table
because
obviously
we
get
logs
from
many
places.
So,
let's
partition
that
table
because
maybe
in
the
future
we
will
need
partition
by
month,
year
weeks
or
days
and
use
some
condition
to
fulfill
the
empty
fields.
A
A
A
A
A
A
So
for
the
record,
we
will
let
the
functions
the
pl
sql
function
on
the
slide,
so
you
can
take
take
them
and
use
them
later
as
an
example.
But
you
can
see
here
that
we
can
query
the
data
using
select
distance
based
on
the
container
field
in
the
kubernetes
log,
and
we
will
see
that
we
have
five
records,
but
there's
one
with
an
empty
field.
Maybe
we
need
to
look
into
this
later
and
see
that
some
objects
will
not
come
with
the
container
name
or
something
like
that.
A
But
appeal
sql
function
can
be
more
complicated
because
sometimes
we
may
want
to
send
the
data
into
more
than
one
just
table.
Okay,
so
let's
see
a
function
that
can
speed
the
data
depending
on
the
tag.
If
the
input
doesn't
match
attack,
send
it
to
a
default
table,
so
we
may
be
ending
having
three
tables,
so
the
following
function
will
show
you
how
to
split
between
tags
used
with
apache
and
tags
or
the
data
that
comes
with
kubernetes
inside.
A
A
Obviously,
we
need
to
group
by
a
field
in
this
case
container
image
and
see
that
we
have
a
lot
of
container,
but
maybe
we
want
to
know
what,
if
we
split
the
containers
and
count
them
by
host,
that's
easy,
just
add,
cost
and
group
by
host,
then
we
have
all
the
containers
split
them
using
host
so
yeah.
This
is
a
nice
and
this
is
cool.
But
let's
see
something
else.
A
In
this
case,
we
are
going
to
select
all
the
fields
from
the
kubernetes
logs,
where
the
container
image
is
fluent
bit.
Okay,
there's
a
lot
of
data.
Maybe
that's
not
so
useful,
you
know
well,
but
you
can
see
it's
sql.
So
let's
do
something
different.
What
about
distinct,
host
and
count
all
the
fluent
bit
image
running
on
different
hosts.
A
A
A
A
A
A
A
A
A
A
Okay,
you
will
see
that
there's
a
lot
of
data
that
didn't
fit
in
our
functions,
so
it
was
stored
on
the
default
table.
That
was
the
sense
of
having
a
default
table
because
you
may
not
want
to
lose
any
data
at
any
moment
of
your
process.
So
let's
see
some
data,
for
example
data
for
the
cpu.
As
you
can
see,
it's
just
like
everything
on.
A
Fluid
as
you'll
see,
we
can
do
a
lot
of
stuff
with
this
plugin
and
once
the
data
is
on
pulse
sql,
but
there's
some
ideas.
You
may
want
to
experiment
right
now
like
graph
the
data
using
rafana,
which
is
a
very
useful
tool
to
create
graphs
split
the
data
per
weeks,
not
just
month
or
maybe
per
year.
If
you
have
that
amount
of
time,
maybe
a
day
would
be
useful
to
start
testing
stuff,
create
some
script
to
out
for
automatics
reports
per
month
or
per
year.
That's
also
some
idea
you
can
get
from
here.