►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Security Logging Use Cases: Building an Open-Source SIEM- Jonah Kowall, Logz.io
Understanding threats is the goal of a SIEM, which collects, enriches, and correlates events and threats. Learn how to build an open-source SIEM with Fluentd and Fluent Bit. SIEMs primary challenge is data collection and scale. Security infrastructure generates diverse data. We will cover data sources along with how to parse and security signals. We will provide real-world examples of how these data collection systems are used to bring together security data into an open-source SIEM. Learn how extracting metrics from logs with Fluentd can provide additional data to understand your organization’s security posture. The EFK Stack is very popular for log analytics. This includes the life cycle of collection, indexing, and storing them. Log data is valuable, but use cases for logging are operational for observability and debugging. The security world typically uses other tools, but building it on top of the same logging stack is efficient. SIEM takes a centralized approach to collection, enrichment, and analysis. As we know in today’s environments, we must ultimately federate this work to the edge to reduce data volumes and take action faster. While this is not something that SIEMs do today, it’s something that the future SIEM and technologies like Fluentd will provide.
A
Good
for
me
to
start
cool.
Thank
you
thanks.
Everyone
for
joining
me
today.
My
first
fluentcon
happy
to
be
here
talking
about
a
topic
that
I've
spent
a
lot
of
time
with
in
the
last
couple
of
years,
but
it's
not
really
my
background,
but
I
wanted
to
share
with
you
some
of
the
findings
and
work
that
I've
seen
happening
out
there
in
insecurity
and
how
fluenty
fluent
bit
can
be
used
to
build
use
cases
around
security
and
sim.
A
My
background
more
of
an
operations
type
person
a
practitioner
for
a
long
time,
I've
run
various
types
of
organizations
I
switched
in
my
career
and
became
an
analyst
to
gartner
for
a
few
years
covering
what's
called
observability
today
back
then
it
was
monitoring.
I.T
operations,
I'm
currently
the
cto
at
logs
io
we're
an
open
source
sas
company.
A
A
So
today,
I'm
going
to
talk
about
all
the
great
food
that
we're
going
to
eat
in
la
over
the
next
few
days.
I'm
sure
all
of
you
have
very
big
agendas
of
asian
food
and
chinese
food
and,
of
course,
mexican
food
that
we're
all
going
to
be
eating
lots
of
tacos
on
my
plan,
because
in
the
east
coast
we
don't
understand
mexican
food.
So
it's
great
to
be
back
here
where
the
best
mexican
food
is.
A
One
could
call
these
logs
events
depends
on
what
you
want
to
call
them,
but
generally
they're
the
same
thing.
You
want
to
store
them
centrally,
analyze
them
alert
people
that
bad
things
are
happening
and
then
folks
investigate
them.
So
the
tool
can
help
people
do
threat,
hunting,
forensics
other
types
of
investigation,
some
sims,
especially
the
newer
style.
We
store
raw
data.
We
store
all
the
logs
in
information.
Older
systems
would
correlate
it
and
store
an
event
like
this
was
something
bad
that
happened,
but
I'm
not
going
to
keep
all
the
data
that
goes
into
it.
A
So
sim
doesn't
necessarily
mean
you're,
storing
everything
all
the
time
and
we're
always
trying
to
figure
out
ways
to
make
this
more
efficient,
because
obviously
it's
super
expensive
to
store
all
of
this
security
data.
So
there's
a
lot
of
challenges
in
the
market,
so
there's
a
lot
of
different
types
of
food.
There's
a
lot
of
use
cases.
I've
outlined
many
of
the
ones
that
I
hear
about
regularly
and
then
a
lot
of
users.
These
are
different
users.
I
know
most
folks
here
live
in
devops
and
operations,
but
totally
different
users.
A
A
We
have
about
two
customers,
one
of
which
is
a
really
large
company
that
have
devsecops
going
so
by
and
large.
This
is
not
a
thing
that
people
do,
even
though
it
would
be
way
more
efficient
for
them
to
store
this
data
once
and
use
it
for
two
use
cases.
It's
not
really
what
happens
because
the
buyers
are
different,
so,
like
these
users
down
here,
they
have
a
separate
budget.
A
They
have
a
separate
set
of
needs,
they
want
things
to
be
in
a
different
language
and
so
the
market's
generally
quite
different
from
each
other
today
and
the
things
that
they're
doing
are
also
quite
different.
When
you
look
at
some
of
these
and
I'll
just
pick
out
one
of
the
use
cases
to
give
you
an
example,
let's
say:
threat
hunting
one
can
say
how
is
threat
hunting
any
different
than
doing
analysis
of
an
incident
or
an
outage.
A
The
issue
is
that
when
I
do
threat
hunting,
I
want
other
information
to
be
brought
into
the
tool
that
gives
me
more
context
from
a
security
perspective.
What
was
the
attacker's
ip
address?
What
network
did
they
come
from?
What's
the
as
number
do
I
know
anything
else
about
when
that
same
attacker
did
anything
to
my
network.
A
A
A
Most
of
them
are
on-prem
legacy
appliances.
This
is
the
market,
but
yet
people
spend
three
billion
dollars
a
year
trying
to
solve
this
problem.
It's
crazy
how
old
it
is
in
terms
of
the
technology,
proprietary
agents,
no
open
source,
really
to
speak
aside
from
one
company,
maybe
one
and
a
half
that
do
a
little
bit
of
open
source.
A
A
Elastic's,
obviously
gone
a
proprietary
route
over
the
last
year.
They
don't
really
have
an
open
source
option.
Log
stash
is
commonly
used,
and
so
we
always
recommend
people
use
fluency
and
fluent
bit.
For
these
types
of
use,
cases
there's
also
a
new
elastic
search
replacement,
called
open
search.
A
lot
of
people
are
starting
to
use
that
if
they
want
to
keep
things
in
the
open
source,
community
there's
also
a
project
I'll
talk
about
called
the
hive.
A
That's
a
really
cool
tool
for
helping
do
incident
management,
workflow
and
some
enrichment
that
I'll
talk
about,
but
it's
not
really
a
sim.
So
you
have
to
do
a
lot.
If
you
want
to
build
this
yourself
and
I'll,
give
you
some
ideas
of
how
you
could
do
something
like
that
so
similar
to
a
lot
of
the
discussions
today
about
enrichment,
some
really
interesting
talks
and
some
great
insights
that
I
picked
up.
A
Most
of
the
data
types
for
sim
are
still
syslog.
You
know
we're
talking
udp
1980s
technology
here
and
then
there's
log
files,
but
there's
a
lot
of
syslog
out
there
still,
especially
from
network
devices,
and
it
it's
still
a
big
challenge.
So
this
this
stuff
is
is
very
old
technology
in
general.
What
people
want
you
to
support
in
your
system
and
then
there's
threat,
there's
a
metadata
which
I'll
talk
about
so
there's
insecurity,
threat
feeds
are
critical.
A
There's
all
these
companies
that
create
feeds
that
say
these
domain
names,
these
ip
addresses
these
subnets
or
as
numbers
depending
on
how
you
want
to
look
at
it,
constitute
botnets
and
people
that
are
doing
attacks
on
the
networks
and
with
sim.
You
want
to
take
this
data
and
stitch
it
in
real
time.
So
threat
feeds
are
really
critical
to
sim
because
they
provide
great
context
and
then
there's
a
lot
of
lookup
lists,
whether
it's
go
ip
that
one's
easy.
A
A
So
you
want
to
make
sure
that
hosts
are
not
compromised.
You
have
to
understand
the
integrity
of
the
host.
These
are
all
critical
things
in
compliance.
They're
critical
in
security,
so
that
you
understand
when
changes
occur
and
then
there's
lots
of
cloud
data
sources,
everything
from
our
typical
cloud
providers,
the
the
big
three
listed
up
there
to
really
specific
ones
like
I
want
to
get
my
sales
force
data
and
understand
how
that's
being
changed.
I
want
to
understand
my
office
365.
A
A
So
fluent
d
and
fluent
bit
obviously
have
a
huge
amount
of
plug-ins
and
technologies
to
make
this
easier.
There
are
sometimes
some
weird
ones
where
we
have
to
still
go
to
logstash
because
they
have
plugins
that
do
things
like
office
365,
where
there
aren't
good
options
in
the
open
source
world
today,
unfortunately,.
A
And
many
of
these
get
pulled
via
api.
So
if
you
want
to
get
audit
data
from
salesforce,
you've
got
a
hidden
api
grab
the
data
it
gets.
It
gets
a
bit
tricky
some
of
the
newer
systems
out
there
and
we
do
this
on
our
platform.
You
can
subscribe
to
a
stream
and
I
know
aws
pushes
this
a
lot.
It's
way
cheaper
for
you
to
subscribe
to
a
topic
and
get
information
than
querying
an
api
over
and
over
again
where
they
charge
you
every
time
you
hit
it.
A
So
stream
processing
is
very
new
in
sim,
almost
no
one's
doing
it
today,
but
I
think
that
that's
the
way
of
the
future
because
of
the
volume
of
data
and
the
fact
that,
when
we're
polling
apis,
it's
extremely
inefficient
and
expensive
and
the
cloud
providers
are
making
us
pay
for
that.
So
anytime
we
can
stream
data
in
it's
it's
way
more
efficient,
but
it's
a
harder
problem
to
solve
today.
A
So
the
next
thing
is
parsing
the
data
looking
at
all
the
ingredients
that
go
into
our
guacamole
over
here
on
the
right,
all
the
important
stuff
you
have
to
understand,
schemas.
So
a
big
challenge
in
sim
is
on,
and
in
logging
in
general,
is
understanding
what
a
message
is.
If
I
get
the
same
type
of
message
from
a
firewall,
a
switch
and
a
host,
how
do
I
know
that
this
is
the
username
that's
being
authenticated?
A
The
answer
is
it's
completely
inconsistent
and
there's
been
a
lot
of
attempts
at
creating
schemas
ecs,
which
is
the
elastic
common
schema,
is,
is
open
source.
It's
one
way
that
people
try
to
normalize
the
data
open
telemetry
we
just
incorporated-
and
I
do
some
work
on
hotel
as
well.
We
just
incorporated
flexible
schemas
into
otel.
A
That's
going
to
be
a
way
to
do
the
same
type
of
thing,
normalize,
the
data
so
that
I
can
say
this
is
a
user,
that's
authenticating
and
we
can
start
to
understand
different
types
of
telemetry
without
necessarily
needing
to
build
custom
parsers
in
security,
there's
also
a
ton
of
frameworks
for
compliance.
Here's
a
few
examples.
These
will
classify
different
types
of
attacks.
A
They
will
tell
me
about
compliance
needs
that
my
organization
have
and
there's
many
other
ones
that
are
specific
to
an
industry
like
pci,
for
payments
hipaa
for
healthcare,
and
it
goes
on
and
on
and
on
when
customers
come
to
us
and
they're
under
a
compliance
requirement.
They
want
out
of
the
box
reporting
they
want
best
practices
in
the
sim.
A
So
I
mentioned
enrichment,
so
the
most
common
one
that
everyone
here
deals
with
with
fluent
be
fluent
bit
and
fluent
d
is
enrichment.
I
want
container
data,
I
want
service
data
and
something
that
is
definitely
a
big
focus
for
open
telemetry,
we're
just
starting
to
make
recommendations
of
using
ebpf
and
open
telemetry
to
provide
enrichment.
A
A
A
That
is
in
question
about
the
user
in
question,
and
these
all
increase
the
size
of
the
data,
but
they
make
it
way
more
valuable,
similar
to
spices
or
herbs.
As
the
picture
shows
here
we
put
herbs
into
our
food
because
it
it
gives
us
a
great
set
of
flavor.
It
gives
us
way
more
depth
than
what
we're
eating,
and
this
is
the
same
type
of
thing
so
very
similar.
A
The
other
way,
the
other
challenge
with
enrichment,
is
some
of
this
data.
You
want
to
centralize.
So
when
the
data
comes
in,
I
want
to
enrich
it
as
it's
before
it's
stored
right
on
the
edge
of
that,
but
in
many
other
use
cases
it's
better
to
do
this
at
the
edge,
because
it's
more
scalable
so,
for
example,
based
on
the
scale
of
the
environment
or
how
much
data
you're
bringing
in
the
enrichment
takes
a
lot
of
cpu.
A
A
A
Maybe
I
want
to
keep
it
when
one
user
authenticates
several
times
fails
and
then
authenticates
that
could
show.
Maybe
they
figured
out
something
with
a
password
or
they
figured
out
some
other
type
of
two-factor
authentication.
So
we
have
to
think
about
these
use
cases.
If
we
do
that
on
the
edge,
we
can
actually
reduce
the
data
volumes
that
we're
sending
to
a
central
place
without
sending
data
and
then
discarding
it
centrally.
A
So
enrichment,
I
think,
is
going
to
be
a
key
for
fluency
and
fluent
bit
in
the
future
and
where
differentiation
can
come
from
in
open
source.
It's
I
I
think
it's
an
interesting
area
for
sure,
especially
with
cost
reduction,
cost
control.
We
all
deal
with
these
challenges
so
a
little
bit
about
ingestion
and
storing,
and
I
had
to
pick
like
someone
eating
a
taco
for
ingestion.
A
So
so
the
filtering
is
important.
The
pipeline
is
really
critical.
A
lot
of
folks
us
included,
use
a
lot
of
kafka
to
help
manage
pressure
on
the
back
end.
To
do
various
things
in
our
pipeline.
We
use
it
for
building
machine
learning
models.
We
use
it
for
doing
real-time,
alerting
to
circumvent
the
fact
that
we
have
to
store
it
and
then
analyze
it.
We
try
to
analyze
things
in
the
stream
as
much
as
possible
and
and
then
you
need
a
lot
of
custom
code.
A
If
you
want
to
build
this
yourself,
how
do
I
stitch
the
data
in?
How
do
I
archive
data
restore
data
because
in
compliance
some
of
this
data,
like
pci,
you
have
to
keep
seven
years
of
transactional
data?
That's
expensive.
I
want
to
put
that
on
glacier,
keep
it
really
inexpensive,
but
I
need
to
be
able
to
reconstitute
it
when
I
have
to
do
an
investigation.
A
A
But
hotel
is
very
early
in
logging
today,
so
we're
going
to
keep
working
on
it
and,
and
hopefully
it'll
improve
over
the
next
year
and
beyond
so
storage.
As
I
mentioned,
some
most
organizations
keep
90
days
of
log
data,
that's
kind
of
the
standard.
If,
if
you
were
to
ask
me
of
those
customers,
what
do
they
keep
it's
usually
90
days?
They
want
to
be
index
searchable
available.
A
That
doesn't
mean
they're
going
to
search
it
all
the
time
they
mostly
are
searching
five
days
of
data
or
less,
but
they
want
90
days
in
case
it
could
be
slower.
It
could
take
a
while
to
return,
and
so
that's
the
challenge.
It
gets
really
bloated.
It's
a
lot
of
data,
someone
sending
you
five
terabytes
of
data
a
day
for
90
days,
you're
talking
about
a
huge
amount
of
storage,
a
lot
of
cost.
A
A
A
These
are
where
we've
actually
been
investing
a
lot
working
with
amazon
on
ultra
warm
capabilities
in
the
open
source.
So
this
lets
you
store
data
in
low
cost,
but
yet
keep
the
indices
in
hot
storage.
So
you
can
do
some
basic
searching
and
it's
a
big
challenge,
but
I
think
open
source
tooling
is
going
to
get
a
lot
better
here,
because
everyone
complains
about
cost.
It's
the
big
challenge
when
you're
dealing
with
logs.
A
And
today
this
is
usually
done
centrally,
but
I
think
it
needs
to
be
done
in
a
more
distributed
manner.
I
think
we
have
to
start
thinking
about
correlation
at
the
edge,
because
it
is
a
big
challenge
and
everyone
does
it
centrally
today
and
I
don't
think
it's
very
efficient
and
it
can
also
take
a
little
while
when
you
think
about
you
being
under
attack.
If
the
delay
is
one
or
two
minutes
before
you
know
about
it,
they
could
already
have
your
information.
They
could
have
that
crit
those
critical
assets
already
completely
controlled.
A
A
So
most
of
the
correlation
today
is
rule-based
and
you
can
go
in
the
exhibit
hall
tomorrow
and
talk
to
all
these
vendors
that
tell
you
they
have
some
magic,
ai
correlation
system
and
they
may
have
a
little
bit
of
more
advanced
correlation.
But
there's
always
a
rule
set
underneath
and
the
rule
set
is
generally
regular
expressions
or
some
type
of
parsing
of
strings.
To
figure
this
out
in
opens
open
source.
There
is
a
an
alerting
system,
that's
part
of
open
distro,
and
it
does
some
pretty
pretty
nice
anomaly
detection.
A
Some
interesting
padding
pattern
matching.
So,
if
you're
looking
at
building
something
in
open
source,
it's
a
pretty
cool
way
to
go,
but
it's
still
missing
a
lot
of
things
like
multi-query
correlation
we've
actually
built
some
of
this
in
our
product,
but
I
don't
see
anything
in
open
source
that
that
does
a
lot
of
the
more
complex
alerting
today.
A
A
So
how
does
the
user
use
the
data?
I
do
want
to
touch
on
a
couple
of
use
cases
here.
So
how
do
you
iden?
How
do
you
simplify
the
problem,
which
is
billions
of
log
messages
down
to
a
number
log
messages
or
events
down
to
hundreds
of
alerts
down
to
dozens
of
incidents?
So
that's
kind
of
the
goal
is
we
want
to
simplify
the
problem,
make
it
digestible
to
the
security
operations
team
and
a
good
example
here,
which
is
what
we
call
user
behavior
anomaly
detection
over
on
the
right,
so
we
have
all
the
authentications.
A
A
A
So
the
idea
here-
and
this
is
zooming
out
so
there's
sim
and
then
on
the
bottom
is
soar.
That's
basically
the
opera.
That's
the
orchestration
and
automation
platform
for
security,
so
you'll
see
companies
today,
many
vendors
that
build
sims,
they're,
also
building
or
buying
stores,
to
try
to
own
not
only
the
the
inve,
the
incident
investigation,
but
the
automation
to
try
to
remediate
the
incident
and
that's
what
soar
is
about.
So
the
idea
is
the
sim
is
the
intelligence,
the
brain
it
takes.
A
All
the
data
distills
it
down,
creates
an
incident,
opens
a
ticket
or
an
incident
in
the
tool,
and
they
can.
They
often
have
some
basic
incident
management
in
the
tool
or
you
could
use
an
external
tool,
maybe
a
service
now
or
jira
or
security
specific
tools.
There
are
many
of
them
out
there
that
do
ticketing
and
workflow
like
the
hive,
which
does
that
and
then
you
can
create
automation.
So
a
lot
of
our
more
advanced
customers.
They
want
to
change
a
firewall
rule
lock
down
a
port,
do
something
on
an
end,
user's
machine.
A
It's
not
easy
to
do.
It
takes
a
long
time.
How
do
you
scale
it
and
manage
it?
Do
all
the
alerting
everything
else
in
between
never
easy
or
you
could
try
to
use
some
open
source
technologies,
and
do
it
yourself
lots
of
good
ways
to
ingest
data,
lots
of
ways
that
you
can
put
stream
processing
in
place,
trying
to
aggregate
it
so
most
folks
that
are
using
open
search,
elastic
search,
usually
run
kafka
if
they're
at
scale.
So
all
of
this
kind
of
comes
together,
it's
really
exciting
to
see
that
kafka's
simplifying
I'm
sure.
A
All
of
you
hate
zookeeper
and
it's
finally,
almost
going
away
from
kafka.
They
just
released
non-production
version
of
kafka
3
that
has
an
ozu
keeper.
I
really
want
to
upgrade
our
stuff
to
it
because
zookeeper's
a
bit
archaic
to
say
the
least.
So
I
think
kafka
is
making
some
awesome
improvements
thanks
to
the
to
the
community
and
the
folks
at
who
is
it
confluent
right,
yeah,
confluent,.