►
From YouTube: Keynote: Scaling Observability Pipelines in the Enterprise - Anurag Gupta & Eduardo Silva, Calyptia
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Keynote: Scaling Observability Pipelines in the Enterprise- Anurag Gupta & Eduardo Silva, Calyptia
Our infrastructure is continuously growing, generating more data. Extracting value from that growing data and scaling up analysis is a constant pain. Modern tools need to solve both critical performance tasks like data collection as well as pre-processing enrichment and delivery.
Fluentd and Fluent Bit CNCF projects are the de facto Logging and compatible Metrics solution for any cloud environment. In this session you will learn about our journey on scaling up data processing for financial institutions, performance improvements and how Calyptia, the company founded by Fluent creators and maintainers, is taking the Fluent ecosystem to the next enterprise level through cloud services and free tools for our community.
A
So
with
that,
you
know,
hey
everyone,
my
name
again
is
honorable
part
of
calypsia
and
I'm
joined
here
with
ed
eduardo.
We
are
you
know
the
creators
and
maintainers
of
these
projects
and
we
founded
a
company
calyptia
around
these
projects
and
we
really
concentrate
on
what
we
call
first
mile
observability,
where
you're
collecting
all
that
data.
You
are
intrinsically
doing
things
like
parsing,
enriching,
redacting
and
routing
it
to
end
destinations
and
with
that
journey,
there's
a
ton
of
stuff
that
comes
into
play.
As
we
start
evolving
into
microservices.
A
We
start
evolving
into
high
performance
data
streams.
We
start
evolving
into
data
that
just
doesn't
necessarily
look
like
anything.
We've
we've
dealt
with
before
and
so
in
this
session,
we're
going
to
talk
a
little
bit
about
that
problem,
inherently
what
it
is,
how
to
go
and
solve
it,
some
workarounds
you
can
use
today
and
of
course,
some
places
where
colyptia
can
come
in
and
help
with
with
what
we've
built.
So
with
that,
I'm
going
to
pass
it
off
to
edwardo
to
talk
a
little
bit
about
this.
B
So,
as
I
said,
we
started
this
company
called
kalitia
to
solve
all
these
problems
and
challenges
that,
from
the
open
source
perspective
yeah,
we
can
address
some
of
them,
but
when
you
go
to
the
enterprises
even
more
problems
and
to
understand
the
problems,
we
need
to
understand
a
once
a
pain
and
the
other
how
we
are
tackling
that
problem
at
a
low
level.
Technically
speaking
from
the
implementation
and
scalability
is
a
continuous
problem,
we
always
have
more
data,
more
data,
more
data
on.
B
How
do
we
optimize
quick
question
is
like
hey,
they
have
more
nodes
more
clusters,
but
sometimes
that
adds
other
sites
problem.
Now,
if
we
talk
how
things
works,
it's
like
we
were
talking
about.
We
got
data
which
is
called
input
right.
We
process
data
when
this
set
the
data
out,
but
this
is
more
complex
than
that
internally,
dealing
with
an
input
means
to
open
files,
perform
a
lot
of
cisco,
create
sockets
listeners,
allocate
reallocate
memory
free
out
that
memory
and
for
metrics
also
is
the
same
thing.
B
So
it's
a
pretty
complex
and
tough
job
after
that,
if
you're
in
login,
we
need
to
purchase
information
filter
it
out,
serialize
it
right,
because
you're
not
going
to
deal
with
just
text
you're
going
to
have
binary
data.
You
have
to
deal
with
buffers
and,
as
I
said,
we
have
physics
limits.
Right
memory
is
not
unlimited
and
then,
when
you
proceed
all
this
and
filter
this
data
out,
you
want
to
route,
it
create
schedulers,
and
if
this
go
wrong,
you
have
to
retry.
B
So
all
of
these
complexities,
so
you
say:
okay,
if
I
want
to
scale
up
all
my
data
management
right.
How
do
we
solve
this
in
the
low
level
part?
Because
I
cannot
get
rid
of
cisco's?
I
cannot
get
rid
of
memory
handling.
I
have
to
do
that.
That
is
a
continuous
need,
but
also
we
had
to
deal
with
networking,
because,
if
you're
going
to
send
the
data
out,
we
have
this
our
best
friends
called
dns
servers
right
then
it's
always
complex,
dns,
lookups
connect
and
also
you
want
security
right.
We
want.
B
We
don't
want
to
transfer
the
data
in
a
plain
way.
We
had
to
do
tls
hand,
checks
round
trips
certificates
format.
The
data,
because
also
the
destination,
will
not
understand
our
binary
format
that
but
elastic
will
have
its
own
json
format,
splunk
its
own
way
to
get
that
data
and
that
transformation
or
data
also
takes
time
right.
We
can
scale
up
the
services
with
some
threads
some
mechanisms,
but
at
some
point
you
need
something
else
and
when
delivering
things
also
for
the
network
we
have
to
validate,
and
that
also
takes
times.
B
So,
if
you
think
how
to
how
do
we
scale
up
this,
I
will
you
tell
the
users,
the
user
will
tell
you
hey.
I
got
this
problem.
I
cannot
scale
up
a
half
terabytes
of
data
as
probably
maintainers
and
I'm
putting
my
oss
hat.
I
cannot
tell
hey,
take
a
look
at
the
syslog.
Take
a
look
at
the
memory
buffers
that
one
scale
right
as
a
solution
in
a
business.
B
They
need
to
have
something
differently,
ready
to
go,
that
kind
of
that
he
can
approach
and
tackle
the
problem
in
a
in
a
different
way.
So
as
oss
maintainers,
you
got
this
problem
right.
That
goes
from
left
to
right.
From
data
collection,
standpoint,
processing,
filtering
buffering
with
default
final
destination,
which
is
your
database
or
your
cloud
service,
and
this
is
a
as
a
company.
We
come
up
with
this
concept
of
the
first
mile
observability.
B
You
have
a
huge
pipeline
right,
but
we
are
experts
on
the
first
fraction
of
it
so,
and
we
said,
let's
try
to
provide
some
solution
on
that
part
right.
Let's
stop
talking
about
cisco,
cisco,
let's
stop
talking
about
memory
allocations
and
let's
bring
things
to
a
higher
level
for
the
enterprise
where
you
likely,
where
you
have
thousands
of
servers
right,
you
want
to
have
something
that
you
can
put
in
your
architecture
and
work
and
scale
right
away.
A
And
so
our
expertise
here
is
really
in
a
lot
of
the
cloud
native
pieces
and
we
thought
how
would
we
go
architect,
something
like
this
from
an
open
source?
How
have
folks
gone
and
solved
this
today
and
the
first
is
you
have
all
of
this
data
flowing
and
you
might
need
multiple
processes,
instances
to
go
in
and
take
care
of
that.
So
you
might
need
multiple
containers
right
if
someone's
running
in
kubernetes,
you
might
want
five
or
six
different
containers
that
are
reading,
that
data
processing
it
and
sending
it.
A
You
might
want
automatic
load
and
traffic
balancing.
We
have
this
great
idea
of
ingress
routes
within
kubernetes.
It
can
take
it
data
load
balance
it
automatically
for
you
and
and
it's
something
powerful
kind
of
comes
out
of
the
box
with
with
many
kubernetes
distributions,
and
then
you
want
simple
scalability
right.
The
idea
with
kubernetes
is
you
can
treat
these
workloads
somewhat
as
ephemeral
and
you
can
scale
them
up.
You
can
scale
them
down
and,
and
you
can
go
and
configure
it
say
from
a
remote
or
api
type
location.
A
So
this
is
where
we
said.
How
can
we
solve
these
using?
What's
available
today
and
for
us
we
thought
of
the
operator,
we
have
the
fluid
bit
operator,
there's
a
session
on
that
later
today.
There's
ways
that
you
can
do
this
just
running
this
on
top
of
kubernetes
and
then
as
calyptia.
We
wanted
to
package
and
bundle
all
of
that
as
well,
so
we
created
a
new
product
called
klyptia
axle
and
essentially
takes
the
fluent
as
a
service,
but
allows
you
to
deploy
it
within
your
data
center.
Have
it
remotely
managed
and
configurable.
A
A
We
can
just
increase,
say
the
replica
size
and
then
we
can
go
ahead
and
save
that
and
just
like,
we
would
expect
with
any
of
these
cloud
native
architectures,
how
we
go
and
update
and
how
controllers
work
where
it
goes,
looks
at
the
new
configuration
and
starts
to
apply
it.
We
can
come
back
and
see.
Okay,
we
have
you,
know
a
brand
new
replica
size
that
matches
what
we
specified.
So
this
is
our
our
way
of
saying:
hey,
there's
all
these
problems
with
scalability.
A
We
can
solve
them
using
a
lot
of
the
cloud
native
architecture
that
exists
out
there.
It's
a
lot
of
manual
putting
together,
and
if
you
want
something
more
turnkey,
we
have
a
product
that
can
do
that
and
of
course
we're
looking
for
folks
who
are
interested
in
trying
that
out
and
building
that
vision
with
us.
So
yeah.
A
Thank
you
all
for
for
the
time
we're
going
to
go
ahead
and
break
for
the
coffee
break
now,
and
then
we
will
come
back
to
this
session
here
at
about
10
15,
10
20.,
so
we'll
get
that
30
minutes
of
coffee
and
for
us,
so
yeah
we'll
be
up
here.
If
there's
any
questions,
otherwise
we'll
also
be
in
the
slack
where
folks
are
out
asking
questions
and
comments,
but
other
other
than
that
yeah.
Please
have
a
good
coffee
break
and
see
you
soon.