►
Description
The session will focus on design choices and tradeoffs we made to reach the scale of ingesting 1mn events/s to an open source observability tool (SigNoz) Key points to be covered: - Horizontally scaling OpenTelemetry collector in Kubernetes - Designing hierarchy of OpenTelemetry collectors (cluster level/node level/sidecar level) - Extracting metrics from traces using spans2metrics processor - Getting application metrics and infrastructure metrics in the same UI - Why distributed columnar database is efficient for storing telemetry data
A
A
A
A
So,
for
example,
if
you're
trying
to
understand
how
a
person
is
by
trying
to
understand
what
the
temperature
is
right.
So,
if
you
think
about
it,
the
temperature
of
a
person
is
indication
of
something
wrong
which
is
going
on
with
the
person
right
and
hence
that
is
a
a
reflection
of
what's
being
wrong
and
we
use
tools
like
thermometer
to
understand
why
the
temperature
is
high
and
to
measure
the
temperature
and
then
that
is
used
by
doctors
to
infer
what's
come
so
observability
for
computer
systems
and
software
systems
are
is
very
similar.
A
So
that's
probably
what
auxiliary
is
now.
Why
is
it
work
and
more
important
to
know
right?
So
one
of
the
things
which
has
happened
in
the
recent,
maybe
last
two
three
years,
is
that
more
and
more
people
are
shifting
to
microservices
earlier
it
was
the
code
bases
were
primary
monolithium
single
big
code
which
is
running,
and
then
in
this
case
you
can
easily
print
out
the
locks
and
try
to
understand
what's
going
wrong
right.
A
But
if
you
have
a
micro
services
architecture,
you
don't
have
a
single
system
on
which
you
can
print
logs.
There
are
different
components
of
the
system
and
each
components
behave
independently
and
then
to
figure
out
what's
wrong
with
the
whole
system.
You're,
not
sure.
Where
do
you
like,
which
part
do
you
measure
right?
A
So,
as
this
graphic
shows
in
the
monolith,
you
can
just
put
a
thermometer
here
and
then
the
sense
of
what's
wrong.
While
here
there
are
like
different
subsystems-
and
then
you're
not
sure
like
where
to
put
the
thermometer
to
get
a
sense
of
what's
wrong
right
and
hence
we
believe
that
as
more
and
more
systems
are
moving
to
micro
services,
objective
systems
will
become
more
and
more
important
right.
And
that's
like
that's
one
of
the
reasons
why
we
started
working
on
students.
A
So
if
you
think
about
it,
there
are
like
primary
three
pillars
of
observation:
one
is
matrix,
which
is
the
aggregates
of
time.
Series
data
points
which
you
see,
which
shows
up,
shows
what
how
your
systems
are
perform.
For
example,
if
you
want
to
see
latencies,
so
it's
a
time
series
a
graph
of
how
the
different
latencies
are
are
there
and
you
get
a
sense
that
okay
in
in
this
time
something
has
some
latency
has
gone
up
or
called
down.
A
A
A
A
So,
for
example,
this
this
is
a
trace.
This
shows
you
complete,
stack
trash
trace
of
how
a
particular
request
which
was
made
on
this
api
went
through
like
a
different
services,
for
example.
Here
it's
going
from
dispatch,
micros
end
point
to
customer
endpoint,
then
root
endpoint
and
then
maybe
other
different
microservices.
A
A
A
Let's
not
discuss
about
open
telemetry,
so
open
transmitter
is
a
is
a
fairly
recent
project
and
it's
a
collection
of
tools,
apis
and
sdks,
which
help
you
generate
this
and
collect
and
export
this
telemetry
data
right.
So
we
talked
about
like
different
type
of
telemetry
testos,
which
could
be
metrics
logs
and
traces,
so
open
telemetry
is
the
instrumentation
is
like
helps.
You
instrument
your
applications
with
these
and.
A
And
then
you
can
send
it
to
open
telemetry
collector,
and
then
you
can
export
this
telemetry
tags
and
then
you
can
use
a
background,
for
example
like
signals
to
visualize
it
and
store
it
and
like
do
more
analysis
on
top
of
this
right,
so
open
telemetry
is
open
standard
like
top
companies
like
microsoft,
aws,
etc,
are
contributing
to
it,
and
it's
a
fairly
maturing
project.
So
do
have
a
look
at
it.
A
So
some
of
the
key
features
of
open
telemetry
are
like
it
has
all
the
three
key
pillars
of
authority
we
talked
about.
It
has
traces,
it
has
metrics,
it
has
logs.
So
you
can
create
this
telemetry
data
and
also
collect
from
your
services
like
so,
for
example,
if
you're
running
a
web
server,
you
can
create
a
student,
your
web
server
to
send
tray
sales,
metrics
and
logs
and
then
visualize
the
backend
systems.
It
provides
stop
instrumentation,
so
it
has
lots
of
auto
inject
libraries,
which
can
for
principles
like
spring
is
v.net
core
express.
A
So
you
can
just
inject
this
libraries
in
your
code
base
and
then
start
seeing
data
about
it
right,
and
this
is
open
source
and
vendor
neutral.
It
is
supported
by
lots
of
industry
leaders,
as
I
mentioned,
like
microsoft,
aws,
etc,
and
hence
it's
effort
from
the
community
to
make
this
make
the
instrumentation
part
of
the
observability
of
open
standards
so
that
everybody
can
rely
on
it
and
then
you
can
use
different
tools
to
sort
of
visualize
it
right.
So
that's
where,
like
open,
telemetry
fits
in
open
till
and
let
us
cover.
A
So
let's
talk
about
open,
telemetry
collector,
which
is
sort
of
the
one
of
the
key
components
of
open
telemetry.
Let
me
share
a
quick
diagram.
Maybe
that
would
be
more
helpful
for
you
yeah.
So
this
is
a
good
diagram
which
explains
how
the
open
telemetry
works.
So
this
open
telemetry
is
something
called
collector
which
on
which
you
can
send
start
sending
a
data
from
your
applications,
and
then
you
can
visualize
it
through
multiple
metrics
right.
So
now,
open
telemetry
collector
has
three
key
pieces.
It.
It
has
something
called.
A
Something
called
receivers
which
helps
you
receive
metrics
from
different
places
like
your
applications
or
you
can
receive
prometheus
metrics,
and
then
you
can
process
it.
For
example,
you
can
process
the
open
telemetry,
the
metrics
you
receive
through
your
applications
and
then
add
more
data
like,
for
example,
you
can
enrich
with
the
node
metrics
or
you
can
maybe
remove
your
sensitive
data.
You
can
create
a
processor
to
remove
any
sensitive
data
from
the
data
you
received
and
then
try
to
process
it,
and
then
it
has
exporters.
A
For
example,
it
can
export
to
different
databases.
It
can
export
to
different
backend
systems
so
that
you
can
then
export
visualize
it
later
right.
So
that
was
that's.
What
open
telemetry
collector
is
and
we'll
we'll
come
back
to
this
later,
where
we
discuss
more
about
like
how
we
how
this
is
important.
But
let
me
first
give
you
a
quick
intro
of
what's
like
signals,
because
we
use
open
telemetry
heavily
so
so
as
of
now
so
signals
is
an
open
source
objective
platform.
A
A
We
have
very
powerful
tracing
filtering
functionalities.
You
can
aggregate
things
very
well
and
the
core
principle
which
we
have
is
to
make
it
as
easy
as
possible
right.
So
if
you
have,
if
you
see
other
sas
tools
like
new,
really
data
dock,
they
they
let
you
set
up
systems
very
quickly,
but
most
of
the
current
open
source
systems
are
not
as
easy
to
set
up
right.
So
so
one
of
the
things
which
you
want
to
do
with
signals
is
set
up
as
easily
as
possible.
A
In
just
a
few
clicks,
you
should
be
able
to
start
seeing
your
application
metrics
and
we
light
it
over
lights.
It
in
signals
right.
We
use
click
house
as
the
data
store.
It's
a
it's.
A
olaf
data
store
column,
column,
db
and
I'll
also
talk
about
later
on
how
why
columnar
database
is
important
for
authority
data
and
like
being
open
source?
You
can
you
don't
need
to
send
data
to
any
outside
servers?
A
You
can
run
signals
in
your
in
your
cloud
right,
so
so
that's
probably
what
we
do
at
signals
you
can
set
it
up
very
easily
install
get
installed
and
run
in
your
old
cloud
right
now.
Let's
take
a
quick
look
on
like
how
we
have
designed
signals
and
then
this
would
lead
to
what
could
be
the
potential
issues
which
may
face
right
at
investing
in
use.
So
probably
we
use
open.
A
Telemetry,
we
use
open,
telemetry
libraries
to
instrument
applications,
so
you
can
instrument
your
applications
here.
Add
open,
telemetry
libraries.
This
goes
to
open,
telemetry
collector,
which
receives
all
this
libraries.
And
then
you
write
to
clickhouse
store
right,
which
is
the
our
main
data
store,
and
then
you,
the
dashboards,
which
you
see
they
they
call
the
go,
go
backend,
which
is
a
query
service
which
talks
to
click
on
underlying
right.
So
that's
probably
the
architecture,
and
so
now
what?
Let's
talk
about?
A
Come
to
the
topic
of
like
what
could
be
issues
which
my
face
and
on
like,
if
you
try
to
send
huge
amount
of
data
through
signals
and
open
element
right
so
so.
This
is
just
example,
which
we
came
across
when
we
were
trying
when
you're
working
with
one
of
our
users,
who
was
like
sending
huge
amount
of
data
to
us,
and
then
we
wanted
to
test
like
what
what's
the
like
scale,
which
had
signals
can
handle
right.
A
So
so
that's
when
we
started
working
on
this
and
trying
to
really
test
the
limits
of
what
open
telemetry
with
signals
can
handle
right
and
what
we
found
was
really
surprising
and
that's
what
like,
I'm
really
excited
to
share
with
you.
But
let's
to
give
you
a
quick
sense,
let's,
let's
take
a
quick
pause
and
try
to
see
like
where
could
be
the
issues
right
so
cool.
A
So
we
just
saw
that
okay,
so
signals
has
this
architecture
there
like
this,
are
different
applications
from
where
you
start
sending
data
to
open
telemetry
so
say
this
is
that
one?
This
is
app2
now
you're
trying
to
send
this
data
to
this
open,
telemetry
collector,
which
comes
with
signals
right,
and
then
you
try
to
send
this
to
clickhouse.
A
A
A
Is
his
data
getting
dropped
here
right
and
then
second
piece
is
this,
which
is
like?
If
you
send
a
lot
of
amount
of
data
from
hotel
collector
to
click
house,
how
much
is
it
able
to
handle
right?
So
these
were
basically
the
two
questions
we
wanted
to
test
and
and
see
like.
Where
do
we
end
up
right
like
or
how
much
scale
is
this
being
able
to
handle.
A
And
like
couple
of
quick
points
about
open,
telemetry,
so
open
telemetry,
the
collector
which
we
have
is
a
statelet
selector.
So
it
can
be
scaled
by
increasing
the
number
of
replicas.
So
if
you're
using,
for
example,
kubernetes,
you
can
add
a
lot
of
number
of
replicas
to
handle
the
scale
right.
So
so
that's
like
one
of
the
good
features
of
open
telemetry
that
you
can
scale
it
horizontally
and,
just
like
add
multiple
number
of
replicas
to
it
and
and
yeah.
A
Let's,
let's
discuss
now
about
what
could
be
the
different
hierarchy
in
which
we
can
run
open,
open,
telemetry,
collector
right
so
yeah.
So,
let's,
let's
see
like
what
could
be
the
different
hierarchy
for
open,
telemetry,
collector,
so
open
telemetry
collector
can
run
in
like
three
different
modes.
Primarily
one
is
a
side
car
three
modes.
Primarily
one
is
the
side
car
mode
where,
like.
A
Side,
car
mode,
the
second
is
it
can
run
at
each
mode
which
is
like
the
hbm
or
third
is
it
can
run
it
run
on
each
cluster
right.
So
these
are
privacy
modes
in
which
open,
telemetry,
collector
can
run
sidecar
mode
is
nothing
but
you
run
open
telemetry
alongside
an
application.
Suppose
you
have
a
application
one,
and
by
this
I
mean
something
like
say,
a
springboard
application
right,
so
you
have
a
springboard
application
and
then
you
want
to
collect
metrics
from
it.
So
either
you
can
inside
car
mode.
A
A
And
then
maybe
multiple,
if
you
want
to
send
it
to
a
particular
bigger
hotel,
collector
or
they
may
not,
click
click
that
it
will
try
to
send
to
that
right.
The
issue
with
this
approach
is
that
there
are
like
lots
of
hotel
collectors
everything,
so
it's
difficult
to
manage,
also
like,
because
each
total
collector
needs
certain
resources
to
run.
It
takes
lots
of
resources
from
the
system
right,
but
but
the
advantage
that
it's
it's
like
blast
radius
is
very
small.
A
If
one
of
the
open
electric
connectors
goes
down,
only
this
open,
telemetry
collector
goes
down
right.
The
other
approach
could
be.
You
can
run
only
on
a
single
node.
So,
rather
than
are
you
running
on
like
each
collector,
you
don't
run
it
here.
You
run
a
single
collector
on
each
node,
which
is
like
the
vm
here,
and
then
you
start
matrix
sending
metrics
from
here
right.
So
the
advantage
here
is
that
you
will
get
all
vm
metrics
like
memory,
users,
cpuses,
etc.
A
A
More
compared
to
like,
if
you
are
on
a
single
hotel
right,
the
other
approach
could
be.
You
can
just
like
run
a
single
hotel,
collector
and
rather
than
each
application,
sending
to
like
their
site
car.
You
can
just
send
directly
to
here.
So
no,
no,
no
hotel
collectors
at
the
node
level.
Also,
but
just
runs
things
send
instrument
all
applications,
have
your
applications
and
then
keep
on
sending
it
to
the
main
cluster
right.
A
So
there
can
be
different
nodes,
so
this
can
be
vm,
2,
vm1
and
each
vm
can
be
sending
data
or
like
applications
in
each
vm
can
be
sending
data
to
force
and
data
two
of
central
hotel
collector,
which
can
be
on
the
cluster
network
right.
So
these
are
like
different
ways
in
which
open
telemetry
collector
can
scale.
A
A
This
slide
basically
talks
about
how
does
these
different
model
works
and-
and
let's
talk
about
like
what
we
worked
on
right,
so
we
worked
on
ingesting
and
trying
to
test
this
in
a
docker
swap
environment.
So
this
was
just
a
test.
Ideally
we
could
have
done
it
on
a
covenant
disclosure,
but
this
was
just
a
test
on
handling
how
much
resources
can
an
open,
telemetry
collector
handle
right.
So
we
started
generating
data
around
one
million
events
per
second,
the
amount
of
resources
it
took.
A
So
we
had
it
in
a
single
dockers
form
setup,
and
this
was
this
was
not
a
kubernetes
setup.
So
this
was
a
single
set
of
where
you
can.
You
have
a
single
virtual
machine
which
is
like
we
used
a
9
96
cpu,
72,
cpu
virtual
machine,
and
then
we
used
a
16
hotel,
collector
replica
set
so
made
16
replicas
for
it,
and
but
a
single
vm
instance
right,
a
single
click
house
instance
in
the
vm.
A
So
what
we
found
was
that
this
setup
was
able
to
handle
like
one
million
events
per
second
right,
which
is
quite
huge
and
which
is
what
we're
like
very
surprised
with,
and
if
you
see
broadly,
this
is
at
this
skill
this.
This
resource
utilization
is
very
cost
effective.
Like
you
get
a
42
v
cpu
machine
in
aws
at
like
very
easily
like
you
can
get
a
96
cpu
intensive
machine
very
easily,
and
then
you
can
now
in
just
like
one
million
events
per
second
right.
A
You
can
send
so
so
broadly,
if
you
see
to
send
around
1
million
events
per
second,
you
can
you
have
to
run
around
50
replicas,
while
in
our
experience
we
were,
we
found
that,
like
16
or
10,
collectors
were
able
to
handle
that
scale,
so
that's
around
like
50
to
60
000
for
events,
but
that
also
depends
upon,
like
the
number
of
events
which
you
are
sending
and
like
maybe
also
the
type
of
the
events
but
but
like
so
this.
This
is
scaled
decently.
A
Well,
we're
able
to
send
around
one
million
events
per
second
handle
it
with
just
16
replicas
in
the
dock
or
some
container,
and
then
on
the
amount
of
cpu
start
using
like
42
cps
right
so
so.
This
was
like
a
really
surprising
for
us
that
hey
like
a
single
neil
opened
elementary
collector
installation,
with
with
using
a
single
click
house
instance
worked
for
handling.
A
This
is
one
million
units
per
second
right,
so
this
gives
you
lots
of
confidence
like
like
on
how
much
open
telemetry
can
scale
and
also
like
how
click
offs
is
a
good
database.
For
this.
There
are
like
possible
improvements
for
which
we
can
do.
As
I
mentioned,
between
between
open
telemeter
collector
to
click
house,
we
can
have
some
sort
of
queueing
in
unique
house,
of
course,
something
called
car
kafka
engine
in-house
and
then
there's
also
concept
of
buffer
engines.
A
So
maybe
we
can
try
to
improve
this
by
adding
a
collector,
but
like
for
the
one
million
events
per
second
scale.
We
are
not
seeing
any
such
buff
issues
if
you
want
to
have
better
performance
for
the
click
house.
Click
house
is
also
both
horizontally
and
vertically
scalable,
so
you
can
scale
click
house
horizontally
saad.
You
can
also
set
up
data
replica
replication
to
enable
redundancy.
A
A
If
you
want
to
or
run
this
in
production
right
and
and
the
third
piece
is,
you
can
also
do
a
distributor
set
up
of
click
house
so
rather
than
having
a
single
machine
click
on
this
running,
you
can
set
up
distributed
so
that
a
same
query
is
like
distributed
among
different
instance,
and
then
you
can
get
data
from
different
instances
and
then
merge
it
back
right.
So
that
might
be
in
it
like
some
of
the
possible
improvements
which
which
we
think
could
be
made.
A
But
even
this
result
at
like
even
to
test
this,
this
results
is
very
interesting
to
us
that
it
was
able
to
handle
one
million
events
per
second
without
any
issues
right.
So
so,
yeah
and
last
point
like
one
of
the
questions
that
many
people
have
was
like,
why?
Why
did
you
choose
to
use
a
columnar
database
to
us?
A
Columnar
database
might
make
natural
sense
for
observability,
because
you
can
add
a
as
many
like
rows
and
like
attributes,
you
want
and
there's
no
prior
schema,
which
is
needed
in
in
observatories.
Basically,
if
you,
if
you
want
to
provide
good
analytical
capabilities
like
we
do
provide
in
signals,
aggregation
of
queries
is
very
important,
and
this
works
pretty
well
for,
like
aggregate
for
column,
columnar,
database,
right
and
and
so
just
to
give
you
a
sense
of
like
why
this
is
important.
A
Let
me
share
a
quick,
maybe
example
yeah,
so
so
just
a
quick
example
of
on
like
why
columnar
databases
are
important
so
generally
for
observability,
especially
in
the
application
monitoring
space.
The
type
of
data
we
deal
with
is
like,
for
example,
you
are
trying
to
trace
a
request
across
different
systems
right,
so
you
have
a
syst
example
like
get,
and
then
it's
making
query
to
example.
This
is
the
resource
string
and
then
there's
a
like.
You
have
time
like
time
taken
operation.
A
Say
200
so
like,
and
then
you
can
add
different
attributes
here
right
so
now
suppose
if
you
have
different,
so
each
request
is
like
this:
if
you
are
storing
this
in
a
like
columnar
database
format,
you
can
just
do.
Aggregate
queries
like
hey.
Do
tell
me
the
average
time
taken
by
queries,
which
has
this
http
status
code,
equal
to
say,
200
right.
A
So
what
you
have
to
do
is
you
just
have
to
say
like
find
http
queries
which
has
http
code
this
and
then
just
copy
the
like
try
to
get
an
average
of
this
right.
So
so,
if
you
see,
if
you're,
storing
these
as
like
columns,
you
don't
need
to
look
into
other
columns
at
all.
You
can
just
say:
hey
like
what
are
the.
What
are
the
rows
which
has
http
gold
is
equal
to
200?
Then
you
take
the
time
for
those
and
then
just
do
an
average
of
this.
A
If
you
allow
like
users
to
slice
and
dice
the
data
in
different
ways
and
and
columnar
databases
really
seems
very,
very
suitable
for
this
right,
so
so
and
just
to
point
out
and
and
like
if
you
are,
if,
like
all
the
required
queries,
are
in
the
same
locality,
it
also
increases
quite
a
speed,
there's
a
blog
by
honeycomb,
also
on
why
they,
they
chose
a
columnar
database
as
the
main
data
store
and
do
have
a
look
at
it.
It's
very
interesting.
You
mentioned
some
great
points
there,
so
yeah.
A
A
If,
if
you
are
interested
in
the
project,
if
you're,
probably
interested
in
the
ability
domain
check
out,
our
repo
do
do
also
check
out
the
slack
community
hangout
there
we
have
around
800
plus
requester
and
if
you're
interested
in
working
on
it
or
working
on
the
domain,
we
are
always
looking
for
good
people
to
join.
Our
team.
Do
give
me
a
ping
and
yeah
or
if
you're,
interested
in
contributing
to
the
project,
just
have
a
look
at
the
issues,
open
issues
and
then
feel
free
to
say,
hi
cool.