►
From YouTube: Is Kubernetes Monitoring Flawed?
Description
A 3-node Kubernetes cluster with Prometheus will ship around 40k active series by default! Do we really need all that data?? The current state of Kubernetes open source monitoring is in need of improvement. High churn rate of pod metrics, proliferation of metrics with low usage, and configuration complexity are some of the issues that need to be addressed.
I discussed this topic with Aliaksandr Valialkin, CTO at VictoriaMetrics and creator of the open source project. We discussed the common problems, as well as directions and best practices to overcome some of these complexities as individuals and as a community. We also discussed VictoriaMetrics open source project and how it addresses some of these challenges.
Aliaksandr is a Golang engineer, who likes writing simple and performant code and creating easy-to-use programs. Sometimes these hard-to-match requirements work together, like in the VictoriaMetrics case.
The podcast episodes are available for listening on your favorite podcast app and on this YouTube channel.
We live-stream the episodes, and you’re welcome to join the stream here on YouTube Live or at https://www.twitch.tv/openobservability.
Follow us on Twitter @openobserv to get the live stream times and other updates, and to pitch in with your thoughts and comments.
Have you got an interesting topic you'd like to share in an episode? Reach out to us and submit your proposal at https://forms.gle/9LDkYCmegyS5D8Li7
Dotan Horovits
============
Twitter: https://twitter.com/horovits
LinkedIn: https://www.linkedin.com/in/horovits/
Aliaksandr Valialkin
===============
Twitter: https://twitter.com/valyala
LinkedIn: https://www.linkedin.com/in/valyala/
VictoriaMetrics: https://victoriametrics.com/
On GitHub: https://github.com/VictoriaMetrics/VictoriaMetrics
VictoriaMetrics community channels - https://docs.victoriametrics.com/#community-and-contributions
Resources
=========
Why Prometheus cannot query remote storage in an expected way via remote_read protocol - https://github.com/prometheus/prometheus/issues/4456
VictoriaMetrics: scaling to 100 million metrics per second https://www.youtube.com/watch?v=xfed9_Q0_qU
Chapters
========
00:00 show intro
02:07 topic and guest intro
03:13 monitoring microservice system, app and communications
05:43 high churn rate for pod metrics
12:02 Kubernetes produces too many metrics by defaults, most of which are unused
17:06 recommended listing of metrics
21:50 removing unused metric labels to reduce cardinality
24:16 Prometheus native (exponential buckets) historgrams
26:49 Configuration complexity with multiple deployments
33:16 OpenTelemetry and OpenMetrics open specifications
36:11 collecting system metrics and application metrics uniformly
40:20 VictoriaMetrics essentials
48:46 VictoriaMetrics extensions beyond Prometheus
54:06 a full stack monitoring collection, analysis and alerting
56:09 how to join the VictoriaMetrics community
58:05 industry update: 2023 cloud native predictions post by CNCF CTO
59:16 outro
A
Hello,
everyone
and
welcome
to
open
observability
talks,
I'm
your
host,
the
Tom
horvitz
and
here
at
open
observability
talks.
We
talk
about
anything,
devops,
observability
and
open
source,
so
May
the
open
source
be
with
you
and
Happy
New
Year
everyone.
This
is
the
first
episode
of
2023
and
we've
got
a
whole
new.
Look
for
the
episode
for
those
who
remember
the
the
yellow
theme
theme
so
hope
you
like
it
and
do.
Let
me
know
on
Twitter
at
openobserve
or
at
orbit.
A
I
also
just
got
the
end
of
year,
podcast
statistics
from
Spotify
and
I'm
glad
to
share
that.
We
have
listeners
from
over
40
countries.
We
have
more
than
tripled
our
followers,
270
percent
increase
and
several
hundred
listeners
ranked
us
in
the
top
10
podcasts.
So
thank
you
so
much
for
joining
me
on
the
show
throughout
the
years
and
for
following
us
and
giving
us
high
ranking
and
the
show
is
available
on
all
the
popular
podcast
apps,
not
just
Spotify,
so
as
well
as
YouTube.
If
you
want
to
watch
us
chatting.
A
So
if
you
haven't
followed
yet
do
go
ahead
and
join
us.
I'd
also
like
to
thank
our
sponsors.
Logs.Io
the
cloud
native
observability
platform
logsio
takes
the
best
of
breed
open
source
projects
such
as
Prometheus,
open
search
and
Jager,
and
offers
them
as
a
unified
observability
platform
built
for
scales
for
those
joining
the
live
stream
or
on
YouTube
or
twitch,
feel
free
to
share
questions
and
comments
on
the
chat.
This
makes
things
much
more
interesting
for
me
and
the
guests,
and
let's
move
on
to
today's
episode
today,
I'd
like
to
talk
about
kubernetes
monitoring.
A
We
we
actually,
we
had
an
interesting
chat
a
few
months
ago
at
open
source
monitoring
conference
in
Germany
and
I'm
really
glad
for
the
opportunity
to
have
this
fireside
chat
with
you
on
the
on
the
show
today
and
as
I
presented.
Also
the
opening
you
live
and
breathe,
metrics
and
monitoring.
So
maybe
can
you
tell
us
a
bit
about
your
experience
with
monitoring
kubernetes
as
a
practitioner
and
the
challenges
that
that
you've
encountered.
B
So
there's
only
one
application.
Harmonious
and
each
topic
is
this:
application
export
some
kind
of
application,
metrics
and
system
metrics
and
the
number
of
such
metrics
is
fixed.
And
if
you
split
monoliths
into
many
microservices,
then
every
microservice
need
to
export
its
own
set
of
metrics
system.
Metrics
I
mean
CPU
memory,
Network
users
and
so
on,
and
also
everyone
microservice
needs
to
expose
its
own
set
of
application
metrics
and
additionally,
on
top
of
this,
every
maker
service
should
expose
communication.
B
Metrics
I
mean
metrics
related
to
the
communication
between
microservices,
latencies,
RPS
and
so
on,
and
so,
when
you
switch
from
single
note
application
to
microservice
application,
the
number
of
metrics
explodes
many
times
10
times
and
more
and
since
kubernetes
popularizes
microservice
architecture,
as
this
means
that
when
you
move
to
kubernetes
from
a
plain
old
hosting
providers,
hosting
Services,
you're,
usually
start
to
exposing
start.
Exposing
many
more
metrics.
A
And
I
think
there's
also
a
point,
maybe
worth
mentioning
that
it's
also
the
the
frequency
like
it's
not
just
that
we
break
the
application
into
many
many
microservices.
It's
also
that
each
one
has
its
own
life
cycle.
It's
releases,
obviously
making
it
smaller
and
more
agile
means
more
frequent
releases
and
that
that's
a
major
contributor
as
well
to
the
to
the
to
the
aspect
right,
correct.
B
So
this
means
that,
with
every
new
deployment,
old
Matrix
stop
stop
receiving
new
samples
and
a
new
metrics
appear
and
start
receiving.
New
samples
and
suchitation
means
is
named
this
charm
rate,
and
if
this
redeployment
occurs
frequently,
this
leads
to
high
churn
rate
situation
when
an
old
metrics
are
substituted
by
new
metrics
I
mean
time
series
it's
high
rate
and
this
High
rate.
A
B
A
B
A
B
If
we
look
in
kubernetes
deployments,
there
are
web
replica
sets
and
interim
when
you
instant,
State
forces,
and
if
you
use
state
to
send
deployments,
then
every
Port
has
a
consistent
name
naming.
This
name
consists
of
the
original
pod
name
from
deployment
plus
numeric
the
number
of
this
spot
from
zero,
starting
from
zero
to
the
number
of
positive
deployed
in
a
stateful
set,
and
these
names
stay
static
and
don't
change
when
you're
deploying
deployments
in
stateful
sets.
B
I
would
say
that
this
can
be
used
as
some
kind
of
hack
to
use
stateful
set
abuse,
stateful
sets
for
reducing
charge,
but
I
think
that
the
general
solution
should
be
implemented
by
kubernetes
developers
itself.
They
should
provide
such
some
stable
labels
for
usual
deployments,
not
for
replica
size
deployments,
so
we
as
Monitoring
Solutions
could
use
this
stable
labels
instead
of
name
which
changes
with
each
deployment,
and
this
will
all
be
reducing
just
eliminating
the
channel
rate
in
kubernetes.
A
Sounds
interesting,
yeah!
That's
why
I
was
referring
to
like
back
at
biological
name,
because
I
think
replica
set
or
stateful
set
is
not
the
it's
a
hack,
as
you
said,
but
maybe
the
ability
to
attach
a
logical
name
that
would
carry
on
would
be
more
of
a
a
permanent
set
that
can
be
interesting
and
and
I
think
there's
an
interesting
common
comment
here
from
from
the
audience
saying
that
uuid
is
a
label
always
so
another
suggestion
here
from
the
from
the
audience
the
uuid.
A
A
One
of
the
things
that
is
astonishing
is
I.
Think
the
the
number
of
metrics
that
you'd
find
by
just
installing
the
bare
minimum
I,
don't
know
a
tiny
three
node
cluster
and
just
checking
the
default
metrics
that
come
out
of
the
box.
If
you
just
use
all
the
all
the
defaults
there,
I
I
know
that
we've
experienced
that,
with
with
our
with
our
users,
often
that
they
quickly
start
running
out
of
quota.
Just
because
of
that,
but
interesting
to
hear
your
your
experience
with
that.
B
A
B
The
issue
is
that
the
majority
of
such
metrics
isn't
used
anywhere
in
dashboards.
They
aren't
used
in
dashboards,
they
haven't
used
it
in
alerting
rules
or
recording
rules
and
according
to
grafana
study,
it
has
been
appeared
that
only
25
percent
of
this
exposed,
Matrix
use,
actually
use
it,
and
75
percent
of
these
metrics
are
never
used
in
anywhere.
So
it
is
possible
to
just
reduce
a
lot
on
your
monitoring
system
by
fourth,
at
least
by
four
four
types.
By
removing
these
unused
networks.
B
A
B
B
Metrics
these
companions
aren't
included
in
kubernetes
itself.
The
third
party
components
and
I
think
that
all
these
metrics
from
such
companions
should
be
included
in
kubernetes
itself
and
kubernetes
should
expose
this
metrics,
and
this
the
set
of
this
metrics
should
be
carefully
maintain.
It
I
mean
the
list
of
this
metrics,
and
this
list
should
not
contain
metrics,
which
aren't
used
every
anywhere
well,.
A
That's
that's
actually
a
tricky
question,
because
I
wouldn't
expect
most
of
my
users
at
least
the
one
that
I
you
know,
I
encounter
and
I
talk
to,
and
we
see
also
in
my
company,
don't
really
know
what
they
need
and
usually
they
just
go
for
the
defaults,
because
that's
the
safest
bet
and
just
bringing
everything
and
then
seeing
so
I
think
and
also
it's
not
static.
It
keeps
on
growing
I.
Think
one
of
the
most
astonishing
stats
that
I
I
saw
was
that
scene
I
think
you
brought
it
up.
A
For
example,
we
since
there's
no
such
standard
we
at
at
logs
there,
for
example,
curated
the
list
that
is
open
source.
Obviously
it's
a
it's
part
of
the
helm,
charts
that
we
provide,
and
you
have
the
list
for
for
both
the
kubernetes
out
of
the
box
and
also
for
AKA
cks
gks,
all
the
managed
versions
of
the
kubernetes
to
to
recommend
at
least
what
we
find
useful
amongst
our
users.
Should
you
use
the
cube
system
this
or
that
or
DNA
Cube,
DNS
or
others
that
giving
a
curated
list?
B
I
think
that
system
kind
of
metrics
which
such
as
CPU
usage,
memories
and
so
on
can
be
the
list
of
such
metrics
is
isn't
too
big,
and
it
is
quite
easy
to
provide
such
list
of
create,
at
least
of
such
metrics,
from
cabernetes
developers
itself
from
computer
developers
and
to
provide
such
a
single
or
multiple
experts
in
kubernetes
in
every
kubernetes
installation
which
exports
this
system
metrics
and
as
for
application,
Level
metrics.
It's
a.
B
Every
application
has
different
set
of
metrics
and,
as
you
can
hear,
this,
that's
different
sets
of
users
need
different
metrics
and
use
different
metrics
from
our
application,
and
it's
maybe
hard
to
provide
to
the
curator
at
least,
which
fits
everybody
in
this
case,
I
I
think.
The
way
to
go
is
that
every
application,
such
implication,
which
is
used
in
kubernetes,
should
provide
its
own
set
of
curated,
metrics
or
multiple
sets
for
different
use.
Cases
of
the
metrics
well,.
A
That's
an
interesting
question:
I
I'm,
debating
it
myself,
you
know
again,
as
I
said,
we
resulted
to
to
having
on
our
GitHub
repo
Public
Health
charts,
with
with
the
best
practices
that
we
found.
But
then
again
you
have
different
components
in
it,
obviously
ones
that
use
Cube,
State,
metrics,
Prometheus,
node
exporter
and
others,
and
each
one
brings
its
own
set
of
metrics.
And
the
question
is
if
these
can't
come
with
some
sort
of
of
guidance,
at
least
on
my
on
my
sense
of
things.
A
But
it's
a
question
and
I
think
it
definitely
is
something
that
is
worthwhile,
trying
to
at
least
open
this
channel
of
discussion
between
them
and
and
also
that's
on
the
standardization
side,
and
also
on
the
tooling
side,
to
enable
this
removal
of
irrelevant,
metrics
and
and
maybe
adding
to
the
metrics
themselves.
So
there's
a
metric
that
might
not
be
used
all
together.
The
the
other
level
that
I
found,
at
least
at
least
very
useful,
is
also
removing
labels,
because
that
goes
to
the
cardinality
challenge.
A
So
many
of
the
the
built-in
metrics
come
with
lots
of
labels,
some
of
which
are,
we
found
at
least
less
relevant
and
the
question:
do
you
need
the
per
core
I?
Don't
know
CPU
analysis,
or
is
it
fine
for
you
to
just
get
the
overall
CPU
and
things
like
that
and
when
you
remove
redundant
labels,
you
dramatically
decrease
the
overall
load
and
volume
because
the
cardinality
reduces
the.
Have
you
encountered
something
like
that.
A
And
maybe
just
to
explain
to
our
audience
before
you
carry
on
for
the
so
there
are
different
types
of
metrics
like
gauge
and
like
a
counter
and
histogram
is
the
the
more
complex
one
because
it
actually
is
not
one
number
being
aggregated
over
time.
It's
actually
several
numbers
one
per
bucket,
and
then
you
get
for
each
a
point
in
time.
Multiple
data
points
that
are
being
collected
and,
as
you
said,
this
dramatically
increases
the
the
overall
data
on
that
specific
metric.
A
B
I
think
that
the
solution
there
is
no
Universal
solution
for
this
problem,
but
I
think
that
the
way
to
go
or
is
that
every
developer,
Who
develops
exporters
for
such
metrics
should
pay
more
attention
to
Matrix
it
for
this
experts
are
exposes,
and
just
should
think
twice
before
aging
additional
metric
to
this
exporter
and
should
think
more
about
which
metrics
or
labels
can
be
removed
without
hurting
the
observability
of
this
experts.
In
this
application,
yeah.
A
B
A
B
Think
that
exponential
Instagrams,
with
exponential
buckets
with
a
great
way
to
use
much
better
than
other
bucket
types
for
histograms,
because
you
don't
need
to
think
about
about
boundaries
of
this
buckets.
These
boundaries
are
created
automatically
and
you
automatically
cover
all
the
range
of
values
for
measurement.
You
measure
with
the
histogram,
and
the
main
issue
is
an
exponential
bucket
histograms-
is
that
the
number
of
buckets
can
be
quite
big.
A
Yeah
and
and
another
point
that
I
wanted
to
discuss
is
it
comes
up,
especially
when
dealing
with
multiple
deployments.
So
when
you
have
a
large
environment
involving
many
deployments
in
parallel-
and
this
is
the
configuration
challenge,
how
do
you
configure
these
multiple
environments?
They
also
talk
about
how
you
view
this
and
how
you
handle
this.
B
A
B
B
B
Right,
you
need
to
manage
individually,
yes,
and
so
this
increases
a
lot,
both
some
users
who
operate
all
this
stuff,
and
this
increases
a
lot
on
kubernetes
API
server
because
it
needs
it
needs
to
answer
a
hundred
times.
Instead
of
one
time
when
you
have
one
Escape
configuration
and
I
think
that
the
way
to
go
is
to
oh
to
use
some
to
divide
some
standard
for
serious
Discovery
in
kubernetes
for
deployment
and
what
discovering
kubernetes
on
top
of
serious
discoveries
used
in
parameters.
B
Yeah
I
think
that
they
shouldn't
configure
anything
in
ideal
case.
Their
applications,
which
are
on
Imports,
should
just
expose
metrics
in
a
standard
way
in
parameters.
Exposition
format,
for
instance
at
standard,
was
known
paths
and
CP
ports
and
that's
the
way
to
go,
and
in
this
case
the
monitoring
system
can
just
Escape
discover
all
the
ports
in
kubernetes
and
just
scrape
Matrix
from
well-known
endpoints.
B
This
is
the
way
to
go
and
actually
metric
separatory
support,
such
kind
of
strapping,
which
is
close
to
this
video
situation.
This
escaping
configuration
can
be
configured
via
annotations
in
deployments.
You
can
say
that
please
scrape
Matrix
from
this
ports
which,
from
this
person
or
this
person
from
the
sport
and
this
configuration,
is
written
in
annotations
in
every
deployment.
B
This
is
better
than
writing
custom
resource
definitions
for
Matrix
grade
and
perish
deployment,
but
I
think
that
the
even
best
solution
is
to
just
appears
and
some
kind
of
standard,
formatic
Exposition
in
kubernetes
spots,
and
in
this
case
you
don't
need
to
write
anything
metric
related
into
your
deployments
configurations,
and
the
monitoring
system
should
just
cover
all
these
New
Ports
and
discover
metrics
on
this
spot's.
Without
the
need
of
additional
configuration
from
users.
A
So
actually
there
is
a
there's,
a
question
here
from
the
audience:
isn't
the
this?
The
future
is
in
the
future
in
otlp
protocol
otlp
being
the
open
Telemetry
protocol.
Do
you
see
that
as
part
of
the
roadmap
on
this
path,
or
maybe
I
will
add
to
that
question
from
the
audience?
If
you
see
that
within
the
open,
metrics
format,
or
how
do
you
say
that
mapping
to
existing
opens
open
specifications.
B
B
This
metric
Exposition
format
is
very
easy
to
implement
it's
very
easy
to
debug,
because
it's
just
plain
text
and
everybody
knows
how
to
read
it
familiar
with
it.
As
for
open,
Telemetry,
I
think
that
it
can
be
used
for
collecting
traces,
probably
logs
I,
don't
know,
but
as
for
Matrix
I
think
I
see
that
open
Telemetry
is
quite
complicated
compared
to
simple,
open
metrics
format
for
for
Matrix.
A
Yeah
I
think
it's.
It
is
by
the
way
in
in
the
roadmap
of
open
telemetries,
not
just
for
logs
and
tracing
and
definitely
aiming
for
for
metrics
as
well
and
there's
lots
of
work
also
with
Prometheus
projects
and
the
working
groups
to
to
create
the
Synergy
between
the
two
specifications.
So
definitely
I
think
that
it
will
be
interesting
to
see
how
the
open
Telemetry
specification
also
addresses
that,
but
I
guess
we
will
have
to
live
and
see
that
I
want
to.
B
That's
actually
related
to
opens
Elementary
efforts
because
open
Telemetry
popularizes
as
a
common
standard
for
telemetal
data
collection
so
and
the
one
of
the
main
points
for
such
standard
is
to
ease
the
correlation
between
different
kinds
of
collected
Matrix.
The
SS
logs,
so
I
think
that
that's
also
a
way
to
go
to
improve
the
situation
in
a
kubernetes
world.
A
Yeah
so
I
think
we
discussed,
and
we
also
said
that
you
mentioned
about
the
try
to
create
to
I,
guess,
collect
in
a
uniform
manner
the
two
aspects,
both
the
system-
metrics,
that
are
I,
guess
more,
a
close,
more
closed
set
of
of
metrics
and
also
the
application
metrics.
Do
you
want
to
add
anything
about
the
what
you
see
lacking
in
the
in
the
collecting
in
uniform
manner,.
B
I
have
report,
and
there
is
no
something
to
invent
here
in
this
area,
so
this
Matrix
should
be
I
mean
system
metrics
in
kubernetes
should
be
collected
by
provided
by
kubernetes
itself
and
collected
from
exporters
which
are
provided
by
kubernetes
itself.
Oh,
this
is
I,
see.
I
mean
I,
think
that
it's
a
easy
to
solve
a
task.
B
A
Yeah
I
think
there's
also
aspect
of
what
goes
back
to
what
what's
needed,
because
many
of
the
metrics
that
are
exposed
are
not
I
find
not
actionable,
and
it
really
depends
also
on
the
on
the
platform.
For
example,
if
you
compare,
if
you
go
to
the
managed
kubernetes,
for
example,
when
you
collect
system
components,
I,
don't
know
Azure
Defender,
for
instance,
you
pretty
much
probably
collecting
the
the
system.
Metrics
will
be
useless
for
you
as
an
end
user,
because
you
don't
manage
the
underlying
system
anyway.
So
it's
not
actionable
for
you.
A
So
and
still
you
can
find
these
metrics
exposed
by
the
managed
kubernetes
nonetheless,
I
guess
they
didn't
filter
it.
So
what
I
find
is
that
really
focusing?
It
goes
back
to
also
focusing
on
on
and
the
relevant
metrics,
also
in
the
aspect
of
standardization,
so
per
the
environment
that
you
work,
managed
kubernetes
environments
should
also
provide
their
own
exposed
set
that
is
relevant
to
their
end
users
in
a
way
that
will
will
focus
instead
of
defocus
us
as
the
consumers
of
the
service.
A
No
sounds
good
sounds
good.
I
want
to
switch
and
talk
a
bit
about
your
journey
with
Victoria
metrics,
it's
a
very,
very
interesting,
open
source
in
the
field
and
quite
quite
widely
used
these
days
so
interesting
to
hear
you
started.
Actually
your
service
practitioners
working
with
Prometheus-
and
you
said
you
actually
had
great
Improvement,
especially
over
zapix
back
in
the
air
in
the
previous
day.
So
what
what
drove
you
to
start
developing
another
solution.
B
Yes,
as
the
previous
work,
we
started
using
parameters
and
we're
very
happy
with
parameters
compared
to
zotics,
because
we
were
using
zombies
before
the
main
feature
of
parameters
which
we
liked
that
developers
can
add
as
many
as
they
want
metrics
to
their
applications
without
the
need
to
talk
to
devops
and
say:
please
set
this
metric
to
my
application.
This
is
the
way
to
go
the
way
which
we
were
using
in
the
Xanax
error.
B
That's
great.
That
was
great,
but
when
we
started
using
Prometheus,
we
end
up
in
a
very
big
number
of
metrics
and
the
the
at
that
time
there
were
no.
There
was
no
premises
to
zero.
There
was
premises,
one
point
something
and
this
parameters
couldn't
keep
up
with
the
increased
amounts
of
metrics,
which
we
happily
fit
it
even
on
a
pretty
beefy
Hardware.
A
Important
to
note
for
the
listeners
that
are
not
aware
Prometheus
by
Design
is,
is
a
single
node
installation,
so
it
can
scale
obviously
vertically,
if
you
add
more
memory
more
disk
whatever,
but
it
can't
scale
horizontally.
So
you
can't
make
it
into
a
clustered
solution.
So
that's
that's
the
limit.
I
guess
that
that
native
from
it
is
offered.
A
Nice,
by
the
way
we
in
a
previous
episode,
we
also
hosted
the
founder
of
signals,
which
is
also
based
on
on
clicks,
clickhouse
architecture,
very
fascinating,
open
source
clickhouse,
and
we
can
talk
about
it
in
more
in
different
episodes.
But
I'm
curious
actually
to
hear
so
so
you
you
found
yourselves
need
liking
the
Prometheus
concept,
let's
say,
and
then
many
of
the
features,
but
you
needed
to
lack
the
scalability.
A
B
Yeah
with
telematics
the
main
difference
between
Victory
metrics
and
other
systems
on
the
markets,
such
as
Thomas
cortex.
Only
Mir
is
that
Vector
Matrix
is
written
from
scratch.
It
doesn't
use
any
source
code
from
parameters,
so
other
systems
cortex
and
veneer
based
on
parameters
source
code.
They
reuse
it.
So
that's
the
main
difference
between
this
systems.
B
Have
pros
and
cons
the
main
process
of
block
storing
data
to
ordinary
disks
is
that
ordinary
these
usually
have
much
smaller
latencies
much
smaller
error
rates
and
much
higher
bandwidth
compared
to
object
storage.
This
helps
improving
query
performance,
comparing
to
object,
storage
and
the
main
disadvantage
of
local
disk
compared
to
object.
Storage.
There
is
a
local
disks,
have
limited
space
capacity
and
you
need
to
manage
resize
some
kind
to
think
about
how
to
resize
this
disks.
B
When
you,
your
storage,
will
shrink
and
Victor
metrics
can
scale
to
multiple
nodes
in
this
for
solving
this
problem.
So
if
you
fuel
a
single
disk
in
this
Vector
Matrix,
then
you
cannot
more
storage
nodes
to
economics,
and
in
this
case
you
will
get
more
storage
space
and,
additionally,
some
local
disks,
such
as
Google,
persistencies
or
Amazon
block
storage,
can
be
resized
on
the
Fly.
You
can
just
add
more
space
to
this
in
disks
without
the
restarting
of
the
application.
B
Recently,
we
run
a
benchmark
who
in
which,
in
this
Benchmark,
we
when
Jason
100
metrics
per
second
into
a
telemetrics
cluster
during
24
hours,
and
these
metrics
were
collected
from
a
real
node
expert
parameters,
not
experts
or
companions.
This
metrics
weren't
generated
around
only
so
this
Benchmark
was
very
close
to
production
data
from
from
collection
collection
site
and
in
this
Benchmark
we
discovered
that
Victory
metrics
can
scale
to
such
huge
workload
on
that
ingestion
path,
and
this
Benchmark
was
running.
B
A
Nice
yeah
I
remember.
Actually,
we
went
from
from
OSMC
from
open
source
monitoring
conference
in
Germany.
That
was
a
pretty
astonishing
numbers
and
that
you
shared
there.
One
point
that
I
wanted
to
ask
is
that
you
know
many
of
these
Solutions
started
as
a
long-term
storage
for
Prometheus.
I.
Think
that
you
with
Victoria
metrics,
took
some
a
different
approach,
at
least
at
some
point.
You
you
started
I,
guess
diverging
or
extending
beyond
the
Prometheus
ecosystem,
both
in
the
Chrome
ql,
query
language
with
extensions
and
with
apis.
B
Yes,
as
you
said,
we
started
initially,
we
started
as
a
remote
storage
for
parameters
and
later
we
stumbled
with
some
issues
with
Prem
in
premises
ecosystem.
The
first
issue
was
not
so
good
support
for
a
remote
read
protocol
from
Prometheus
site
when
we
created
new
telemetrics
as
remote
storage
for
parameters.
B
We
were
thinking
that
parameters
could
use
Vector
metrics
as
a
storage
and
all
the
queries
can
go
to
Prometheus
and
premises
can
ask
for
the
data
from
electromatics
and
processes
that
and
return
the
query
response,
but
it
has
been
appeared
that
that's
not
that
was
not
working
because
of
different
issues.
I
can
later
put
you
post
your
links
to
these
issues
in
parameters,
issue
tracker
and
that's
why
we
had
to
create
our
own
implementation
of
parameters.
Query
language.
We
start
implementing
it
implemented
it
from
scratch,
so
you
can
query
Victory
metrics
on
itself.
B
Without
the
help
of
parameters
and
later
this
parameter,
Square
language
implementation
has
been
grown,
overgrown
bronchial,
query
language
and
we
renamed
it
to
metic
scale.
This
superset
on
top
of
bronchial
I,
would
say-
and
it
also
has
some
incompatibilities
with
bronchial,
and
this
and
capabilities
are
deliberate.
This.
B
We
are
not
going
to
fix
them
this
kind
of
capabilities
because
of
we're
seeing
that
our
implementation
fits
users
better
than
parameters
implementation.
For
instance,
we
can
look
at
increased
function,
implementation
if
you
use
a
execute
and
increase
function
on
top
of
integer
counter
in
parameters.
You
can
be
surprised
when
the
parameters
will
return.
You
not
non-integer
responders
from,
for
instance,
you
want
to
get
the
number
of
requests
for
the
last
five
minutes,
so
you
write,
increase
over
number
of
requests
for
the
minutes
and
the
premises
could
write
on
you
1.5,
for
instance,
not.
B
Is
tightly
connected
to
increase
function
and
also
Victory
metrics
implements
it's
quite
differently.
For
instance,
when
you
use
rate
over
a
small
time
window
in
parameters
which
is
smaller
than
the
double
of
scrape
intervals
and
prematures,
you
get
an
empty
graph
and
empty
response
and
victory
metrics,
just
Returns
what
most
users
expect
it
Returns
the
rate
well.
A
B
I'm
aware
of
M3
database
and
I
I
know
it
is
also
written
from
scratch.
It
probably
it
uses
some
kind
of
parameters.
Code,
I,
I,
think
that
they
use
parameters
code
for
query
execution
because
they
declare
that
they
are
100
compatible
with
from
Kill.
So,
if
you
want
to
achieve
such
compatibility,
the
easiest
way
is
to
reuse
the
code
for
query
execution.
So
probably
they
use
parameters
code
for
from
kill,
but
the
storage
layer
is
written
from
scratches
as
I
know
in
M3
other
from
this.
A
B
A
If
you
don't
know
it
and
it's
a
question
from
the
audience,
but
it's
fair
to
say,
if
you're
not
familiar,
we're
not
expecting
to
compare
with
all
the
the
products
out
there.
I
I
have
a
I
have
a
question
that
also
because
we
talked
about
a
lot
about
the
storage,
which
is
obviously
the
time
series
database
is
the
core.
But
it's
not
everything,
even
in
Prometheus
it's
for
me.
This
is
not
just
a
Time
series
database.
B
B
A
B
Yeah
we,
we
are
very
glad
to
see
new
users
in
our
community's
community.
We
have
a
slack
shots
like
the
operating
system.
We
encourage
you
to
join
this
workshops.
I
will
post
a
link
to
it,
so
you
can
find
it
when
on
our
victim,
metrics
GitHub
page,
and
we
encourage
users
new
users
to
use
Victory
metrics
to
file
feature
requests
to
file
bug
reports
and
to
send
pull
requests.
We
are
very
happy
when
users
sent
even
has
seen
very
simple,
pull
requests
which
fix
some
tips
in
documentations.
B
B
A
A
We
have
just
one
minute
left,
so
I
want
to
mention
one
on
the
on
the
on
the
industry
updates
section
I
want
to
just
mention
one
very
interesting
thing
that
I
just
saw
a
few
a
couple
of
days
ago
by
the
by
Chris,
the
CTO
of
the
cncf
that
wrote
the
cloud
native
predictions
of
2023
I
found
them
very
interesting,
including
things
such
as
the
cloud
IDs
becoming
normalized
and
the
standardization
of
phenops
and
wasm
webassembly,
Innovation
and
others.
So
I
highly
recommend
checking
this
out.
A
I'll
put
it
on
the
resources
page,
a
fascinating
sort
of
predictions
listing
that
also
is
very
aligned
with
what
we
see.
What
we
discuss
here
on
the
show
and
and
the
topics
that
we
we
keep
on
discussing
very
interesting
on
that
I
want
to.
Thank
you
again,
Alexander
for
joining
me
on
this
show
and
sharing
your
your
insights.
Thank
you
very
much.
A
Thank
you
and,
of
course,
thank
you
to
all
our
listeners
for
joining
that.
As
for
the
episode
and
as
always,
you
can
find
all
the
episodes
on
the
on
your
favorite
podcast
app
or
on
YouTube.
If
you
want
to
see
that
in
the
videos
videocast
format,
if
you
are
listening
to
the
episode
on
on
the
podcast
and
do
know
that
we
stream
the
episodes
live
on
Twitch
and
YouTube
live.