►
From YouTube: The Great Cardinality Disasters of Our Time - Bryan Boreham, Weaveworks & Chris Marchbanks, Splunk
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
The Great Cardinality Disasters of Our Time - Bryan Boreham, Weaveworks & Chris Marchbanks, Splunk
Many Cloud Native tools generate Prometheus metrics; together they form a great combination to operate and monitor your infrastructure. But sometimes things go wrong: a quirk in the metric labels can make the volume of data explode, and, soon after, your Prometheus will explode too. Chris and Bryan will share their war-stories such as receiving 46,000 simultaneous alerts or squashing the source of 100kB label values. Then, they will provide top tips to avoid this happening to your tools in the future.
https://sched.co/Uadv
A
B
A
So,
as
I
mentioned,
I
work
at
we've
works
and
one
of
the
things
we
have
a
product
offering
is
a
Prometheus
as
a
service.
You
know
it's
part
of
our
weave
cloud
offering
and
what
we
let
people
do.
Is
fire
up
agents
on
their
cluster,
which
are
uploading,
metrics
time
series
data
to
our
back-end,
and
anyone
can
do
this.
You
can,
you
can,
you
know,
take
out
a
free
trial.
A
A
A
Okay,
so
you
just
came
for
like
sympathy
problem
shared
yeah,
so
you
may
have
seen
this
message
in
the
docs
either
before
or
after
you
had
the
disaster,
and
this
is
the
Prometheus
documentation
and
it
says
it
says:
don't
do
it
right,
and
so
that's
that's.
Prometheus
I
mentioned
on
the
slide
maintainer
of
cortex,
which
is
a
scalable
variant
of
Prometheus.
That's
how
we
run
our
back-end,
the
you
know
when
we
talk
to
people
at
m3
and
other
other
metric
systems
they're.
Basically,
everyone
runs
into
this
problem.
So
it's
it's
pretty
generic!
B
How
much
is
too
much
in
terms
of
Prometheus
land
there's
a
couple
of
rules
of
thumb
that
we've
developed
one
is
try
not
to
query
more
than
a
hundred
thousand
series
at
any
time
you
can
write
to
more
than
that,
but
trying
to
query
it
gets
really
slow
by
200,000
series.
A
single
query
is
taking
five
seconds.
Eventually
they
just
start
timing
out
or
get
rejected
by
Prometheus
and
UX
kind
of
sucks
there
and
then
a
single
Prometheus
it
can
handle
about
10
million
unique
series
before
really
the
operational
load
becomes
too
much.
B
That's
a
fair
amount
of
serious.
Oh
that's!
A
lot
of
data,
so
you've
got
some
leeway
most
of
the
time,
but
once
that
carnality
starts
kicking
in
like
you'll
start
seeing
startup
times
taking
15
minutes,
20
minutes.
If
you
lose
an
instance,
it's
a
long
time.
I'll
typically
see
heap
on
the
order
of
hundreds
of
gigabytes
beefy
boxes
like
it
and
then
you
know
a
2x
increase
anymore.
That's
really
bad!
B
When
you're
at
a
200
gigabyte
heap
or
something
like
that,
and
then,
where
you
normally
see,
carnality
issues
being
manifested
in
prometheus
is
commonly
its
memory.
Usage
you'll
see
big
spikes
in
memory
usage
or
just
more
than
you
would
expect
out
of
the
box.
So
that's
sort
of
one
of
those
things
I'm
sure
you'll
see
it
a
couple
times
in
our
slides,
but
first
off
good,
pass
it
over
to
Brian
for
the
first
carnelli
disaster
of
our
talk,
yeah.
A
A
So
we're
monitoring
our
our
service
that
we
run,
and
this
is
kind
of
the
the
sign
that
we're
in
some
kind
of
trouble,
the
so
the
bit
of
the
bit
of
cortex
that
executes
the
rules.
It's
same
same
code
as
Prometheus,
but
it's
it's
in
a
standalone
subpart
of
the
system
called
the
ruler
and
this
thing
was
growing
and
growing
and
growing
over
hours.
And
it
doesn't.
It
doesn't
look
good
right.
A
The
the
prognosis
for
this
process
is
is
terminal,
so
so
we
tried
to
dive
in
and
you
know
what
do
you
do
if
you,
if
you've
got
access
memory
utilization,
you
maybe
take
a
profile
heap
profile
and
try
and
figure
out
where
all
the
memory
is
going
and
that
wasn't
entirely
useful.
It
basically
said
it
was
doing
a
lot
of
queries
and
couldn't
really
figure
it
out
from
there.
A
But
you
know,
given
a
few
hints,
we
we
did
eventually
narrow
it
down
and
figure
out
what
was
happening
so
so
one
of
our
customers
had
I've
changed
specifics
of
the
naming
to
protect
the
the
details.
But
but
let's
say
they,
they
had
a.
They
had
a
particular
interest
in
a
concept
of
an
invalid
tag
and
they
set
up
a
alerting
rule.
If
the.
If
the,
if
there
were
any
you
know
greater
than
zero
invalid
tag,
trouble
was
they.
A
The
underlying
data
was
was
kind
of
turning
over
and
don't
just
they
weren't
just
stable
series,
so
these
people
generated
300,000
alerts
in
in
a
few
hours,
so
that
was
that
was
kind
of
painful.
The
good
news
is
Wow.
If
you
like,
Prometheus,
has
a
fixed
sized
Channel
down,
which
it
deliver
the
alerts
on
their
way
to
pager
duty,
that
channel
overflowed
and
in
fact
dropped.
Almost
all
of
the
300,000
alerts,
so
I
would
not
have
been.
You
know
wanting
to
be
the
owner
of
that
that
phone
that
was
receiving
300,000
notifications.
A
So
so
what
can
you
do
about
it?
Yeah
do
be
careful
with
the
with
the
query
in
your
alerts,
so
one
one
way
to
alert
might
might
be
just
to
to
count
how
many
of
these
things
there
are
and
an
alert
if
there's
more
than
zero
of
them.
So
then
the
the
person
who
gets
the
the
page
can
go
figure
out
what
the
detail
is.
A
If
you
want
some
detail
in
your
alerts,
you
might
want
to
consider
a
call
like
top
K,
which
which
will
give
you
the
the
top
in
this
case
at
the
top
10.
So
you
could
generate
10
alerts
if
you
know
if,
if
some
of
the
detail
is
useful
top
a
thousand,
if
4,000
of
them
are
useful
but
but
please
don't
don't
do
300,000.
B
B
B
This
particular
instance
has
200
gigabytes
of
heap
and
start
around
a
hundred
and
for
the
course
of
a
couple
days
went
to
200
gigabytes,
and
maybe
we
should
have
had
an
alert
on
that,
but
you
know
oops
and
then
everything
and
memory
wise,
because
it's
memory,
you
run
H
a
pairs
of
Prometheus.
In
this
case
we
had
three
of
them,
but
it's
memory.
They're
all
scraping
the
exact
same
things
which
mean
they
have
the
same
data
memory
usage
is
the
same
and
they
all
at
the
same
time.
B
Luckily,
we
have
a
Deadman
switch
alert.
We
stopped
getting
our
alerts,
we're
gonna.
What
got
our
other
alert?
This
is
bad
started
debugging
into
it,
and
this
was
hard
to
solve.
It
was
like
this
was
our
only
clue
was
like
what
what
the
heck
there's
a
time
memory
usage
and
we
found
like
okay
going
around
the
thing
we
found
was
symbol
table
size.
It
grew
from
5
gigabytes
to
something
on
the
order
of
60
gigabytes
before
everything
crashed.
What
does
that
mean?
What
is
a
symbol
table
in
Prometheus?
B
B
B
Second
thing
you
can
do,
is
it
might
take
a
little
bit?
You
know
change
that,
so
we
had
a
I
have
to
do
in
the
meantime
is
put
in
a
real
abling
rule
to
just
drop
the
metric.
Once
you
can
actually
identify
the
metric
put
in
a
relay.
Bolling
rule
drop
the
metric.
You
can't
just
isolate
out
and
drop
the
specific
label,
because
then
you
might
have
multiple
metrics
collapse
into
one,
which
is
our
multiple
series
collapse
into
one,
not
a
great
situation,
so
you
have
to
drop
the
whole.
B
The
whole
thing
and
it's
our
key
thing
here,
like
those
messages
were
coming
from
sto
sto,
is
a
very
well
used.
Well
tested
program
has
bugs
in
it.
This
can
happen
to
any
program
out
there.
I
would
like
to
thank
this.
Do
team.
We
reported
this
issue
and
they
fixed
it
within
a
week
and
a
new
version
up
and
running
us
was
good,
so
very
responsive
from
them.
And
finally,
one
other
key
thing
about
the
symbol
table.
Is
it
doesn't
go
away
until
that
block
is
deleted
or
the
data
is
deleted?
B
So
that's
60
gigabyte,
that's
sitting
there.
You
either
have
to
delete
all
that
data
or
you
have
to
wait
for
it
to
aged
out,
and
that's
very
painful,
so
I
definitely
recommend
keeping
an
eye
on
that
and
not
letting
it
to
grow
too
big,
especially
if
you
run
a
system
like
fan
O's,
where
you
have
to
lower
load
all
historical
data,
it
gets
really
really
big
yeah,
as
ACTU
yeah.
A
A
A
My
next
next
next
we
turn
to
buckets
I'm
I'm
singer,
well,
a
lot
of
blank
stares,
maybe
a
couple
of
nods.
So
let
me
just
explain:
Prometheus
histogram
how
that
works.
Histograms
are
really
useful
for
seeing
something
like
the
the
95th
percentile
of
a
trick.
You
know
compared
to
the
the
median
compared
to
the
mean
really
powerful
feature
of
Prometheus
and
and
the
way
it
works.
Is
you
set
up
when
you
define
the
metric
in
your
code,
you
set
up
buckets
and
each
one
of
them
is
a
counter.
A
So
in
my
first
picture
here,
I
have
a
bucket
at
everything
up
to
20
milliseconds.
Well,
there
actually
weren't
any
of
those.
In
this
example,
everything
up
to
40
milliseconds.
There
was
one
of
those
everything
up
to
80
milliseconds,
a
bunch
of
dots
in
that
bucket
everything
up
to
160,
milliseconds
and
everything
above
that.
So
that's
five
buckets.
So
this
is
in
my
histogram
definition
that
I'm
using
for
this
example.
A
So
if
you
work
with
a
histogram
like
this,
what
you
might
find
is
is
is
when
you
do
the
queries,
the
value
the
value
get
out
of
it,
we'll
just
kind
of
jump
around
between
the
40
millisecond
and
the
80
millisecond
point,
because
almost
all
the
data
is
in
that
bucket.
So
you
might,
you
might
find
there's
no
definition
in
that
range.
So
no
problem
add
a
bucket,
in
this
case
I've
I've
added
one
at
60
milliseconds.
A
Now
we
get
that
extra
extra
definition
in
that
range,
where
there
was
a
lot
of
density
now
them
there
might
be
another
reason
to
use
a
the
this
kind
of
data
collection,
which
is
to
check
your
service
level
agreements
or
the
the
thing
you
would
check
is
the
s
Li
and
you
would
compare
it
against
your
internal
SLO
and
you
might
have
to
pay
out
on
your
SLA.
Everyone's
read
the
book
right.
A
So
so,
let's
imagine
our
SL
Oh
is
100
milliseconds.
Well,
we
can't
answer
the
question
with
those
buckets
so
so
better
put
one
in
put
it
in
a
bucket
at
100
milliseconds.
Maybe
you
find
you
don't
have
much
definition
at
the
top
end.
Maybe
you
know
you
want
to
kind
of
know
whether
whether
those
ones
that
are
above
160
milliseconds
are
like
a
little
bit
above
or
a
lot
above,
and
so
you
add
more
buckets
there
and
you
buckets
bucketz
buckets
and
before
before
long
you're,
you're
pretty
much
swimming
in
buckets.
So
the.
A
The
other
thing
that
you
need
to
watch
out
for
is
the
multiplicative
effect,
so
you
might
have
let's
say:
10
buckets.
You
know
that's
a
very
modest
number,
but
then
you
you
put
some
labels
on
it.
You
you
measuring
all
of
those
buckets
per
entry
point
into
your
system
and
you're
actually
measuring
them
on
all
the
different
nodes,
or
maybe
all
the
different
pods
in
the
system.
A
And
so
let's
say
you
had
10
buckets
and
20
entry
points
and
100
pods,
and
maybe
now
we
want
to
separate
out
the
error
codes
that
can
come
back
and
there's
like
seven
different
error
codes.
They're
all
those
numbers
multiply
up
and
and
before
long
Prometheus
explodes
because
you
know
you've,
given
it
10
million
series.
A
This
is
real
code
from
cubed
DNS,
where
they
define
linear
buckets
is
a
is
a
Prometheus
helper
function
and
it
has
a
start
and
an
increment
and
account,
and
so
they
excellent
resolution,
the
you
know
they
had
a
bucket
at
10
milliseconds,
20,
milliseconds,
30,
milliseconds,
40
milliseconds,
five
hundred
buckets.
They
gave
you.
A
Never
mind
somebody
fixed
it.
I
happened
to
be
able
to
find
that
PR
and
I
switched
it
out
to
exponential
buckets,
which
you
know
goes
up
by
multiple,
which
probably
makes
more
sense
a
lot
of
the
time.
You
probably
you
probably
want
the
fine
detail
at
the
low
end,
and
then
you
know
if
your
DNS
request
takes
10
seconds
and
you
probably
don't
care
whether
it
was
10
or
12
or
you
know
it.
You've
lost
hope
at
that
point.
C
B
Act
four:
this
is
the
final
disaster
of
our
presentation.
So
imagine
your
team,
our
account
service,
instrumenting
everything
in
prometheus.
It's
going
great!
So
if
there's
a
service,
it's
like
libraries
they're
using
they
sort
of
support
Prometheus,
but
not
really
they're.
More
designed
for
this
old-style
dot,
notation!
Well,
okay,
but
I
need
some
tags
in
there,
so
put
them
in
the
names
sure
like
they're
there,
but
at
least
you
can
do
stuff.
So
you
get
metrics
that
look
like
my
service,
blah
some
routes,
some
tenant
status,
codes,
etc.
B
B
There's
been
some
improvements,
it
might
not
break
completely
anymore,
but
for
a
while
it,
like
my
oldest
browser,
crashed
on
me
every
time
I
tried
to
load
from
atheists
and
the
reason
for
this
is
prometheus
and
the
ger
fauna.
You
is
if
you're
trying
to
like
put
in
a
new
query
into
the
Griffons
UI.
They
pull
back
every
single
metric
name
for
the
type
of
head
completion.
It's
a
nice
tool,
I,
don't
know
all
the
names
of
every
metric
in
my
system
like
them.
B
That's
way
too
much
cognitive
load
type
it
in
start
getting
there
great,
like
that's
a
really
nice
feature,
but
some
systems
have
a
two
million
metric
names
and
what's
the
number
on
here
I
think
it
is
one
meg,
some
121
megabytes
of
data
that
you
have
to
pull
back
to
just
to
get
your
metric
names
back
from
the
server
there's
a
lot
of
it's
a
lot
of
data
to
pull
back
to
load
a
simple
little
UI.
It's
a
lot
of
JSON.
B
Yes,
a
lot
of
JSON
so
clearly
in
good
news,
we're
not
the
only
ones
with
who
have
come
across
this
issue
and
there's
actually
I.
Think
I've
seen
three
currently
open
prometheus
issues
around
this
behavior
and
what
to
do
about
it.
Hopefully
some
of
those
will
get
worked
on
before
too
long
yeah,
so
do
not
put
label
values
into
your
metric
names,
not
a
good
idea,
lots
of
different
in
efficiencies.
This
can
also
happen
from
systems
that
are
misconfigure.
You
might
have
a
stats
d
exporter
or
a
jmx
exporter.
B
You
just
grab
a
regex
and
put
it
into
the
name
and
oops.
Now
I
have
weird
data
in
prometheus.
This
is
another
case
of
label.
Names
are
forever
that
you
pulled
it
back
from
all
the
blocks
in
your
database.
So
if
you
do
this
once
you
can
have
to
wait
for
your
attention
to
age
out
for
them
to
disappear
from
type-ahead
or
you
have
to
delete
the
entire
series
with
the
admin
API.
But
there
are
ways
around
this.
Hopefully
you
can
fix
that
the
source,
but
if
not,
you
can
write
some
great
relabeling
rules
too.
A
How
to
travel
a
bit
of
fun
talking
about
the
the
good
times
in
the
in
the
trenches
and
the
sre
trenches,
the
so
yeah?
How?
What
we
wanted
to
do
is
leave
you
with
some
tips
and
ideas
for
what
to
practicality
practically
do
in
the
in
the
system.
Oh
yeah!
So
that's!
That's!
That's
that's
my
message.
It's
unpleasant
and.
A
A
We
we
have
a
query
with
no
metric
name,
so
it's
just
curly
brackets,
nothing,
nothing
before
that
and
then
what's
inside
the
curly
brackets
is
we're
going
to
match
on
name
and
we're
gonna
match
it
with
a
regex
which
is
dot
plus
for
reasons
and
then
and
then
we
count
those
by
name
and
that
way
we
could.
We
could
find
out
what
the
highest
cardinality
is,
but
in.
B
Big
clusters
that
stops
working
if
I
try
to
do
that
query
on
our
Prometheus
instances,
it
just
sits
there
and
spins
and
spins,
and
if
I'm
lucky
it
returns
in
58
seconds
when
our
timeout
is
one
minute
and
if
I'm
not
lucky
you're
on
this,
not
my
smaller
instances
that
sit,
never
gonna
happen.
So
what
do
you
do
instead
on
these
bigger
instances?
Well,
a
few
questions
I
like
asking
myself.
First,
how
many
series
are
in
memory
gives
me
a
good
snapshot.
B
There's
a
metric
Prometheus
exports
for
that
Prometheus
T
used
to
be
head
series
nice
and
helpful
typical,
that's
where
I'll
typically
make
sure
I
try
and
keep
under
that
10
million
mark
that
goes
much
above
10
million,
probably
in
for
a
bad
time
before
too
long
and
ii
talked
about
a
bit
earlier.
How
much
space
do
my
unique
strings?
Take
as
I
said,
TS
DB
symbol
table
size
very
helpful
bit
mysteriously
named,
but
now
you
know
what
it
does.
B
Third
I,
really
like
looking
at
how
many
samples
are
scraped
by
job
there's,
a
internal
metric
called
scrape
sample
scraped
at
every
target.
Tells
you
how
many
and
then
you
can
summit
by
job
you
can
also
do
there's.
Also
a
scraped
set,
scrape
series
added
that
you
can
do
a
sum
over
time
to
see
churn
and
these
really,
instead
of
doing
full
names
and
the
name
matches.
This
really
looks
into
well.
Now,
I
actually
have
a
name.
So
it's
a
much
smaller
data
set
and
this
will
return
Italy.
B
A
A
B
Can
have
a
good
week
ago
is
when
I
released
it
as
a
non
release
candidate.
There's
a
TS
DB
status,
page
built
into
the
UI.
That
will
quickly
tell
you
many
many
different
metrics.
So
now,
let's
take
an
example
of
well,
you
don't
always
have
to
have
carnality
disasters
to
learn
more
about
the
carnality
of
your
system.
So
last
week
you
know
I'm
prepping
for
a
cube
top
con
talk.
It's
like
well,
I
gotta
have
something
so
like
looked
at.
What
the
carnality
of
my
system
is
and
was
like.
B
Whoa,
that
simple
table
side
is
still
like
gig
in
this
environment.
It
goes
for
gigs
in
a
different
environment
where
the
hex
coming
from
well
now,
prometheus
UI
comes
with
this
nice
page,
use
click
status,
runtime
information
and
there's
a
bunch
of
sections.
One
of
those
sections
is
label
names
with
highest
cumulative
label
value
length.
So.
B
A
new
what's
telling
you
is
what
is
the
label
that
has
the
most
values?
This
is
what
we
had
to
add
to
at
that
time,
as
a
TST,
be
analyzed
command
line
tool
to
figure
out
where
those
error
messages
were
coming
from,
like
what
what
label
has
all
of
this
symbol
sighs.
Well,
last
week
we
found
there
is
one
metric
or
one
label
log
file,
with
two
orders
of
magnitude,
more
data
than
anything
else
in
our
cluster.
As
you
can
see,
one
of
those
other
things
is
the
container
ID,
just
literally
just
a
sha.
B
So
that's
a
lot
of
data,
two
orders
of
magnitude,
more
so
great.
Okay.
That
gives
me
a
great
place
to
start
now.
Instead
of
an
empty
name,
regex
match
I
can
do
a
count
of
where
log
file
does
isn't
empty.
So
tell
me
what
all
series
come
back
that
have
this
log
file.
There
are
two
of
them
they're,
both
from
M
tail.
B
For
those
of
you
who
don't
know,
M
tail
is
a
program
that
will
monitor
log
files,
and
you
can
do
some
regex
matching
on
the
log
entries
and
they'll
convert
them
into
metrics
that
you
can
then
scrape
so
for
things
that
maybe
don't
expose
error
message:
errors
as
Prometheus
metrics.
You
can
use
em
tail
to
grep
for
these
error
messages,
whatever
you
really
want,
just
just
regex
matching
and
export
them
as
metrics
super
useful
for
some
cases.
B
Don't
know
if
you've
seen
container
log
pads
and
kubernetes
goes
something
like
var,
log
containers,
pod
name,
container,
name,
no
name,
space,
container
names,
container,
you--it,
dot
log,
it's
pretty
pretty
long
that
that
could
definitely
explain
why
log
file
is
taking
up
so
much
space
so
made
em
tail
all
makes
great
blog
files
that
be
actually
cared
about,
reduce
this
to
seventy
nine.
Instead
of
fifty
thousand
pretty
good-
and
this
is
the
result
you
can
see
on
this
graph-
reduce
the
number
of
containers
we
are
climbing,
climbing,
climbing
and
then
symbol
table
size
starts
decreasing.
B
As
I
said,
you
have
to
wait
and
wait
until
blocks
actually
age
out,
so
it
takes
a
few
days,
but
like
now,
we
are,
we
have
in
this
cluster
I
believe
is
fifty
percent
or
so
or
I.
Guess
it's
a
30%
decrease
so
far
and
one
of
our
other
clusters,
the
symbol
table
just
over
the
last
four
days,
has
decreased
by
almost
50
percent.
That's
a
big
difference
just
for
something
that
was
misconfigured
and
these
carnality
metrics
pointed
towards
this
misconfigured
thing.
B
A
A
Well
yeah,
so
one
of
the
things
about
cortex
is
because
we
have
all
this
history
of
random
people
sending
us
random
data
on
the
Internet.
We
actually
have
a
whole
suite
of
soft
limits
which
are
actually
configurable
power
tenant.
So
you
know
what
you'll
find
if
you're,
if
probably
someone
the
rooms
tried
this
already.
If
you
know,
if
you
sign
up
for
a
free
trial
and
start
sending
us
a
million
series,
you
will
you
just
will
send
you
errors.
A
B
B
There's
a
TST
be
analyzed
tool
that
is,
it
is
included
with
the
in
the
Prometheus
repo
I.
Don't
think
it
is
currently
in
the
Prometheus
docker
file,
though,
but
if
you
pull
down
the
tar
file
it
will
be
in
there.
It
is
soon
to
be
integrated
into
prom
tool
as
well.
So
within
the
next
couple,
release
cycles
we'll
just
be
in
prom
tool
and
then
you'll
actually
have
it
in
your
container
that
you're
running
as
well
yep
beyond
code
reviews.
A
C
B
Is
discussion
of
adding
some
more
limits
into
Prometheus
itself,
such
as
don't
allow
hundred
thousand
kilobyte
error
message?
It
label
values,
so
that
is
something
we
are
discussing
at
least
and
hopefully
we'll
make
it
into
core
Prometheus
at
some
point,
just
some
protections
around
this.
There
are
also
some
things
around
there's
a
scrape
limit
you
can
put
on
to
scrape
jobs
in
Prometheus
to
limit
no
target
can
have
in
this.
D
A
C
A
B
So
the
so
there's
two
options
for
cleaning
the
symbol
table.
One
is
delete
a
block,
not
great.
The
second
is
you
can
figure
out
which
series
are
causing
the
large
symbol
table
and
there's
a
TS
DB
admin
API
that
allows
you
to
delete
based
on
mashers,
so
you
will
have
to
enable
that
API
go
in
very
carefully.
Only
delete
the
certain
ones
from
the
specific
time
range
that
you
care
about,
and
then
you
have
to
do.
B
A
second
API
call
to
say:
do
a
compaction
now
and
thick,
and
then
the
compaction
process
will
work
on
that
data
and
remove
it
permanently
from
your
database,
at
which
point
this
it.
The
symbol
table
is
for
those
strings,
is
in
trees
in
trees,
for
those
strings
are
no
longer
necessary
and
you
will
be
in
a
better
spot.
That's
the
safer
way
to
do
it,
be
careful.
It's
an
admin,
API
for
a
reason,
all.