►
From YouTube: Loki Community Meeting 2022-04-07
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
All
right
welcome
everyone
to
today's
episode.
Was
it
definitely
not
counting
months
on
my
fingers
april,
7th
2022.,
so,
first
off
we
have
the
2.5
release
coming
out
imminently.
What
do
you
say
next
couple
days.
B
It's
gonna
be
today,
no
matter
what
oh,
okay,
so
wow,
we
were
kind
of
playing
a
little
game
of
so
there's
a
go
release,
sort
of
ominously
that
has
a
private,
secure
and
undisclosed
cve.
That
is
part
of
it,
and
that
was
supposed
to
come
out
today.
So
we
were
kind
of
keeping
our
eye
on
that
and
then
they
just
updated
the
mailing
list
to
say
that
they're
having
trouble
with
their
release
tooling.
So
that's
going
to
be
tuesday.
B
So
we're
not
going
to
wait
for
that
anymore.
So
now,
as
I
was
joking
earlier,
I
just
need
to
get
off
of
this
call
and
a
call
or
two
after
that
and
then
we'll
circle
back
around
and
tag
it.
But
it's
ready
release
notes
are
ready
and
getting
reviewed.
The
branch
is
ready
and
everything
that
we
wanted
to
backboard
is
back
ported,
so
the
I
copied
the
release
notes
in
here.
B
So
if
this
won't
publish
probably
before
the
release,
I
think
just
those
that
are
here
a
lot
of
improvements
for
performance
in
this
release,
which
is
really
exciting.
So
we've
got
a
forked
version
of
the
go
standard,
lib
regex
library,
which
makes
a
number
of
improvements
around
how
loki
uses
regular
expressions
owen.
Did
this
work
here
to
make
binary
operations
significantly
faster
by
leveraging
the
query
splitting
and
sharding
for
multiple
parts
of
the
binary
operations?
B
There's
new
schema,
so
v12
schema
if
you're
running
on
mostly
probably
will
only
be
interesting
to
you.
If
you're
running
on
s3
s3
has
rate
limits
that
apply
on
a
prefix
and
the
the
11
and
older
schemas
I'll
put
all
of
the
chunks
in
one
prefix,
so
you
can
run
into
rate
limits,
usually
on
reads:
if
you're
pulling
in
something
like,
I
was
at
5500
requests
a
second
which
you
can
actually
do
with
a
big
enough
loki
cell
with
enough
parallelization.
B
So
the
v12
schema
breaks
up
the
storage
into
more
prefixes
so
that
you
can
get
more
parallelism
without
getting
rate
limited.
That
was
also
ported
to
the
file
system
store,
mainly
because
you
know,
we've
seen
feedback
and
I
totally
get
it
we're
having
all
of
the
chunks
in
one
directory
is
sort
of
non-optimal
operator
experience.
If
you
have
millions
of
files
in
a
directory,
linux
gets
a
little
grumpy.
A
Right
so
the
v12
schema
was
very
it's
just
an
easy
way
to
hook
in
to
the
to
the
ability
to
fix
it.
Moving
forward,
yep.
B
So
just
took
advantage
of
that
hedging
storage
request
is
pretty
neat.
I
looked
for
the
weirdo
because
the
tempo
did
this
first
and
I
thought
they
had
a
blog
post
directly
about
it,
but
I
can
only
find
their
release
notes,
but
the
idea
is
you
make
a
request
to
pull
a
chunk
from
the
object
store,
which
you
might
do.
Thousands
or
tens
of
thousands
of
in
a
big
query,
sometimes
they're
slow,
and
we
basically
you
know
we
do
them
a
lot
in
parallel.
B
But
at
some
point
you
always
have
to
wait
for
some
amount
of
ordered
returns
before
you
can
continue,
and
you
can
see
that
like
if
most
requests
take
100
milliseconds,
but
one
takes
like
three
or
four
seconds
that
that
three
or
four
seconds
often
dominates
the
the
query
performance.
So
what
hedging
does?
Is
you
largely
configure
it
around?
B
What
your
kind
of
p99
normally
is
say
like
500,
milliseconds
or
200
milliseconds
or
like
what
your
sort
of
normal
request,
maybe
not
maybe
like
a
p50
and
if
a
request
takes
longer
than
that
value
or
that
time
it'll
send
another
one
with
the
theory
that
that
one
will
likely
return
quicker,
and
so
then,
whichever
returns
quicker
gets
the
result.
So
it
basically
gives
the
loki
a
chance
to
say
hey.
B
Prom
tail
get
a
bunch
of
new
ways
to
ingest
logs,
including
docker
daemon
service
discovery.
So
this
was
added
in
upstream
prometheus.
You
can
use
it
a
couple
ways
you
can
actually
use
it.
The
way
we
typically
do
file
service
discovery.
So
you
can
tell
it
to
ask
the
doctor
damon
what
the
name
of
the
containers
are
and
use
that
to
construct
pass
to
tail
files,
but
it
also
has
the
ability
to
essentially
query
the
the
logs
from
the
docker
api.
So
there's
a
logs
endpoint
in
the
docker
api.
B
So
you
can
do
it
that
way
as
well.
I
believe
the
pr
mostly
focuses
on
that
approach.
Then
you
don't
have
to
play
around
with
access
to
the
file
system
and
then
there's
ways
to
pull
in
logs
from
cloudflare
as
well
as
the
greylog
extended
log
format
you
can
set
up
a.
I
can't
remember,
I
think
it's
a
udp
listener
to
receive-
maybe
it's
both
tcp
and
udp,
but
receive
logs
in
that
format
and
ingest
them
and
then
last
things
from
prom
tail
are
the
ability
to
do
rate
limiting
on
the
client
side.
B
So
if
you
wanted
to
limit
how
much
one
individual
prom
tail
can
send,
I
know
lots
of
folks
have
environments
where
the
you're
sort
of
operating
infrastructure
on
behalf
of
other
people,
and
you
can't
always
get
them
to
do
what
they
want,
and
you
know,
depending
on
how
the
multi-tenancy
setup
is
in
loki
you.
This
is
a
way
to
prevent
a
user
of
a
shared
tenant
infrastructure
from
overwhelming
the
limits
that
might
be
applied
on
the
server
side.
B
So
this
gives
a
couple
more
options
there
for
people
operating
and
running
low-key
infrastructure.
That's
it
2.5.
Anybody
have
any
questions,
a
bunch
of
fixes
in
there
too,
like
a
whole
lot
of
fixes.
I
strongly
recommend
upgrading,
oh
if
you
upgrade,
like
probably
99
of
you,
are
gonna
find
that
we
made
one
change.
B
That
is
a
bit
intrusive,
which
is
there's
a
configuration,
called
split
queries
by
interval
and
it
was
previously
could
be
defined
in
two
places
in
the
query,
range
config
block
and
in
the
limits
config
block
in
loki
2.5
loki
will
fail
to
start
if
it's
defined
in
the
query
range
config
block.
So
it
kind
of
went
back
and
forth
on
how
to
do
this.
We
went
with
the
kind
of
pull
off
the
band-aid
approach
and
basically
you
need
to
define
it
only
in
the
limits
config
section.
B
So
what
we
did
is
update
loki
to
read
from
the
limits
config
section
for
this
value
they'll
be,
and
this
is
in
the
upgrade
guide.
You
don't
have
to
remember
this,
but
but
if
you
start
it
and
it's
in
the
spot,
it
actually
just
spits
out
an
error
message,
and
it
tells
you
what
you
need
to
do
aside
from
that,
there
are
no
other
changes.
That
should
be
sort
of
you
know
noticeable
or
stop.
B
You
know
invasive,
I
guess,
but
we
did
change
a
lot
of
defaults
around
how
parallelism
between
2.4
and
2.5
we've
really
pushed
the
direction
of
making
parallelism
by
default
even
in
the
single
binary
modes.
So
any
queries
that
you
run
will
get
broken
up
into
pieces
and
parallel.
So
in
a
single
binary,
it's
all
within
you
know
threads
on
the
same
machine,
so
you
probably
will
see
increases
in
memory
and
cpu
on
queries,
specifically
the
configs
that
we
changed
are
in
the
upgrade
guide
too.
B
A
Or
there's
like
a
big
trend
here
right
of
trying
to
get
closer
to
more
opinionated
and
saner
defaults,
and
we
learn
a
lot
about.
A
You
know
what
makes
more
sense,
the
more
that
experience
that
we
get
building
loki
and
operating
it,
but
I
think
a
lot
of
these
will
be
generally
very
helpful
for
people.
The
honestly,
like
the
biggest
one
that
I'm
looking
for
here
for
from
this
release,
asides
from
all
just
like
the
performance
improvements
bug
fixes
just
like
all
these
quality
of
life
benefits
is
the
new
schema.
That's
coming
out
right.
We've
had
this
problem
with
aws.
A
B
Anybody,
that's
not
done
schema
upgrade
before
it's,
you
add
a
new
entry,
so
the
schema
is
a
roll
forward
type
setup,
so
anything
that
you've
created
to
date
would
be
addressed
and
found.
With
the
current
schema.
You
would
add
a
new
entry
in
the
config
with
a
new
start
date.
That
start
date
needs
to
be
in
the
future.
B
So
you
want
it
to
be
say
tomorrow
and
then
it
rolls
over
at
gmt,
it'll
start
reading
the
new
schema
and
creating
things
with
the
new
schema
and
then
we'll
handle
sort
of
querying
back
and
forth
between
the
old
and
new.
Don't
really
have
a
way
to
migrate
between
schemas
you
you
actually
can.
I
have
done
this
there's
a
migrate
tool
in
the
loki
repo
that
I
made
and
there's
a
big
old
warning
at
the
top
of
it.
That
says,
like
you,
know,
caveat
mtor.
Basically,
so
it
does.
B
You
know
just
it's,
you
can
do
it.
It's
just
maybe
don't
start
with
your
most
valuable
data.
Actually.
A
You
know
do
this,
make
sure
that
you
ping
at
ed
welch
on
public
slack.
He
will
make
sure
to
take
time
to
address
any
concerns.
B
So
a
bunch
of
folks
joined
after
we
started
talking
so
I'll
go
circle
back
around
2.5.
Isn't
actually
I
haven't
cut
the
tag.
Yet
it's
going
to
happen
and
probably
another
I'll
say
two
hours,
because
I
get
a
call
after
this
too.
So
in
a
few
hours
time,
they'll
be
released
for
images
and
tags
for
2.5.
B
We're
gonna
have
a
blog
post
about
it
tomorrow,
so
maybe
tomorrow,
and
so
I
could
do
a
nice
friday
upgrade
sounds
like
fun
to
me.
Yeah.
C
B
C
A
You
can
also
ask
questions
at
the
end
too,
totally
fine
yeah.
So,
let's
I
did
not
do
as
good
of
a
job
with
this
description,
as
ed
did
notably
missing
a
lot
of
links
here.
Is
there
an
example
of
the
new
structure
in
s3?
A
Yes,
I
can
find
one
after
no,
what
actually,
let's
see.
B
I
think
the
long
and
short
of
it
is
that
the
stream
fingerprint
hash
is
now
part
of
the
prefix,
so
it
used
to
be
index.
Slash,
sorry,
you
would
be
tenant,
slash,
tenant
id
or
was
it
actually
called
tenant?
Sorry,
you
feel
like.
I
should
know
this,
but
there's
a
high
level
folder.
I
think
maybe
chunks,
slash
tenant
id
slash
all
of
this
in
id
chunks
and
now
it's
going
to
be.
Are
you
looking
at
it.
B
You
got
it
yeah
and
then
so
the
fingerprint,
so
we
actually
played
around
with
a
lot
of
versions
of
this
and
like
having
more
and
we
went
with
the
approach
of
maybe
taking
it
simpler
to
start,
but
basically
every
stream
now
gets
a
folder
within
the
tenant
or
the
user.
In
this
case
it
says
folder.
B
It
makes
it
seem
like
it
was,
you
know,
really
simple.
A
lot
of
work
went
into
sort
of
playing
around
with
yeah
and
testing
and
benchmarking
and
trying
to
say
we
were.
I
don't
remember
what
the
number
of
reads
per
second,
we
did
do
some
benchmarking
on
this.
I
don't
know
this.
D
A
D
This
simpler
approach,
too,
was
great
because
we
did
some
forward-looking
thinking
and
experimenting
as
well
and
as
far
as
you
know,
we
could
tell
this
supports
the
vast
majority
of
use
cases
for
even
large
low-key
deployments
on
using
s3.
But
it
was
good
for
the
loki
team
as
well
to
experiment
and
learn
more
and
we
even
have
future
ideas.
If
we
get
to
a
scale
where
somehow
we
need
to
even
support
even
better
request,
parallelization
s3
so.
A
Yep
the
big
difference
there
being,
even
though
it's
very
small
change
right.
We
just
basically
replaced
a
colon
with
a
slash
s3's.
The
way
that
s3
prefixes
work
turns
out
to
make
this
much
better
for
rate
limiting
purposes
and
it
hopefully
we'll
see.
This
problem
largely
disappear
for
the
for
the
users
who
have
been
having
that
problem.
A
Okay,
yeah,
so
we've
been
in,
for
I
don't
know.
Maybe
a
year
ago
we
started
kind
of
identifying
some
bottlenecks
and
kind
of
issues
with
the
loki
storage
layer
and
had
come
up
with
a
couple
kind
of
internal
designs
for
what
we
could
do
and
recently
that
materialized
in
a
plan
to
to
write
a
new
index
for
loki
largely
based
on
tsdb,
which
is
a
part
of
the
prometheus
project.
A
I
guess
two
months
now
or
so,
and
it'll
function
very
similarly
to
how
bolt
db
the
voltage
shipper
works
in
that
it
lives
on
in
object,
storage,
alongside
with
your
chunks
and
so
it'll,
be
again
operationally
very
much
simpler
compared
to
the
old
way
of
operating
loki
where
you
had
a
dedicated
index
store,
but
the
benefits
here
being
that
it
should
be
much
much
faster
and
should
require
less
resources
to
run
so
we'll,
particularly
on
some
of
our
larger
clusters.
A
A
A
This
is
relevant
for
the
series
series
endpoint
in
loki,
which
can
basically
for
a
set
of
matches,
can
give
you
all
of
the
resulting
series
that
satisfy
that
request,
and
this
is
used
in
things
like
grafana
autocomplete,
so
historically
we've
the
long
tail
on
these
can
be
the
long
tail
latencies
can
be
very
high,
and
now
we
should
be
able
to
run
these
off
the
index
alone
and
hoping
we.
I
can
share
some
more
actual
performance
numbers
when
we
get
that
far.
But
the
expectation
is
that
it'll
be
it'll.
A
Another
thing
that
we're
going
to
be
getting
out
of
tscv
this
tsb
work
is
the
ability
to
do
dynamic
sharding.
So
one
of
the
ways
in
which
loki
makes
queries
fast
is
by
parallelizing
it
across
across
the
all.
The
labels
of
your
series
and
historically,
a
shard
factor,
is
coded
into
the
index
itself
generally.
This
is
16
by
default
for
the
recent
for
the
recent
schemas.
A
That
is
very
helpful,
a
lot
of
the
time
but
depending
on,
but
it's
a
cluster-wide
control,
which
means
that
if
you
have
a
tenant
with
a
very
small
amount
of
data,
maybe
it
would
be
sub-optimal
to
shard
at
all.
Maybe
you
don't
even
need
to
have
that
optimization.
It
might
work
well
for
some
tenants.
Then
some
tenants
might
be
large
enough
that
16x
isn't
really
enough
for
some
of
the
queries
that
they
run.
A
So
the
idea
here
being
that
we
can
move
this
from
a
cluster
level
problem
to
a
request
level
problem
so
that
we
can
say
okay
loki
can
process.
You
know
for
an
average
query:
400
megabytes
a
second.
It
can
redone
any
logs.
So
what
we
actually
want
to
do
is
we
want
to
figure
out
how
much
data
we
are
operating
on
and
then
shard
down
until
we
get
close
to
that
number
and
then
by
doing
that
at
a
request
level,
it
accounts
both
for
tenants
which
have
different
throughputs.
A
A
The
other
thing
is
that
this
paves
the
way
for
a
whole
host
of
future
optimizations,
which
I
just
talked
about
some
of
around
query
planning,
but
also
things
like
features
which
can
run
off
the
index
alone,
meaning
that
they'll
be
very
cheap
to
run
because
you
don't
have
to
pull
a
lot
of
logs
from
s3
or
whatever
your
object.
A
Storage
is
and
that'll
be
very
attractive
for
things
like
label
and
cardinality
analysis,
which
I
think
would
be
really
great
features
in
the
future,
as
well
as
like
downgrading
our
existing
log-volume
histogram
queries,
which
are
a
relatively
new
addition
to
grafana,
which
will
basically
show
you
the
distribution
of
logs
for
some
query
over
a
period
of
time
and
often
or
not
often,
but
occasionally
the
long
tail
again
here
is
very
problematic.
So
you
can
have
a
query.
That
is
maybe
some
filter
query,
which
seems
very
reasonable.
A
So
there's
kind
of
a
lot
of
things
that
we'll
be
able
to
do
afterwards
and
it
should
allow
us
to
build
new
richer
features
on
top,
which
I'm
very
excited
about.
When's.
It
going
to
be
ready,
owen,
yeah,
hoping
to
have
it
running
internally
on
some
grafana
clusters
end
of
month,
but
I
need
to
need
to
close
slack
a
little
bit
and
get
some
of
that
work
done.
Yeah.
B
B
We
started
by
using
a
nosql
store
and
requiring
it
separately,
so
you
had
to
have
cassandra
or
bigtable
or
dynavodb,
but
there
was
always
this
bolt
dbe
version
that
you
could
use
in
the
local
config
so
that
in
sandeep
it's
a
bit
late
in
the
day
for
for
him
to
join,
did
a
bunch
of
work
with
what
we
call
bolt
db
shipper
that
takes
the
bolt
db
file
and
just
uploads
it
to
the
object
store
and
that's
how
we
largely
run
loki.
B
However,
it's
basically
a
key
value
store
that
we've,
you
know,
used
to
store
our
data
and
then
you
have
what
is
the
tsdb
from
prometheus?
That
is
a
database
built
around
the
sort
of
label
model
and
data
that
we
store.
So
it's
a
highly
tailored
index
for
exactly
what
we
do,
except
it
stored
postings
offsets
to
blocks
that
go
into
the
prometheus
tstb
format
or
something
like
that.
Some
of
those
words
are
right.
B
Anyway,
we
changed
it
to
support.
Looking
at
where
our
you
know,
s3
objects
are,
but
what
it
does
really
well,
which
is
nice.
Is
it
interns
all
the
label
names
in
one
place,
so
you
get
a
huge
size
reduction
when
you
have
so
in
our
current
index.
You
know,
there's
some
amount
of
deduplication
or
de-amplification
done
on
labels,
but
it's
not
nearly
as
effective
as
what
the
string
interning
can
do
in
prometheus
tsdb,
so
indexes
get
a
lot
smaller.
B
We
should
be
able
to
support
a
lot
more
streams
and
it
can
answer
a
lot
of
the
queries
like
was
talking
about
much
quicker
about
things
like
series
and
labels
and
values,
and
then
we're
going
to
punch
a
bunch
of
metadata
in
there
to
be
able
to
help
do
more
query,
planning
and
things
so
very
exciting,
quite
a
bit
of
work
to
go,
but
it
is
actually
kind
of
real.
I
mean
it
is
there's
real
code.
It
really
works.
A
B
Quite
in
a
cluster
yet
but
very
exciting,.
A
Yeah,
we
are
pretty
happy
with
the
the
kind
of
where
tsdb
is
for
us
right
now,
we're
using
kind
of
a
fork
at
the
moment,
definitely
looking
to
at
ways
in
which
we
can
recontribute
a
lot
of
this
stuff
upstream,
but
for
the
moment,
we're
working
off
of
a
fork
and
all
that's
in
loki
right
now.
It's
just
not
wired
up
to
the
rest
of
the
components
in
our
existing
store
structures
and
all
of
the
really
unfun
work
that
I've
got
to
do
left.
B
We
so
for
sort
of
for
like
in
the
similar
time
period,
but
the
2.5
release
actually
has
no
dependency
on
cortex
anymore.
We've,
forked,
the
remaining
software
that
was
shared,
so
we
used
within
cortex,
primarily
the
chunks
store
which
cortex
deprecated
use
of-
probably
I
don't
know
a
year
ago,
or
so
when
they
moved
to
the
tstb
block.
B
B
We
then
went
through
and
forked,
basically
everything
else,
largely
for
self-serving
reasons,
because
we
vendor
internally,
when
we
make
so
we
have
the
enterprise
version
of
loki
that
bolts
on
some
other
features
and
of
our
other
projects
and
the
vendoring
of
those
was
as
a
nightmare
so
that,
basically
it
was
you
know
loki
depending
on
cortex
grafana
enterprise
metrics,
depending
on
cortex
those
both
depending
on
this
proprietary
code,
and
it
became
so
difficult
to
keep
those
things
updated
that
the
kind
of
few
packages
that
weren't
they
were
still
overlapped.
B
Like
the
query
front
end
in
the
ruler,
we've
forked
them
into
loki,
so
we
may
look
to
see
if
we
can
break
them
back
out
into
a
shared
package
with
mirror
or
we
might
just
keep
them.
It's
not
a
huge
amount
of
code
at
this
point
that
was
that
was
shared.
So
a
lot
of
these.
A
B
Yes,
if
you
want
to
build
a
distributed
system
and
use
a
hash
ring
to
solve
all
your
problems,
like
we've
done,
there's
the
grafana
ds
kit
repo,
which
is
where
the
code
for
that
stuff
lives.
It
actually
has
some
other
useful
stuff,
like
the
runtime
overrides
that
we
use
for
reloading
configs
at
runtime
and
the
services
model
that
loki
uses
for
essentially
bootstrapping
components
internally
with
services.
All
that
stuff
went
into
a
shared
library.
To
that
we
use
with
all
of
our
projects.
A
Amen
all
right
unless
there
are
no
more
questions,
I
think
we're
going
to
talk
about
helm,
charts,
okay,
alex.
G
Hello,
can
you
guys
hear
me
yeah,
yeah,
yeah,
first
time,
caller
a
long
time,
lurker
glad
to
be
part
of
the
call.
I
just
wanted
to
ask.
G
I
posted
a
question
on
the
slack
loki
dev
channel,
maybe
about
a
month
ago
in
our
environment,
we're
trying
to
deal
with
some
mtls
challenges,
and
there
was
one
little
kind
of
kind
of
strange
issue
that
we
found
when
we
were
trying
to
secure
the
querier
to
the
query,
front-end
communication
and
we
we
tried
to
make
some
kind
of
a
you
know,
a
quick
patch,
and
I
pushed
it
up
to
the
repo
or
up
to
my
own
repo,
and
I
put
it
into
the
issue,
but
I'm
just
not
sure.
G
If
there's
some,
I
don't
know
some
suggestions,
anyone
can
offer
for
how
we
might
go
about
fixing
this.
Is
this
a
ds
kit
problem,
I
guess
with
cortex
being
completely
forked,
I
don't
know
if
we
could
contribute
something
to
the
loki
repo
now,
not
sure.
If
anyone
is
familiar
with
the
issue,
it's
number
5592.
B
I
do
know
that
there
are
problems
trying
to
do
mtls
between
all
of
the
components
and
it's
it's
all
largely
possible,
but
the
configs
weren't
exposed
everywhere.
Maybe
that's
what
you
ran
into
as
well,
so
I
mean
it
sounds
like
we
just
probably
need
to
get
your
pr
merged
or
get
apr
with
your
changes
merged.
If
it's
not
already
in
a
pr,
because.
G
The
interesting
yeah,
because
the
interesting
thing
is
that
it
does
work
for
most
of
the
services
it
just
it
seems
like-
and
I
was
looking
into
this
like
a
month
or
two
ago,
so
I'm
kind
of
forgetting
the
details,
but
I
think
there
was
one
one
communication
that
was
going
through
this
ds
kit,
library
and
that
library
ended
up,
creating
doing
this
kind
of
forces,
the
the
dns
lookup
of
the
host
name
and
then
connects
with
the
ip
address.
G
So
when
we
were
trying
to
do
this
with
mtls
with
certs,
then
everything
was
failing,
but
everything
else
seems
to
go
through
just
the
regular
grpc
libraries
like
I'm,
not
a
goal-line
expert,
so
you
know
bear
with
me.
I.
A
A
And
are
you
running
without
a
query
scheduler.
G
G
Are
you
running
a
kubernetes
for
this
one?
No
okay,.
A
In
kubernetes,
we
use
something
in
an
option
on
the
services
themselves
which
allow
you
to
basically
publish
addresses
that
are
not
considered
ready
yet,
and
so
there's
chicken
and
egg
problem
between
whatever
the
queues
are
hosted.
In
your
case,
this
would
be
the
query
front
and
the
queries
which
are
responsible
for
for
dequeuing
work
from
the
cues
and,
and
you
know,
executing
the
queries
where
the
front
ends
or
the
or
the
schedulers.
A
G
Yeah
I
mean
this
is
all
kind
of
basically
bare
metal
set
up,
so
the
host
names
are
are
valid.
It
just
seems
to
that.
It
gets
resolved
to
the
ip
address,
and
it's
in
this.
So
I'm
trying
to
remember
where
this
was.
B
So
the
code
that
you
referenced
in
this
issue
is
forked
now,
so
it
probably
would
be
fairly
easy
for
us
to
add
the
config
I'm
just
looking
like
it
looks
like
you're
disabling
the
dns
lookup.
To
so
I
mean
I
guess,
I
need
to
understand
a
little
bit
more
about
what
actually
caused
that,
but
I
don't
know
it
seems
like
we
ought
to
be
able
to
do
this.
F
F
Okay
and
the
fix
is
the
dns
watcher
that
might
be
able
to
fix
this.
G
Yeah
because
I
think,
like
our
environment,
like
at
least
for
this,
this
setup
that
we're
trying
to
get
working,
it's
it's
relatively
static.
So,
like
you
know,
just
from
under
looking
at
the
code
like
this
dns
watcher,
it's
kind
of
assuming
a
pretty
dynamic
environment
where
this
could
change
and
the
ips
get
resolved.
But
in
our
case
we
just
want
to
get
like
a
relatively
simple.
G
G
Yeah
exactly
yeah,
that's
so
we're
able
to
get
we're
able
to
make
this
work
around.
We
can
do
the
tls
and
secure
skip
verify
to
true,
and
then
everything
works,
but
it's
connecting
with
the
ip
address.
Kaiji.
B
Yeah
that's
kind
of
an
interesting
problem
because
the
yeah
right.
I
think
this
is
what
owen
you
were
saying,
because
we
use
the
dns
entry
to
resolve
a
pool
of
front
ends,
and
then
you
would
make
individual
connections
to
each
of
the
front
ends,
which
is
why
it
does
it
by
ip
and
not
dns,
because
the
dns
would
resolve
so
that
yeah
like.
If
you
wanted
to
run
more
than
one
front
end
worker,
then
I
don't
think
you
can
do
this
this
way
or
we
need
another
way
to
do
that.
A
Is
why
we
don't
see
this
in
other
components
that
connect
to
the
ips
themselves?
Well,.
A
F
If
there
was
a
way
to
configure
this
to
pass
it
through,
but
yeah
I
mean
our
problem.
We
don't
run
into
this
because
we
don't
do
mutual
dls.
So
it
doesn't
matter
that
we're
connected
to
the
ips
right,
because
there's
no
certs
to
check
the
host
name
against.
A
Yeah,
but
if
you've
and
if
we
have
mtls
working
distributor
to
ingest
or
require
to
ingest
or
right,
I'm
curious
why
it's
not
working
here
right,
because
I
think
it's
the
same
problem
so
just
be
right
like
we
can
just
copy.
There's,
I'm
guessing,
there's
a
difference
there.
Then
we
don't
need
to
solve
it
in
a
different
way
right.
It
should
just
work
yeah.
G
As
far
as
I
can
tell
from
what
I,
I
think
the
connection
is
different,
so
it's
just
from
the
query
front
end
to
the
query.
Sorry,
from
the
core
ears
to
the
query
front
end
that
it
goes
through.
I
believe
this
ds
kit
library,
whereas
I
think
some
of
the
other
connections
they
seem
to
be
just
using
the
straight
jrpc
connection.
As
far
as
I
could
tell,
and
I
think
that
code
path
is
where
this
ip
gets
resolved.
As
far
as
I
could.
A
That
would
that
kind
of
I
mean
that
would
be
consistent
with
what
we
were
talking
about
right,
like
the
other
two
work,
and
this
one
has
a
slightly
difference
in
code
path.
That's
probably
where
we
should
look.
B
So
if
you
have
one
front
end,
which
is
probably
what
you
have,
then
this
is
simpler,
but
we
usually
run
multiple
front
ends
and
to
make
the
mtls
happy
the
certificate
will
either
have
to
contain
the
ip
address
of
what
you're
connecting
to
or
a
dns
entry
of
what
you're
connecting
to,
which
means
a
separate
dns
for
every
front
end
or
the
ip.
So
yeah
there's
not
an
easy
way
that
I
can
think
of.
F
That
would
be
like
a
config
change
and
we'd
have
to
pass
that
all
the
way
down-
and
I
don't
know
if
it's
worth
it
like
support
this
case.
But
I
think
that
would
be
the
simple
solution
here,
because
you're
right
anything
what
you
actually
want
to
get
the
addresses
and
the
certs
and
everything
all
that
to
line
up
and
match
is
gonna
change.
Your
topography.
A
B
Yeah
I
mean
the
distributors
are
going
to
go
to
the
ring,
pull
back
all
the
ips
and
make
a
connection
on
ips
so
for
mtls
to
work
there.
You
still
have
to
have
a
cert
that
I
don't
know
if
you
can
create.
This
is
not
something
like.
Can
you
create
a
an
mtls
cert
that
accepts
ips
and
ranges
of
ips
or
to
say
I
don't
know
that's.
The
question
is
for
someone
to
make
that
work.
They'd
have
the
same
problem,
though
you're
right.
I.
B
Long
as
you
had
like
the
cm
ask
this
question,
which
is
we're
just
kind
of
because
there
wasn't
much
else
on
the
agenda
just
exploring
this,
because
it's
interesting,
but
anybody
has
anything
else
or
maybe
we
should
touch
in
on
the
we're.
Probably
gonna
have
to
pull
this
one
offline
and
figure
out
what
we
can
do
with
it.
But
it's
it's
a
great
question
alex
I
don't.
Mtls
is
really
annoying.
B
I
mean
it's,
it's
tough
because
loki
has
numerous
interconnections
between
components.
So
yeah
it's
a
bunch
of
work.
So
that's
you
know,
do
you
have
it
working
between
distributors
and
I.
G
I
mean,
I
believe,
it's
working,
I
hope
so
otherwise,
but
yeah,
I'm
pretty
sure
it's
working.
I
I'm
sorry.
I
didn't
I'll
I'll
try
to
confirm
my
understanding
because
it
was
a
month
or
two
ago
that.
B
Was
the
one
spot
that
I
remembered
correctly
was
the
because
you
have
the
ingestor
client
and
I
don't
remember
if
we
exposed
the
there's
an
adjuster
client
config
that
you
needed
to
expose
the
place
to
add
a
cert
to,
and
my
recollection
was
that
we
hadn't
passed
that
config
through
so
that
was
one
place
you
couldn't
get
mtls
that
may
have
changed.
Somebody
may
have
it
was.
It
was
a
change
to
just
expose
the
config,
but
so
someone
may
have
done
that
and
I
didn't
notice
it.
B
But
yeah
I
mean
it
is
on
our
list,
like
lots
of
people
for
good
reason
want
to
have
tls
and
mutual
tls
between
components.
I
can't
say
that
it's
anywhere
on
the
short
part
of
that
list,
so
it's
nice
that
folks
are
helping
us
move
that
forward.
G
F
Always
this
is
real
short
vlad
wanted
me
to
to
mention
some
openshift
support
that
he
recently
added
to
the
home
chart.
So
if
you're
deploying
via
helm
and
openshift,
you
can
now
specify
the
right
pod
security
policies
and
get
the
right
service
accounts
through
that
you
need
for
it
to
actually
work
and
just
in
general
there
you
know
the
helm.
Charts
have
always
been
sort
of
a
second
class
citizen.
F
If
you
will,
because
we
don't
use
them
ourselves,
but
there
are
a
lot
there's
a
lot
of
activity
going
on
there,
there's
new
enterprise,
logs,
simple
scalable
and
and
there's
improvements
happening
to
the
other
ones.
So
if
you
do
use
helm,
take
a
look
at
them
because
they're
getting
in
better
shape
than
they
have
been,
and
if
you
don't
use
helm,
I
don't
believe
you.
A
All
right
does
anyone
have
any
any
other
questions
or
points
to
discuss
today.
B
B
Didn't
change
the
vlog,
it
was
hard
for
me
because
we
introduced
bugs
and
then
fix
them
between
the
releases
and
kind
of
narrow
those
down,
but
no
the
improvements
there
are
some
fixes
in
there
are
some
bugs
in
the
querying
around
around
duplicates.
A
Yeah
are
we
gonna
backboard
that
I
actually
need
to
finish
that.
B
B
So
deduplicating
stuff
is
already
kind
of
its
own
effort
or
it's
you
know
if
you
use.
So
if
you
have
a
replication
factor
on
and
you
have
a
lot
of
entries
that
have
the
same
timestamp
and
the
same
stream,
so
lookie
will
accept
entries
with
the
same
timestamp
for
the
same
stream
as
long
as
the
values
are
different
and
there
were
some
issues
where
we
weren't
properly
deduplicating.
All
of
that,
so
that's
exists.
Still
in
2.5,
that'll
be
out
in
a
2.5.1,
not
too
long,
but.
B
But
there
were
some
other
issues
around
somebody
discovered
along
the
way
that
if
you
take
a
query,
should
we
write
this
into
our
canaries.
But
if
you
do
the
sum
of
a
rate
of
say,
name,
space
equals
foo
and
then
divide
that
by
the
sum
of
the
rate
of
namespace
equals
foo.
You
would
expect
the
result
to
be
one
and
it
turns
out
in
some
cases
it
wasn't
so
we
fixed
a
lot
of
the
bugs
related
to
that.
B
A
Yeah
tall
aspiring
programmers
out
there,
this
is
a
trick
that
cyril
taught
me
you
make
sure
to
include
a
bug
so
that
when
you
fix
it
later,
you
look
very
good.
B
B
A
Kind
of
tested
this
somewhere,
I
didn't-
I
didn't
include
numbers.
I
don't
think.
B
But
you
know
lots
of
fun
in
performance
improvements.
The
regex
stuff
that
brian
boreham
has
been
playing
around
with
is
pretty
neat
like
goes
regex,
parser
is
sort
of
not
known
to
be
super
fast.
It
was
known
to
be
sort
of,
I
guess
consistent
and
reliable
or
whatever,
but
we
found
or
should
say
brian.
I
can't
take
any
credit
for
this
and
then
that
link
somewhere.
B
F
A
Yeah,
the
the
binary
operations
are
all
on.
Our
tests
are
about
10
times
faster
by
the
way.
So,
if
you're
using
those,
you
know
a
over
b
and
now's
your
chance,
okay.
A
Any
any
any
last
words
from
folks,
otherwise,
I'm
gonna
go
pretend
to
work
for
the
rest
of
the
day.
A
See
statically
good.
Is
there
a
subsection
of
this
that
you
particularly
care
about?
I
think
all
of
them
are
are
updated.
B
Oh
no,
well,
I
started
writing
this
blog
post.
I
have
the
ed
welch's
concise
guide
to
labels,
v3,
which
is
a
lot
of
what
the
contents
from
that
came
from,
and
I
want
to
give
some
more
so
like.
After
the
you
know,
our
operating
practices
and
what
we've
seen
and
I
can
give
some
better
guidance
on
labels,
which
is
what
it
looks
like
that
best
practices
digs
into
a
lot.
But
the
general
advice
is
still
probably
the
same.
You
know
use
the
labels
that
you
need
try
not
to
use
more.
B
What
think
has
been
interesting?
I've
been
trying
to
figure
out
how
to
put
it
into
words.
A
little
bit
is
actually
a
lot
of
folks.
I've
listened
really
well
and
when
we
added
the
way
to
do
removing
the
out
of
order
constraint,
like
we've
seen
things
go
the
other
direction
where
there's
too
much
data
in
a
single
stream,
loki's
horizontal
charting
mechanism,
is
the
stream.
B
So
if
you
put
all
of
your
data
in
one
stream,
we
can't
split
it
more
than
one
machine,
so
labels
are
the
tool
that,
let
you
you
know
tune
that.
So
if
you
have
too
many,
you
have
too
many
or
too
many
combinations
of
labels
and
values.
You
have
too
many
tiny
streams
and
that
gets
inefficient
and
if
you
have,
if
you
really
have
too
few,
which
most
people
don't
but
it
does
happen,
then
you
can't
char
those
for
scale
either
so,
but
generally
just
the
labels
that
always
work.
B
The
best
are
like
apps
environments.
You
know
cluster
name
things
that
you
would
use
for
querying
things.
You
would
think
about
how
you
go
find
your
logs,
but
some
of
that
best
practices
predates.
I
think
that,
like
the
loki
2.0
release
which
had
lockfield
v2
so
now
like
you
can
extract
any
content
from
the
log
line.
So
there's
really
no
reason
you
need
to
embed
anything
in
the
labels
to
do
aggregations
on.
B
A
B
The
advice
that
I
think
it
helps
people
the
most
is.
Our
experiences
show
that
if
you
can
keep
the
number
of
unique
streams
in
a
24
hour
period,
because
that's
what
our
index
bucket
is
so
that
becomes
kind
of
relevant
at
less
than
200
000
you're
not
gonna,
have
any
trouble
so
and
that
would
be
per
tenant
too.
So
you
can
actually
have
multiple
tenants
with
200
000
labels
a
day
and
it's
fine.
B
So,
like
our
internal
ops
cluster
is,
I
don't
know
100
megs
a
second
typically,
so
it's
like
8
to
10
terabytes
a
day
and
we're
pretty
aggressive
with
labeling,
because,
like
all
the
pods
get
a
label
and
we'll
roll
tens
of
thousands
of
pods
in
a
day.
Sometimes
so
you
know,
you
see
a
hundred
thousand
streams
in
a
day
to
200
000
streams
a
day
and
that's
fine.
If
you
start
getting
a
lot
more
than
that,
like
more
than
500
000
streams.
B
In
a
day,
the
index
gets
big
because
you
have
to
store
all
the
labels
and
values
in
there,
which
isn't
great
and
the
query
performance
degrades
in
a
kind
of
an
interesting
way,
which
is
over
the
course
of
the
24
hour
period.
The
range
scans
on
the
nosql
table
start
taking
longer,
so
that
is
kind
of
a
rule
of
thumb
of
like
keep
it
under
200
000
streams
a
day
so
and
in
terms
of
active
streams,
it's
a
little
bit
more
driven
on
memory.
B
We
have
some
ingesters
that
will
have
20
000
50
000
active
streams
on
them.
You
know
it's
it's
if
the
limit,
when
we
run
a
hundred
thousand
something
like
that.
So
I
don't
know
this
isn't
the
best
advice
on
the
fly.
But
if
you,
if
you
are
over
optimizing
to
get
down
to
like
10
or
20
like
you,
have
way
more
room,
you
have
way
more
overhead
right,
like
if
you're
trying
to
get
under
100
streams
or
things
like
that.
You
do
have
a
lot
more
overhead
than
that,
because
I
think
I
might
have.
A
B
B
B
The
pod
is
pretty
ephemeral
for
a
lot
of
times,
and
that's
that's
fine,
but
I
I
want
to
try
to
get
this
kind
of
documented
a
bit
better,
but
I
would
say
that
that's
probably
what
I
would
say
if
you
and-
and
I
think
most
in
our
experience
most
people
don't
have
that
many
streams
like
be
hard
to
come
up
with
a
hundred
thousand
streams
in
a
day
unless
you
have
thousands
of
pods
that
you're
rolling
right
like
because,
if
you
stick
to
that
idea,
like
owen
said,
which
is
you
know
the
file
that
it
comes
from
being
a
stream,
then
you
know
you
have
to
have
a
lot
in
your
environment
to
get
higher
than
that.
B
G
G
H
About
this
indexing
improvement,
so
so
thank
you.
Thank
you
so
much
for
putting
that
out.
C
E
B
A
B
H
B
B
To
be
here
thanks
all
right,
should
we
call
it
owen.
A
Yeah,
I
think
so,
thanks
for
coming,
everyone
really
appreciate
it,
and
hopefully
we'll
see
y'all
next
month
take
care
thanks.