►
From YouTube: Loki Community Meeting 2021-12-02
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
A
Then
I
was
saying
that
I'll
record,
but
doing
the
usual
spiel.
If
you
would
not
like
to
be
recorded,
you
may
mute
yourself
and
turn
off
video
leave
kind
of
whatever
is
most
appropriate
for
your
situation.
A
Okay,
so
yeah.
This
is
probably
relatively
light.
Today,
after
2.4
we've
been
kind
of
turning
a
lot
of
the
loki
teams,
efforts
in
different
places,
and
so
there's
not
not
a
ton
of
updates
here,
but
I
think
we
can
probably
start
off
with
steve
who's
aggregating
loki
rules
into
a
single
configuration.
C
D
On
what
I
sent?
Apologies
no.
D
No,
it's
fine.
I
can.
I
was
trying
to
look
at
the
name
of
the
of
the
of
the
sidecar
piece,
but
fundamentally
you
know
loki
brilliant.
You
know,
2.4
has
made
up
made
a
massive
difference
and
I
was
speaking
to
jen
about
some
of
the
some
of
the
potential
improvements
and
you
know
we
build
kubernetes
as
a
platform.
So
we
we
stick
kubernetes.
So
we
stick
loki
in
each
of
our
clusters
for
nice,
localized
logging.
D
The
current
design
means
that
we
need
to
know
all
of
our
loki
rules
when
we
build
the
cluster
rather
than
you
know,
like
the
prometheus
model,
where
we
can
add
rules
dynamically.
So
the
initial
question
was
whether
there
was
any
intention
for
putting
a
loki
operator
in
with
a
custom
crd
and
then
the
follow-up
question
was
actually.
I
can
see
a
pattern
to
use
a
configmap
generator
as
a
sidecar.
D
A
A
There
are
rules
yeah,
so
we
implement
a
lot
of
the
same
endpoints
that
prometheus
itself
actually
does,
and
you
can
configure
loki
to
use
a
a
back
end,
which
is
dynamically
reloaded
and
you
can
also
put
stuff
against
it,
get
delete.
You
can
all
your
normal
crud
methods
and
sorry.
D
It's
more
whether
you
can
reload
loki
so
there's
a.
I
forget:
the
exact
name
of
the
sidecars,
the
kiwi
grid
side,
car
that
watches
config
maps
of
a
specific
specification
merges
them
into
a
single
config
map,
which
would
obviously
be
a
known
value
that
could
be
mounted
into
the
loki
rule
pod,
but
the
output
of
that
action.
Changing
the
conflict
map
is
a
web
hook,
and
it's
where
the
loki
has
a
web
hook
to
reload
or
whether
you
need
to
kill
the
pod
and
start
it
again.
A
Okay,
so
this
particular
model
would
need
to
be
reloaded
that
the
whole
process
would
need
to
be
reloaded
here.
Actually,
I
need.
I
need
to
actually
double
check
that
off
top
my
head.
We
might
hook
into
the
same
code
like
if
you
change
the
the
the
file
on
disks
will.
D
A
A
For
instance,
like
we
run
a
lot
with
like
a
gcs
bucket
as
a
back
end,
which
means
that
we
can,
and
that
will
be
dynamically
reloaded
and
also
dynamically
resharded
across
a
number
of
rulers,
so
that
you
can
kind
of
horizontally
scale
that
sub
service
for
for
using
config
maps,
specifically
I'm
not
sure
off
the
top
of
my
head.
A
A
No,
although,
like
that,
the
idea
of
reloading
regularly
based
on
the
local
file
system
or
like
mounted
configmap,
seems
completely
reasonable
to
me
right
since
we
already
regularly
reload
things
like
back
end
object,
store
buckets
for
rural
configurations.
We
could
do
the
same
thing
on
localhost
and
I
wouldn't
see
a
problem
there.
A
D
Yeah,
so
if
you
know,
if
we
can
confirm
it
works,
I'm
happy
to
put
a
pr
in
on
the
leaky
disputed
chart,
which
is
the
one
that
we
use
to
put
an
optional
side
current,
to
scrape
complex
maps
together
into
a
single
config
map.
That
means
the
rule.
Can
you
can
operate
similar
to
me.
C
Yeah,
it
should
reload.
So
if
you
mount
the
config
map
as
a
file
and
use
it
as
a
local
file
rule,
it
looks
like
we
do
rescan
that
periodically
there's
a
timer
that
does
it.
So
you
would,
I
don't
think,
there's
a
web
hook,
but
there's
a
timer
that
runs.
I'm
not
sure
if
that
interval
is
configurable,
but
it
should
reload
the
rules.
If
the
config
map
changes
awesome
I'll,
give
it
a
go.
A
Into
the
whatever
that
timer
is,
that
would
basically
turn
your
delay.
Interval
to
there's
cubelet
has
a
timer
as
well
for
like
maximum
config
map,
resync
interval,
and
I
can't
remember
exactly
what
that's
called
but
it'll,
be
that
plus
whatever
our
internal
timer
is,
would
be
the
maximum
lag
yeah.
C
It
is
configurable,
it's
called
the
poll
interval
and
it's
wonderful.
I
don't
know
what
the
default
is,
I'm
guessing,
5
or
10
seconds.
Let
me
see.
C
One
minute,
actually
that's
pretty
long,
so
the
default
poll
interval
is
one
minute
for
rules,
but
that
could
be.
I
don't
see
any
reason
why
that
couldn't
be
five
or
ten
seconds,
although
I
guess
I'm
sure,
there's
a
trade-off
there
too,
with
what
it
does,
but
but
yeah.
I
think
that
if
you
load
as
a
config
map
that
it
should
auto
reload,
let
us
know
how
that
works,
though,
because
seems
like
a
reasonable
thing
to
be
able
to
do
for
not
doing
it
already.
It
seems
like
it's
most
of
the
codes
already.
A
Again
not
the
specific
question
that
was
asked,
but
this
is
kind
of
a
nice
tie-in,
something
that
I've
wanted
to
add
for
a
while.
Oh,
I
need
to
put
that,
as
we've
got
this
new
low
hanging
fruit
tag
in
in
the
loki
repo
that
I'm
trying
to
use
to
bootstrap
things
of
limited
scope,
we've
got
some
possible
to
do
work
work
here.
A
A
The
ability
to
like
basically
dynamically
do
this
stuff
and
you
know,
persist
it
to
like
an
object,
storage
backend.
That
sort
of
thing
was
added
and
so
there's
a
flag
to
enable
the
api,
but
the
file
system
actually
does
not
have
an
implementation
to
support
it,
but
we
could
definitely
enable
the
api
on
top
of
a
file
system
back
in
and
I've
got
this
request
a
number
of
times
to
be
totally
reasonable.
C
The
loki
operator,
so
I
guess
I
don't
want
to
spoil
the
the
we'll-
have
a
better
announcement
around
this
once
it's
it's
sort
of
in
action,
but
the
awesome
folks
at
red
hat
have
built
a
loki
operator
that
they're,
using
with
an
open
shift,
that's
going
to
be
upstreamed
into
the
loki
project
for
a
general
kubernetes
operator.
So
you
know
there'll
be
a
sort
of
a
lot
more
to
come
on
this
over
time
in
terms
of
like
how
it
evolves
and
how
it
gets
used.
C
But
it's
primarily
targeting
probably
the
exact
use
case
that
that
you're
using
steve,
where
they
basically
want
to
make
it
easy
to
provision.
You
know,
relatively,
I
would
say,
sort
of
small
medium
size,
loki
cells
in
an
easy
fashion.
So
I
guess
stay
tuned
to
that
we
are
going
to
say
we
will
be
upstreaming
it
into
they're,
contributing
it
to
the
project.
So
we're
excited
about
that.
So
we'll
see
more
about
that
soon.
A
Does
that
all
answer
any
of
your
questions?
Steve?
No!
That's!
That's!
Brilliant!
Thank
you.
Thank
you
very
much.
All.
D
C
I
put
this
on
here
just
just
to
sort
of
a
couple
things
that
we've
noticed.
I
don't
know
what
to
do
about
the
first
one
here
that
it
has
been
inconvenient
for
a
lot
of
folks
because
we
enabled
the
wall
by
default
and
that
directory
wasn't
mounted
in
the.
C
So
if
your
config
doesn't
specify
where
the
wall
directory
is
it
just
defaults
to
a
directory
named
wall
wal
which
isn't
going
to
be
mounted
by
anybody
and
isn't
going
to
be
writable
on
almost
any
container
like
our
loki
containers,
I
can't
write
to
that
so
so
that
has
been
a
bit
frustrating
for
people,
so
you
basically
need
to
specify
the
directory
and
yeah
on
the
newer
configs
that
we've
built
that
are
part
of
the
opinionated
configs
do
this,
but
if
you
have
an
existing
like
values,
file
for
your
helm,
chart,
for
example,
that
has
a
config
in
it.
C
It's
not
going
to
do
this
and
you're
going
to
get
this
error.
I
think
we've
merged
some
prs
to
make
the
error
more
descriptive
to
try
to
improve
this,
because
I
know
the
error
is
a
bit
nasty
too.
It
just
can't
make
their
wall
and
it
doesn't
really
help.
You
understand
what
to
do
so.
If
you
run
into
that,
if
you've
seen
that
it
basically
means
you
need
to
specify
in
the
adjuster
config
where
you
want
the
wall
to
be,
and
you
have
to
make
sure
that
you
have
in
your
container
that
mounted
as
writable.
C
Usually
people
just
specify
the
same
container,
you
put
the
you
know,
bolt
tv
index
files
and
the
perhaps
chunks
files
like
slash
loki
is
already
writable
by
the
container,
all
new
containers,
so
you
can
put
it
there,
but.
A
C
I
mean
yeah
like
the
the
reason
we
broke
it
or
what
broke.
I
guess
is
because
we
we
in
code
now
enable
the
wall
and
in
config,
if
you
don't
specify
the
wall
directory,
it's
so
like
we
could
encode
force
the
wall
directory
to
one
that
we
know
that
works
in
the
docker
image,
but
I'm
not
sure
if
that's
a
better
side
effect.
C
It's
basically
just
change
the
default
from
wall
to
slash
loki,
slash
wall,
and
that
would
fix
it
in,
but
what's
annoying
about,
that
is
that
the
helm
chart
mounts
all
of
the
stuff
at
a
different
location
than
the
docker
image.
Does
so
the
docker
image
mounts
everything
at
slash
loki,
but
then
the
helm
chart
overrides
that
to
slash
temp
or
no
slash,
var,
local
or
something
like
that.
It's
a
mess!
I
don't
know
how
to
unmess
it
honestly,
but
it's
just.
C
So
sorry
I
mean
I
wish
I
knew
a
better
way,
but
we
want
the
wall
enabled
by
default,
because
I
think
it's
a
value
to
everybody
like
they
should
have
it.
Unfortunately,
defaulting
the
directory
becomes
really
really
difficult,
because
the
helm
chart
has
different
defaults
than
the
json,
which
has
different
defaults
than
the
image
itself.
And
I
don't
know
how
to
reconcile
that.
C
I
updated
the
helm
chart
to
the
the
default
values.yaml
file
that
comes
in
the
helm.
Chart
now
specifies
the
wall
directory
to
match.
Whatever
was
specified
in
that
particular
helm
chart
because
it's
actually
different
between
the
loki
home
chart
and
the
distributed
home
chart
the
rabbit
hole
gets
deeper,
so
each
of
those
were
updated,
but
typically
most
people
have
a
values,
file
that
they've
curated
and
don't
probably
go
reference
to
see
if
the
one
that
ships
with
the
helm
chart
changed,
and
I
think
that's
what
caused
a
lot
of
the
trouble
is.
D
D
This
pain
then
steve
you're,
saying
yeah.
We
adopted
2.4
the
day
it
was
released,
so
it
took
me
a
couple
hours
to
a
couple
hours
get
it
working.
C
Sorry
about
that,
what
what
I
guess
is
is:
is
there
something
that
we
could
do
to
help
others
at
this
point
still
or
you
know,
where
did
you
go?
Look
that
you
didn't
find
the.
D
So
I
mean
first
principles:
if
you
ship
minio
in
the
distributed
chart
and
ship
it
with
the
bulk
db
shipper
turned
on,
the
config
will
be
closer
to
what
most
people
should
be
using,
and
then
you
wouldn't
need
the
extra
copy
and
readme.
But
I
think
I
think,
there's
just
there's
some
missing
missing
continuity
in
the
dark.
So
then
you
can
read
what
something
does,
but
the
conflicts
obviously
are
dependent
on
each
other,
the
wow
being
a
new
concept
that
was
introduced
and
yeah.
C
C
There
I'm
terrible
at
home,
and
so
whenever
I'm
messing
around
on
the
helm
charts,
I
don't
know
I
feel
like
I
make
it
worse
than
better
sometimes,
but
the
readme
is
a
good
one,
though
I
should
look
at
that.
D
B
C
Absolutely
yeah
the
the
helm,
charts
we've
always
been
in
a
bit
of
a
challenge
here,
because
internally
we
run
loki
with
jsonnet
and
that
effectively
makes
us
the
only
ones
in
the
world
that
do
that
everybody
largely
that's
not
true.
I
mean
a
lot
of
people
use
json
it,
but
I
know
we're
a
bit
unique
there.
So
we
don't
have
a
lot
of
hands-on
with
the
helm,
charts
and
like
the
loki
distributed.
Chart
was
very
graciously
created
and
donated
by
the
community
member
and
for
the
most
part
the
charts
are
community
maintained.
C
So
absolutely
would
take
you
up
on
that
if
you,
if
you'd
like
to
be
involved
with
helping
with
the
helm,
charts
and
helping
me
understand
how
to
not
break
them,
would
be
nice
too.
C
Or
slack.grafanow.com
there's
a
loki
channel
or
just
kind
of
dm
me,
and
we
can
kind
of
figure
out
how
to
get
you
involved.
I
would
really
appreciate
that
yeah,
that's.
C
Another
thing
that
is
that's
come
out
is
that
we
did
miss
some
configs
in
2.4
for
getting
the
most
parallelism
out
of
of
loki,
so
I
gotta,
I'm
not
sure
so.
The
paralyzed
charitable
queries,
I
know
for
a
fact,
is
still
defaulted
to
false
and
that's
gonna
be
a
big
one,
because
it's
gonna
enable
sharding
on
charitable
queries
which
allows
them
to
be
split
basically
16
times
farther.
So
that's
a
huge
config.
C
Probably
it
matters
more
when
you
have
multiple
instances
running
because
you're,
you
know,
if
you're
only
running
one
instance,
parallelization
is
largely
limited
by
your
cpu
cores
and
throughput,
but
you're
gonna
want
that
on.
We
will
update
this
in
the
next
release,
so
these
will
get
fixed
and
probably
be
2
2.5
to
be
basically
correct
defaults
and
I
pasted
a
couple
others
here.
I
should
have
added
more
notes
to
these.
Maybe
I'll
circle
back
around
the
split
queries
by
interval.
C
C
We
actually
probably
run
30
minutes
in
some
of
our
clusters
too.
You
don't
you
can
go
less
than
that,
but
there's
diminishing
returns
on
splitting
by
time,
because
you
end
up
duplicating
a
lot
of
work.
I
I
would
be
hesitant
to
run
15
minutes
honestly,
should
we
just
say
30.
A
C
C
Yeah,
the
sort
of
way
that
works
right
is
it's.
If
you
think
about
how
much
data
is
in
one
chunk
and
if
you
tend
to
you
know
max
chunk
ages,
one
hour
or
maybe
two
hours
and
you
don't
write
a
lot
of
data
and
you
don't
end
up
filling
chunks.
If
you
split
by
five
minutes,
it
just
means
you're
gonna.
Take
that
two-hour
chunk
and
break
it
up
into
five-minute
pieces
and
every
query
is
going
to
be
operating
on
the
same
chunk.
It's
not
really
going
to
be
faster.
C
If
you
have
streams
that
are
riding
10
chunks,
a
second
or
something
that
would
be
a
huge
amount,
but
splitting
those
on
five
minute
intervals
would
effectively
paralyze
a
lot
better.
I
mean
someday
loki
will
be
smarter
at
this.
You
know
we'll
know
the
throughput
of
a
stream
and
be
able
to
split
automatically,
but
for
now
it's
a
bit
of
a
tunable
that
30
minutes
is
probably
a
pretty
reasonable
default.
D
Sorry,
my
voice,
you
can
ask
a
question
yeah,
please.
So
I
said
we
we
build
and
ship
loki
as
a
platform,
so
we've
got
some
consumers
and
we're
taking
on
some
new
internal
consumers
on
this
internal
platform
and
one
of
our
consumers
turn
their
chunk
age
to
24
hours
to
because
they
read
the
docs.
That
said,
unfilled.
Chunks
are
bad.
D
C
C
2.4
included
a
number
of
those
sort
of
opinionated
changes
that
we've
learned
on
how
to
run
loki
for
reasonable
use.
Cases
like
we,
you
know,
force
the
chunk
target
size
now
and
the
max
chunk
age.
So
this
this
is
an
area
that
we
want
to
put
more
time
into
where
we're
trying
to
get
a
lot
better
at
to
make
it
easier
to
just
have
loki
work.
C
The
optimization
that
they
would
get
from
the
information
that
they
read
is
going
to
offset
the
sort
of
trouble
you
might
have
like.
It's
not
that
big
a
deal
to
flush,
small
chunks,
but
it's
depends
a
bit
on
the
storage
you're
using,
and
you
know
I'll
talk
about
this
in
a
second.
Maybe
if
we
have
time
at
the
end
about
what
we
just
started
conversations
about,
how
we're
going
to
iterate
on
the
storage
to
make
some
of
these
situations
better.
But
what
ends
up
happening
is
that
same
thing
right?
C
C
Well,
that's
not
totally
true,
I
would
say,
like
you
know,
one
or
two
hours
is
probably
fine
and
you're
increasing
it
more
than
that.
It's
probably
not
going
to
gain
you
a
lot
in
the
long
run.
It's
not
that
big
a
deal
if
you
flush,
you
know
six
chunks
versus
two,
but
it's
going
to
be
better
for
querying.
If
you
probably
keep
it
at
one
or
two
hours
for
junk
age,.
D
I
think
the
reason
I
ask
on
this
because
I
think
it's
related
to
split
queries,
but
also
it
would
be
yeah
communicating
the
defaults
somewhere
to
look
up.
You
know
whether
that's
in
the
helm,
charts
or
or
somewhere
for
to
see
what
the
standards
are.
I
think
some
of
the
defaults
are
non-documented
for
2-4.
I
think
they
were
internal
defaults,
but
if
you
had
a
config
already
where
you
set
them,
I
think
your
you
know
what
you
had
before
would
would
be
kept
with
the
way
that
the
the
conflict
said.
Isn't
it
here.
D
C
This
could
this
discussion
has
come
up
in
the
sort
of
like
a
next
step,
which
would
be
a
config
auditing
tool
or
something
that
we
would
do
to
answer
that
question
for
everybody
right
already
running
like
what
what
does
the,
because
we
simplified
the
config
and
in
doing
so
exactly
what
you
said
we
internalized
it.
We
didn't
really
document
it.
So
I'm
sure
karen's,
if
she's
still
on
the
call
ears,
are
burning
because
she's
our
tech
writer
and
has
been
asking
us
for
a
long
time
to
help
solve
some
of
these
problems.
C
Let
me
take
that
as
a
good
note
here
that,
if
nothing
else,
we
actually
recently
talked
like
what
are
good
reference.
Implementations
like
we've
started
some
work
on
this.
It
gets
tricky
because
there's
so
many
ways
you
can
run
loki
with
so
many
stores,
but
some
of
these
configs
would
be
the
same
regardless.
So
we
could
definitely
document
those.
A
Steve,
can
you
share
a
little
bit
about
how
you're
running
loki
like?
Are
you
running
like
single
binaries
per
tenant?
Are
you
using
the
new
single
scalable
deployment
model?
Are
you
running
microservices.
D
So
we're
running
the
distributed
helm
chart
backed
on
s3.
We
haven't,
got
any
memcache
caches
at
the
moment
because
we're
running
in
cluster,
so
you
know
we're
talking
about
most
clusters
are
sort
of
I'd,
say
about
50
nodes,
so
we're
you
know
we're
running
all
of
our
observability
in
cluster
as
a
control
plane
with
the
potential
to
then
aggregate
those
in
something
separate.
D
You
know
maybe
refine
a
cloud
or
equivalent
outside
of
it,
so
not
huge
setups
but
the
the
micro
service
model,
because
when
we
previously
ran
in
single
binary,
we
did
see
the
the
differences
around
the
around
the
read,
write
and
you're
really
happy
with
it.
I
think
you
know
we
used
on
some
clusters
previously
we're
just
now
shipping
to
every
single
one
of
our
clusters
with
a
new
pattern,
and
some
of
them
have
different
use
cases
which
might
you
know
where
we
might
not
been
called
out
before
running
running
what
we
had.
D
A
A
Yeah,
I
I
think
it's
a
lot
better
than
the
defaults
like
we
also
do.
As
that
said,
we
have
another
overlay
which
kind
of
lives.
On
top
of
this,
the
open
source,
one
as
well,
which
is
kind
of
tailored
to
some
particulars,
for
how
grafana
cloud
specifically
does
it,
and
there
is
definitely
some
improvements
to
be
made
for
migrating
some
of
those
changes
up,
but
I
do
think
this
is
a
right
like
it's
immediately
here.
It
is
not
it's
not
by
no
means
a
good
answer,
but
it
is.
C
Yeah,
throw
it
in
there
for
sure
just
so
that
people
get
but
yeah
the.
I
will
say
it's,
it's
very
present
in
our
minds
to
continue
improving
this.
Like
it's,
I
know
it's
I
apologize.
It's
been.
C
C
C
2.4
that
I
was
going
to
talk
about
so
and
if
anybody
else
had
any
questions
or
feedback
on
their
experiences
with
2.4
going
once
going
twice,
nice
cyril
wasn't
able
to
make
it.
So
I'm
just
going
to
mention
this.
It's
not
released
yet,
but
we
did
add
support
for
the
as
a
udp
receiver
for
the
gray
log
extended
log
format
so
that
you
can
ship
grey
log
jelf.
I
don't
know
if
it's
pronounced
gelf
or
delph.
C
The
grand
enterprise-
oh
my
god,
so
the
the
the
yeah
greylock
extended
log
format-
will
be
available
from
udp
and
it's
effectively
json.
So
after
it's
received
via
udp,
you
can
use
the
prom
tails
pipeline
stages
to
manipulate
the
json
to
pull
the.
Maybe
they
call
them
tags.
I
can't
remember
into
labels
if
you
want,
but
so
another
cool
way
to
get
logs
into
loki
and
this
one
daniel
yeah.
This
is
super
exciting.
C
F
If
you
click
on
the
the
tempo
pr,
they
show
what
impact
it
had
on
on
their
side.
We
would
probably
get
larger
gains.
I
would
imagine
so
you're
already
excited
to
see
this
happening.
F
Yeah
yeah,
so,
oh,
and
if
you
go
to
the
first
pr
in
the
first
or
in
the
yeah
that
one
pr
one
that's
got
the
tail
at
scale
paper
link
in
it
there
yeah.
So
the
idea
behind
this
feature-
and
this
was
implemented
by
cyril.
F
So
it's
another
item
taken
credit
for
cyril's
work
this
week,
this
unreleased
feature
as
yet
so
when
you
are
fetching
chunks
from
object
storage,
if
your
object
store
is
experiencing
some
latency
that
can
really
push
up
the
the
latency
of
the
overall
query
right.
So
if
you
put
a
query
and
that's
got
to
fetch,
100
chunks
and
one
of
those
chunks
is
a
bit
slow
to
retrieve
that
that
increases
the
latency
of
the
overall
request.
F
So
what
this
change
introduces
is
the
ability
to
hedge
a
request,
so
we
will
you'll
typically
run
this
in
your
environment.
For
a
while
and
then
figure
out.
What
is
your
sort
of
p99
of
your
requests
to
your
object
store?
Then?
You
can
configure
this
hedge
request
feature
so
that
when
you,
when
a
request,
exceeds
that
p99
time,
it
will
actually
cancel
that
request
and
issue
a
new
request
and
the
idea
around.
F
That
is
that,
if
you're
using
a
cloud
service
that
that
one
request
that
one
chunk
in
the
object
storage
could
have
hit
a
node.
That
was
a
little
bit
overloaded
or
there's
some
other
kind
of
network
congestion,
and
you
issue
another
request
and
that
usually
completes
faster
and
so
there's
two
pull
requests
there.
F
The
one
is
implementing
the
base
feature
and
the
second
pr
is
implementing
the
rate
limiting
and
the
reason
why
we
need
the
rate
limiting
is
because,
if
the
object,
storage
service
as
a
whole
is
slow,
then
you're
effectively
going
to
be
hedging
every
single
request
and
that's
going
to
push
up
your
request,
latencies
quite
ex,
quite
massively
so
yeah.
Those
two
features
together
should
provide
some
some
pretty
awesome
gains.
F
So
we're
going
to
be
running
it
in
our
cloud
environment
over
the
next
few
weeks
and
experimenting
with
that
and
seeing
what
what
kind
of
benefit
that
gives
us.
But
it
gave
tempo
quite
a
good
improvement
on
their
side,
so
we're
expecting
to
see
something
like
that
as
well.
F
I
think
I
think
yeah
they
configured
their
hedge
time
at
500
milliseconds.
If
I
remember
correctly,.
A
Okay,
so
it
looks
like
they
brought
the
so
p99
is
generally
your
slowest
request
right,
your
99
percentiles,
and
this
is
what
we
kind
of
talk
talk
about
by
the
tail.
It's
the
tail
of
distribution,
meaning
the
slowest
of
the
request
that
you
actually
sent
out.
It
looks
like
they
reduced
theirs
from
about
10
seconds
to
about
two
and
a
half,
so
it's
a
factor
of
four
there
which
is
really
impressive
and
when
you
think.
D
A
That
this
can
can
function
a
bunch
of
different
layers,
so
it
can
be
like
the
one
request
that
you
send
to
loki,
for
instance,
but
it
can
also
be
that
loki,
splits
and
charge
and
paralyzes
one
request
into
a
thousand,
and
maybe
the
thing
that
holds
up
your
your
your
whole
query
from
returning
is
you
know
one
of
those
thousand?
A
The
idea
is
that
we
can
take
this
and
reduce
it
by
a
large
factor,
and
so,
if
you're
spending
a
lot
of
the
time
on
on
your
queries,
just
waiting
for
something
to
happen
waiting
for
the
tail
to
finish.
This
is
another
way
to
mitigate
that.
F
A
C
It's
super
fun,
I
mean
the
the.
If
you
look
at
loki's
traces
you'll
see
a
pattern
right
where
we,
we
limit
parallelism
to
sort
of
save
resources,
but
that
does
mean
we
have
to
wait
for
at
some
point
every
one
of
say
the
16
queries
that
were
parallelized
at
the
same
time
to
return,
and
if
one
of
those
is
waiting
two
seconds
for
one
chunk,
then
your
query
ends
up
being
the
sum
of
the
slowest
requests
and
so
by
hedging.
C
A
B
B
G
B
Go
for
it
so
yeah
I'm
starting
to
to
change
my
career
into
acting.
So
I
got
this
video
going.
It's
really
excited
about
it.
I.
B
Yeah
yeah,
I
played
I
played
myself.
Actually
I
know
so
there's
a
so
a
lot
of
the
work
that
that
I
contributed
to
to
2-4
was
around
the
simple
scalable
models.
So
these
two
new
targets
that
my
I'm
not
just
me.
I
mean
a
lot
of
people
worked
on
on
these
new
targets
that
read
and
the
right
targets.
So
I
put
together
a
video
on
getting
started
with
those
and
then
there's
also
a
helm
chart.
B
That's
in
pr
and
I'd
love,
some
reviews
from
the
community
on
that,
because
I
am
in
the
same
boat
as
ed
when
it
comes
to
helm
and
not
feeling
like,
I
know
what
I'm
doing
when
I'm
getting
in
there
so
yeah.
I
love.
If
anyone
has
tried
out
this,
these
two
targets,
love
feedback
and
yeah.
That's
it.
B
B
The
the
problem
that
we've
seen
with
single
binary
is
when
there's
a
problem.
It
becomes
a
really
big
problem.
So
if
you
have
like,
if
you're
queriers,
for
example,
that
will
actually
take
down
your
injectors,
because
it's
single
binder
so
on
the
same,
but
we
thought
that
we
could
maybe
get
a
little
bit
better.
B
But
then
the
problem
with
microservices
obviously
is
just
it's
complicated
right,
there's
a
lot
of
moving
pieces,
and
so
for
people
who
are
maybe
just
kicking
the
tires
with
loki
or
like
setting
it
up
for
maybe
like
a
smaller
load.
Maybe
it's
for
like
a
hobby
project
or
something
like
that.
We
wanted
to
find
a
middle
ground
between
the
two
where
it
was
simpler
to
deploy
in
terms
of
the
topology,
but
was
a
little
bit
more
rugged
and
robust
than
than
single
binary
and
so
yeah.
B
That's
that's
the
the
simple
scalable
it
it
sort
of
encouraged
a
a
a
lot
of
other
work
that
I
think
is
going
to
be
really
valuable,
like
the
query
schedulers
having
a
ring.
So
now,
even
in
single
binary
mode,
the
queries
can
be
started
and
then
we
also
got
a
little
smart
with
the
compactor
with
a
ring
so
that
the
the
ring
determines,
which
instance
the
compactor
would
run
on
and
making
sure
it
only
runs
once
so.
I
think
it's
it's.
B
G
B
How
well
we've
done
until
people
run
it
that
way
so
give
it
a
try
and
let
us
know
what
works
it
doesn't
work.
C
Yeah,
the
only
thing
that
I
would
say
is
that
I
would
say
the
ssd
mode
is:
is
really
actually
intended
for
people
that
want
to
run
gigabytes
or
even
a
couple
terabytes
a
day
like
you,
you
can
scale
it
like
you're
you're,
making
a
trade-off
on
complexity
versus
sort
of
brutal
efficiency
that
you
can
get
with
microservices.
So
so
that
was
the
idea
right
like
in
a
number
of
cases.
C
Most
people
don't
want
to
know
what
an
index
gateway
is
or
query
scheduler
or
query,
front-end
or
queriers,
and
even
the
caching
layers,
and
then
we
hid
all
of
that
inside
of
these
new
read
and
write
targets.
So
it
includes
in-memory
caching
and
includes
the
query.
Scheduler
and
actually
the
bind
single
binary
does
now
too.
So
you
get
that
parallelization.
C
B
C
C
G
Yeah,
so
basically
I
just
wanted
to
mention
here
that
yesterday
we
had
a3
release
that
comes
with
long
awaited
and
unwanted
feature,
which
is
what
volume
or
you
might
know
it
under
the
name.
Full
range
log
histogram
and
this
feature
is
still
under
feature
toggle.
So
if
you
would
like
to
try
it
out,
you
need
to,
in
your
custom,
ini
file
toggle
this
feature
and
let
me
find
out
what's
the
name,
maybe
I
should
mention
it
here.
So
the
name
is
full
range
logs
volume
I
will
maybe
oven.
G
C
Would
love
feedback
on
this
word,
I
I
would
say
like
we're.
You
know
the
two
big
concerns
or
there's
two
sort
of
questions.
One
is
what
the
ux
is
like
and
how
do
people
like
it
in
our
experience
running
it's
it's
nice
to
have
the
full
range
histogram.
C
C
So
we've
hedged
that
by
limiting
it
to
10
seconds,
which
sort
of
is
a
way
to
say,
if
you're
querying
a
really
long
time
that
it,
you
know,
doesn't
consume
a
huge
amount
of
resources,
and
this
is
the
part
that
I
think
is
going
to
probably
be
a
bit
contentious
and
what
that
experience
looks
like
so
I'm
interested
to
see
we're
gonna,
you
know
run
it
on
our
cloud
environment.
We're
gonna,
do
sort
of
a
beta
version
of
it.
C
C
G
Yeah
we
can
either
create
like
one
issue,
or
I
was
thinking
that
if
anyone
has
any
kind
of
issue
feedback
just
like
create
issue
for
that,
specifically
in
graphene
already,
I'm
wondering
if
it
should
be
graphene
ripple
or
loki
repo
as
it's
kind
of
interconnected.
But
what
like,
if
you
do
it
in
any
of
these
repositories
like
we
will
get
the
feedback
and
definitely
okay.
C
Yeah,
it's
a
good
question.
I
would
say
grafana
though,
just
because
we're
largely
targeting
the
sort
of
user
experience
and
the
front-end
part
is
what
we're
looking
for
feedback.
But
as
an
operator,
I
guess
we
could
have
one
for
loki
too
right,
like
you
know,
what's
your
experience
like
as
an
operator
of
a
loki
cluster
when
you
turn
this
feature
on,
we
could
maybe
do
both
and
reference
each
other.
G
G
A
A
And
otherwise
we
can
probably
cut
this
a
little
bit
short
and
give
everyone
back
some
time.