►
From YouTube: Grafana Mimir Community Call 2022-05-26
Description
3rd Grafana Mimir Community Call, in which we discussed just-released Mimir 2.1 and ongoing work.
Join us on the last Thursday of every month. Please see the linked document for meeting notes, agenda and details on how to join: https://docs.google.com/document/d/1E4jJcGicvLTyMEY6cUFFZUg_I8ytrBuW8r5yt1LyMv4/view
A
A
B
A
Cool
we
have
a
list
of
top
new
features
in
the
document,
so
these
are
an
arm
image
for
my
mirror,
particularly
multi-arch.
Docker
images
there's
a
new
mode
for
the
ruler
that
allows
you
to
use
the
full
capabilities
of
the
mirror,
query
sharding
and
all
that,
so
it
should
improve
performance
of
rule
evaluation.
A
C
Yeah
in
linear
2.0
we
introduced
what
we
called
the
custom
trackers
for
active
series.
Basically,
you
can
specify
a
list
of
label
matches
to
track
active
series,
only
matching
the
specific
label
matches.
For
example.
Let's
say
you
want
to
track
the
number
of
active
series
by,
I
don't
know
the
job
label.
E
A
C
C
C
A
A
There's
a
few
other
kind
of
major
efforts
underway.
So
one
is
the
new
backfill
api,
I'm
not
sure
if
anyone
on
the
working
on
that
is
on
the
call
here
today,
but
this
one
is
about
being
able
to
backfill
metrics
data
into
a
mirror.
So
say
you
have
tsdb
blocks
from
some
other
system
or
you're
migrating,
for
example,
from
an
on-prem
vemire
to
reform
cloud,
that
sort
of
thing
being
able
to
backfill
those
metrics
into
the
database.
A
All
right,
marco,
you
had
a
note
here:
yeah
about
the
runtime
config
file,
yeah
high
level.
Runtime
config
allows
you
to
configure
things
at
runtime
rather
than
needing
to
restart
processes
to
configure
them.
So
the
some
of
the
features
we
were
talking
about
earlier,
the
penta
pertinent
custom
trackers,
is
one
of
those
you
can
configure
without
restarting
anything
pretty
neat,
oh
yeah
anything
else,
but
the
backfill
api.
A
All
right
there's
a
another
one
in
process
which
is
adding
a
thread
pool
to
the
store
gateway.
So
this
one
is
trying
to
address
some
issues.
We've
seen
with
map
page
faults
going
on
and
store
gateways,
causing
store
gateways
to
basically
become
unresponsive,
so
we're
experimenting
with
adding
a
thread
pool
to
basically
isolate
those
page
faults
to
a
known
number
of
threads
so
that
they
can't
block
all
the
go
routines
attempting
to
serve
requests.
A
So
there's
more
details
in
that
issue,
but
it's
a
thing
that's
been
going
on
for
a
few
weeks
now:
testing
in
our
production
environments.
I
think
we've
seen
a
little
bit
of
improvement.
I'm
not
sure.
Does
anyone
have
like
a
knowledge
of
what
the
expected
changes
are
in
terms
of
performance
or
whatever
availability.
C
C
C
E
C
C
A
Yeah,
I
think,
but
what
I
remember
is
basically
they
were
finding
that,
yes,
they
were
able
to
prevent
the
store
gateway
from
completely
locking
up,
but
it
didn't
necessarily
improve,
like
overall
query
evaluation
like
because
the
stronger
areas
ended
up
slower
as
a
result
of
triple
something
like
that,
so
I'm
sure
they'll
keep
working
on
it
and
you're
right.
We
have
a
backup
plan
cool.
A
The
next
item
is
this
errors,
catalog,
which
this
is
about,
trying
to
make
it
a
little
bit
easier
to
understand
the
various
different
kinds
of
errors
that
come
out
of
a
mirror
and
having
a
central
place
to
you,
know
kind
of
understand
how
you
can
fix
those
things.
There's
a
few
different
issues
linked
here.
I
don't
know.
I
think
marco
there's
another
one
that
you
worked
on.
If
you
want
to
say
anything
else
about
that.
C
C
Or
when
you
run
a
query,
which
is
you
know,
forbidden
for
some
reason
because
of
the
configurable
because
of
some
limits
come
configured
and
we
want
you
to
be
able
to
google
the
hair
already
and
just
get
a
round
book
explaining
what
does
it
mean
and
how
to
resolve
it?
For
example,
if
you,
if
you
hit
the
limit,
we
give
you
suggestions
on
first
of
all,
which
is
the
configuration
option
controlling
that
limiter
or
if
there
are
other
options
to
solve
it
without
increasing
the
limit.
C
D
A
A
bunch
of
work
going
on
in
the
coming
weeks
and
months
on
the
helm
chart
across
a
lot
of
different
areas,
but
we
tried
to
highlight
a
few
things
here.
So
basically,
the
first
thing
that
we're
going
to
do
is
we're
actually
going
to
move
the
helm
chart
from
the
grafana
helm
charts
get
repo
into
the
mirror,
git
repo
itself.
A
We
think
this
will
allow
us
to
make
sure
that
we're
kind
of
keeping
the
helm
chart
up
to
date.
You
know
on
every
pr
effectively
keep
making
it
as
first
class
citizen
as
everything
else,
the
documentation,
the
jsonnet
deployments,
all
that
kind
of
stuff
we're
going
to
keep
them
as
like
one
unit,
the
helm
chart
repository
itself
will
be
unchanged.
So
the
plan
is
for
the
actual
installation
experience
to
be
the
same
shouldn't
be
any
breaking
change
there.
A
As
far
as
the
move,
but
yeah
that's
that'll,
be
the
first
thing,
and
then,
after
that,
we're
kind
of
doing
a
lot
of
thinking
about
the
release
process.
How
we
can
test
the
chart
just
generally
trying
to
make
sure
that,
as
we
accept
both
internal
and
community
contributions
to
the
chart,
we
can
be
more
confident
that
we're
not
going
to
break
anything
understand
how
it
impacts
things
like
functionality
and
performance
over
time.
A
We're
not
sure
you
know
how
they're
going
to
work
yet
we'll
be
a
part
of
that
beta
chart
that'll,
give
users
a
chance
to
install
them
quickly
and
give
feedback
without
us
needing
to
commit
to
long-term
support
in
amir
distributed,
which
is
used
not
only
by
open
source
users,
but
also
by
our
enterprise
customers.
So
we
have
a
little
bit
higher
bar
of
kind
of
long-term
sustainability.
D
A
The
mirror
distributed
chart
versus
the
very
distributed
beta
chart,
which
we
will
be
able
to
make
a
little
bit
more
kind
of
breaking
changes
and
move
more
quickly
as
we're
learning
things,
and
then
there's
also
beyond
that.
Like
so
that's
kind
of
all,
just
like
the
infrastructure
work,
I
would
call
it
for
the
home
chart
to
make
sure
that
it
is
something
that
we
can
sustain
and
provide
long
term.
A
On
top
of
all
of
that
kind
of
like
foundation,
we
want
to
make
sure
that
the
helm
chart
is
the
best
way
to
install
my
mirror.
So
that
means
all
of
the
you
know.
High
performance
features
like,
for
example,
query
results,
cache
query,
sharding,
multi-zone
ingestion.
All
of
that
kind
of
stuff
needs
to
be
a
part
of
the
help
chart,
so
the
plan
is
kind
of
once
we
have
that
foundation
of
testability
and
a
release
process.
We
can
start
to
really
quickly
add
those
sorts
of
features
make
the
home
chart.
A
You
know
as
capable
as
any
other
way
of
deploying
the
mirror,
and
then
the
only
last
thing
that
I
haven't
mentioned
is
openshift
support.
This
is
the
thing
that
several
different
users
have
asked
for,
and
enterprise
customers,
and
so
that's
one
of
those
things
that's
like
kind
of
a
bigger
change,
and
so
this
is
part
of
what
inspired
us
to
think
of.
A
Maybe
we
need
a
beta
chart
because
it's
it's
difficult
to
test
like
the
scale
of
change
required
to
support
something
like
open
shift
in
the
main
chart
with
the
release
process.
We
have
now
because
we're
releasing
on
every
commit
and
we
don't
want
to
break
things
and
we
know
we're
going
to
need
to
experiment
a
bit.
F
F
I
think
the
main
message
is
that
we
really
want
to
make
the
hem
charts
first
class
this
season
and-
and
you
know,
speed
up
the
development
which
is
a
bit
could
be
perceived
as
a
bit
sluggish
at
the
moment,
because
we
need,
you
know
a
kind
of
stability,
but
in
the
in
the
same
time
we
are
having
a
lot
of
requests
that
that
introduce
breaking
changes
so
yeah,
but
good
summary.
Thank
you.
A
G
Hey
folks,
thanks
michael
hey,
hey
folks,
my
name
is
giri.
I
work
with
this
startup
called
true
foundry,
based
on
depending
we
were
about
to
work
on
setting
up
a
metric
stack
when,
like
literally
the
I
think,
the
day
that
I
joined,
I
saw
that
member
called
purchase.
Thank
you
so
much
for
doing
that.
G
It's
been
like
I've
spent
the
last
couple
of
months
trying
to
just
wrap
my
head
around
it,
and
it's
been
great
so
far
we're
looking
to
take
into
production
over
the
next
couple
of
days.
We
mostly
we
struggle
with
different
paths
here
and
there,
but
it's
mostly
because
just
trying
to
run
it
in
as
minimal
configuration
as
possible.
I
saw
that
loki
has
a
simple,
simple
head
chat.
It
would
be
great
if
we
could
maybe
have
one
of
those
as
well.
G
I
tried
setting
up
something
with
some
with
essentially
running
the
local
setup
on
you
know
and
create
and
chat
out
of
it,
but
it
didn't
really
work
out
the
way
I
was
expecting
it,
and
I
see
that
I
think
the
allotments
are
conflict.
Yeah
remember
can
actually
be
disabled.
Now,
with
the
fix.
That's
coming
up
in
you
know
in
2.1,
I
think
like
in
our
current
setup.
G
One
thing
I
would
like
to
add
to
the
metrics
themselves:
I'm
still
trying
to
work
out
how
I
really
make
the
most
use
of
them
like,
for
example,
currently,
the
memory
dashboards
like
setting
a
memory
data
source
on
graphing
itself
requires
that
I
mention
a
current
idiom.
Is
there
an
admin
view
that
I
can
have
of
all
the
tenants
as
such,
without
probably
doing
something
too
funky.
C
C
Of
dashboards,
which
are
called
the
tenant
dashboard
and
top
tenants
dashboard,
I
I
guess
I
guess
you
already.
You
already
know
them
which
gives
you
you
know
some
insights
on
a
per
tenant
usage.
So
typically,
when
it
comes
to
to
mimir
itself,
what
you
want
to
monitor
is
the
entire
health
of
the
system.
So
that's
why
most
of
our
dashboards
don't
allow
you
to
trim
down
into
pertinent
lieutenant
androids,
except
for
a
couple
of
dashboards.
I
I
I
just
mentioned.
C
G
Okay,
okay,
I
I
think
I
was
I'm
trying
to
just
approach
it
from
the
perspective
that
say
in
case
a
tenant
ends
up
or
is
essentially
trying
to
blow
up
the
number
of
labels
that
they're,
you
know,
passing,
maybe
passing
something
that's
going
to
create
very
high
cardinality.
I
just
noticed
that
there
was
a
there
was
a
function,
for
I
think
meters
in
in
one
of
the
release
notes
just
now.
G
Maybe
that
is
something
that
can
actually
help
us
with
I'll
check
out
that
and
I
have
been
having
some
trouble
setting
up
the
metrics
dashboard
specifically
for
member
itself.
I
think
the
default
dashboards
expect
the
four
five
requirements,
I'm
just
trying
to
figure
out,
if
maybe
like.
When
I
tried
to
run
them,
I
got
some
errors,
I'm
still
trying
to
work
out.
G
If
maybe
those
flags
are
not,
maybe
those
labels
are
not
passed
by
default
or
something
maybe
some
sort
of
default
configuration
for
prometheus
or
something
that
essentially
provides
hey.
You
know
what
these
are.
The
labels
that
will
essentially
help
meme
run
the
way
that
you're
expecting
that
that
would
also
help
again.
I'm
really.
D
G
C
The
the
dashboards
in
the
alert
make
some
assumptions
on
some
labels.
Each
of
your
metric
has
in
the
system
like,
for
example,
we
require
a
label
which
is
called
cluster
which
identifies
the
keeper
with
this
cluster.
What
is
running
here
and
then
namespace,
which
is
the
kubernetes
namespace
job
label,
is
another
one
that
we
require.
G
G
Yeah,
that
is
actually
something
that
I
think
the
like.
Any
help
would
be
awesome.
I
in
fact
I'd
love
to
be
able
to
even
maybe
contribute
to
some
other
documentation,
because
I
like,
for
example,
with
the
tutorials
and
things
like
I
found
that
maybe
some
hints
here
or
there
would
definitely
help
me-
maybe
reduce
the
amount
of
time
that
I
ended
up.
Spending
researching
for
it.
F
Okay,
let
me
make
a
note
of
this,
and
we
can,
I
think,
corroborate
once
we
once
we
have
this
move
done
of
the
chart
and
we
have
a
place
in
the
documentation
you
know
specified
by
our
doc
team.
I
I
think
we
can
start
up
and
you
know
we
can
call
upon
on
at
least
reviews
or
if,
if
you
are
like,
you
know,
you
would
like
to
try
out
what
we
have
in
that
in
in
the
future
getting
started
guide.
That
would
be
very,
very
good,
like
great
stuff.