►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Grafana Loki: Like Prometheus, But for logs. - Tom Wilkie, Grafana Labs
Loki is a horizontally-scalable, highly-available log aggregation system inspired by Prometheus. It is designed to be cost effective and easy to operate, as it does not index the contents of the logs, but rather labels for each log stream. Loki initially targets Kubernetes logging, using Prometheus service discovery to gather labels for log streams. As such, Loki enables you to easily switch between metrics and logs, streamlining the incident response process - a workflow we have built into the latest version of Grafana. In this talk we will discuss the motivation behind Loki, its design and architecture, and what the future holds. Its early days after the launch at KubeCon Seattle, but so far the response to the project has been overwhelming, with more the 4.5k GitHub stars and over 12hrs at the top spot on Hacker News.
https://sched.co/MPbj
A
All
right,
how's
everybody
doing
we're
gonna
go
ahead
and
get
started.
My
name
is
Tim
Hendricks
I'm
from
open
policy
agent,
Styron
I'm
your
track
host,
so
just
like
always
we're
gonna
save
at
least
five
minutes
for
Q&A
at
the
end,
please
go
ahead
and
stay
don't
get
up
early.
It's
pretty
disruptive
when
you
leave
also
remember
that
at
the
end
of
the
talk
go
ahead
and
rate
it
you
just
log
into
sched
to
do
that.
A
B
Hello,
everybody.
This
is
quite
a
big
room,
quite
daunting,
so
my
name's
tom
I
work
for
fan
labs
I,
also
welcome
from
atheists.
I
was
to
work
on
a
project
called
cortex
as
well,
and
there
was
a
couple.
I
gave
a
couple
of
talks
about
it
cubic
on
past
few
days
and
the
new
thing
I'm
working
on
is
Loki
when
I
do
get
spare
time,
which
is
very
infrequently
I
like
to
make
my
own
bit.
B
I
thought
I'd
include
that
for
some
reason,
I'm
actually
like
I'm,
going
off
script
here,
I'm
in
a
Twitter
war,
with
my
wife,
she's,
a
journalist
and
so
she's,
getting
way
more
followers
than
me
and
so
I'm
massively
whoring
myself
out
here
by
asking
so
yeah
I've
never
spoken
to
this
bigger
audience
before
face
it.
So
if
I
could,
if
I,
could
you
know
past
2,000
followers
on
Twitter
me
so
happy
and
my
wife?
So
sad,
so
you
know
make
that
what
you
will
so
that's
pretty
much
gonna
be.
B
This
is
the
last
talk
I'm
giving
a
cube
concert?
This
is
the
kind
of
you
know.
This
is
what
you're
gonna
get
basically,
so
we're
gonna
do
some
audience
participation
so
who
here's
already
using
Prometheus
hands
up?
Oh
wow,
that's
so
cool
who
is
already
tried,
Loki,
okay,
so
mostly
people
who
happen
great.
Who
here,
like
I,
gave
a
very
similar
talk
to
this
at
FOSDEM
in
February,
and
we
had
a
lot
of
feedback
that
people
liked
it.
But
I've
said
some
people
here
have
already
seen
it
so
who's.
B
Seen
the
FOSDEM
talk,
oh
okay,
so
I've
summarized
the
Fossum
talk.
I
might
just
give
the
pause
and
talk
again.
You
haven't
I'll,
go
to
the
summer
to
see
what
happens
and
then
finally,
who
here
cuz
of
the
keynote
like
okay,
that's
pretty
good
yeah.
Having
that
opportunity
to
speak
to
suddenly
people
was
was
so
awesome,
so
low-key,
what
is
Loki
Loki
is
a
horizontally
scalable
highly
available,
multi
tenant.
You
know
you
name
it.
It
basically
does
everything
log
aggregation
system
inspired
by
Prometheus.
We
started
the
project
quite
a
long
time
ago.
B
Actually,
but
it
went
for
a
big
long
where
we
didn't
really
have
enough
time
to
work
on
it
and,
and
we
only
really
got
to
working
on
it
and
kind
of
pushing
out
the
first
release
on
the
way
to
Cuba
con
us
I
mean
if
you
go
and
look
at
the
get
history
most
of
the
code
was
written
on
the
plane
on
the
way
to
Cuba
con
us.
So
it's
it's
six
months
in
it's
six
months
since
we
open
sourced
it.
The
the
like
the
response
has
been
astounding,
like
we
were.
B
We
spent
the
first
like
12
hours.
We
are
after
announced
at
the
top
of
hacker
news
where
over
6,000
github
stars.
Now
we
like
we've
just
had
so
many
people
coming
up
to
me
and
saying
they
love
it
and
they
use
it,
and-
and
this
is
like
the
first
time
I've
ever
experienced
that
so
that's
so
awesome.
We've
spent
the
last
six
months
like
listening
to
everyone's
feedback.
We've
hired
a
bunch
of
people,
our
graph,
our
labs,
to
work
on
it.
We've
got
a
load
of
open-source
contributors.
B
I
wanted
to
put
together
a
load
of
stats,
but
you
can
all
don't
get
up
and
see
them
like
there's
like
80
90
contributors
to
Loki
now,
which
is
just
amazing
for
such
a
new
project
and
we've
added
some
really
cool
features,
so
Loki
your
knowledge
to
make
a
lot
of
sense
later
and
when
I
demo
it.
But
Loki
takes
a
different
approach
to
log
aggregation
and
we've
added
some
of
the
things
that
people
want.
B
That
brings
it
a
little
bit
closer
and
allows
you
to
walk
the
trade
off
a
bit
better,
Gotham
I'm,
not
sure.
If
he's
here,
I
think
he's
in
the
other
room,
he
has
been
working
on
query
performance
or
crew.
Performance
has
come
on
leaps
and
bounds
in
the
last
six
months.
It's
a
little
work
to
do,
but
we're
getting
there.
And
then
we
started
developing
our
own
query
language
as
well,
and
then
finally,
like
today,
I
want
to
show
you
that
we've
added
the
ability
to
see
context
we've
added
the
ability
to
do
live
tailing.
B
That
was
like
one
of
our
most
requested
features,
but
the
doc
up
there,
the
Google
Doc.
That
was
the
design
doc
I
wrote
over
a
year
ago
now,
I
like
to
start
of
a
project
with
a
design
doc.
So
you
know
it's
public,
go
and
read
it
now,
give
you
the
wheel
in-depth.
So
what
are
we
doing
today?
Well,
that
was
the
intro
we're
gonna.
Do
a
summary
at
the
FOSDEM
talk,
it's
so
much
fun,
I
thought
I'd!
B
Do
it
again
and
then
we're
going
to
talk
about
what's
new,
where
we're
going
and
hopefully
do
a
demo
and
some
questions
later,
the
FOSDEM
talk,
I
just
recommend
you
go
and
watch
it
if
you're,
if
you
want
a
bit
more
in
depth,
I
think
it
was
kind
of
mid
day
if
anyone's
ever
been
to
Foss,
then
I
may
have
had
a
few
beers
by
this
point.
It's
a
good.
It's
a
good
conference
and
it
was
just
such
a
great
opportunity.
It
was
a
it
was
a
lecture
style.
B
Cuz,
it's
run
in
a
university,
so
it's
a
lecture
style
conference
like
everyone
was
like
up
in
front
of
me.
It
was
so
cool
anyway
we,
but
we,
the
FOSDEM,
talk
basically
focused
on
three
things
about
Loki.
It
focused
on
how
we've
tried
to
deliver
on,
like
the
simple
and
effective
to
our
own,
simple
and
cost
effective,
to
operate.
Promise
that
we
made
all
this
should
start
with
zero
by
the
way.
B
The
second
thing
we
talked
about
was
how
we've
integrated
it
with
existing
observability
tools,
as
I
kind
of
alluded
to
in
the
keynote
it's
all
about
that
joined
up
workflow
and
like
joining
the
pillars
together
and
then
the
airplane
mode,
as
I
said,
like
I
built
locate
on
a
plane,
I've
had
to
do
a
lot
of
traveling
recently
so
like
having
an
airplane
mode
for
your
software
as
well
as
making
it
cloud
later.
I
think
is
kind
of
neat
and
makes
makes
Loki
cool.
So
first,
what
I
mean
by
simple
to
scale?
B
Well,
most
log
aggregation
systems
will
index
the
contents
of
your
logs,
so
they'll
go
through
they'll.
Take
every
line
in
your
logs,
they'll
tokenize.
It
and
they'll.
Stick
it
in
a
massive
inverted
index
and
your
index
is
your
logs,
and
this
is
really
cool
right.
This
makes
it
makes
it
super
powerful.
It
makes
it
really
easy
to
ask
questions
like
how
many
log
lines
had
the
word
error
in
them
right,
which
a
lot
of
people
want
to
do
with
their
log
aggregation
system.
This
is
also
kind
of
how
Google
search
works
right.
B
It's
how
they
index
the
internet.
It's
the
same
technology.
The
challenge
here
I
think
for
me.
The
challenges
these
inverted
indexes
are
hard
to
scale.
You
know-
and
this
is
a
bit
bit
contentious
for
sure,
like
I,
think
other
people
would
say
they're
not.
This
is
my
experience
and
I
think
there's
a
bit
to
why
I
think
they're
hard
to
scale.
If
you
think
about
an
inverted
index,
you
take
a
log
line,
you
split
it
up
into.
You,
know
1015
tokens
and
then
you're
going
to
like
those
tokens
to
your
index.
B
B
Similarly,
if
you
want
to
optimize
it
for
reading
you
kind
of
want
to
localize
your
reads
so
you've
got
these
two
opposing
trade-offs.
When
scaling
an
inverted
index,
so
a
low
key,
we
we
do
it
completely
differently.
We
still
do
some
amount
of
indexing
as
I
kind
of
alluded
to
in
the
keynote
there's
degrees
of
indexing.
You
can
do
you
know
something
like
okay
log
by
Peter
eschewed.
All
forms
of
indexing
didn't
want
to
do
anything
like
wanted
it
to
be
as
simple
as
possible.
B
We
want
to
index
metadata
and
labels
about
where
the
logs
came
from
in
Prometheus
words,
we'd,
say
these
are
the
target
labels
and
we
give
you
this
little
bit
of
index
that
points
to
basically
a
stream
of
logs
and
the
stream
of
logs
internally
isn't
indexed.
This
allows
you
in
the
workflow
that
we'll
show
you
later.
This
allows
you
to
kind
of
filter
what
job
you
want
to
see.
What
Metin
you
know,
maybe
what
log
level
you
want
to
see,
what
host
what
server
etc
get
the
logs
for
that,
but
then
the
rest
of
it.
B
B
The
other
thing
is
these
streams
compress
really
well
like
all
the
logs
from
a
single
host.
You
get
a
lot
of
locality
in
the
stream.
You
get
great
compression
out
of
them
again.
I
kind
of
ran
out
of
time
to
get
real
numbers
here,
but
I'll
do
a
blog
post
in
the
next
week
or
two
and
I'll
actually
give
you
facts
then.
B
The
second
second
argument
is:
is
integrating
with
existing
systems,
so
I
work
for
a
company
called
Cortana
tirana,
labs,
I,
don't
actually
work
on
granite,
I
work
on
back-end
stuff,
so
I
welcome
Prometheus
on
cortex
on
Loki
this.
We
a
long
time
you
know
18
months
ago
or
more
two
years
ago,
I
built
this
workflow
I
built
this
slide.
In
fact,
they
kind
of
showed
what
my
day-to-day
activity
was.
As
a
DevOps
engineer,
you
know
I'd
get
an
alert
me
not
in
slack,
but
I
needed
a
picture
right.
I
get
a
lot
of
my
phone.
B
You
know,
I'd
follow
the
link
it'd,
take
me
to
a
dashboard
and
that
that's
what
I
click
around
you
know
maybe
use
the
dashboard.
Maybe
it's
a
red
style
dashboard
to
kind
of
hierarchically,
go
through
my
services
and
find
the
one
that's
causing
the
errors
causing
the
high
latency
and
then
I'd.
You
know,
then
I'd,
basically,
click
on
the
dashboard,
I'd
click
Edit
and
I'd
copy.
B
The
prompt,
ql
expression
out
into
the
Prometheus
expression
browser,
then
I'd
fiddle
with
it
I'd
fiddle
with
it
to
get
more
information
I'd
fiddle
with
it
to
find
maybe
the
exact
path
of
the
request:
that's
failing
or
the
exact
instance,
or
the
exact
host
and
whatever
it
was
I'd
end
up
fiddling
with
the
expression
and
that
copy
and
paste,
and
that
fiddling
was
sonic.
We
really
wanted
to
streamline
ingre
fauna
when
I
when
I
joined
Gore
fauna.
So
this
is
I.
B
Think
David
did
he
added
the
Explorer
mode,
which
allows
you
to
click
on
this
graph
and
and
go
basically
into
a
recreation
of
the
Prometheus
expression
browser
inside
Ravana
and
allows
you
to
do
that
fiddling
and
in
a
nice
way,
because
the
Explorer
mode,
I,
don't
know
why
I'm
talking
so
much
about
the
Explorer
mode?
It's
just
cool,
but
the
Explorer
mode
now
has
kind
of
cool
tab
completion
and
it
gives
you
suggestions
of
ways
in
which
you
can
improve
your
query
and
and
really
helps
you
at
that
first
step.
B
But
I
guess
the
reason
we
really
wanted
this
Explorer
mode
is
because
we
wanted
that
second
step.
We
wanted
to
be
able
to
once
we've
isolated,
maybe
the
instance
or
the
job
or
the
host,
that's
causing
the
high
latency
or
the
error
rates
we
wanted
to
be
able
to
with
one
click,
show
you
the
logs
for
that
host
and
that's
the
whole
motivation
for
Loki.
You
know
and
then
distribute
tracing,
obviously
as
kind
of
something
we'll
do
pretty
soon,
I'm
pretty
sure.
So
how
do
we
do
this?
B
Well,
we
deploy
an
agent
to
all
of
your
hosts.
We
need
a
log
collection
agent,
so
we
build
our
own.
It's
called
prom
tale.
We
don't
want
to
build
our
own.
No
one
wants
to
build.
You
know,
there's
a
few
things
in
world.
You
should
never
do,
should
never
build
a
database.
This
would
be
the
third
I've
built.
You
should
never
build
a
log
collection
agent.
Luckily
we
managed
to
hire
someone
who
did
a
really
good
job
of
making
this
one
better.
B
Anyway,
we
built
prom
tail.
What
prom
tail
does?
Is
it
embeds
the
Prometheus
service
discovery,
libraries
embeds
these
libraries
and
uses
the
Prometheus
integrations
to
collect
metadata
about
your
jobs?
So
it's
a
pod
name,
the
labels
associated
with
the
pod,
maybe
the
image:
that's
the
pods
running
this
stuff,
and
then
it
associates
that
data
with
the
stream
of
logs
it's
following
on
disk,
so
it'll
tail,
the
files
that
are
in
like
volleyball
or
docker
somewhere
it'll
tell
them
you'll,
associate
the
labels
with
it
and
send
that
to
Loki
the
reason
we
did
it.
B
You
just
have
to
use
the
same
conflict
Prometheus
users.
That's
why
we've
called
it
prom
tale
because
it
kind
of
really
heavily
Prometheus
inspired,
that's
how
we
keep
the
labels
consistent
and
by
keeping
the
labels
consistent.
We
can
enable
that
kind
of
experience
of
switching
between
the
two
seamlessly.
So
hopefully
everyone
gets
that
mystery.
Think
nope
right,
so
airplane
mode
is
the
next
one.
I
realized
I
started
off
with
zero
one
one.
Three,
that's
embarrassing
to
do
so
many
slides.
So,
as
I
said,
I
needed
to
be
able
to
run
this
on.
B
My
laptop
I
needed
to
be
able
to
develop
quickly
on
my
laptop,
but
also
I
worked
on
another
system
called
cortex,
and
when
we
built
cortex
we
you
know
we
wanted
to
do
it.
The
cloud
native
way:
microservices
docker
eyes
cloud
dependencies
BigTable,
you
know,
s3,
you
name
it.
We
wanted
to
use
all
that
all
the
cool
stuff.
The
problem
we
then
found
out
is
that
it
was
really
hard
to
run
like
you
had
to
have
a
kubernetes
cluster
running
in
one
of
these
clouds
took
to
run
it.
B
You
had
to
provision
everything
configure
everything
correctly,
and
there
was
just
a
lot
of
moving
parts
in
cortex.
So
when
I
wrote,
Loki
I
wanted
to
avoid
that
experience
like
I
wanted
to
give
people
a
really
nice
out-of-the-box,
one
command
runs
Loki
and
it
just
works,
and
this
was
also
like
you
know.
Kind
of
selfishly
I
also
wanted
more
people
to
use
Loki
than
ever
ended
up
using
cortex
I
also
wanted
more
people
to
contribute
to
Loki
than
ever
contributed
to
quartet,
and
so
we
tried
this.
B
We
Loki
uses
a
lot
of
Saint
coders
cortex
to
achieve
all
of
the
kind
of
dhts
and
and
eventual
consistency
and
all
of
the
kind
of
algorithms
are
just
cortex
algorithms,
but
we
package
it
up
as
a
monolith
and
I
know.
Like
someone
had
a
cloud
native
conference
talking
about
monoliths,
I'm
gonna
get
sued
off
stage
soon,
I
think
so
we
packaged
it
up
as
a
monolith
and
we
made
it
so
one
command.
You
could
run
it
on
your
laptop.
B
We
also
added
stubs
to
to
allow
you
to
run
with,
like
local
databases
like
bolt
DB,
an
embedded
database.
You
know
so
you
could
store
the
logs
on
disk
and
this
way
we
gave
you
an
experience
where
yes,
I
could
run
out
on
an
airplane,
but
you
can
also
run
it
on
your
laptop
or
on
your
on-prem.
You
know
servers
or
you
know
in
your
kubinashi's
cluster,
that
isn't
running
in
the
cloud
or
even
if
it
is
running
in
the
cloud.
B
Maybe
you
don't
want
to
spin
up
a
big
table
instance,
because
big
table
can
be
expensive
and
stuff
like
that,
so
we
built
this
monolith,
but
now,
of
course,
we
run
trendy
and
we're
only
building
long
lists,
so
we
added
a
flag
to
the
monolith
to
say,
oh
by
the
way,
we
also
want
you
to
behave
like
a
micro
service,
so
you
can
optionally
choose
to
do
the
complicated
cortex
architecture.
This
is
how
we
run
low
key
in
production.
We
deploy
I,
think
it's
like
9
or
10
services
that
together
make
up
low
key
right.
B
You
don't
have
to
use
one
one,
but
we
run
10
and
you
know
if
you
want
to
run
10
as
well.
You're
welcome
to
do
that.
It's
just
a
flag,
the
cool
thing
about
this
architecture
and
the
reason
you
know
we
did
it
with
cortex
and
the
reason
Mike
services.
So
a
good
idea
is
we
get
to
isolate
the
query
path
from
the
right
path.
For
instance,
we
get
to
scale
them
independently.
You
know
if
you
get
a
query
of
death.
B
If
you
get
someone
it
sends
as
a
query
that
decides
to
load
a
terabyte
into
a
16
gig
pod
like
it
will
kill
that
pod.
You
know,
but
it
won't
kill
the
ingestion
path.
So
you
can
kind
of
separate
concerns
about
reads
and
writes
and
separate
the
reliability
of
the
two.
We
we
have
this
nice
little
model
where
we
like
to
move
quickly.
We
like
to
merge
stuff,
we
what
we
deploy
earthmaster.
B
We
continuously
deploy
this,
but
when
you
touch
the
in
gestures,
which
are
kind
of
the
stateful
bit
and
a
bit
sensitive,
we
like
to
do
a
bit
of
extra
code
review,
but
because
the
query
is
and
the
query
front
ends
and
so
on,
they're
all
like
stateless
and
if
they
break
we
just
roll
them
back
like
we
like
to
move
a
bit
quicker.
We
like
to
L,
do
TM
stuff
a
bit
quicker
there,
so
yeah
Loki's
kind
of
optionally
micro
service-oriented
if
you
want
so
that's
the
really
brief
summary
of
the
the
FOSDEM
talk.
B
The
FOSDEM
talk
was
I,
think
40
minutes.
So
if
you
want
more
details,
I
just
recommend
you
go
and
read
that
and
I
think
we
will
do
a
write-up
of
it
as
well.
So
simple
and
cost-effective
I've
used
number
two
here
like
I,
should
have
integrated
with
existing
observability
tools
and
an
airplane
mode,
but
also
being
cloud
native.
B
So
halfway
through
what's
new
I
mean
that
was
kind
of
six
months
ago
right
that
was
where
we
started.
We've
had
a
huge
amount
of
feedback
like
it's
been
overwhelming,
I've
never
been
involved
in
anything
this
successful,
so
we've
started
to
started
to
respond
to
some
of
it.
You
know
we
had
to
hire
a
few
more
people.
I
can't
you
know,
write
all
this
code
myself
and
we've
got
to
the
position
now,
where
I
think
we're
doing
pretty
well
we're
like
this
close
to
being,
like
you
know,
ready
for
our
first
release.
B
I
was
really
hoping
to
stand
here
and
announce.
We're
doing
you
know
low-key,
naught
point
one,
but
just
a
little
bit
so
I
tell
you
what
we've
done.
The
four
things
I
want
to
show
today
that
we've
added
two
loci,
our
log
filter
chaining.
This
is
our
first
step
towards
our
own
query:
language
for
loci.
The
most
common
thing
people
want
to
do.
You
know
to
fulfill
that.
Just
give
me
grep
is
like
filter
their
logs,
but
also
like
filter
them
in
a
negative
sense,
filter
them
in
a
positive
sense.
B
Not
everyone
likes
writing,
regular
expressions.
So
sometimes
you
just
want
equality,
so
we've
added
log
filter
chain.
You
can
have
as
many
of
these
as
you
want
extracting
labels
from
logs
I
mentioned
that
earlier.
So
we
we
run
internally.
I
hope
the
demo
works.
I
really
do
we
run
internally.
We
extract
for
the
debug
level
from
the
logs,
so
this
now
you
know
we
include
the
debug
level
in
our
index.
B
Every
stream
that
comes
in
gets
split
into
four
streams,
and
this
is
kind
of
cool
because
it
allows
you
a
little
bit
more
power
like
the
work.
The
use
case
I
really
want
to
show
you
know
in
the
next
few
weeks,
is
the
ability
to
add
an
annotation
to
your
pod.
That
says,
like
oh
I,
just
want
to
record
in
low
key.
B
Like
you
know,
warnings
and
errors
and
info
I
can
I
want
to
report
debug
and
then,
if
you
see
an
error,
you
can
like
go
along
and
just
by
changing
the
annotation
in
your
pod,
you
can
change
it
to
do.
You
know
a
full
debug
logging
back
to
low
key.
You
don't
have
to
change
your
code,
but
it
will
just
be
a
done
through
Prometheus
relabeling
rules.
So
that's
like
next
time,
so
we
can
now
extract
labels
from
logs
one
of
the
biggest
things
we
were
asked
for.
B
His
live
tailing
like
people
love
log,
cube,
catalogues,
F.
They
really
like
that
workflow,
and
so
they
wanted
to
see
that
in
brief
honor.
So
we've
got
that
/.
If
we
can
show
you
that
in
context,
you
know
again
modeled
after
grep
people
like
to
build
pipelines
of
greps
the
grep.
Has
that
see.
You
know,
copper
context.
You
know
I
want
to
see
three
lines
before
a
match
and
four
lines
after
a
match,
so
we've
added
that
to
Loki
yeah
and
there's
much
more.
So
let
me
see
how
good
the
Wi-Fi
is.
B
Yeah,
don't
remain
oh,
so
I'm
just
we're
just
gonna
go
to
our
instance
of
the
Wi-Fi
seems
good
we're
going
to
go
to
our
instance
of
loki,
that's
running
in
a
dedicated
cluster
inside
refine
labs
and
we
we
run.
I
don't
know
the
final
count.
It's
like
fifteen
to
thirty
kubernetes
clusters
distributed
around
the
world.
We
know
we've
got
ones
for
dedicated
specific
applications.
We've
got
one
dedicated
to
specific,
like
customers
and
regions
and
so
on,
and
we
like
to
get
a
global
view
of
all
our
logs
in
one
place.
B
So
we
run
an
OP
stores
cluster.
Where
we
do
that
I
know,
you
didn't
see
a
login,
that's
because
it's
behind
Google's
IP
proxy.
So
good
luck!
So
let's
do
the!
Let's
do
the
workflow
I'm
we've
got
this
demo
app
called
the
TNS
demo.
Let's
actually
do
a
port
forward.
This
was
me
trying
to
run
it
locally
earlier.
Didn't
work,
cube,
cuttle,
config,
use
context,
ops
tools,
one
port
forward,
namespace
TNS
demo
come
on
come
on
there.
B
We
go
service,
80,
80,
80,
okay,
so
we
built
this
demo
because
I
feel
like
like
whenever
people
show
you
demos,
it's
a
fake
thing
like
it
doesn't.
Have
a
UI
and
I
really
want
my
demos
to
have
real
things.
So
so
I
built
a
hacker
news,
clone
they've
all
got
fake
stories.
These
are
not
real
stories.
They're
supposed
to
be
a
joke.
Don't
take
them
too
seriously.
B
Like
the
links
work,
but
so
just
to
show
you
there
is
something
going
on,
but
sometimes
it
fails.
Why
is
that?
So?
We
can
go
to
our
demo
app
and
see
that
sometimes
it's
failures
are
low,
so
we're
going
to
go
and
well
we've
seen
that
the
database
is
periodically
failing,
so
I
can
I
can
go
and
load
up
the
Explorer
view,
and
this
shows
me
yes
there's
definitely
now
you
can
see
that
I
can
kind
of
include.
You
know.
Maybe
I
want
to
see
it
by
instance.
B
You
know:
okay,
there's
only
one
instance.
That's
not
helpful
and
I
can
also
say
well,
I'm
interested
in
only
those
500s,
so
you
can
click
there,
and
this
is
really
designed
for
people
who,
like
maybe
aren't
that
confident
with
prompt
QL
and
we
can
just
help
them,
build
it
and
find
what
they
want
and
when
you're
kind
of
getting
paged
8:00
a.m.
just
before
you're
about
to
go
on
the
keynote
stage.
This
is
really
useful
that
actually
happened
by
the
way.
B
So
now
we
can
zoom
in
on
one
of
the
particular
spikes,
because
we
can
see
there
was
a
spike
five
QPS
of
500s
at
that
point,
and
we
go
up
here
and
we
can
say,
I
want
to
see
the
logs
for
that
and
then
there
you
go.
There's
the
logs.
You
can
see
various
errors
in
there.
That's
pretty
cool,
that's
the
logs.
Just
for
that
we
can
see.
B
Oh
that's
all
for
the
same
label,
so
doesn't
doesn't
do
anything,
but
this
is
like
there's
a
lot
of
stuff
in
here
right,
so
maybe
I
just
want
to
see
errors
and
I
can
filter
down.
Just
by
era,
I
tried
to
demo
this
on
someone
else's
keyboard,
but
it
was
a
Norwegian
keyboard
and
I
couldn't
find
the
you
know
the
pipe
operator.
So
here
we
go
now.
We've
seen
its
filter
by
us
and
we
can
even
like
do
pattern
that
you
are
moment
we
can
do.
Multiple
of
these,
we
can
do.
B
So
there
we
go
and
then
one
final
thing
we
can
do
is
we
can.
We
can
do
some
kind
of
ad
hoc
analysis
of
the
thousand
results
we
got
in
the
browser.
This
is
done
browser
side
right
now.
We
really
want
to
do
well.
I'll
talk
about
a
second
I'll
show
what
we
want
to
do
in
a
second,
but
one
of
the
things
that
is
kind
of
cool.
So
you
see
we're
in
this
view
and
we
filter
by
eros
or
only
seeing
errors
right,
but
I
want
to
see.
B
Maybe
that
error
is
going
to
be
preceded
by
a
stack
trace
that
doesn't
include
the
term
error,
so
I
want
to
see
the
context
that
might
include
that
stack
trace,
and
so
we
can
click
on
show
context,
and
this
is
then
going
back
to
Loki
doing
a
couple
of
queries,
and-
and
this
is
your
grep
see,
kind
of
workflow
and
I.
Just
think
this
is
so
cool
like
I
should
actually
include
stack
traces,
so
it
works
even
better
yeah.
B
What
was
the
other
thing?
I
was
going
to
show
you
our
life
tailing.
Yes,
so
let's
go
to
the
last
15
minutes
and
let's
get
rid
of
that
and
then
I've
never
actually
done
this,
because
I
was
told
by
David
that
it's
working
and
I
trust
him
that
much
so
here
we
go.
We
click
here
and
we
click
live
and
then
what
happens
so
received
and
it
takes
10
seconds
because
for
various
reasons
you
have
to
leave
a
buffer.
You
have
to
have
a
delay
to
get
things
in
order
and
then
work.
B
B
Yeah
it
didn't
work.
Did
it
yeah?
Well,
there'll,
be
a
better
demo
online
I
promise.
So
the
other
thing
I
want
to
show
you
showing
you
dog,
training,
the
extracting
label
some
logs.
Have
we
got
a
log
level
in
our
thing,
yeah
we
we
do
not
know
well,
two
out
of
four:
that's
not
bad
for
a
demo.
So
let's
carry
on
so
that's
the
demo
to
our
for
what's
next.
So,
as
I
said,
like
the
biggest
question
we
get
about,
Loki
is:
when
can
I
use
it
in
production
we
use
it
in
production.
B
B
I
wasn't
expecting
you
to
laugh
at
that,
so,
but
now
it's
really
stable.
Actually,
like
we're
really
happy
with
it,
we
were
going
to
cut
the
beta.
The
first
Peter
like
a
naught
point,
one
version
really
soon,
but
don't
let
any
of
this
stop
you
like
go
and
use
the
master
branch.
It's
good!
It's
stable!
It
mostly
works.
As
you
see
we
I
alluded
to.
We
want
to
do
that
kind
of
aggregation
view
that
you
saw
showing
you
how
what
percentage
of
errors
were,
which
message
we
want
to
do
that
server-side.
B
B
This
is
just
some
ideas
like
it
might
not
look
like
this
there's
a
design
doc,
that's
linked
in
the
top
right,
I'll
upload,
the
slides
afterwards,
and
you
can
go
and
check
out
the
design
doc
we're
like
I,
think
Cyril
may
I've
got
like
20
unread
messages
from
Cyril
right
now,
so
I
think
he's
implementing
this
right
now
and
he's
probably
also
telling
me
my
life
tailing
is
not
working
so
yeah
we
wanna.
We
want
to
be
able
to
implement
some
basic
aggregations,
so
you
can
start
using
Loki
to
plot
graphs.
B
So
you
can
start
using
Loki
to
do
numeric
analysis.
Loki
will
never
be
for
business
analytics.
You
know,
that's
not
what
we're
trying
to
build
we're
trying
to
help
developers
debug
and
troubleshoot
their
applications,
so
we're
never
going
to
be
able
to
execute
these
queries
as
fast
as
like
the
fully
indexed
log
aggregation
systems,
but
it
doesn't
mean
we
can't
try,
and
so
we've
got
plans
around
like
this
fully
parallelized
streaming
query
engine
that
we're
building
which
I
think's
gonna,
be
really
cool.
I'm
really
excited
about
that.
B
So
once
we've
got
that
kind
of
numeric
aggregation
in
log
QL,
we
want
you
to
be
able
to
use
the
Prometheus
machinery
to
build
an
alerts
off
of
it
like
it's,
you
know,
I'm,
pretty
opinionated
about
this
stuff.
I,
don't
think
you
should
be
using
logs
to
do
you
alerts,
but
you
know
not.
Everyone
lives
in
a
panacea,
and
sometimes
your
application
will
only
log
error
messages
and
sometimes
it
won't
expose
metrics.
So
we
wanted
to
support.
We
want
to
help
you
kind
of
that's
kind
of
use.
B
Cases,
especially
actually
somebody
came
up
to
me
earlier-
gave
me
a
really
good
example
of
like
kernel
drivers.
You
know
I
doubt
we'll
ever
get
the
Prometheus
instrumentation
libraries
into
the
kernel
and
a
lot
of
drivers
are
only
ever
write,
error
messages
to
D
message,
so
we
want
low
key
to
support
the
use
case
where
we
can
count
those
error
messages
that
are
happening
in
D
message
and
now
you
to
alert
on
them.
B
So
this
was
a
really
good
idea
that
someone
came
to
the
came
to
the
booth
in
the
hall
and
said,
like
I,
really
want
to
use
low
key
for
auditing,
but
I
want
you
to
like
add
this
whatever
that
means
signature
changing,
so
we
can
verify
that
they've
not
been
tampered
with
I,
don't
know
how
to
do
this.
I'm
gonna
have
to
go
away
and
do
some
research,
but
if
any
of
you
know
how
to
do
this,
Piazza
welcome.
B
You
know
we
want
to
do
this,
I
think
it's
a
cool
use
case.
We
talked
about
airplane
mode
and
bolt
DB,
but
in
production
you
still
probably
born
will
be
using
something
like
BigTable
or
GC
or
what
DynamoDB,
and
so
we
want
to
remove
that
dependency.
We
want
you
to
be
able
to
run
in
production
on
bolt,
DB
I,
think
that'd
be
really
cool,
and
then
we
want
to
launch
the
first
peter
as
soon
as
possible
and
with
that
I'm
done.
Thank
you
very
much.
C
First
question:
thanks
for
the
talk
and
thanks
for
the
tool,
I
think
it's
very
awesome.
My
question
would
be
you
mentioned
that
you
don't
like
alerting
out
of
logs.
Unfortunately,
some
of
my
colleagues
are
using
metrics
out
of
the
logs.
Do
you
think
Loki
can
help
me
in
this
I?
Don't
like
it,
but
I
have
to
support
that
ya.
B
D
B
Know
so,
can
we
take
logs
and
give
them
a
bit
more
structure,
sure
like
in
the
agent
prom
tale,
we
already
have
the
ability
to
like
ingest
log
format,
logs
or
JSON
logs
use
a
JSON
path
expression
to
take
like
a
field
out
of
that
logs
and
promote
it
into
the
index.
So
that's
how
we're
doing
log
level
internally
that
I
can't
show
you
so
yes,
it's
short
answer
is
we
can
take
some
things
now.
E
B
Our
back,
did
you
say
her
back.
Yes,
role
based
access
control,
yes,
yes,
Loki's
is
a
multi-tenant
system
already
and
it
kind
of
didn't
cover
it
in
the
talk.
But
when
you
push
data
into
Loki,
you
can
specify
a
tenant
ID
and
that
will
keep
it
isolated
from
other
tenants
and
then,
when
you
specify
queries
again,
when
you
send
queries,
you
specify
ten
ideas
at
HTTP,
header
and,
and
we
already
already
do
roll,
but
we
already
do
multi-tenancy
now,
role
based
access
control
isn't
the
same
as
multi-tenancy.
B
Obviously
like
you
need
to
be
able
to
put
a
gateway
in
front
of
that
and
really
like
the
expectation
is,
you
could
put
a
gateway
in
front
of
that
to
enforce
authentication
and
authorization
to
do
that.
One
of
the
things
we've
been
toying
with
the
idea
of
is
putting
like
whitelist
and
blacklist
filters
in
two
loci
to
say
that,
like
this
tenant
can
only
access
you
know
met
logs
that
match
this
set
of
matches
because
I
think
the
match
has
syntax
is
really
nice
to
specify
these
things
in,
but
yeah
like
ideas
are
welcome.
B
B
We
don't
yeah,
it's
really
hard
right
like
that's
why
we
did
context
right
so
that
you
can
see
for
that
stream.
What
came
before
and
after
it
like
I
I,
don't
know
enough
about
how
other
people
deal
with
it,
but
I've
not
seen
a
nice
solution.
To
that
I
mean
then
I
get
ya
know
we
don't
like
that's
what
context
is
for.
G
Hi,
first
time,
hearing
about
lucky:
fantastic,
really,
nice!
Thank
you
very
much,
so
we
using
else
else
like
for
log
base,
monitoring
and
exporters
permutation
graph
owner
for
metric
based
monitoring.
My
question
is:
would
you
have
such
feature
that
loki
is
besides,
you
know
this
kind
of
querying
and
seeing
logs
and
metrics
side
by
side
on
graph
on
a--
with
you
generate
some
automatic
URL
that
could
point
back
to
cabana
and
you
could
basically
search
based
the
similar
filters
there
as
well
to
see
the
logs.
So
I.
G
B
Don't
know
the
exact
answer
to
that,
I
don't
work
on
the
front
end.
I
know
we
are
working
on
adding
an
elastic
data
source
for
the
explorer
view.
So
maybe
it's
not
via
URLs,
but
maybe
you
can
build
the
same
workflow
with
elastic
and
other
monitoring
systems,
but
then,
like
the
point
of
loki,
is
that
that
workflows
automatic,
like
it
just
works
right,
whereas
in
other
systems
you
have
to
go
and
set
them
up
and
configure
them
so
that
the
labels
make
sense
in
both
of
them.
B
G
B
And
a
good
question
like:
what's
the
next
step
after
exploring
logs
I
mean
what
I
really
want
to
do,
I
mean
we,
the
whole
of
the
loci
stack
and
tracing
stacking
beds,
open
tracing
and
I
now
have
to
go
and
head
open,
telemetry
as
well,
and
they
emit
in
the
log
lines,
trace
IDs
and
so
that
workflow
I
alluded
to
in
the
keynote
about
clicking
on
a
trace,
ID
and
going
to
your
Zipkin
or
yeag
or
whatever.
That
I
think
is
a
next
step
for
us.
The
exemplars
is
another
great
example.
B
We
don't
have
that
yet,
but
I
really
want
to
add
that
support
in
graph
honor
and
basically
I
want
graph
honor
to
do
like
trace,
visualization
or
at
least
link
to
other
tracing
systems.
That's
probably
where
we're
going
to
go
in
the
next
kind
of
six
to
twelve
months,
but,
like
suggestions,
are
welcome
right.
I
B
You
can
do
it
now
with
Loki.
You
can
Prometheus
yes,
so
your
you
can
set
filters
in
prom.
Tell
and
pom
tell
will
export
the
number
of
logs
that
match
a
certain
regular
expression,
for
instance,
and
that
then
you
know
you
can
scrape
that
in
Prometheus.
You
can
rate
that,
and
you
can
say
if
the
rate
of
errors
is
greater
than
X
far
and
low,
but
I
want
to
make
it
better
right.
I
want
to
be
able
to
do
it
server-side
as
well
a
problem
with
doing
it
like
this.
B
J
Hello,
hello,
rest
of
all,
thank
you
for
lucky.
It's
really
great
and
the
second
question
you
little
bit
cheating
on
the
first
slide,
so
you
show
that
your
work
flow
means
that
you
have
an
alert
and
in
the
lucky
for
now
you
don't
have
any
errors.
What
do
you
think
about
this
and
how
would
it
should
be
shown-
and
maybe
we'll
use,
use,
method
or
read
method
for
this
scheme?
So
it's
very
interesting.
B
Well,
so
we
we
have
alerts
off
of
prometheus
alerts
right,
we
use
them
expensively
and
so
the
first
step
of
really
hey.
We
get
an
alert,
then
we
go
and
look
at
the
dashboard.
That's
really
talking
about
that
kind
of
workflow
button,
but
no
yeah.
We
don't
have
a
locally
alerts
yet
and
it's
something
we're
working
on
like
fair
point.