►
From YouTube: TGI Kubernetes 081: Grafana Loki
Description
Join Christian Dickmann as he hosts his first TGIK exploring Grafana Loki. Index-Free log aggregators, like Loki, promise drastic reduction in storage cost, while serving the logging needs in dynamic Kubernetes environments.
A
All
right,
hi
everyone-
this
is
Christian
I've,
been
with
VMware
for
11
years
11
and
a
half,
almost
yeah,
I
think
so,
and
I've
been
working
on
v,
Sanford
lost
eight
and
a
half
years
V,
since
our
enterprise,
storage,
product
and
Joe
invited
me
a
few
weeks
ago
to
host
one
of
these
TGA
case,
and
there
was
somebody
on
the
chat
who
was
asking.
What
does
it
stand
for
its
thank
God?
It's
Friday
or
in
this
case
thank
God,
is
kubernetes
and
so
yeah.
A
A
B
A
A
This
looks
better
yeah
I
wanted
to
talk
to
you
about
the
final
loci
today,
but
before
we
do
that
we
do
the
usual,
which
is
sort
of
a
community
update
and
and
what's
happening
out
there
in
kubernetes
I'm,
actually
not
an
expert
yet
in
the
community.
I'm
just
starting
I
actually
commented
on
my
first
pull
request
or
not
on
somebody
else's
pull
requests
about
a
cap.
Just
a
week
ago,
my
first
contribution
to
kubernetes
so
I'm,
still
ramping
up
I
was
able
to
attend
Kubek
on
EU.
A
That
was
exciting,
so
I'm
new
to
this,
and
our
experts,
who
have
been
with
kubernetes
for
years,
have
actually
helped
me
set
up
for
today.
So
they
have
been
setting
up
a
bunch
of
things
that
we
can
talk
about
a
little
bit
more,
in
particular,
actually
zoom
in
a
little
bit
clearer,
going
through
a
little
bit
of
an
update.
Actually
given
that
I'm
new
to
the
community
as
well
probably
newer
than
most
of
you
on
on
this
feed.
A
I
wanted
to
just
do
a
shout
out
to
those
projects:
I'm,
actually
not
not
an
expert
on
any
of
these,
but
Q
Khan
Shanghai.
Just
just
happened
or
is
happening,
and
all
the
videos
are
already
starting
to
pour
in
this
is
pretty
awesome.
I
saw
the
same
thing
with
Q
Connie.
You
was
really
exciting
to
see
everything
up
really
quickly
afterwards.
So
I
will
personally
be
checking
out
the
videos
over
the
weekend,
probably
or
next
week,
to
see
what
what
was
presented.
I
didn't
get
a
chance
to
do
it
live
this
week.
A
So
this
going
to
be
interesting.
Also,
the
CFP
process
is
open
for
San
Diego
and
it's
actually
coming
up
on
July
12th.
It
will
close
and
actually
I
need
to
get.
My
personal
stop
mission
in
I,
hope,
I
have
an
interesting
topic
and
it
will
get
selected,
but
you
should
all
you
know,
take
a
look
at
that:
Hydra
Strasberg,
another
European
person
joining
really
late
in
the
day,
so
appreciate
that,
thanks
for
that.
A
So
alright,
so
basically
that's
let's
go
through
the
other
thing.
So
this
week
there
was
no
kubernetes
release
which
is
weird
to
me,
I'm
from
VMware.
We
do
release
this
every
like
six
months.
We
thought
that
was
really
fast.
We
used
to
do
18
months
or
24
months
when
I
joined
the
company.
So
the
notion
that
it's
big
news
that
there
was
no
release
this
week
is
actually
a
big
deal
for
me,
but
there's
some
interesting
articles
about
1:15
pouring
in
and
personally
the
the
future
of
Sierra
DSS
has
been
really
interesting
to
me
because.
A
We're
we're
working
on
a
few
CR
ds2
on
the
kubernetes
front,
and
you
know
they
are
they're
still
a
little
bit
immature
I
guess:
they're,
there's
a
lot
of
things
that
we
can
improve
about
CRTs
and
how
they're
done
and
there's
a
lot
of
work
that
actually
went
into
1:15
and
there's
a
great
article
about
this
and
I
will
personally
be
checking
out
more
about
all
the
updates
on
that
front
and
try
them
out.
I
haven't
seen
it
yet.
A
I've
mostly
saw
it
on
Twitter
that
there's
a
lot
of
improvements
and
they
sound,
really
exciting,
there's
also
obviously
116,
starting
now
with
115
just
hitting
the
market
so
check
out.
You
know
a
lot
of
that
material
about
you
know.
What's
coming
on,
116i
I
will
again
being
new
to
the
community.
I
will
actually
do
this
as
well
and
start
checking
out.
A
We
have
the
CNC
of
doing
a
webinar
on
on
port
security
policies
and,
if
I'm
not
mistaken,
we
actually
did
a
TGI
K
on
port
security
policies
just
two
weeks
ago.
This
is
an
interesting
topic
to
me
as
well,
because
you
know
as
more
and
more
production
workloads
are
on
kubernetes
I
guess,
there's
a
lot
more
need
for
strong
security,
and
our
Beck
has
been
one
of
the
things
I've
been
looking
at
and
I
think
there's,
there's
lots
of
interesting
stuff.
That
is
already
happening
in
that
that
could
be
happening
so
this.
A
A
A
So
I
guess
this
is
a
Raspberry
Pi
with
seven
modules,
and
you
know
the
seven
modules
basically
run
around
seven
kubernetes
nodes,
I
guess,
and
so
we
have
a
whole
bunch
of
stuff
all
running
in
in
such
a
small
form
factor.
So
that's
that's
pretty
exciting
and
it's
pretty
small
form
factor.
So
any
hardware,
geek
will
will
probably
be
super
excited
about
what's
happening
in
that
space.
I
know
that
for
vmware
products,
home
labs
has
always
been
a
big
thing.
So
you
know
getting
down
to
that.
A
Size
is
actually
quite
exciting
and
I
think
something
that
as
vmware
we
haven't,
you
know
we
haven't
scaled
all
the
way
down
to
that.
Yet
I
believe
so.
This
is.
This
is
interesting
and
another
community
update
here
is
this
see
a
CD,
a
blog
post
series,
and
this
is
actually
something
I
want
to
check
out,
because
CSID
is
something
that
I
personally
worked
on
at
VMware
for
several
years.
We
build
a
lot
of
homegrown
systems
because
at
the
time
that
we
did
them,
there
wasn't
much
out
there.
A
But
you
know
these
days,
there's
a
lot
lot
happening
and,
in
fact,
I
think
who
Bonelli's
offers
an
opportunity
to
do
a
lot
more
on
CS
CD
in
terms
of
actual
integration.
Testing
between
you
know,
putting
together
a
whole
application
with
all
its
micro
services,
and
so
I
will
personally
be
checking
out
this
blog
series
and
and
see.
A
What's
going
on
to
me
a
really
exciting
topic
that
I
think
yeah
will
evolved
quite
a
bit
over
the
next
few
years
as
well,
and
and
finally,
we
have
an
article
about
debunking
security,
myths
and
I
guess
this
is
one
of
the
interesting
things
about
kubernetes.
We
have
a
large
community
and
there's
a
lot
of
information
out
there
and
a
lot
of
information
that
you
might
not.
Oh,
the
sound
is
low.
A
Can
other
people
confirm
that
my
volume
is
low
because
earlier
we
had
it
a
little
bit
louder
and
people
suggested
that
I
should
tune
it
down
by
the
way
hi
Sweden,
no
Spain
Frederico
thanks
for
joining
all
right
sounds
like
it's.
It's
a
bit
low.
Alright!
So
let
me,
let
me
see,
let
me
make
it
slightly.
A
A
Hi
India
guys
you
may
need
to
sleep
like
India,
must
be
middle
of
the
night
right
now
like
midnight
or
or
something
like
that,
so
excited
for
you
to
be
here,
but
make
sure
that
you
sleep
sleep
is
important,
so
so
yeah.
Basically
you
know
a
bunch
of
myths
around
security
and
someone
from
our
team.
Actually,
the
Oh
1:40
a.m.
all
right,
that's
even
worse
than
I
thought.
A
Yeah,
the
the
settings
getting
them
right.
This
has
been
very
tricky
thanks,
Joe
and
in
fact
you
know,
I'm
a
digital,
nomad
I,
actually
don't
have
a
real
setup,
so
I'm
in
Palo
Walter
this
week
and
I'm
using
our
office
here
to
do
this
at
all,
because
I
try
to
do
this
from
home
with
the
equipment
I
have
at
home
and
that
just
didn't
work
at
all.
A
So
you
know
had
to
scrounge
up
the
equipment
and
you
know
I'm
not
set
up
like
Joe,
who
has
basically
a
full
recording
studio
at
this
point,
so
you
know
I
hope
this
setup
is
working
for
everybody,
but
yes,
so
so
bunch
of
myths
around
here
around
security.
You
know,
port
security
policies
are
apparently
not
hard.
I
have
verified
this
myself,
but
you
know
a
bunch
of
interesting
things
here
that
I
will
be
checking
out
as
well
and
so
take
a
look
at
this.
A
A
So
with
that
I
think
I
will
I
will
start
diving
into
into
what
I
have
well
semi
prepared
for
today.
You
know.
For
me
this
is
weird
not
being
fully
prepared,
not
having
scripted
everything,
but
Joe
tells
me
that's
the
whole
point
of
this.
This
format
that
you
see
me
life
and
you
see
me
screw
up.
So
that's
the
exciting
part
of
about
this.
A
A
That's
super
interesting
I
should
actually
do
an
analysis
and
then
I
thought
well,
if
I'm
checking
it
out
if
I'm
doing
analysis
well,
actually
I
should
be
sharing
this
with
the
community,
and
I
should
be
using
this
as
a
chance
to
do
a
first
T
gik.
So
here
we
are,
and
essentially
what
Loki
is,
is
a
index
free
log
aggregator?
A
My
first
question
to
tomboys:
what's
index
free
about
it,
and
so
basically
the
notion
is
that
existing
log
aggregation
systems
like
Splunk
or,
if
you
use
like
elasticsearch,
they
are
all
pretty
much
sort
of
strong
on
on
being
able
to
search
for
anything.
Everything
that
comes
in
all
the
strings
that
come
in
are
heavily
tokenized
so
that
you
can
do
an
arbitrary
query
and
it
returns
in
in
you
know
just
a
snap
and
that's
awesome,
and
it's
awesome,
especially
when
you
do
business
intelligence.
A
If
you
do
machine
learning
on
this
on
this
and
and
you
you
want
to
be
able
to
run
these
arbitrary
queries,
and
you
want
to
get
some
insight
from
that-
and
that's
I
guess
a
really
important
thing
these
days.
But
you
know
in
kubernetes
we
are
a
large
community
of
developers
and,
quite
frankly,
most
of
the
logs
I
produce
I
have
a
hard
time.
Imagining
that
somebody
can
really
get
business
intelligence
out
of
it.
I
really
want
my
logs
to
do
debugging,
and
if
you
want
your
locks
or
debugging,
do
you
really
to
tokenize
everything?
A
Or
could
you
know
a
good
old
graph,
actually
be
a
solution
and
I?
Think
low
key
basically
takes
a
interesting
approach
in
that,
and
it
basically
says
every
pod
that
you
run
or
even
within
a
pod.
You
know
you
have
a
few
streams
of
data
and
you
basically
just
like
Prometheus.
You
take
all
the
metadata
associated
with
a
pod
like
the
pod
name
in
any
key
value
pairs
that
you
put
in
kubernetes
and
that's
your
metadata.
A
Your
time
range
is
your
metadata
and
that's
what
you
can
do
a
search
on,
but
everything
else
doing
doing
the
actual
text
search
you
just
use
grab
and
the
promises
that
this
is.
You
know
10
X,
52,
X
hundred
X,
cheaper
than
the
competing
solutions
that
actually
do
the
full
indexing,
and
that
sounds
appealing
you
know
at
the
end
of
the
day,
we
all
care
about
costs.
So
so
that's,
let's
get
into
it.
A
A
For
today,
I
created
an
AWS
account:
I
actually
did
not
have
one
and
I
created
one
too
to
be
able
to
run
this
demo,
because
I
wanted
to
use
Amazon,
s3
and
I
wanted
to
try
it
out
and
I
I
just
needed
a
little
bit
of
an
ec2
instance,
and
so
I
created
an
account
and
I
will
actually
start
with
setting
things
up.
So
let
me
launch
an
instance
for
what
we're
going
to
do
today
and
I
think
I
wanted
to
have
no
Bunty
I
wanted
to
have
the
18:04
LTS.
That
sounds
good
and
I.
A
Think
I
decided
for
the
demo
to
do
a
t2
large
to
V
CPUs
and
an
8
gigs
of
ram,
we're
actually
going
to
later
evaluate
a
little
bit,
whether
that's
a
good
choice
and
and
what
my
choices
are.
But
let's,
let's
start
with
that,
and
let's
launch
that
so
yes,
I
know
about
SSH
all
right,
so
this
is
happening
and
well
this
is
happening.
A
Let
me
actually
start
how
any
developer
starts,
which
is
you,
google
things
grafanello
key,
so
we
get
to
the
github
page,
so
I
will
actually
set
up
a
little
OneNote
kubernetes
on
on
that
one
ec2
instance,
and
when
I
prepared
for
this
I
I
just
got
here-
and
you
know,
I
have
to
admit
in
good,
develop
profession,
I
didn't
even
read
this
full
wiki
page
I
just
searched
for
the
blue,
getting
started
Docs.
So
let's
get
started
and
I'm
not
using
local
docker
I'm
going
to
use
kubernetes
and
I
mean
we're
going
to
use
helm.
A
C
A
It's
initializing
so
I
guess
this
is
the
awkward
moment
where
I
need
to
wait
for
things
and
just
talk
while
we're
waiting.
One
thing
as
a
vmware
guy
that
has
been
interesting
using
AWS
is
hey
Christopher
hi
to
Germany.
One
thing
that
has
been
interesting
to
me
is
that
when
you,
when
you
use
the
VMware
software
one
of
the
nice
things
you
get,
is
you
get
the
interactive
screen
of
a
VM
and
that's
super
cool?
A
And
that's
that's-
that's
really
useful,
but
what
you
get
here
is
actually
sort
of
you
click
on
connect
which
I
thought
was
going
to
get
me
the
screen,
but
it's
actually
about
you
know
connecting
SSH
and
as
a
developer.
I
really
appreciate
this,
and
this
was
new
to
me,
and
this
actually
makes
a
lot
of
sense
because
I
actually
don't
want
to
use
the
screen
to
set
things
up.
I
want
to
connect
your
SSH,
so
I
like
this.
A
A
A
A
Interesting
on
Azure,
you
can
get
a
serial
console.
That's
that
sounds
useful
Hydra
Virginia
at
least
we
have
one
person
from
the
US
spending
their
their
afternoon
with
us
and
not
just
Europeans
and
Indians,
who
should
be
in
bed
or
sheared.
At
least
people
out
on
a
Friday
night
doing
something
more
fun
involving
beers.
Oh
I
see
William,
you
wouldn't
talk
about
me
cool.
So
let's
take
a
look
here.
So,
let's
see
click
on
connect,
let's
get
our
instance
and
it's
connected.
A
A
Basically,
I
decided
for
today,
given
that
you
know
I
I'm,
a
newbie
and
two
other
people.
You
know
kubernetes
is
a
growth
area,
were
you
know,
there's
lots
more
people
ramping
up
every
day
and
so
I
decided
for
today,
I'm
actually
going
to
start
add
start
from
scratch:
I'm,
starting
with
a
fresh,
fresh
VM
fresh
instance,
and
I'm
actually
going
to
install
everything,
including
kubernetes
from
scratch,
and
so
I'm
setting
up
Python,
I'm,
studying
restore
services.
Sure.
B
A
C
B
A
Alright,
so
apparently
you
shouldn't
be
enabling
DNS
too
fast
after
enabling
kuvera's
itself.
So
so
what
did
I
do?
I
I
just
used:
sudo
staff,
install
micro,
kate's
and
then
I
used
micro,
Kate's,
enable
and
enabled
DNS,
which
was
weird
to
me.
I
felt
like
DNS
is
such
a
basic
service
that
should
always
be
running,
and
so
it
tripped
me
up
in
to
prepare
that
it
wasn't
just
enabled
but
figure
that
out
all
right.
So
now
we
have
a
kubernetes,
which
is
great.
So
actually
let
me
set
a
cute
cuddle
by
the
way.
A
Because
I
can
say,
cube
CTO,
but
I
don't
like
to
write
code
or
C.
You
see
me
fair
fingering
everything
get
nodes
and
we
have
one
node
and
that's
all
we're
going
to
need
for
today.
So
so,
basically,
I
think
this
should
give
me
a
sufficient
setup
right.
Oh
no,
it
doesn't.
Actually
the
next
thing
I
want
to
do
is
I
want
to
I,
also
need
help.
C
A
A
A
B
C
C
A
B
C
A
C
A
A
And
this
is
another
crazy
thing
to
me
that
you
know
I
created
the
VM,
that's
on
the
public,
Internet
and
so
I
will
just
create
a
firewall
rule.
Put
my
IP
address
in,
and
this
is
all
safe,
apparently
I'm
too
much
of
an
enterprise
guy
by
now
to
feel
like,
okay,
just
putting
the
PM's
on
the
public,
Internet
and
I'm,
pretty
sure
that
this
at
least
two
or
three
people
on
this
call
right
now
or
on
this
on
this
feed
right
now
trying
to
hack
me
so
go
at
it.
C
C
C
A
Let
me
know
if
the
font
size
is
for
Griffin
and
everything
also
work
out.
I
think
this
looks
ok,
ish,
but
let
me
know
if
it
doesn't
so
we
go
to
OHANA.
We
say
we
want
to
add
mopey
and
you
know
now
we're
actually
within
the
kubernetes
networking
and
thank
god
I
enabled
dns
earlier.
So
this
should
work
and
it
does
alright
and
and
gharana
has
actually
been
recently
adding
this
explore
mode.
So
you
know
what
I:
what
we
need
to
note
is
we're
basically
from
what
I
understand
running
top
of
free
versions.
A
Right
now
and
I've
been
doing
this
over
the
last
several
weeks,
using
the
explore
feature
and
depending
on
the
day,
sometimes
it
works
better
than
other
days.
It's
generally
moving
into
the
right
direction,
but
occasionally
there's
a
few
regressions.
So,
let's
see
what
we
actually
get
today
so
go
to
this
Explorer
mode,
and
the
cool
thing
is
that
you
know,
while
her
father
was
traditionally
more
of
a
dashboarding
tool
where
you
create
your
dashboards
and
you
and
that's
how
you
looked
at
your
environment,
Explorer
is
really
more
interactive.
A
It's
supposed
to
be
for
your
interactive
troubleshooting.
Both
for
Prometheus
as
well
as
now
for
Loki
and
they
actually
both
in
this
Explorer
mode,
and
so
let's
take
a
look.
The
labels
have
already
been
been
populated
and
what
we
can
see
is
actually
you
know
what
is
it
earlier?
What
helm
was
deploying
ways
was
Loki
prom,
tail
and
Goffin
and
we're
going
to
talk
about
what
these
are
in
a
moment.
But
you
know
the
graph
Anna
we
just
deployed
and
that's
what
we're
using
right
now.
A
A
B
A
So
let's
try
this
again
it's
this
in
my
browser
command
line,
so
we're
supposed
to
be
able
to
do
this
and
it's
supposed
to
filter
by
that.
It
doesn't,
though,
and
I've
definitely
seen
a
demo
where
this
worked.
I've
definitely
tried
this
out
a
few
weeks
ago
where
this
worked,
but
I
would
think
that
only
this
line
should
be
showing
up
and
it
should
be
correctly
doing
a
grep
on
that.
So
it
didn't.
A
B
C
A
A
A
B
A
Cool
alright,
so
yeah
Paul.
Thank
you.
This
yeah!
This
is
working
now
cool,
so
so
now,
let's
actually
use
this.
So
what
I
was
thinking
that
we
could
do
was
going
to
be
that
we
use
cube,
cuddle,
applying
and
we're
starting
a
new
pod
and
so
I
prepared
a
new
pod
and
I
just
want
to
use
this
very
simple.
This
very
simple,
pod
I
hope
these.
These
font
sizes
also
work
out
and
and
this
and
the
colors,
but
essentially
this
part
is
called
example.
A
A
A
A
So
with
that
what
did
I
want
to
do
next?
So
so,
basically,
we
we've
used
it
and
this
seems
to
be
working.
So
what
does
this
actually
do
underneath?
Let's
actually
check
this
out
so
so
we
already
looked
at
the
fact
that
there
were
config
maps
and
pods,
and
so
let's
take
a
closer
look,
so
there
is
a
there's,
a
low-key
pod
there's
a
graph
on
a
pod.
These
are
the
two
ones
that
we
expect.
So
the
low-key
pod
is
the
one
that's
actually
implemented
in
the
backend.
A
A
A
And
so
what
this
does
is
basically
on
every
node
it
will.
It
will
run
prom
tale
and
prom
TLD.
The
idea
of
the
name
is
basically
that
it's
it's
it's
collecting
things
the
same
way
that
Prometheus
collects
metrics.
It
does
that
for
logging.
So
it's
it's,
the
Prometheus
style
tale
and
basically
tailing
means
it's
basically
monitoring
all
the
all
the
containers
running
and
so
for
that
it's
it's
mounting
in
a
bunch
of
things
from
the
host
path.
So
it's
mounting
in
the
run
directories
the
container
directories
under
under
a
volunteer
viola.
A
So
basically
it
gets
access
to
the
host
runs
in
a
more
privileged
mode
and
hence
it
it
has
access
to
to
what's
happening,
and
so
it
can
actually
monitor
for
all
the
parts
coming
up
like
we
did
for
our
example.
Pod.
It
can
see
them,
it
can
start
streaming.
Then
it
has
to
access
to
the
metadata
and
it
can
do
all
these
cool
things.
So
that's
awesome.
So
so
that's
prom
tale.
A
To
the
API
the
push
API
for
Loki,
so
this
is
actually
something
I
checked
out,
because
this
is
there's
a
REST
API
that
you
can
also
call
directly.
So
you
don't
need
to
use
prom
tail,
but
prompt
hill
in
the
Promethea
in
the
menendez
context
is
particularly
useful,
but
it
can
actually
connect
to
it
from
anything.
And
you
can
you
can
stream
logs
into
Loki
and
that's
actually
going
to
be
something
that
we
for
next.
So
basically
what
I
now
want
to
do.
So
we
have
Loki,
it
works.
A
People
say
it's
awesome
cool,
but
I
wanted
to
actually
understand
from
a
storage,
respect
forgiving
that
I'm,
a
storage
person
and
I'm
an
infrastructure
person
I
wanted
to
sort
of
analyze
this
from
an
architecture
perspective
from
a
design
perspective
and
understand
the
cost
better
right.
Now
the
the
github
pages
don't
get
half
a
cost.
Modeling
and
I
asked
on
on
Twitter
whether
that's
something
that
exists,
but
it
doesn't
doesn't
really
exist
yet.
A
A
A
The
low
key
container
also
exposes
a
slash
metrics
REST
API
for
Prometheus
and
I'm,
going
to
just
start
a
Prometheus
configuration
with
15-second
scrape
interval,
doing
HTTP
to
the
low
key
at
the
same
location,
where
we're
getting
that
we're
actually
connecting
from
graph
on
to
and
I'm
going
to
do
a
deployment
that
runs
Prometheus
container
exposed
on
port
90
90,
using
this
configuration
and
I'm
creating
a
service
for
that.
So,
let's.
A
B
A
See
hi
Bucharest
cool,
so
so
I
guess
the
the
the
usage
document
is
just
a
little
bit
out
of
date,
and
this
has
actually
been
updated.
Interesting,
ok,
that
that
makes
sense.
That's
also
why
I
I
was
pretty
sure.
I
had
seen
a
demo
that
does
do
this,
so
I
will
add
prometheus
1990
and,
let's
see
see
fantast
this
worked,
and
so
that
should
mean
that
under
Explorer
I.
A
A
That
the
fauna
is
a
bit
experimental,
I
think
the
entire
explore
mode
is
experimental.
You
you
only
get
it
on
the
master
branch.
You
don't
get
it
in
a
released
version
and
low
key
itself
seems
to
be
quite
experimental.
So
you
know
everything
we
we
look
at
here
seems
pretty
fresh
off
off
the
open
source
boat.
A
With
that,
let's
look
at
well
I'm
still
not
set
up
alright,
so
I'm
I'm,
an
idiot,
so
another
thing
I
need
to
do
is
I.
Have
this
s3
configuration
here
were
this
is
the
Loki
configuration
and
I
prepared?
This
were
basically
I'm
telling
I
still
want
the
built
in
ball.
Tb
and
bolt
is
similar
to
sequel
light.
I
did
some
research
on
this.
A
A
And
that
holds
the
configuration
being
a
secret
which
is
good,
but
it
we
cannot
really
see
it
nicely
here.
So
what
I'm
going
to
do
is
pretty
crude
I'm
going
to
delete
the
secret,
and
now
you
will
see
that
I'm
a
developer
like
everybody
else,
which
means
I,
don't
actually
know
how
to
create
a
secret
hi
Google.
A
A
A
Let's
see
so,
basically,
what
we're
doing
here
is
we
are
forming
a
JSON
payload
and
the
JSON
payload
is
basically
updates
one
stream
at
a
time,
and
we
put
our
labels
that
we
warned
and
then
we
create
a
number
of
entries
and
each
entry
is
basically
a
timestamp
and
and
some
line
that
we're
putting
in
and
and
so
the
input
here
is
how
many
of
these
log
lines
we
want
to
push
in
one
go
and
and
I'm
just
going
to
return.
You
know
the
total
number
of
bytes
that
I
actually
put
in.
A
A
So
that's
50,
so
each
one
will
do
roughly
25
and
they
will
each
call
this
and
job
locks
functions
so
and
at
the
end
it
prints
how
many
megabytes
it
actually
produced
and
logs
so
simple
script,
but
it
should
be
useful.
So
by
now
this
should
also
be
ready.
Yep.
This
is
ready
cool,
so
it's
called
push
up.
I.
A
A
C
C
A
A
This
alright
cool,
so
what
we're
seeing
here
and
we
can
do
an
auto
refresh
every
five
seconds.
So
we
see
now
that
number
of
chunks
that
were
ingested
is
going
off
and
that's
great,
that's
what
we
wanted
and
we
can
also
see
that
if
I
look
at
my
s3
bucket
I
should
start
seeing
some
data
popping
up.
Let's
see,
if
that's
true.
B
A
So
on
June
28th,
that's
right
now
we're
creating
and
given
that
these
long
lines
are
pretty
similar,
we're
producing
roughly
260
kilobytes
for
a
given
chunk.
They
are
compressed.
I
actually
tried
downloading
these,
but
they
use
a
binary
format
that
wasn't
very
easy
for
me
to
decode.
I
actually
got
a
pointer
to
the
binary
format
form
for
me
to
poke,
but
I
was
hoping
that
I
could
just
get
them
and
take
a
look
and
just
see
the
raw
log
lines.
A
A
C
C
C
A
What
was
the
first
time
Stan
there
were
a
bunch
of
error,
so
you
know
I,
don't
want
to
debug
this
right
now
and
I
guess
this
is
one
thing:
I
don't
yet
fully
understand
how
the
how
the
buffering
and
any
potential
back
pressure
works,
whether
I
overwhelmed
Loki
and
and
how
it
dealt
with
that
I'm
I'm
actually
an
entire
issue
about
that.
But
essentially
it
took
roughly
two
minutes
to
to
push
these
250
megabytes.
B
A
We
have
250
megabytes
divided
by
roughly
2
minutes,
which
I
should
have
been
able
to
do
in
my
brain,
which
is
roughly
2
megabytes.
A
second
I
think
when
I
didn't
get
some
of
these
errors,
I
was
actually
able
to
push
more
than
that,
but
you
know,
let's
assume,
for
a
second,
it's
2
megabytes,
a
second
I
assume
it's
actually
going
to
be
more.
But
let's,
let's
go
with
that
number.
So.
A
Right
so
it's
basically
the
same
thing
in
terms
of
numbers,
and
we
see
that
we
basically
on
the
on
the
s3
put
objects.
We
had
how
many
50,
so
we
pushed
50
objects
which
makes
sense
because
I
created
sort
of
50,
50
jobs
and
they
they
had
didn't,
have
enough
log
lines
to
actually
create
multiple
chunks,
and
so
they
they
all
ended
up
being
a
chunk
that
was
pushed
oh.
B
A
C
C
C
C
A
C
A
Right
so
so,
what's
not
showing
up
here,
I'm,
not
entirely
sure
how
from
getting
the
timestamps
right,
but
essentially
this
should
also
be
showing
that
that
I
will
eventually
need
to
do
get
requests
instead
of
just
put
requests,
and
they
don't
seem
to
be
showing
up
yet
so
I
will
I
will
not
dive
into
that
too
closely,
but
when
I
was
preparing
for
that,
that
definitely
worked
and
some
of
the
get
requests
were
being
served.
So
some
of
the
logging
words
was
served
by
these
by
these
older
things.
So
so,
maybe
if.
A
Up,
so
that's,
okay:
now
what
I
wanted
to
do
was
I
want
to
go
back
to
the
architecture
perspective
where,
if
I,
if
we
look
at
what
what
we've
done
is
we
let
me
see
so
so?
Basically
we
had
we
had
this
picture
here
and
in
this
picture
we
had
the
in
gesture
filling
up
memory,
and-
and
now
we
should
be
thinking
about
this,
what
do
we
want?
A
What
was
it
point,
eight
five
course
what
point
eight
five
of
a
core
and
we
were
running
at
two
megabytes-
a
second
so
for
every
two
megabytes,
a
second
that
we
want.
We
need
to
spend
almost
a
core,
and
so
this
this
equation
basically
tells
us
what
our
CPU
efficiency
is.
And
so,
if
we
want
to
ingest,
let's
say
a
terabyte
a
day,
we
can
break
it
down
to
how
many
megabytes
per
second
and
we
can.
We
can
calculate
our
CPU
consumption
and
so
what
Loki
then
can
do.
Is
it
can?
A
Actually,
if
that
needs
more
instances
of
the
in
gesture,
well,
it
can
scale
out,
so
it
can
add
more
in
gestures,
and
so
that
can
be.
You
know
actually
many
in
gestures.
Let's
make
them
so
that
I
can
draw
many
of
them,
and
so
you
actually
get
many
and
jesters,
and
let
me
just
check
whether
I'm
still
streaming
correctly
and
whether
everything
is
happy
seems
like
people
are
happy
all
right,
and
so
basically
we
we
we
can
scale
the
in
gestures.
A
The
other
reason
that
we
actually
want
more
in
gestures
is
because
we
we
might
want
to.
We
might
want
to
protect
against
loss.
So
if
one
a
gesture
dies,
it
takes
all
the
logs
that
were
still
buffered
in
it
that
weren't
written
out
as
three.
Yet
they
are
actually
going
to
be
lost,
and
so
one
tactic
here
is
that
we
can
actually
make
the
distributor
do
a
replica.
A
So
the
pod
would
also
stream
its
data
into
into
another
and
gesture
and
that
other
in
gesture
would
also
set
it
to
s3,
and
so
we
actually
have
multiple
in
gestures,
getting
the
same
memory
and
you
know
for
an
enterprise
storage
person,
I
cringe
when
I
see
this,
because
you
know
total
power
failures
in
my
experience
do
happen
and
you
need
to
be
protected
against.
Actually,
all
your
memory
being
lost,
and
but
I
guess
for
logging
and
from
a
cost
optimization
perspective.
This
is
actually
a
perfectly
fine
trade-off
to
do,
and
we
can.
A
We
can
put
it
into
three
pieces
of
memory
and
we
can
do
that
and
what
I
heard
was
loki
is
also
working
on
a
mode
was
actually
doing
this
persistently
on
disk,
but
obviously
writing
every
piece
of
log
to
two
disk,
instead
of
just
a
memory,
is
going
to
increase
our
storage
cost.
So
this
is
an
interesting
trade-off
here
and
probably
a
perfectly
appropriate
one
for
the
use
case.
So
so
this
is,
this
is
redundancy
and
this
is
sort
of
the
sizing.
So
so
with
that,
let's
actually
look
at
this.
A
If
we
say
that
we
have
one
terabyte
per
day,
which
means
1,024
megabytes
per
day,
divided
by
24
to
get
hours
and
divided
by
1000
3,600
we're
actually
not
getting
that
much
and
instead
right
well,
okay,
I'm,
dumb,
a
terabyte
is
more
than
1024
megabytes.
It's
actually
1,024
times,
20,000
24.
So
it's
12
megabytes
a
second.
So
if
we
assume
the
numbers
that
we
saw
and
again
I
I
think
in
a
prior
experiment,
I
was
actually
getting
a
little
bit
more
than
that.
A
A
A
So
so,
let's
go
with
these
two
numbers
and
we're
producing
one
terabyte
a
day,
which
means
that
we
are
producing
30
terabytes
per
a
month
and
let's
say
that
we
want
to
actually
keep
logs
around
for
one
month,
so
so
with
that
lets
actually
scope
out
how
much
this
would
cost
us
as
a
very
rough
approximation.
I
know
this
is
not
the
full
scientific
thing,
but
this
is
probably
close
enough.
D
A
A
So
we
said
that
we
need
six
course
times
three.
So,
let's,
let's
go
with
eight
course
and
32
gigs
of
ram
is
somewhere
in
the
middle
of
what
we
said
we
would
need.
So
let's
just
go
with
that.
Let's,
let's
assume
this
is
the
instance.
We
need
and
leave
it
at
that,
and
so
that
would
you
know,
equate
to
730.
A
Month
for
our
ec2
instances
and
let's
look
at
s3
well,
we
wanted
to
store
terrabytes
okay,
but
aside
from
storing
30
terabytes,
we
need
to
also
think
about
how
many
requests
are
we
going
to
do
so.
Let's
do
some
math
on
that
as
well,
so
let's
actually
scroll
a
little
bit
here
to
get
a
little
bit
more
real
estate.
A
A
I'm
ending
up
with
90
terabytes
that
that
I
want
to
store
unless
I'm
doing
some
deduplication
and
I
think
that
that
is
something
that
the
Loki
folks
are
also
interested
in
exploring
further.
But
for
the
moment,
let's
assume
we're
not
doing
that.
So
we
have.
We
have
that,
and
so
let's
assume
we
have
90
terabytes
and
we
we
divide
that
by
250
kilo
bytes.
How
many
objects
is
that?
Well,
let's
take
a
look
and
yes
I
know
that
I'm
butchering
units
so
98
times
1,024.
A
A
So
so
these
are
our
key
numbers
for,
and
this
is
for
the
ingest
and
if,
if
we
do-
and
now
it
all
depends
on,
how
often
do
you
query
right
like
how
often
do
you
have
incidents
where
you
actually
need
to
need
to
query
and
how
many
will
that
cover
and
I
don't
have
a
good
model
for
that
right
now,
but
you
know
you
will
occasionally
need
to
need
to
unzip
in
the
courier.
You
need
to
run
graph
and,
and
so
I
don't
know.
A
Doing
a
lot
of
get
requests
was
not
going
to
cost
us
that
much
there's
going
to
be
CPU
cost.
So
you
know,
let's
assume
from
a
CPU
perspective,
and
we
don't
need
as
much
memory
for
for
the
carriers
I
guess.
But
let's
assume
that
we
have
another
cost
like
this.
So
so,
let's
assume
that
you
know
maybe
a
very
rough
estimate-
and
you
know
I
hope
this
is
something
that
somebody
does
a
little
bit
more
scientifically
with
some
real-world
experience.
A
But
let's
assume
that
we
have
roughly
6000
that
we
calculated
up
there
where
we're
going
to
to
budget
another
three
for
hon.
Let's
say
we
don't
need
all
that
redundancy.
So
so,
let's,
let's
budget
another
200
for
for
the
query
and
for
the
actual
queries,
that's
also
budget
a
hundred.
So
you
know
in
total,
we
end
up
with
something
like
6500
and
and
if
you,
if
you
look
upwards,
Splunk
or
or
log
Z
or
others,
they
definitely
a
range
in
in
in
the
many
ten
thousands
of
dollars
for
the
same
log
volume.
A
So
so
definitely
this
is
interesting,
and
this
is
this
is
something
that
comes
out
at
a
much
more
reasonable
price
or
in
much
lower
priced
and
and
what
you
would
get
from
log
Z,
for
example,
or
from
Splunk
or
or
if
whether
you
run
that
on
pram
or
or
in
the
cloud
so
so
I'm,
hoping
that
the
Loki
project
sort
of
comes
up
with
with
with
some
real
life
number
is
just
will
as
sort
of
a
calculator
for
this.
But
this
all
looks
quite
reasonable
to
me.
A
A
A
Hope
that
those
calculations
for
for
for
the
costs
are
interesting
and
makes
sense.
I
was
definitely
interested
in
in
what
this
comes
out
to
be,
given
that
at
VMware
we
produce
a
lot
of
logs,
and
you
know,
sort
of
storage
costs,
our
bread
and
butter,
but
I
really
didn't
have
any
any
good
notion
for
for
what
this
would
cost.
That's,
let's
actually
do
this.
A
Daily
capacity,
so
I
wanted
Enterprise
contact
sales,
cool,
choose
your
daily
volume.
Let's
say:
I
do
a
hundred
gigabyte
of
daily
volume,
so
a
hundred
gigabyte
of
daily
volume
would
be
would
be
five
thousand
five
hundred.
So
if
I,
if
I
do
this
times,
10
that
would
be
fifty
thousand,
so
so
I
guess
the,
but
the
rough
math
were.
You
know
the
low-key
presentations
claimed
to
be
roughly
a
factor.
10
seems
to
seems
to
pan
out
roughly
from
here.
Obviously
I
assume
that
you
know
the
Enterprise
version
has
some
additional
features.
A
I
assume
that
you,
you
also
in
the
Enterprise
version,
get
get
some
volume
discount,
but
roughly
this
actually
seems
to
pan
out
cool.
So
so
this
is
what
I
have
today
thanks
everyone
for
joining
and
letting
me
ramble
for
an
hour
and
a
half
and
and
sorry
for
that
little
bit
of
a
hiccup
were
my
macbook
just
decided
that
I
was
done
and
thanks
everybody
for
staying
up
late,
just
a
word
of
advice.
A
If
you're
in
Europe
and
it's
Friday
night
get
a
beer,
what's
the
recording
I
won't
be
offended
or
I
assume
Joe
won't
be
offended,
and
if
you're
in
India
and
it's
1:00
a.m.
you
should
sleep,
sleep
is
important
with
that.
Thanks
everybody
for
joining
this
was
fun
and
I
hope
to
do
this
again.
I
really
enjoyed
this.
Thank
you.
I
know.
I
need
to
figure
out
how
to
stop
this.