►
From YouTube: Got a Need for Speed? Accelerate Your Prometheus Dashboard Using... - Shilla Saebi & James Ranson
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Got a Need for Speed? Accelerate Your Prometheus Dashboard Using Trickster - Shilla Saebi & James Ranson, Comcast
We live in a world where high performance and speed are essential. A few extra seconds of response time on a dashboard can be a deal breaker. Many dashboards request the entire time range of data from the time series database, every time a dashboard loads or reloads. This can result in slower rendering times, and different results depending on when the request is made. We are proud to announce that Trickster, a new open source project, was developed to address this very issue. Written in Go, Trickster is a reverse proxy cache for the Prometheus HTTP APIv1 that considerably accelerates dashboard rendering times for any series queried from Prometheus. This is possible because of the delta proxy, step boundary normalization, and fast forward features. In the presentation, we will discuss how Trickster was developed at Comcast, and we will show you a live demo of the software.
To learn more: https://sched.co/GrZ2
A
My
name
is
James
Ranson
I
am
an
architect
at
Comcast.
I.
Do
software
architecture
for
our
video
products
and
I
have
created
a
software
application
called
trickster?
That
is
an
accelerator
for
Prometheus
based
dashboards.
So
a
little
bit
more
about
me.
If
this
is
gonna
work,
the
one
thing
we
didn't
test
so
a
little
bit
more
about
me:
I
enjoy
mountaineering
I
like
to
watch
football,
I
played
piano
for
35
years
and
all
kinds
of
good
nerdy
stuff,
like
that,
my
favorite
video
game
is
Zelda
and
with
that
I'll
introduce
Sheila
who's.
B
Hi
I'm,
Sheila,
savvy
and
I
am
an
open
source
program
manager
in
the
open
source
program
office
at
Comcast.
I've
been
in
this
role
for
about
two
years.
Prior
to
this
role,
I
was
a
cloud
of
operations
engineer
working
on
the
OpenStack
team
and
yeah
I,
like
dogs,
I
also
play
piano,
and
this
is
my
first
koukin
cloud
native
conference,
mine.
B
All
right
so
a
little
bit
about
my
team
and
how
it
even
ties
in
to
trickster
and
how
it
ties
ties
in
to
James
and
other
engineers
at
Comcast,
so
the
open
source
program
office
opened
doors
in
2017
and
we
support
over
7,000
developers
internally.
We
also
have
over
one
hundred
and
forty
one
project
repos
on
github.com
and
we're
also
a
four-person
team,
so
I
focus
on
program
management
with
community,
and
we
also
have
an
open
source
program
manager
who
focuses
on
compliance.
B
We
have
a
software
engineer
who
does
compliance
toolings,
who
just
recently
joined
us
and
as
well
as
our
leader
and
many
of
you
may
be
asking
what
contest
is
even
doing
with
open
source
when
we
started
how
long
we've
been
in
the
game.
So
this
slide
is
going
to
detail
that
I
won't
get
into
every
single
point
on
the
slide,
but
I'll
go
through
a
few
highlights.
B
In
2006
we
were
very
large
consumers
of
open
source.
This
predates
me,
I've,
been
with
the
company
for
eight
years,
and
so
I
actually
went
around
and
asked
different
leaders
and
different
folks
from
different
organizations
and
departments.
What
they
could
tell
me
about
the
open
source
history,
2013
is
one
that
sticks
out.
We
are
open
source
Advisory
Council
was
formed.
This
is
a
group
that
consists
of
our
legal
team
software
engineers,
Network
architects,
security,
architects,
as
well
as
the
open
source
program
office.
B
This
is
the
team
that
James
actually
had
to
go
through
to
get
his
project
approved
and
other
engineers
have
to
go
through.
This
team
has
evolved
since
then,
and
we
trying
to
make
the
process
easier
for
our
engineers
to
open-source
stuff.
2015
was
a
big
one.
We
we
have
a
project
called
traffic
control,
which
is
now
a
top-level
Apache
project.
Recently
at
2018
it
became
one
traffic
control
was
open
sourced
at
that
time,
and
in
2016
we
that
was
when
we
donated
it
to
the
Apache
Software,
Foundation
and
2018,
which
will
fast
forward
to
today.
B
About
a
month
ago,
we
had
an
open
source
summit
in
Philadelphia,
which
was
open
to
folks
outside
of
Comcast.
It
wasn't
just
an
internal
thing.
We
plan
on
having
another
one
in
2019.
Also
that's
when
Trixter
was
born
and
then
we
also
have
fellowship
pilot
where
we
we
encourage
our
engineers
to
work
on
open
source
projects
for
20%
of
their
time
and,
as
you
can
see,
we're
really
doubling
down
on
open
source.
B
B
We
encourage
you
to
also,
if
you
want
to
open-source
something
to
do
the
same
and
I'll
dive
deeper
and
took
some
of
the
community
stuff
later
once
once
James
talks
more
about
trickster,
but
the
basic
concept
is
that
your
source
code
is
open,
but
also
you
want
to
cultivate
a
good
community
by
having
nice
welcoming
documentation
and
just
really
prioritizing
it
a
few
facts
and
I'm
sorry
metrics
about
trickster.
We
have
two
hundred
and
twenty
two
hundred
thirty
three
commits
we
actually
just
pulled
up
that
number.
Recently
it
keeps
going
up.
We
have
fifteen
releases.
B
Two-Thirds
of
our
contribution
tribute
errs
are
outside
of
Comcast's.
We
really
value
that
we
want
to
bring
diversity
globally
from
different
companies
as
well
different
thoughts.
So
we're
super
excited
that
they're
not
just
Comcast
contributors,
and
then
you
know
you
can
see
here.
The
number
of
stars
and
forks
over
five
hundred
stars
and
with
that
I
will
pass
over
the
mic
to
James
he's
gonna
dive
deeper
into
trickster.
For
you,
oh.
A
My
god
Thank
You
Sela,
so
I'll
back
up
one
one
slide
just
talking
about
this,
a
little
more
in
the
very
first
keynote
on
Tuesday
morning.
Somebody
said
we
love
stars,
I
love
stars,
so
please
start
my
project
if
you
like
it,
I'd
really
appreciate
having
more
of
those.
But
let's
talk
about
what
trickster
is
so
let
me
get
a
show
of
hands
who
actually
has
used
trickster,
whereas
using
trickster
in
their
in
their
systems.
Awesome
we
have
any
trickster
contributors
in
the
house
awesome
who
has
no
idea
what
trickster
is
before
today.
Awesome!
A
That's
what
I
want
to
see.
You're
gonna
love
talking
about
it.
So,
as
Sheila
mentioned,
I
created
trickster
last
year
at
Comcast
and
I'll
talk
about
the
journey
as
to
why
I
felt
like
I
needed
to
make
it.
But
what
it
is
in
essence,
is
basically
a
reverse
proxy
cache
and
actually
call
it
an
accelerator,
because
it
does
more
than
just
caching
that
sits
between
your
dashboard
application
and,
as
you
can
see
on
this
diagram,
I
have
a
graph
on
an
icon,
but
it'll
actually
work
for
any
dashboard
that
has
a
back
end
to
Prometheus.
A
So
here's
a
common
problem,
if
you
ever
worked
in
any
kind
of
industry
where
you
work
with
users
at
large
in
America,
you're,
probably
familiar
with
the
shape
of
this
chart
and
I'll
just
show
you
real,
quick,
it's
up
there,
so
I
can't
see
it
nevermind.
So
you're
probably
me
with
the
shape
of
this
chart.
A
A
Charge
of
some
systems
at
Comcast,
and
so
I
was
building
this
dashboard
to
show
charts
like
this
to
show
failure
rates
and
those
kind
of
things
across
some
of
our
platforms
and
realized
that
I
didn't
like
the
speed
of
our
dashboard
performance
so
and
I'm
gonna,
explain
kind
of
like
what
I
discovered
under
the
hood
as
I
started.
Looking
at
things.
So
I
started
out
with
Griffin
ax
and
I
noticed
that
every
time
that
you
actually
refresh
your
dashboard
page,
the
entire
data
set
is
reloaded
all
the
way
from
end
to
end.
A
When
really
all
you
need
to
collect
for
a
refresh
is
that
last
little
sliver,
that's
in
purple
on
this
screen
now,
because
that's
the
only
thing
on
the
changes
that
paint
that
thing
on
the
page
that
changes
between
refreshes
of
your
dashboard
charts.
So
what
I
set
out
to
do
is
find
a
way
to
make
our
dashboard
applications
only
request.
Those
little
slivers
instead
of
the
entire
data,
set
every
time
the
dashboard
refreshed.
A
So
that's
what
trickster
does
and
I'll
go
into
kind
of
how
that
works
out
in
a
minute,
but
I'll
go
through
a
little
more
things
here,
so
each
time
not
only
does
do
I
refresh
that
dashboard
am
I
collecting
the
whole
data
set,
but
so,
as
everybody
else,
who's
also
looking
at
that
dashboard,
and
so
what
I
found
is
on
a
typical
graph
on
a
dashboard.
Each
line
on
the
screen
typically
takes
about
300
data
points.
That's
gonna
be
240
to
320
depending
upon
how
Griffin
it
decides
to
do
the
step
values.
A
So
in
essence,
every
time
your
dashboard
refreshes
you're,
asking
for
300
times
more
data
than
you
actually
need
to
put
that
little
sliver
back
on
the
screen
to
refresh
that
dashboard.
So
we
do
that
in
a
few
ways
with
trickster.
The
first
thing
we
do
is
we
have
what's
called
the
delta
cache.
So,
as
you
can
see
on
this
screen
here
in
blue,
imagine
that's
data.
That's
already
in
the
trickster
cache
and
there's
orange
lines
that
are
outlined
on
the
left
and
highlight
and
solid
on
the
right.
A
Those
are
data
points
that
trickster
needs
to
complete
a
user
request
that
doesn't
have
those
data
points.
So
a
trickster
does.
When
you
make
a
request
to
your
from
your
dashboard
to
Prometheus.
Is
it
looks
at
your
request
and
the
time
stamps
at
your
requesting
data
between
and
it
determines
what
it
has
in
that
data
set
and
what
it
needs
and
so
it'll
adjust
the
upstream
queries
to
Prometheus
to
only
grab
the
data
points.
A
You
need
to
fill
the
cache
reconstitute
the
data
set
to
the
user
and
send
that
back
to
the
user,
and
what
that
means
is
those
queries
can
be
accelerated
from
about
maybe
a
second
half
a
second
to
about
forty
times
faster
than
that
and
we'll
show
that
in
the
demo,
in
just
a
minute,
your
dashboards
really
do
go
about
40
times
faster
with
trickster
in
the
middle.
The
second
piece
of
that
is
what
we
call
step
down
or
no
to
normalization.
A
The
reason
this
is
important
is
because,
if
you've
ever
used
a
graph
on
ax-
and
you
have
other
people
using
Gravano
next
to
you-
and
you
kind
of
sometimes
see
their
charts,
look
slightly
different
from
yours,
a
couple
little
artifacts
and
squirrely
things.
That's
just
the
way.
Gravano
works
in
that
it
looks
from
the
start
time
or
the
the
newest
time
that
works
backwards.
And
so,
if
you
are
13
seconds
behind
on
refreshing,
your
dashboard
from
somebody
else,
you're
gonna
see
slightly
different
results
on
your
screen.
A
It's
really
hard
to
cache
when
everybody's
data
points
are
slightly
different,
so
what
step
boundary
normalization
does?
Is
it
actually
aligns
all
of
the
data
points
to
very
well-known
step
boundaries
like
the
top
of
the
minute
or
the
top
of
four
minutes
or
six
minutes,
depending
upon
how
big
your
graph
is
and
those
kind
of
things,
and
so
as
step
boundary
normalization
happens
that
allows
everybody's
data
points
and
not
just
your
data
points
to
all
share
the
same
cache,
because
everything
is
normal
or
crap
normalized
across
the
same
boundaries.
A
So
this
the
third
feature
called
fast
forward,
will
actually
go
out
and
that
newest
data
point
at
all
times
and
insert
it
into
the
data
set
as
it
returns
that
back
to
the
customer,
so
that,
in
addition
to
just
having
all
those
nicely
normalized
lines
on
the
boundaries,
you
get
that
last
data
point.
That's
always
the
most
recent
data
point.
So
not
only
your
graphs
are
really
really
fast,
but
they're
still
as
up-to-date
as
possible.
A
couple
use
cases
for
trickster
the
first
one
is
what
I
call
cardinality
creep.
A
A
Are
you
have
no
control
over
when
Apple
releases,
iOS
4.1
12.1
was
one
point
one
two,
and
so,
as
those
label
values
start
to
creep
out
of
your
control,
that
could
make
your
dashboards
are
a
little
slower.
So
the
great
thing
about
trickster
is
since
we're
reducing
that
data
set
that
were
requesting
from
Prometheus
down
to
just
a
small
amount.
Those
queries
are
still
really
fast,
even
when
you
have
a
very
deep
amount
of
cardinality
in
your
from
a
theist
labels.
Secondly,
any
incidental
heavy
dashboard
usage.
A
A
lot
of
you
probably
are
aware
that
whenever
you
have
outages,
your
dashboards
become
the
most
popular
thing.
People
are
looking
at
because
they
want
to
find
out
what
the
hell
is
wrong,
and
so,
when
you
have
these
major
outages
like
a
step,
one,
for
example,
it's
our
name
or
collector
at
Comcast.
Typically,
you
then
get
your
VPS
your
directors
and
all
these
people
looking
and
loading
your
dashboards.
A
The
trickster
solution
will
also,
in
that
case,
accelerate
dashboards.
A
quick
example
of
that
many
of
you
are
probably
familiar
with
in
the
spring
of
this
year.
Github
was
acquired
by
Microsoft,
and
so
we
had
it
turned
out
quite
a
few
people
decided
to
defect
from
github
and
go
to
get
lab
and
get
lab
systems.
A
How
to
have
a
monitoring
system
for
all
their
users
to
watch
started
to
fall
over
because
of
the
heavy
usage
after
that
move,
and
so
they
put
trickster
in
the
middle
and
resolved
a
lot
of
their
issues
with
scaling
and
their
Prometheus
infrastructure.
So,
in
those
heavy
dashboard
usage
situations
also
a
great
fit
I
think
we
have
a
live
demo.
A
So
here
is
a
quick
dashboard,
we've
intentionally
obscured
what
this
data
is
since
its.
That
is
not
good.
Let's
see
if
this
will
help
there,
we
go
alright.
So
this
is
just
a
random
generic
dashboard
of
data
comcast.
We
have
scored
the
data,
so
you
can't
know
what
it
is.
Just
you
know
that
just
so
you
know
that
it's
really
slow.
So
if
I
press
this
reload
button,
what
you'll
see
there
is
I'm
gonna,
give
it
one
more
reload
up
here.
A
A
So
for
this
I
just
have
a
quick
run:
trickster
script.
If
I
were
to
cat
that,
for
you
you'd
see
it's
just
a
quick
docker
run,
that's
all
it
is
so
now
and
then
in
here,
so
I
don't
typos
and
that
thing
or
something
I
have
a
little
file.
That
gives
me
my
URL
to
copy/paste.
So
let
me
cat
that
real,
quick
and
then
I
type
with
the
word
cat.
A
A
Nice,
low
dashboard,
still
and
now
you'll
see
I
have
a
new
data
source
called
trickster
in
here
when
I
click
on
that
you're
gonna
see
it's
going
to
be
a
little
slow,
the
first
time,
because
if
it
is
a
cache,
so
I
just
have
to
actually
cache
the
data
one
time.
So,
let's
clear
that
and
that
was
the
wrong
button
there
clearer
there.
It
goes
and
now,
if
I
hit,
that
refresh
Wow
we've
gone
from
four
seconds
down
to
655
milliseconds
for
this
lowest
one.
A
If
I
were
to
click
refresh
on
that
again,
you'd
see
just
as
fast
700
that
time
and
that's
over
LTE
right
now,
we've
had
that
we
had
to
hotspot
into
the
phone
because
the
slow
Wi-Fi
here
in
the
building-
and
you
can
see
each
one
of
these
graphs
looks
identical.
So
how
were
to
flip
it
back
over
to
the
Prometheus
instance
slow
again
same
charts,
flip
back
to
trickster
back
to
being
fast
again.
A
That's
our
demo
for
that
you
can
try
that
same
demo.
We
have
the
instructions
right
on
the
readme
in
our
github.
Something
else
about
the
github
I
wanted
to
mention.
Is
we
just
this
morning
added
the
who's
using
trickster
on
to
our
readme
I'd,
encourage
everybody
to
if
you're
using
it
too.
Please
go
to
a
PR.
Add
your
name
to
the
list
and
we'll
get
you
on
the
list
there
and
then
our
big
readme
file
roadmap
here
going
forward.
A
A
It's
really
important
to
us
because
we
want
to
contribute
this
project
to
CN
CF
at
some
point.
Some
point
that
I
definitely
believe
that
supporting
all
of
these
upstream
types
is
important
before
you
make
that
contribution.
Secondly,
we
want
to
incorporate
distributed
tracing
support,
because
a
lot
of
people
have
asked
for
that.
There's
been
a
lot
of
talk
in
the
community
about
you
know
which
way
to
go.
A
It
looks
like
we
now
in
2018
have
started
to
get
solutions
that
are
fully
interoperable
between
all
the
different
options
for
tracing.
So
we
want
to
get
that
and
incorporated.
So
if
you
have
any
experience
but
distribute
rates,
we'd
encourage
you
to
contribute
that
as
well
and
then.
Lastly,
high
availability
is
something
we
want
to
look
at
for
things
like.
Let's
say
you
have
two
different
Prometheus
servers
and
a
high
availability
notes:
they're
both
scraping
the
same
data
endpoints.
Well,
what
happens
if
one
goes
down
for
a
little
bit,
but
then
it
comes
back.
A
B
Right
really
fabulous
on
James,
really
proud
of
the
work
that
you're
doing
so
we're
trying
to
continue
cultivating
this
community
building
it.
So
some
of
the
things
trickster
has
done
since
the
beginning.
I
would
like
to
share
with
you
all,
because
it's
been
really
helpful
and
it's
been
working.
So
it's
really
important
to
having
a
very
friendly
and
welcoming
onboarding
process
for
your
contributors.
B
Make
sure
that
your
documentation
is
up
to
point
James
put
together
a
contributor
guide,
there's
a
readme
license
files
and
just
having
the
right
attitude
for
your
contributors
is
really
really
important,
also
provide
clear
and
consistent
support,
sometimes
I
wonder
if
James,
even
sleeps
or
what's
going
on
with
him
I
see
requests
getting
merged,
pull
request
getting
merged
just
in
the
middle
of
the
night,
3
a.m.
5
a.m.
I
know
you're
in
Denver,
but
still
it's
typically
late
anyway.
B
So
he's
he's
been
doing
a
fabulous
job
of
just
providing
that
support,
but
also
we
quickly
realized
that
one
maintainer
is
not
enough.
So
now
we're
proud
to
say
that
we
have
another
maintainer
who
is
outside
of
Comcast.
That's
also
on
this
project
so
make
sure
that
you're,
really
supportive
of
your
new
contributor,
is
welcoming
and
you're
clear.
B
Also
James
came
all
the
way
from
Denver,
so
I
run
the
Prometheus
meetup
in
philadelphia.
If
anybody
is
on
the
East
Coast
or
around,
please
follow
our
group
on
meetup
comm
we'd
love
to
have
you,
but
we
have.
We
had
our
first
prometheus
meetup
back
in
the
fall
and
James
came
and
he
did
a
talk
on
trickster.
It
was
very
well-received.
We've
also
done
this
talk
at
the
open
source
summit
in
North
America
we're
gonna,
do
the
same
thing.
B
We're
gonna
continue
doing
it
and
then
also
the
go
meet
up,
so
it
doesn't
have
to
necessarily
be
relevant
to
either
project
I'm.
Sorry,
it
doesn't
have
to
actually
be
the
Prometheus
Meetup.
It's
just
as
long
as
it's
relevant
to
your
project.
Then
you
can
find
meetups
to
go
speak
at
also.
We
have
a
trickster
Oh,
trickster
io,
a
Twitter
account.
Please
feel
free
to
follow
us.
We
any
releases
that
come
out.
We
make
sure
to
tweet
out
any
kind
of
testimonials
we
have
or
news
and
updates.
They
also
go
out
on
our
Twitter
account.
B
If
you're
interested
to
learn
more
about
other
open
source
projects
at
Comcast,
you
can
visit
Casta
github
do
we
featured
trickster
and
the
number
of
other
projects
at
the
top
of
our
carousel,
that
we
really
want
you
to
look
at,
but
also
please
contribute
back
to
if
you're
interested
and
then
I
sort
of
found.
This
interesting
borrowed
the
slide
from
metroburg.
He
put
out
a
blog
post
about
the
return
of
investment
on
stickers
and
I.
B
Won't
read
all
of
this,
but
the
key
takeaway
is
that
stickers
are
for
between
5x
and
76
X
greater
return,
and
with
that
I
will
say
that
we
do
have
a
bunch
of
stickers
here
for
you
all,
if
you're
interested,
we
have
them
in
the
back.
We
have
them
here
on
the
stage
and
you
can
come
ask
us
for
one
if
you'd
like
and
thank
you,
we
have
some
time
for
questions.
Yeah.
C
So
we've
learned
about
a
few
other,
exciting
add-ons
for
Prometheus
in
this
conference.
Cortex
and
Fae
nose
as
well.
So
it
sounds
like
maybe
there's
some
feature
overlap
with
some
of
the
caching
and
performance
improvements,
but
then
also
some
things
that
are
totally
different,
like
the
durability
and
long-term
storage
that
some
of
those
other
solutions
provide,
so
I
would
be
interested
to
know.
Is
there
operability?
C
A
So
we
do
have
users
who
use
the
trickster
in
front
of
Thanos
already,
and
you
can
actually
take
a
look
at
our
issues.
We've
actually
worked
through
issues
to
make
that
interoperable
already
so
most
of
the
situations
where
we
have
interoperability
issues,
people
submit
those
issues
with
us
and
we
fix
them
pretty
quickly.
So
we
do
have
all
of
those
things
working
correctly.
A
A
good
question,
so
trickster
is
highly
configurable
one
of
the
first
things
I
did
as
I
built.
The
app
was
to
build
in
a
lot
of
configurability
in
terms
of
TTLs,
not
only
on
individual
data
points,
but
also
on
the
entire
cache
of
a
particular
query
itself.
That's
one
of
the
great
things
about
Richter
I
didn't
mention
is
it
is
adaptive
to
all
of
your
query.
So,
if
you're
using
things
like
reporting
rules,
that's
great
but
I
highly
encourage
those.
There
are
situations
where
you
might
not
control
your
own
dashboard
data
sources
right.
A
You
might
be
packaged
everything
you
might
just
run
the
dashboard
as
a
service
and
in
those
cases
it's
really
important
to
be
able
to
cache
random
queries
that
any
user
of
your
dashboard
that
creates
those
can
cache
and
so
I
do
adapt
those
queries
adaptively
each
one
is
stored
in
its
own
little
hashed
entry
and
each
one
of
those
hatches
has
a
configurable
TTL
on
it,
as
well
as
how
far
back
in
time
the
data
points
can
last
and
both
of
those
are
independently
configurable.
We
do
also
offer
a
multi
origin
solution
with
trickster.