►
From YouTube: Sponsored Keynote: Upstream-First, High Scale Prometheus Ecosystem - Kemal Akkoyun, RedHat
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Sponsored Keynote: Upstream-First, High Scale Prometheus Ecosystem - Kemal Akkoyun, Software Engineer, RedHat
Welcome to the beautiful world of cloud-native observability. Where things that your application is doing are understandable, clear, and non-surprising. Where you get notified only when the user is impacted or will be soon. Where you can in minutes drill down to what’s wrong, where the impact is and why. Where running the observability does not mean deploying 4 separate big, expensive systems with different UI and endpoints.
This is what is achievable right now with open source projects, we help maintain at Red Hat. Prometheus is no longer just one binary. It’s the way of thinking, a way of simplifying observability on all fronts. Join us to learn more about how Red Hat runs Prometheus and other open-source projects that share the same principles and patterns.
A
Hello,
everyone,
I
hope
you
are
enjoying
the
conference
so
far
today.
I
am
going
to
talk
about
how
we
use
prometheus
and
its
friends
for
openshift
offering
in
reddit,
and
I
am
going
to
tell
you
about
the
system
that
we
built
for
remote
health
monitoring
for
the
openshift
clusters,
also
how
we
gather
telemetry
data
to
make
data-driven
decisions
in
reddit
without
further
ado.
Let's
start
I
work
for
openshift,
observability
and
monitoring
team.
As
a
software
engineer
also,
I
am
a
thanos
maintainer
and
a
prometheus
contributor.
A
As
a
team,
we
are
building
a
platform
to
collect
and
store
observability
signals.
Also.
We
have
sre
responsibilities
and
we
are
on
call
for
the
internal
platform
that
we
are
building
the
system
that
we
build
is
called
telemeter.
So
what
is
the
limiter
in
simple
terms?
Telemeter
is
an
open
source,
remote
health
monitoring
system,
remote
health
monitoring
works
by
sending
a
carefully
chosen
subset
of
data
back
to
the
telemeter
service.
A
A
So
the
the
primary
information
collected
by
telemetry
includes
the
number
of
updates
available
per
cluster.
The
number
of
errors
that
occur
during
an
update
the
progress,
information
of
running
updates,
health
condition
and
status
of
openshift
component
that
is
installed
on
the
clusters.
The
name
of
the
platform
that
openshift
deployed
on
such
an
aws.
A
The
data
we
collect
enables
us
to
provide
a
lot
of
benefits
to
the
end
user,
otherwise,
would
be
impossible
with
the
help
of
the
telemetry
data.
We
can
observe
events
that
may
seem
normal
to
a
single
and
a
single
end
user,
but
with
the
perspective
of
seeing
those
events
across
the
fleet
of
users,
we
can
provide
more
insights
with
the
connected
information.
We
can
improve
the
quality
of
the
releases
and
react
more
quickly
to
the
issues
found
in
the
clusters.
A
So
how
we
build
telemetry
in
red
hat,
we
have
an
absolute
upstream
first
mentality,
so
telemetry
is
based
on
the
open
source
tools
on
the
promet
use
ecosystem.
We
deploy
and
maintain
highly
available
prometheus
alert
manager
and
several
thanos
components
in
each
and
every
cluster.
We
run
how
we,
how
do
we
deploy?
A
We
use
prometheus
operator
to
deploy
prometheus
alert
manager
and
several
thanos
components
into
openshift
clusters.
So
let
me
briefly
explain
what
it
is
for
the
ones
who
aren't
familiar
with
the
prometeus
operator.
The
prometeus
operator
provides
kubernetes
native
deployment
and
management
of
prometus
and
related
monitoring
components.
The
purpose
of
this
project
is
to
simplify
and
automate
the
configuration
of
a
prometeuse-based
monitoring
stack
for
the
kubernetes
clusters
by
using
prometeos
operator
out
of
the
box.
We
monitor
critical
cluster
components
and
alert
on
the
metrics
we
collect.
A
A
In
the
first
version
of
the
system,
the
incluster
pro
meteors
was
collecting
the
data
from
the
workflows
in
prometus
format,
and
then
a
dedicated
component
was
scraping
the
federated
endpoint
of
prometheus
every
four
and
a
half
minutes.
Then
the
component
was
cleaning
the
metrics
and
anonymizing
the
data
and
then
sending
the
metrics
in
prometus
federate
data
format
to
the
server-side
telemeter
server.
A
On
the
server
side,
we
had
the
telemeter
server
component.
It
was
receiving
the
data
on
the
federate
data
format
and
store
the
data
into
its
hashing
of
numerous
replicas,
and
then
the
telemetry
servers
had
been
scraped
by
the
two
replicas
of
prometheuses
ingesting
all
of
the
all
of
the
data
twice
the
hashing
was
super
primitive.
A
A
Tunnels
help
us
to
build
a
cost
efficient
store
for
historical
metric
data
while
retaining
fast
query
latencies.
We
introduced
a
new
custom
central
metrics
collection
pipeline
for
telemeter,
using
tunnels
to
make
it
even
more
scalable
red
invested
in
adding
a
new
component
to
thanos
called
receipt.
At
that
point,
this
was
a
novel
idea
for
thanos
because
it
had
changed
thanos
model
from
a
pool
based
solution
to
a
push-based
one.
This
effort
was
started
in
june
2019
and
we
ended
up
building
a
sas
offering
for
reddit's
internal
usage
for
internal
customers.
A
So
how
does
the
niv
stack?
Look
like
all
the
all?
The
ink
cluster
bits
remain
the
same
and
we
converted
the
telemeter
server
to
a
mere
authentication,
proxy
and
data
transformer
for
the
legacy
endpoint,
all
the
uploaded
data
now
being
converted
to
the
prometeus
remote
data
format
and
send
send
it
over
to
the
thanos
receiver
stream.
A
We
added
a
new
endpoint,
though,
with
the
new
endpoint,
we
added
the
ability
to
support
direct
descending
metrics
from
the
prometus
through
telemeter
server
using
a
remote
remote
drive
api
in
the
upcoming
openshift
versions.
We
are
planning
to
move
completely
to
writing
directly
from
prometheuses
to
telemetry
servers.
A
But
how
about
the
other
signals?
I
mentioned
that
we
collect
them
right,
so
this
diagram
shows
what
we
have
now
metrics
and
logs,
and
what
are
we
planning
to
build
soon,
tracing
and
profiling
to
extend
the
functionality
of
telemetry?
We
decided
to
build
yet
another
open
source,
observability
system.
Our
main
goals
were
to
provide
multiple
observability
signal,
support,
correlation
between
signals,
seamless,
multi-tenancy,
authentication
and
authorization
and
improved
security
thanks
to
prometheus
ecosystem
and
the
other
systems
that
were
built.
A
Similarly,
this
was
a
relatively
easy
task,
so
we
based
our
design
on
two
major
points:
schema-less
labels
and
object
storage
support.
We
recently
started
to
provide
logging
solution
using
grafana
to
our
internal
customers
at
reddit,
because
this
system
observatorium,
you
can
think
observatorium
as
a
distribution.
A
We
have
packed
a
reference
architecture
of
thanos
loki
and
soon
others
that
allows
easier
installation
configuration
and
operating
of
several
observability
systems
that
we
found
the
most
useful
and
practical.
And,
of
course
all
of
these
efforts
are
open
source.
You
can
check
out
everything
we
have
done
so
far
by
visiting
our
website.
Please
do
so
so
what
we
learned,
while
doing,
while
building
this
platform
in
such
a
big
organization
like
ours,
being
able
to
offer
an
internal
sauce
for
the
less
experienced
teams
extremely
useful.
A
Moreover,
the
flexibility
of
deployment
and
design
is
must
have
requirements
and
priorities
change
overnight,
and
we
had
the
support
from
one
cpu
cluster.
To
many
many
huge
clusters
using
schema-less
labels
to
correlate
signals
was
paid
off.
Api
driven
model
helped
us
to
extend
the
system
easily
and
cost
matters
and
on
an
even
medium
scale.
It
takes
millions
of
dollars
to
gather
all
the
log
lines,
traces
or
events
so
focusing
on
actionable
metrics,
helped
us
to
scale
seamlessly
and
also
relying
on
the
object
storages
for
long-term.
Retain
retention,
helped
us
to
control
all
the
costs.