►
From YouTube: Sponsored Keynote: Monitoring Production Clusters at Scale - Filip Petkovski, Sr. Software Engineer
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Sponsored Keynote: Monitoring Production Clusters at Scale - Filip Petkovski, Senior Software Engineer, Red Hat
In this talk, Filip Petkovski & Ian Billett, engineers from the Red Hat Monitoring Group will explain how Red Hat leverages Prometheus and its ecosystem components to monitor tens of thousands of clusters running critical customer workloads. The audience will learn how tens of billions of metrics are gathered from user applications & infrastructure components using Prometheus, how this data is stored at scale, and how developers, admins and business stakeholders use this data to ensure that OpenShift is delivering a first-class experience for customers of Red Hat.
This talk will outline problems faced when running this infrastructure, and explain how we have managed challenges such as multi-tenancy, scale, authorization and more
A
Hey
everyone.
Thank
you
for
joining
the
prom
con
keynote
today,
I'm
looking
forward
to
all
of
the
exciting
talks
that
we
have
in
line
this
year,
I'd
like
to
kick
off
the
conference
by
exploring
the
prometheus
ecosystem
and
show
that
there
is
much
more
to
it
than
just
prometheus.
We
can
easily
do
that,
but
by
going
through
the
monitoring
stack
that
we
deploy
as
part
of
openshift
and
for
those
of
you
who
are
not
familiar
with
openshift,
you
can
think
of
it
merely
as
a
kubernetes
distribution.
A
So
everything
that
we
build
as
part
of
the
openshift
monitoring
stack
very
easily
applies
and
translates
to
kubernetes
itself
I'll.
Just
briefly
introduce
myself.
My
name
is
philip.
I'm
a
member
of
the
ratchet
openshift
monitoring
team.
Besides
that,
I'm
also
fairly
active
in
upstream
projects,
most
notably
in
the
prometheus
operator
and
cube
state
metrics,
all
right,
let's
jump
right
into
it
in
order
to
illustrate
how
we
would
build
a
full
bone
monitoring
solution
by
just
leveraging
what's
available
there
in
the
ecosystem,
let's
start
by
a
vanilla,
kubernetes
distribution.
A
Sorry
I
have
a
little
kubernetes
cluster
and
see
what
we
can
build
so
for
us,
at
the
core
of
the
monitoring.
Stack
is,
of
course,
no
surprise
prometheus.
We
deploy
prometheus
using
the
prometheus
operator,
and
we
do
that
because
the
operator
can
do
a
very
good
job
at
upgrading
prometheus.
So
whenever
we
need
to
deploy
a
new
version
into
the
stack,
we
are
confident
that
the
operator
is
going
to
do
the
right
thing
and
is
going
to
make
sure
that
the
upgrade
is
successful.
A
A
Metrics
gives
us
metrics
about
the
kubernetes
environment,
whereas
node
exporter
gives
us
real
time,
information
about
resource
usage
of
nodes
and
processes
or
containers.
So
we
can
see
that
with
just
a
handful
of
components,
we
have
what
is
traditionally
referred
to
as
infrastructure
monitoring,
and
this
setup
is
already
useful.
I
mean
you
can
query
a
lot
of
metrics.
If
you
have
a
dashboarding
system,
we
can
create
dashboards,
we
can
create
recording
rules
and
so
on.
A
In
addition
to
what
we
have
here,
we
also
allow
what
we
consider
core
openshift
components
to
register
their
targets
with
this
primitive
instance,
and
they
do
that
by
using
the
service
and
pod
monitors
that
are
provided
by
the
prometheus
operator.
So
the
operator
then
scrapes
those
resources
and
configures
prometheus
to
scrape
core
openshift
components,
and
this
kind
of
setup
up
until
now
has
all
of
the
metrics
that
are
required
for
monitoring
openshift
as
a
platform,
and
this
is
the
reason
why
we
call
this
setup
the
platform
monitoring
premises.
A
We
also
allow
users
to
monitor
their
applications,
but
we
spin
up
a
new
instance
for
them.
We
want
to
separate
the
two
failure
domains
so
that
if
a
user
application
accidentally
causes
a
cardinality
explosion
and
the
platform
metrics
are
still
protected
and
admins
or
anyone
with
access
to
those
metrics
can
troubleshoot
and
see.
What's
going
on
in
the
platform,
of
course,
for
routing
alerts,
we
deploy
alert
manager
again
through
the
operator
and
an
extremely
important
component
for
us
is
the
thundersquare.
A
So
the
query
or
the
time
square
serves
two
functions
for
us
one.
It
provides
a
single
query
endpoint
to
users
and
therefore
it
serves
as
an
api
essentially
for
the
monitoring
stack,
so
users
don't
have
to
think
like
which
prometheus
do.
I
need
to
query
for
which
metric
they
simply
use
a
single
endpoint
and
two
it
also.
So
the
query
can
join
metrics
across
prometheus
instances,
which
means
that
users
can
issue
queries
that
have
metrics
both
from
the
platform
and
from
their
applications
all
right.
A
A
What
we
want
to
do
is
we
want
to
carry
over
this
multi-tenant
property
into
the
monitoring
stack
and
we
want
to
say
tenants
should
only
be
able
to
access
metrics
that
they
essentially
have
access
to
that.
They
that
belong
to
them,
so
multi-tenancy
in
openshift
is
defined
on
a
space
basis
and
if
tenant
one
has
access
to
namespace
one,
only
they
should
only
be
able
to
see
metrics
from
namespace
1..
A
So
we
specifically
configure
permeable
proxy
to
inject
the
namespace
label,
but
this
is
an
average
configuration
specific
to
us
and
so
when
a
tenant
issues,
any
prompt
query
in
this
case,
for
example,
cube
pod
info
from
level
proxy,
is
going
to
inject
the
appropriate
names
to
this
label,
and
so
tenants
don't
actually
know
that
this
transformation
is
happening.
They
only
yeah
they
would.
A
They
basically
make
the
query
as
they
would
anywhere
and
the
kind
of
these
underlying
plumbing
scopes,
the
query
to
return
metrics
that
belong
to
them
all
right,
so
this
is
kind
of
this
kind
of
setup
has
worked
for
us
for
a
fairly
long
time.
We
have
this
kind
of
stack
deployed
across
probably
tens
of
thousands
of
clusters
at
the
moment.
What
we
are
seeing,
though,
is
that
some
some
users
are
growing
out
of
this
setup
and
they
do
that
by
requiring
either
a
bit
more
flexibility
or
a
bit
more
resilience.
A
We
see
here,
for
example,
that
we
have
two
prometheus
instances,
but
we
don't
necessarily
have
to
have
two.
I
mean
these
two
number
two
is
fully
arbitrary
and
has
worked
for
us
until
now,
but
what
we
want
to
do
next,
to
improve
the
resiliency
and
also
the
flexibility,
is
to
slim
down
this
prometheus
instance
that
we
call
platform
monitoring,
primitives
and
use
it
for
absolutely
critical
metrics
that
are
that
are
needed
for
monitoring
a
plane,
kubernetes
cluster,
and
then
we
would
delegate
the
monitoring
of
each
core
component
to
its
own
monitoring
stack.
A
We
would
also
like
to
allow
users
to
spin
up
such
stacks
with
very
minimal
configuration,
so
this
kind
of
we're
still
exploring
this
approach
and
we're
the
beginning
of
the
project,
but
once
we
launch
it,
we
would
like
to
make
it
kind
of
a
fully
open
source
project
that
anyone
can
contribute
to
and
anyone
can
use.
So
we
will
want
to
have
to
provide
kind
of
a
well
a
good
way
to
run
a
prometheus-based
stack
in
isolation.
A
So,
in
summary,
this
is
how
we
leverage
everything
that
the
community
has
built
and
is
available
in
the
community
to
kind
of
compose
a
very
robust
and
very
good
monitoring
stack.
Of
course,
we,
as
a
monitoring
team,
also
actively
contribute
to
almost
all
of
the
projects
that
we
went
through
here
and
all
the
projects
that
we
pull
in
for
our
stack
and
if
you
are
someone
who
enjoys
being
part
of
this
community
and
would
like
to
keep
it
vibrant
would
like
to
keep
contributing
to
it.