►
From YouTube: How Attackers Use Exposed Prometheus Server to Exploit... David de Torres Huerta & Miguel Hernández
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How Attackers Use Exposed Prometheus Server to Exploit Kubernetes Clusters - David de Torres Huerta & Miguel Hernández, Sysdig
Prometheus has become the standard for monitoring Kubernetes services. It comes with a set of helpful exporters, and Kubernetes offers several metrics endpoints directly through the API. These features enable monitoring and troubleshooting of most situations that SREs face on a daily basis. But, what if an attacker accesses your Prometheus server? How much information can they get for fingerprinting the cluster? Kernel versions, IP addresses, instance types, library versions…the list goes on and on. In this session, you will learn how attackers use this information in the first part of reconnaissance, to see if you are vulnerable. The speakers will share - What secrets they collect to fingerprint your Kubernetes cluster (hint: they're not after your timeseries) - How to leverage this information internally to secure your cluster - How to prevent the exposition of sensitive information No matter how many safety best practices you apply, you must be aware of every link of the chain.
A
Hello,
everyone,
I
think,
it's
time
to
start
or
talk.
Anyone
is
here.
Thank
you
all
of
you
for
coming
here
or
and
listen
us
in
this
talk.
It's
really
nice
to
see
faces
not
in
virtual
and
thank
you,
the
organizer
on
the
cnca
for
ascertain
or
to
be
to
be
here.
We
are
to
talk
about
how
attackers
viewers
expose
prometheus
and
server
to
exploit
kubernetes
cluster.
A
My
name
is
miguel
hernandez.
I
work
as
a
security
researcher
for
the
last
five
years
in
different
topics
like
cloud
security
or
synthetic
content
to
social
engineer
or
also
flood
detection,
and
I
work
with
davide
torres
he's
manager
of
prometheus
team
at
csd
and
we're
here
to
try
to
explain
this
mem.
It's
like
we
put
more
and
more
liars,
more
and
more
software,
but
if
you
fail
in
one
point,
everything
is
broken
and
with
that
is
sports
matrix.
Okay,
it's
a
typical
insecurity.
A
A
We
don't
like
to
release
any
new
vulnerability
or
cve
to
kubernetes
or
prometheus.
Here
is
for
common
sense
to
educational
purpose
and
we'd
like
to
provoke
any
attack
for
real
customers,
and
also,
we
recommend
to
follow
always
the
security
best
practices
in
kubernetes
and
also
to
use
promotions
to
monitor
everything,
but
please
don't
open
the
door
to
attackers
or
expose
your
credential
for
free,
because
it's
very
easy,
if
you
are
an
attacker
to
get
to
get
access,
why
is
important
or
why
someone
likes
to
do
coronary
fingerprinting?
A
Well,
if
you
are
a
pen,
testing
or
ethical
hacking
or
you
work
in
security,
the
first
thing
to
do
is
when
you
have
a
target
is
to
gather
all
information
as
you
can.
Okay,
you
need
this
information
to
choose
the
best
tool,
the
best
technique
to
get
access
or
exploit
these
things,
and
it's
normal
to
match
with
cvs
or
non-vulnerabilities
with
the
exactly
tool
and
you
get
access.
It's
the
first
step
to
initial
access.
You
need
to
gather
information,
and
we
try
to
explain
in
this.
A
A
It's
something
I
don't
know
you
you
like
to
protect
everything
and
you
follow
the
securities
practices
and
you,
I
don't
know,
use
a
lot
of
security
techniques
or
application.
If
you
expose
your
credential
in
clear
text,
doesn't
matter
everything
you
that
you
access
for
free
to
your
cloud
provider
and
start
to
crypto
mining,
for
example,
but
this
is
one
example
of
the
things
you
can
do
and
I
like
to
remark
this,
because
in
spain.
A
Like
you,
you
work
in
the
incident
response
team
or
digital
forensics,
the
first,
the
most
important
thing
to
to
do
when
you
create
the
report
is
to
to
know
what
happened
with
this
in
an
incident,
and
you
need
to
learn
from
the
past
to
don't
repeat
them
again
in
the
future,
and
this
is
important,
because
what
happened
when
prometers
you
expose
again
and
leak
the
credential
or
not.
Well,
also,
we
like
to
mention
that
this
is
not
the
first
thing
or
the
first
guys
that
talk
about
this.
You
do
a
quick
search
in
google.
A
You
can
see
an
issue
from
this
shop
that
discusses
the
same
topic.
It's
like,
if
you
use
prometheus
and
expose
the
the
dashboard,
is
a
liquor
credential
or
you
have
a
problem
in
the
future
or
not.
Also
gifrock
write
this
amazing
article
about
this
problem,
but
more
focus
on
the
secrets,
not
in
the
pivot.
You
know
other
information,
but
this
is
really
nice
and
finally,
we
are
not
the
first
speaker
talking
about
hacking
monitoring.
This
is
something
that
is
cool
but,
as
I
mentioned
before,
there
is
repeat
and
repeat,
and
they
it
doesn't
matter.
A
If
you
explain
once
sure
doesn't
matter,
the
first
thing
to
do
is
a
real
scenario
to
access
to
a
promiscuous
spot
to
the
internet.
Well,
you
need
to
configure
it's
not
by
default
the
dashboard
open
to
the
world,
but
the
configuration
is
not
enabled
and
if
you
configure
the
dashboard,
ensure
you
configure
the
authentication
or
is
free
to
the
attackers
to
make
queries
or
access
to
the
prometheus
metrics.
A
And
if
you
do
a
simple
google
docking,
you
can
see
some
several
results
with
real
prometheus
server
expose
and
also,
if
you
are
in
different
security
conferences.
This
is
very
typical
use.
Other
engines
like
soda
and
sensors
or
fofa
to
see
if
the
impact
is
a
real
impact,
because
there
are
a
lot
of
servers
suppose
or
not
with
solan
is
more
simple
because
you
use
to
filter
the
the
favicon.
But,
as
you
can
see,
a
lot
of
servers
are
exposed.
B
B
B
C
C
C
C
So
we
will
have
also
information.
Also,
these
are
from
the
k,
cube
state,
metrics
exporter.
We
have
also
information
about
the
hostname,
with
the
cube
underscore
node
and
the
sql
info.
We
can
even
make
some
blackmagic
with
from
ql
to
filter
all
the
service
of
the
kind
load
balancer.
As
you
know,
these
load
balancer
services
are
the
one
that
the
service
provider
implement
behind
something
to
expose
to
the
public,
and
we
can
also
have
information
about
the
ingressors
that
we
have.
This
is
interesting
because
the
ingresses
are
telling
us
which
reverse
proxy
we
are
attacking.
C
C
We
are
starting
to
have
more
information
now
we
are
actually
starting
to
make
the
most
important
thing.
That
is
the
rule
of
attack.
We
know
that
behind
these
example.com
slash
api,
slash
whatever
it
goes
to
as
ingress
and
that
ingress
go
to
a
service
and
the
service
go
to
a
pot,
and
we
can
do
this
with
different
endpoints
of
the
api.
C
But
let's
go
further
with
ksm
we
can
have
the
namespaces,
we
can
have
the
workloads
deployment.
The
stateful
sets
even
the
pods
the
container
inside
the
pods,
so
we
are
starting
to
have
real
information
of
what
are
the
names
of
the
applications.
What
are
the
name
of
the
workloads
that
we
have,
whether
the
name
of
the
containers?
What
are
the
names
of
the
cron
jobs
that
are
being
executed?
C
C
Let's
change
the
dimension,
we
are
talking
about
kubernetes,
but
let's
talk
about
is
behind
the
machine.
We
can
have
information
about
both
the
image
that
the
node
is
running
and
the
kernel
version-
and
this
is
important
because
we
are
adding
more
non-kubernetes
not
only
of
the
cover
letters
cluster,
but
also
from
both
the
kernel
version
and
the
image
of
the
operative
system
that
it's
using.
C
Let's
go
farther.
Let's
go
for
applications
we
are
having
cubestead
metrics
is
giving
us
information
about
the
image
and
the
tag
of
the
containers
running
in
the
cluster.
So
we
know
which
image
it
is
running
which
stack
they
are
using,
which
hash
it
is,
and
even
we
know,
also
the
registry
that
they
are
using.
C
If
we
even
want
more
information,
some
applications,
they
are
even
exposing
the
more
specific
information
about
the
version
or
even
the
configuration
options
in
some
custom.
Metrics
like
prometheus,
underscore
build
underscore
info,
and
now
it's
when
it's
starting
to
get
interesting,
because
we
are
adding
even
more
normal
variabilities
for
each
application
that
we
can
find
in
the
cluster.
C
And
last
but
not
least,
we
are
also
having
information
about
the
secrets.
You
might
say.
Okay,
you
only
know
the
name
of
the
secret,
yes,
but
we
are
also
having
information
about
the
annotations
and
the
content
of
the
annotations
of
the
secrets
why
this
is
important.
I
don't
know
if
you
know
that
in
some
versions
of
cube,
ctl
cube
ctl
was
adding
an
annotation
of
the
current
applied
configuration
that
was
exposing
in
plain
text
directly
the
content
of
the
secret.
C
So
we
already
have
a
lot
information
and
we
already
have
a
nice
map
of
the
cluster
that
we
want
to
attack,
and
you
know,
do
you
want
to
know
something
funny?
We
gather
all
this
information,
most
probably
without
no
one
noticing,
because
prometheus
has
the
capacity
of
logging,
all
the
queries
made
by
the
api
on
the
front
end.
But
unfortunately
this
is
disabled
by
default
and
you
can
even
check
if
this
is
disable
or
enable
with
this
metric
like
prometheus,
underscore
engine
underscore
query
underscore
log
underscore
enable
if
this
is
a
zero.
C
A
Drive
from
here,
okay
see
next
thing
to
do.
Okay,
we
have
all
this
information
is
quite
simple
to
make
queries
gather
all
the
information
map.
Everything
and
you
don't
know
you
don't
need
to
initial
access
to
know
if
these
components
are
vulnerable
or
not,
if
you
can't
escape
or
not,
because
you
have
the
information
yet
all
right,
we
like
to
expose
three
different
examples
and
how
this
is
possible
or
this
fictitious,
but
not
really,
it's
possible
to
achieve
the
goal
of
the
attacker.
A
It's
like
I
like
to
leak
data,
or
I
like
to
crypto
mining
or
like
run
somewhere
in
depends
of
the
goal
you
use
one
or
other
techniques,
or
you
can
perform
different
paths
to
get
the
access,
and
I
think
it's
interesting
to
see
what
happened
with
all
this
information.
The
first
scenario
is
with
like
data.
Okay.
This
is
an
example
of
a
real
scenario.
Okay,
everything
is
fine.
We
follow
the
security
best
practices.
We.
A
The
first
thing
to
do
is
use,
for
example,
a
sql
map
to
try
to
do
some
injection
in
the
api
and,
if
possible,
to
do
or
not
it
depends,
but
in
case
you
can't,
the
next
step
is
well.
I
know
what
is
the
registry
is
third
party
or
or
private.
I
know
the
images
and
I
can
perform
some
social
engineer
to
get
access
to
a
third-party
registry,
for
example,
because
I
like
to
upload
a
malicious
images
in
the
continuous
delivery
to
get
access,
because
the
other
way
is
impossible
to
initial
actors.
A
In
some
cases
for
sensing
scenarios
we
craft
the
the
imaging
with
a
vulnerability
to
get
that
example
from
the
other
steps
to
levy
privilege
or
something
like
that.
But
in
this
case
you
need
to
focus
on
the
supply
chain,
because
the
other
way
you
need
to-
I
don't
know
zero
days
or
something
like
that.
But
in
this
case
we
need
to
to
do
by
chain
attacks
and
you
can
use,
I
don't
know
beef
to
create
a
fishing
attack
or
you
can
do
something
similar
to
typo
squatting
to
let
a
new
container
with
similar
names.
A
In
this
scenario,
the
crypto
mining-
you
like
the
earliest
credential,
and
you
need
to
do
these
stairs-
to
get
access
to
the
node
and
following
the
or
seeing
the
config
files
or
in
the
environmental
variables
if
the
audio
is
key
or
the
cloud
provider
key
are
in
this
place.
Like
the
example
of
kubernetes
dashboard
is
clear
in
display
in
this
case,
we
need
to
do
these
steps.
This
is
a
long
path.
If
you
like
to
do
very
short,
you
can
do
the
same
example
with
kubernetes
it's
possible.
A
If
you
use
aws-
and
you
configure,
your
secrets-
is
exposed
by
default
in
the
dashboard,
and
you
make
the
query
and
get
access
to
the
credential
and
it's
fine,
you
don't
need
to
do
you
don't
need
to
perform
any
initial
access
or
any
other
step,
because
it's
in
clear,
okay.
This
is
example
that
is
in
the
documentation.
Please
be
careful
if
you
are
supposed
the
prometheus
is
possible,
that
suppose
your
secrets
and
it's
something
known
for
the
for
the
community
and
again
don't
open
the
door.
A
Okay
and
finally,
another
scenario:
ransomware
is
a
bit
strange
in
in
kubernetes.
Cluster
is
more
for
traditional
environmental
database
that
you
cipher
the
data
and
ask
for
a
rescue,
and
in
this
example
it's
a
bit
different.
You
know
an
application
have
a
vulnerability
and
the
initial
act
is
via
spring
cloud.
A
You
can
inject
a
command
beyond
hdb
and
access
to
the
to
the
container,
and
also
you
detect
with
prometheus
that
one
component
target
level
like
one
tv
and
you
can
use
to
change
the
namespace
or
get
access
to
the
secrets
and
access
to
the
data
in
the
database.
The
problem
here
is
to
ask
for
the
money,
but
okay,
I
don't
like
to
to
give
you
any
advice
or
anything
to
know
who
is
possible
to
ask
for
the
money
in
this
case,
because
you
need
something
to.
A
I
don't
know,
deploy
a
pod
with
user
interface
or
something
like
that.
But
it's
something
similar
you
get
information
from
prometheus
and
prepare
the
journey
to
to
get
access.
Okay.
This
is
a
summary
of
these
three
attacks
or
three
scenarios,
and
I
like
to
summarize
that
is
for
it's
important
to
protect
all
the
information.
You
are
exposed.
Okay,
that
doesn't
matter
if
there
are
a
metric
or
it's
a
credential
or
it's
a
server
application.
I
don't
know,
but
it's
important,
because
insecurity
is
a
long
battle.
A
It's
red
team
bursts
blue
team
and
all
the
time
we
are
fighting
or
cat
and
dogs
is
different
ideas,
but
a
long
battle.
If
you
expose
your
credential
or
your
sports
or
prometheus
or
browse
the
dashboard,
you
change
to
a
spear
run.
Okay,
it's
very
simple
and
we
like
to
finish
with
the
normal
recommendation
in
each
step.
Okay,
all
this
talk
is
about
well
follow
the
securities
practices
and
kubernetes
follow
the
security
practices
in
class
in
the
code
in
the
deployment
you
have
these
layers,
and
sometimes
we
forget
the
metrics
and
the
security
metrics.