►
From YouTube: Fleeting Metrics: Monitoring Short-lived or Serverless Jobs... Bartłomiej Płotka & Saswata Mukherjee
Description
Fleeting Metrics: Monitoring Short-lived or Serverless Jobs with Prometheus - Bartłomiej Płotka & Saswata Mukherjee, Red Hat
Prometheus is the leading open-source monitoring solution when it comes to metrics and alerting. It is a single binary that provides you with all you need to monitor your infrastructure and services. It has seen the shift from on-prem to cloud environments and has proven to be successful for users with all kinds of use cases. Prometheus was always designed to aggregate long-living metrics. However, this does not always go along with the solutions that are emerging in the CNCF ecosystem. Short-living workloads are increasingly common in form of Kubernetes batch jobs and serverless platforms like OpenFaas or Lambda and many more. This leads to the question, how and if we can use Prometheus to monitor and troubleshoot those kinds of jobs? In this talk, you will learn about the potential solutions that are emerging in the Prometheus ecosystem. Bartek and Saswata will dive into this problem and propose a set of solutions that could help in monitoring those short-living workloads using the Prometheus data model. The audience will see a demonstration of a solution that uses best practices to capture fleeting metrics and integrates them with Prometheus.
A
A
But
before
that
short
question
to
the
audience,
raise
your
hand
if
you
know
what
the
word
fleeting
means
do
you
know
what
fleeting
means
only
couple
of
hands:
okay,
good!
Thank
you.
I
was
worried
that
everybody
will
know
what
this
means,
because
I
didn't
know
when
I
was
proposing
this
talk
with
saswata
because
yeah.
I
never
heard
this
work
in
my
life.
A
So,
let's
start
with
a
short
explanation,
so
the
dictionary
says
that
fleeting
means
something
that
passes
very
quickly
and
and
generally
is
going
sooner
than
you
expect,
and
I
found
this
amazing
cartoon
art
by
artist
called
mickey
bach.
That
explains
the
word
very
well.
In
my
opinion,
he
actually
did
a
word
a
day
cartoon
series
around
1940s
and
I
think
I
have
to
start
doing
those
to
kind
of
learn.
English
better.
A
For
example,
imagine
wilbur
have
a
lot
of
those
fights.
Perhaps
we
want
to
learn
with
what
ratio
he
is
winning
or
losing
those
fights,
or
perhaps
we
want
to
know
the
median
duration
of
those
fights
right
per
day,
as
you
can
imagine
it's
hard
to
get
those
information
from
our
pool
wielberg,
who
potentially
passed
out
already
after
a
lost
fight
and
maybe
forgets
what
what
happened
and
generally.
This
is
what
we
will
be
speaking
about
today,
so
gathering
metrics
from
those
fleeting
processes,
let's
say
short
disclaimer.
We
won't
solve
all
your
problems
there.
A
It's
it's,
there
is
no
silver
bullet,
essentially,
but
hopefully
we'll
get,
will
give
you
more
context
of
what
what
are
the
options
and
why
it's
not
trivial,
the
problem,
trivial
problem
and
where
are
potential.
You
know
pitfalls
of
of
this
solution
of
this,
of
the
solutions
where
are
available
in
the
space.
But
before
that
short
introduction
together
with
me,
I
have
saswata
on
stage
who
flew
like
over
20
hours,
with
with
two
stops
from
india.
First
time
in
europe,.
B
Everyone,
my
name,
is
sasha
mukherjee,
I'm
a
software
engineering
intern
at
red
hat
on
the
monitoring
team,
and
I
was
a
gsoc
21
student
developer
under
baltic's
mentorship
in
the
thames
project.
I
love
working
with
distributed
systems
and
observability
related
technologies
and
when
I'm
not
working,
I
love
to
read
and
go
out
for
long
walks.
A
And
my
name
is
bartek
brodka
and
I'm
principal
software
engineer
at
red
hat.
I
maintain
various
projects
in
open
source,
mostly
written
in
golang,
including
prometheus
and
thanos,
and
I'm
cncf
tag.
Observability
techlet
and
I
also
write
a
book
with
aurelie
called
efficient
go
and
I
put
way
too
much
content
about
observability,
so
it's
kind
of
related
as
well
so
yeah
so
sweater.
Let's
go.
B
So
let's
first
understand
how
prometheus
normally
monitors
your
workloads.
So
in
any
process
you
can
have
numerous
events
occurring
as
it
programmatically
does
things
that
you
needed
to
do.
You
can
even
see
that
every
single
cpu
instruction
is
an
event,
and
many
of
these
events
or
groups
of
events
are
of
interest
to
people
like
us
who
are
responsible
for
running
these
processes,
as
this
can
tell
us
if
the
system
is
behaving
as
we
expect
it
to,
and
we
want
to
generate
numeric
data
from
these
events
by
aggregating
them
over
time.
B
So
with
this
pull
based
model
of
prometheus
in
mind,
let
us
try
to
understand
how
the
infrastructure
and
the
functionality
that
we
control
has
changed
over
time
and
why
our
metrics
are
now
fleeting.
In
the
past.
We
would
prepare
physical
servers
for
software
to
be
deployed
on
which
would
involve
installing
the
operating
system,
hardware,
upgrades
device,
drivers
and
so
on
and
so
forth.
As
there
was
strong
coupling
between
hardware
and
software,
this
could
be
called
as
static
infrastructure.
B
Afterwards,
virtual
machines
emerged
so
now
you
could
deploy
your
software
to
a
kind
of
simulated
physical
server.
This
provided
a
lot
more
flexibility
and
made
deployments
repeatable
and
started
a
shift
towards
dynamic
infrastructure
and
reduced
the
amount
of
functionality
that
the
user
needed
to
control,
but
there
were
still
some
overhead
and
limitations
with
this,
and
that
is
when
several
containerization
tools
and
technologies
started
to
be
born
and
made
it
possible
to
run
several
different
applications
on
the
same
system
without
any
of
them
interfering
with
each
other.
B
The
surface
area
of
user
control
further
reduced
with
this,
and
then
we
now
have
serverless,
which
abstracts
a
lot
of
this
away.
Users
now
no
longer
have
to
care
about
where
their
code
is
running
or
even
when
their
code
is
running
as
this
taken
care
of
by
serverless
solutions,
and
even
if
it
is
not
serverless
applications
with
predefined
completion
states
have
existed
for
a
while
now,
such
as
bad
jobs
and
cron
jobs.
B
Now
this
is
sort
of
an
oversimplification
of
a
very,
very
long
time
line
of
changes,
but
highlights
a
very
important
trend,
which
is
that
users,
like
you
and
me
now
have
way
less
control
over
the
code
that
we
run
now.
This
is
especially
relevant
in
monitoring,
as
we
do
not
have
the
full
overview
of
the
situation
in
such
environments.
We,
author
the
code,
but
we
don't
run
it.
B
So
what
exactly
is
the
nature
of
such
short
living
or
fleeting
workloads?
So
when
we
talk
about
short-living
jobs,
we
usually
indicate
some
processes
like
serverless
functions,
batches,
bat
jobs
or
cron
jobs
or
even
ephemeral
containers.
And
when
we
talk
about
such
processes,
they
usually
usually
are
running
and
doing
the
work
for
a
limited
but
arbitrary
amount
of
time.
But
short
living
is
kind
of
a
wrong
name
here
or
a
misnomer.
B
Here
we
cannot
define
a
hard
limit
or
a
boundary
for
when
a
process
goes
from
short
to
long
say
if
a
job
runs
for
10
hours,
is
it
long
or
is
it
short?
So
then?
How
do
we
make
the
distinction
here?
Well,
we
can
define
it
as
a
process
that
has
a
well-defined
completion
state,
any
process
which
has
some
completion
state,
after
which
we
don't
expect
it
to
perform
any
operations,
and
it
can
be
terminated,
gracefully
can
be
termed
as
a
short
living
process.
B
So
now
that
we
know
what
a
fleeting
workload
really
is,
let
us
try
to
understand
how
monitoring
is
different,
based
on
the
type
of
processes
for
longer
living
workloads,
which
are
by
far
the
most
common
workload
that
is
monitored
by
prometheus,
like,
for
example,
a
web
server.
It
is
relatively
easy,
as
it
can
be.
B
However,
for
workloads
composed
of
only
fleeting
jobs,
prometheus
pool,
based
model
becomes
problematic
and
effective
monitoring
cannot
be
achieved.
A
scrape
might
miss
the
fleeting
job
completely,
in
which
case
it
simply
cannot
collect
any
metrics,
and
in
the
case
it
does
coincide
with
a
running
fleeting
job.
B
So
to
summarize,
we
have
three
major
problems
when
it
comes
to
monitoring
fleeting
jobs
using
prometheus
semantics.
Fleeting
processes
can
miss
prometheus
grapes
completely
or
partially.
Health
and
success
can't
be
deduced
from
upness
of
the
process
alone.
And
finally,
it's
hard
to
aggregate
data
correctly,
while
you're
only
aware
about
a
single
event,
which
is
a
point
that
we'll
be
talking
about
later
on
and
before
this
talk,
we
also
discuss
the
same
problems
with
open,
fast
and
key
native
communities.
A
Okay,
hello,
hello,
yes,
okay!
So
what?
If
you
are
a
promiscuous
user
or
you
want
to
be
a
prominent
use
user,
but
you
have
a
lot
of
those
fleeting
jobs
like
serverless
functions,
bad
jobs
or
any
kind
of
short
living
processes
as
we
define
them.
What
do
you
do?
Well
initially,
there
are
two
categories
of
solution
that
are
often
mentioned,
maybe
as
an
ultimate
solutions,
but
the
truth
is
they're.
Just
you
know
some
options
in
the
space
and
let's
go
through
those
and
and
those
options
solve
some
of
the
challenges
saswada
mentioned
with
fleeting
jobs.
A
So
let's
focus
on
the
first
one,
which
is
simply
relying
on
event-based
observability
and
deriving
metrics
from
it.
As
we
mentioned
process,
our
processes
are
full
of
events,
and
some
of
them
are
useful
to
observe
and
count.
So
then
we
can
create
alerts,
monitor
dashboards
and
really
use
that
for
observability,
particularly
when
we
look
on
butt
jobs
and
fleeting
processes.
A
You
know
request
with
even
stored
in
just
one
place.
We
can
process
that
information
and
provide
really
any
metric.
We
often
call
those
jobs,
rulers,
aggregators
generators,
sometimes
exporters
and
the
ruler
can
then
deliver
those
this
data
to
promote,
use
or
promotes
compatible
system
by
allowing
promote
use
or
or
some
kind
of
collector,
to
scrape
it,
or
maybe
you
know,
backfill
through
usdb
block
or
something
like
that.
A
To
sum
up
with
this
solution,
we
know
about
all
the
events,
so
it's
kind
of
easy
to
aggregate
and
find
common
dimension
for
for
your
metrics
and
there's
no
regular.
You
know,
interval
of
you
know,
scrape
happening
so
process
can
have
whatever
lifetime
a
lifespan.
It
might
be
fleeting
short,
medium
long
long
living
as
long
as
we
are
able
to
deliver
reliably
this
single
trace
from
each
about
each
event.
A
To
some
back
end,
we
should
be
fine
and
there
are
many
solutions
in
open
source
that
you
can
grab
and
they
will
do
that
work
and
it's
still
growing
right.
So,
for
example,
you
could
use
grafana
loki
with
loki
ruler,
you
can
have
graphana
tempo
and
you
know
there
is,
I
think,
just
I
don't
know
like
I
seen
that
like
two
weeks
ago
there
was
like
tempo
metric
generator
and
there
are
ideas
to
to
have
some
rulers
as
well.
A
So
there
are
some
solutions
there
are
more
and
of
course,
there
are
vendors
who
who
try
to
solve
that
for
for
yeah
for
some
some
same
amount
of
money
and
their
downside
as
everything
and
because
of
all
the
engineering
is
really
about
trade-offs.
First,
one
is
enormous
cost
that
really
usually
results
from
complexity
of
such
system.
You
need
to
gather
every
interesting
even
from
all
fleeting
jobs,
all
fitting
processes
then
process.
You
know
and
derive
metrics
from
those
and
then
so
then
you
can
alert
and
monitor
and
do
the
whole
observability
story.
A
That's
a
serious
traffic
per
second
just
for
observability
data
and
and
and
then
store
all
this
data
in
your
in
your
back
end.
Now.
Imagine
yourself
as
function
does
one
thing
one
event
and
then
admits:
perhaps
few
spawns
tracing
spawns
about
the
situation
and
wants
to
send
it
to
some
collector
right.
A
That's
probably
a
few
kilobytes
of
data
and
some
network
call,
and
so
it
feel
maybe
okay
from
the
first
glance,
but
serverless
are
meant
to
scale
from
zero
to,
like
you
know,
thousand
qrs
per
second
in
milliseconds
second
span
and
your
receiving
pipeline
for
those
events
have
to
have
to
scale
in
the
same
way,
and
you
have
to
pay
for
that.
That's
a
lot
of
compute
power
and
engineering
time,
so
it
means
larger
cost.
A
Not
only
that,
but
you
have
to
store
and
reach
index
and
process
that
data
in
your
back
end
to
derive
meaningful
information
from
it.
Metrics
and
you
are
in
very
much
in
the
big
data
world
this
day.
So
then
you
need
to
pay
and
again,
on
top
of
that,
also
a
metric
system
that
will
use
those
metrics.
So
generally,
it's
more
more
costly
solution
right
and
at
the
end,
is
very
tempting
to
use
this
for
all
observability
as
a
silver
bullet.
A
But
it's
not-
and
I
presented
some
ballpark
calculation
on
the
conference
in
uk
two
weeks
ago
and
just
tracing
back-end
cost
for
some
vendor
you'll
pay
30
times
more
than
just
for
your.
You
know.
Compute
power
for
your
application
logic,
and
I'm
not
counting
here
even
additional
cost
of
collecting
all
of
this
and
sending
to
this
vendor
right.
So
you
can
imagine
it
can
go
very
sideways.
If
you
really
are
you
know
short
of
of
of,
maybe
I
don't
know
you
want
to
save
some
money
right.
A
It's
a
very
popular
technique
of
filtering
information
about
events
to
only
send
those
are
that
are
that
matters
like
requests
that
were
failing
and
maybe
very
slow
or
maybe
interesting
for
some
monitoring
purposes
and
sampling
is
really
must
have
to
have
like
reasonable
cost
of
tracing
even
nowadays,
but
then
it
means
we
cannot
really
derive
metrics
from
that
very
easily,
because
even
storage
or
collector
has
only
partial
information
of
what
happened,
like
maybe
10
percent
of
things.
So
you
don't
have
a
big
picture
and
it's
kind
of
inaccurate
right.
A
So
it's
right
kind
of
defies
the
solution,
but
in
some
way
having
just
the
solution
like
full
sampling
solution
for
just
fleeting
jobs,
which
maybe
is
a
small
part
of
your
overall
system,
maybe
makes
sense.
Maybe
it's
fair
solution
for
you,
but
with
complexity.
Reliability
is
another
problem
and
there
is
a
reason
why
promote
use
is
so
popular.
A
It's
just
a
single
binary
that
just
works
and
close
to
the
workloads,
and
here
there
are
so
many
stages
that
something
can
go
wrong
and
it
makes
your
pipeline
makes
your
monitoring
just
less
reliable
and
some
of
this
you
can
mitigate,
of
course,
by
pushing
more
money
and
engineering
time,
but
yeah.
It's
just
a
trade-off
and
last
but
not
the
least.
A
Something
that
is
maybe
overlooked
is
that
we
treat
serverless
function
as
a
single
quick
job,
and
you
really
do
really
want
to
make
another
tcp
connection
to
send
observability
signal
and
wait
for
it
and
pay
for
that.
Like
price
for
function,
function
depends
on
its
execution
time,
like,
for
example,
in
lambda.
You
pay
for
megabytes
per
second,
you
use.
A
There
are
other
problems
too,
like
again,
you
cannot
buffer
and
batch
this
monitoring.
You
know
data
together
to
maybe
send
it
more
efficiently,
so
it's
much
more
expensive
and
slower
than
if
you
would
have
you
know,
maybe
longer
living
job,
and
and
what
do
you
do
with
you
know
if
tracing
collector
is
down?
How
do
you
buffer
those
things,
doesn't
have
state.
B
So
yeah,
so
let
us
take
a
look
at
another
category
of
solutions
that
exist
in
the
ecosystem
and
allow
us
to
push
partially
aggregated
metrics
instead
of
just
events
and
how
this
might
be
useful.
So
let
us
start
again
at
the
beginning,
where
we
have
a
fleeting
process.
We
have
multiple
fleeting
processes
that
have
some
events
internally,
that
we
are
interested
in.
However,
as
normal
prometheus
scrape
does
not
work
here,
we
need
to
collect
that
data.
B
Somehow
something
that
we
can
do
is
aggregate
some
of
the
recent
data
about
these
interesting
events
within
the
process
itself.
This
partial
aggregation
cannot
give
us
the
full
context
that
we
need,
but
can
only
give
us
the
very
small
context
of
events
within
a
fleeting
process
as
prometheus
is
not
a
push
based
monitoring
system.
B
B
B
However,
as
these
aggregations
can
only
give
us
the
very
small
context
of
a
single
fleeting
workload,
we
cannot
use
it
as
this
is
not
an
overall
view,
as
you
can
see
here,
with
the
fu
counter
metric,
which
is
sent
by
each
fleeting
process
after
it
has
been
incremented
to
values
like
one
two
or
three,
but
for
us
to
get
an
overall
view.
The
metric
can
be
aggregated
at
the
gateway
or
collector
level
to
the
value
of
six,
and
this
is
what
we
can
term
as
a
distributed
aggregation.
B
B
The
buckets
can
be
added
up,
and
if
the
bucket
boundaries
are
mismatched,
the
result
can
be
the
union
of
all
buckets.
So,
as
you
can
see
here
with
when
two
http
request,
durations
of
pi
seconds
or
3.14
seconds
are
pushed
they
get
added
to
6.28
and
for
gauges.
Metrics
can
also
be
added
similarly
to
counters,
but
we
might
want
to
provide
different
semantics
for
aggregations.
B
For
example,
this
cpu
temperature
gauge
would
benefit
from
aggregating
the
last
push
temperature,
instead
of
just
simply
adding
up
all
of
them.
So
several
open
source
solutions
can
help
us
in
building
such
a
partial
metric
push
based
architecture
for
our
fleeting
processes,
all
of
them
with
varying
semantics,
implementations
and
configurations,
and
each
with
its
own
trade-offs.
Also,
functionality
like
distributed
aggregations
are
complex
to
implement
so,
for
example,
prometheus
push,
gateway,
supports,
push
and
cache
functionality,
but
that
but
not
distributed
aggregations.
B
Other
solutions
like
we've
works
from
aggregation
gateway
does
but
aren't
really
customizable,
and
there
is
even
a
new
rust
based
aggregating
gateway.
That
does
this
via
labels,
and
it's
built
by
an
awesome
engineer
at
cloud.
Fair,
called
colin
douche,
who
is
who
we
actually
spoke
to
before
this
talk,
and
he
is
giving
a
kubecon
talk
about
the
same,
and
you
can
rely
on
ecosystems
of
open,
telemetry
and
statsd
as
well
for
partial
metrics,
push-based
architectures
too.
Thus,
you
need
to
carefully
consider
what
fits
your
workloads
now.
There
are
also
problems
around
metric,
stillness
semantics.
B
Metric
stainless
is
not
handled
well
by
such
push-based
solutions.
Solutions
like
the
push
gateway,
never
forget
the
time
series
that
is
pushed
to
it
and
will
expose
them
forever
to
prometheus
until
you
manually
delete
it
via
the
push
gateways
api
which
might
make
it
frustrating
to
use-
and
there
is
this
is
also
the
case
with
aggregation
gateway,
but
it
does
not
even
have
a
metric
deletion
api.
B
As
a
result,
it
becomes
really
easy
to
explode
with
cardinality
with
inconsistent
labels
and
dimensions,
and
this
can
also
lead
to
some
pretty
serious
accuracy.
Issues
which
can
be
really
hard
to
fix
so
with
push.
Reliability
also
takes
a
hit.
Push
request,
may
fail
from
your
fleeting
processes,
and
the
gateway
not
being
scalable
in
the
majority
of
cases
might
become
a
bottleneck
by
dropping
push,
requests
or
simply
not
being
able
to
process
them
and
getting
oomed
as
one
metrics
are
pushed
to
them.
B
Thus,
with
such
solutions,
there
are
way
more
things
that
can
break
and
need
to
be
fixed.
So,
as
bartek
mentioned
earlier,
push
pushing
in
any
form
might
mean
an
enormous
latency
hint,
as
you
have
to
initialize
some
tcp
or
udp
connection,
while
partially
aggregated
metrics
are
easier
to
push
instead
of
actual
event
data.
This
might
already
make
it
too
slow
for
your
environment,
and
we
also
need
to
worry
about
scalability
for
many
of
the
existing
open
source
solutions,
the
gateway
or
collector
components
are
not
horizontally.
B
Scalable,
like
the
prometheus
push
gateway
or
the
aggregation
gateway.
Multiple
instances
often
cannot
be
run
together
or
simultaneously.
So
these
components
will
likely
become
single
points
of
failures
in
your
monitoring
architecture.
As
you
start
to
push
more
and
more
metrics
from
your
applications.
A
Okay,
hello,
hello,
can
you
hear
me?
Okay,
as
we
learned
the
two
mentioned
solutions,
pushing
events
and
pushing
metrics,
they
have
their.
You
know
smaller
or
bigger
problems,
and
the
main
theme
is
complexity
and
it
might
be
fine
for
your
requirements
by
the
way.
But
maybe
there
is
something
simpler
we
could
do.
A
We
might
ask
wilbur's
partner,
perhaps
wife,
that
you
know
we're
watching
all
those
fights
she
might
know
not
to
know
exactly
what
was
on
wilbur
mind,
but
she
saw
the
whole
situation
right,
and
this
is
really
some
lesson
from
our
requirements.
Gathering
discussions
with
openfast
community,
it's
funny
kind
of
we
met
with
the
maintainers
and
to
you
know,
including
alex
who
is
here,
and
we
asked
hey
with
the
promising
use
community.
What
do
you
need?
What
do
you
need
to
make
open
first
better
and
we're
like?
What
do
you
mean?
A
We
are
happy.
I
was
like
okay,
why
you
are
happy
and
the
reason
why
they
are
happy
is
they
build
their
monitoring
and
they
get
this
mentor
monitoring
information
cheaply
from
surrounding
of
of
those
fleeting
processes,
which
also
mean
there
is
no
instrumentation
manual,
instrumentation
needed
it's
auto
instrumented.
A
If
you
think
about
the
smart,
there
are
many
long
living
processes
around
your
fleet,
for
example,
orchestration
system
like
kubernetes,
you
have
cube
state
metrics.
So
if
you
have
bad
job,
you
don't
need
to
instrument
back
job.
You
can
tell
it's
we,
you
know
kind
of
successful
rate
or
durations
by
just
cube.
State
metrics
right
the
network,
proxies
or
gateways
are
part
of
your
request,
so
you
can
deliver.
A
You
know
they
can
provide
metrics
about
from
network
traffic
perspective,
so
you
can
totally
build.
You
know
red
method,
monitoring
like
rates
errors,
duration,
all
of
it
just
you
know
from
you
know,
service
meshes.
Finally,
our
os
knows
ever
kernel
knows
about
everything.
Every
cpu
instruction
I
o-
that
is
happening
with
your
serverless
function.
So
also
memory
usage
and
performance,
and
so
so
everything
else
you
can
kind
of
derive
from
existing
tools
like
c
groups
ebpf.
A
The
thing
that
you
might
miss
is
a
it's
an
open
box
monitoring,
so
things
that
are
with
really
within
the
context
and
logic
and
of
the
function
itself.
But
because
function
is
super
super
small.
The
context
is
usually
little
very
small.
So
the
question
is:
do
you
really
need
that?
Right
and
second,
I
don't
think
I
have
time
for
that.
We
can
chat
about
that
later.
But
essentially
it's
like
recommendation
for
serverless
architecture
as
a
user
or
serverless.
You
cannot
do
more,
but
we
have
to
skip
that.
A
Unfortunately,
but
all
of
this
yeah
you
know
the
recommendation.
I
didn't
mention
and
also
the
closed
bus
monitoring.
What
you
can
do,
if
you
are
just
a
user
of
like
lambda
or
amazon,
lambda
or
cloud
run
or
any
of
those
existing
solutions
right
in
some
way.
The
reality
is
that
you
know
clouds.
Maybe
cloud
vendors
want
to
lock
you
in
in
some
way,
so
users
are
left
without
kind
of
possibility
to
use
normal,
prompt
use,
metrics
normal
code
registries
and
instrument,
normal
instrumentation
and
and
those
close
box
monitoring
things.
A
So
it's
really
about
you
know
also
incentivize
those
vendors
to
integrate
more
with
open
source
and
cncf
kind
of
monitoring.
So
I
guess
it's
some
time
for
some
feature
requests
and
here
and
there,
if
you
are
users
of
those
of
those
platforms,
so
really
yeah,
that's
the
some
of
it,
and
the
big
learning
here
is
that
there
is
no
silver
bullet
here.
A
Unfortunately,
both
even-based
solution,
pushing
metrics,
close
box
monitoring
has
their
pros
and
cons,
and
hopefully
we
learned
today
about
some
trade,
volt
trade-offs
and
implications,
and
hopefully
all
of
this
will
give
this
important,
debugging
monitoring
and
observability
data
to
wilbur
who
can
improve.
You
know
feast,
duels
or
ideally,
better
negotiation
skills
to
solve
disputes
in
a
more
friendly
way
and
yeah.
That's
it.
Thank
you
very
much.
D
B
Yeah
yeah,
so,
as
far
as
I
understand
it,
push
gateway
is
not
recommended
for
those
kind
of
solutions.
It
does
not
have
any
such
distributed
aggregation
capabilities,
so
it's
only
recommended
for
service
level,
ephemeral,
bad
jobs
and
running
it
on
like
a
daemon
set
or
even
multiple
replicas
of
it
isn't
recommended.
I
think
there
was
some
mailing
list
about
it
too,
where
brian
commented
that
no,
it
isn't
safe
to
run
it
like
that.
So
yeah.
A
A
That
schedules
those
those
functions
and
maybe
only
one
type
of
the
function,
and
then
only
you
scale,
those
long
living
processes
that
orchestrates
those
functions,
because
those
functions
should
be
super
small.
There
is
overhead
to
even
start
the
process,
so
it
would
be
good
because,
finally,
you
have
some
long-living
worker
that
will
share
some
state
that
then
you
can
batch
different
things.
A
Not
only
monitoring
but
but
you
know
maybe
other
state
as
well,
for
for
even
for
your
logic,
right
and
yeah,
but
the
state
of
of
the
world
right
now
is
that
you
know
they
they
natively
use
kubernetes.
So
they
would
need
to
move
a
little
bit
away
from
that.
So
you
know
it's
just
a
recommendation
from
from
you're
someone
that
looks
at
this
architecture
from
monitoring
perspective,
so
yeah.