►
Description
Panelist: Lenka Bočincová
Speakers: Eric D. Schabell, Goutham Veeramachaneni, Petr Stýblo, Miro Adamy, Zdeněk Váňa
Welcome to the panel discussion from Kubernetes Community Days Czech & Slovak 2023, recorded on Saturday, May 20th. This video captures an insightful dialogue between leading minds in the Kubernetes field, brought together to explore the complexities and nuances of Kubernetes technology. Engaging in a lively exchange of ideas, the panelists offer their unique perspectives, drawing upon years of experience and deep knowledge in their respective areas of expertise.
They delve into intricate technical aspects, offering clear explanations that make even complex concepts accessible. The panelists' broad range of views ensures that topics are examined from multiple angles, fostering a comprehensive understanding for viewers. With a collective commitment to quality speech and thoughtful analysis, this discussion promises to engage, educate, and inspire.
Whether you're new to Kubernetes or an experienced professional, you'll find invaluable insights in this panel discussion that will help you navigate the Kubernetes landscape. Watch and join in our journey of discovery and learning at the Kubernetes Community Days Czech & Slovak 2023.
A
I
think
we
can.
We
can
slowly
start
good
afternoon.
Everyone
Welcome
to
our
panel
discussion
about
observability
I,
think
you
noticed
that
the
observability
is
nowadays
a
big
topic
also
within
this
conference.
So
many
talks
on
on
it
and
I'm
really
like
that.
I
can
be
here
today
with
all
the
experts
who
had
also
talks
or
help
us
with
the
organization
and
be
part
of
this
this
panel,
and
before
we
start,
we
would
like
to
ask
you
a
few
questions,
maybe
so
how
many
of
you
are
here?
A
Okay,
perfect
and
so,
and
do
we
have
many
people
who
are
working
on
daily
basis
on
observability,
okay,
okay,
thank
you!
So
I
guess,
like
we
have
a
nice
crowd
here
so
and
before
we
start,
please
don't
be
shy
to
ask
questions.
We
have
a
slider,
so
I
will
try
to
check
it
but
put
questions
there
and
we
will
get
back
to
them
at
the
end
of
the
panel,
and
you
can
also
ask
questions
if
you
want
to
speak
up.
Okay
and
now.
A
B
I
work
for
company
po3,
which
you
probably
never
heard
about
it's
Canadian
company
I
work
for
the
Spanish
Office
of
the
for
the
European
office,
which
is
in
Spain
of
the
company.
We
do
many
things
and
I
particularly
work
in
the
infrastructure
and
Cloud
Department
I'm,
some
sort
of
like
devops
SRE
role,
the
you
can
find
us
on
the
web
at
pivot3.com
and
the
people
that
explain
much
better
than
I
would
what
we
do.
B
C
Hi
I'm
I'm
working
for
prusa
research
as
a
cloud
and
devops
team
lead
and
in
my
free
time,
I'm
organizing
the
community
in
the
Prague
I'm
in
the
cloud
native
Prague,
meetups,
and
also
this
event.
So
maybe
it's
the
reason
why
I'm
here,
because
one
of
my
responsibility
is
also
the
obso
about
this
Tech.
So
that's
it.
D
Hi
everyone,
I'm
Gautam
I
work
at
grafana
Labs,
but
I'm,
also
I
started
off
as
a
Prometheus
maintainer
like
six
years
ago,
worked
on
the
storage
engine
and
on
the
general
Prometheus
maintenance
for
a
while
and
then
on,
Loki,
Tempo
and
quite
recently
on
open
Telemetry,
but
nowadays
I'm
doing
rough
screens,
because
I
am
a
new
product
manager,
but
I
still
am
super
plugged
into
the
observability
side
of
things.
So
yeah
I.
That's
why
I'm
here.
E
F
A
Thank
you.
Thank
you.
I
love
you
so
I,
as
you
can
see
like.
We
have
a
really
group
of
people
who
are
part
of
different
companies,
so
maybe
the
the
like,
the
term
observability,
can
can
be
different
and
also
like.
We
have
people
in
the
audience
who
are
new
to
it.
So
can
some
of
you
maybe
give
us
some
ground
about
what
observability
means
and
how
it
fits
into
the
kubernetes
world.
So
so
we
have
like
common
ground
for
the
discussion
who
would
like
to
start
I'll.
F
Do
the
academic
part?
There
is
actually
a
definition
here,
so
if
you
can
look
it
up,
I
did
I.
Did
that
too.
So
it's
not
from
the
top
of
my
head,
but
actually
observability,
is,
is
defined
in
some
way.
I
think
it
has
been
defined
exposed
after
we
have
started
using
observability
and
people
trying
to
find
the
right
meaning.
F
There
are
some
defined
signals
that
we
that
we
commonly
agreed
on
Tracy's
metrics
logs,
but
there's
also
many
other
signals
that
have
not
made
it
to
the
observability
definition,
so
those
are
less
less
less
pronounced,
such
as
profiles,
there's
also
events.
This
is
this
is
where
I
come
from
event
management.
So
event
is
not
a
log
message
and
there
are
some
people
that
that
consider.
It
also.
E
Originally
we
started
out
with
you
know:
server
racks
data
centers
and
you
were
actually
monitoring
what
a
known
environment,
a
known
size,
very,
very
easy
to
Target
application
layers
and
all
this
different
stuff
that
you
were
doing,
whether
it's
a
system
or
not.
You
went
to
the
second
generation,
you
started
getting
VMS
and
it
gets
a
little
bit
more
complicated,
that's
kind
of
when
you
saw
everything
you're,
seeing
now
around
8
pm
and
and
those
kind
of
monitoring
platforms
looking
at
applications
and
stacks
and
messaging
layers,
and
things
like
that.
E
And
then
you
have
the
cloud
native
world
where
things
are
Auto
scaling
with
kubernetes
and
the
data
explosions.
That
you're
going
to
hear
us
talk
about
here
today
and
that's
that's.
That
requires
a
third
generation
of
of
monitoring
and
observability
and
that
that's
what
the
cloud
native
observability
platforms
are
that
you're
talking
and
hearing
about
today.
They
have
to
give
the
customers
the
ability
to
to
control
the
the
speed
with
which
things
are
happening,
the
the
cost.
E
D
However,
sorry
about
that,
it's
a
good
problem
to
have
I
have
a
lot
of
non-technical
friends
who
call
me
up
when
their
volume
stops
working
in
Windows
and
I
tell
them
to
restart,
and
they
ask
me
what
do
you
do
and
I'm
not
going
to
show
them
graphs,
but
like
I
kind
of
explain
that
you
know
I
kind
of
help,
people
understand
what
their
code
is
doing
and
this
could
be
any
code
and
it
could
be
any
form
of
understanding.
For
example,
people
say
logs
metrics
traces,
but
just
errors
and
error.
D
C
More
types
of
the
opposable
birthday
tools,
so
when
I
should
somehow
describe
to
my
child
what
the
absolute
is
so
I
just
mentioned
that
you
have
the
application
when
it
should
running
somewhere.
So
you
should
know
how
the
application
is
behavior
and
the
behavior
of
the
application,
and
not
only
the
app
but
also
about
the
platform
about
the
network.
C
So
I
have
to
collect
some
data
about
it
and
also
I
like
to
somehow
make
some
visualization
of
this
data
and
also
make
some
other
solution
that
I
wanted
to
know
that
something
is,
you
know,
I
see
a
different
behavior
of
the
trends
in
the
in
these
metrics
or
logs.
So
it's
just
about
the
tools
for
getting
these.
We
can
call
it
metrics
and
the
way
how
we
can
store
it
there
somewhere
and
how
we
can
make
some
decision
from
them.
B
Well,
only
maybe
two
things
that
has
been
described
pretty
thoroughly
and
I
I,
don't
really
have
to
add
too
much
to
it.
There's
one
aspect
that
wasn't
mentioned
is
that
when
you
talk
about
observability,
it's
not
only
about
the
tools,
because
at
the
end
there
is
some
kind
of
notification.
There
is
a
Opus
Genie
notification
there
or
page
page
activity
or
a
slack
message
or
something
whatever
you
have,
but
there's
also
important.
What
do
you
do
as
a
result?
B
A
Thank
you
Miro.
You
touched
maybe
on
the
on
the
next
question,
a
bit
because
like
for
me,
it's
easier
to
understand
when
I
hear
some
practical
examples
of
how
how
devops
teams
or
SRE
teams
or
whatever
you
call
it
nowadays
use
it.
So
can
you
give
me
some
more
practical
examples
of
observability
and
how
it
is
used?
Well,.
B
We
use
pretty
much
the
standard
tools,
then
we
use
Prometheus
and
kubernetes.
We
use
grafana,
and
but
we
also
still
have
some
data
centers
from
the
past,
which
we
are
in
process
of
migrating
to
Cloud.
Therefore,
we
inherited
our
own
share
of
some
older
tools
and
it's
it's
kind
of
challenging
to
streamline
those
this
data
and
make
sure
that
the
same
process
of
handling
these
notifications
can
be
used
when
it
comes
to
a
cloud
and
the
distance
Center,
because
usually
you
have
different
teams
involved
and
different
kind
of
skill
set
required
for
dealing
with
it.
B
So
I
I.
Don't
think
that
my
particular
experience
is
very
typical
for
the.
If
you
are
really
living
in
the
cloud,
then
the
the
Practical
example
is
well
pick
the
something
that
is
well
aligned
to
your
organizational
structure,
because
you
don't
want
to
just
install
chromito,
so
that
will
generate
like
lots
and
lots
of
information,
lots
of
alerts
you
need
also
to
Define,
which
of
these
are
important,
and
how
are
you
going
to
handle
them?
What
are
you
going
to
do
as
a
result
of
these
things?
B
If
you
don't
Define
that,
and
if
you
don't
clearly
separate
important
things
from
unimportant
things,
you
may
not
get
the
results
that
are
expected,
because
the
really
important
information
will
get
drowned
in
the
in
the
noise
of
those.
Those
monitoring
system
can
tell
you
a
lot
and
they
can
you
sometimes
tell
you
too
much
and
it's
equally
important
as
making
sure
you
pick
the
the
important
events
is
not
to
pick
the
events
that
are,
for
example,
in
kubernetes.
B
There
are
a
lot
of
things
that
are
self-healing
and
you
want
to
know
about
them,
but
you
don't
want
to
start
a
relatively
expensive
process
of
notifying
and
and
raising
people
during
off
hours.
Just
for
something
that
you
know
from
history
that
will
correct
within
like
two
minutes
or
one
minute,
if
it's
still
within
the
SLA,
that's
that's
perfectly
okay.
If
it
doesn't
fix,
then
you
want
to
to
start
the
whole
process.
So
that's
Eric.
A
E
E
What's
going
on
as
fast
as
possible,
you
prefer
to
be
worn
before
it
happens,
but,
okay,
maybe
it
happens,
and
then
you
get
warned,
but
you
know
as
quickly
as
possible,
you're
not
waiting
for
your
visualization
to
load,
because
there's
so
much
back
data
and
stuff
piled
up
that
you
can't
get
your
your
images
in
front
of
your
face.
You
want
to
be
able
to
triage
this
stuff
as
quickly
as
possible,
whether
that
means
remediate
or
whatever
it's
going
to
be.
E
You
want
to
be
able
to
take
action
and
take
a
sensible
action
and
you
want
to
be
able
to
go
back
and
understand
really
deeply
what
happened.
So
it
doesn't
happen
again.
You
can
take
measures,
so
we
talk
about
no
triage
and
understand.
That's
not
logs
metrics
tracing,
that's
not
digging
down
into
kubernetes
pods
and
all
this
other
kind
of
stuff.
It
doesn't
matter
where
it
came
from
where
that
information
presented
itself
as
long
as
you're
able
to
Corral
it
and
use
the
the
information.
D
D
For
me,
for
example,
I
could
look
at
a
slow
query,
get
the
trace,
ID
look
at
the
trace
and
I
could
understand.
Okay,
it
took
18
seconds
because
promptql
took
three
seconds
because
talking
to
S3
or
bigtable
took
five
seconds,
so
it
kind
of
helped
me
understand
what
was
slow.
I
mean
I
still
couldn't
optimize,
but
at
least
I
understood
what
was
slow
and
why
things
are
working
the
way
they
are.
What
are
these?
All
these
components
and
how
they
work
together,
yeah.
So
that's
one
practical
example
of
observability.
C
Okay,
I
can
give
you
some
examples
of
usage,
so
we
can
space
it
up
to
two
things.
One
one
is
the
platform
and
the
second
one
is
the
app
itself.
So
when
we
speak
about
the
the
platform
so
I'd
like
to
get
some,
you
know
I'm
running
the
kubernetes
cluster,
so
I
wanted
to
get
some
metrics,
so
I
can
use,
for
example,
node
node
exporter,
to
get
some
metrics
through
Primitives,
and
then
I
also
wanted
to
get
some
events
and
clocks
for
the
platform
itself
and
when
we
move
to
the
application.
C
So
when
I
run
the
application,
so
I
like
to
have
some
the
application
metrics
like,
for
example,
I
wanted
to
know
how
many
users
are
connected
to
my
website.
Right
I
also
wanted
to
collect
some
logs,
so
I
can
use
solution
like
Rocky,
for
example,
or
elk
stack
to
to
get
the
locks
yeah,
but
there's
also
the
second
type
of
the
logs
and
the
there's
the
exceptions
right.
C
So
we
wanted
to
know
once
there's
something
strange,
so
you
can
use
Solutions
like,
for
example,
Sentry
to
to
get
these
metrics
and
you
know
see
some
Trends
and
set
some
hours
about
it
yeah,
but
when
your
application,
sometimes
there
is
no
issue-
I
mean
from
the
logs
purpose
perspective,
but
you
can
see
that
your
application
is
slowing
down
yeah.
So
you
wanted
to
know
what
what's
happening
there.
So
the
thing
is
that
you
like
to
get
some
traces
I
mean
the
analysis
of
the
whole
flow.
C
What
would
happen
when
you
call,
for
example,
exact
endpoint?
So
you
should
have
some.
You
know
tools
like
Jagger
or
Tempo
to
collect
these
metrics,
but
you
have
to
also
make
them
together,
yeah,
because
if
you
have
only
one
of
the
type
of
these
events,
so
it's
useless.
So
we
have
to
somehow
make
some
visualization,
for
example,
via
grafana,
and
then
we
have
to
set
some
other
thing
and
then
you
can
use,
for
example,
the
manager
from
the
permative
stack.
C
But
the
last
thing
is
that
you
for
this
all
solutions.
You
have
to
use
a
different
tool
and
it's
quite
common
that
you
store
it
in
different
tools.
So
maybe
you
should
realize
that
you'd
like
to
have
also
higher
a
bit
more
and
also
some
one
store
for
all
of
these
types
yeah,
because
if
we
speak
about
the
metrics
and
The
Primitives,
so
you
can
use
tools
like
Tunnels
for
the
high
availability
mode
yeah.
C
A
Thank
you.
You
ended
the
question
right
away
because
I
wanted
to
ask
about
this.
So
so
you
all
mention
different
tools
within
the
cncf
landscape.
There
is
a
thousands
of
them
and
it's
so
difficult
to
you
know,
figure
out
which
to
use,
and
maybe
Eric
or
Eric
mentioned
that
sometimes
the
technology
is
not
something
that
you
should
put
like
front
when
you
are
implementing
it.
So
what
criteria
the
the
people
should
should
have
when
they
are
thinking
about
implementing
it.
F
F
There,
let
me
try
I,
know
I'll,
give
it
a
try
if,
if,
if
it's
wrong,
somebody
else
will
take
over,
but
my
understanding,
if
the
question
is
right
and
whoever
raised
the
question,
feel
free
to
correct
me.
But
if
you
mean
like
is
observability
like
the
tooling,
the
instrumentation
slowing
down,
basically
kubernetes
making
the
less
performant
is.
That
is,
that
is
the
question
about
how
do
we
make
I
came
online
or
okay
so
anyway?
F
If,
if
that's
the
question,
I
think
it
has
been
already
asked
in
some
previous
sessions,
it's
really
a
matter
of
choice
like
what
you
want
to
observe
and
it
you
know
we
can
think
of
it.
As
you
know,
in
from
many
angles,
but
one
first
angle
would
be
don't
overdo
it
like
there
is
no,
no
observability
that
will
get
you
everything
from
a
single
Source
or
you
know
a
collect
all
agent.
There
may
be
some
efforts
to
do
that.
Even
you
know
mentioning
open
Telemetry.
F
The
ambition
is,
is
really
huge
like
to
to
to
to
to
provide.
You
know
a
you
know
a
solution
for
all,
but
it
will
never
be
the
case
because,
yes,
you
will
kill
the
the
system
just
by
observing
it.
So
the
right
thing
as
I
think
the
right
approach,
the
reasonable
approach-
and
this
this
has
been
proved
like
in
the
history.
You
need
to
be
selective
at
the
at
the
beginning.
What
what
do
you
need?
What
are
you
interested
to
see?
What's
your
business
reason
to
to
do
the
observability?
F
What
are
the
outcomes
that
you
want
to
get
out
of
observability
and
collect
just
the
data
that
you
need
start
start
low,
observe
the
system
how
it
performs?
If
you
can
do
more,
if
you
need
more,
you
know
erase
your
granularity
and
proceed
gradually,
simply
so
otherwise.
Otherwise,
of
course,
there
will
not
be
yeah
the
observability
itself,
I
mean
it
can
be.
It
can
be
a
pretty
pretty
intensive
process
so
be
cautious
on
on
choosing
the
tooling
and
choosing
the
amount
of
data.
B
If
I
may
add,
I
would
emphasize
different
aspects
of
that.
It's
not
only
performance
and
performance
is
usually
not
a
big
deal.
It's
kind
of
a
few
percent
of
the
the
whole
output
it
it
can
get
worse
if
you
go
overboard
and
try
to
post
process
those
things
and
Aggregate
and
summarize
and
predict
and
do
all
all
a
kind
of
complicated
processing
on
it.
B
They
are
fairly
lightweight,
but
they
quite
often
are
responsible
for
a
lot
of
more
observability
know
is
kind
of
the
observability
system
is
observing
itself
and
Reporting
on
itself,
a
lot
which
actually
defeats
the
purpose.
You
want
to
be
notified
first
about
the
business
law.
It's
not
about
the
that
this
part
from
observability
cannot
talk
to
that
other
part
from
observability,
which
is
usually
subscribe
itself.
So
this
is
the
kind
of
weird
effect
of
if
you
put
a
lot
of
up
Tooling
in
place,
that
tooling
will
keep
itself
busy
for
a
considerable
amount
of
time.
F
One
more
common
is
like
performance.
I
know
it
has
been
related
to
kubernetes,
but
you
can
be
more,
you
can
Mark
make
it
more
generic.
The
thinking
like
the
performance
is
not
just
the
performance
of
the
system
itself,
but
you
can
think
of
business
performance.
You
can
think
costs.
You
know
those
are
things
that
you
need
to
take
into
consideration,
especially
with
tracing
I'm
sure
this
is
the
case
with,
with
with
metrics
cost
of
storage
are
becoming.
You
know
a
differentiator
at
some
point.
F
E
I
also
think,
that's
a
reason,
you're
seeing
quite
a
bit
of
the
larger
users
of
these
kind
of
platforms,
Outsourcing
this
to
a
SAS
provider,
which
is
what
we
also
do
together
with
the
grafana
labs.
Does
the
same
thing
there's
a
real
reason
for
this.
We
did
an
observability
report
in
2023
and
I
mentioned
this.
E
Managing
the
infrastructure
and
the
sres
that
put
their
hands
up
can
tell
you
all
about
this.
It
becomes
a
full-time
effort
and
out
of
that
research
report
which
I
linked
to
in
my
presentation.
If
you
want
to
go
back
and
look
at
it,
one
of
the
most
shocking
things
for
me
is
is
25
of
their
time.
These
Engineers
is
being
spent
on
issues
and
triaging
problems.
E
So
if
you're
spending
you
know
10
hours
a
week
doing
this
are
you
happy?
Is
that
what
you
signed
up
for
is
that?
Is
that
really
what
you
enjoy
doing
and
really
quickly?
You
see
when
it
gets
to
be
a
large-scale
operation?
One
of
our
customers,
for
example
during
the
the
pandemic,
doordash
delivery
of
food
right.
So
when
everybody
got
locked
down
in
the
pandemic,
they
started
delivering
food
like
crazy,
and
this
scaled
up
so
fast
and
so
hard.
It
became
more
than
a
full-time
job
right.
E
You
need
to
Outsource
that
kind
of
infrastructure
and
get
back
to
doing
your
core
business
to
make
make
the
money
where
you
want
to
make
it
and
getting
people
off
of
10
hours
a
week
of
triaging
issues
and
and
which
leads
to
nothing
more
than
a
bunch
of
statistics
about
burnout
and
why
people
don't
want
to
don't
want
to
work
there
anymore,
and
this
is
this-
is
not
how
you
want
to
live
your
life.
This
is
not
how
you
want
to
retain
your
talent
and
your
talent,
you're,
actually
hired
for
something
else.
D
There
was
another
interesting
statistic
in
his
talk:
it's
it's
a
it's
quite
common
that
the
cost
of
observability
is
more
than
the
cost
of
infrastructure.
So
I
have
a
question
for
you.
How
much
do
you
think
observability
should
cost
so,
if
you're
paying
a
thousand
dollars
for
infrastructure?
How
much
are
you
okay,
paying
for
observability?
Is
it
five
dollars?
Is
it
five
hundred
dollars?
Is
it
five
thousand
dollars
yeah?
D
No,
it's
it
it.
In
the
end.
It
comes
down
to
the
business
question.
You
first
need
to
understand.
Okay,
what
are
the
questions?
I
would
like
to
like
to
answer
like
to
ask
and
how
quickly
do
I
need
them
answered,
and
some
people
would
be
like
I
need
to
answer
all
the
questions
in
a
millisecond,
and
that
comes
with
the
business
trade-off
that
this
is
going
to
be
ridiculously
expensive.
So
you
need
to
kind
of
make
the
while
choosing
the
tools,
while
choosing
the
providers
and
stuff
you
need
to
make
understand.
D
Ten
percent
I'll
probably
go
for
one,
but
like
it's
a
business
to
value
question
when
you're
picking
tools
and
that's
kind
of
how
I
would
evaluate
like
do,
I
really
need
ebpf
profiles.
Probably
not
would
it
be
good
to
have
definitely
would
I
pay
for
them
now
and
questions
like
this
when
you're
picking
tools,
yeah.
D
I
mean
yes,
there's
always
challenges,
you
could
always
be
too
good
and
write
a
bunch
of
alerts
and
then
you'll
developers
are
getting
paged
all
the
time.
That's
a
very,
very
common
challenge,
too
many
alerts,
and
then
you
black
hole
all
of
them
to
with
an
email
filter,
and
then
you
sleep
peacefully
through
the
night.
Your
customers
don't,
but
that's
a
different
story.
Yeah
the
two
challenges
I
see
with
implementing
observability
I
mean
three
I
would
say:
is
one
cost?
D
B
Just
to
add
to
that
this,
the
cost
of
storing
data,
it's
important
things,
but
especially
with
traces.
There
is
this
retention
window
that
is
kind
of
exp
important.
You
don't
really
need
traces,
except
when
you
need
them
as
as
a
result
of
some
event.
It's
like
the
rule
that
five
five
hours
before
each
fire,
the
fire
department,
needs
to
inspect
the
state
of
the
of
those
machines
and
so
on.
So
essentially,
you
need
to
establish
reasonable
trade-off
between
the
the
cost
you
are
going
to
invest
on
ongoing
basis
and
how?
B
How
much
you
aggregate
this
information.
There
is
one
window
where
you
want
to
keep
the
traces
all
of
them
for
certain
amount
of
days
and
that's
after
the
new
release,
because
when
you
are
releasing
new
version
of
the
software,
it
doesn't
have
to
be
a
problem
with
your
software.
It
can
be
problem
with
interaction
with
environment,
with
some
hidden
assumption
and
in
these
cases
the
the
traces
are
absolutely
Priceless.
So
this
this
can
mean
days
in
terms
of
being
able
to
react
quickly
and
fix
the
issue.
That's
that's
the.
E
E
So
that's
like
really
rich
people
digging
into
their
pocket
and
what
do
they
pull
out?
It's
not
the
same
amount
of
change,
I
pull
out
of
my
pocket
right
and
for
them,
65
million
at
the
time,
was
definitely
Pocket
Change.
You
know
they
were
making
oodles
and
noodles
more
than
that,
so
they
didn't
really
care,
and
that
goes
around
back
to
what
he
was
saying.
Observability
costs,
tooling
all
the
stuff
you're
using
nobody's
going
to
care
at
all
if
you're
making
money.
E
If
customers
are
having
good
experience,
if
you're
getting
done
what
you
want
to
get
done
and
everybody's
able
to
discover
where
the
problems
are
and
fix
them
in
time
right,
we
will
pay
anything.
It's
like
Super
Bowl
ads.
Why
do
people
pay
that
much
money
for
30
seconds
on
TV
because
it
achieves
what
they
want
to
achieve?
They
don't
care
what
it
costs
the
people
that
advertise
it
just
don't
care
right.
They
can
afford
that.
E
This
is
the
problem
with.
When
you
look
at
this
space,
you
can
easily
get
just
swamped
with
what
happens
in
the
bill
where
it
just
surpasses
everything
you're
doing
normally
in
your
business.
But
if
that's
acceptable
to
you,
fine
nobody's
going
to
make
a
noise
or
make
a
problem
out
of
this
or
change
anything
that
they're
doing.
E
But
as
soon
as
you
stop
making
enough
money
on
this
side,
then
this
is
going
to
become
a
huge
problem
right
and
that's
going
to
be
the
first
thing
they
start
looking
at
I
mean
it's
full
stop
I
think
it
was
a.
There
was
a
pretty
neat
statistic
last
year
right
towards
the
end
where
I
was
watching
a
cloud
report
and
an
analyst
was
talking
about
how
in
their
research
they
found
out
that
83
of
I
think
it
was
83.
E
Maybe
it's
give
or
take
a
few
percent,
but
it
was
in
the
80s
would
have
organizations
would
have
a
financial
operations,
a
fin
Ops
position
by
the
end
of
2023.
and
that's
pretty
crazy
right.
That's
who's.
They
can't
figure
out.
The
bottom
line
is
basically
your
CIO
I
think
owns
or
your
CTO
owns
the
budget
right.
That's
the
end
responsibility
for
what
you're
doing
with
your
platforms.
E
The
problem
is,
you
have
developers
sneaking
stuff
in,
you
have
Engineers
sneaking
stuff
in,
and
nobody
really
knows
who
pushed
that
button
in
that
cloud
providers.
Marketplace
to
add
this
new
load
balancer:
that's
not
adding
tags
and
Metric
cardinality!
That's
just
blowing
up
your
your
budget
right.
You
need
to
have
a
position
that
that
fills
that
space,
that
that
quantifies,
what's
going
on,
puts
processes
in
place
all
that
kind
of
stuff,
apparently
within
the
next
couple
of
months,
everybody's
going
to
have
one
in
Europe
anyway.
C
C
And
the
second
thing
is
how
long
you
wanted
to
store
them
yeah,
because
so
many
companies
tries
to
keep
their
locks
for
almost
a
year,
but
just
realize
that
when
you
last
time
when
you
look
at
through
more
than
maybe
one
month
or
other
walks
yeah,
so
why?
Why
are
restoring
them?
So
you
pay
so
so
lot
and
you
have
to
find
a
different
tooling,
because
when
once
you
have
these
locks,
so
it's
quite
hard
to
find
them
because
it's
it
takes
time.
C
So
maybe
the
challenge
is
to
find
some
agreement,
I
mean
with
the
developers
and
the
management
about
it.
What,
and
why
are
you
storing
this
data
yeah,
the
Second
Challenge?
Is
we
have
a
lot
of
tools
for
you
know,
making
the
observability,
but
still
it's
quite
hard
to
put
that
together,
yeah
so
yeah?
You
can
collect
logs
metrics
traces,
but
there
is
no
easy
way
how
you
can
see
on
the
one
panel,
all
these
data
and
you
can
easily
to
find
the
root
cause
of
the
issue
so
yeah.
A
D
D
Can
we
blame
observability
I
hope
not
because,
but
it's
typically
process
I
would
say.
For
example,
in
kubernetes
we
had
a
big
outage
where
all
our
pods
were
suddenly
being
killed
by
kubernetes,
and
nobody
could
figure
out
why
for
a
long
time-
and
it
turns
out
we
configured
health
checks,
but
we
never
activated
the
health
check
in
the
applications.
So
communities
was
thinking.
This
pod
was
unhealthy
and
it
was
killing
it
regularly
and
yeah,
like
we
kind
of
weren't
prepared
for
that.
E
E
That
requires
quite
a
few
things
to
happen
and,
and
you
know
to
be
really
good
at
what
you're
doing
I,
don't
think
it's
it's
it's
fair
to
to
really
pick
on
that
specific
outage,
especially
since
the
complete
report
hasn't
been
put
out
about
the
incident.
You
know
they
put
something
out,
but
not
not
all
of
it
yet
so
you
kind
of
want
to
wait
and
find
out
exactly
what
happened
and
I
mean
those
are
the
most
interesting
things
to
read
right
because
everybody
can
learn
from
those
things.
E
So
we
hope
everybody's
going
to
be
open
and
honest
about
what
happened.
That's
that's!
What
you're,
looking
for
and
I
think
something
seriously
to
evaluate
when
you,
when
you
look
at
companies
that
are
that
are
providing
these
kind
of
services
for
you.
How
do
they
handle
this
stuff
in
the
past?
How
open
are
they
with
you
or
they
just
do
that
with
select
customers,
you
know:
do
you
have
to
have
a
certain
tier
before
you
get
good
response
times
and
things
like
that
I
think.
E
F
Just
on
that,
of
course,
of
course,
I
don't
have
an
example,
because
I
work
for
IBM,
it
doesn't
happen
to
to
IBM
no
but
seriously,
I
think
this.
This
is
this
is
referring
to
the
point
that
we
heard
in
the
beginning,
there's
a
lot
of
things,
but
that
we
don't
learn
that
we
don't
know
from
from
the
Press
because
of
observability.
You
know
how
many
incidents
have
been
avoided
by
by
thanks
to
thanks
to
having
right
tools
right,
right
processes,
right
observability
in
place.
F
You
will
never
hear
the
good
news.
This
is
actually
no
news,
I
mean
you
will
not
hear
it
in
the
news
right.
Nobody
writes
about
things,
but
those
are
the
those
are.
Those
are
really
the
the
stories
that
we
should
hear
like
how
many,
how
many
things
they
they
managed
to
avoid.
This
is
a
single
tone.
This
is
a
disaster
that
happens
to
everybody.
This
will
happen
to
everyone
and
yeah.
It's
it's.
It's!
It's
very
sticky
with
the
Press.
A
D
We
do
have
some
machine
learning,
but
we
try
to
not
make
make
it
automatic
like
we
do
have
a
product
where
you
can
say:
hey
run
these
algorithms
on
these
metrics
and
kind
of
show
the
trends.
And
then
you
can
alert
on
the
trends
and
the
deviations
from
the
trends.
But
we
try
to
make
this
manual
so
that
developers
think
and
know
what
they're
doing
when
they
set
up.
That
alert.
D
I
think
it's
really
hard
for
machine
learning
to
get
all
the
edge
cases
correctly
and
you
might
end
up
in
a
case,
it's
very
common
that
you
miss
the
important
alerts
or
you
get
a
lot
of
alerts.
So
you
black
hole
everything
and
then
you
miss
everything
so
yeah
we
we
do
some
machine
learning,
but
we
haven't
found
like
a
Magic,
Bullet
I,
don't
think
it's
possible
to
find
a
Magic
Bullet
with
our
complicated
architectures
and
applications.
D
F
Yeah,
well,
we
do
have
a
tool
in
in
our
portfolio.
I,
don't
want
to
talk
about
that
tool,
but
yeah
I
think
this
is
really
the
the
the
the
future
that
we
are
going
to
look
at
in
observability
the
machine
learning
some
some
call
it
AI,
because
this
is
really
about
searching
the
needle
in
the
haystack.
F
You
know
you
cannot
really
pick
everything
from
the
screens,
no
matter
how
well
Consolidated
how
well
correlated
the
the
stuff
is
simply
there's
so
many
things
happening
and
there's
so
many
symptoms
associated
with
it
with
with
a
single
failure.
So
the
root
cause
is
perhaps
one
but
there's
lots
of
indicators
on
that.
So
we
definitely
need
some
some
AI,
some
machine
learning.
I
agree.
It
is
a
complex
area
we
do
we
do.
F
We
do
have
solutions
for
for
machine
learning
for
unlocks
also
on
on
metrics,
but
this
is
pretty
pretty
hard
yeah.
This
is
really
rocket
science
almost
and
yeah,
and
it
depends
on
actually
on
the
customer,
their
use
cases.
So
no
matter
you
know
we
may
develop.
We
do
have
generic
models.
For
you
know
many
typical
scenarios,
typical
situations,
but
still
you
will
have
lots
of
lots
of
customer
specific
cases,
so
basically
I
think
we
are
looking
at
some
tooling
that
will
provide
some
general
purpose.
F
B
If
I
may
I'm
not
an
AI
expert,
but
we
are
having
this
active
discussions
on
on
this
theme
about
how
to
connect
these
areas-
and
we
see
a
couple
of
challenges
and
one
of
the
challenges
is
a
problem
of
training,
so
you
can
have
it
enough
training
data
from
the
system
you
are
running
but
though
that
data
needs
to
be
the
the
you
need
to
create
a
trained
model,
and
you
know,
in
order
for
the
model
to
be
good,
you
don't
need.
You
need
to
look
back
on
certain
amount
of
time.
B
So
therefore,
that
means
that
you
need
to
train
that,
for
with
the
data
that
are
going
long
back
in
the
in
the
history,
which
means
you
are
supposed
to
be
spending
a
lot
of
money
on
that
training
and
it's
kind
of
hard
to
justify
the
expense
of
running
the
continuous
trainings,
potentially
very
expensive.
Big
machines
versus
the
assumed
benefit
of
the
thing.
So
there
is,
it
is
kind
of
I
would
say
that
at
this
stage
it
is
a
excellent
solution.
Looking
for
a
good
problem,
so.
D
I
also
just
want
to
add
to
his
point,
I
think:
there's
a
lot
of
value,
AI
can
add
I
just
wouldn't
run
your
primary
alerting
on
top
of
AI
I
would
still
run
primary
alerting
on
top
of
slos
and
once
you
get
alerted,
then
maybe
you
can
look
at
AI
for
possible
courses
to
this.
That
can
maybe
then
start
to
okay.
What
is
the
anomaly
compared
to
last
week,
and
why
did
this
alert
fire?
D
That
could
be
something
that's
useful
and
you
might
not
even
need
AI,
but
it
could
look
AI
like,
for
example,
all
you
need
is
like
a
history
of
all
the
alerts
that
fired
all
the
incidents
that
happened
and
when
alert
fires.
You
could
basically
look
through
that
history
and
be
like
the
last
time
this
alert
fired.
These
incidents
were
declared
and
these
were
the
root
causes.
Can
you
check
these?
D
E
Well,
I
touched
a
little
bit
on
it
with
a
stat,
but
there's
quite
a
bit
of
stuff.
That's
come
out
in
the
last
six
months
around
this
and
there
are
organizations
you
could
find
online
like
LinkedIn
and
stuff
that
have
fin
Ops.
You
know
like
practices
and
things
like
that.
I
know
in
North,
America
they're,
quite
heavy
in
that
I
was
reading
an
article
yesterday
about
someone
that
was
interacting
with
Google
and
just
launched
something
around
some
fin
Ops
programs.
D
F
So
you
can
definitely
leverage
observability
to
give
you
the
inputs
to
correctly
size
your
infrastructure,
your
virtual
machines,
your
ports,
your
kubernetes
clusters,
Etc,
because
you
probably
know
it
better
than
me
and
that
it's
a
it's
always
a
problem
to
to
make
to
size
the
cluster
to
size
to
size.
The
pods
right.
You
never
know
upfront
in
advance.
F
What
will
be
the
correct
sizing
so
observability
there's
a
very,
very
tight
connection
to
the
finops
aspect
of
it,
like
managing
costs
by
right
sizing,
the
workloads
so
that
you
don't
overdo
it
and
also
don't
you
know
underdo
it
so
that
you
have
underperformance.
So
you
don't
want
to
spend
more
than
you
have
to.
C
C
So
there
are
some
solution,
for
example
by
the
Google
yeah
and
it
works
somehow
could
accept
PVCs,
of
course,
but
you
know
it's
not
just
about
the
kubernetes,
because
you
wanted
to
calculate
the
whole
infrastructure
and
also
maybe
split
it
by
the
application,
but
sometimes
it's
if
we
are
in
the
kubernetes
space
right.
So
the
application
doesn't
mean
that
it's
run
on
a
on
one
namespace,
so
how
you
will
calculate
it,
because
these
tools
are
based,
for
example,
by
the
namespace
or
some
text
right
so
yeah
the
current
solution.
C
B
I'm
not
directly
involved
with
that
group,
but
what
I
know
is
that
we
are
using
predominantly
this
solution
and
the
partner
solutions
for
that.
It's
something
like
Cloud
Checker.
That
gives
you
information
about
which
parts
of
the
which
accounts
are
underutilized
and
also
now
the
kubernetes
is
deploying
some
solutions
by
they're
using
spot,
which
is
essentially
trying
to
utilize
the
support
capacity
of
Amazon
Cloud.
These
are
the
two
that
that
run
on
the
top
of
my
mind.
B
But
generally,
if
you
are
running
kubernetes
in
a
multi-tenant
mode,
then
cost
distribution
is
a
non-trivial
problem,
because
this
is
it
is.
If
you
have
everything
split
by
the
namespaces,
then
it's
relatively
easy
most
of
the
tool
gives
you
a
cost
allocation
based
on
namespaces,
but
they,
if
you
are
sharing
infrastructure
for
multiple
tenants,
then
it
is
none,
not
an
obvious
which
portion
of
the
cost
belongs
to
room
over.
Obviously
you
could
run
one
kubernetes
cluster
per
customer,
or
even
one
Amazon
account
per
customer,
which
makes
it
extremely
easy
to
calculate.
E
Can
we
we
actually
have
a
talk
that
we
shared
it's
called
single
pane
of
glass
and
they
spell
pane,
pay
p-a-I-n
right,
I!
Think
when
you
look
at
an
organization
the
complexities
that
you're
dealing
with
across
the
organization
depending
upon
who
you
ask
in
your
organization,
what's
important
to
them
is
pretty
much
representative
of
the
view
they
want
to
have
of
what's
happening
with
the
business
right
and
so
I.
D
D
This
is
basically
take
all
the
communities
logs
pipe
them
into
a
file
run,
a
script
which
gets
the
trace,
IDs
and
the
slow
queries
and
all
of
that
into
a
Google
spreadsheet
and
then
says
then
sort
the
Google
spreadsheet
by
the
slowest
queries
and
then
look
at
the
trace,
IDs
and
then
open
them
in
Jaeger
UI.
Now
you
can
imagine
how
painful
that
was,
but
before
that
we
had
nothing.
D
Okay,
people
want
metrics
to
logs
to
traces
in
the
same
in
in
an
easy
way
to
jump
between
the
three,
because
they
give
you
three
different
kinds
of
insights
that
you
will
be
using
together
to
solve
problems,
and
there
are
many
many
tools
that
kind
of
give
all
that
insights
in
one
place
like
if
you
take
data
dog
or
New
Relic
and
other
observability
SAS
vendors,
all
of
them
are
in
the
same
place
and
you
are
able
to
jump
from
A
to
B
to
C
and
I.
Think
that's
quite
important.
D
You
can
obviously,
if
you
want
to
do
it
with
open
source.
You
can
do
that
with
grafana,
and
this
is
also
something
that
we
are
seeing
as
Eric
said
like.
If
you
take
a
large
organization,
it's
really
hard
to
get
all
the
data
in
one
place,
there
will
be
five
different
monitoring.
Vendors,
five,
different
logging,
vendors
and
getting
trying
to
get
them
into
the
same
place
is
really
hard
and
trying
to
do
that
longer
term.
D
F
Again,
a
quick
one
for
me:
I
agree,
you
know
everything
screens
are
important
and
I
think
this
is
also
a
a
place
for
the
future
consolidation
of
the
tools.
So
again,
if
you
are
talking
about
the
future
of
observability,
this
is
this
is
the
area
that
will
take
place.
Inevitably,
another
comment
regarding
the
the
the
availability
of
data.
I.
Think
it's
really
crucial.
You
know
you,
you
can
have
good
screens,
but
is
it
really
that
you're
looking
at
the
screens
all
the
time?
F
Would
you
wouldn't
you
rather
prefer
to
have
you
know
the
right
information
at
the
right
time,
where
you
need
it
at
the
right
context?
So
first
thing,
you
know
I.
Think
users
will
appreciate
tools
that
give
them
the
context
at
you
know
not
necessarily
by
looking
you
know
at
craft,
but
maybe
maybe
clicking
in
the
in
the
right
place
in
the
graph
topology,
showing
things
in
context
and
clicking.
You
know
drilling
down
in
a
connected
way.
B
If
I
may
add,
the
I
wouldn't
see
it
as
a
having
unified
view
from
an
operations
perspective,
because
if
the
action
of
that
of
the
result
of
the
having
that
view
is
actionability,
then
with
a
large
enough
organization,
you
have
always
different
teams
for
different
systems
with
different
skills,
and
the
team
that
is
managing
your
Azure
cloud
is:
doesn't
actually
care
about
any
alerts
from
Amazon
and
vice
versa.
So
this
is
just
Destruction
for
them.
B
So
having
unified
view
is
actually
worth
what
you
may
want
as
a
unified
are
some
sort
of
reports
for
a
decision
makers
that
are
not
as
essentially
to
be
doing
something
about
that
result.
But
using
that
data
in
order
to
plan
and
in
order
to
to
make
a
decision
so
for
like
decision
support
system
and
in
this
case
it's
important
to
have
unified
view,
but
it
can
be
done
significantly
cheaper
and
significantly
less
obstrusive
way
as
to
for
towards
the
whole
organization
to
use
the
same
tool.
C
I
saw
it
so
many
times,
so
the
thing
is
that
one
way
is
just
force
them
to
use
some
such
solution
like
retro,
New,
Relic
and
stuff,
like
that
I
have
some
experience
with
that
and
I
understand
that
it's
you
for
someone,
but
when
you
wanted
to
do
this,
you
know
modification.
So
maybe
you
realize
that
it's
not
the
way
or
the
other
thing
is
that
the
Window
Rock
is
also
not
the
best
way.
C
A
E
E
What
else
would
be
good
choosing
as
many
tools
as
you
can
come
up
with
that
they'll?
Let
you
buy
now,
but
the
the
all
the
joking
aside,
I
think
the
the
the
worst
practices
that
are
going
on
right
now
and
what
what
people
fail
to
understand
is
the
amount
of
data
that
is
flooding
into
your
organizations
being
collected.
It's
being
stored
is
what's
running.
Those
costs
up
right,
we've
already
covered
that,
but
being
able
to
find
a
way
to
Corral
that
is
is
is
immensely
difficult.
E
E
The
second
thing
is
I
think
it's
really
important
not
to
get
too
fixated
on
tools
and
all
this
other
stuff.
It's
it's
mostly
about
trying
to
to
figure
out
what
your
business
is
doing.
What
you
you
know
when
you
design
your
visualization
whoever's
the
business
behind
you
having
to
design
that
visualization
to
keep
track
of
what
they're
doing
they
know
best,
what
they
want,
what
they
need.
E
You
just
have
to
have
the
ability
to
to
narrow
down
and
only
use
what
they
need
and
not
pay
for
one
penny
more
or
one
label
more
or
one
sub-dimension
more
of
any
of
your
metrics
right.
That's
that's
such
a
strict
policy.
You
need
to
you
need
to
get
set
up
as
soon
as
you
can,
because
it'll
snowball
out
of
control
in
no
time
and
it
just
it
just
gets
exponentially
worse.
When
you
get
successful
in
a
cloud
native
space.
F
F
If
you
approach
to
replace
an
existing
monitoring
solution
which
has
been
working
for
you
for
ages,
with
some
sexy
new
observability
platform
with
just
tracing
all
the
metrics
and
actually
basically
have
to
force
the
customer
to
redo
all
their,
you
know
all
the
setup
in
order
to
to
be
able
to
consume
Prometheus,
metrics
and
so
on.
So
I've
seen
I've.
Seen
enough
such
attempts
I
I
concerned
really
that
I've
seen
like
disasters
happening
I.
B
I
would
add
to
these
not
so
brilliant
ideas.
Kind
of
one
idea
is
to
using
multiple
delivery
channels
for
notification,
like
you
post
message
into
multiple
slack
channels
and
send
an
Ops
Genie
notification
and
an
email
just
in
case,
which
will
essentially
make
sure
that
your
team
will
be
dealing
with
the
same
thing
three
or
more
times
and
lose
a
lot
of
times.
This
is
kind
of
one
of
those
ideas.
B
The
other
is
that
you
define
the
very
loose
response
groups
or
escalation
levels,
which
means
that
everybody
knows
that
everybody
else
has
also
got
this
notification,
and
everybody
assumed
that
somebody
will
take
care
of
it
so
when,
in
which
case,
you
will
end
up
with
much
longer
response
time
as
if
you
know
that
well,
it
is
my
responsibility
this
today
and
if
the
next
escalation
is
this
guy
and
this
guy
and
you
that
this
way
things
get
will
happen.
But
there
is
a
large
group
supposed
to
be
taking.
B
C
See
two
of
them
are
hurts
that
you
know
it
depends
that
the
developers
and
also
obscene
just
stop
focusing
on
it
yeah,
because
they
are
that
okay,
there
is
something
I
know
it,
and
it's
quite
common.
So,
and
the
second
thing
is
what
I
mentioned
in
the
beginning
of
the
the
talk
is
storing
data
which
you
never
used
and
how
the
store
a
Precision
of
the
data,
which
is
not
necessary.
For
example,
if
you
collect
every
second
I
mean
geometric,
so
maybe
it
doesn't
make
sense
for
all
of
your
metrics.
D
A
couple
of
things
that
I
see
in
large
organizations
is
like
centralized
firefighting
teams,
so
you
write
an
alert,
but
the
alert
gets
routed
to
a
central
team
and
then
they
triage
it
and
then
they
assign
it
to
the
right
team.
That
is
just
a
bad
position
to
be
in,
but
it's
also
quite
quite
common.
This
is
what
SRE
teams
are
for.
Are
there
for
in
some
companies
the
other
issues
I
see.
Is
you
have
the
data,
but
you
don't
have
access
to
the
data.
D
For
example,
you
could
be
on
call
and
your
data
could
be
in
Splunk,
but
you
don't
have
access
to
Splunk.
You
need
to
talk
to
a
different
team
to
get
access
to
Splunk
for
an
hour,
and
this
is
also
quite
common
yeah.
So
just
make
sure
if
you
write
an
alert,
it
gets
to
you
and
not
someone
else
and
if
you're
on
call
you
have
if
you're
collecting
data
you
have
access
to
the
data
at
the
right
time
at
the
at
the
right
place.
Yeah.
E
That
just
triggered
something
really
quickly,
I
wanted
to
mention,
too,
that
I
saw
last
that
I
was
pretty
impressed
with
when
I
saw
it
was
designing
dashboards
to
include
all
the
links
and
help
and
stuff
you
need
from
that
point.
So,
no
matter
how
experienced
you
are
you're
being
led
down
the
same
path
as
an
experienced
person
would
do
so.
A
D
Yes,
it's
called
an
APM.
This
is
like
before
distributed
systems,
Cloud
native
whatever
tracing
existed
in
the
form
of
APM
it
traces.
Like
all
your
fun.
All
the
important
function
calls
how
long
it's
a
request
took
in
each
function,
call
and
I
think
that's
highly
useful
information.
If
a
request
is
taking
five
seconds,
you
want
to
know
where
are
these
five
seconds
happening
and
I
think
traces
are
a
really
really
good
way
of
answering
that
and
yeah
it?
It's
quite
useful.
B
There
is,
there
is
also
one
aspect
of
monolith:
that
okay,
let's
assume
that
your
traces
will
reveal
something
now
for
youth
in
order
to
fix
it
in
code
means
to
do
a
full
release,
but
with
the
micro
Services,
you
can
do
this
kind
of
you
replace
just
one
micro
service
and
there
is
much
less
this
disturbance.
So
essentially
the
your
ability
to
act
upon
discovery
of
something
important
is
very
different
from
this
from
the
micro
services.
B
So
I
would
say
it
is
also
important
to
put
traces
around
the
monolith,
because
quite
often
the
situation
is
not
that
the
monolith
has
a
problem,
but
something
that
is
around
it.
Something
in
the
environment
changed
and
suddenly
the
communication
doesn't
work
and
if
you
are
tracing,
if
you
are
putting
traces
around
on
the
on
all
the
end,
points
that
the
monolith
talks
to
you
may
find
out
that
all
you
have
to
do
is
just
simply
to
restart
or
restart
one
of
the
services,
and
you
are
fine.
F
F
One
thing
regarding
microservices,
you
know
and
and
and
tracing
that
we
see
today
there
have
been
you
know,
architectures
before
and
I
think
we
still
call
them
monoliths.
Those
were
called
in
the
past
like
distributed
architectures
like
SOA
service,
oriented,
architectures
I
think
they
still
qualify
as
monolith
in
a
way
and
what
I
want
to
say
is
that
there
was
tracing
before
we
didn't
call
it
tracing.
We
call
it
the
distributed
monitoring
which
evolved
into
into
tracing
that
we
know
today.
So
this
is
not
a
New
Concept.
F
So
if
you,
you
know,
if
you
have
in
mind
like
those
three
classical
three
tier
three
three-tier
architectures
and
some
service
bus
around
all
the
all
the
Legacy
stuff
that
have
been
there
has
been
tracing
before
only
a
little
bit.
You
know
a
little
bit
a
little
a
little
bit
more
difficult
to
work
with.
C
B
He's
an
author
of
Taos
famous
operating
system
textbook
and
the
author
of
Minix
that
inspired
Linux
and
so
on.
It's
a
kind
of
it
was
the
one
of
the
celebrities
when
I
was
going
to
University.
He
once
said
the
definition
of
the
truly
distributed
system
is,
that
is
the
system
where
a
failure
of
a
component
that
you
didn't
know
it
existed
can
cause
the
non-functioning
of
the
whole
system.
A
C
D
Charitivity
no
just
kidding
again
I
agree
with
the
new
OP
stuff
with
ebpf.
It's
super
promising,
like
you
just
focus
on
your
business
logic
and
you
drop
an
agent
in
the
kernel
and
you
get
full
insights
into
your
application
of
the
granularity
that
you
want
without
affecting
performance.
I
think
that's
Utopia
and
I.
Think
but,
like
I,
think
we're
very,
very
close
to
that.
Yeah.
F
Gtt
or
AI
Ops
AI
enabled
monitoring.
This
is
the
future
for
sure
I
think
we
haven't
mentioned
the
aspect
of
security.
I
think
this
is.
This
is
becoming
even
more
relevant
recently,
so
our
security
should
be
really
a
dimension
in
in
the
whole
observability
discussion.
So
maybe
maybe
a
topic
for
next
time.
B
I
really
don't
have
anything
more
to
add
and
I
can
say
that
I
certainly
always
will
be
somebody
else's
problem
and
that
observability
and
all
those
SRE
will
go
the
way.
How
serverless
replay
was
kind
of
continuation
of
the
classic
servers
so
essentially
something
that
scales
on
its
own
runs
on
its
own
and
the
it's
essentially
no
Ops,
as
as
Linux
said.
E
A
You
everyone,
it
was
a
great
panel.
Thank
you
for
all
the
questions
from
the
audience.
You
were
very
engaging
and
helped
me
with
the
running
the
the
discussion
so
and
I
hope
that
you
will
continue
in
it
after
this
session,
as
we
will
have
a
coffee
break
and
you
can
get
together
with
the
with
the
speakers
and
ask
them
some
follow-up
questions.
So
thank
you
so
much
and
thank
you
all.