►
From YouTube: Monitor:APM Weekly Meeting - 2019-12-04
Description
Weekly team meeting for the Monitor:APM group. Discussion included:
- Naming
- Ongoing P1 issue
- Metrics dashboard overview
- Using the "workflow::planing breakdown" label
- Deprecation of features in upcoming releases
- Review issues
B
C
C
A
E
Well,
I'm,
located
in
cluj-napoca
in
Romania,
it's
situated
in
Transylvania
I.
Think
you
heard
about
this
area
from
pop
culture
and
yeah
I've
been
working
as
a
front-end
developer
for
the
past,
say
five
years
and
I've
been
in
design.
Also
I
pretty
much
see
the
web
as
the
next
Guttenberg
in
revolution,
so
I'm
pretty
enthusiastic
to
work
in
this
domain.
A
A
C
I
told
Matt
that
this
thing's,
like
this
or
both
my
superpower
and
my
kryptonite
I,
do
like
I
do
like
looking
at
these
kind
of
these
kind
of
issues.
So
if
you
look
at
that
issue,
37
387,
it's
about
Prometheus,
metrics
er,
are
missing
for
some
sidekick
processes
and
it
took
me
a
little
while
to
catch
up
with
with
Andrew
and
the
others,
but
I
finally
did,
and
so
what
what
the
problem
was.
Was
that
handy
here.
C
C
These
sidekick
jobs
running
it
at
any
time,
but
we're
only
seeing
like
four
come
back
from
from
this
psychic
pipeline
number
eight
here
and
the
the
money
line.
Is
this
kid
thing
and
I
didn't
know
about
this
bid
until
looking
at
it,
but
but
every
in
the
Prometheus
client
that
we
use,
we
assume
you
can
see
my
screen
in
the
Prometheus
client
that
we
use.
C
E
C
F
C
F
G
C
If
we
look
on
down
aways,
there's
a
there's,
a
company
there's
a
conversation
about
a
there's,
a
bunch
of
metrics
files
that
have
been
deleted.
If
you
start
looking
at
our
Prometheus
client
that
we
use
that
thing
creates
a
file,
so
memory
mapped
file
for
providing
metrics
so
that
it's
kind
of
you
know
global
to
the
machine
type
thing.
C
C
With
LSO
F,
minus
W
and
grip
for
the
sidekick
directory,
so
it'll
be
like
directory
dot
sidekick.
So
so
there
are
no
deleted
files
out
here.
Yet
if
we
wait
around
a
few
minutes
and
look
back
out
here,
one
of
those
some
of
these
files
are
gonna
start
getting
deleted
on
my
local
box,
which
was
lucky
for
me,
because
I
didn't
have
had
production
access
to
go
around
to
diagnose
this
thing.
So
that
was
that's
the
first
thing.
I
always
do
is
say:
hey
can
I
get
this
thing
going
on
my
local
box.
C
Can
I
reproduce
this
on
the
local
box?
So
so
now
there's
a
bunch
of
deleted
files
out
there
I
look
at
our
old
friend.
This
thing
called
prometheus
cleanup
multi-prong
der
service.
That
thing
just
goes
out
there
and
says:
delete
some
files
delete
all
of
the
files
that
are
associated
with
you
know
the
Prometheus
stuff
that
were
storing
somewhere
in
Unix.
We
need
delete
a
file
and
somebody's
already
got
that
file
open.
Then
that
are
somebody.
Some
process
has
got
it
open.
That
process
still
has
is
open.
C
It
still
keeps
reading
that
file
or
writing
that
file,
regardless
of
the
fact
that
some
of
the
process
deleted
and
when
they
closed
it
the
file
goes
away.
So
that's
the
way
you
know
typical.
Ok,
you
know
thing:
it's
like
hey
my
servers
running
out
of
disk
space
and
I,
don't
have
and
I've
deleted
all
the
files
you
can
delete.
The
files
with
the
disk
space
doesn't
come
back
until
the
writing
process
closes
it,
but
no
other
processes
can
see
it.
C
So
what
I
think
is
happening
here
is
that
we
fire
up
eight
processes
on
this
box
to
run
some
sidekick
jobs
and
the
last
one
deletes
all
a
lot
of
the
files
from
the
first
few
and
those
are
not
getting
reported
through
to
Prometheus
anymore,
because
the
file
show
is
deleted,
but
they're
still
open,
they're
still
being
written
to,
but
they're
just
not
accessible
to
to
read.
So
what
I
did
was.
C
C
A
A
Can
fix
it
cool
yeah?
That's
it's
very
strange
technical
problems.
It
also
brings
up
a
point
where
I
put
the
link
to
possibly
related
one,
so
we're
using
a
older
forked
version
of
the
Prometheus
client,
Ruby
client
right
where-
and
was
this
so
you
said
you
found
this
fix
in
or
something
similar
to
this
in
that
and
oh
you're,
a
mutant,
I.
C
Looked
in
that
file
to
see
how
it
was
manipulating
these
files
and
what
it
was
doing
with
the
files,
because
it
does
storm
in
files,
because
the
regular
Ruby
client
for
every
process,
just
though
I
think
it
stores
the
the
metrics
in
memory,
so
there's
never
a
situation
where
something's
gone.
But
in
that
case
you
know
say:
we've
got.
Eight
sidekicks
were
running
on
one
box.
We
have
to
have
eight
different
end
points
to
read
metrics
from
on
that
one
box,
and
you
know
anything's
on
that
box.
C
A
C
B
C
C
F
C
F
A
C
Would
not
I,
don't
think
yeah
I,
don't
think
this
issue
would
have
even
we
would
have.
We
would
have
seen
this
issue
at
all,
and
it's
just
because
you
know
we
we
made
a
few.
You
know
design
choices
that
you
know
we're
not
necessarily
shortcuts,
but
they
were.
You
know
things
that
would
gave
us
the
ability
to
see
to
satisfy
our
unique
needs
in
the
get
lab
comm
world.
A
A
A
I'll
show
it
to
as
a
engineering
organization,
we've
been
having
a
kind
of
a
renewed
focus
on
some
of
our
metrics,
so
I
put
together
all
right
put
together
this
dashboard,
with
help
from
a
number
of
other
teams,
have
been
providing
some
input
to,
but
I
just
wanted
to
make
sure
everyone
has
seen
this
and
I
can
kind
of
talk
through
a
little
bit
about
some
of
the
stuff.
We're
measuring.
A
A
So
you
can
see
the
red
line
here
is
like
the
team
size
and
the
green
is
the
average.
Mr
square
engineer:
purple
is,
or
just
a
count
which
is
matches
what's
up
here.
So
that's
one
thing
then
there's
some
information
about
which
bugs
we
have
so
we
have
a
lot
of
open
bugs
and
it
breaks
it
down
by
priority
and
severity.
Some
of
this
part
of
the
drive
for
this
dashboard
we
used
to
have
a
quality
dashboard.
It
was
called
that
had
a
lot
of
this
information
on
it
that
was
used.
A
A
So
I
added
some
information
about
bugs
there's
looks
like
there's
one
right
now
that
doesn't
have
priority
or
severity
didn't
just
a
little
more
kind
of
drilling
down
into
some
details
like
merge,
requests
that
don't
have
group
APM
labeled
on
them
group
APM
is
kind
of
the
label
we
use
for
all
of
these
metrics
at
the
top.
So
it's
missing
that
then
we're
not
counting
everything
and
also
if
it's
missing
a
one
of
these
kind
of
types
feature
backstage
bug
your
security.
A
A
So
these
are
kind
of
two
different
ways
just
to
measure
how
long
it
mr
has
been
in
review.
Basically
they're,
not
perfect,
but
it's
a
way
to
get
there,
and
these
are
all
just
for
our
team
yeah
and
then
the
bottom
is
just
how
big
our
team
is
and
how
it's
been
growing
over
time
from
an
engineering
side.
A
A
A
number
of
other
teams
are
creating
very
similar
or
cloning,
this
or
working
on
similar
things,
but
it
would
be
nice
if
you
could
just
filter
it
by
automatically
and
get
lab,
because
all
periscope
I
think
is
doing
is
dumping
like
data
from
the
either
there
that
gala
I
don't
think
it
uses
the
get
Lab
API
I
think
it
pulls
information,
and
maybe
it
is
from
the
API,
but
pulls
it
into
like
a
data
source
that
periscope
you
use,
but
yeah.
All
of
this
we
could
just
create
the
same
thing
with
just
a
gate.
A
Current
get
lab
API
is
because
it's
just
looking
at
issue
date,
mostly
just
issue
data
and
merge
request
information
so
so
yeah.
If
anyone
has
ideas
of
things
that
we
should
would
like
to
track
or
follow
up
on,
feel
free
to
reach
out
to
me-
and
let
me
know
I
the
stuff
that's
on
here
now
is
kind
of.
Are
things
that
are
related
to
kind
of
KPIs
that
we
track
as
an
engineering
organization?
A
So
that's
kind
of
why
I
started
with
that,
but
we
can
certainly
expand
this
to
include
any
other
information
that
would
be
useful
for
our
team,
so
just
wanted
to
share
just
so.
Everyone,
at
least,
could
see
that
we
have
this
data
and,
if
you
ever
are
curious,
you
can
kind
of
take
a
look
at
it.
Yeah.
A
Good
yeah
Ben
I've
been
a
crash
course
in
periscope.
So
basically,
all
of
these
charts
are
just
kind
of
just
sequel:
queries
against
different
data
sources.
So
in
the
back
end
it's
not
that
complicated
and
you
just
put
the
whatever
kind
of
chart
you
want
on
it.
So
but
yeah,
let
me
know
if
you
need
some,
no,
it's,
okay.
A
B
A
A
The
idea
for
that
workflow
label
and
I
like
there's
some
place
in
the
handbook
which
maybe
I,
have
to
fix
but
fix
the
link,
but
in
the
handbook,
under
the
product
development
flow
kind
of
talks
about
all
these
different
workflow
statuses.
And
one
of
them
is
this
planning
breakdown,
which
seems
to
say,
like
hey,
we
kind
of
done
with
the
planning
of
this
issue,
but
now
we
have
to
figure
out
the
implementation
details.
So
my
recommendation
is
that
our
team
starts
using
this
too,
as
a
first
step
in
breaking
issues
down
further.
B
B
And
then
solution,
validation
and
then
cleaning
break
down.
Some
features
that
we
are
building
are
going
through
this
entire
validation
crack.
So
you
may
end
up
seeing
what
research
issues,
so
this
is
where
normally
myself
and
Amelia
and
UX
researcher
are
doing
the
work
for
researching
an
issue,
for
instance
tracing
total
minimum,
and
we
know
that
we
want
to
get
there.
We
don't
know
exactly
what
features
are
required
in
order
for
tracing
to
be
minimal,
so
we
started
with
the
problem:
validation
track.
We
are
interviewing
customers.
B
Once
we
will
interview
the
customer,
we
will
know
better.
We
will
move
from
problem
validation
to
solution
validation,
to
better
understand
what
are
the
features
that
we
want
to
be,
and
then
we
will
go
with.
We
will
apply
the
label
for
planning
breakdown
where
we
are
breaking
down
each
feature
into
into
iteration.
So
this.
B
Bill
the
bill
track
start
to
the
big
face.
This
is
the
entire
flow,
so
just
to
give
the
perspective
of
where
is
it
coming
from?
So
some
issues
starts
from
planning
breakdown.
Some
issues
start
immediately
in
the
beta
phase,
but
if
you
want
to
go
by
the
book,
issues
should
start
with
this
validation
flow
and
all
the
way
to
the
bill
track.
D
A
The
addition
of
this
step
before,
because
I
think,
as
we
talked
about
before
I,
think
we've
struggled
a
little
bit
with
braking
issues
down
from
a
kind
of
an
engineering
perspective,
getting
them
to
like
iterable
driving
iteration
and
getting
small
pieces
and
I
think
this
could
help
that
a
lot
so
I'm
just
wanted
to
so
people
have
thoughts
or
comments
feel
free
to
comment
on
this,
mr,
and
we
can
work
on
that,
and
then
this
is
against
the
handbook.
For
so
we
can
just
try
using
this
and
see
how
it
goes.
B
B
B
We
can
discuss
this
big
chicken
days
like
articles
that
define
how
how
monitored
from
being
just
like,
let's
measure
the
metrics
evolve
into
logs
traces,
and
we
can
see
a
consolidation
in
the
market
where
tech
elasticsearch,
as
example
as
an
example,
started
with
logs,
add
matrix
in
outraces.
Take
data
dog
started
with
matrix,
then
traces
in
our
logs.
They
purchase
magmatic
few
years
ago,
so
there
is
even
Splunk.
So
there
is
a
lot
of
consolidation
in
the
market,
realization
that
monitoring
had
evolved
beyond
just
being
monitoring.
B
This
is
why
this
whole
concept
of
observability
emerge
a
couple
of
years
ago
and
yeah
I
mean
stuff
reading
about
it
and
I
think
makes
makes
a
lot
of
sense.
Given
the
changes
that
we
see
today,
the
technologies
changes
that
we
have
yeah
so
may
I
just
use
like
a
few
lines,
just
a
few
lines
and
one
time
you
have
to
explain
that.
B
B
Is
actually
in
in
the
next
major
lists
we
are
going
to
duplicate,
probably
two
things:
one
is
the
interest
again.
This
is
something
that
I
guess
we
miss
to
do
earlier
this
year
and
apparently
and
I
wasn't
well
of
that
Toyota
for
politics
we
use
in
flux.
Now
we
have
connectors,
there's
no
need
for
it
hooks,
so
we
in
these
12.6
we
need
so
first
of
all
the
process
on
duplicating
something
indeed,
lab
is
first,
you
need
to
announce
on
that
and
only
in
the
major
release
you
can
duplicate
it.
B
So,
ideally,
the
announcement
for
the
depression
in
crooks
could
should
have
announced
earlier,
but
we'll
do
it
on
12.6
and
apparently
in
the
discussion
that
I
had
with
Ruben,
he
suggested
today
deprecated
the
ruby
string,
interpolation
and
he's
in
suggesting
to
replace
it
with
it's
called
the
liquid
template.
Something
like
that.
B
Basically,
the
change
is
that,
using
this,
this
new
template
customers
will
be
able
to
pass
variables
so
indicating
the
custom
dashboard.
However,
the
format
need
to
change
the
format
with
the
percentage
that
you
see
in
the
in
the
description
here.
This
is
not
something
that
liquid
template
can
use
so
for
now
we
will
support
both
cases,
so
we
support
both
string,
interpolation
and
the
liquid
template
format,
but
even
when
we
will
deprecated
the
string
interpolation,
this
means
that
we
have
users
or
customers
that
are
using
that.
What
not
so
I
wanted.
B
You
understand
if
you
think
that
there
is
any
any
risk
here
as
we
do,
that
I
mean
we
are
going
to
announce
that
we
are
going
to
make
sure
custome.
Well,
it's
it's
rather
simple
change
who
want
to
make
sure
if
something
is
broken,
we
change
it.
It's
just
just
to
remove
the
percentage
and
I'd
like
the
curly
bracket.
That's
it!
That's!
That's
the
entire
change,
but
I
wanted
to
get
your
opinion
on
that
as
well.
B
F
I
just
wanted
to
clarify
a
point
regarding
the
supporting
multiple
orgs,
so
I
think
we
already
had
a
small
discussion
regarding
this
particular
thing,
but
basically
is
that
we
have
to,
as
in
areas
for
supporting
multiple
alerts.
One
is
that
we
have
multiple
metrics
inside
a
single
panel
and
each
color
will
correspond
to
a
single
metric
there's
another
scenario
where
you
have
only
one
metric,
but
you
want
multiple
alerts
for
that
given
metric.
F
So,
for
example,
you
want
maybe
to
have
multiple
alerts
that
warn
you
about
memory
consumption,
so
you
have
one
for
one
gigabyte,
one
for
two
gigabytes,
one
for
another
for
40
bytes.
So
in
this
case
the
latter
one.
We
cannot
support
it
because
the
back
end
is
not
supported
if
I
understood
ryan
correctly,
but
the
former
one
we
can
support.
So
for
this
duration
we
could
ship
the
former
one,
but
I'm
not
sure.
If
that's
what
we
want
to
do
at
this
release,
that
I'm
sure
yeah.
B
Matic
pearl
yeah:
that's
that's
good,
because
I
think
the
previous
one
is
we
at
the
support
of
multiple
metrics
on
chart.
What
the
next
thing
is
to
make
sure
users
can
allowed
on
those
metrics.
So
this
is
more
important
than
having
like
the
same
alert
on
the
same,
so
multiple
alert
on
the
single
mapping.
B
F
I
Yeah,
so
you
need
some
back
and
help
with
updating
the
environment,
Search
API.
So
a
little
bit
of
background
about
this,
so
the
environments
drop
down
in
the
monitoring
dashboard
currently
only
shows
many
items,
and
if
there
are
more
than
20
environments,
you
won't
be
able
to
see
that
so
in
order
to
improve
the
experience,
we're
adding
a
search
box
of
that
to
look
up
for
environments.
However,
the
backend
API
right
now
only
returns
the
name,
but
we
need
more
attributes
as
well,
along
with
the
name.
I
A
B
Would
be
valuable,
I
think
I
think
you
know
I
know
that
in
the
last
few
meetings
we
went
over
the
newly
open
issues,
but
I
think
that
it
is
less
productive
unless
other
any
other
one
think.
Otherwise
it's
worth
talking
about
what
people
are
working
today,
maybe
go
over
the
existing
issues
in
206
quickly
and
see
locust
or
any
concern
that
sounds
great.
I
can
maybe
just
before
putting
the
list.
You
know
if
anyone
has
any
concern
about
any
issues
that
is
working
on
them
at
the
moment.
H
B
B
E
H
A
A
B
I
C
C
There's
a
front-end
back-end
component
to
it.
The
front
end
is
just
it's
just
passing
the
query
text
back
to
the
back
through
to
the
backend
we
may
need.
We
may
need
some
user
interface
assistance
on
this.
If,
because,
when
we
throw
the,
when
we
throw
the
feature
flag,
then
we'll
need
the
ability
to
pick.
If
you
want
custom
metrics
or
do
you
want
just
pick
one
metric
and
and
alert
on
that,
but.
B
A
Okay,
bogs,
endpoint,
there's
a
Ryan's
working
on
this
one
to
talk
to
him
earlier
this
week
about
it.
I
think.
Oh,
this
was
the
original
mr,
but
this
is
the
one
that
he's
working
on.
So
he
hasn't
mres
is
it's
still
kind
of
in
progress
so
once
in
progress
as
well,
I
think
my
collage
is
looking
at
this
one.
B
Think
Christianly
think
around
the
license
here
and
I
think
this
is
the
ask
about
the
front
end
issue
which
yeah
open
your
schedule.
The
next
iteration
the
comment
here,
but
they
saw
there
is
an
EMA
on
that.
So
I'm
not
sure
if
someone
is
working
on
it
for
the
situation
and
if
so,
let
me
know
so
because
this
is
also
doesn't
hurt
that
is
pose
if
it
will
have
like
both
fundamental
can.
This
is
definitely.
B
B
A
F
F
C
B
C
B
A
B
G
G
F
F
A
A
B
C
But
so
I
made
a
a
reactive
caching
for
doing
chunk,
reactive
caching,
so
that
the
front-end
can
pitch.
You
know
the
rows
that
you
know
the
rows
that
have
changed.
You
know
the
new
rows
from
the
logs
instead
of
having
to
pull
the
whole
set
of
logs
every
time
so
probably
a
day
away
from
having
that
to
the
point
where
it's
got
this
and
it's
ready
to
be
reviewed,
but
I've
discussed
a
little
bit
with
Adrian
on
the
approach.
I'm
taking
something
I
guess
got
legs.