►
Description
Update issue - https://gitlab.com/gitlab-org/incubation-engineering/apm/apm/-/issues/32
A
Hello
joshua
here
full
stack
engineer
in
the
incubation
engineering
department,
I'm
working
on
apm
application,
performance,
monitoring
management
and
observability
for
the
gitlab
devops
platform.
This
is
my
weekly
update
video.
You
can
see
here
my
demo
update
issue
if
you
like
for
today
covering
the
last
week.
A
A
One
solution
could
have
been
that.
Maybe
this
was
sending
that
high
cardinality
data
much
at
a
much
lower
rate
in
the
agent,
but
I
actually
found
it
was
an
issue
whereby
I
wasn't
deduplicating
or
grouping
the
data
correctly,
based
on
the
entire
unique
tag
set
coming
through
in
the
series
data
from
datadog.
A
Now
this
means
that
we
were
losing
chunks
of
data
and
it
was
sort
of
a
an
ordering
or
race
condition
where
certain
bits
of
data
would
be
dropped,
whereas
all
the
bits
will
be
kept
and
I've
done.
I
fixed
that
I've
added
more
documentation
around
it
now
so
I'll
show
you
some
of
the
docs
that
I've
added
into
our
project
here.
So
there's
a
better
data
dog
agent
docs.
Here
I
thought
I
better
better
put
these
in.
A
While
I
was
doing
that
work,
so
there's
documentation
around
the
series
endpoint
that
I'm
talking
about
about
here.
So
this
is
an
example
post
and
information
about
how
we
handle
metrics
and
custom
metrics
information
around
the
validation
rules
that
we
expect
based
on
the
datadog
api,
and
I've
got
tests
around
all
these
now
to
make
it
a
bit
more
robust
and
more
specifically,
how
we
convert
that
series
format
into
our
measurement
format
that
we
use
in
the
click
house
database
and
I've
got.
A
So
it's
easier
to
to
refresh
on
that
information
and
the
key
bit
here
being
that
it's
grouped
on
this
unique
set
here
of
the
host
timestamp
measurement
and
the
unique
tag
set,
which
is
the
bit
that
I
was
missing
before
that
sort
of
transfers
into
this
goal
line
structure
here.
So
I
can.
I
can
show
you
the
a
running
database
now,
just
to
give
an
example
of
the
sort
of
cardinality
of
data
we've
got
in
there.
So
if
I
have
a
look,
this
is
running
on
my
machine
at
the
moment.
A
Just
and
you
can
see,
I've
done
a
count
of
metrics
there,
individual
metric
measurements,
that's
a
measurement
name
and
it
has
a
tag
set
and
it
has
a
set
of
field
values
associated
with
that
measurement.
You
see,
there's
quite
a
lot
there
quite
a
significant
amount.
That's
because
it
you
know
it
generates
a
lot
of
data.
If
we
look
at
the
distinct
measurements
in
metrics,
these
are
all
the
different,
unique
top
level
measurements
that
we
that
we're
getting
and
there's
a
there's
a
lot
of
stuff.
A
A
And
you
can
see
the
cpu
metrics
there
and,
as
an
example,
you
you,
this
doesn't
have
any
tags
at
all.
These
are
the
text
fields
here,
but
it
does
have
a
lot
of
different
fields.
So
in
some
ways
it's
relatively
high
cardinality.
Those
are
those
are
all
fields
associated
with
one
cpu
record,
and
these
are
the
the
array
values
that
are
stored
with
it,
but
as
an
example
of
a
high
cardinality
tag
set
data
source.
A
What
you
can
see
here-
and
this
is
just
one
record
on
this-
to
the
tag
center
for
that
record-
you've
got
a
lot
of
different
fields
associated
with
it,
so
you've
got
container
id
container
name,
display,
name,
docker,
image,
image,
name,
image,
tag,
container,
name,
cube,
name,
space,
part,
name,
pod,
phase,
short
image,
so
a
lot
of
fields
and
there's
a
lot
of
different
variations
of
this
sort
of
thing,
and
I
was
seeing
way
less
than
I
expected.
So
I
managed
to
fix
that
issue.
A
A
So
back
to
the
other
thing
that
we
were
doing,
we
were
getting
the
datadog,
endpoints
and
general
sort
of
series,
time
series
endpoints
integrated
with
gitlab,
so
that
we
could
tie
together
an
api
key
provided
by
an
agent
with
a
a
gitlab
project,
an
optional
environment,
and
I
showed
you
a
bit
of
this
work
last
week.
We've
made
good
progress
with
this
is
nearly
functional.
A
So
just
as
a
reminder,
we've
got
some
documentation
here
as
part
of
this
issue,
talking
about
adding
these
environment
variables
as
as
global
tags
in
the
case
of
datadog,
but
this
will
put
with
other
agents
as
well
and
what
we
actually
discovered
while
we're
doing
this.
These
don't
end
up
on
every
request
with
datadog
there's.
Actually,
an
initial
intake
request
that
ends
up
with
these
host
tags
specified.
A
So
we
need
to
be
able
to
strip
those
out
of
that
request
and
then
pair
those
up
with
the
api
token
and
what
will
and
that
the
last
bit
we
need
to
do
with
that
is
pair,
that,
together
with
those
access
rights
and
store
that
in
some
sort
of
session
storage
about
the
solution.
So
I'm
thinking
of
having
a
simple
redis
cache
in
there
for
now,
so
this
stuff
can
have
a
one-way
hash
on
it
and
we'll
store
that
that
information
based
on
the
api
key
and
the
host
name.
A
A
So
we've
got,
I
showed
you
before
this
example
of
the
gitlab
api.
I've
updated
this
for
the
intake
endpoint
that
we
were
talking
about
and
what
we
do
and
what
we've
got
implemented
as
well
is
when
we
get
a
project
back
successfully
to
attach
to
a
metric
set.
We
just
check
that
the
permissions
part
of
the
response
is
set,
and
then
it
has
an
access
level
equal
to
or
above
30,
which
means
that
the
api
token
is
associated
with
a
developer
level
account
and
the
assumption
there
is
a
developer
level,
accounting
git.
A
A
So
that's
good,
that's
implemented
now
and
what
was
the
next
bit?
The
other
part
of
that
is
for
local
development.
We
we
created
a
gitlab
stub
service,
so
we
don't
have
to
have
a
gitlab
instance
running
or
all
well-known
keys
configured
in
gitlab.com
or
whatever
for
local
dev.
I've
just
created
a
stub
for
the
endpoints.
We
need
again
I've
documented
this
a
bit
better
as
well
as
part
of
the
documentation,
so
the
local
stub
endpoints
you
can
see
here.
These
are
the
ones
we've
we
implemented,
and
these
are
the
ones
we
need.
A
So
we
we
need
to
get
the
project's
id,
get
a
project
by
id
to
work
out
what
these
permissions
are.
Optionally,
get
the
environment,
so
we
can
validate
that
environment
variable
and
then
we
use
the
version
endpoint
to
do
basic
api,
key
validation
and
that's
part
of
the
standard
gitlab
api.
So
when
the
agent
does
a
verification
request
for
the
token
we
can
just
check
against
that
endpoint
as
well,
so
that
makes
it
nice
and
simple
I'm
trying
to
think
if
it's
anything
else.
I
think
that
mostly
covers
what
we've
been
up
to.