►
Description
Weekly APM Single Engineer Group update video.
Demo issue - https://gitlab.com/gitlab-org/incubation-engineering/apm/apm/-/issues/19
A
A
A
So
that's
where
we
are
looking
at
storing
metrics,
logs
and
traces
initially
via
the
datadog
agent.
So
a
bit
of
history
on
that.
We've
previously
done
some
analysis
of
the
data
job
agent
in
a
sandbox
environment.
A
To
make
sure
it's
doing
the
sort
of
things
we
expect
and
a
bit
of
validation
around
that
we've
also
performed
quite
an
in-depth
evaluation
of
click
house
and
benchmark
for
metric
storage
and
we've
designed
a
flexible
metric
schema
for
click
house
as
well,
and
you
can
see
that
in
the
previous
videos
and
we
benchmarked
that
as
well
with
some
good
results.
A
So
what
I'm
work?
What
I've
been
working
on
in
the
last
week
is
trying
to
push
forward
infrastructure
for
the
apm
project,
so
it
will
be
running
slightly
separately
from
gitlab
initially
because
it
it
integrates
with
the
gitlab
api,
so
it
will
do
and
we
can
run
it
in
its
own
google
cloud
project.
A
So
we
can
more
easily
monitor
that.
So
we
managed
to
get
incubation
members
to
be
able
to
create
cloud
sandboxes
now,
so
we
can
create
google
cloud
and
aws
sandboxes
to
build
and
test
our
our
software
if
we
need
to
so
I'll,
be
using
one
of
those
sandboxes
to
set
up
a
test
environment
for
this,
and
we
we've
sort
of
got
a
fairly
good
handle
on
now.
The
sort
of
access
requests
we
need
to
make
to
get
proper,
staging
and
production
environment
set
up.
A
I'm
going
to
do
that
once
I've
spent
some
time
iterating
on
the
infrastructure
for
this
to
make
sure
it
works.
As
I
expect
it
should.
A
That's
using
the
newly
developed
click
house,
storage
schema
and
you
know
it's
it's.
A
very
initial
version
needs
a
lot
of
work
to
harden
it
and
make
it
acceptable
for
production
use
a
git
web.
So
I
need
to
work
on
that
as
well.
So
we've
got
a
an
open
issue
here,
which
is
the
the
sort
of
overall
iteration
link
to
the
submit
backend
there
and
then
some
cross
references
here.
So
we've
got
the
the
metrics
design
that
we've
completed
and
some
of
the
components
here
that
we're
looking
at.
A
So
what
we've
done
so
far
in
this
project
is
set
it
all
up
to
to
be
kubernetes
first,
so
it's
all
sort
of
cloud
native
setups
and
that's
even
in
development
we're
using
mini
cube
so
that
when
we
move
through
environments
into
testing
staging
and
so
on,
it
should
be
a
lot
easier
for
us
to
to
accomplish
that.
We
don't.
We
won't
be
building
all
that
from
scratch,
and
so
I'm
that's
the
way.
I'm
testing
the
application
locally.
A
So
we
we
merged
in
a
helm
deployment
with
click
house.
So
for
click
house
we've
gone
down
the
route
of
using
this
alternative
click
house
operator.
I've
seen
quite
a
lot
of
mentions
of
this
for
a
while,
and
it
seems
to
be
using
a
few
different
organizations
successfully.
A
It's
got
a
lot
of
documentation
and
this
should
allow
us
to
create
a
click
house
deployment
on
communities
where
we
can
manage
shards,
replicases
and
migrations
of
data,
and
things
like
that
quite
well.
So
there's
there's
a
lot
of
functionality
in
there.
It's
got.
You
know
built-in
configuration
for
monitoring
maintenance
tasks
and
you
can
do
a
lot
of
complex
setups
with
it.
So
it
looks
quite
useful,
I'm
using
that
both
in
development
as
well
to
set
up
the
base
basic
case.
A
What
else
do
we
do?
The
development
environment
also
sets
up
data
dogs,
so
we
can
test
that
locally.
That's
running
in
the
in
the
mini
cube
instance,
and
we've
also
got
a
test
version
of
grafana
in
there
as
well
to
run
with
that
and
the.
A
This
issues?
That
is,
the
one
that
we're
tracking
the
actual
series
endpoint
implementation
in.
So
we've
got
some
details
of
that
here
and
we're
working
on
that.
You
know
in
another
merge
request
here
for
the
draft
gateway
service,
so
I'll
show
you
that
running
that
runs
from
this
project.
It's
a
simple
installation
via
helm
that
we
can
run
locally
through
mini
cube.
A
So
here's
the
canines
interface
that
we
can
look
in
our
local
cluster.
Looking
at
the
pods
in
the
cluster
here,
so
you've
got
a
click
house
cluster.
There
single
shard,
single
replica
cluster,
just
for
development
that
you
can
see.
There's
the
click
house
operator.
We've
got
our
dev
gateway.
We've
got
the
datadog
agent
running
that
has
the
main
agent
and
some
of
the
components
in
there.
A
So
I've
been
able
to
reuse
quite
a
few
of
those
components,
and
we
also
got
cube
statement
cube
state
metrics
running
there
that
data
dog
uses
to
get
kubernetes
metrics,
so
that
is
deployed
in
quite
a
simple
way.
All
the
images
are
built
automatically
and
deployed.
So
it's
quite
easy
to
use.
A
I
need
to
set
up
ci
to
do
some
builds
on
there
as
well,
and
you
can
see
as
a
quick
example
if
we
refresh
that
this
is
the
data
from
that
click
house
database
with
the
newly
designed
metrics
schema,
and
this
is
system
cpu
and
user
cpu
being
rendered
in
here
and
those
are
you
can
see
the
queries
down
here.
A
Excuse
me,
you
see
the
queries
in
place
down
there
that
grab
and
display
the
data.
So
there
we
go
grabbing
the
measurement
system
cpu
and
getting
an
average
of
the
of
the
system
field
names
from
there.
So
this
this
kind
of
splits
out
the
data
into
a
nice
format
for
us.
So
that's
good!
That's
positive!
Like
I
said,
there's
a
lot
of
work
still
needs
doing
there
just
to
productionize
that
that
service-
and
you
can
see
all
the
code
for
that
in
this
in
this
project.
A
I've
added
some
interesting
links
there,
including
the
the
operator
so
next
week.
I
need
to
continue
improving
that
implementation,
adding
some
test
cases
making
sure
it
works
for
different
combinations
of
metrics,
doing
some
exploration
with
it
and
also
to
continue
iterating
on
the
test
infrastructure
as
well.