►
From YouTube: Tamland: How GitLab.Com Uses Long-Term Monitoring Data For Capacity Forecasting. - Andrew Newdigate
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Tamland: How GitLab.Com Uses Long-Term Monitoring Data For Capacity Forecasting. - Andrew Newdigate, GitLab, Inc.
Speakers: Andrew Newdigate
Tamland is a capacity planning tool built by GitLab to provide long-term forecasts of potential capacity issues across the services running GitLab.com. It's built on top of the long-term metric storage capabilities of Thanos, which provides utilization and saturation metric data stretching back over a 1 year period. From this, a predictive forecast model is constructed and used to predict future growth trends across hundreds of saturation points over the coming months. This practical talk demonstrates how we capture long-term metrics data in a scalable way using Thanos, how we use Facebook's Prophet library for building forecast models, and how we integrate this with Jupyter to generate a report complete with visualizations. It discusses the benefits of switching to a data-driven and repeatable approach to capacity planning, as well as some of the practical challenges of building the tool. Tamland is an open-source project and attendees have access to the project source if they're interested in digging deeper into our implementation.
A
Hi
everybody
I
hope
you're
all
having
a
great
conference.
Thank
you
for
coming
to
my
talk
about
Tam
land
or
how
goodlab.com
uses
long-term
monitoring
data
for
capacity
forecasting.
My
name
is
Andrew
nudigatz
I'm,
a
distinguished
engineer
in
the
infrastructure
departments
at
gitlab,
where
I
focus
on
gitlab.com.
A
Let's
start
with
a
bit
of
a
backstory,
this
Story
begins
in
mid
2021
gitlab.com
experienced
a
series
of
S1
database
incidents
related
to
high
resource
utilization
on
our
primary
postgres
cluster
within
the
company,
concerns
were
raised
about
the
growth
rate
of
gitlab.com
and
the
ability
of
our
primary
database
to
support
this
growth.
If
the
database
hit
maximum
capacity,
it
might
lead
to
further
outages
due
to
saturation
issues.
A
A
A
It
would
be
much
better
if
we
could
confirm
that
the
database
would
have
sufficient
capacity
allowing
us
to
stick
to
the
original
date,
giving
the
take
that
giving
the
team
the
space
to
carry
out
the
migration.
With
due
caution,
we
needed
an
accurate
and
data-driven
assessment
of
the
available
capacity
in
our
main
postgres
cluster.
Using
a
new
tool
called
tameland,
we
were
able
to
carry
out
a
capacity
review
with
timeland.
We
analyzed
all
potential
saturation
points
within
the
cluster
and
plotted
their
individual
growth
forecasts
to
predict
which,
if
any,
would
hit
saturation.
A
We
found
that
if
a
few
small
changes
were
made
in
the
short
term,
we'd
have
enough
Runway
to
support
the
delivery
of
the
functional
decomposition
over
the
multi-month
time
frame.
The
project
was
expected
to
take,
so
it
turns
out
that
this
is
a
pretty
boring
story.
As
predicted
we
had
enough
capacity
to
perform
the
migration
safely.
The
migration
got
completed
in
time
without
hitting
capacity
issues
that
we
had
been
concerned
about,
and
everyone
lived
happily
ever
after,
but
in
software
infrastructure,
boring
is
normally
a
good
thing.
A
A
I
found
this
quote
from
a
paper
called
SRE
best
practices
for
capacity
management
by
Louis,
Casada,
Torres
and
Doug,
kolish
and
I
think
it
captures
the
goal
of
capacity
management
really
well.
The
goal
of
capacity
management
is
controlling
uncertainty
in
the
midst
of
the
unknown.
The
service
must
be
available
now
and
continue
to
run
in
future.
A
challenging,
but
rewarding
and
delicate
balance
of
trade-offs
is
in
play.
Efficiency
versus
reliability,
accuracy
versus
complexity
and
effort
versus
benefits
before
I
continue
I'll
Define.
A
few
terms
that
I'm
going
to
use
throughout
this
talk.
A
The
first
one
is
the
definition
of
a
resource.
A
resource
can
be
anything
that
can
be
consumed
and
has
a
limit.
A
lot
of
capacity
planning
literature
focuses
on
Hardware
resources,
for
example,
CPU
memory
and
network
bandwidth,
and
these
are,
of
course
very
important
examples,
but
there
are
a
lot
of
other
software-defined
resources
that
are
equally
important
to
our
capacity
planning
process.
A
A
A
Etc
capacity
is
for
any
given
resource
the
maximum
utilization
decibel,
for
example,
the
maximum
number
of
open
files
that
are
processes
allowed.
The
total
capacity
of
a
disk
in
bytes
or
the
maximum
number
of
database
connections
allowed
in
a
pool
utilization
percentage
is
the
utilization
expressed
as
a
percentage
of
the
capacity
of
a
resource.
A
Finally,
saturation
is
what
happens
when
the
utilization
of
a
resource
nears
or
reaches
a
capacity
of
that
resource?
In
some
cases,
saturation
or
at
least
prolonged
saturation
will
adversely
impact
the
performance
of
the
system,
for
example,
through
increased
queuing,
higher
latencies
errors
or
possibly
complete
system
failure,
planning
for
and
hopefully
avoiding
saturation
is
the
goal
of
capacity
planning.
A
Foreign
here
are
some
examples
of
saturation
events
and
the
typical
times
that
a
full
mitigation
might
take.
In
some
cases,
the
low
impact
of
saturation
or
the
ease
of
mitigation
means
that
we
don't
need
to
focus
on
these
items
as
a
priority
in
our
capacity
planning
process.
For
example,
in
kubernetes
pod
CPU
saturation
should
be
mitigated
automatically
within
minutes
by
the
horizontal
pod
Auto
scaler.
So
it's
unnecessary
to
focus
on
these
resources
in
a
capacity
planning
review.
A
A
If
one
of
these
resources
unexpectedly
became
saturated,
it
could
have
a
detrimental
impact
on
the
availability
of
the
system
for
weeks
or
even
months.
That
could
mean
that
requests
to
the
service
start
experiencing
latency
issues
or
the
application
could
fail
completely.
For
example,
in
the
case
of
a
postgres
transaction
ID
wraparound
event
in
which
reaching
saturation
will
automatically
shut
down
the
database
and
a
disaster
recovery
operation
will
need
to
take
place.
The
goal
of
our
capacity
planning
process
is
to
focus
on
Resources,
with
high
risk
or
long
mitigation
times.
A
Let's
discuss
the
general
approach
that
we
use
to
capacity
planning
on
gitlab.com
before
diving
into
further
details,
a
lot
of
capacity
planning
advice
that
I've
seen
focuses
on
the
system
as
a
whole,
using
demand
signals
such
as
monthly
active
users
or
requests
per
second.
They
talk
about
the
current
utilization
in
times
in
terms
of
these
signals
and
the
total
capacity
expressed
in
the
same
terms.
In
other
words,
the
current
and
maximum
utilization
of
the
entire
system
are
represented
by
a
single
metric.
A
A
The
first
approach,
and
by
far
the
most
widely
used,
is
to
guess
what
the
maximum
the
first
approach
and
by
far
the
most
widely
used,
is
to
guess
what
the
maximum
the
first
approach
and
by
far
the
most
widely
used,
is
to
guess
what
the
maximum
utilization
capacity
of
your
service
is.
This
is
known
as
the
back
of
the
envelope
method
or
even
the
seat
of
your
pants
method.
A
It's
not
scientific,
and
it's
not
Based
on
data
only
intuition
it
has
many
problems.
It's
almost
certainly
not
going
to
be
correct.
You
can't
systematically
improve
the
process
behind
it,
so
it
never
gets
any
better.
It's
not
scalable
and
you
can't
automate
it.
This
means
that
you're
less
likely
to
carry
out
capacity
planning
since
it's
a
manual
process.
A
The
second
approach
to
determining
utilization
capacity
is
through
load
testing.
This
is
much
much
better
than
guessing,
but
this
approach
also
has
several
downsides.
If
you're
running
a
large
system,
setting
up
an
appropriately
sized
test,
bed
can
be
prohibitively
expensive,
if
possible
at
all.
Additionally,
you'll
need
to
create
a
large
custom
cluster
of
clients.
Additionally,
you'll
need
to
create
a
large
cluster
of
clients
to
generate
the
workload.
This
adds
to
the
cost
and
complexity.
A
Approach
is
using
modeling.
This
approach
involves
building
executable
performance
models,
possibly
in
a
spreadsheet,
Jupiter
notebooks
or
a
regular
programming
language.
The
model
attempts
to
capture
the
performance
and
scalability
characteristics
of
the
system.
This
approach
can
be
very
complex.
Additionally,
it
can
be
very
difficult
to
validate
whether
the
model
is
predicting
the
maximum
utilization
correctly.
A
A
A
The
second
approach
is
determining
capacity
through
load
testing.
This
is
much
much
better
than
guessing,
but
this
approach
also
has
several
downsides.
You're
running
a
large
system
setting
up
an
appropriately
sized
test
bed
can
be
prohibitively
expensive
if
possible
at
all.
Additionally,
you'll
need
to
create
a
large
cluster
of
clients
to
generate
the
workload,
and
this
adds
to
the
cost
and
the
complexity.
A
The
third
approach
is
using
modeling.
This
approach
involves
building
executable
performance
models,
possibly
in
a
spreadsheet
or
jupyter
notebooks,
or
a
regular
programming
language.
The
model
attempts
to
capture
the
performance
and
scalability
characteristics
of
the
system.
This
approach
can
also
be
very
complex.
A
So
estimating
capacity
is
really
hard
to
make
matters
worse.
The
application
is
evolving
very
quickly,
as
is
the
production
environment
in
a
monolithic
application
such
as
gitlab,
multiple
teams
may
be
making
changes
to
the
environment
and
the
applications
simultaneously,
making
it
even
more
difficult
to
control
for
change.
Additionally,
experiments
are
running
feature,
flags
are
being
toggled
and
user
activity
patterns
are
constantly
changing.
A
Any
maximum
capacity.
Estimates
that
are
produced
quickly
go
out
of
date
and
lose
their
meaning
constantly
lagging
behind
the
true
state
of
the
environment.
This
means
that
the
capacity
estimates
need
to
be
constantly
revised
through
frequent
updates.
Unfortunately,
none
of
the
methods
for
capacity
estimation
are
easy
to
perform
and
efficiently.
Automating
them
is
very
hard
or
not
possible.
A
The
approach
that
we
use
on
gitlab.com
is
closest
to
modeling.
However,
instead
of
trying
to
estimate
the
capacity
of
the
system
as
a
whole
and
expressing
that
using
a
single
demand
indicator
such
as
active
users,
we
treat
each
Service
as
a
collection
of
related
resources
and
monitor
and
track
them
independently.
So
the
resources
for
each
service
are
treated
as
a
collection
of
independent
but
grouped
capacity
planning
metrics.
A
All
of
these
metrics
have
different
growth
rates
thresholds
and
periodicity.
Each
new
change
introduced
to
the
system,
be
it
through
application,
changes
or
infrastructure
changes
or
even
user
activity,
will
impact
the
resources
differently.
We
do
not
attempt
to
aggregate
these
resource
capacities
back
into
a
single
system
capacity,
as
this
is
error-prone
and
difficult
to
verify.
A
Unlike
when
attempting
to
estimate
total
system
capacity,
estimating
the
capacity
of
individual
resources
is
fairly
straightforward,
for
example,
for
disk
space.
The
capacity
is
the
disk
size
for
open
file
descriptors.
The
capacity
is
the
maximum
number
of
file
descriptors
and
for
database
connection
pools.
The
capacity
is
well
defined
in
configuration.
A
The
biggest
downside
to
this
approach
is
that
we
have
to
track
and
forecast
a
lot
more
metrics,
but
many
services
share
the
same
types
of
resources.
For
example,
many
stateful
services
will
share
disk
space
and
inode
resource
types.
Many
ruby-based
services
will
share
a
resource
metric
representing
Global
interpreter
lock
utilization.
A
We
do
this
by
constructing
recording
rules
in
Prometheus
all
the
resource
utilization
recording
rules
use
the
same
name,
gitlab
resource
utilization,
colon
ratio,
but
they
have
different
labels
to
distinguish
them
one
label
for
the
name
of
the
service
and
another
for
the
name
of
the
resource.
Each
combination
of
service
and
resource
will
have
one
series.
A
A
These
recording
rules
are
fairly
easy
to
construct
for
any
given
resource.
They
are
simply.
The
current
utilization,
divided
by
the
current
capacity
here,
are
three
examples.
From
the
gitlab
code
base
for
gcp
quota
resources,
we
obtain
current
quota
utilization
and
current
limits
from
gcp
quota
exporter
for
Prometheus
for
open
file.
Descriptometrics
many
Prometheus
client
libraries
automatically
publish
these
metrics
out
of
the
box,
so
they're
very
easy
to
obtain.
A
Third
example
demonstrates
how
persistent
volume
claim
disk
usage
can
be
monitored
and
using
metrics
exported
by
cubelet
approach
results
in
a
lot
of
utilization,
metrics
being
generated
using
the
open
file
descriptor
utilization
metric
on
gitlab.com.
There
are
close
to
5
000
series
for
this
at
present.
This
is
too
much
data.
We
need
to
reduce
it
down
by
aggregation
so
that
we
can
have
a
single
representative
metric
per
service
resource
pair.
Now
there
are
many
different
ways
that
we
can
do
this.
A
For
example,
we
could
use
an
average
or
quantile,
but
in
most
cases
we
find
that
the
best
way
of
aggregating
these
metrics
is
by
the
maximum
value.
This
isn't
always
the
case,
for
example,
for
some
kubernetes
metrics,
where
we
might
have
very
high
cardinality.
We
might
use
the
99th
percentile
removing
outlier
values.
A
However,
in
the
majority
of
resources
we
aggregate
using
Max.
The
reason
we
use
this
is
that
highest
utilization
is
the
one
that
leads
to
saturation
and
instability
in
the
system.
For
example,
if
you
have
four
servers
in
a
cluster
and
three
of
them
have
very
low
disk
utilization,
but
one
of
the
servers
has
a
disk.
A
Each
of
these
resource
service
combinations
has
a
set
of
recording
rules,
alert
threshold
values,
alerts
and
grafana
dashboards
associated
with
it.
Early
on
this
configuration
was
hand
coded,
but
this
soon
became
unmanageable
with
too
many
combinations
of
resources
and
services.
To
get
around
this,
we
migrated
to
configuration
language
called
jsonnet.
This
allows
us
to
manage
a
single
configuration
for
each
resource,
including
metadata
such
as
documentation
or
whether
or
not
the
resource
is
horizontally
scalable.
This
slide
shows
an
abbreviated
example
of
that
configuration.
A
We
can
then
validate
the
configuration
before
using
it
to
generate
all
the
recording
rules
and
alerts
for
dashboard
generation.
We
use
a
jsonet
library
maintained
by
grafana
called
grafonet.
This
allows
us
to
generate
all
our
dashboards
from
the
same
configuration
and
ensure
that
they're
always
up
to
date.
In
fact,
most
of
the
dashboards
for
gitlab.com
are
now
generated
from
a
graphonic
definition.
A
This
slide
shows
the
main
utilization
dashboard
for
a
service,
in
this
case
our
primary
postgres
cluster.
Each
of
these
series
represents
the
worst
case
for
a
given
type
of
resource.
This
allows
us
to
quickly
review
resource
utilization
across
dozens
or
even
hundreds
of
resources
that
are
being
consumed
by
the
service.
If
one
of
the
metrics
requires
further
investigation,
we
can
navigate
to
detailed
dashboards
that
are
generated
for
the
resource.
A
This
is
what
some
of
those
panels
look
like
for
each
resource.
We
generate
a
full
grafana
panel.
These
six
resources
belong
to
our
main
postgres
instance.
Once
again,
the
aggregated
recording
rule
value
is
represented
by
the
thick
yellow
line
and
the
de-aggregated
underlying
values
are
visualized
alongside
those
two.
This
allows
us
to
quickly
dig
down
into
the
source
of
utilization
or
saturation
issue
and
determine
the
problem.
A
Of
course,
we
generate
alerts
for
each
utilization
metric
too
generally.
The
alerting
threshold
for
each
resource
is
somewhere
below
100
percent.
The
actual
alerting
threshold
depends
on
the
nature
of
the
saturation
metric
and
the
way
in
which
it's
measured.
In
particular,
the
time
period
over
which
samples
are
collected.
A
These
thresholds
are
used
for
real-time
monitoring
alerts,
but
also
for
forecasting
capacity
issues
as
we'll
see
later
now
that
we're
collecting
utilization
data
for
resources
in
a
normalized
form,
we
can
start
analyzing
it
over
long
periods
of
time.
Looking
for
trends,
luckily,
gitlab.com
uses
Thanos
for
long-term
metric
storage.
This
means
that
all
the
resource
utilization
data
that
we
use
for
short-term
monitoring
of
gitlab.com
is
retained
for
several
years
in
object,
storage.
A
A
Our
first
attempt
at
forecasting
was
not
a
huge
success.
We
attempted
to
use
linear
regression
to
predict
future
growth.
Prometheus
has
a
predict
linear
function
that
we
can
employ.
So
our
first
somewhat
naive
attempt
at
building
a
forecasting
engine
was
built
on
this
function,
since
the
data
could
be
processed
within
Thanos.
This
approach
was
very
easy
to
implement
and
to
experiment
with.
A
We
could
see
promise
in
the
approach,
but,
unfortunately
the
forecasts
were
very
inaccurate.
One
of
the
main
reasons
for
this
is
that
linear
regressions
failed
to
take
seasonality
into
account.
In
this
example,
the
linear
regression
is
able
to
determine
the
single
long-term
growth
Trend
in
the
utilization
data,
but
fails
to
take
into
account
weekday
Peaks
averaging
the
trend
with
weekend
off
peak
times.
A
A
Luckily,
around
this
time
we
became
aware
of
an
open
source
project
out
of
matter
called
profit.
It's
a
library
written
in
r
in
Python
for
performing
forecasting
and
predictions.
It
seemed
like
a
much
better
fit
for
our
purposes
than
linear
regression
could
ever
be,
and
we
started
experimenting
with
a
proof
of
concept.
A
It
quickly
became
clear
that
it
was
well
matched
to
the
task
at
hand.
Here
are
some
of
the
reasons
we
really
like.
Profit
forecasting
is
notoriously
difficult
to
do.
It
generally
requires
specialist
skills.
Profit
has
been
designed
to
make
forecasting
easier
without
the
need
for
data
science,
expertise,
profit,
works
on
time,
series
data
and
can
produce
very
good
forecasts
without
needing
specific
tuning
and
customization
on
a
per
series
basis.
A
This
allows
us
to
scale
it
up
to
hundreds
of
forecasts,
it's
very
fast
at
generating
full
cost
too,
which
also
helps
when
you
have
as
many
as
we
do.
It's
able
to
recognize
seasonal
patterns,
for
example,
traffic
activity
over
hours
in
a
day
or
days
in
a
week.
Our
utilization
data
is
strongly
seasonal.
We
see
the
same
traffic
patterns
week
in
and
week
out.
A
A
We
jokingly
called
the
proof
of
concept
timeland
after
brick,
time
land,
the
kind
but
simple-minded
weatherman
from
the
Anchorman
movies
played
by
the
actor
Steve
Carroll.
We
selected
the
python
version
of
The
Profit
library,
because
we
have
more
expertise
within
the
team
in
Python
than
in
r.
Profit
is
well
suited
for
use
with
jupyter
notebooks,
but
we
wanted
to
automate
the
process
of
generating
the
reports,
so
we
use
the
python
Library
called
Jupiter
book,
which
is
designed
to
generate
static
websites
from
jupyter
notebooks
or
from
markdown
documents.
A
The
utilization
data
is
combined
with
metadata,
such
as
service
catalog
information
and
resource
metadata,
to
augment
the
forecast
with
useful
context
and
descriptions
in
the
gitlab
CI
pipeline.
We
import
the
utilization
data
from
Thanos,
run
forecasts
and
generate
a
static
sites,
including
plotly
graphs,
showing
Trends
and
forecasts
over
time.
We
then
published
the
site
using
gitlab
pages.
This
pipeline
is
run
automatically
on
a
weekly
basis.
A
A
Here's
an
example
of
a
resource
from
a
service
in
the
report
at
the
top,
we
have
a
description
of
the
resource
being
monitored.
This
is
important
because
in
some
cases,
such
as
for
inodes,
it's
very
easy
to
understand
the
utilization
metric,
but
in
other
cases
some
of
the
resources
that
we
measure
can
be
pretty
abstract
and
having
a
clear
explanation
of,
what's
being
tracked,
can
really
help.
A
A
This
is
used
for
prioritization
and
alerting.
Finally,
we
have
the
most
important
component
the
forecast
chart.
This
is
a
Time
series
chart
plotted
over
a
270-day
period,
that's
six
months
into
the
past
and
three
months
into
the
future,
because
of
that
the
current
date
is
always
two-thirds
across
the
time
series
for
the
forecast.
We
present
the
median
confidence
line
and
an
80
confidence
interval
range
around
that,
depending
on
the
variance
in
the
data,
this
confidence
band
will
be
wider
or
narrower.
A
This
chart
measures
our
total
git
disk
utilization
across
all
80
or
so
of
our
git
servers.
Recording
this
as
a
resource
metric
gives
us
a
pretty
good
idea
of
when
we'll
need
to
provision
additional
git
servers.
The
provisioning
of
these
servers
doesn't
take
particularly
long,
but
there
can
be
a
lead
time
in
getting
quota
limits
raised
for
storage.
So
it's
important
that
we're
planning
ahead.
For
these
events,
here's
an
example
of
what
profit
calls
a
change
point.
A
A
change
point
is
an
inflection
when
a
trend
changes
in
this
case
we
see
a
sudden
Step
Up
in
resource
consumption,
because
we
don't
have
much
Headroom
on
the
service
and
the
lead
time
to
mitigating
the
saturation
is
long.
The
decision
was
made
to
investigate
what
had
led
to
this
change.
A
regression
in
the
application
was
discovered
and
corrected,
avoiding
a
saturation
problem
further
down
the
line.
A
Here's
one
more
example:
this
resource
represents
redis
memory
on
our
reader
session
storage
cluster.
The
cluster
stores
session
state,
if
we
hit
saturation
redis,
will
start
to
evict
old
session
state
with
the
least
recently
accessed
being
evicted
first,
if
that
happens,
you
may
need
to
re-log
in
to
the
gitlab
web
application
sooner
than
you
otherwise
would
have
needed
to.
This
is
caused
by
a
bug
in
the
gitlab
application,
coupled
with
some
unusual
user
activity.
A
The
issue
on
the
right
was
created
around
about
the
time
of
the
event
and
demonstrates
how
the
team
were
able
to
pick
this
issue
up
very
quickly
within
a
short
period
of
time,
from
the
start
of
the
event
and
long
before
the
issue
became
critical.
Unfortunately,
in
this
case,
the
forecast
wasn't
accurate
in
that
it
didn't
predict
saturation
because
it
had
not
yet
picked
up
the
change
point,
but
it
was
picked
up
via
manual
review
with
plenty
of
time
to
mitigate
the
problem.
A
Here
are
some
figures
from
our
current
deployment
of
temland.
We
monitor
about
360
different
service
resource
combinations,
but
this
is
growing
all
the
time.
Coincidentally,
we've
investigated
about
the
same
number
of
potential
issues
within
the
last
12
months.
The
timeline
report
generation
job
takes
around
90
minutes
to
run
on
a
dedicated,
N1
standard,
8
Runner.
A
Tamlan
is
an
aesthetic
project
and
we're
improving
it
all
the
time
to
make
it
better
and
more
useful
for
forecasting
for
engineers
at
gitlab.
Here
are
some
of
the
improvements
that
we're
considering
implementing
in
future
tamlan
started
off
as
an
internal
project,
and
it
was
highly
coupled
to
our
specific
infrastructure
and
monitoring
systems
we'd
like
to
continue
to
evolve
the
project
in
a
way
that
would
allow
other
people
to
benefit
from
it.
This
would
require
decoupling
it
and
allowing
more
configuration
options.
A
A
Another
goal
is
better
ownership
to
shift
the
ownership
left
to
product
teams.
You
can
do
this
by
sending
alerts
directly
to
those
teams
and
finally,
it
would
be
worthwhile
evaluating
whether
other
forecasting
libraries,
including
neural
profits,
which
is
a
pytorch
neural
network
forecasting,
Library
based
on
profit
and
linkedin's
gray
card
Library,
could
work
better.
A
In
summary,
we
use
tamlan
to
generate
a
weekly
capacity
planning
report.
This
capacity
plan
is
tightly
coupled
with
our
engineering
scheduling
processes,
the
same
metrics
that
we
use
for
short-term
resource
utilization
monitoring
and
alerting
also
get
used
for
long-term
forecasts,
thanks
to
our
ability
to
retrieve
them
from
Thanos.
A
Profit
is
a
very
useful
tool
which
makes
forecasting
very
easy.
I
would
highly
recommend
you
give
it
a
try
for
forecasting
if
you're
interested
in
exploring
further
here
are
some
links
to
the
timeline
source
code.
Unfortunately,
the
report
itself
is
no
longer
public
due
to
restrictions
in
publishing,
forward-looking
statements.
Now
that
gitlab
is
a
public
company,
but
all
the
source
code
is
still
available
and,
of
course,
if
you're
interested
in
this
sort
of
thing
please
reach
out,
we
have
a
large
number
of
vacancies
open
in
the
infrastructure
team
and
that's
the
talk.