►
From YouTube: How We Built a Framework at Twitter to Solve Service Ownership & Improve Infrastructure Utilization
Description
How We Built a Framework at Twitter to Solve Service Ownership & Improve Infrastructure Utilization at Scale [I] - Vinu Charanya, Twitter
Twitter is powered by thousands of microservices that run on our internal Cloud platform which consists of a suite of multi-tenant platform services that offer Compute, Storage, Messaging, Monitoring, etc as a service. These platforms have thousands of tenants and run atop hundreds of thousands of servers, across on-prem & the public cloud. The scale & diversity in multi-tenant infrastructure services makes it extremely difficult to effectively forecast capacity, compute resource utilization & cost and drive efficiency.
In this talk, I would like to share how my team is building a system (Kite - A unified service manager) to help define, model, provision, meter & charge infrastructure resources. The infrastructure resources include primitive bare metal servers / VMs on the public cloud and abstract resources offered by multi-tenant services such as our Compute platform (powered by Apache Aurora/Mesos), Storage (Manhattan for key/val, Cache, RDBMS), Observability. Along with how we solved this problem, I also intend to share a few case-studies on how we were able to use this data to better plan capacity & drive a cultural change in engineering that helped improve overall resource utilization & drive significant savings in infrastructure spend.
About Vinu Charanya
Vinu Charanya is a Senior Software Engineer at Twitter where she works in the Compute Platform building Twitter’s internal cloud infrastructure management platform. She is also a core team member of Women who code, a non-profit organization dedicated to inspiring women to excel in technology careers. She is also a part-time teacher and a mentor helping students learn iOS and Android development at CodePath. Vinu received her Masters in Computer Science and Engineering from University at Buffalo, where she worked on the PhoneLab Testbed research group under Prof. Geoff Challen and Prof. Steve Ko.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
My
name
is
vino
and
I'm.
A
software
engineer
on
the
cloud
infrastructure
management
team
at
Twitter,
I'm
also
proud,
being
a
member
of
Twitter
women,
tech,
women
and
women.
A
code
today,
I'll
be
talking
about
how
we
built
a
set
of
tools
and
processes,
a
framework
to
improve
the
utilization
of
infrastructure
resources.
I'll
start
by
sharing
with
some
history
and
context
around
the
Twitter
stack
and
deep
dive
into
Twitter's
chargeback,
a
service
that
helped
gain
visibility
and
build
accountability
around
infrastructure
resource
utilization.
A
We
learned
a
ton
while
building
charge,
FAQ
service
and
I'm
super
excited
to
share
more
details
on
the
next
iteration
of
it
called
kite
our
service
lifecycle
manager.
It
aims
to
address
some
fundamental
challenges:
around
cracking,
managing
and
ownership
attribution
of
resources
in
a
dynamic
environment
where
micro
services
or
deployed
at
scale
I'll
close
out
with
some
impact
and
future
work
in
both
kite
and
charge
pack,
let's
start
with
history
and
context.
A
If
you
didn't
already
know,
Twitter
migrated
from
a
large
monolithic
rail
service
into
micro
services
to
address
the
developer
and
reliability
productivity
concerns.
Then
this
chart,
although
a
bit
old,
is
a
great
representation
of
the
service
graph
that
powers
Twitter
and
its
associated
products.
What
you're
seeing
here
is
the
RPC
trace
for
a
request
to
create
a
tweet.
The
the
highlighted
service
is
our
ingress
service
and
reverse
proxy
called
TFE.
A
This
routes,
the
request
to
numerous
downstream
services,
which
further
fan
out
to
even
more
services
if
it
hasn't
already
crossed
your
mind,
twitter
has
a
lot
of
services
when
I
joined
the
company.
I
was
at
off
at
the
breadth
and
the
depth
of
the
stack
over
time.
As
my
team
and
I
worked
on
resource
occasion,
utilization
and,
more
importantly,
ownership
attribution
to
these
kazillion
services,
I
transition
to
more
or
less
of
this.
A
Anyways
twitter
has
invested
heavily
both
in
in-house
and
open-source
technologies
that
power
its
core
infrastructure
parameters,
such
as
compute
storage,
databases,
etc.
In
fact,
the
focus
has
been
in
enabling
multi-tenancy
right
from
the
start.
This
introduces
unique
resource
allocation
capacity,
planning
and
ownership
challenges.
A
Just
to
close,
the
thought
here
is
a
slide
that
captures
at
high
level
Twitter's
platform
and
infrastructure
world
at
the
bottommost
layer.
We
have
the
data
center
management,
we
run
mesas
aurora
for
long
running
services,
hadoop
for
batch
compute
and
use
Manhattan
for
key
value
storage.
These
services
make
up
core
infrastructure
primitives
moving
up
the
stack
we
have
services
such
as
cache,
observe
leti,
messaging,
etc,
built
on
top
of
these
core
primitives.
A
So
it
shouldn't
be
a
surprise
that
these
services
pretty
much
make
2/3
of
our
entire
server
footprint.
Twitter
runs
in
multiple
data
centers
and
has
hundreds
and
thousands
of
service
servers
planning
capacity
for
such
large
services
is
a
challenge
by
itself.
Given
the
footprint
and
complexity
around
multi-tenancy,
we
started
asking
ourselves
how
to
get
visibility
into
the
footprint
of
individual
jobs
and
datasets
within
these
shared
platforms:
how
to
attribute
resource
consumption,
to
teams
or
organizations
how
to
incentivize
the
right
behavior
to
improve
efficiency
of
resource
usage
in
these
platforms.
A
So
one
of
the
first
challenges
for
my
team
was
to
model
a
resource
catalog
for
a
set
of
disparate
infrastructure
services.
It
became
clear
that
there
was
a
need
for
a
consistent
way
to
identify
an
inventory
these
resources.
A
resource
in
this
context,
means
either
of
physical
or
virtual,
an
entity
that
is
consumed
by
applications,
thus
driving
the
overall
capacity
of
the
underlying
infrastructure
or
platform
service.
A
We
had
to
support
resource
fluidity
with
the
utility
computing
model.
The
resources
are
no
longer
a
permanent
entity,
both
from
infrastructure
or
service
owners
perspectives.
It
is
much
easier
to
model
capacity
and
understand
cause
when
everything
was
running
under
bare
my
bare
metal
machines.
In
the
current
service-oriented
architecture,
we
need
to
treat
resources
in
a
more
granular
way,
break
them
down
to
primitives,
like
CPU
RAM
memory,
stored,
reads
or
writes
per
second
etc
and
create
abstract
resources
as
we
go
higher
up.
A
The
stack,
for
example,
tweets
per
second
infrastructure
resources
evolve
over
time
and
our
model
captures.
This
second
challenge
is
part
of
supporting
a
diverse
set
of
infrastructure
services
is
the
me
to
track
client
and
ensure
their
map
to
owners
without
correct
ownership.
It
is
difficult
to
ensure
accountability.
I
talk
about
how
we
solve
this
problem
later
during
the
talk.
A
Let's
look
at
the
entity
model,
the
unit
price
of
a
resource
is
captured
and
offering
measure
cost.
In
our
case,
we
have
a
process
to
review
and
change
the
unit
price
per
quarter.
An
offering
measure
captures
the
resource
itself,
example
course
per
day
or
service
per
day,
etc.
This
alone
cannot
define
what
we
are
truly
measuring
and
costing
out.
There
is
a
need
for
context
around
the
resource
in
the
form
of
offering
infrastructure
service
and
providers
providers
can
represent
either
internal
data
center
or
public
cloud
providers.
A
Here
is
an
example
of
Aurora
Aurora
offers
compute,
which
is
measured
in
Coe
days,
where
compute
is
the
offering
in
this
case.
On
the
contrary,
an
infrastructure
like
Hadoop
offers
clusters
that
store
and
process
data
as
separate
offerings
with
measures
such
as
GB
RAM,
number
of
accesses
per
day,
etc.
We
were
able
to
onboard
other
infrastructures
like
Manhattan
blobstore,
in
a
similar
way,
as
her
entity
model
can
represent
these
disparate
infrastructures
in
a
consistent
way.
A
A
A
A
So
here
is
an
example
of
how
we
price
Twitter's
compute
platform
for
Aurora.
We
started
defining
unit
cost
from
the
bottom
up,
that
is,
from
the
bare
metal
layer
where
we
incurred
true
cost
to
buy
physical
machines.
We
calculated
cost
per
server
per
day,
which
includes
capex
and
OPEX,
cost
of
machines
license
Headroom
utilities,
inefficiencies
and
the
human
cost
to
operate
that
resource.
It
is
important
to
keep
in
mind
that
TCO
evolves
over
time
to
account
for
depreciation
in
order
to
track
these
changes.
A
Our
entity
model
is
designed
to
capture
these
unit
prices
with
a
time
varying
dimension.
These
form
the
basis
to
help
calculate
other
service
unit
prices.
As
we
move
up
the
stack
next,
we
looked
at
the
total
used
course
in
Aurora.
This
is
a
combination
of
both
cores
used
by
production
and
non
production
jobs.
We
use
the
default
container
stat
matrix
exported
by
missiles.
A
Next,
we
account
for
excess
course
that
are
due
to
incorrect
container
sizing
and
extra
quota
allocated
to
account
for
say,
traffic
spikes.
The
goal
here
is
to
incentivize
folks
to
right-size
their
jobs
and
ensure
they
are
requesting
enough
quota
and,
if
possible,
tax
them
for
holding
excess
course.
A
Finally,
the
unit
price
also
accounts,
for
course,
used
by
the
compute
team
themselves
for
testing
operation
and
maintenance.
We
treat
this
as
a
general
tax
that
is
equally
distributed
across
the
pricing
of
all
the
charge.
Backers
I
know
that's
a
lot
of
components
to
consider
for
a
unit
price
computation,
but
we
wanted
to
ensure
that
that
we
keep
the
cost
true
to
the
model
and
not
some
funny
money.
We
charged
out
to
customers
to
ensure
resource
utilization.
The.
A
We
worked
with
every
infrastructure
service
owner
to
export
a
resource
usage
data.
These
are
classic
ETL
pipelines
that
extract
data
from
different
sources.
Example:
Aurora,
scheduler,
Hadoop
master
that
are
transformed
and
are
loaded
into
a
generalized
chargeback
schema
called
the
raw
fact.
Our
contract
is
pretty
simple.
The
raw
fact
expects
the
client
identifier
offering
measure
volume
and
additional
metadata,
such
as
job
environment
or
directory
paths
when
it
comes
to
file
accesses
and
the
timestamps.
A
This
is
now
transformed
by
the
data
transform
service,
whose
job
is
twofold:
one.
It
resolves
ownership
of
a
client
identifier
from
a
raw
fact
to
an
appropriate
team
or
organization
and
number
two.
It
uses
offering
measure
and
volume
to
compute
total
cost.
The
transform
data
is
then
stored
in
a
result
fact
table
before
consuming
by
the
reports.
A
A
Thus
we
invested
in
a
system
early
on
that
looks
at
the
data
and
sends
out
alerts
when
there
is
missing
data
and
our
huge
deviations
of
usage
or
cost.
The
system
is
currently
being
extended
to
do
notification
to
service
owners
directly
when
a
given
project
shows
huge
deviation,
so
they
can
either
take
immediate
action
or
to
resolve
any
issues.
The
result
fact
is
a
centralized
chore
that
is
used
to
generate
a
variety
of
reports.
A
We
have
a
numerous
internal
customers
and
each
have
different
requirements.
For
instance,
the
infrastructure
service
owner
required
information
on
overall
cluster
growth
and
Kline
growth
over
time
to
help
plan
capacity.
They
also
use
this
data
to
figure
out
how
each
client
is
doing
in
terms
of
the
allocated
capacity
versus
utilization.
A
We
generated
P&L
reports
for
all
the
infrastructure
services
that
have
been
on-boarded
into
the
charge
vac
system.
This
shows
the
income
and
expense
per
infrastructure
service
and
gives
us
information
on
whether
they
are
efficient.
For
instance,
we
don't
want
these
services
to
make
a
profit
instead
be
close
to
a
zero
margin.
If
it
is
on
the
profit
side,
we
know
it
can
be
due
to
some
optimization
on
the
infrastructure
side,
subsequently
reducing
the
unit
prices
for
everyone's
benefit.
A
A
Here's
a
view
of
team
spend
per
infrastructure,
we
provide
detailed
bills
per
team,
but
org
on
the
various
platform
services.
Each
section
is
a
platform
service
with
offers
defined.
We
shared
usage
numbers
and
show
the
overall
cost
for
obvious
legal
reasons.
We
cannot
show
real
numbers,
although
this
gives
a
good
understanding
of
how
the
team
is
spending
on
each
infrastructure
service.
A
This
view
is
not
good
enough
when
you
wanted
to
work
on
optimization
projects,
so
we
created
a
drill
down
view
where
they
get
a
unique
view
of
the
project
level
spend
even
further
Delta
drill
down
to
client
identifiers.
So
you
see
you
have
an
A
cost
incurring
in
Aurora.
You
can
go,
find
the
exact
project
and
the
exact
job
that
is
incurring
the
cost
to
see
whether
is
it
necessary
or
not.
It
is
much
more
effective
for
engineers
to
know
the
exact
data
set
or
job
contributing
to
their
spend.
A
One
of
the
most
important
lessons
learnt
is
that
the
data
must
be
trustworthy.
As
mentioned
earlier,
the
charge
book,
the
chargeback
reports
are
used
for
a
variety
of
use
cases,
so
it
is
all
the
more
important
to
ensure
data
is
accurate
and
trustworthy.
We
were
also
able
to
extend
the
service
to
notify
when
there
is
data
inconsistencies,
for
example,
sudden
increase
or
decrease.
As
I
mentioned
earlier,
ownership
accuracy
of
client
identifier
is
extremely
important.
When
we
first
deployed
the
chargeback
system
for
a
physical
service,
over
60%
of
the
identifiers
were
attributed
to
unknown.
A
A
One
of
the
reasons
ownership
problem
is
hard
to
solve
is
because
of
the
lack
of
clear
grouping.
Primitives
ownership,
mappings
team
seems
to
course,
while
ownership
mapping
to
client
identifiers
seems
too
granular.
There
was
a
need
for
some
form
of
logical
group
owners.
A
project
is
a
logical
group.
We
decided
upon
so
and
it's
a
grouping
of
resources
such
as
services,
krons,
ad-hoc
jobs,
etc.
It
provided
an
easy,
manageable,
perceptible
unit.
A
Everything
changes,
as
mentioned
earlier.
Tco
evolves
over
time
to
account
for
depreciation
projects
and
identic
can
be
handed
off
to
other
teams
as
the
priorities
of
the
team
changes.
In
order
to
track
these
changes,
we
had
to
customize
the
entity
model
to
capture
the
change
history
of
resources
using
time
varying
dimensions.
A
A
This
includes
setting
up
a
project
creating
identify
O's
provisioning
certificates,
acquired
capacity
and
quota
to
use
infrastructure
services
and
finally
view
the
cost
and
usage
we
started
working
on
this
in
q4
of
last
year
and
as
of
today,
we
have
integrated
with
eight
infrastructure
services,
aggregated
10,000
identifiers
created
over
thousand
projects
that
are
used
by
over
hundred
teams.
The
kite
we
went
UI
serves
as
a
single
pane
of
glass
and
allows
every
engineer
in
the
company
to
manage
their
projects.
A
Their
client
accounts
for
infrasonic
wire
Kota
for
aurora
charge
back
exposed
us
to
the
difficulty
around.
Creating
and
managing
ownership
of
client
identify
us
/
infrastructure.
One
of
the
things
we
focused
with
kite
early
on
is
to
build
an
identity,
identity
system
that
explicitly
captures
org
structure
and
exposes
team
group.
Client
identifier,
us
into
projects
associate
team
with
projects
and
a
low
management
operations
on
these
entities.
A
To
expand
on
the
model
itself
from
the
bottom
client
identifier
saw
scoped
per
infrastructure
and
they
are
associated
with
a
service
account.
The
backend
that
powers,
the
service
account,
is
modeled
to
be
pluggable.
We
use
LDAP
groups.
Service
accounts
are
primarily
used
for
authentication
expected
to
be
eventually
used
for
authorization,
client,
identifier
or
large
logically
group
does
projects
projects
belong
to
a
team
which,
in
turn
belongs
to
a
cost
center
or
department.
Essentially,
some
entity
that
can
be
responsible
for
the
dollars
spent
here
is
an
example.
We
have
revenue,
which
is
one
of
our
cost.
A
Centers
and
one
of
their
teams
is
called,
adds
prediction
that
team
owns
several
services,
one
of
which
is
prediction
prediction
and
runs
under
the
service
account.
As
prediction,
a
service
can
have
multiple
service
accounts
based
on
the
access
control
needs
example,
one
for
every
different
environment.
A
The
same
model
can
support
all
the
infrastructures
beat
Hadoop
or
Manhattan
or
any
other
platform
intra
service
by
default.
Every
entity
is
time
varying
dimension.
Thus,
if
a
team
were
to
get
free
or
tomorrow
or
as
a
project
removed
from
team,
a
to
team,
be
the
entity
model
supports
each
of
this
chain
each
of
these
changes,
which
means
at
any
point
in
time.
As
long
as
we
have
this
data,
we
can
say
which
team
owned
this
project
and
what
was
the
cost
incurred
by
that
team
on
a
specific
day.
A
The
impact
through
hi
through
kite
has
been
enormous.
For
example,
kite
currently
manages
over
10,000
identifiers
here
is
a
split
by
infrastructure
service.
Clearly,
Hadoop
observed,
laity
or
monitoring
service
contains
the
largest
set
of
client
identifiers.
While
databases
has
a
small
set.
This
allowed
us
to
improve
ownership
significantly
and
reduce
unknowns.
A
A
The
identity
service
powering
the
kite
UI
enables
everyone
in
the
company
to
search
and
discover
owners
for
projects,
client
identify
us,
etc.
So,
if
you
are
a
site,
reliability
engineer
and
at
any
point
in
time,
you
want
to
identify
whom,
who
does
this
job
belong
to?
They
can
easily
find
out
from
Qaeda
chargeback
data
has
enabled
teams
to
understand
their
team,
spend
an
attribute,
cost
and
attribute
pressed
and
Crouse
to
actual
events.
A
For
example,
let
us
walk
you
through
some
of
the
things
our
team
was
able
to
improve
or
understand
over
the
period
of
time
about
a
year
back.
We
realized.
We
had
some
underutilized
resource
and
we
were
able
to
release
those
back
into
the
resource
pool
I
mentioned
earlier
unit,
price
updates
are
done,
and
that
is
the
depth
after
the
unit
price
update
in
q2.
A
A
If
you
wanted
to
more
graph,
if
you
wanted
more
granularity
than
the
team
level
to
know
how
exactly
a
particular
project
is
spending
over
the
period
of
time,
then
our
overview
page
has
project
level
spent
as
well.
So
the
idea
is
as
an
engineering
manager
or
a
tightly
you
care
about
the
team
and
how
the
team
is
performing
or
utilizing
the
resources
over
the
period
of
time
as
a
developer,
who's
working
on
one
or
two
projects
belonging
in
the
team.
A
They
can
keep
an
eye
out
for
how
their
project
is
performing
over
the
period
of
time.
We
recently
enable
quota
management
from
kite
for
jobs
running
in
Aurora.
A
user
need
not
go
to
different
user
interface
to
acquire
quota
for
their
production
service,
but
can
directly
acquire
this
from
kite
and
and
get
a
sense
of
how
much
it
costs
this
forces
developer
to
be
a
bit
more
conscious
when
they
design
and
plan
release
of
their
system
such
that
they
are
not
over
provisioning
resources
for
future
work.
A
After
the
success
of
Aurora
quota
manager,
we
plan
to
extend
to
include
Hadoop,
storage
and
other
system.
This
way,
service
owners
have
a
single
place
to
go
to
request
and
manage
quota.
The
other
advantage
of
this
is,
as
I
mentioned
earlier
right
now.
It
is
a
resource
creation
and
then,
after
the
fact,
come
claim
model,
but
with
provisioning
resources
right
from
Kight.
We
attribute
proper
ownership
to
the
client,
identify
us
right
from
the
start,
resulting
in
zero,
unknown
costs
and
a
conscious
decision
by
developers
when
requesting
any
form
of
resource.
A
The
second
one
is
project
depreciation,
similar
to
production
review
process.
It
is
important
to
have
a
proper
deprecation
review
process.
Kight
provides
a
list
of
resources
associated
with
the
project
and
with
a
chargeback
daily
usage
pipeline,
we
can
validate
and
notify
the
release
of
the
resources
back
into
the
pool
to
the
users.
We
want
to
build
a
process
based
on
a
deprecation
policy.
Why
is
this
important?
A
lot
of
time
think
about
running
an
Aurora
job
or
running
an
Aurora
service
that
has
cash?
A
You
go
request
Aurora
and
then
you
have
requested
cash
separately
and
when
the
time
is,
when
the
time
is
there
to
kill
the
Aurora
service,
you
may
never
have
released
your
cash
pool,
so
your
cash
is
assigned
to
you
unutilized
and
with
a
proper
project
deprecation
service.
We
now
know
all
the
resources
associated
with
the
project.
We
can
inform
you
of
what
resources
that
you
haven't
released
and
we
can
help
you
with
that
last,
but
not
the
least
capacity
planning.
Every
time
there
are
conversations
about
capacity
planning.
The
requests
are
collected
with
a
top-down
approach.
A
Service
owners
usually
add
some
extra
Headroom,
along
with
their
actual
request
at
the
end
of
the
day,
as
this
request
travels
down
and
comes
to
the
infrastructure
owners
and
when
they
receive
their
requests
and
go
to
the
capacity
planning
meeting.
We
have
this
all
those
extra
Headroom
requests
added
up
at
each
layer,
not
to
mention
the
extra
Headroom
added
by
the
infrastructure
service
owners
themselves.
The
outcome
of
this
is
over
provisioning
and
over
requesting
of
resources.
A
A
Although
we
were
able
to
find
underutilized
resources
without
proper
ownership
model,
we
weren't
able
to
attribute
or
act
upon
these
underutilized
resources
with
service
identity.
We
came
up
with
a
model
that
captures
the
organization
and
ties
these
resources
to
team.
Our
reports
had
their
customers
release
unused
resources,
including
our
own
team.
Our
current
model
is
claim
after
you
create.
We
want
to
minimize
this
effect
effort
through
resource
provisioning,
where
we
can
capture
the
ownership
at
the
time
of
creation
with
charge
pack
and
kite
ownership
model.