►
Description
How We Built a Chargeback System for Twitter’s Compute Platform (Mesos/Aurora) That Incentives Higher Resource Utilization [I] - Jeyappragash Jeyakeerthi, Twitter Inc.
Twitter is powered by a collection of diverse, multi-tenant infrastructure and platform services. We run Mesos/Aurora for general compute, Manhattan for storage, Hadoop for batch compute and many more. We also run many platform services and one such example is our custom observability (monitoring) service.
Given Twitter scale, these services should always be available, reliable, scalable, well utilized, cost effective and flexible. The multi tenant nature of these services brings increased management complexity and a host of challenges, for both the service owner and end-users. Here are a few, that we will be covering (along with uses cases) during the first half part of the talk:
- resource discovery
- capacity planning
- resource ownership
- resource provisioning through self-service
- resource utilization and cost tracking
In the next half, we will talk about our solution i.e the Cloud Platform Management System: A simple way to catalogue, identify, plan, provision, meter usage and chargeback resources of multi-tenant infrastructure services. We will also share a case study on how we improved the overall utilization of the general compute platform (Mesos/Aurora) after being on-boarded into the Cloud Platform Management System. Finally, we will share our vision for the next generation cloud platform for multi-tenant services.
About Jeyappragash Jeyakeerthi
Jeyappragash leads Twitter’s Cloud Infrastructure Management Platform. This hybrid cloud platform helps developers manage and launch their services providing detailed visibility about resource usage and for infrastructure owners, to offer their infrastructure or platform as a true multi-tenant offering. Previously he worked as a Distinguished Engineer at Motorola, leading efforts to build their Notification Infrastructure, their Software Upgrade services and a Prospective Search based Content Delivery Service. Jeyappragash graduated from IIT Madras with a Masters in EE.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
And
then
what
is?
What
is
the
one
that
connected
between?
So
the
key
keating
that
I'm
tryna
convey
in
this
talk
is
about
what
what
is
important
in
thinking
to
oppose
when
you
break
down
a
service
in
to
find
Jane
services
and
what
is
required.
What
is
required
for
running
a
multi-tenant
service
that
coexist
with
other
multi-tenant
services
and
then
along
the
way?
We
will
talk
about
service
identity,
which
is
one
of
the
key
binding
blocks
between
running
the
service
on
a
multi-tenant
service.
So
I
couldn't
needs
no
introduction,
it's
a
network
of
now.
A
That's
where
you
go
to
learn
that
your
privacy
is
lost
as
of
like
two
days.
Back
so
I
mean
twitter
is
a
platform.
It's
very
open.
People
use
it
for
multiple
different
reason
and
we
just
go
through
a
little
bit
of
a
history
on
twitter.
To
just
give
you
a
perspective
of
why
we
did
all
the
infrastructure
changes
that
we
did
behind
the
scenes.
A
So
we
still
kept
building
the
platform
we
optimized
and
then
there
was
a
speed
from
ln,
the
selfie
from
oscar,
which
was
like
the
most
retweeted
tweet,
of
course,
that
then
go
without
a
glitch
and
they
followed
through
with
a
tweet
saying
they
are
sorry
about
the
output
and
the
next
biggest
peter
is
like
one
from
brazil.
Germany.
There
was
like
this
had
a
decided
eat
a
total
feet
of
six
hundred
and
sixty
thousand
feet
per
minute
somewhere
around
that
change.
A
I
think
the
number
here
is
wrong,
but
it's
still
in
our
blog
post,
where
you
can
go
check
it
out
in
terms
of
how
how
we
did
and,
of
course
the
zing
go
without
a
glitch.
There
are
issues
with
that
that
we
act
of
faith
and
we
still
kept
improving
the
platform
and
then
I
think
toad's
februari
of
2016,
when
Leonardo
DiCaprio
God
has
Oscar.
Finally,
I
think
we
had
about
four
hundred
and
forty
thousand
beats
per
minute
and
it
went
to
Lake
smooth,
but
people
externally
didn't
know.
A
A
So
in
February
of
2010,
we
had
this
issue
and,
like
any
other
application
services,
we
went
into
optimizing
the
head
code
of
the
application
trying
to
get
the
maximum
CPU
maximizing
the
CPU,
maximizing
the
memory
and
then
including
the
I/o
that
you
have
and
throw
that
coat.
The
things
you
will
see
that
the
setting
up
I
mean
optimizing
on
server
utilization
was
like
a
high
priority
in
one
form
or
the
other.
A
So
this
one
was
about
memory,
management
and
cpu,
like
kind
of
maximize
the
memory
and
CPU
within
all
of
the
services
in
Twitter.
It's
primarily
optimized
cash.
Optimization
happened
here
and
the
next
was
paralyzing
the
workload
that
was
then
using
the
unicorn
framework,
which
is
still
on
Ruby
on
Rails
and
yes,
along
Ruby
and
then
along
the
way.
We
migrated
out
to
java
based
scholar
based
systems,
but
then
you'll
see
here
the
world
cup
tweet
that
generated
a
blog
post,
so
every
blog
post
that
gives
to
get
generated.
A
That
time
was
about
like
why
we
fail.
This
is
how
we
failed,
and
this
is
so.
This
is
what
we
are
doing
to
fix
and
we
kept
going
on
in
terms
of
turning
pro
improve
the
platform
improve
the
utilization
of
the
service,
so
the
user
base
still
kept
growing,
and
then
we
grow
out
of
capacity
in
Arcola.
So
we
did
the
move
to
our
own
data
center
and
started
off
in
july
of
2010,
with
a
plan,
and
in
a
year
in
a
little
over
a
year.
I
think
we
did.
A
We
did
a
really
smooth
migration
over
to
the
data
centers,
so
this
one
I
mean
ask
the
user
best
kept
growing.
It
kept
outpacing
the
pace
at
which
we
are
improving
the
platform
and
then
improving
the
applications
that
run
on
the
platform,
and
by
this
time
here
we
are
like
way
into
building
fine
grain
services
and
then
each
fine
grain
services
that
you
build
is
going
to
have
the
same
set
of
problems
in
terms
of
scaling
and
scale
management,
and
we
did
build
finagle
and
smaller
sensors.
A
So
that's
a
that's
an
RPC
framework
which
abstracts
away
all
your
problems
for
dealing
up,
dealing
with
ex
scaling
your
services,
whether
it's
supplying
back
pressure
or
talking
to
another
service
or
trying
to
find
out
where
your
next
service
is
located.
So
all
those
problems
are
dealt
with
that
clinical.
So
while
this
is
still
happening
and
tryna
make
a
plan
to
make
writing
services,
easy
writing
services
be
more
scalable.
A
A
So
now
you
get
any
luck
with
relocatable
workload,
which
is
called
containers,
know
so
I
think
it's
a
term
that
now
many
people
are
familiar
with
and
many
people
associate
with
utilization
of
server,
but
it
was
primarily
to
facilitate
workload
migration
at
that
time,
and
it's
a
pretty
good
hardware
abstraction
that
I
think
many
people
now
know
about,
so
that
so
all
those
things
happened
and
now
now
we
have
large
need
of
servers.
Several
hundred
thousands
of
servers
and
I
think
more
than
more
than
forty
percent.
I
think
line
renaud
may
so
flustered.
A
So
this
is
this
keeps
expanding,
and
then
we
will
still
keep
expanding
on
the
server
capacity,
but
we
still
keep
working
on
optimizing.
Our
platforms
and
infrastructures
that
support
support,
running
services
and
the
other
example
where
we
have
hair
on
which
is
a
highly
optimized
stream
processing
engine,
and
we
recently
the
least
open
source
version
of
this.
A
So
what
you'll
notice
is
like
the
primary
objectives
for
hardware
infrastructure
is
maniac
lee
optimized
for
cost
like
finds
the
cheapest
possible
processors,
find
cheapest
possible
hardware
and
make
it
more
beam
available
and
more
efficient
and
for
platform
such
as
the
primary
objective
that
the
optimized
works
is
keeping
the
SL
laser-like.
Trying
to
build.
The
multi-tenant
systems
can
optimize
to
make
sure
that
it
sends
on
any
kind
of
flaky
hardware
and
that's
their
primary
objective
for
platform.
Infrastructure
and
application
framework
will
heavily
optimized
towards
making
the
developer
productivity
higher.
A
So
there
are
a
lot
of
definition
about
fine
grain
service
model,
but
I
will
try
to
give
my
version
of
it
when
you
start
off
at
the
monolithic
service
and
try
to
break
it
down.
Your
expectation
is
sort
of
like
you'd,
get
a
really
clean
interface
and
then,
let's
sort
of
a
beard
like
this,
where
you
can
actually
talk
to
each
service
talk
to
each
other
and
everything
is
understandable
and
not
cleaned.
But
what
you
really
end
up
with.
A
Is
this
you're
going
to
get
thousands
of
services
each
talking
to
each
other
and
you're
going
to
have
to
manage
this
mess
in
one
form
of
the
other?
So
there
are
tools
that
will
get
built.
There
is
cataloguing
system
and
everything
that
will
get
built
around
the
around
just
to
manage.
This
complexity
know
that
we
have
created
because
of
Michael
because
of
fine
grain
services
bit,
but
this
is
essential,
of
course,
for
improving
resiliency,
improving
developer
productivity
and
then
I
think
that
needs
no
explanation.
No
I
think
many
of
you
are
familiar
with
that.
A
Obviously,
you
can
go
crazy
and
then
make
every
function
call
service,
but
it
needs
to
be
broken
down
into
borrowing
from
the
genius.
So
it's
a
service
services
should
be
broken
down
to
a
granularity,
that's
as
fine
as
possible,
but
no
finer
and
that's
a
hard
line
to
draw
and
I
think
everybody
who's
done.
Microservices
would
have
gone
through
this
iteration
of
their
all
in
one
form
of
the
other
like
when
do
I
break
the
service
is
down
so
they
how
about
the
principles
are
like
isolation
and
security.
A
Boundary
is
one
thing
that
you'd
have
to
really
pay
attention
to.
So
if
there
is
a
cell,
if
there
is
a
service
and
that
deals
with
the
data
that
you
wanted,
that
service
to
deal
with
that,
that
should
not
leak
into
other
services,
so
you
should
contain
any
part
of
services
that
deal
with
some
part
of
data
that
you
think
needs
to
be
isolated
and
kept
needs
to
be
in
a
single
cell.
So
you
don't?
You
want
to
bake
them
into
multiple
services.
A
Then
you'd
have
to
deal
with
like
these
source
permissions
and
granting
permissions
between
these
services.
That's
going
to
be
harder
when
you
like,
take
a
model
when
you
take
a
big
code
base
and
then
you
try
to
walk
through
debug
through
your
performance,
you're
going
to
interpret
like
some
hot
spots
in
the
code,
and
you
can
easily
at
that
time
say
that
that's
a
piece
that
needs
to
be
broken
down
into
the
service,
so
that
can
be
independently
scale
degree
less
without
having
to
worry
about
the
scale
of
these
out
take
services.
A
So
that's
one
thing
that
I
think
cez
want
for
people
to
think
too
and
I
understand
that
that
needs
to
be
broken
down.
But
you
do.
I
do
see
a
lot
of
people
tripping
around
the
isolation
and
security
bounty.
I
think
that's
like
one
of
the
hard
harder
problems,
the
other
thing
that
that
we
need
to
pay
attention
to
is
fault,
localization
and
isolation,
so
fault,
localization
and
isolation
or
two
different
parts
of
the
same
problem:
the
effective
isolation.
We
call
the
stresses
like
the
impact.
A
The
thing
that
you
do
is
lick
localization,
so
any
fault
that
you
do
you'd
have
to
click,
consume
consumers
within
your
service
and
then
respond
in
a
way
that
you
don't
actually
fail.
These
calling
service
or
if
you
call
a
service
and
then
you
detect
like
some
measure
and
then
you'd
have
to
like
gracefully
shell
government
fails
so
you'd
have
to
try
hard
to
keep
your
false
local
for
as
Finnegan
as
a
framework
provides
that
ability
like
quite
a
bit.
So
we
rely
on
Finnegan
quite
a
quite
a
bit
on
that.
A
A
So
we
did
like
a
rough
assumption,
4002
even
figure
out
if
it's
a
chargeback
as
a
virtual
project
to
go
about,
and
then
we
put
together
trying
to
find
out
the
total
cost
for
what
we
would
end
up
saving
if
we
did
charge
practices-
and
this
is
not
used
in
current
TCO
model.
Yet
but
a
version
of
some
form
or
some
version
of
this
is.
A
To
the
multi-tenant
infrastructure
that
supports
renault,
a
tenant
infrastructure
is
key
to
supporting
fine
grain
services.
The
third
multi-tenant
infrastructure,
fine
grain
services,
asleep
almost
pointless
I-
mean
I.
Guess
you
could
use
like
a
fine
Jane
service
on
like
a
real
emo
nah
non
multi-tenant
service,
but
you
don't
get
there
and
if
occasionally,
you
don't
get
the
identity
propagation
over
to
the
platform
services
and
then
like
cracking
auditing
and
everything
becomes
harder
or
even
charging
back
becomes
harder.
A
So
proper
layered
multi-tenant
system
is
something
that
could
have
a
full-on
or
hardware
infrastructure
that
has
the
container
and
a
vm
and
RP
supported
by
our
PC,
and
you
build
platform
epa's
off
of
the
car
pc
framework.
And
then
you
have
an
application
api
that
you
build
on
top
of
platform.
Api
is
that
you
can
expose
to
the
hindus
wraps.
A
If
you
are
living
in
this
world
the
talking's
here
and
then
we
are
all
happy,
but
the
variations
in
layers
is
like
quite
quite
a
bit.
I
mean
quite
complex,
so
you
find
like
some
infrastructures
that
are
running
on
our
pc
framework
powered
by
platform.
Apa's
some
are
running
in
containers
on
without
RTC
frameworks,
but
it
will
just
use
a
platform
with
PA
and
then
there
are
platform,
a
page
that
will
be
built
directly
on
top
of
hardware.
This.
A
These
are
called
dedicated
services
in
our
world,
but
you
have
you
have
variety
of
this
and
now
we'd
have
to
deal
with
I
mean
you
cannot
know,
localize
your
problem
of
service
identity
to
just
either
the
finagle
framework
or
container.
So
now
you
have
to
understand
all
layers
of
stack
and
how
you
actually
propagate
the
identity
for
like
varying
types
of
multi-tenant
infrastructure
and
have
you
how
you
identify
them,
how
you
deal
with
them?
A
There
are
few
principles
that
that
facilitate
a
proper
multi-tenant
neighbors
and
let's
talk
about
little
little
little
more
about
like
the
complexity
around
critters
infrastructure
excel.
So
we
have.
We
have
a
lot
of
multi-tenant
infrastructure
within
critter.
They
see
each
of
them
have
their
own
identity,
identity,
way
to
identify
clients
if
they
have
a
way
to
identify
clients
and
some
have
stronger
dedications
some
object.
Basically,
I
have
an
AP
a
go
at.
A
It
call
do
whatever
you
want
rate
and
authorization
is
inconsistent
and
most
of
the
systems
don't
do
like
a
fine-grained
of
the
authorization,
even
if
they
do
and
that's
that's
an
inconsistent
implementation
and
I
am
guessing
many.
Many
people
that
are
going
to
have
to
deal
with
like
polyglot
of
multi-tenant
in
infrastructure,
I
have
to
go
through
this
in
EV
and
rate
limiting
or
color
management
is
all
over.
A
The
map
like
some
are
some
know
how
to
limit
and
isolate
that
isolate
the
rates
and
Kota
total
I
a'n't
and
some
actually
don't
come
infrastructures,
don't
even
care.
So
we
have
plethora
of
multi-tenant
infrastructures.
I
mean
these
methods
have
made
those
offers
like
multiple
offerings.
You
have
like
a
proud
offering
in
a
spot
instances,
type
of
offering
and
Hadoop
has
its
own
talk
and
cold
store.
A
So
you
have,
you
have
choices
to
pick
which
ones
you
want,
but
you
don't
necessarily
are
given
any
guidance
in
terms
of
this
is
or
this
is
how
you
pick
each
and
one
of
the
things
that
happened
as
a
result
of
chargeback
was
we'd,
be
able
to
provide
them
with
the
real
cost
in
real
dollars
to
see.
If
you
pick
this,
this
is
what's
going
to
be
the
impact
which
most
for
most
of
you,
working
on
clothes
given
and
taken
for
granted
link.
You
already
know
what
your
unit
unit
prices
are
going
to
be.
A
We
have
infrastructure
is
consuming
other
infrastructures,
which
is
sort
of
like
this,
so
we
so
these
are
the
complexities
that
that
are
that
exist
and
trying
to
have
a
polyglot
of
multi-tenant
services
that
are
available
for
building
on
microservices,
so
we'll
go
a
little
bit
deeper
into
what
the
security
model
modulus
and
before
that
I
just
wanted
to
go
over
the
understanding
of
what
security
exercise.
I
mean
security
is
like
completely
overused
and
abused
these
days,
at
least
that's
what
I
feel
so.
A
People
use
security
when
they
want
to
talk
about
authentication
people,
your
security,
and
there
is
going
to
say
authorization,
enforcement
and
many
times.
Protection
is
completely
confused
for
security
when
you
say
like
a
system
that
has
the
same
cryptid
and
data
stored
at
rest
is
the
protected
system.
But
it's
not
a
secure
system.
I
mean
unless
you
have
the
ability
to
say
that
the
person
who's
allowed
to
access
the
data
is
the
only
person.
That's
allow
that's
accessing
the
data,
it's
not
a
fully
secure
system.
It's
so
and
same
thing
with
privacy
and
security.
A
Is
security
essentially
provides
trust
like
you
just
basically
trust
the
system.
If
it's
a
secure
system,
you
just
trust
the
system,
you
do
what
you
are
required
to
do,
and
then
the
system
just
does
what
it.
What
is
the
right
thing
for
you?
So
at
the
core
of
security
is
service
identity.
So
these
are
all
tropics
by
itself.
There
is
a
lot
of
let
each
one
can
have
have
a
one-hour
talk
or
20
a
talk
by
itself.
A
So
what
I'm
trying
to
do
is
cut
to
everything
to
get
the
service
identity
as
a
problem,
and
then
we
can.
We
can
delve
down
into
a
little
bit
deeper
on
that
right,
so
and
I
want
to
leave
more
time
for
Q&A.
So
it
will
be
a
much
more
interesting
conversation
because
I
can
mean
cater
to
what
you
want
to
hear,
rather
than
giving
you
like
a
broad
overview.
So
service
identity
is
one
key
problem.
A
A
A
Identity,
I
highly
encourage
you
to
go.
Take
a
look
at
spiffy:
that's
not
what
we
do
a
quitter,
but
that's
like
close
enough
that
you
can
follow
for
your
production
environment
that
will
taught
through
so
so
what
do
we
get
by
solving
identity?
I
mean,
obviously
you
get
security
which
solves
authentication,
authorization
and
enforcement
and
accountability
which
helps
you
solve
Kota
budget
and
charge
back,
and
it
provides
you
auditability
for
compliance
and
transparency.
A
So
these
are
the
things
that
you
get
by
solving
service
identity,
I'm
not
going
to
go
into
how
you
actually
solve
service
identity.
If
anybody
is
interested,
I
can
talk
about
it
for
hours
with
you,
but
but
what
I
wanna
highlight
is
that's
one
key
thing
that
I
think
we
need
to
think
through
in
terms
of
there.
Isn't
it
doesn't
a
complete,
valid
exact
solution,
that's
out
there
and
at
the
end
of
it.
A
A
A
Multi-Tenant
services
should
basically
be
it
should
be
able
to
authenticate,
provided
you
have
a
valid
token,
but
it
is
not
the
thing
that
is
going
to
store
your
user
ID
any
time
any
system
has
a
user
database
within
it
then
you've
already
like
broken
the
contract
of
like
a
good
name,
multi-tenant
system,
because
you
now
have
to
deal
with
how
you
need
hope.
You
do
translations
from
one
identity
that
that
the
service
owns
to
trying
to
create
a
user
account
that
you
try
to
map
to
that
system
right.
A
So
you
don't
want
to
deal
with
that
mess
and
the
other
biggest
stuff.
Is
you
want
to
externalize
the
policy
management
externalize
the
authorization
decision?
How?
But
the
system
should
basically
deal
with
enforcing
that.
So
how
do
you
generate
your
auth
token?
Doesn't
really
matter,
but
the
fact
that
the
auth
token
is
exists
and
is
allowed
to
do
something
is,
is
the
logic
that
that
should
meet
that
should
exist
with
the
multi-tenant
service
and,
of
course,
maintaining
qos
keywords
is
important
it.
A
This
is
sort
of
like
I
mean
when
you
go
buy
something
from
farmers
market.
You
don't.
Actually
you
take
your
you,
don't
give
your
credit
card
out
to
that
person
or
that
could
that
person
doesn't
issue
you
credit
card
for
a
physical
store
that
you
go
to
late,
so
you
take
it
the
record.
Your
authentication
happens
somewhere
else,
and
all
he
needs
to
know
is
you
have
five
dollars
in
your
credit
card
that
I
can
use
because
you
bought
this
item
plate.
A
So
that
is
the
level
of
that's
a
level
of
loose
loose
coupling
that
you'd
have
to
get
to
to
to
allow
really
coexisting
multi-tenant
neighbors.
So
these
problems
need
to
be
externalized
solved
externally,
and
then
the
interfaces
need
to
get
defined,
much
more
clear,
which
does
not
exist
right
now,
but
I
think
we
are
you're
slowly
getting
there.
A
So
the
heat
take
away
in
verses
for
any
large-scale
enterprise
that
you're
dealing
with
the
large
systems
of
servers,
you're
going
to
be
many
actually
focused
in
trying
to
maximize
resource
usage,
and
it's
not
necessarily
minimizing.
It's
like
trying
to
use
the
workload
to
provide
the
value
at
the
at
a
cost,
efficient
ways
right
so
and
all
the
hardware
platform
and
application
together
will
drive
towards
maximizing
value
from
a
workload.
But
while
minimizing
costs,
a
good
multi-tenant
system
is
key
to
enabling
secure,
reliable
and
available
infrastructure.
A
A
The
most
important
stuff
is
whether
it
whether
you
are
running
it
on
Prem
or
whether
you
are
running
it
on
prim
plus
one
other
cloud
and
I
hate.
The
word
flow
portability,
I.
Think
cloud.
Interoperability
is
a
much
more
practical
thing
to
do.
The
include
portability
you're
not
going
to
be
able
to
put
a
function
from
one
close
to
the
other.
That
is
again
like
if
anybody
is
interested
in
topping
I
talking,
I
can
talk,
but
what
you
really
need
to
enable
cloud
portability
is
again
going
back
to.
A
You
need
like
a
loosely
defined
set
of
taught
to
protocols
that
can
actually
enable
seamless
functioning
between
your
on-prem,
our
cloud
or
on-premise,
multiple
clothes
so
and
I
didn't
it
starts
from
identity.
Then
you
build
the
access
management
layer
on
top
and
those
all
should
be.
An
external
externalized
system
compared
to
the
quality
or
multi-tenant
system
is
so
that's
mostly
what
I
had
available
for
sessions
and
thanks
thanks
for
me.
Thanks
for
listening.