►
Description
Managing Lifecycle of Microservices at Scale [I] - Micheal Benedict, Pinterest
(This is an extended presentation of my lightning talk on "Managing micro-services at scale" from KubeCon/CloudNativeCon 2016 - https://youtu.be/Q1CEgPw7CG8)
As more organizations transition from a monolith to a more micro-services architecture, organizations are finding significant challenges around governance and lifecycle management of micro-services.
For example, how often have you (developer, ops, leadership) have asked one or more of the following questions?
1. What does it take to create and manage a new micro service? (Metadata Management, governance)
2. How do we identify a micro services canonically across infrastructure/platform services? (Identity)
3. How do we allocate resources for a micro service? (Resource provisioning)
4. What does it take to operate a micro service? (Deploy pipelines, orchestration, monitoring)
5. How do we measure resource utilization and cost of operating a micro service? (Metering and Chargeback)
These questions persist independent of an organization's container strategy or public/private cloud strategy.
Through this talk, I will deep dive further into the above challenges, the impact and share details on the need for a governance system that manages the lifecycle of micro-services. The talk will focus on the following areas:
1.Metadata Management (project info, team ownership info, operational info such as dashboards, alerts)
2. Identity Management (canonical service identifiers, secrets provisioning, distribution and management)
3. Resource Management (provisioning of primitive resources such as CPU, MEM or abstract resources such as RPS)
4. Metering and Chargeback
At the end of the talk, I'll share case studies from Twitter and Pinterest on how they implemented portions of these systems and its impact.
About Micheal Benedict
Micheal Benedict leads Product Management for Pinterest's Infrastructure Platform Teams. Previously he lead products for Twitter Cloud Platform building the next generation compute infrastructure that spans internal and the public cloud. He and his team built Kite - Service lifecycle manager and Infrastructure Metering & Chargeback system. Prior to that he was an engineer building systems that power Twitter's Observability and Monitoring stack. Micheal has a Masters degree in Computer Science from SUNY at Buffalo.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
Therefore,
the
internal
cloud
platform
building
the
Twitter
compute
platform
today
I
will
be
talking
about
micro
services,
lifecycle
management,
some
of
the
learnings
from
working
at
these
companies
on
how
to
build
and
scale
micro
services
up
and
more
specifically,
with
the
advent
of
the
public
cloud
where,
where
it's
almost
everywhere
available
the
availability
of
the
on-demand
resources,
it's
kind
of
critical
to
kind
of
understand
how
you
can
leverage
those
resources
with
some
set
of
governance.
I
think
one
of
the
biggest
problems
today
is
that
it's
it's
a
wild
wild
west
right.
There
is
no
structure.
A
There
is
no
process
around
how
you
can
leverage
these
resources
and
using
our
experience,
working
both
on
the
data
centers
and
the
public
cloud.
I
wanted
to
share
some
of
the
learnings
and
how
we
kind
of
try
to
tame
that
beast
of
managing
micro
services
having
some
governance
around
it
without
annoying
any
of
the
engineers
with
a
lot
of
set
of
processes
right.
So
that
is
kind
of
the
context
of
this
whole
talk.
A
I
will
describe
what
a
job
is
as
well
in
this
context,
moving
on
what
is
governance
and
the
challenges
around
that
when
you're
actually
running
you
know
the
hundreds
and
thousands
of
micro
services
and
finally,
the
future,
and
how
we're
actually
trying
to
solve
this
problem,
at
least
at
Pinterest
at
this
point,
so
how
many
of
you
have
actually
used
Pinterest
any
of
you?
Okay,
great,
so
it's
an
amazing
company,
I
recently
joined
Pinterest
and
I've
been
surprised
at
how
big
they
are
at
least
the
scale
of
their
systems.
A
Pinterest
is
a
it's
a
curated
list
of
ideas
across
the
world.
It's
a
collection
of
objects,
items
images,
videos,
bead
what
but
created
by
humans,
I
think
as
of
last
September,
they
have
about
75
billion
pins
pines'.
Their
you
know,
unit
of
resource
very
similar
to
how
tweets
are
for
Twitter
and
Sam
for
billion
is
a
pretty
big
number
and
one
of
the
key
things
of
the
key
challenges
is:
how
do
you
discover
these
bill?
How
do
you
discover
these
pins
within
this
huge
catalogue?
A
The
way
we
see
it
is
Pinterest
is
more
of
a
visual
discovery
search
engine
not
similar
to
not
similar
to
Google,
where
you
can
go
and
ask
a
question
and
get
a
very
objective
answer,
but
more
about
subjective
answers.
For
example,
how
should
I
decorate
my
house
or
what
is
a
recipe
to
cook
certain
things,
or
how
can
I.
A
How
can
I
like
get
a
new
car
or
color
my
car,
or
things
like
that,
like
what
kind
of
accessories
can
I
get
from
my
car?
So,
given
such
very
subjective
questions,
how
do
you
discover
an
answer?
So
it's
a
process
of
discovery
going
through
multiple
pins,
looking
at
curations
how
people
have
curated
ideas
together,
getting
inspired
and
trying
to
actually
get
it
done
as
well.
A
They
recently
launched
a
feature
called
tried
it
where
you
can
actually
look
at
an
idea,
and
once
you
try
it,
you
can
actually
let
people
know
that
you've
tried
it
and
get
feedback
on
that
as
well.
Anyways
back
in
2010,
that's
when
was
Pinterest
was
started,
and
it
looks
something
like
this
very
primitive
I'm.
A
It's
a
pretty
cool
experience
and
a
lot
of
companies
are
trying
to
solve
that
problem,
especially
Google,
but
then
Pinterest
has
come
with
one
step
ahead
simply
because
of
all
this
metadata
input
by
users
about
what
an
item
really
is
in
this
example
of
this
gif
really
played,
you
would
see
various
categories
of
things
like
it's
a
clock.
It's
a
wooden
clock,
it's
got
Roman
numerals
and
those
would
be
all
the
categories
and
every
show
list
of
relevant
clocks
which
are
very
similar
to
those
categories
and
what
it's
worth?
A
A
We
have
seen
some
significant
scale
challenges
which
led
to
moving
to
the
micro-services
world,
so
we
still
have
the
large
monolithic,
django
application
running
at
pinterest,
but
at
the
same
time
we
have
a
lot
of
micro
services
around
it,
supporting
specific
functions
of
the
large
monoliths,
specifically
with
the
core
services
which
serve
the
pins
or
taking
the
rights
for
the
pins
or
serve
the
boards
and
those
specific
primitives.
But,
along
with
that,
we
also
enable
other
engineers
to
build
a
lot
of
different
services.
A
You
know
supporting
the
adds,
function
or
discovery
functions
and
things
about
to
kind
of
give
you
a
sense
of
scale.
You
know
with
this
is
the
number
of
micro
services
while
number
of
offline
jobs.
It
goes
in
the
tens
and
thousands
and
the
VM
instances
again
it's
pretty
huge,
just
kind
of
site
tracking
from
here
and
looking
at
how
Twitter
was
so
previously.
I
used
to
work
at
Twitter,
like
I
mentioned
before
Twitter,
has
a
completely
different
set
of
challenges
and
problems
to
solve
it's
a
real.
A
It's
a
real-time
messaging
system,
more
specifically
a
broadcast
mechanism
rather
than
one
to
one.
So
there's
a
whole
list
of
challenges
around
how
you
fan
out
messages
to
a
certain
list
of
users.
How
quickly
can
you
do
it
and
such
Twitter
early
on
moved
from
the
monolith,
which
is
the
large
Ruby
on
Rails,
which
we
used
to
call
Apple
II
the
monorail
back
in
the
day
to
microservices
after
an
incident
at
2010?
A
This
was
when
the
FIFA
World
Cup
happened
and
I
think
Spain
won
at
that
point,
but
for
every
goal,
scored
on
literally
every
minute.
Twitter
would
go
down
simply
because
it
could
never
sustain
that
level
of
per
second
and
that
really
prompted
the
company
to
rethink
and
how
you
enable
reliability
at
that
scale,
and
that
prompted
us
to
all
think
about
how
you
build
microservices
or
move
in
that
direction,
where
individual
systems
take
ownership
of
reliability,
efficiency
and
things
of
that.
A
This
is
one
of
my
two
bow
slides,
always
to
kind
of
explain
the
complexity
of
how
Twitter
was.
These
are
like
more
than
2000
micro-services.
What
you're
specifically
seeing
is
is
the
TFE
or
the
twitter
front
end,
which
usually
takes
in
the
requests
and
fans
that
out
to
multiple
services
downstream.
So
that
gives
you
a
sense
of
how
complicated
or
how
big
the
Twitter
micro
services
ecosystem
is.
A
All
right
so
TL
DR.
What
did
we
learn
like
you
know
the
obvious
benefits
of
having
micro
services
is.
You
know
you
have
a
clear
sense
of
fencing
and
ownership
and
you're
working
on
the
monolith.
It's
really
hard
there,
100
or
200
plus
api's,
and
there
would
be
specific
owners
to
each
of
those
api.
Is
you
cannot
really
organize
your
team
around
the
specific
API?
A
It's
all
trying
to
contribute
code
to
the
same
repository
or
the
same
monolith,
then
continuous
integration
testing,
all
of
that
becomes
a
mess
and
the
release
engine
of
the
release
cycle
itself
is
really
hard
to
kind
of
orchestrate.
For
what
it's
worth
so
microservices
does
give
you
a
sense
of
isolation
and
ownership
and
fencing
by
you
owning
individual
systems
and
services.
It
also
gives
a
sense
of
reliability,
because
if
one
system
goes
down,
it
doesn't
mean
it
has
to
take
down
like
the
entire
site.
Right.
A
A
So
micro-services
does
enable
organizations
to
scale,
so
these
companies
move
from,
like
hundreds
of
Engineers
slick
or
even
less
than
attends
to
hundreds
of
engineers
to
like
thousands
of
engineers,
the
ability
to
kind
of
move
from
five
to
six
teams
to
like
tens
and
hundreds
of
teams.
However,
you
know
things
are
not
always
what
they
seem
right.
If
it
was
so
simple,
then
we'd
all
be
very
happy
and,
and
everything
would
be
working
fine.
A
So
it's
really
not
what
it
seems
and
that's
really
what
I'm
gonna
be
talking
about
so
before
I
get
into
those
specific
challenges.
Let
me
just
describe
what
is
the
life
cycle
of
a
job
of
a
service?
I've
been
interchangeably
using
these
words,
but
let
me
define
what
it
actually
is
and
he's
in
the
world
of
Pinterest
Engineering
a
job
can
be
one
of
these
things.
It's
a
long-running
service.
It
could
be
a
bad
job
which
is
doing
certain
MapReduce
or
model
training
problems.
A
You
know
it
runs
within
a
certain
set
of
time
and
it
dies,
and
it
keeps
doing
that
again
and
again,
like
a
cron
job.
It
could
also
mean
an
experiment,
not
many
people,
think
of
experiments
as
a
job,
but
now,
as
companies
have
invested
significantly
in
experimentation
frameworks,
machine
learning
frameworks,
it's
kind
of
critical
to
identify
that
as
another
entity
as
well.
You
have
to
think
about
how
do
you
scale
like,
for
example,
experiments
have
like
a
finite
set
of
time,
but
they
run
like
long-running
services.
A
They
don't
serve
all
100%
of
the
traffic,
but
probably
some
percentage
of
the
traffic
collecting
data
and
pushing
back
to
some
offline
data
processing
system.
So
you
have
to
really
think
about
that
as
another
entity
as
well.
So
I
am
kind
of
gonna,
be
entertained,
I'm
going
to
be
using
job
going
forward
now,
but
it
could
mean
any
of
these
regardless.
The
problems
around
the
management
is
still
the
same.
A
The
governance
problems
are
still
the
same,
so
this
is
really
primitive,
but
it
in
a
shin
and
really
like
a
concise
30,000
foot
view
is,
like
you
create
a
job.
There
are
certain
steps
you
take
to
create
a
job,
and
then
there
is
the
release
phase,
the
monitoring
phase
and
the
deprecation
phase.
The
deprecation
phase
is
something
we
usually
never
talk
about
right,
like
how
many
of
you
have
actually
deprecated
services,
I,
don't
I,
don't
think
a
lot
of
people
have
done
that,
so
the
process
of
our
deprecation
is
like
black
magic.
A
Essentially,
you
really
don't
know
what
to
do.
You
know
where
to
go
and
remove
the
coat
or
shouldn't
even
be
even
doing
that,
so
how
many
people
talk
about
it,
but
I
think
it's
important
kind
of
expanding
on
the
central
portion
of
things
is
you
know
what
does
the
release
phase
looks
like
and
what's
the
monitoring
phase
looks
like
and
I
believe
this
is
my
personal
opinion
is
that,
as
a
community,
we've
spent
a
lot
of
time
defining
structure
and
governance
around
these
particular
areas.
A
A
You
need
really
know
like
what
kind
of
stack
you
have
to
setup
to
kind
of
build
a
complete
monitoring
stack
so,
but
the
unknown
portions
are
really
like
the
create
and
the
duplicate
phase
and
which
I
think
like
has
another
portion
on
top
of
it
called
the
management
phase
are
more
like
operations
or
governance,
again
I'm
interesting
to
be
using
these
words,
because
each
of
them
mean
different,
but
I'll
kind
of
like
bring
all
of
that
together
towards
the
end.
So
what
does
the
manage
phase?
Actually
look
like?
A
A
You
have
to
tell
you
know,
set
up
a
repository,
probably
add
some
kind
of
owners
file
telling
that
you're
the
person
who
wrote
that
code
you
have
to
go
through,
you
know,
AWS
I
am
or
Google
Cloud
I
am
or
whatever,
that
is
to
kind
of
set
up
the
relevant
service
accounts,
and
things
of
that
you
have
to
define
some
metadata
in
some
configure
files
or
JSON
files,
and
that
metadata
could
be
as
simple
as
telling
that
hey.
You
know.
A
If
there's
something
wrong
with
the
service,
this
is
the
slack
Channel
you
have
to
head
out
or
our
primitive
information,
such
as
that
dashboard
links,
alert
links.
The
other
one
is
the
biggest
one,
which
is
like
planning
capacity.
Just
out
of
curiosity,
how
many
of
you
actually
planned
capacity
for
services?
How
many
of
you
think
about
that?
Okay?
So
it's
a
huge
problem,
it's
one
of
the
largest
problems
which
many
people
fail
to
really
think
about,
and
we
feel
like
orchestration
systems,
kind
of
help
manage
that
to
a
large
degree.
A
Basically,
you
just
go
back
and
write
a
post-mortem
saying:
I
didn't
plan,
sorry
and
the
final
part
is
the
budgeting
and
the
spent
portion
as
well
I'm.
Sure
many
of
you
have
been
asked
by
engineering
managers
or
leaders
within
your
company
as
to
how
much
does
your
service
cost
or
how
much
does
a
job
costly?
How
many
resources
are
using
and
that's
a
hard
thing
to
answer
right
like
the
public
cloud?
They
don't
really
don't
provide
you
information
at
a
service
or
a
job
level.
A
They
give
you
how
many
instances
you
used,
how
many
s3
prefixes
you
have
or
things
of
that.
But
how
do
you
tie
all
that
together
to
the
concept
of
a
job
or
a
service,
so
that,
according
to
me,
is
really
the
large
problem
to
solve?
And
this
is
what
I'm
gonna
be
talking
about
now
for
the
next
10
to
15
minutes
so
I
think
in
short,
again,
this
is
an
amazing
gift
which
I
saw
by
the
practical
dev.
A
It's
it's
the
chaos
essentially,
when
you
have
like
hundreds
of
thousands
of
micro
services
running
in
your
company,
where
it
becomes
really
hard
to
kind
of
figure
out
what
is
happening
where
I
won't
share
this
link,
it's
it's
worthwhile
to
go
and
watch
alright,
so
we
discovered
early
on,
like
governance,
is
needed
back
at
Twitter
when
we
had
like
over
thousand
plus
micro
services.
It
used
to
be
like
a
constant
thing
for
the
compute
teams
to
figure
out
like
who
are
the
top
customers.
A
You
know
like
how
much
resources
are
they
using
because
we
used
to
run
multi-tenant
systems
like
our
compute
platforms
were
all
multi-tenant.
It
ran
on
apache,
Arora
and
mesas
back
when
I
was
there
at
least
we
used
to
run
more
than
200,000
containers.
The
biggest
challenge
over
there
is
like,
since
it's
a
multi-tenant
system.
We
were
responsible
kind
of
to
scale
or
clusters
out,
and
we
have
to
figure
out
how
our
customers
are
gonna
scale
in
within
the
clusters
and
that
and
again
Twitter
around
our
data
centers.
A
It
was
not
the
public
lot,
so
you
could
really
cannot
go
and
acquire
10,000
more
instances.
So
we
had
to
pre-plan
buying
machines
and
things
of
that,
so
it
was
really
crucial
to
kind
of
on
this
and
who
are
your
top
customers?
How
much
resources
are
they
using?
So
that's
how
this
this
whole
thing
triggered
off
like
because
we
have
so
many
services.
We
wouldn't
understand
like
okay.
What
do
we
need
to
do
to
kind
of
solve
that
problem?
And
that's
when
we
realized
governance
is
needed,
because
we
were
also
answerable
to
our
leaders.
A
Just
say
why
the
Apache
or
mesas
cluster
grew
by
50x,
and
you
cannot
just
point
your
hands
to
say
a
certain
team
and
say
hey
they.
You
know,
ask
for
like
10,000
cores
or
something,
but
it's
it's
a
mutual
thing
and
you
need
to
like
make
sure
the
accountability
is
at
the
right
place
through
some
sort
of
visibility
so
before
I
get
into
those
things.
I'm
just
gonna
have
like
this.
You
know
simple
service.
A
This
is
not
a
real
service
at
Pinterest,
but
I
just
want
to
talk
about
how
these
things
look
right,
like
you
have
a
let's
imagine
that
if
you're
writing
a
pin,
write
service,
it's
a
deploy
package
and
it
requires
certain
resources
on
the
computer
itself.
First
thing
you
do
is
definitely
define
some
names
right.
A
You
have
to
define
the
namespace,
you
have
to
define
some
cluster
names
to
say
this
is
who
I
am
right
and
that
needs
to
happen
again
through
some
config
files
or
or
you
go
to
some
self-serve
UI
to
do
that
or
you
literally
like
email,
someone
or
ping
someone
and
tell
that
hey
set
this
up
for
me,
you
have
to
set
up
your
sort
of
discovery
paths.
You
need
to
define
a
name
again
over
there,
and
then
you
obviously
require
storage,
blob
storage
or
some
key
value
stored.
A
Now
you
have
to
go
to
those
individual
services
and
again
define
instances
for
your
application.
I'm
sure
everyone
has
done
that,
and
everyone
has
given
different
names
in
each
of
these
different
steps
all
the
time
and
what
happens
then
is
the
operators
of
these
individual
systems
have
a
hard
time
figuring
out
who
actually
owns
this
particular
instance
of
a
fate?
You
know
like
a
prefix
in
the
blob
store
or
a
namespace
in
the
key
value
store.
How
do
you
tie
that
back
to
this
package,
which
is
actually
the
pin
write
service?
A
So
the
first
question
was
always
literally
who
owns
what
like,
who
is
the
owner
for
prefix
X
or
key
value
store
Y,
and
this
was
I
gonna
give
everybody
gets
a
spreadsheet
right
like
each
of
these
teams.
I
remember
like
used
to
have
these
large
spreadsheets
having
all
these
different
identifiers.
X'
mapped
to
some
person's
name
and
for
what
it's
worth
like.
You
know,
companies
as
large
as
Twitter
or
Pinterest
people
leave
people
come
in
new
york's
happen.
Many
things
happen
and
it's
really
hard
like.
A
You
cannot
expect
a
storage
team
or
a
compute
team
to
kind
of
manage
their
spreadsheets
on
a
daily
basis.
The
whole
point
of
multi
canton
systems
is
to
like
help.
People
move
fast.
You
come
in,
you
launch
something,
and
you
know
you
know
that
person
is
using
x
resources
and
you
charge
them
and
all
that
stuff.
So
I've
seen
like
many
AMS
and
our
teams,
like
literally
managing
these
spreadsheets
on
a
daily
basis,
they
track
like
resources
asked
by
this
user.
A
They
add
a
link
to
a
ticket
or
some
email,
but
it's
a
pain
like
you.
Don't
want
to
be
spending
your
time
actually
managing
these
spreadsheets.
So
that
was
one
of
the
first
problems
we
set
out
to
solve,
and
I
pinterest
like
one
of
the
things
we
started
doing
is
like
thinking
about
okay.
What
is
that
entity
model?
Look
like
how
do
you
solve
this
problem?
If
you
were
to
like
automate
this
into
a
system
right
and
you
know
it's
an
ownership
directory
essentially,
so
how
do
you
define
ownership?
A
And
how
do
you
tie
that
back
to
a
notion
of
a
service
I
job
and
that's
what
this
entity
model
describes?
Ownership
could
be.
It
starts
at
a
business
owner
level.
You
know
you
have
your
adds
organizations,
and
I
think
I
do
have
an
example.
Yes,
so
it
kind
of
maps
well,
at
least
in
the
two
organizations.
You
know
I've
try
to
like
build
this
out
it
maps
well
to
the
larger
business
organization,
the
individual
teams
and
the
projects
they
own,
and
it
ties
well
to
a
notion
of
an
identifier,
an
identity.
A
In
my
example
of
the
service
which
we
were
trying
to
build,
the
pinned
writer
service
was
the
service
identifier,
a
canonical
name
which
we
wanted
to
find,
which
is
something
I
think
I'll
talk.
Next
is
how
do
you
provision
those
identifiers
per
resource
for
a
job?
Like
I
said
we
usually
put
in
different
names
for
our
same
service
as
and
when
we
are
requiring
acquiring
different
resources?
A
If
you're
launching
VM
instances,
you
probably
have
some
different
key
value
store
to
describe
ownership
when
you're
launching
your
RDS
or
a
storage
cluster,
you
probably
have
some
different
names
to
find
over
there.
So
how
do
you
tie
those
together?
How
do
you
kind
of
categorize
or
build
an
inventory
of
resources
you
want
to
provide
to
your
engineers
within
the
company?
A
How
do
you
build
that
and
tie
that
to
a
notion
of
an
identifier,
because
at
the
end
of
the
day,
you
don't
want
engineers
to
be
writing
these
names
regardless,
let's
be
honest,
like
naming,
is
really
hard.
So
that's
one
of
the
hardest
problems
in
CS
at
least
I,
believe
so
you
don't
want
any
of
these
folks
to
be
writing
those
names.
So
how
do
you?
How
do
you
kind
of
automate
that?
A
How
do
you
figure
out,
like
a
plan
to
solve
that,
so
that
was
the
next
step
right,
so
this
is
the
same
service
I
talked
about
like
so
pulling
those
things
out
like
so
there's
always
gonna,
be
some
canonical
name,
defining
all
these
resources
to
a
particular
job
regardless.
If
it's
a
Hadoop
job,
it's
gonna
be
some
prefixes
from
some
store.
A
You're
gonna
pull
you're
gonna
do
MapReduce
and
you
know
you're
gonna
be
pushing
it
out
to
some
other
data
store
or
consume
it
through
a
service
or
if
it's
like,
just
general
compute
like
or
even
if
it's
in
Cuban
at
ease,
you
are
going
to
define
some
canonical
name
and
you're.
Gonna
have
an
identifier
for
that
individual
system.
I
think
that's
the
first
thing
we
identified
like
we
ended
up
building
a
notion.
A
We
ended
up
building
an
identity
service
back
at
Twitter,
which
has
specific
API
is
which
each
of
these
infrastructure
services
would
just
talk
to.
Well,
essentially
meant.
Is
we
centralize
the
provisioning
process,
so
people
engineers
literally
go
into
a
console
to
provision
the
resources
they
need
for
either
Hadoop
or
compute
or
storage
or
whatever?
That
is,
and
they
never
go
to
these
individual
systems
to
go
and
provision
something.
A
So
that
way
we
control
how
we
want
to
start
naming
things
downstream
and
there's
just
a
single
name
at
the
top,
and
that
way
we
just
solve
that
problem,
and
we
also
take
away
the
spreadsheets
which
each
of
these
operators
will
be
owning.
If
the
operate
needs
any
information,
they
can
just
look
back
into
the
identity
service.
To
figure
out.
You
know
who
owns
a
certain
UUID
of
my
key
value
store.
A
So
that's
one
of
the
first
things
right
like
so
who
owns
what?
Then,
how
do
you
identify?
How
do
you
create
an
identifier,
a
provisional
identifier,
so
an
identity
manager?
The
next
step
was
like.
We
realized
that
okay,
you
have
an
identifier
now,
but
many
a
times.
There's
access
controls
associated
to
these
identifiers
in
storage,
especially
you
want
to
make
sure
that
the
right
service,
you
know
using
a
certain
name,
space
of
a
key
body
store
or
a
prefix
and
s3,
and
you
want
to
make
sure
those
that's
audited.
That's
logged!
A
A
We
were
building
it
into
the
RPC
layer
itself,
so
as
your
as
your
service
comes
up
and
it
starts
to
communicate
with
any
of
these
downstream
systems
based
on
the
identifier,
the
downstream
systems
can
block
your
requests
and
things
of
that.
So
we
could
enable
service,
authentication
and
service
authorization.
A
A
Now
the
next
thing
we
started,
we
had
a
problem
with
with
like
where
to
look
up
job
metadata
and
configuration
again.
These
exist
mostly
either
in
the
source
code
in
some
llamó
files
and
JSON
files,
or
sometimes
in
some,
my
sequel
database
within
operators
like
so
the
use
cases
we
had
were
like
RS
re,
or
at
least
the
reliability
engineering
team
would
like
to
know
across
the
fleet,
like
all
the
services
really,
where
can
I
find,
if
there's
anything
wrong
with
a
service?
What
is
the
batch
board?
A
I
need
to
go
and
look
at
what
is
the
alert,
I
need
to
go
and
see
who
should
I
ping
or
rich
slack
channel
or
which
HipChat
channel
should
I
go
in
and
talk
to
or
like?
What's
the
email
of
that
team,
and
that
was
like
usual
operational
metadata,
you
want
Agony's
the
ownership
side
of
things
at
the
same
time
like
the
links
to
the
batch,
pods
and
alerts.
So
where
do
you
store
all
that
information
like?
How
do
you
tie
that
to
the
notion
of
a
of
a
job?
A
So
at
Pinterest
we
use
a
system
called
telephones.
We
are
more
importantly
like
we
deploy
using
our
inbuilt
deployment.
Orchestrator
called
teletraan,
it's
an
open
source
project.
It
looks
a
bit
complicated
right
now,
it's
changed
over
time,
but
at
Pinterest
we
do
VM
based
deployments.
We
we
are
in
the
process
of
containerization
and
that's
a
another
story,
and
hopefully
I
can
make
a
talk
on
that.
But
right
now
it's
still.
A
We
M
based
deployments
and
one
of
the
things
is
this
tool
actually
has
a
host
of
buttons
on
the
Left
configure
where
you
have
to
like
specify
over
70
fields
on
what
kind
of
metadata
is
there
for
a
service?
Now,
that's
painful,
but
again
that's
model
based
on
how
the
how
the
system
works
and
it's
all
required.
It
includes
auto
scaling
groups.
It
includes
all
that
information
which
is
required,
but
the
thing
is,
it
is
not
tied
to
a
notion
of
the
canonical
identifier.
A
So
you
want
to
start
tying
metadata
off
a
service
it
to
the
canonical
identifier
and
that's
another
problem
to
solve,
and
we
set
out
to
solve
it
by
building
a
metadata
manager,
the
single
source
of
truth
for
all
job
metadata,
and
this
is
basically
going
to
be
tied
back
to
that
initial
entity
model
which
I
showed
you
guys
about
ownership
and
identity.
The
idea
is
like
you
can
always
have
a
key
value
at
the
job
level,
but
then
it's
a
hierarchical
model,
so
you
would
deference
to
find
on
the
project
side
of
things.
A
You
don't
have
to
have
multiple
metadata
at
almost
every
table,
but
this
this
this
works
at
least
pretty
well
for
the
use
cases
we
had-
and
this
is
the
part
I
think
which
is
kind
of
this
and
there's
another
one
behind
it.
It's
like
I
think
these
are
like
very
important
as
well,
like
whatever
I
explained
right
now,
we're
all
primitives
to
kind
of
help
to
this.
How
do
you
allocate
infra
resources
consistently
cross
across
your
company?
A
If
you
are
like
part
of
the
cloud
teams
for
your
own
organization,
I'm
sure
you've
thought
through
this
problem.
One
of
the
things
is
like
as
part
of
governance.
You
don't
want
to
open
the
gates
to
everyone.
You
don't
want
to
like
provide
the
AWS
console
access
to
every
engineer
in
the
company
or
the
GCPD
X
console
axis.
You
don't
want
to
have
some
gates
around
it.
You
don't
want
to
define
a
set
of
instance
types
you
would
like
to
allow.
A
You
want
to
define
a
set
of
primitive
services
or
even
high-level
services
like
lambda,
whatever.
That
is
which
engineers
can
use.
It's
it's
not
to
restrict,
but
it's
all
it's
all
about.
Like
providing
some
constraints,
so
it's
ease
of
management
at
the
infrastructure
and
the
operations
level
and
also
reduces
a
lot
of
cognitive
overhead
for
the
engineers
to
choose
and
decide
when
you
limit
options,
if
there's
really
truly
a
requirement,
we
can
always
have
a
conversation
separately
and
onboard
that
as
a
resource
to
offer
to
the
entire
company.
A
So
one
of
the
things
we
had
to
do
at
least
at
Pinterest
we're
doing
right
now
is
we're
defining
those
resources.
Again,
like
I
said
it
was
a
VM
based,
mostly
VM,
based
that
everyone
gets
access,
to
instance,
is
pretty
directly.
We
do
have
a
concise
set
of
instance
types
we
allow,
but
then
again
we
want
to
start
thinking
like
for
the
future
like
if
we
were
to
move
to
the
containerized
world.
How
would
that
look
like
you
know?
We
don't
want
people
to
think
in
terms
of
instance,
is
going
forward.
A
We
want
to
think
them
to
think
in
terms
of
CPU
memory
and
gigs,
at
least
for
their
compute
jobs,
probably
something
different
for
their
bad
jobs,
because
over
there
it's
more
about
how
much
memory
can
you
crunch
over
a
certain
millisecond
of
time,
at
least
in
Hadoop?
Those
are
the
metrics
we
care
about,
so
we
want
to
define
those
at
the
resource
types
for
these,
these
infrastructure
services.
A
So
it's
basically
the
in
mentoring
and
the
provisioning
of
resources,
and
this
ties
at
this
to
tie
very
closely
with
the
identity
system,
because
at
one
point
like
I
said,
one
of
the
initial
problems
is
given
a
prefix
in
s3
who
owns
it
and
given
a
prefix
in
s3.
How
much
does
it
cost?
You
really
want
it
at
that
level
or
even
like,
if
you
were
to
think
from
a
Hadoop
perspective,
given
a
job
or
multiple
jobs
belonging
to
a
workflow?
What
other
resources
it
touches?
A
It
could
be
an
s3,
it
could
be
in
key
value
or
my
sequel.
Where
does
it
touch?
What
are
the
resources
it
touches,
and
where
does
it
store
something?
So
you
want
the
dependency
graph
as
well,
so
you
have
to
tie
whatever
resources
you're
defining
over
here
to
that
identity
model,
to
kind
of
build
those
kind
of
graphs.
A
So
yeah,
like
I,
said
we're.
Gonna,
have
we're
building
this
resource,
centralized
resource
provisioning
system
which
will
do
the
cloud
provisioning
for
us.
We
don't
let
users
themselves
do
this,
but
we
are
that
intermediate
service
which
will
actually
provision
those
resources
on
the
cloud
for
them
and
that
way
we
can
now
have
behind
the
scenes.
Either
we
still
keep
running
on
VMs
or
you
know,
change
it
to
Kuban
at
ease
or
maysa
or
whatever
that
is,
and
the
user
need
not
know
at
all.
A
How
much
does
it
cost
the
company-
and
this
was
one
of
the
hard
problems
to
solve,
because
there's
no
clear
way
of
doing
this,
like
I've,
heard
multiple
solutions
about
this.
Like
you
know,
if
you're
in
the
public
lab,
you
could
probably
have
different
accounts,
you
know
like
map
to
teams
and
they
can
do
whatever
they
want,
and
you
have
budgets
around
that,
but
in
a
true
like
as
we're
moving
to
this
whole
multi-tenant
world-
and
you
don't
want
to
like
put
that
burden
onto
your
engineer,
the
user.
A
A
Based
on
that
now
things
get
a
bit
different
when
you're
running
from
the
DC
world
in
the
public
cloud
world,
so
in
pintura
is
one
of
the
things
we're
seeing
is
that
you
know
it
abused
already,
sends
us
a
bill
and
how
you
figure,
or
how
do
you?
How
do
you
map
that
to
the
notion
of
resources
and
jobs
which
I've
been
talking
about,
so
you
can
give
the
right
information
to
the
engineering
teams
tell
them
that
this
is
exactly
what
you're
using
this
is
how
much
you're
costing,
and
you
can
actually
do.
A
Xyz
changes
to
kind
of
reduce
that
cost
or
improve
utilization.
Like
I
said
many
times
what
happens
is
since
there
is
no
clear
capacity
planning
process
and
that
this
comes
back
to
governance
as
well.
People
often
oversubscribe
what
they
need
like
they
would
always
ask
for
thousand
plus
more
instances
than
they
would
require
and
probably
not
end
up
using
it
for
the
next
three
months
till
they
actually
go
and
deploy
the
service
for
the
first
time
till
then
you
will
probably
be
spared
paying
that
money
for
those
thousand
instances
regardless.
A
So
you
want
to
like
catch
all
those
things
early
on
the
the
biggest
pain
point
people
have
expressed
is
that
it
takes
a
long
time
to
provision
those
things
or
you
just
don't
want
to
think
about
it
right
now,
so
you
just
get
whatever
you
can
right
now
and
think
about
it
later.
So
one
of
the
things
is
like
this
enables
visibility
and
accountability,
and
we
define
like
some
canonical
schemas
and
how
we
want
to
meter
this.
A
We
in
fact
leverage
our
internal
observability
and
visibility
systems
to
kind
of
meter
utilization
and
subsequently,
like
tell
users
that
hey,
you
asked
for
hundred
instances
but
you're,
actually
using
only
X
CPUs
from
these
instances,
and
this
caused
the
company
so
much
or
in
base
stitch.
We
do
something
about
that,
and
this
is
key
like
at
least
the
Twitter.
We
saw
some
significant
gains
on
our
compute
infrastructure
when
we
kind
of
started
publishing
such
numbers.
For
example.
A
This
was
one
of
the
bills
like
how
it
used
to
go
out
at
Twitter,
so
this
was
to
the
tweet
service
team,
where
we
would
send
out
hey
you're
using
Aurora,
and
this
is
how
many
instances
you're
using
part
of
Aurora
and
here's
the
amount
you're
actually
costing
so
much.
What
this
doesn't
include,
and
actually
there
are
detailed
reports
behind
this
is
how
much
are
you?
Are
you
actually
utilizing
from
those
resources?
Manhattan
was
the
key
value
data
store
at
at
Twitter,
so
this
was
all
allocation.
A
What
the
users
requested,
there's
also
a
usage
component
which
usually
doesn't
show
up
here,
but
in
detail
reports,
so
so
I
think
I
spoke
about
like
the
need
for
governance
across
these
different
areas,
and
I
spoke
about
multiple
individual
services
on
how
we're
kind
of
tackling
these
problems,
but
they
all
come
together
right.
They
all
come
together
into
like
an
internal
cloud
platform,
but
you
want
to
provide
to
your
users,
so
they
can
like,
essentially
not
worry
about
managing
the
infrastructure
or
dealing
with
it
directly,
but
rather
through
this.
A
A
It
also
helped
with
the
capacity
planning
to
show
what
your
historical
mid
utilization
of
the
job
has
been
and
what
is
the
projected
utilization
and
finally,
and
most
importantly,
like
telling
you
how
much
you're
costing
for
that
particular
job,
because
that
is
the
most
actionable
item.
Engineers
can
get
it's
telling
what
their
job
or
their
service
is
costing
or
in
terms
of
utilization.