►
Description
Pinterest's Journey from VMs to Containers [I] - Michael Benedict, Pinterest
Pinterest helps you discover and do what you love. A visual discovery engine at heart, Pinterest guides you through a billion possibilities to quickly discover & get inspired to do something. With over 150MM MAUs across the globe contributing & combing through a billion pins, Pinterest's Infrastructure is built to cater to this scale with very unique requirements -- Today, I'll be talking about how a company operating on the public cloud on VMs since its inception decided to move to containers.
This talk will primarily focus on four things:
1. Pinterest Infrastructure Overview (Offline Compute / Online Serving)
Pinterest was born on AWS. As of today, we operate tens and thousands of instances and process tens and hundreds of PBs of data. Data is the cornerstone of our business where freshness & relevance is key. We will deep dive into our processing & serving stack.
2. VMs vs. Containers - The Pros and Cons
In this section, we will cover the challenges along four key pillars:
a. Developer Velocity - We will discuss the overall job lifecycle workflow i.e build, setup, deploy, operations when using VMs or Containers.
b. Service Reliability - Constraints around resource isolation and standardization across health checks.
c. Infrastructure Governance - Attribution of resources both on utilization & Spend, Quotas
d. Efficiency - Specifically around auto scaling -- our learnings from using ASGs at scale & how this impacts VM vs. Container from an efficiency & operations perspective.
3. Move to Containers
Here we will discuss the use of Docker at Pinterest and more importantly the steps we took around evaluating various orchestration systems. I'll share the various dimensions we evaluated and our learnings when running on a public cloud environment. For ex, docker integration, scheduling, networking, community, stateful support, big data support, security support
4. Vision of the Compute Platform at Pinterest
Finally we will close out with the larger vision (next 18 months) for the Compute Platform at Pinterest.
About Micheal Benedict
Micheal Benedict leads Product Management for Pinterest's Cloud & Data Infrastructure. He and his team are building Pinterest's next generation multi-tenant compute platform for stateless and stateful services. He also manages Infrastructure Governance at Pinterest. Previously, he lead products for Twitter Cloud Platform. Prior to that, he was an engineer building systems for Twitter's Observability Stack. Micheal has a Masters degree in Computer Science from SUNY at Buffalo.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Okay,
I
think
we're
good
to
start
hello,
everyone.
My
name
is
Michael
Benedict
I'm,
a
p.m.
on
the
cloud
in
the
data
infrastructure
team
I
work
on
the
container
orchestration
team
at
Pinterest
and
I
also
manage
the
infrastructure,
governance
and
the
capacity
planning
team
I
have
with
me
here.
My
colleague
Lida,
who
is
the
tech
lead
on
the
container
orchestration
project.
A
A
Then
you'll
talk
about
the
larger
compute
platform,
which
you
are
building
at
Pinterest
start
from
the
high
level,
the
vision
and
the
scope
to
get
us
set
the
context
as
to
what
what
it
all
means
and
then
eventually
go
into
the
meat
of
the
presentation
which
is
about
the
orchestration
system
and
what
we
are
doing
and
how
did
we
get
there
and
finally,
I'll
close
out
with
the
future
as
to
the
next
steps
going
into
2018
as
well
all
right.
So
with
that
how
many
of
you
use
Pinterest?
A
Okay,
that
is
great.
So
here's
a
quick
story
so
Pinterest
was
born
back
in
2010.
It
started
off
as
a
simple
Django
application
running
on
the
cloud
from
day
one.
This
is
what
it
looked
like.
The
goal
for
Pinterest
at
that
point
was
to
collect
and
organize
a
lot
of
ideas.
People
pinned
from
searching
on
the
Internet
fast
forward
to
seven
years
in
2017,
Pinterest
has
grown
to
become
Pinterest,
has
evolved
into
a
server,
produces
evolved
into
a
discovery
engine.
A
Today
we
focus
more
about
discovering
pins,
which
are
relevant
to
users
focused
more
on
their
interests
graph,
rather
than
just
generic
pins.
As
you
can
see,
we've
also
advanced
a
lot
into
image,
detection
and
detecting
objects
or
items
within
an
imagine
recommending
relevant
pins
and
categories.
For
that
particular
object.
A
Today
we
do
about
200
million
users,
monthly,
active
users
and,
what's
more
interesting,
is
that
these
users
are
pinners,
as
we
call
them
have
so
far
pinned
about
hundred
billion
objects.
To
put
that
into
context,
I
mean
that's
a
huge
number,
that's
it
so
the
interesting
part
about,
even
that,
is
that
each
pin
has
a
rich
set
of
metadata
about
who
posted
that
pain
rebel.
A
Is
it
extracted
from
you
know
what
site
or
image
or
video
is
that
and
the
specific
tagging
which
the
user
themselves
have
done
relevant
to
that
pin
a
single
pin
usually
can
be
pinned
by
multiple
users.
So
you
have
a
lot
of
context
or
interest
graph
generated
as
part
of
a
single
pin
to
power
that
we
have
tons
of
services
behind
the
scenes.
So
this
beautiful
bubble,
visualization
essentially
just
shows
all
the
serving
services
which
are
categorized
by
the
amount
of
V
CPUs
they're,
essentially
using
like
I,
said:
Pinterest
runs
on
the
public
cloud.
A
A
We
started
off
with
our
Django
monolith,
which
have
which
has
now
been
breaking
down
to
large
set
of
micro
services.
We
do
have
in
the
tune
of
about
thousand
micro
services
running
on
our
infrastructure.
The
hosts
are
tens
and
thousands
of
hosts,
but
a
large
portion
of
Pinterest
infrastructure
is
actually
on
the
data
processing
side.
We
do
use
Hadoop
and
spark
streaming
as
well
as
tensorflow,
largely
as
part
of
our
machine
learning
platform
provide
some
context.
A
We
have
like
about
10,
raise
to
5
number
of
jobs
running
at
any
given
point
in
time,
specifically
on
model
training
analytics
pipeline,
just
general
compute
work
as
well,
and
our
footprint
on
the
block
storage
system
is
also
pretty
immense
in
the
tens
of
hundreds
of
petabytes.
We
not
only
use
it
as
a
storage
mechanism.
We
also
leverage
it
mostly
like
a
transactional
analytical
back-end
as
well,
so
we
do
really
push
the
limits
on
some
of
these
systems.
A
The
current
state
of
the
world
looks
something
like
this,
so
Pinterest
over
the
years
has
been
adding
multiple
layers
of
infrastructure
to
suit
certain
use
cases
and
means,
and
what
has
happened
is
that
that
we've
come
with
these
diverse
set
of
tools
and
platforms
to
support
those
growing
needs.
As
of
today,
we
already
have
like
multiple
mechanisms
of
provisioning
resources
on
top
of
AWS.
We
have
different
compute
platforms
to
help
users
run
certain
set
of
workloads
on
these
on
these
platforms,
and
our
use
cases
are
also
diverse
and
growing
over
a
period
of
time.
A
A
They
have,
to
probably
do
a
whole
different
set
of
steps
to
do
the
same,
and
you
can
imagine
the
permutation
and
combination
which
will
enhance
this
complexity
for
a
developer,
and
it
really
leads
to
like
less
velocity
in
terms
of
when
you're
going
from
an
idea
into
production
even
worse.
It
adds
a
lot
of
support
and
operations
challenge
on
the
infrastructure
teams
which
are
managing
these
systems.
A
Unfortunately,
it
is
also
hard
to
migrate
the
system,
these
jobs
or
workloads
off
of
these
existing
systems,
because
I
think
over
a
period
of
time,
we've
just
lost
all
that
context
and
knowledge
about
how
they
work
and
for
what
it's
worth
they've
been
just
running
there
for
the
longest
time.
So
these
are
like
a
lot
of
these
challenges
which
we
had
to
go
through
and
think
through
a
bit
more
in
coming
with
coming
up
with
our
you
know,
strategy
for
the
next
few
years
we
have
to
support.
A
A
The
one
more
key
point
to
add
over
there
is
it's
also
very
difficult
to
actually
implement
governance
around
these
platforms.
When
you
have
such
a
diverse
set
of
platforms
to
truly
know
who's
using
what
and
what
are,
the
resources
being
used
is
extremely
extremely
difficult.
You
just
have
these
blocked
chunks
and
you're
Bill,
saying
that
this
was
the
maximum
see
two
instances
used
by
certain
tags,
which
makes
it
really
hard
to
have
like
decent
ROI
conversations
with
the
actual
users
of
those
systems.
A
Alright,
so
with
that,
like
I,
just
want
to
quickly
run
through
what
our
vision
for
the
compute
platform
is,
this
helped
us
id8
a
bit
more,
and
we
came
to
the
conclusion
that
our
goal,
regardless
of
Orchestrator,
is
regardless
of
which
cloud
we
are
running
on
or
whatever
that
is,
it
should
be
to
take
to
help
a
user
provide
a
fastest
path
from
idea
to
production,
without
really
worrying
about
anything
on
the
underlying
infrastructure.
Our
key
focus
areas
were
divvied
up
into
three
things.
A
We
wanted
to
simplify
the
end-to-end
developer
experience
as
part
of
building
deploying
launching
a
service
and
operating
it,
not
only
a
service,
but
also
a
data
job
or
any
other
type
of
jobs.
We
wanted
to
provide
an
integrated
infrastructure
platform,
whether
you're
running
a
bad
job
or
a
long-running
services,
metrics
logs
health
checks
or
even
job
statuses.
All
of
them
need
to
be
provided
to
a
user
through
a
single
experience.
A
To
give
you
an
overview.
This
is
our
interpretation
of
what
a
developer
usually
goes
through
in
terms
of
building
and
launching
a
job
and
subsequently,
these
are
also
individual
problem
areas
which
need
to
be
resolved
for
what
it's
worth
over
the
last
seven
years.
Pinterest
has
built
multiple
solutions
across
many
of
these
boxes
and,
like
I
said
they
were
built
for
Suffolk
needs
at
that
point
in
time,
which
has
just
been
added
over
a
period
of
time
today,
where
integration
across
all
of
them
is
extremely
extremely
difficult.
A
So
from
our
perspective,
this
is
really
the
problem
domain
for
the
compute
platform
and
as
our
goal,
to
really
stitch
a
narrative
around
it
and
provide
a
solution.
But
that
said,
we
have
to
start
from
the
from
the
bottoms
up,
so
it
comes
down
to
actually
the
cloud
resources,
the
orchestration
systems
and
even
those
perimeters
around
containers
or
VMs,
which
need
to
be
defined
first
before
we
can
go
up
anywhere.
A
Another
context
to
be
added
earlier
is
Pinterest
has
been
running
on
the
VM
virtual
machines
for
the
longest
time.
We
even
are
there
ec2
instances,
at
least
on
the
serving
side,
and
each
service
runs
on
their
individual
instances.
So,
with
that
in
mind,
I
think
one
of
the
first
things
we
had
to
just
figure
out
was:
how
do
you
containerize
this?
So
this
was
really
like
a
timeline
of
this
of
the
initiation
of
the
project.
A
We
had
to
then
productionize
darker
and
adopt
it
over
the
company
I'm,
going
to
particularly
talk
about
this
one
on
how
we
actually
went
about
from
that
phase
to
going
into
an
orchestration
system.
So
many
companies
are
facing
this
today
in
terms
of
how
you
actually
evaluate
I
know.
Many
of
you
must
be
already
using
Cuban
IDs,
so
you're
really
not
here
to
learn
about
this,
but
I
do
want
to
give
you
an
overview.
A
As
to
the
pain
involved
in
going
through
this
process
and
coming
to
a
decision,
because
this
is
not
something
easy
easily
done
at
a
company
of
the
scale
so
TLDR
is
we
had
to
build
this
evaluation
framework
where
we
decided
what
the
the
various
options
we
had
in
terms
of
choosing
something
the
pocs
we
had
to
do
and
subsequently
what
was
the
outcome?
This
was
done
in
a
span
of
eight
weeks
from
an
implementation
perspective,
but
the
discussions
have
been
happening
for
more
than
two
years.
A
So
with
that
said,
the
criteria
around
how
we
wanted
to
choose
an
orchestration
system.
The
most
important
thing
for
Pinterest
literally,
was
the
integration
cost.
What
does
it
actually
mean
to
move
from
a
vm's
world
into
into
this
new
world,
specifically
around
the
darker
support
side?
How
do
you
support
sidecars
Pinterest
has
a
lot
of
site
cars
running
alongside
boat
in
the
host
level,
as
well
as
in
the
application
level.
How
can
we
support
that?
A
The
runtime
extensibility,
the
networking
integration,
so
Pinterest
has
already
invested
heavily
on
the
networking
side
of
using
Ian's
eyes
and
having
security
groups
assigned
to
those.
So
we
have
a
pretty
robust
system
already
in
place.
How
do
we
integrate
with
that
which
is
both
easy,
simple
and
easy
to
debug
and
used
reason
out
with
as
well?
The
other
aspects
of
what
was
really
important
for
Pinterest
was
also
the
resource
and
the
task
scheduler
inside
the
flexibility
around
how
we
can
like
place
tasks.
Co-Locate
has
given
some
of
our
services
and
just
a
multi-tenancy
aspect.
A
This
comes
from
the
infrastructure,
governance
piece
which
I
spoke
about
because
we
know
for
a
fact
today.
Many
of
our
service
is
not
not
effectively
utilizing
any
of
their
instances,
so
we
really
want
to
push
that
limit
in
terms
of
protectively
utilizing
those
resources.
The
other
aspects
around
the
criteria
are
just
as
regular
scalability
and
the
performance
of
that
orchestration
system.
A
Staple
service,
support,
which
is
extremely
important
for
us,
given
that
a
large
portion
of
our
infrastructure
is
all
about
data
processing,
the
ecosystem
in
the
community
and
finally,
the
cluster
operations
and
the
support
our
choices
were
pretty.
We
had
like
lot
more
than
this.
We
had
to
condense
it
down
where
we
actually
did
like
a
detailed
POC
on
all
of
this.
So
obviously
kubernetes
was
there.
We
also
looked
at
mesas
and
Aurora,
mesas
and
Marathon
and
even
like
thought
about
a
custom
scheduler.
A
If
you
were
to
build
up
again,
mesas
is
a
pretty
battle-tested
system
and
we
also
have
some
experience
operating
mesas
within
our
company
for
our
real-time
streaming
platform
and
finally,
even
EDA
AWS
as
ECS
offering.
We
did
look
at
that
one
was
there
just
to
kind
of
get
a
sense
of
what
is
the?
What
is
the
difference
in
terms
of
a
hosted
solution
versus
like
a
managed
solution,
and
what
would
it
mean
for
the
teams
operating
I'm,
not
gonna,
get
into
the
depths
of
POC
leader,
we'll
cover
more
details
on
that,
but
the
TLDR
is.
A
We
do
find
some
commonalities
around
many
of
these
systems
and
we
do
detailed
pocs
on
each
of
that,
especially
on
the
networking
and
IM
say
again,
leader,
we'll
cover
that
at
the
later
part
of
the
presentation.
And
finally,
the
outcome
was
like:
it
was
a
conscious
choice.
We
did
see
some
shortcomings
as
part
of
making
this
choice,
but
it
was
a
conscious
choice
that
we
could
actually
work
through.
B
Hello:
okay,
that's
cool!
That's
great!
So
thank
you!
Michael
and
now,
though,
everyone
so
I'm,
Lida
I'm,
a
software
engineer
at
the
cloud
management
platform,
a
teen
choice,
so
I
think
Michael
has
give
you
a
high-level
view.
What
our
goal,
what
we
are
doing
and
I
will
bring
you
drill
down
to
the
technical
detail
with
what
we
have
done
for
our
container
platform
and
what
we
are
currently
doing
for
our
container
actuation
effort
with
kubernetes.
B
So
this
slide
gives
you
a
view
that,
before
we
are
doing
any
conversation,
we
are
running
our
stately
service
on
the
ec2
instance.
So
we
have
layered
a
Mis,
build
on
it,
where
we
have
puppy
dimension,
our
machine
configuration
and
we
have
thyratron,
which
is
our
in-house
deployment
system,
which
manages
the
service
piece
deployment
beyond
that
you
have
various
process
management
and
running
the
service.
So
these
things
had
several
pain
points
for
us,
so
the
engineers
has
to
be
involved
in
to
the
ami
building.
B
B
So
then
I
will
talk
a
detail
about
under
the
hood,
what
we
are
doing
for
this
docker
plan.
For
so
first,
we
introduce
a
Pinterest
service,
specific
description,
language,
which
you
can
see
it's
on
the
on
and
after
side
in
the
slide.
So
there
are
several
motivations
behind
why
we
give
this
service
description
language,
so
first
we
want
to
the
engineers
only
care
the
configuration
they
care
about.
So
then
the
underlying
system
can
expand.
This
configuration
to
handle
things
like.
B
What's
the
required
mounted
volume,
our
environment
variable
required
for
the
infrastructure,
so
engineers
know
don't
have
to
set
them
and
understand
how
to
set
that.
The
other
thing
we
want
to
have
here
is
that
we
want
to
provide
an
abstraction
layer
to
the
underlying
technology.
We
are
using
no
matter,
we
choose
the
missiles
or
choose
the
communities
later
on
under
engineer,
don't
have
to
rewrite
any
of
their
service
configuration
language
and
also
on
the
depth
developers
workflow.
B
We
did
some
work
on
the
optimize,
some
image
building
on
our
largest
share
image,
to
reduce
the
build
time
and
the
image
size
and
the
last
we
are
running
docker
registry,
as
we
use
the
easier
as
our
main
registry.
But
meanwhile,
we
have
a
replication
yourself
host
the
registry,
combine
them
together
which
give
us
the
high
availability
in
the
production
environment
considering
our
scale.
B
So
this
is
what
it
looks
like
under
the
hood
in
the
application
runtime.
So
there
are
a
bunch
of
the
sidecars
around
the
application
to
provide
all
kind
of
service,
so
all
of
them
are
running
in
containers,
so
we
provide
a
tool
which
called
terrific,
which
basically
manages
the
dependency
among
these
containers
and
also
in
your
application.
It
might
be
composed
with
multiple
containers.
B
So
terrific
not
only
manages
the
dependency
between
the
sidecar
and
the
application
also
manages
the
dependency
inside
the
application
and
even
between
the
different
aside
cars,
and
we
are
running
around
the
NATO
network
model
in
the
host
and
network
model
which
give
us
the
best
performance
and
our
darker
engine.
We
use
the
library
stall
so
to
ensure
that
the
docker
engine
restart
won't
call
the
all
the
container
being
killed
and
restarted
and
we're
running
out
of
this
to
overlay
to
file
system.
We
also
build
the
garbage
collection
in
our
local.
B
You
need
to
keep
the
a
configurable
number
of
the
image
for
every
kind
of
service
on
the
local,
so
so
that
we
can
roll
back
and
roll
forward
fairly
fast
and
last
thing
we
build
some
parallel
prefetching.
This
is
many
because
our
deployment
system
has
a
new
tow
exclusive
lock
on
the
host
side
and
you
given
time.
We
only
allow
one
development
to
happen,
but
for
this
particular
darker
environment,
we
want
to
parallel
pull
all
the
image,
so
it
can
make
our
deployment
be
faster.
B
So
this
is,
after
we
have
built
this
platform.
We
start
the
migration
sometime
in
the
June,
so
the
up
to
today
we
have
75%
of
the
state
of
the
service
already
in
this
docker
container
platform,
and
so
we
spend
a
lot
of
time
for
the
migration
and
in
the
last
few
months
our
team
is
basically
working
like
this.
B
So
there
are
several
learnings
I
want
to
share
with
you
about
this
migration
process,
how
to
make
sure
the
service
there's
no
outage
e.
During
this
migration,
we
have
hundreds
of
service
there,
so
I
think.
The
first
thing
is
that
we
should
invest
in
your
tools
in
this
migration.
You
need
to
have
a
deployment
tools
that
be
able
to
run
your
container
and
the
nun
container
at
the
same
time,
and
you
can
adjust
the
ratio
and
you
need
the
tools
to
be
able
to
compare
the
metrics
coming
from
this
to
work
load
and
most
important.
B
This
tool
must
be
very
easy
to
use
for
your
engineers
that
are
not
in
the
plan
form
team.
They
should
be
able
to
use
these
tools
when
they
start
to
migrate
your
service,
and
also
you
should
automated
all
the
deployment
and
migration
and
runtime
configuration
validation.
When
you
are
doing
this
kind
of
peak
migration,
make
sure
you
have
all
the
I
am
role
security
group.
All
the
surfaces
coverage
settings
all
correct
and
also
another
learning
is
that
container
is
a
very
fast
changing
area.
B
So
you
need
understanding
how
your
company
works
on
your
team,
dynamic
works
so
that
you
can
build
the
team
that
can
doing
this
work
most
effectively
at
Pinterest.
We
have
an
invented,
I--sorry
motor,
so
basically
the
eyes
already
sit
together
and
work
together
with
our
development
team.
This
helps
a
lot
in
this
project,
so
we
can,
whenever
we
hit
a
bug
in
the
docker
engine
which
I
have
to
upgrade
the
OS
kernel
and
have
to
reboot
the
ami.
B
We
get
a
very
fast
turnaround
time
on
that
and
the
last
thing
I
want
to
call
out
in
that
you
want
to
try
to
migrate
a
complex,
complex
service
earlier,
so
we
decided
to
migrate.
Our
most
important
service,
which
our
API
fleet,
which
is
one
of
our
largest
fleet
hosts,
are
our
mobile
client
and
web
client
at
a
very
early
stage.
B
It
take
a
while,
but
it'll
help
you
to
cover
every
corner
of
the
planet
for
make
sure
it's
really
production
ready,
and
meanwhile
it
gives
the
confidence
for
the
other
engineers
to
move
their
service
into
this
new
platform,
because
we
have
already
moved
this.
Everybody
known
this
huge
fleet
and
they
will
it's
very
easy
to
persuade
them
to
move
to
the
new
platform.
B
So
next
I
will
talk
about
our
current
container
actuation
effort,
our
oxidation
evaluation
and
what
we
have
built
for
the
kubernetes
right
now.
So
first
I
will
talk
about
the
evaluation,
so
we
have
done
it's
a
very
hard
work
to
do
the
evaluation,
it's
a
very
time-consuming
and
we
builded
a
POC
which
POC
I
means.
We
have
a
demo
service
which
are
our
integration
with
our
environment,
for
both
missiles
plus
our
aura
and
for
the
kubernetes.
B
So
for
the
missiles
we
evaluate
the
different
container
Iser
different
schedulers,
and
what's
our
executor
strategy,
we
try
to
write
our
customized
executor
in
this
POC
and
the
kubernetes
is
like
there
are
a
lot
of.
There
are
long
list,
a
lot
of
out
of
box
features.
We
actually
try
to
do
the
POC,
antacid
I
think
that,
more
importantly,
I
have
a
common
ground
of
the
things.
I
think
are
very
important
when
you
try
to
build
up
a
container
extrusion
platform.
B
There's
a
networking,
your
security,
how
you
are
doing
the
service
discovery
and
how
you're
doing
the
law,
the
matrix
security
management
I
mean
at
Pinterest,
since
we
have
been
several
several
years
and
we
have
a
larger
fleet,
we're
all
ready
for
some
of
them.
We
already
have
the
solution
and
some
solution
are
easy
to
migrate
to
the
container
world
like
surf
discovery
we're
using
zookeeper
as
our
service
discovery
platform.
I
know,
as
you
give
the
container
a
routable
IP
address.
Probably
it's
a
minimum
code
change.
B
B
They
have
been
battle
tested
and
they
have
been
proved
to
be
able
to
support
our
current
scale.
Even
there
might
some
glitch
in
them.
We
still
want
to
keep
keep
the
moving
parts
as
small
as
possible,
so
today,
I
will
focus
on
talking
about
two
things
and
networking
and
that
I
am
what
we
have
done
for
kubernetes
cluster
and
I.
Think
this
is
the
most
relevant
and
useful
to
share.
B
So
networking,
naturally,
is
the
most
hard
part,
and
we
spend
a
lot
of
time
on
it
and
even
today,
I
wouldn't
say
we
have
a
fully
satisfied
option.
So
today,
especially
you're
running
on
the
AWS
I
think
there
is
no
standard
solution
we
should
use
today
and
first
we
build
a
kind
of
our
networking
platform
to
be
more
flexible,
so
we're
using
the
AWS
Ani
and
but
also
we
support
different
kind
of
the
cni
plugins
that
we
targeted.
B
So
essentially,
we
use
the
cni
for
our
long-running
service,
which
the
service,
the
hosting
you
have
a
part.
You
have
to
have
an
incoming
traffic
when
using
the
ECS
there's
a
open,
a
the
best
open
source
that
ECS
eni
planning,
which
give
a
dedicated
yennai
to
a
running
pod,
so
we're
using
that
for
longer
a
new
service.
B
The
major
motivation
is
that
it's
simple
and
basically
everything
it's
more
very
leaning
towards
to
what
the
VM
looks
like,
and
but
this
has
a
major
limitation.
That
is
the
number
of
the
year
nights
you
can
attach
to
a
single
ec2
instance
is
Limit
Data
and
at
the
current
moment,
we're
looking
at
our
stately
service.
Most
of
them
are
running
on
it
cause
our
service
will
cost.
We
think
it
can
service
for
a
while,
but
meanwhile
we're
collaborating
with
AWS
looking
at
their
latest
release
the
CNI
plugin.
B
We
also
track
the
progress
from
the
Netflix
and
the
lift
that
they
have
some
shared
E&I
strategy
in
the
future.
We
may
change
that,
but
meanwhile,
our
structure
also
support
different
again
I.
We
today
we
are
running
our
batch
job
and
our
basically
worker
in
the
actually
in
the
bridge
mode,
which
is
very
like
the
talker
bridge
because
they
don't
need
to
serve
incoming
traffic
as
long
as
you
can
go
out
that
can
work.
So
we
support
the
I
am
low
and
a
security
group
on
the
yen
I.
B
So
this
diagram
gives
you
the
detail
about
how
we
are
going
to
the
networking
set
up.
One
app
has
started
so
when
the
community's
pod
is
started.
It's
that
the
coop,
the
complete
the
basic
tree
to
the
pods
container
and
start
to
call
the
CNI.
So
we
create
a
proxy
CNI
plugin
which
basically
invoked
the
next
the
daemon
which
we
have
a
daemon
running
on
the
host,
so
this
demon
will
actually
cure
the
kubernetes
get
to
the
palace
back
with
the
user.
Basically,
we
specified
the
network
model
in
the
palace
back.
B
So
whenever
this
demon
gets
the
pulse
back,
it
start
doing
the
NI
management.
So
so
even
we
are
using
the
EECS
unite,
but
we
don't
have
the
ECS
control
plan,
so
we
have
to
build
our
own
control
plan
basically
to
the
yen
I'm
Angie,
you
know
allocation,
so
this
is
all
in
this
demon,
so
this
demon
will
basically
look
at
the
current
ini
status
and
the
host
the
network
model
of
the
part
of
the
currently
an
eye
on
the
house.
B
If
it's
ask
for
that,
que
te
ni,
I
look
at
the
year
now,
if
necessary
and
as
I
know,
year
night,
so
we'll
return,
which
CNI
plugin,
that
you
should
invoke
and
what's
the
input
parameter,
the
proxy
and
I
actually
invoked
the
CNI.
Meanwhile,
this
demon
always
keep
reconciled
the
host
estate
and
updated
the
kubernetes
status.
B
B
So
the
jump
basically
listens
on
the
host
and
network
interface
and
whenever
so,
whenever
a
party
that
we
actually
configure
the
IP
table
rule
to
redirect
to
the
meta
proxy
car
to
the
local
interface
and
the
drum
actually
gets,
the
request
is
a
check
of
the
IP
address
and
also
it's
talked
to
the
local,
complete
and
point
to
get
all
the
pod
status
and
configurations
a
little
bit
different
here
that
we
also
the
drum,
will
call
out
things.
We
call
the
roll
assumed
service.
B
This
main,
the
main
reason
we
are
doing,
that
is
for
security
reason.
We
don't
want
to
give
the
base
rail
all
the
assumed
role
permission
in
the
UC
to
set
up
so
by
this
way.
So
whenever
a
person,
the
met
approximate
cuesta,
will
redirect
to
the
local
drum
and
gets
the
right,
I
am
credential.
This
work
for
both
the
ini
and
the
bridge
mode.
B
A
Thank
You
Lydia,
so
we
do
have
more
details
on
that,
so
we
will
be
at
coupon
for
the
next
two
days.
Our
team
is
also
here.
So
we
love
to
talk
more
about
people
who
are
working
on
similar
problems,
especially
on
AWS,
with
that
plug
I
just
want
to
quickly
go
back
to
the
larger
scope.
We
are
working
towards
again.
This
is
not
like
a
you
know,
three
month
or
a
six
month
project.
We
do
expect
like
a
lot
more
to
come
out
of
this.
A
As
we
build
out
our
orchestration
platform,
we
also
intend
to
kind
of
have
abstraction
in
terms
of
facilitating
job
deploys
and
orchestrating
those
deploys
as
well.
So
this,
like,
like
I,
said
earlier.
We
do
have
multiple
systems
today
to
do
that
for
bad
jobs
or
long-running
services.
So
one
of
our
priorities
essentially
is
going
to
be
kind
of
consolidating
that
and
have
a
single
place
of
facilitating
job
deployments.
So
some
of
the
things
we
are
planning
for
2018
is
obviously
production.
Izing
are
cube,
entities
cluster
and
adopting
some
initial
use
cases
we
already
have.
A
Our
team
has
been
working
really
hard
on
getting
Jenkins
and
those
kind
of
workflows
already
running
on
it,
and
we
also
experimenting
with
running
some
non-critical
long-running
services.
We
are
already
also
experimenting
on
trying
SPARC
to
run
on
Cuban,
Ares
I
think
that's
going
to
be
the
biggest
thing
for
us
and
we
intend
to
see
some
progress
on
there.
At
the
same
time,
like
I
mentioned
earlier,
our
job
definition
abstraction
works
pretty
well.
A
We
do
want
to
normalize
that
across
all
of
Pinterest
and
have
a
single
job
submission
service
which
will
facilitate
deployments
either
to
Cuban
ADIZ
today
to
Hadoop
yarn
or
something
else
in
the
future.
So
that
way
at
least
a
spec
still
stays
the
same,
there's
a
translation
which
is
happening,
and
we
intend
to
also
do
ownership
checks,
quota
management
on
all
of
that
stuff
it
at
the
job,
submission
layer
and
finally,
just
close
out
like
the
service
identity
portion,
so,
like
I,
said
make
sure
that
the
identity
of
the
service
is
validated
and
checked.