►
From YouTube: Diego Runtime Overview - Eric Malm, Pivotal
Description
Have you heard about Diego for a while now, but wonder what it means to you as an application developer or operator? Have you experienced the magic of pushing an application to Cloud Foundry and want to know what makes it all possible behind the scenes?
Over the past year, the Diego container runtime has made tremendous strides, reaching feature parity with the existing DEA runtime, making its deployments easily upgradable, and scaling to manage larger and larger container workloads. In this talk, the project lead for the Diego team will review the components of the Diego system, how they interact with each other, and how they integrate with existing Cloud Foundry deployments to allow a straightforward transition to the new runtime. We will also discuss how the Diego runtime enables many powerful new platform features, such as CF Tasks, TCP routing, and SSH access to containers, and preview other upcoming features that CF teams are actively working on and exploring today.
After watching this talk you’ll be ready to upgrade to Diego with confidence and take advantage of the many improvements it brings to Cloud Foundry.
Eric Malm
Pivotal Software
Eric works at Pivotal Software as the Product Manager for the CF Runtime Diego team, and prior to that was a software engineer on the Diego team and on the CF Runtime team. He also holds a Ph.D. in Mathematics from Stanford University.
A
All
right
good
afternoon,
everyone
thanks
for
coming
I,
hope
you
all
had
a
great
lunch
break.
My
name
is
Eric
Maugham
from
pivitol
and
I'm
the
product
manager
for
the
CF
front,
I'm
Diego
team
and
today
I'd
like
to
give
you
an
overview
of
the
new
Diego
runtime
system.
So
what
is
Diego
Diego
is
cloud
foundries,
new
container
runtime
and,
as
you
may
have
noticed,
if
you've
looked
at
it,
it
introduces
a
few
new
components.
A
Actually
there's
a
lot
more
components,
so
I'd
like
to
give
you
some
sense
of
how
we
can
make
sense
of
this
entire
new
system.
That's
going
to
live
at
the
heart
of
cloud
foundry.
So,
even
though
there
are
a
lot
more
components,
they
each
have
very
specific
responsibilities
and
we
can
organize
them
into
coherent
subsystems.
A
images
and
windows
apps
and
integrate
with
that
cc
bridge
to
help
us
run
that
workload
and
then
some
of
the
other
components
that
we've
introduced
have
been
to
interact
with
and
to
extend
the
notions
of
routing
on
the
platform.
So
let
me
give
you
a
visual
depiction
of
how
these
subsystems
are
connected
together
in
an
existing
cloud.
Foundry
deployment.
A
So
you
can
see
on
the
edge
here:
we've
incorporated
all
of
these
subsystems
with
the
existing
ones
that
we're
used
to
from
Cloud
Foundry.
In
this
case
the
cloud
controller
and
the
go
routers
I
mentioned
the
Diego
core.
It's
responsibilities
are
to
take
containerized
workloads
and
determine
determine
how
best
to
place
those
across
the
distributed
system
and
how
to
manage
their
lifecycle
and
to
do
that
effectively.
A
They
need
a
consistent
distributed
store
to
collaborate,
their
components
across,
but
the
core
of
Diego
itself
is
not
responsible
for
creating
these
containers.
Instead,
it
delegates
that
to
its
garden
dependency,
so
that
is
the
component
that
knows
how
to
actually
create
a
container
and
how
to
execute
inside
of
it.
A
A
So
in
that
core,
there's
really
three
types
of
VMs
and
down
to
the
bottom.
We
have
a
large
group
of
VMs
that
we
call
the
cells
in
the
Diego
deployment,
and
these
are
where
we're
actually
going
to
be
creating
containers
and
executing
workloads
in
them.
So
on
each
cell,
there's
a
local
garden
process
running,
but
that
can
be
garden.
Linux,
a
green
garden
windows
and
we're
all
looking
forward
to
the
coming
garden
run,
see
as
the
next
generation
of
what
garden
Linux
means
but
garden
itself.
A
Although
it
knows
how
to
create
containers
and
how
to
run
processes
in
them.
It
doesn't
participate
in
this
broader
distributed
system,
and
so
the
other
Diego
component
that
lives
on
the
cells
is
something
that
we
call
the
rep,
and
this
is
responsible
for
broadcasting.
The
presence
of
that
cell
to
the
rest
of
the
system
to
receive
work
from
that
system
and
to
report
it
back
to
the
rest
of
the
system,
as
well
as
to
drive
garden
to
actually
make
the
containers
okay.
A
A
Okay,
so
even
though
the
BBS
knows
a
lot
about
the
lifecycle
policy
for
these
individual
units
of
work,
it
in
and
of
itself
is
not
responsible
for
making
the
placement
decisions.
Instead,
it
delegates
that
responsibility
to
a
service
that
we
call
the
auctioneer.
So
this
is
responsible
for
communicating
with
all
those
cells
that
are
registered
in
the
deployment
and
making
optimal
placement
decisions,
telling
them
what
work
to
do
in
what
order.
And
then.
A
Last
but
not
least,
we
have
the
converter,
and
this
is
responsible
for
periodically
analyzing
the
global
state
of
the
work
that's
running
in
the
diego
core
and
taking
any
corrective
actions.
If
there
are
missing
instances,
then
it
will
recreate
them.
It'll
tell
the
BBS
you're
missing
some
stuff.
Please
make
this.
If
there
are
extra
instances,
it'll
tell
the
system
to
stop
them
and
if
there
is
work
that
is
ready
to
be
rescheduled
it'll
sweep
through
until
the
system
to
do
that
as
well.
A
A
In
contrast
to
that,
the
other
type
of
work
that
we
scheduled
on
Diego
is
the
notion
of
a
one-off
task,
and
this
is
almost
the
exact
opposite
of
a
long-running
process.
These
are
expected
to
terminate
at
some
point
and
the
data
that
we
track
around.
Those
tasks
will
detect
whether
or
not
they've
terminated,
successfully
or
not,
and
if
they've
terminated
successfully
can
convey
a
small
results
back
to
the
client.
A
These
are
inherently
non-scalable.
We
intend
them
to
be
one
offs
and
we're
also
more
careful
about
trying
to
schedule
them
consistently
to
make
sure
that
we're
only
running
them
once
so.
There
are
more
opportunities
for
the
system
to
say:
I
can't
be
sure
that
I
scheduled
this
exactly
once
so,
maybe
I'll
defer
to
scheduling
it
zero
times
and
let
the
client
deal
with
that.
A
So
these
two
types
of
work
are
abstracted
from
the
types
of
work
that
we
know
that
we
run
already
in
Cloud
Foundry.
So
these
long
running
processes
are
an
abstraction
of
the
notion
of
the
app
process
that
Cloud
Controller
wants
to
run
to
run
a
build
pack
app,
but
also
now
a
doctor
after
a
Windows
app,
and
these
one-off
tasks
are
a
representation
of
the
kinds
of
staging
tasks.
A
A
So
one
of
the
things
that
we
need
to
know
to
run
this
type
of
work
as
a
container
on
the
backend
is
what
are
the
file
system
contents
of
this
application?
So
if
you're
a
built
pack
app,
then
all
of
them
are
based
on
the
same
common,
consistent
root
filesystem.
This
is
something
that
cloud
before
refers
to
as
a
stack.
A
So
it
knows
it's
only
as
CF
Linux
FS
2
for
the
trusty
based
stack
and
when
it's
running
on
Diego,
we
have
an
option
to
preload
those
onto
some
of
the
cells,
and
so
when
we're
identifying
that
root
filesystem,
we
use
this
pre-loaded
schema
to
indicate
that
this
should
already
be
present
on
some
of
your
cells,
and
they
will
know
that,
but
that's
not
sufficient,
just
to
run
the
particular
type
of
application.
We
also
need
to
load
in
the
contents
of
that
application
into
this
base.
A
Root
file
system,
so
that's
the
notion
for
a
build
pack
app
of
a
droplet.
So
for
this
build
back
out
the
file
system.
Contents
of
the
container
are
the
union
of
those
two
sets
of
information.
But
then
we
also
need
to
know
what
to
run
in
that
file
system
context
and
in
this
case,
for
a
rails
app.
Maybe
we
want
to
run
rails
and
server
mode
with
the
correct
gem
context.
A
So,
for
example,
let's
say
we
want
to
run
the
library
Redis
image
from
dr.
hub.
Well,
that
tells
us
all
of
the
information
we
need
for
the
contents
of
that
container.
The
whole
thing
comes
from
docker
hub
and
then
the
action
will
be
whatever
the
entry
point
specifies
in
that
docker
image,
which
we
can
extract
during
a
staging
task
before
we
run
it
as
an
app
and
then
likewise,
maybe
we
have
different
limits
that
we
want
to
place
on
this
container
when
it's
running.
A
We
might
need
less
memory
for
this,
because
we
know
how
much
data
we
intend
to
store
in
this
Redis
instance,
but
it
might
need
more
disk
because
we're
having
to
take
the
entire
contents
of
that
docker
image,
and
we
don't
know
that
it's
necessarily
shared
with
any
other
containers
like
we
do
for
the
preloaded
route
offenses
and
then.
Finally,
maybe
we
only
want
to
run
one
instance
of
this
because
these
registers
are
not
clustered
together,
and
so,
if
we
run
multiple
copies
and
load
balance
across
them,
we
would
get
inconsistent
behavior.
A
A
So
the
first
step
is
for
cloud
controller
through
the
bridge
to
specify
that
the
DVS
I'd
like
to
run
this
Ruby
based
application
and
give
me
three
instances
of
it.
So
the
VBS
records
that
into
Stata
base
it
records
the
specification
of
that
work
entirely,
but
it
also
makes
slots
reach
one
of
those
instances
index
zero,
one
and
two
and
as
them
greyed
out,
because
it
hasn't
assigned
them
to
anything.
Yet
nothing
in
the
system
has
claimed
them
as
running.
A
So
it's
next
step
is
to
communicate
to
the
auctioneer.
There's
some
outstanding
work
that
we
need
to
place
somewhere
to
try
to
start
running
and
the
auctioneer
receives
that
request
and
batches
up
requests
for
new
work
to
place.
So
it
could
be
in
the
middle
of
placing
some
existing
work
and
it's
not
going
to
step
over
itself
trying
to
place
a
new
work
in
the
middle
of
that.
Let's
say
this
is
the
complete
batch
of
work
that
the
auctioneer
has
been
told
to
run
so
here's
how
it
places
them.
A
Its
first
step
is
to
contact
all
the
cells
that
are
registered
in
the
deployment
and
to
get
a
snapshot
of
their
current
state
and
from
there
it
can
make
decisions
about
optimal
placement
and
update
those
in
real
time,
as
it
makes
its
decisions
so
its
first
going
to
decide
where
to
place
that
index
0
instance
of
this
new
lrp
and
it's
looking
across
the
capacity
of
those
cells,
and
it
identifies
that
second
cell
as
the
best
place
to
put
that
instance,
because
it's
running
the
fewest
containers.
Maybe
it's
also
using
the
least
amount
of
memory.
A
Ok,
and
now
it's
going
to
place
that
index
1
instance
and
it's
looking
now
and
it's
updated
representation
of
what
these
cells
are
running
and
one
of
the
constrain
that
the
auctioneer
operates
by
is
that
it
wants
to
spread
these
instances
of
the
same
lrp
out
as
widely
as
possible.
So,
even
though
it
looks
like
those
two
linux
cells
are
tied,
it's
going
to
prefer
the
first
one
for
placement,
because
it's
not
running
any
other
instances
of
this
LRP.
A
Okay.
So
now
it's
got
one
final
instance
to
place
the
index
to
one.
If
we
were
going
solely
by
anti
affinity,
it
would
want
to
place
it
on
that
last
cell
because
it's
not
running
any
instances
of
it,
but
that
cell
is
ineligible
to
run
this
workload,
because
it's
a
Windows
cell
and
not
a
Linux
cell.
So
instead
it
decides
to
put
that
second
instance
on
that
second
cell
or
that
index
to
instance.
A
So
now
that
the
auctioneer
has
made
all
of
those
decisions
it
transmits
them
just
to
the
cells
that
need
to
run
new
work
and
those
cells
immediately
reserve
space
for
these
individual
instances
and
now
the
auctioneer
is
done
and
the
cells
are
in
charge
of
managing
the
local
work
that
they've
been
assigned.
So
their
next
task
is
to
communicate
back
to
the
BBS
and
to
double-check
that
nobody
else
has
claimed
that
workload.
A
While
this
is
happening,
so,
let's
say
cell
number
two
does
that
first
and
it
tells
the
BBS
I'm
trying
to
claim
these
two
units
of
work.
Nobody
else
has
so
the
BBS
says
great
you're
good
together
and
at
that
point
this
cell
can
start
creating
those
containers
and
then
once
they're
created,
run
these
processes
inside
of
them.
So
cell
number
one
does
the
same
thing
claims
that
in
index
one
instance
and
starts
it
up.
So
let's
say
that
cell
starts
that
container
really
fast
and
a
detects
that
this
process
is
healthy.
A
So
then,
reports
back
to
the
BBS
not
only
have
I
claimed
that
but
I'm
actively
running
and
the
BBS
updates
its
representation
of
that.
Okay,
let's
say
cell
number
two
does
not
have
such
a
good
time
and
ends
up
crashing
that
index
to
instance.
So
it
also
reports
that,
back
to
the
BBS
to
say
this
one
crashed
sorry
and
then
it
clears
up
its
container.
A
Okay,
so
we've
seen
that
many
of
these
components,
even
in
the
core
of
Diego,
are
crucial
for
it
to
operate
successfully.
We
can't
get
any
work
placed
if
we
don't
have
the
auctioneer
available
to
make
decisions
about
it,
so
that
means
in
any
kind
of
real
deployment.
We
want
to
run
multiple
copies
of
that,
but
we
also
need
to
recognize
that
it
has
this
important
global
responsibility
and
we
wouldn't
want
multiple
copies
of
it
running
simultaneously
and
independently
and
making
decisions
that
conflict
with
each
other.
A
So
this
is
where
these
components
coordinate
with
each
other,
to
make
sure
that,
even
though
we
have
multiple
ones
deployed,
only
one
is
active
and
the
rest
are
dormant
and
they
do
that
via
our
console
dependency.
So
these
auctioneers
contend
over
a
particular
key
and
console
that
represents
a
lock,
so
the
both
try
to
grab
it,
but
only
one
of
them
will
succeed.
That's
where
we
use
the
consistency
of
the
RAF
based
key
value
store.
A
A
Well,
let's
say
that
we're
now
updating
the
deployment.
Maybe
this
is
a
controlled
update,
we're
using
Bosch
to
update
all
the
software
in
the
cluster,
so
we're
gracefully
shutting
down
the
auctioneer
and
as
part
of
that,
it
will
release
its
lock
in
consoles
key
value
store.
So
now
the
standby
auctioneer,
which
is
continuous,
trying
to
grab
it,
will
succeed
and
become
active.
A
So
then,
when
Bosch
brings
back
that
second
instance
of
the
auctioneer
it'll
be
in
that
dormant
mode,
okay!
Well,
let's
say
that
something
more
catastrophic
happens
and
that
first
auctioneer
just
crashes
well
eventually
within
seconds
console,
will
release
that
lock
from
the
system
because
nobody's
maintaining
it
and
then
that
second
auctioneer
can
come
in
and
grab
it
again
and
become
active
within
just
a
few
seconds.
So
we
use
this
pattern
for
the
auctioneer
to
coordinate
them.
A
So
in
general,
if
you
want
to
know
how
to
run
an
H,
a
deployment
of
these
services,
you
need
at
least
two
of
these
to
be
sure
that
one
can
fail
over
to
the
other
appropriately
and
in
general,
if
you're,
deploying
by
availability
zone,
we
recommend
that
you
have
at
least
one
per
availability
Xin,
so
we
also
use
consul
for
other
purposes.
We
use
it
to
broadcast
stateless
services
through
consul,
DNS
and
also
the
cells
store,
their
presence
records
in
there,
so
the
auctioneer
can
find
them.
A
Okay.
So
that's
a
detailed
tour
of
the
core
of
Diego
and
let
me
give
you
some
context
on
what
some
of
these
intermediate
layers
do
as
well:
the
CC
bridge
and
the
routing
layers.
So
if
we
look
at
the
CC
bridge,
there's
a
lot
of
little
components
on
there,
but
they
each
have
very
tightly
focused
responsibilities.
A
So,
for
example,
the
stager
component
is
responsible
for
receiving
staging
requests
from
cloud
controller
and
translating
them
into
those
Diego
tasks
to
submit
to
the
BBS,
and
then
it
also
registers
itself
as
the
place
to
receive
a
call
back
when
that
task
is
done.
So
if
that
task
succeeds
or
fails,
the
stager
is
notified
about
it
and
then,
let's
see
si
know
about
it.
A
Likewise,
when
cloud
controller
wants
to
run
an
app
on
the
Diego
back
end,
it
doesn't
talk
directly
to
the
BBS.
It
talks
through
this
instinct
listener
component
and
it
tells
it
to
start
the
application
and
that
translates
it
into
a
long-running
process
for
Diego
to
manage,
and
then
what
collector
for
wants
to
know
how
that
process
is
doing,
how
its
instances
are
running,
it'll
talk
to
the
TPS
listener
and
that
will
ask
the
BBS
about
the
lrpc
state
okay.
A
So
there
are
also
a
couple
of
components
on
this
bridge
that
have
more
global
responsibilities
and
the
most
important
one
is
what
we
call
the
instinct
volker.
So
it
will
periodically
ask
the
BBS
what
el
RPS
are
you
running
on
behalf
of
Cloud
Controller
and
then
it'll?
Also
ask
hot
controller
what
apps
should
Diego
be
running
on
its
back
end
and
it
will
reconcile
those
treating
cloud
controller
as
the
source
of
truth.
So
if
there
are
any
missing,
LR
PS
it'll
make
them.
A
So
because
this
is
a
global
responsibility,
it's
a
lot
of
data
and
it
could
be
potentially
making
decisions
that
conflict
with
itself
if
it's
running
in
parallel.
This
also
has
one
of
these
global
locks,
that's
managed
through
console
and
then
last
and
maybe
least,
we
have
the
TPS
watcher
and
it
has
one
job.
It's
going
to
watch
for
instances
the
crash
and
translate
those
into
crash
events
for
cloud
controller,
to
report
to
end-users.
A
Ok.
So
in
the
last
few
minutes
let
me
give
you
an
overview
of
how
we've
integrated
the
routing
tears
into
this
new
Diego
back-end.
So
we
have
the
familiar
go
routers
down
here
in
the
lower
right,
and
then
we
have
this
component.
We
call
the
route
emitter
that's
deployed
alongside
that,
so
the
router
meter
is
paying
attention
to
all
the
work
that's
scheduled
in
the
BBS.
So
when
we
specify
a
new
lrp,
not
only
can
we
specify
what
should
run
in
the
container,
but
we
can
specify
some
networking
and
information
associated
to
it.
A
So
in
particular,
we
can
tell
Garden
I
want
to
expose
ports
in
this
case,
say
8080
and
9999,
and
then,
as
part
of
that
specification,
Cloud,
Controller
or
any
other
client
can
record
that
it
wants
to
associate
some
of
those
ports
with
certain
types
of
routers.
So
in
this
case
for
the
go
router,
we
have
a
tag
called
the
CF
router
and
it
can
specify
the
domain
name
that
that's
associated
with
and
then
the
port
the
traffic
should
come
into
the
container
on.
A
So
the
router
meter
detects
this
new
work
is
desired
and
starts
building
up
its
routing
table
in
memory.
So
it
knows
at
this
point
that
something
is
interested
in
routing
foo.com
requests
see
instances
of
this
app
or
of
this
lrp,
but
it
doesn't
have
any
instances
yet
so
when
one
shows
up
some
additional
information.
That's
present
when
it's
running
is
the
address
of
the
cell
that
that
instance
is
running
on
and
the
port
mapping.
A
So
if
something
outside
of
the
cell
wants
to
communicate
with
this
container
on
container
side
port
8080,
it
should
instead
talk
to
this
other
port.
That's
exposed
on
the
Sun,
so
the
router
matter
gets
that
update
too,
and
now
it
effectively
doesn't
inner
join
on
the
LR
P's
ID
and
that
port
to
register
that
new
endpoint
with
this
domain,
and
then
we
get
when
it
gets
these
updates.
It
sends
out
of
information
out
to
all
the
go
routers
well.
This
is
now
an
extensible
mechanism
that
supports
other
types
of
routing
on
the
platform.
A
So
in
this
case
it
knows
it
wants
to
associate
with
this
lrp
traffic
destined
for
port
9999,
with
a
particular
routing
group
and
a
particular
external
port,
and
then
I
can
update
those
TCP
routers
dynamically,
with
the
back-end
information
as
those
instances
come
online
or
die
off
okay,
so
that
is
a
large
part
of
the
functionality
of
what's
in
Diego
release
and
how
it
works.
And
let
me
tell
you
a
little
bit
about
the
next
steps
that
we're
planning
on
taking.
A
One
of
the
bottlenecks
that
we've
identified
as
fcd
itself
in
version
2,
so
we're
almost
done
switching
Diego
over
to
run
with
a
relational
store,
which
is
a
much
better
fit
for
the
kinds
of
data
operations
we
need
to
do
on
the
workload.
We're
managing
you've
also
heard
a
lot
about
the
persistence
effort.
So
in
fact
we
saw
a
demo
this
morning
of
WordPress
running
scaled
up
on
Cloud
Foundry,
which
is
pretty
amazing
and
the
persistence
team
is
now
working
on
ways
to
attach
different
types
of
storage
instances
such
as
individual
block
storage
for
instances.
A
We
also
have
a
container
to
container
networking
team.
That's
integrating
with
Diego
to
provide
richer
types
of
networking
between
containers
and
policy
to
enforce.
So
these
are
all
things
that
are
happening
on
teams
that
are
adjacent
to
Diego,
but
on
the
core
Diego
team
itself
we're
very
interested
in
exploring
notions
of
active
rebalancing
within
the
cluster.
So
we
can
make
greater
use
of
the
available
space
on
cells
by
doing
more
sophisticated
placement
decisions.
Right
now
we
give
absolute
primacy
to
running
instances.
A
A
Scripts
are
the
primary
uses
to
create
a
Diego
deployment
that
integrates
against
a
CF
and
there's
a
broader
plan
to
take
CF
release
and
break
it
into
these
kinds
of
subsystems,
Cloud
Controller,
UA,
the
routing
releases,
and
so
the
release
integration
team
is
working
on
their
efforts
to
produce
a
better
system
for
manifest
generation
that
will
incorporate
the
existing
CF
release,
components
as
well
as
the
Diego
ones,
so
we'll
have
a
unified,
manifest
generation
system
coming
soon.
But
for
now
it's
a
separate
deployment,
manifest
yeah.
A
So
right
now
we
don't
support
any
notion
of
cross
locality
for
that
scheduling.
All
those
scheduling
decisions
are
independent,
I
think
that's
another
interesting
placement
direction
to
go
in
so
now.
Maybe
you
have
a
system,
especially
in
the
presence
of
this
container
networking
policy,
where
two
different
LR
PS
should
be
communicating
with
each
other.
So
that's
one
direction
that
we
could
look
at
taking
placement
policy
depending
on
the
use
cases.