►
Description
Scaling Organizations Using Kubernetes [I] - Richard Fliam, Comcast VIPER
Kubernetes enables us to build software that spans hundreds or thousands of machines in a simple manner. Often overlooked, but just as critical, is that Kubernetes can enable us to build software that scales across hundreds of developers. Kubernetes can enable us to scale organizations.
If you have gone from 5 to 250 people in 5 years like Comcast VIPER serious challenges will arise in deploying, monitoring, debugging, and preventing outages. The default solution for these challenges is process. However, with appropriate tooling and leveraging the underlying API’s of Kubernetes it is possible to automate much of this process. We can lighten the burden of developing large, multi component features and services. We will explore the steps VIPER is taking to leverage Kubernetes to effectively scale our organization.
About Richard Fliam
Richard Fliam is a software engineer with years of industry experience in developing large-scale distributed systems. His work ranges from CDNs, to video packagers, to multi-terabyte per second cloud video recorders. He now leads VIPER’s Engineering Efficiency group. He has previously presented at Gophercon.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Okay,
what
about
20,000
and
if
you're
in
Google,
you
should
raise
your
hand
all
right,
so
I
am
Richard
climb,
I
lead
the
comcast,
I
/
engineering,
efficiency
team
and
those
of
us
who
are
lucky
enough
to
be
among
the
founders
of
a
really
successful
organization.
We
all
face
a
problem
scaling
software
is
hard
and
communities
helps
a
scale.
Software
scaling
organizations
is
actually
harder
and
Cooper
Nettie's
can
help.
You
scale
your
organization
as
you
grow
too.
So
seven
years
ago,
I
joined
a
three
engineer:
research
team
within
comcast
for
IP
video.
A
It
is
doubled
in
size
every
year,
since
that
organization
called
Viper
is
hundreds
of
engineers.
Today
we
have
doubled
the
number
of
micro
services
we
support.
Every
year
we
served
tens
of
millions
of
customers.
We
sell
our
products
to
other
providers.
We
deployed
tens
of
thousands
of
instances
into
more
than
50
data
centers
and
if
you
can't
read
that
architecture,
don't
worry
you're
not
supposed
to
like
oh
good
documentation,
it's
out
of
date,
anyways
and
I'm,
not
Q.
A
Today,
to
tell
you,
though,
about
all
the
wonderful
things
we
do,
because
that
would
be
very
educational
or
entertaining
instead
I'm
here,
to
tell
you
all
the
wonderful
ways
you
can
screw
up
growing
your
organization's
because
growing
organizations
as
I
said
it's
hard,
really
hard
and
hopefully
I
can
share
some
insights
on
how
to
fix
them.
I'm
going
to
start
a
little
bit
with
our
story.
All
numbers
are
approximate.
A
So
when
Viper
released
its
first
real
customer
application,
we
followed
a
fairly
familiar
pattern.
We
installed
Hardware
in
a
data
center
set
up
a
sophisticated
virtualization
platform,
created
some
VMS
on
it
and
started
up
copies
of
micro
services.
It
takes
about
five
engineers,
no
big
deal
so
customer
demand
spikes
for
our
product.
We
rapidly
set
up
more
machines
and
more
sites.
Anyone
who's
done
operations
on
hundreds
of
physical
and
thousands
of
virtual
instances.
Probably
knows
one
thing:
something
has
broken
all
the
time
at
every
hour
of
the
night
and
day.
A
A
Take
it
to
another
year
later,
for
us
customer
demands
spike
again.
Our
ever
industrious,
dev
team
has
added
two
new
products
and
another
development
team.
Now
we
need
thousands
of
instances
running
in
tens
of
data.
Centers
deployment
has
become
a
huge
problem
with
too
many
manual
configurations.
Many
deployments
have
missed
their
targets
and
left
things
and
broken
and
hard
to
debug
ways.
A
A
Take
it
another
year
and
you're,
seeing
the
effects
of
doubling
the
business
has
asked
for
a
huge
feature
set.
So
a
very
aggressive
roadmap
and
a
sophisticated
and
entirely
new
product
line
called
cloud
DVR.
We
now
serve
millions
of
users
every
day.
Our
products
include
dynamic
advertising
in
analytics
pipeline,
client-side
players
and
a
home
gateway
at
this
deployment.
At
this
point,
deployments
have
gotten
risky,
even
with
an
eye
toward
simplicity.
We
have
a
very
big
set
of
products
that
are
interconnected
and
many
things
in
the
stack
rely
on
each
other.
A
Subtle
changes
in
the
scaling
of
our
operate,
scaling
or
operations
of
our
AP
is
mean
outages.
Only
the
developers
themselves
can
troubleshoot,
which
is
hugely
problematic,
so
ensure
we
have
a
quality
problem,
particularly
a
multi
component
integration,
quality
problem.
So
what
do
you
do
when
you're
a
traditional
technology
organization?
A
You
hire
a
dedicated
to
18,
/
end
and
integration?
Oh
and
you
need
to
train
them
all
and
on
what
they'll
be
working
on
so
we're
not
about
a
hundred
engineers,
including
our
QA
folks,
one
year
later,
we're
now
so
successful.
People
in
the
industry
are
buying
and
deploying
art
components.
Instead
of
others,
we
need
an
Operations
team
just
to
handle
our
SAS
customers,
not
just
our
own
internal
infrastructure.
A
We
now
run
tens
of
thousands
of
virtual
instances
in
more
than
50
data.
Centers
are
internal
and
external
customers
are
demanding
new
features
and
pushing
hard
on
our
ability
to
even
scale
or
software.
The
sales
team
is
delaying
onboarding
new
customers,
because
we
simply
can't
keep
up
with
the
pace
necessary
for
deployment.
All
products
now
have
large
engineering,
QA
and
operations
teams.
New
hires
must
receive
information
about
all
technologies
and
products
and
that
information
about
new
technologies
and
products
that
needs
to
be
disseminated
to
everybody
in
the
organization
for
us
to
work
effectively.
A
A
Second,
organizational
units
will
become
boggled
bottlenecks
and
third
organizations
become
coupled
so
I'm
gonna
have
some
fun
with
math
here
I'm
going
to
write
a
recurrence
relation
for
organizational
communication,
assuming
a
complete
graph,
that's
one
in
which
all
nodes
are
connected
by
an
edge
for
those
of
us
who
haven't
done
this
in
a
decade.
We're
going
to
try
to
figure
out
the
flat
form
of
a
recursive
equation
will
call
this
update
function.
A
U
and
its
input
n
will
be
the
number
of
people
in
the
organization
for
my
organization
to
stay
in
sync
on
changes
with
any
people.
I
need
to
communications
with
everyone,
except
for
myself.
This
adds
the
term
you
see
on
the
right
there
or
mobile
and
2n
minus
11
communication
of
the
change
and
one
back
confirming
the
impact
or
the
lack
thereof.
So
every
other
team,
when
going
through
this
change,
needs
to
do
the
same
thing,
and
this
adds
the
recurrence
term:
u
of
n
minus
1,
so
we
obtained
the
recursive
relation
for
our
organization.
A
Update
function:
u
of
n
equals?
U
of
n
minus
1,
plus
2
n
minus
1,
and
if
you,
if
you
remember
the
master
method
and
you
apply
it
here-
you
get
n
squared,
so
changing
anything
in
our
org
and
validated
unit
to
impact
is
Big
O
of
N
squared
and
that's
Big
O,
because
that's
for
a
complete
graph.
If
we
have
a,
we
have
a
less
connected
graph,
we
can
do
better
than
that.
A
A
New
features
deep
in
the
stack,
must
be
communicated
and
tested
with
other
changes
in
operations
and
QA
trained
on
its
use.
So
the
more
connected
the
graph,
the
more
difficult
it
becomes
to
change
the
more
people.
You
must
work
with
and
inform
the
higher
of
the
bandwidth
of
that
communication
traffic,
and
this
is
a
nonlinear
relationship
right.
It's
N
squared,
even
if
we
do
that
successfully.
A
The
volume
of
that
information
is
going
to
exceed
the
ability
of
the
individuals
in
the
organization
to
process
it
which
encourages
us
to
bank
high
level
release,
notes
to
batch
changes
and
that
significantly
increases
our
risk,
which
decreases
our
ability
to
deploy,
which
decreases
our
ability
to
be
chained.
To
make
change
to
be
effective.
A
So
now,
worse,
our
velocity
of
now
so
awesome,
it's
breaking
QA.
They
can't
test
component
integrations
when
we
are
to
meeting
three
times
a
day,
put
products
with
major
features.
These
cycles
are
going
to
run
long
and
also
is
going
to
have
trouble,
deploying
new
versions.
Oh
and
the
port
infrastructure
teams
that
support
this.
Everyone
needs
an
EMS
and
hardware
from
them
all
of
the
time,
but
those
VMs
when
their
Idol
our
pain,
to
manage
and
keep
true
up
huge
chunks
of
resources,
and
they
must
be
continually
managed
by
those
teams.
A
This
is
because
our
organizations
all
rely
on
someone,
just
as
your
software
becomes
coupled,
so
our
do
our
organizational
units,
the
more
complete
that
connection
graph,
the
harder
it
is
to
do
anything.
An
organization's
ability
to
make
a
change
is
limited
by
the
ability
of
others
to
change.
This
decreases
the
ability
of
in
unit
to
take
effective
action.
A
A
Now
I
could
probably
pick
out
a
few.
A
DevOps
evangelist
out
there
that
are
jumping
about
about
to
jump
out
of
their
seats
and
say
developers
should
operate
your
software.
Then
you
shouldn't
have
to
train
operations
and
fight
deployments,
but
we
include
local
variance.
There
are
tens
of
thousands
of
channels
being
broadcast
to
literally
millions
of
users.
All
day
every
day.
In
the
comcast
footprint
we
do
cloud
DVR
recordings
in
the
tens
of
petabytes
per
day
heck
we
have
a
team,
that's
solely
responsible
for
replacing
failed
hard
drives,
and
that's
not
a
joke.
A
Here's
the
thing:
if
you
want
your
developers
to
operate
what
they
build,
they
have
to
be
even
capable
of
it.
Puppet
modules
and
VMS
doesn't
make
an
operable
infrastructure.
That's
like
running
thousands
of
machines
with
your
only
capability
being
to
edit
in
a
dot.
B
scripts.
Don't
get
me
wrong,
puppets
a
great
product,
but
it's
not
sufficient
for
that
kind
of
scale.
It
takes
a
five-man
team
with
an
organization
to
make
a
component
that
millions
of
people
tens
of
millions
of
people
use.
A
We
do
that's
not
that
big
of
a
deal
when
you
do
to
scale
the
effectiveness
of
the
work
we
do
and
simply
being
able
to
make
big
changes.
Also,
is
it
enough
you
need
to
be
free
to
do
so.
We
need
to
remove
the
limiters
on
our
ability
to
act
and
increase
the
effect
of
those
actions,
which
is
why
M
by
M
communications,
bottlenecks
and
organizational
coupling
are
so
bad.
They
all
limit
our
ability
as
engineers
to
act,
so
coop
can
help
us
here.
The
khoob
website
refers
to
itself
as
a
container
orchestrator
I.
A
Think
of
it
as
a
distributed
operating
system,
it
helps
abstract
us
from
the
hardware
right.
It
provides
sophisticated,
multi
user
permissions,
especially
with
1.6.
So
let's
talk
about
solving
that
huge
operational
overhead.
The
biggest
reason
we
need
a
dev
ops
team
for
an
application
is
simply
the
number
of
instances
we
need
to
care
for.
No
development
team
could
interact
with
all
the
necessary
machines
in
an
efficient
manner
sure
we
could
register
them
all
and
zookeeper
provide
necesita
to
do
automated
routes
and
pups
have
puppet
enforcing
their
changes
happen.
A
A
A
So
in
Cuba
Nettie's
PSN
system
cuddle
functionality
is
effectually
handled
by
cube
cuddle.
So
what's
iostat
is
griffin
out
or
sis
big
or
CC
adviser,
there's
tons
of
great
stuff
out
there
for
this
without
journal
cuddle,
that's
your
elk
or
splunk
stack,
and
they
just
keep
saying
cuddle
on
this
slide.
So
what
about
estrace?
That
is
something
like
open
tracing
for
us
or
Zipkin.
A
But
it's
not
enough
that
those
tools
exist.
They
need
to
just
work.
I,
don't
need
to
modify
my
programs
to
use
iostat,
it's
part
of
the
platform.
It's
well
documented
and
that's
a
key
both
up
taken
effectiveness.
Any
organization
common,
easy-to-use,
tooling,
that's
available
to
everyone
and
just
like
UNIX
I,
also
need
sophisticated
permissions
and
process
isolation.
I
can't
operate
independently
of
others
on
a
shared
platform.
If
I
could
accidentally,
let's
go
with
accidentally
forth
on
the
system.
That
means
I
need
you
guessed
it
resource
limits
and
cuban
the
cloud
native
environment
provides
these.
A
It
provides
us
a
data
center
operating
system,
okay,
so
now
you've,
given
our
developers
a
platform,
they
can
use
to
operate
our
system,
but
what
about
deploying
new
ones?
What
about
meeting
the
ever-growing
number
of
customers
who
use
our
software?
Let's
look
at
how
to
solve
this.
It
Kiva's
the
Linux
of
our
data
center
I
need
an
RPM
and
young
for
deploying
and
managing
installed
software.
A
So
organizations
sit
down
to
write
these
for
ourselves.
We
needed
a
way
to
avoid
concrete
versions
of
a
whole
micro
service
system
that
can
be
tested,
upgraded
and
migrated
online.
So
we
call
those
two
items:
Geronimo
and
ticklish
internal,
the
comcast,
but
we're
hoping
to
open
source.
Soon
we
have
Geronimo,
it
interacts
with
the
queue
API
to
store
the
versions
of
deployed
software
track
across
clusters.
A
What
software
is
deployed
and
abstract
environmental,
specific
configs
Geronimo
via
the
cube
API,
provides
automated
deployment
and
rollback
hooking
into
arbitrary,
develop
validators
and
those
are
defined
in
what
we
call
our
Geronimo
file.
They
can
monitor
production
production,
metrics,
run
smoke,
tests
upgrade
database
and
well
anything
that's
what
arbitrary
means
now
that
we
have
a
way
to
package
and
deploy
software
in
a
repeatable
manner,
dalinar
p.m.
we
can
vastly
simplify
our
QA
and
release
process.
A
So,
let's
be
sized
on
that
for
a
second,
the
cube
API
lets
us
label
deployed
pods
and
software.
So
in
those
labels
the
tool
writes
the
version
the
system
we
expect.
You
saw
that
in
the
screenshot
here
to
verify
that
the
things
are
what
we
expect
them
to
be.
We
store
a
cryptographic
hash
of
all
related
JSON
items
like
replication
controllers,
endpoints,
etc.
In
the
system
annotations,
which
annotations
are
not
mutable
at
runtime,
each
system
is
kept
in
a
Geronimo
file,
which
is
a
wrapper
around
the
cube
format.
A
It
describes
the
environment,
independent
format
of
the
system.
Remember
I,
said
we're
in
over
50
data
centers,
and
that
has
to
do
with
pushing
video
very
close
to
the
edge.
An
environment
file
has
been
combined
with
the
system
file
and
it
contains
environment.
Specific
configs
things
like
replication,
counts
secrets
for
that
environment,
endpoints
in
that
environment
that
are
specific,
etc.
So,
when
Geronimo
goes
to
upgrade
a
system,
it
stands
up
a
new
name
space
with
the
new
version.
A
A
But
what
if
we
want
to
install
whole
groups
of
software
that
are
dependent
on
each
other?
Again,
we
need
a
young
leveraging.
The
cuban
Nettie's
api,
if
we
are
developing
to
cushion
to
integrate
these
into
digital
systems
in
detestable
services
it
like
yummm,
resolves
and
installs
all
the
tendencies
of
a
system
based
on
version.
Indeed,
like
yums
group
install,
it
becomes
possible
to
simply
install
a
linear
video
line
that
consists
of
quite
literally
hundreds
of
components
and
then
we'll
also
warn
you
statically
as
you
build
the
system
if
dependencies
are
not
satisfied.
A
So,
to
cushion
begins
with
a
component,
that's
a
whole
artifact
like
a
docker
container.
It
layers
on
this
the
concept
of
its
system,
as
in
a
geronimo
system
which
is
composed
of
those
components
that
might
be
something
like
the
set
of
micro
services,
we
use
for
cloud
DVR
recordings.
Each
system
is
assumed
to
be
a
flat
working
namespace
and
a
has
a
methodology
to
deploy
like
Geronimo.
A
Those
systems
are
then
linked
together
to
form
a
service.
The
service
is
something
like
video
on
demand,
and
video
on
demand
is
a
really
big
thing.
It's
advertising
systems,
it's
video,
back-end
systems,
configuration
systems,
a
lot
of
things
and
those
are
managed
by
huge
groups
that
are
disparate
from
each
other
and
usually
often
in
separate
namespaces
with
separate
permissions.
A
So,
just
like
an
RPM
can
provide
and
require
certain
files.
We
allow
systems
to
provide
and
require
configurations
and
linkages.
This
lets
us
statically
determine
the
wiring
of
a
system,
that's
kind
of
like
if
dr.
compose
was
applied
to
coover
Nettie's
namespaces.
We
also
allow
it
to
link
and
compose
arbitrary
services.
Things
like
cameras
for
playback
testing,
physical
hardware,
databases
that
we
can't
get
into
cube
yet
and
those
kinds
of
items
so
notice.
A
If
you
want,
you
just
grab
the
technician
file
for
linear
and
we'll
describe
to
you
not
me
how
everything
is
deployed,
but
how
it's
linked
together
for
something
like
video
on
demand,
so
the
amount
of
necessary
communication
in
our
organization
drops
dramatically
and,
as
all
the
developers
here,
I'm
sure
would
love
the
amount
of
necessary
documentation
drops
dramatically.
The
system
is
self
documenting
with
this.
A
Our
infrastructure
team
no
longer
has
to
work
on
the
behest
of
each
application
came
with
cube
to
they
deployed
bare
metal
provisioning
that
provides
us
with
the
capability
to
bootstrap
clusters
in
a
single
go.
Our
crew
vanetti
self,
deploy
onto
hardware
as
it's
plugged
in
as
infrastructure,
has
changed,
rebooted
or
otherwise
keeps
ability
automatically
utilize
or
withstand
failures,
utilize
the
new
equipment
or
withstand
sailors.
What's
these
groups
move
quickly
without
necessarily
notifying
and
working
the
teams
which
run
on
their
stocks?
A
Think
about
a
AWS
for
those
of
you
who
use
it
had
to
notify
you
every
time
they
reverted
a
note
in
this
new
world
for
us
they're,
free
to
focus
on
owning
and
operating
that
infrastructure.
They
develop
an
independent,
dev
ops
team
from
bare
metal
provisioning
I'm
going
to
deep
dive
on
that
per
second
boxes,
only
need
to
have
power
and
network
cables
plugged
in
they
kick
see.
A
A
That
means
they're
immutable,
which
is
important
because
it
means
our
test
environments
match
production,
no
point
in
testing,
something
that
doesn't
match.
Production
and
drift
of
the
configuration
of
the
underlying
platform
is
much
more
difficult
to
have
happen.
This
process
decouples
our
platform
teams
from
our
hardware
deployment
teams.
How
would
the
point
it
doesn't
even
have
to
notify
them
when
it's
done?
A
Hud
will
just
appears
in
our
platform
inventory
as
it's
powered
and
cabled
in
think
of
the
power
of
that
compared
to
I
plugged
it
in
please
go
install
the
OS
and
set
up
the
network
correctly
in
the
DNS,
and
all
of
that
now
we
just
come
up
so
to
D
couple
our
platform
teams
from
our
software
teams.
We
need
pop
forum
updates
to
be
low
risk
and
repeatable
and
frequent.
A
So
how
do
you
accomplish
that?
Cipd?
The
same
CIC
d
platforms,
the
deployer
software
now
deploy
our
platforms.
First
deployment
is
done
to
test
an
environment,
the
two
a
test
environment,
that's
running
a
production
like
config
and
production
like
applications
as
that
deployment
runs
a
sophisticated
and
aggression
integration
test
suite
runs
it
does
things
like
verified
doctor
doesn't
hang
offer
those
of
you
who've
run
into
that
networking
works,
no
performance
degradation
have
happened,
logging
still
works,
etc.
A
So
when
that
piece
of
continuous
integration
is
done,
we
begin
the
continuous
deployment
part
rowing
upgrades
begin
in
production.
We
again
run
that
integration
seat
sweet
as
we
deploy
production
and
we
also
have
can
hook
into
production,
metrics
and
auto
back
out,
deploys
that
are
failing
or
causing
applications
underneath
us
to
die,
so
that
makes
her
platform
teams
capable
of
deploying
freely
and
often
and
with
confidence.
A
They've
also
provided
features
like
user
allocated,
high
performance
load
balancers
for
our
platforms
for
details
on
that,
you
should
go
to
my
colleague,
Kim's
talk
tomorrow
in
this
new
world
teams
need
far
less
communication
about
the
new
releases.
The
register
contains
all
that
information.
This
makes
what
communication
we
do
have
far
more
efficient
on
group
between
the
groups,
it's
much
more
focused
on
the
technical
operation
or
that
technical
details,
rather
than
the
operations
of
the
components
everything
I
have
described
here,
is
about
decoupling.
A
The
operations
of
these
teams
at
an
organizational
level
and
giving
those
teams
a
platform
to
manage
their
own
large
complicated,
distributed
systems
as
easily
as
I
can
administer
my
local
linux
box.
Having
also
encoded
the
information
about
the
current
running
state
of
the
system
in
the
system,
it
becomes
self
discoverable
and
self-describing.
A
So
these
are
the
tenants,
the
drive,
our
ability
to
scale
teams,
discovery
and
dissemination
of
actionable
knowledge,
the
ability
to
create
impactful
change
and
the
safety
and
the
freedom
and
the
comfort
to
do
so.
We
want
to
empower
people,
provide
them
with
independence
of
action
in
unity
of
direction.
To
do
that.
A
For
the
organization
operating
a
large
distributed
system,
we
must
have
a
complete
platform,
a
platform
that
is
easy
to
use
and
transparent
in
its
operation,
a
platform
that
automates
as
much
work
as
possible
from
the
users
but
lets
them
configure
it
as
necessary
from
process
management
to
metrics
to
deployment
and
upgrades
through
teams
will
also
focus
on
decoupling
from
others.
Acting
is
complete
platforms
themselves
from
other
teams,
which
was
mentioned
today
in
the
keynotes,
as
our
infrastructure
team
has,
they
can
dramatically
increase
their
effectiveness.
A
They
no
longer
need
to
continually
work
and
do
that
pole
with
everybody
else
around
them,
and
this
is
why
cube
is
such
an
effective
tool
for
scaling
and
organization,
because
it
allows
teams
to
operate
independent
of
each
other,
allows
us
to
provide
simple
declarative
definition,
nervous
system
and
it's
sophisticated,
yet
simple,
api's
make
building
tooling
to
meet
your
own
needs
very
simple,
and
that's
that's
very
key.
The
cube
api's
are
a
huge
part
of
building
a
successful
organization,
just
as
the
Linux
kernel
api's
are
a
huge
part
of
building
successful,
Linux
application.
A
So
most
of
the
data
centers
are
actually
used
by
all
teams
contests,
as
I
mentioned,
we
have
a
lot
of
users
and
we
also
own
the
network.
So
it's
in
our
own
best
interest
to
push
that
video
traffic
as
close
to
the
users,
as
we
can
so
most
of
the
data
centers,
like
actually
all
teams
deploy
to
all
data
centers.
A
So
we
don't
now
there
are
what
we
call
national
data
centers
and
we
don't
have
obviously
to
keep
1.6
Khattak
stuff
deployed.
Yet
that's
a
that's
a
major
portion
underway.
We
also
have,
as
I
mentioned,
done,
deployment
of
tens
of
thousands
of
puppet
VMs,
that's
run
by
our
operations
team
and
we're
slowly
making
the
change
to
try
to
move
as
much
as
that
possible
into
cube.
Those
operators
can
focus
on
operating
the
system
more
like
a
whole
system
and
less
as
a
little
services.
A
So
we
don't
that
we
don't
use
an
inventory
tool
today
for
platform
inventory
to
cushion
acts
like
a
young
server,
in
that
it
tells
you
what
what
components
are
available
to
be
deployed
as
far
as
it
also-
and
this
is
where
it's
not
like
yum,
it
actually
is
probably
the
piece
that
managers
platform
inventory
the
most,
though
those
are
usually
individual
components.
Unfortunately,
most
of
our
platform,
most
of
the
platform
inventory
stuff
out
there
wouldn't
work
with
what
we
do
like
I
mentioned
via
physical
cameras
that
do
playback
testing.
A
A
B
B
A
Okay,
so
first
question:
not
everything
can
be
put
in
containers
and
that's
absolutely
true.
We
have
onions
of
products
that
don't
fit
in
containers,
I
mentioned
transcoders
and
we
tens
of
thousands
of
those
they
don't
generally
fit
well
in
containers
because
they
run
on
asics
the
way.
The
way
you
deal
with
that
is
try
your
best
to
make
it
look
like
containers,
which
is
what
you
do
with
puppet
and
VMS
at
scale.
A
Anyways
puppet
enforces
that
it's
a
container,
it
can't
be
changed
and
you
have
vm
images
that
come
up
and
come
down
in
auto
scale
and
deploy
there's
not
much.
You
can
do
beyond
that
and
then
try
to
make
it
look
like
containers
for
the
stuff
that
can't
be
put
in
containers.
The
other
thing
we
we
face.
A
As
I
mentioned
collision,
you
can't
put
an
iPad
in
a
container
I
need
to
physically
watch
the
iPad
and
I
can't
like
scrape
the
screen
of
what
should
be
playing,
because
that
would
break
our
drm
and
I
need
to
make
the
ER
dr
m's
working
so
that
that
kind
of
stuff
you
write
managers
on
top
of
again
your
own
little
ones
and
try
as
best
as
you
can
to
integrate
our
infrastructure.
Your
second
question
was
sorry
OpenStack.
A
So
comcast
has
three
separate
virtualization
platforms
internally,
and
it
depends
on
the
application
for
the
video
stuff,
which
is
what
Viper
does
we
tried.
It's
been
as
much
of
that
on
bare
metal,
because
that
is
all
I
owe
bound
and
virtualization
platforms,
not
always
the
best
at
I/o
bound
tasks
like
the
reason
our
initial
deployment
was.
Tens
of
boxes
is
because
egress,
even
in
the
smallest
site,
was
more
than
10
gigabits.
A
second,
so
more
than
a
single
Nick
would
normally
run
in
really
large
sites.
A
It's
hundreds
of
gigabits
per
second,
so
running
that
off
of
virtualization
platforms
is
a
huge
overhead,
which
is
why
we
have
so
many
bare
metal
provisioning
and,
as
I
mentioned,
Kim's
Park
actually
goes
in
a
bit
into
depth
on
on
what
we
do
for
load,
balancing
and
stuff
for
that
we
do
have
open
stock
and
we
do
have
these
fear
for
virtualization
of
more
kind
of
normal
applications
and
those
are
run
by
our
clouds.
We
have
an
internal
cloud
came
as
well.
A
We're
very
poly
got
I
think
we
have
least
six
or
seven
deployed
languages,
and
we
don't
actually
were
not
like
a
lot
of
big
organizations.
We
don't
declare
official
languages,
so
people
use
whatever
we
have
Scala
we
had
Java.
We
have
go
people
out
to
go,
we
have
see,
we
have
c++,
we
have
it's
it's
whatever.
Those
teams
want
to
use.