►
Description
Case Study: Kubernetes at Comcast - David Arbuckle, Comcast VIPER
"As of September 1st, Comcast VIPER has deployed Kubernetes to 8 distinct physical locations. Three of these locations are production sites, running applications that provide video services to paying subscribers.
When VIPER began development on its Cloud DVR application early last year, it quickly became apparent that the scalability and uptime requirements were going to incur a high operational cost. With upwards of 15 components in a micro services architecture, we placed a bet on Kubernetes to help us develop this application quickly, and to quickly deploy it and maintain deployments in upwards of 10 regionally-located datacenters.
VIPER’s solution integrates log aggregation, monitoring, and load balancing as first class features of our Kubernetes deployments. These features, and the convenience of deploying into Kube, have helped to drive adoption of Kubernetes platform with other teams and products.
But growth is challenging. We’ll discuss the social aspect of Kubernetes in the enterprise. How we deal with developer and management expectations, and how features like DNS, beta extensions, and configmaps have inspired spirited debate within our organization."
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
I'm
gonna
give
a
talk
called
kubernetes
at
Comcast,
really
just
sharing
our
experiences
getting
kubernetes
deployed
into
production.
What
it
took
to
get
that
done
and
and
kind
of
how
we
how
we
managed
to
to
do
it.
I'm
gonna,
try
to
breeze
through
this,
because
there's
a
lot
of
detail
in
here
and
not
a
lot
of
time
and
I'd
like
to
leave
some
time
for
questions
at
the
end.
So
I'll
start
with
with
a
quick
introduction
to
to
myself
I'm
a
software
engineer.
A
So
cloud
DVR,
some
of
you
guys,
might
have
a
DVR
cloud.
Dvr
is
DVR
in
the
cloud.
We
have
a
legal
requirement
to
store
a
unique
copy
for
every
user,
so
when,
when
one
user
hits
record,
another
hits
record
we
have
to.
We
have
to
store
two
copies
of
that
video
and
that
actually
gives
us
some
really
interesting
technical
requirements.
We
have
to
commit
seven
terabits,
a
second
of
data
to
disk
and
when
we're
playing
stuff
back,
we
have
to
play
each
subscribers
copy
back
unique
copies.
A
So
we
don't
get
to
leverage
any
of
the
caching
strategies
like
a
CDN
to
deliver
content
to
our
subscribers.
For
this,
because
each
subscriber
has
to
get
a
unique
set
or
slice
of
a
bit.
So
we
we
egress
300
gigabits,
a
second
our
database
that
backs
this
service
needs
to
support
a
million,
writes
a
second
and
two
and
to
make
this
especially
crazy.
We
had
to
deploy
this
to.
A
We
have
six
components
in
here
that
would
be
deployed
into
kubernetes,
plus
some
back-office
components
and
five
external
dependencies,
things
that
interact
with
the
system
in
order
to
to
store
content,
store,
metadata
and
and
and
provide
provide
content
to
the
cloud
give
you
our
system,
this
being
our
most
complex
architecture,
we
knew
we
were
going
to
have
a
hard
time
with
it,
and
the
reason
for
that
is
that
our
existing
deployment
processes
once
so
great
we
had
a
virtualization
layer.
We
had.
A
We
had
vmware
in
production
that
under
cloud
vmware
in
all
of
these
different
data,
centers
was
what
data
centers
was
experiencing
configuration
drift
so
everywhere
that
we
had
deployed
it.
We
were
running
a
different
version,
as
our
cloud
team
would
would
roll
a
new
point
release
of
vmware
out
by
the
time
they
made
it
through
the
entire
infrastructure.
We
were
already
several
releases
behind.
Existing
projects
required
a
full
time
release
management
because
our
our
applications
have
this
matrix
of
compatibility.
All
of
these
different
services.
A
What's
running
in
one
one
market
versus
another,
and
and
how
do
we
manage
that
in
adding
more
components
and
creating
a
more
complex
architecture
means
that
that
you
know
we
would
have
to
double
down
on
that
existing
people
process
and
and
and
really
like,
invest
in
in
and
something
that
we
didn't
want
to
invest
in
provisioning.
New
VMS,
getting
new
capacity
was
a
ticket
based
system,
and
we
didn't
have
a
way
to
transition
from
the
lab
to
production.
A
Developers
would
would
provision
a
VM
in
the
lab
and
they
would
just
run
new
versions
of
their
application
on
it
and
and
when
it
came
time
to
go
to
production,
we
just
throw
the
app
over
the
wall.
Here
you
go
figure
it
out.
We
got
the
documentation
on
the
wiki
good
luck,
so
so
all
of
this
kind
of
left
us
with
a
with
low
confidence.
You
know
the
existing
infrastructure.
The
existing
deployment
model
wasn't
successful.
So
so
we
had
to
study
to
find
a
new
a
new
model.
A
These
are
our
high-level
requirements,
and-
and
these
are
you
know,
pretty
simple
I-
think
manageable
deployments
lab
production
parity
managed
risk.
You
know
these
are
real
products
with
real
customers.
When
we
roll
this
stuff
out,
it's
not
like
we're
we're
a
scrappy
startup
who
is
slowly
ramping
up
the
customers
and
can
solve
these
problems
at
over
time
like
we
have
to
we
roll
these
things
out
to
a
large
existing
customer
base,
and
and
because
of
that
it
has
to
work
and
it
has
to
work
right
away
and
ultimately,
one
one
faster
release
cycles.
A
So
we
kind
of
had
two
options
here.
We
could
scale
up
our
existing
people
processes
or
we
could
find
something
new
and,
and
there
were
a
lot
of
new
options
at
the
time
this
was
about
a
year
ago,
year
and
a
half
ago,
you
know
we
had
meso
stalker
swarm,
some
sort
of
Frankenstein's
monster
of
different
hasha
Cort
utilities
hum
all
like
tied
together
into
something
that
works,
and
then
there
was
kubernetes
which
had
just
hit
one
dot,
o
and
kubernetes
gives
us
containers
and
encapsulation
and
kubernetes
provides
containers
at
scale
really
like
declarative
deployments.
A
I
can
define
my
deployment.
I
can
run
it
on
on
kuba,
local
or
mini
cube,
something
that
actually
didn't
exist
when
we
started
this
and
then
I
can
use
that
same
deployment
when
I
go
to
production
or
into
the
lab
things
like
service
discovery
are
a
big
deal
for
us
because
running
multiple
sets
of
load
balancers
means
that
any
troubleshooting
exercise
you
know
requires
an
investigation
into.
Is
that
the
load
balancer
or
the
component
or
the
next
load
balancer
in
the
chain?
And
then
you
look
looking
towards
the
future.
A
Kubernetes
is
very
promising
with
respect
to
providing
multi-tenancy
and
if
we
used
it
correctly,
it
wouldn't
necessarily
lock
sin,
and
that
was
a
big
selling
point.
As
we
didn't
have
to
architect
and
design
or
applications
in
containers
with
kubernetes
specifically
in
mind,
we
could
design
a
12
factor,
app,
dump
it
in
a
container,
run
it
in
kubernetes,
and
if
it
worked
that's
great
and
if
it
didn't
we
could,
we
could
scramble.
We
could
go
back
to
the
drawing
board
and
figure
something
out.
So.
A
A
We
had
to
answer
the
question
of
load
balancing
very,
very
early
on
and
we
had
to
solve
monitoring
and
log
R
again,
these
things
needed
to
be
centralized,
because
you
know,
as
we
deploy
to
all
of
these
different
facilities,
we
don't
want
to
have
to
hire
a
team
to
operate
every
single
deployment.
We
wanted
to
be
able
to
to
have
one
team
that
understood
the
applications,
use
one
single
pane
of
glass
to
actually
manage
and
operate
these
things
things
like
docker
needed
to
work.
A
So,
with
respect
to
our
deployment,
automation
and
getting
kubernetes
out
there,
you
know,
kubernetes
is
great
because
it
solves
configuration
drift
for
applications,
I
declare
what
I'm
gonna
have
and-
and
it
just
runs
in
kubernetes
and
I-
can
reuse
that
declaration
in
in
different
kubernetes
clusters.
But
we
had
to
solve
that
for
kubernetes
itself
and
getting
that
deployed
and-
and
we
kind
of
had
some
principles
with
respect
to
that.
A
So
you
know
we
had
some
goals
what
you
wanted
to
get
to
be
our
source
of
truth
and
and
we
wanted,
we
really
wanted
for
our
deployments
and
our
upgrades
to
follow
the
same
process
so
so
know.
Like
you
know,
yum
install
or
yum
update
type
of
thing
happening.
We
wanted
to
use
the
exact
same
process
for
deploying
a
net
new
server,
as
we
were
using
for
upgrading
an
existing
server,
keeping
in
mind
that
we
were
deploying
this
on
two
on
two
bare
metal
machines
and
our
own
data
centers.
A
Now
we
also
needed
to
support
other
applications,
things
that
exist
outside
of
the
context
of
kubernetes
entirely,
because
we're
deploying
a
much
larger
application
that
entails
storage,
object,
storage,
databases,
things
like
that,
and
we
kind
of
wanted
to
think
about.
Well,
how
do
we?
How
do
we
manage
this
change
over
time
as
well?
You
know
we
have.
We
have
a
lot
of
facilities
when
these
applications
that
we
deploy
are
going
to
exist
for
a
long
time.
You
know
it's,
it's
pretty
easy
to
get
a
demo
running
and
say
yeah.
This
is
good
enough.
A
Let's
roll
out
to
production,
but
but
figuring
out
how
to
how
to
manage
that
long
term
and
ensure
that
a
year
and
a
half
from
now
we're
not
three
point
releases
behind
the
the
main
line
of
kubernetes
is
important
to
us.
So
you
know
we
we
had
this
goal:
let's
get
an
immutable
immutable
infrastructure.
Let's,
let's
declare
what
we
have.
A
You
know
we
treated
the
entire
set
of
configurations
for
our
under
cloud
as
a
as
a
deployable
entity
and
we
roll
that
out
to
the
lab
and
then
our
other
labs
and
then
into
production,
and
this
is
something
that
reduces
errors,
so
outcomes
for
what
we
did
here.
The
systems
that
we
that
we
built
have
successfully
deployed
a
thousand
physical
servers
in
production
today.
A
Capacity
augmentation
for
kubernetes
is
fully
automated.
In
fact,
when
we
were
first
trialing
the
system,
our
deployment,
engineering
teams,
racked
and
stacked
fifteen
additional
servers-
and
they
didn't
tell
us
about
it-
and
we
found
them
in
the
kubernetes
api
when
we
were
wondering
why
why
we
had
additional
capacity,
which
was
a
good
gap.
The
issues
with
this
model,
its
its
fleet,
is,
is
not
a
great
way
to
deploy
software.
It's
an
all-or-nothing
approach.
We
can't
just
roll
a
change
through
that
cluster
and
or
through
through
a
cluster.
A
We
have
to
turn
a
service
off
and
turn
a
new
one
back
on
and
vendor
products
are
challenging
to
integrate
when
you
buy
a
service
or
a
system
from
a
vendor,
it's
tough
to
to
provide
them,
there's
a
requirement
that
all
we
have
to.
You
need
to
allow
us
to
instrument
an
update
of
your
software
through,
through
a
variety
of
through
a
series
of
reboots
vendors.
Don't
really
like
that
and
we're
working
on
a
version
two
of
this
that
we
intend
to
open-source.
A
So
another
challenge
that
we
had
to
overcome
was
load.
Balancing
I
just
came
from
a
pretty
good
talk
on
ingress,
ingress
didn't
exists
and
we
had
some
pretty
egregious
bandwidth
requirements.
We
couldn't
just
deploy
a
couple
of
VMs
with
a
che
proxy
and
be
done
with
it.
We
needed
something
that
would
scale
across
the
entire
cluster
and
let
us
omit
a
lot
of
data.
Our
requirements
are
really
scale:
reliability,
no
VMs
and
having
some
sort
of
failure.
Semantics
in
the
event
that
that
a
master
fails
over.
A
If
you
guys
haven't
heard
of
IP
vs
or
you
used
it,
it's
a
really
great,
really
great
tool
to
have
in
your
in
your
toolbox.
Briefly
how
it
works.
You
have
a
high
PBS
master.
This
is
a
component
that
that
publishes
a
VIP
on
its
on
its
public
interface
to
the
router.
All
of
your
inbound
traffic
comes
to
this
master.
It
routes
traffic
to
the
back
end,
which
has
an
anycast
address
and
that
anycast
address
and
that
back-end
replies
directly
to
the
client.
A
So
you
end
up
with
a
ingress
bandwidth
equal
to
whatever
the
throughput
is
on
that
master,
node
and
egress
equal
to
whatever
the
sum
is
of
all
of
the
throughput.
On
all
of
your
back
ends
to
make
this
work
in
kubernetes,
we
couldn't
just
deploy
a
PBS.
We
wanted
to
make
it
really
easy
for
developers
to
add
a
new
load
balancer
and
to
do
that
quickly
and
to
know
what
we
had
configured
in
ipbs.
A
So
we
created
a
tool
that
uses
the
config
maps
to
store
a
representation
of
of
what
VIPs
were
assigned
to
what
what
IP
vs
services.
So
our
master
would
get
the
config
map,
it
would
get
the
nodes
and
it
would
configure
IP
vs
ATM
with
all
of
the
rules
that
were
necessary
in
order
to
load
balanced
traffic
to
all
of
those
different
backends.
And
then
on
the
backend
side.
A
We
would
get
the
config
map,
which
would
have
a
VIP
and
a
port
and
a
service
in
the
namespace,
and
it
would
take
that
data
and
it
would
use
it
to
write
IP
tables
rules
in
the
in
the
NAT
table
that
would
route
packets
that
were
inbound
on
a
VIP
to
a
specific
kubernetes
service.
So
a
very
high
performance
low-level
load,
balancer
that
that
actually
works
and-
and
it
works
pretty
well.
A
The
the
drawbacks
of
this
are
are
that
it
is
limited
to
a
single
broadcast
domain
that
that
model
of
replying
directly
to
a
client
requires
that
all
of
the
nodes
are
layer,
2
adjacent
so
so
currently
we're
limited
to
like
255
nodes,
but
we
have.
We
have
plans
to
just
scale
that
out
even
further,
potentially
by
using
bgp
or
by
using
some
of
the
the
networking
technologies
that
are
featured
out
in
the
in
the
hallway.
A
So,
like
the
last
thing
that
we
really
needed
to
do
to
make
kubernetes
useful
for
our
developers
was
logs
and
monitoring
and
and
this
kind
of
ties
into
into
like
bringing
a
DevOps
model
into
an
enterprise
team
to
the
dev
team
that
was
developing
this
cloud.
Dvr
application
I
didn't
have
operational
experience.
You
know,
we've
deployed
some
applications,
but
we
throw
those
apps
over
the
wall,
and
that
was
it
and
this
development
team
was
very
new.
A
A
So
any
any
system
that
we
introduce
in
order
to
get
developers
to
use
kubernetes
and
use
these
systems.
You
know
we
really
had
to
deliver
immediate
value,
because
new
tooling
requires
a
cultural
shift
and
a
lot
of
indoctrination,
which
you
might
not
have
time
to
do
if
you're
just
struggling
to
get
kubernetes
running
so
some
other
high-level
goals.
You
know
we
wanted
to
reuse
existing
implementations
and
we
needed
we
needed
to
to
support.
A
You
know
deployment
to
multiple
data
centers
and
to
get
that
single
pane
of
glass
like
if
we
have
20
places
that
we
need
to
look
for
monitoring
data.
We
need
20
people
to
operate
that
product,
so
so
two
approaches
for
logs.
We
reused
existing
services,
so
our
our
containers
running
in
docker
on
a
kubernetes
node.
That's
what
this
big
box
is.
They
write
their
logs
to
standard
out
using
Dockers
JSON
log
driver.
We
initially
tried
turning
that
off.
A
Right
and
and
like
one
of
the
keys
to
this
is
that
we're
not
asking
developers
to
to
ship
their
logs
to
a
TCP
port.
It's
write
your
logs
to
standard
out,
write
your
container
12
factor
and
we'll
figure
the
rest
out
for
you
and
I
think
that
works.
That
model
works
really
well
at
getting
people
to
adopt
something
new
like
kubernetes.
Now,
monitoring
is
a
little
bit
different
for
monitoring.
We
replaced
the
existing
solution
because
there's
the
existing
solution
doesn't
have
anything
in
place
like
what
we're
using
to
monitor
now.
A
Comcast
doesn't
have
a
good
mechanism
for
monitoring
containers.
You
can
do
buddy
containers
which,
which
I
strongly
disagree
with.
You,
can
run
multiple
processes
in
a
container,
but
but
you
know,
ideally,
we
want
a
system
that
doesn't
require
instrumentation.
So
we
found
this
monitoring
tool
that,
just
like
does
some
clever
stuff
with
system
calls
and
ships
them
off
to
a
single
pane
of
glass
and
that
that
actually
worked
really
great
for
us.
A
Worth
mentioning
too
that
when
we
started
the
project,
prometheus
I,
don't
think
existed
in
a
in
a
usable
form
and
I'm
still
a
little
bit
skeptical
about
that.
So
you
know
was:
was
this
a
success?
Kind
of
our
teams
were
meeting
hundreds
of
gigabytes
of
logs
per
day
per
site
like
we,
we
gave
them
log
aggregation.
We
gave
it
to
them
in
the
labs
and
then
they're
like.
Oh
this
stuff's
free,
let's
ship,
all
of
our
logs,
let's
let's
emit
twice
as
many
logs
as
we
need,
and
they
did
it,
but
largely
a
success.
A
A
So,
let's
talk
about
some
problems
that
we
ran
into,
we
ran
into
problems
with
everything.
Kubernetes
kubernetes
is
great,
but
we
did
run
into
into
three
kind
of
serious
issues
very
early
on.
We
had
the
user
space
load
balancer
in
in
place,
and
that
was
the
default
service
load.
Balance
of
recruitment
at
ease.
Our
very
first
load
test
knocked
that
over
it
was
a.
It
was
a
single
process,
load
balancer
and
you
couldn't
send
more
than
five
gigabits,
a
second
of
traffic
into
a
single
service.
A
The
last
thing
that
we
that
we
run
into
or
we
ran
into
but
had
to
do
with
nap,
hairpin
mode,
so
a
pod
if
it
sends
a
request
to
its
own,
its
own
host
on
a
node
port
to
another
pod.
Another
container,
that's
running
on
that
node
the
packet
would
just
get
lost,
it
would
hang
and
there's
actually
a
configuration
parameter
in
the
coop
proxy
that
needs
to
be
set
in
order
to
support
that
sort
of
traffic.
He
said.
Well,
why
would
you
do
that?
A
I,
don't
know,
I,
don't
have
a
node
on
that,
but
but
it's
one
of
the
things
that
that
you,
you
might
do
if
you're
sending
sending
requests
into
a
into
a
load
balancer
other
problems.
We
ran
into
pertaining
to
docker
most
of
the
issues
that
we've
run
into
our
are
issues
of
a
docker,
and
it's
something
that
we
see.
These
are
all
I.
Think
all
three
of
these
or
race
conditions.
Yeah.
A
We
see
race
conditions
all
the
time
when
you
start
using
containers
at
scale
when
you're,
when
you're
you're
churning
pods
very
quickly
on
every
single
node
in
your
cluster
you're
gonna
get
some
strange
errors,
so
this
first
one
this
this
container
doesn't
exist.
There's
a
race
condition.
We're
calling
docker
stop
at
the
same
time
as
your
container
process
dies
triggers
triggers
this.
A
A
Then
this
happens
when
whenever
we
do
a
deployment
because
kubernetes
being
something
that
will
restart
our
containers,
if
they
go
down,
that's
a
really
that's
a
really
powerful,
like
semantics,
for
designing
a
system,
because
if
something
goes
wrong
and
my
container
like
my
pod,
can
just
commit
suicide
and
kubernetes
will
recreate
it,
which
our
developers
did.
But
then
every
time
they
did
a
deployment,
they
would
hose
a
kubernetes
cluster
and
we
would
have
to
go
in
and
restart
docker
across
the
board.
A
The
second
issue
docker
attached
hanging
containers.
This
was
an
issue
triggered
by
our
use
of
hecka,
so
hecka
use
docker
attached
to
read
the
log
data
coming
off
a
standard
out
and
if
hecka
couldn't
keep
up
or
if
that
process
stopped,
the
standard
up,
pipe
would
fill
and
the
program
that's
running
in
your
container.
It
would
hard
luck.
A
A
Essentially
this
was
a
bad
interaction
between
core
OSS
network
D
and
docker
in
that
ticket
docker
says:
oh,
this
is
a
core
OS
issue,
not
a
bug
and
there's
no
defensive
programming
to
check
to
see
if
the
you
know
is
that
is
that
device
up
this
network
device
up
before
we
add
it.
Essentially,
what
would
happen
is
you
would
network
te
would
be
creating
this
V
device
and
then
docker
would
add
it
to
the
dock
or
zero
bridge.
A
But
if
that
V
device
wasn't
created
in
time
that
add
to
the
bridge
would
fail
and
he'd
be
done.
So
essentially,
your
containers
would
exist,
but
there'd
be
no
network
bridge
there'd,
be
no
networking
and
kubernetes
would
think
that
the
containers
were
healthy.
So
you
know
you
had
to
had
to
figure
it
out
yourself.
A
We
we
may
need
that
with
code
written
in-house
similar
to
the
logging
thing,
it
reads
a
list
of
contain
from
the
docker
API
and
then
checks
to
see
you
know
is
this:
is
this
containers
beef
device
in
the
bridge
so
three
fun
problems
there?
The
the
other
thing
and
I
think
this
is
the
last.
The
last
sort
of
issue
that
we
ran
into
was
actually
a
kernel
bug.
So
this
is
a
flame
graph
of
where
our
our
CPU
time
is
going.
A
We
had
a
bug
that
that
essentially
locked
40
cores
on
a
node
at
a
hundred
percent
in
case
soft.I
rqd,
so
it's
an
interrupt
request
as
consuming
one
other
percent
of
40
cores
on
a
system
whenever
more
than
20
connections
were
moving
traffic.
This
locked,
our
our
our
network,
throughput
to
one
gigabit
a
second.
It
turns
out
that
this
was
related
to
interrupts
being
processed
on
all
cores
instead
of
on
the
core.
A
That
was,
that
was
responsible
for
the
packet
and
we-
and
we
got
a
good
workaround
for
this,
essentially
reducing
the
number
of
received
hues
from
40
down
to
two.
You
know
whatever
I'm
ever
many
physical
cores,
there
are
on
a
single
socket,
but
that
was
that
was
a
pretty
serious
crisis
and
if
we
hadn't
resolved
that
we
we
probably
would
have
had
to
abandon
kubernetes
or
maybe
just
deploy
it
on
to
Santo
s6.
A
A
We
co-located
that
team
with
the
developers,
so
the
team
that
was
responsible
for
the
kubernetes
infrastructure
was
sitting
in
the
same
room
as
the
developers
who
are
writing
the
location
that
was
gonna
run
on
this
infrastructure
and
when
the
developers
ran
into
problems
they
could
talk
to
us
and
and
and
we
could
push
back
if
we
needed
to,
but
but
really
it
came
down
to
the
team
it
was.
It
was
the
right
combination
of
of
software
engineers,
systems,
engineers,
network
engineers,
people
who
understood
every
single
layer
of
abstraction
in
the
stack.
A
But
it's
hard
to
make
those
work
for
you
and
make
the
mortgage
scale.
We
also
took
a
very
conservative
approach
to
using
features
within
the
stack,
even
if
ingress
had
exist
when
we
started
as
a
beta
feature.
We
wouldn't
use
it,
and
that's
just
because
you
know
we
we
we
want
to
be
sure
that
the
features
that
we
use
every
additional
feature
that
we
use
has
a
cost.
We
have
to
understand
how
to
use
it.
A
We
have
to
support
it
and
if
there
are
problems
in
it,
we
have
to
understand
why
those
problems
are
occurring
and
attempt
to
fix
them.
So
the
fewer
the
fewer
features
that
we
use,
the
better
things
like
Sky,
DNS
and
cube
to
sky.
We
don't
use
them
and
the
reason
for
that
is
that
that
adds
an
additional
layer
of
abstraction.
A
That
makes
it
difficult
to
understand
what
our
applications
are
doing
and
I
think
that's
it
for
success
factors
we
attempted
to
DevOps
and
we're
still
doing
that
so
stumbling
blocks
problems
that
we
had
along
the
way
they're,
mostly
technological.
There
they're,
you
know,
issues
with
all
of
the
technology
and
we
kind
of
talked
about
them,
but
they're
they're
problems
that
we
can
solve.
A
If
we,
if
we
put
our
minds
to
it
and
really
understand,
what's
going
on
one
of
the
big
unsolved
problems,
I
think-
and
this
is
something
that
I
see,
is
as
more
teams
within
VIPRE
attempt
to
use
or
want
to
use.
The
stack
is
that
state
management
is
is
an
issue.
Where
do
we
store
data?
Where
do
we
store
the
volumes
that
your
database
is
running
on?
Do
we
need
to
run
something
that's
attached,
or
can
we
run
a
distributed
file
system?
I?
A
Don't
think
there
are
great
solutions
to
that
right
now,
and
especially
when
you're
running
on
on-premise,
you
know
you
can't
just
use
s3,
you
need
to
deploy
an
object,
storage
system
alongside
your
kubernetes
and
and
that's
pretty
hard,
so
things
that
we
need
to
do
better.
Devops
is
hard,
and
it's
it's
really
hard
in
a
in
an
organization
like
as
big
as
Comcast,
where
you
have
a
lot
of
these
kind
of
entrenched
teams
and
business
processes
like
Conway's
laws.
A
Is
this
idea
right
that
our
or
application
architectures
mirror
the
structure
of
organization
and
that's
something
that
actually
happens?
That's
something
that
I've
witnessed
and
and
the
DevOps
model
and
declarative
infrastructure
and
having
developers
do
deployments
like
that
kind
of
flies
in
the
face
of
that.
So
it's
it's
really
hard
to
get
people
to
think
in
those
mindsets
and
to
and
to
take
on
these
problems
and
this
responsibility,
especially
when
it's
things
that
that
other
teams
have
historically
been
responsible
for
responsibility
for
infrastructure.
A
So,
where
we're
going,
you
know
six
products
within
Viper,
in
addition
to
the
one
that
I
showed
you,
our
our
I've
either
been
developed
for
kubernetes
or
migrated
to
it.
The
immutable
deployment
model
is
something
that
we're
rolling
out,
we're
looking
to
the
future
towards
dealing
with
multi-tenancy
kind
of
building
a
crappy
version
of
AWS
within
Comcast,
doing
stuff,
with
ipv6
and
hybrid
six
to
four
clusters
and
then
scaling
out
beyond
200
nodes.
That's
that's
really
where
we're
headed
with
that
and
and
I'm
very
excited
for
for
our
future.
That's
it.
A
Right
so
the
question
was:
how
did
he
deal
with
state
in
the
cloud
since
you
were
running
on
bare
metal
so
for
this
pilot
application
being
in
cloud
DVR
and
having
these
these
excessive
throughput
requirements?
You
know
we
managed
State
externally
we're
running
in
in-memory,
distributed
database
specific
to
this
application
and
we're
running
an
object.
Storage
that
gets
deployed
alongside
at
most
of
the
1,000
servers
that
we've
deployed
are
actually
pertaining
to
that.
A
So
the
question
was:
what's
driving
ipv6,
as
we
show
a
lot
of
things,
there's
a
big
push
within
the
company
to
to
move
towards
ipv6
and
I.
Think
that
long
term
communication
with
external
clients,
set-top
boxes
and
things
like
that
within
the
Comcast
network
are
intended
to
be
ipv6
only
there
also
some
pretty
cool
ypb
six
technologies
like
segment
routing
and
any
cast
IPs
I,
don't
think
I'm
getting
the
name
right
that
that
our
architecture
astronauts
are
and
have
in
mind.
A
30
gigabytes
a
second,
so
the
question
was
when
we
say
we
can
write
okay.
So
the
question
was:
how
much
traffic
can
we
handle
per
container
per
server?
So
so
we
use
Network
teaming
on
the
individual
nodes,
so
per
node,
it's
20,
cubits,
a
second
and
and
the
bandwidth
across
the
cluster
is
the
width
of
all
of
the
nodes
that
we
have.
A
A
The
question
was:
what
was
the
churn
rate
of
this
environment?
You
know
I,
don't
know,
I
think
it's
application
dependent.
We
do
deployments
once
a
week
which
turns
every
single
pot
in
the
node
or
every
single
pot
on
the
cluster,
but
I
believe
that
we
also
turn
on
the
order
of
50
to
100
pods
per
hour.
Just
generally.
A
That's
that's
kind
of
tough
to
answer,
but
I,
but
I
really
think
it
came
down
to
the
to
the
fact
that
your
kubernetes,
the
artifacts,
the
things
that
you
push
into
the
API
or
declarative
and
stateful.
So
it
made
it
really
easy
to
take
a
configuration
that
we
tested
in
the
lab
and
deploy
that
into
a
production
environment.
Things
like
the
the
service
load
balancers
and
the
extensibility
of
the
API,
the
accessibility
of
data
from
the
API,
where
we're
also
pretty
crucial.
A
So
I
believe
the
question
was:
it
sounds
like
a
successful
project.
Have
you
been
successful
at
influencing
the
the
culture
in
Comcast
outside
of
Viper
I?
Think
that
you
know
that's
that
remains
to
be
determined,
but
I
think
that
that's
the
direction
that
we're
heading
I've
seen
a
lot
of
interest
from
other
teams
throughout
Comcast
and
in
what
we've
been
doing
and
people
looking
to
mirror
the
success.
A
All
right,
I
think:
that's
it,
oh
one
more
all
right.
So
the
question
was
how
important
is
network
security
between
micro-services
right
now,
not
important,
but
I
think
that
as
we,
we
look
at
using
kubernetes
and
a
more
multi
tenant
model
having
many
applications
from
any
developers
running
in
in
these
clusters
that
it
becomes
much
more
important
all
right.
Thank
you.
Oh
one,
more.