►
Description
Keynote: Kubernetes: As Seen On TV - Erik St. Martin, Systems Architect, Comcast
"Kubernetes is not just a product to be used solely as it comes out of the box, it’s also a platform, a series of building blocks that can be tweaked, tuned, and modified to fit more specific use cases.
At Comcast we are building a geographically distributed system to stream linear video to millions of cable customers around the country, from nearly 1,000 physical locations, within 27 regional networks. We have unique requirements and constraints which have benefited incredibly from the versatility of Kubernetes. In this talk Erik will walk you through how Comcast leveraged existing Kubernetes components as well as developed custom components to build this system and how you can too."
About Erik St. Martin
Erik St. Martin is a Systems Architect at Comcast where he works on distributed systems and fail-over strategies for linear video streaming. He's a long time Go developer, Kubernetes user, and the initial author of skydns. In his off time he is an author, podcaster, and organizer of GopherCon.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
All
right,
so
we've
only
got
20
minutes
and
I
kind
of
have
to
explain
a
little
bit
about
cable
for
everybody
to
kind
of
understand
what
we're
doing
so
with
that
I'm,
assuming
most
of
you,
are
all
pretty
familiar
with
what
these
things
are.
But
have
you
ever
wondered
how
they
work?
I
mean
how
does
cable
actually
come
across
this
and
get
to
your
to
your
house?
A
So
we've
been
talking
a
lot
about
kubernetes
the
past
few
days,
but
we're
gonna
quickly
detour
for
just
a
couple
of
minutes
to
talk
about
how
cable
works
and
circle
back
for
kubernetes,
so
television
actually
started
out
is
just
a
couple
of
channels
broadcast
using
radio
waves
and
picked
up
by
antennas
attached
to
your
television
set
and
probably
depending
on
the
age.
How
old
you
are
in
kind
of
the
city.
A
But
if
you
lived
in
a
hilly
area
without
line-of-sight
to
the
transmission
tower,
you
really
didn't
have
the
ability
to
get
cable
until
a
man
in
the
fort
I
think
was
back
in
the
40s,
a
gentleman
that
was
selling
TVs.
So
our
TVs
had
a
problem
selling
TVs
because
he
couldn't
sell
them
the
customers,
just
they
got
no
cape
or
they
got
no
TV
reception
due
to
hills
from
the
mountains,
so
basically
went
up
into
the
mountains.
Set
up
a
tower,
ran,
coax,
cable
down
to
his
store
and
then
eventually
out
to
his
customers.
A
And
then
we
got
cable
TV.
It
was
known
as
community
antenna,
TV
back
then,
and
by
the
80s.
It
was
available
throughout
mostly
United
States.
If
we
fast-forward
to
the
late
90s
digital
cable
was
introduced,
oops,
so
digital
cable
was
introduced
and
it
offered
a
number
advantages,
first
of
which
was
that
we
could
multiplex
in
many
channels
under
the
same
analog
frequency
allowing
for
everybody
to
get
more
stations.
It
also
offered
things
like
error,
correction,
compression
and
encryption,
and
it
brought
a
welcome
to
end
to
dial-up
internet.
A
A
For
short,
so
within
those
networks
are
they're
actually
composed
of
a
number
of
locations,
two
primary
locations
where
all
of
our
internet
phone
and
cable
backbones
come
into
and
those
are,
the
locations
are
collectively
referred
to
as
the
headend,
and
they
provide
kind
of
redundancy
at
the
backbone
layer,
level
in
case
of
fiber
gets
cut
or
power
outages
on
one
side
of
the
other.
In
addition,
within
those
locations
we
have
a
piece
of
hardware.
A
This
kind
of
industry,
specific
known
as
a
cap
or
the
specific
model,
is
a
cap
1000,
and
this
piece
of
hardware
is
responsible
for
taking
multiple
video
streams,
injecting
localized
ads,
so
that
on
your
side
of
town,
you
get.
You
know
the
car
salesmanship
that's
around
the
corner
and
not
the
one.
That's
on
the
other
side
of
town.
It
then
multiplexes
those
streams
together
and
creates
one
stream
encrypts
them
and
then
sends
them
out
to
the
smaller
locations
that
we
refer
to
as
hub
sites.
So
we
typically
have
a
redundant
set
of
these
cap.
A
One
thousands
on
each
side
of
the
head
end
both
multi
casting
towards
the
hub
site.
So
in
case
one
gets
cut
off.
There
can
be
any
number
of
these
hub
sites.
I
only
have
a
couple
of
them
from
the
Seattle
area
represented
here.
So
once
the
stream
reaches
the
hub
site
and
makes
its
way
to
another
piece
of
hardware
known
as
the
apex.
So
what
the
apex
does
is
it
takes
this
multicast
traffic.
A
After
that,
an
optical
converter
inside
the
hub
site
will
actually
convert
it
to
light
and
send
it
over
fibre
cables
and
get
it
down
closer
to
your
neighborhood
and
within
the
neighborhood,
there's
typically
another
optical
node.
That
will
convert
it
back
to
RF
and
send
it
across
the
coax
cables
down
your
street.
A
So
what
comes
across
the
cable
in
your
house
is
actually
just
a
series
of
streams
broken
up
into
six
megahertz
with
up
to
ten
streams
within
each
band,
depending
on
whether
high
def
or
standard
definition
channels
and
they're,
always
a
constant
thirty.
Eight
point:
eight
megabits
per
second
for
one
six
megahertz
band.
So
when
you
change
the
channel
in
your
set-top
box,
what
you're
really
doing
is
actually
turning
to
the
radio
frequency
associated
with
the
channel
and
then
demultiplexing
out
the
specific
program
within
it.
A
A
Well,
we
find
ourselves
in
the
middle
of
a
new
transition
and
that's
from
qualm
based
IP,
our
qualm
based
delivery
of
video
to
IP
based
delivery
of
video
with
the
always-on
nature
of
these
streams.
That
cable
is
constantly
pushing
around
three
gigabits
per
second
of
data,
depending
on
the
neighborhood
you
live
in,
for
I
mean
just
think
about
that
three
gigabits
per
second
for
all
those
channels.
A
So
our
newer
set-top
boxes
are
actually
IP
base
and
they
retrieve
the
linear
content
on
demand
which
allows
for
only
channels
being
watched
to
consume
bandwidth.
They
use
a
technology
called
MPEG
DASH,
which
is
a
dynamic
adaptive
bitrate
streaming
over
HTTP,
it's
a
whole
bunch
of
words,
but
we
can
sum
it
up
with.
Basically,
the
set-top
box
reaches
out
over
HTTP
retrieves,
an
XML
manifest
file
that
describes
the
stream
the
different
representations
of
it
and
formats
there'll
be
different
versions
of
audio
and
it'll.
A
A
We
can
offer
more
content
in
more
formats
like
4k
and
we
can
offer
it
to
more
devices
like
tablets
and
and
and
it
provides
us
a
ubiquitous
IP
network,
as
opposed
to
like
our
proprietary
network,
that
only
cable
boxes
are
connected
to
so,
however,
there's
still
millions
of
these
qualms
based
set-top
boxes
out
there
and
that
they're
not
going
anywhere
anytime
soon.
So
that
leaves
us
with
a
problem
while
supporting
both
of
these
device
types,
we
actually
have
duplicate
traffic
for
all
of
our
content.
A
We
have
the
multicast
UDP
traffic
for
our
qualm
based
devices,
and
then
we
have
the
HTTP
traffic
for
our
IP
based
devices.
So
the
goal
of
our
project
is
actually
to
replace
this
cap
1,000
up
in
the
head
end
that
does
the
multiplexing
with
software
that
retrieves
the
content
over
the
IP
infrastructure,
but
delivers
it
to
the
quam
infrastructure.
Just
as
this,
this
industry,
specific
hardware
did.
A
So
with
an
organization
this
size,
many
of
the
different
regions
and
divisions
have
their
own
failover
strategies
and
preferences
and
constraints
as
to
where
the
hardware
and
streams
can
exist
due
to
kind
of
rack
availability,
their
own
networking
topology.
So
we
need
a
solution
with
as
little
complexity
so
that
we
can
hand
it
out
to
these
teams,
but
as
much
consistency
as
possible,
so
that
we
can
kind
of
maintain
kind
of
like
a
canonical
approach,
but
we
also
need
the
versatility
for
them
to
be
able
to
make
some
of
the
decisions
on
their
own.
A
So
with
a
bit
of
history
and
high
level
understanding
of
the
cable
infrastructure.
Let's
take
a
quick
look
at
our
requirements.
First,
we
must
be
highly
available.
There
can
be
no
single
points
of
failure
and
of
which
we
also
need
to
make
sure
that
we
have
a
hot
hot
configuration
of
these
streamers
so
that
in
case
a
fiber
gets
cut.
A
They
need
to
be
able
to
pick
these
things
because,
like
at
some
hub
site
locations,
you
may
have
capacity
for
failover,
but
in
others
you
may
only
be
able
to
tolerate
a
minimal
number
of
nodes
failing
before
you
need
to
fail
up
and
just
say,
like
a
bigger
regional
data
center
at
the
cost
of
a
little
more
back,
a
little
more
bandwidth
across
the
backbone.
We
need
static,
IP
addresses
because
those
apex
devices
use
something
called
source-specific
multicast,
which
they
need
to
be
pre-programmed.
A
With
the
IP
address
of
this,
the
streamer
that's
going
to
be
sending
them
their
their
content.
We
also
need
to
be
able
to
update
the
stream
configurations
on-the-fly.
If
we
don't
want
channel
outages
just
to
make
slight
changes
to
the
way
the
stream
is
working,
change,
kind
of
twisting
knobs
for
a
network
jitter
and
things
like
that,
we
have
real-time
CPU
constraints
and
we
can
have
absolutely
no
over-commitment
of
resources
on
a
node,
because
that
would
result
in
video
degradation
for
all
the
channels
that
are
being
streamed
from
that
node.
A
We
want
deterministic
updates
of
stream
or
software.
We
we
want
to
make
sure
that
we're
not
updating
channels
during
key
prime
moments.
For
that
specific
channel.
You
know,
as
with
anything
we
throw
out
into
production,
especially
in
the
enterprise
world.
We
need
tools
for
debugging
and
monitoring
and
ops.
All
of
that
plays
a
role
in
it.
It
can't
just
serve
the
purpose.
Well,
we
need
to
be
able
to
continue
to
work
with
it.
A
So
when
I
start
thinking
about
the
system
that
we
might
need
to
build
to
solve
these
problems,
there's
only
one
thing
that
comes
to
mind:
I
mean:
can
you
imagine
trying
to
design
a
system
like
this
from
scratch
like
using
zookeeper
and
your
own?
A
homogeneous
pool
of
workers
and
writing
your
own
scheduler
it'd,
be
a
massive
effort
and
it'd
have
a
ton
of
edge
cases,
but
it
probably
comes
as
no
surprise
standing
here
talking
to
you
and
at
a
Cooper
Nettie's
conference,
the
kubernetes
has
actually
solved
most
of
these
problems
for
us.
A
So
let's
take
a
quick
look
at
how
we
implemented
our
solution.
So,
first
using
kubernetes
third-party
resources,
we
can
divine
us
our
own
stream
resource
that
represents
these
multiplex
channels,
and
so
we
have
a
spec
file
to
manage
that,
and
it
has
everything
that
we
need
to
do
that.
The
individual
knobs
to
turn
for
our
multiplexing
software,
the
dash
URLs
to
fetch
content
from
the
static
IP
address
that
our
pod
needs
to
contain.
A
So
this
is
actually
submitted
by
a
web
user
interface
that
we
provide
to
our
video
engineering
team
into
the
kubernetes
api.
Once
it's
submitted
into
the
kubernetes
api,
we
have
our
own
stream
controller,
that
watches
the
kubernetes
api
for
changes
to
these
resources
and
reconciles
the
cluster
based
on
the
changes.
It's
just
that
it
observes.
A
First,
the
stream
controller
creates
a
config
map
and
the
config
map
basically
contains
all
of
the
information
you
saw
in
that
little
MUX
part
of
the
the
stream
spec,
which
is
the
the
configuration
specific
to
that
multiplex
software.
It
also
creates
a
replica
set
for
our
stream
controller
mapping
in
the
config
map.
A
As
a
volume,
the
stream
controller
serves
an
additional
role,
which
is
that
and
continues
to
update
status
information
on
that
on
the
stream
resource,
similar
to
the
way
that,
if
you
were
to
look
at
our
replication
controller
or
a
pod
after
it's
already
been
deployed,
you
would
see
an
additional
value
called
status.
That
has
a
lot
more
information
about
the
the
container
and
things
like
that,
and
this
allows
us
to
not
have
to
be
worried
about
the
kubernetes
primitives
that
make
up
this
stream
that's
out
in
the
wild,
but
we
can
look
through.
A
We
can
we
get
all
our
systems
can
looked
only
at
the
stream
API
kind
of
what
version
of
software
the
streaming
software
it's
running.
What's
the
current
bit
rate?
How
many
bytes
is
this
thing
transferred
since,
since
it's
been
deployed
so
then,
within
each
pod
we
have
a
sidecar
process
that
we
refer
to
as
the
streamer
agent.
So
the
streamer
agent
constantly
monitors
the
config
map
volume
looking
for
changes
and
when
it
observes
a
change,
it
signals
the
streamer
to
reload
its
configuration
from
disk.
A
A
So
if
you
recall,
as
a
requirement,
we
talked
about
divisions
and
regions
needing
to
make
their
own
decisions
about
what
hardware
a
particular
stream
is
served
from
due
to
their
own
processes
and
constraints,
and
this
is
actually
relatively
simple
to
solve.
After
the
one
for
release
using
nothing
more
than
labels
and
annotations,
we
can
define
our
cran
as
a
as
our
cluster
boundaries,
with
different
locations
having
a
unique
label
and
I
know.
This
looks
like
this
isn't
typical
practice
and
I
would
agree
with
you.
A
Cluster
boundaries
are
usually
drawn
within
a
physical
location,
but
we
have
a
thousand
locations,
some
of
which,
with
only
a
handful
of
nodes-
and
we
also
have
the
benefit
of
having
our
own
backbones
with
redundant
fibre
between
every
connection
and
all
the
locations
within
one
cluster
are
fairly
close
geographically
like
in
this
cluster.
This
is
really
only
the
Seattle
market.
It
only
spans,
probably
30
or
50
miles
in
any
direction.
A
So
first
thing
we
can
do
is
we
can
have
the
administrator
label
nodes?
They
prefer
a
stream
destined
for
a
given
ad
zone
width
with
an
ad
zone.
One
label
set
to
primary,
and
then
we
can
pick
a
location
that
may
have
more
failover
Hardware
because
of
Rackspace
availability
with
an
ad
zone
or
an
as
in
one
label
set
to
secondary.
A
We
then
only
need
to
apply
an
annotation
to
the
pod
that
our
stream
control
are
created,
and
this
annotation
designates
first
that
a
node
is
required
to
have
an
ad
zone,
one
label
with
a
value
set
to
primary
or
secondary,
to
be
eligible
for
our
stream
to
be
scheduled
on,
but
is
preferred
to
run
on
a
node
with
an
ad
zone.
One
label
set
to
primary
what
this
does
is.
A
An
example
of
this
is
is
a
small
hub
site
may
have
a
single
blade
server,
so
we
could
tolerate
a
couple
nodes
failing,
but
what
happens
if
the
whole
chassis
goes
out?
There's
not
enough
rack
space
to
have
an
entire
backup
chassis
there,
but
we
can
do
better,
though
we
don't
want
our
redundant
hot-hot
pods
to
end
up
on
the
same
nodes
or
even
in
the
same
location.
This
would
create
single
points
of
failure.
A
So,
with
the
release
of
one
four,
we
now
have
functionality
notice,
inter
pod
affinity
and
anti
affinity.
This
allows
us
to
further
tweak
our
annotations
so
that
we
can
require
the
pod
to
be
scheduled
onto
a
node
that
does
not
already
have
the
same
stream
running
on
it
and
prefer
that
it
doesn't
get
scheduled
into
a
zone.
That's
that
already
has
that
stream
running
in
there,
and
all
of
this
is
free
just
from
applying
the
right
labels
and
to
our
nodes
and
then
adding
annotations
to
our
pods.
A
This
feature
actually
allowed
us
to
remove
our
own
scheduler
completely.
It
wasn't
it
wasn't
more
than
a
couple
months
ago
or
we
had.
We
got
to
delete
our
whole
idea
of
our
own
scheduler,
but
the
power
of
this
comes
in
administration,
because
now
the
team
responsible
for
managing
these
video
streams
get
to
focus
on
the
part
that
matters
to
them.
What
channels
are
configured
together?
What
are
the
format's
that
are
going
into
it?
A
What
frequencies
are
these
things
getting
multicast
on
and
they
don't
have
to
care
about
the
underlying
hardware
and
the
team
responsible
for
the
hardware
can
change
the
hub
sites
that
are
responsible
for
an
ad
zone
at
will
if
a
new
facility
gets
built
or
has
more
capacity,
it's
just
a
matter
of
shifting
labels
around,
and
this
is
a
huge
shift
from
the
way
things
currently
work.
Today.
The
video
engineering
team
needs
to
know
about
every
single
one
of
those
cap
and
apex
devices.
They
maintain
spreadsheets
of
these
things
and
log
into
them
by
IP
address.
A
So
when
a
channel
needs
to
move
they
log
into
one
cap
device,
remove
it
from
that
log
into
another
cap
device,
login
update
that
and
then
they
log
into
the
apex
device
to
change
the
source
address
of
the
cap
device
where
it's
coming
from
and
sometimes
even
comes
down
to
physically
moving
Hardware
or
cables
and
with
the
new
kubernetes
implementation.
They
no
longer
need
to
be
concerned
with
any
of
these
details,
it's
just
a
matter
of
omitting
a
request
for
a
stream
telling
the
cluster.
A
What
ad
zone
this
needs
to
be
broadcast
to
and
then
letting
the
cluster
do
what
it
do
its
thing.
So
we
actually
get
a
lot
of
free,
tooling,
too,
and
I'm
a
Kubb
CTL
guy.
You
won't
hear
me,
say:
coop,
cuddle,
I'm,
sorry,
so
we
can
use
coop
CTL
just
to
get
a
list
of
streams.
We
can
filter
it
down,
get
trying
to
figure
out
what
hosts
are
running
particular
channel
lineups
I
mean
we
can
do
a
search
to
see
what's
running
an
outdated
version
of
our
multiplexer
software.
A
We
can
drain
hosts
so
that
we
can
take
them
down
for
maintenance.
We're
able
to
take
a
highly
complex
system
with
many
constraints
and
turn
it
into
a
solution.
That's
relatively
straight
simple
and
straightforward,
and
there's
tremendous
beauty
in
that.
But
it's
not
without
its
stumbling
blocks
either.
Our
multiplexing
process
actually
requires
real-time
threading.
In
order
to
keep
time,
so
we
need
to
actually
account
for
CPU
real
time
run
time,
and
one
of
the
issues
that
we
had
is
that
in
the
resource
quotas
for
both
kubernetes
and
docker,
this
isn't
accounted
for.
A
A
The
other
issue
we
have
is
source-specific
multicast
and
networking.
So,
as
I
said,
the
apex
requires
that
we
already
know
what
the
IP
address
is,
which
means
that
we
can't
use
the
typical
overlay
Network
approach
of
basically
assigning
a
block
of
IPs
to
a
host.
And
then,
when
a
pod
comes
up,
it
gets
an
available
IP.
An
IP
needs
to
be
pre
assigned
for
a
channel
lineup,
an
ad
zone
combined
and
no
matter
where
it
pops
up
in
the
cluster
and
needs
to
always
have
that.
Ip
address.
A
A
Bandwidth
based
scheduling
right
now,
we're
currently
just
inflating
our
CPU
and
memory
constraints
so
that
we
we
can
ensure
that
we
don't
saturate
a
link
on
a
blade,
but
it'd
be
nice.
If
we
could
actually
have
networking
knobs
there
to
where
we
can,
where
we
can
supply
how
much
networking
a
pod
consumes,
because
in
our
case
it's
always
going
to
be
thirty.
A
To
do,
there's
no
doubt
that
kubernetes
has
changed
the
way
we
deploy
and
manage
application
and
all
the
incredible
talks
that
you've
heard
over
the
past
two
days
are
a
testament
to
that.
The
kubernetes
can
be
just
as
impactful
as
a
framework
for
building
your
own
applications.
You
can
save
yourself
complexity
and
development
time
by
leveraging
functionality
and
tools
that
already
exist.
We
can
also
create
clean
abstractions
between
teams.
By
writing.