►
Description
Keynote: Your Philips Hue Light Bulbs Are Turned On By Kubernetes - Mark van Straten, Senior Software Architect, Q42
Philips Hue is one of the leading global Internet of Things platforms and has integrations with Amazon Echo, Google Home, IFTTT & Nest to make your lights work for you. The platform has grown considerably during the past years and to accommodate this has been migrated towards Google Container Engine at the end of 2015.
We needed to find out how to do zero downtime deployments, grasp how scaling works in Kubernetes, provide ourselves with monitoring and logging and above all find out how to do all these things and more while staying agile in the process. Come join me if you want to start using Kubernetes in production but are not sure yet what you are going to be facing.
About Mark Van Straten
Love to talk about Kubernetes, Rx or Coffee. Technical lead of the Philips Hue Cloud
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Good
afternoon,
everyone
I
hope
you
had
fun
today
and
you
have
learned
a
bunch.
My
name
is
Mark
van
Statten
and
I
work
at
Q
42
in
the
Netherlands
I
am
the
driving
force
behind
Philips
you
being
ran
on
kubernetes
I,
attended
this
same
conference
in
2016
in
London
to
make
a
technical
assessment
if
we
could
start
use
it
in
production.
At
that
moment
in
time
the
1.2
release
of
kubernetes
was
almost
around
the
corner,
and
I
was
looking
for
others
like
me,
who
are
trying
to
use
it
in
production.
A
What
I
found
out
was
that
that
meant
not
many
people
were
using
it
in
production
or
if
they
did
not,
that
much
load
was
putting
on
was
being
put
on
it,
but
what
I
did
feel
was
lots
of
compassion
progression
and
the
tremendous
effort
of
the
community
to
bring
kubernetes
where
it
is
today.
So
when
I
came
back
to
my
office,
I
started
our
own
research
and
our
own
migration
of
the
Philips.
U
cloud!
What's
kubernetes,
this
talk
is
the
culmination
of
our
last
year.
A
A
You
have
it
at
your
home
or
at
your
company
and
Philips
has
originally
developed
this
product,
and
we
have
been
all
using
it
for
many
years,
but
Philips
envisions
that
life
can
be
so
much
more
that
you
could
interact
with
light.
The
data
could
provide
you
with
what
you
need
when
you
need
it
being
it
that
you
wake
up,
that
you
come
home,
that
you
have
a
party,
so
they
started
transforming
the
product
and
they
released
Philips
you
in
2012
it
was
the
world's
first
smart
lightbulb
cloud
connected
and
it
could
change
intensity
and
color.
A
It
is
still
the
most
popular
smart
home
product
today.
The
introduction
of
Phillips
you
has
did
has
done
something
very
special.
It
has
made
something
we
all
take
for
granted
relevant
again
using
sensors
routines
integrations
or
Phillip
you
compatible
products.
You
will
never
have
to
touch
a
wall
switch
again
when
we
started
the
original
development
in
2012
on
the
Philip
you
cloud,
we
knew
the
value
it
would
bring
to
you
all
as
the
customers,
but
the
question
which
we
could
not
answer
was:
how
fast
would
you
all
catch
on?
A
How
fast
fast
would
we
need
to
scale?
So
we
were
searching
for
a
solution
which
would
provide
us
with
a
minimal
cost
per
connected
home
and
could
skill
rapidly
and
we
ended
up
using
Google
App
Engine.
We
built
a
monolithic,
app
using
Java
and
the
Play
Framework
version
1,
so
you
could
interact
with
your
light
bulbs
at
home
while
you
were
not
while
you
were
not
at
home.
That
was
the
primary
use
case
at
that
moment.
A
So
latency
was
not
a
big
concern
because
you
could
not
observe
the
responses
in
2014
we
started
to
observe
growing
pains
because
more
and
more
integrations
were
coming
along
with
the
Philips.
U
cloud
and
also
we
needed
to
iterate
on
our
infrastructure,
because
the
the
Play
Framework
version
1
was
deprecated,
so
we
ended
up
on
compute
engine,
which
is
now
called
Google
App
Engine
flex.
It
still
has
the
great
scalability
metrics
of
Google
App
Engine,
but
it
allows
you
a
custom
runtime
and
we
ended
up
with
no
GS.
A
We
broke
down
a
monolith
into
two
tiers:
the
application
facing
cloud
API
and
the
home
facing
bridge
API,
and
we
glued
them.
Together
with
the
Redis
database,
then
kubernetes
came
along
when
Google
announced
Google
container
engine.
We
were
very
interested
because
this
great
combination
would
mean
that
we
could
you
to
read
the
the
effort
you
use
the
you,
the
benefits
of
kubernetes,
of
building,
true
micro
services
and
utilize,
the
underlying
resources,
much
better,
while
still
keeping
our
operational
efforts
to
a
minimum,
and
this
brings
us
to
the
present
day.
A
We
started
the
migration
of
our
platform
at
the
beginning
of
the
summer
2016,
and
we
also
started
breaking
apart
our
application
even
further
into
smaller
micro
services,
which
is
an
effort
which
continues
until
today.
All
applications
which
want
to
interact
with
your
home
need
to
go
through
the
new
cloud
API.
A
This
means,
if
the
standard
Amazon
Alexa
Google
home,
but
also
the
official
new
mobile
app
utilizes
this
cloud,
and
when
you
put
a
request
to
the
cloud,
for
instance,
to
turn
on
your
lights
at
home,
we
will
at
first
validate
if
the
token
that
you
have
supplied
us,
is
valid
and
is
authorized
to
connect
with
your
home.
If
it
does
it's
all
well,
we
will
push
your
command
into
our
Redis
database,
which
acts
like
a
pub
sub
broker
and
has
very
basic
information
about
your
connection
and
last
we
will
notify
our.
A
We
will
utilize
our
back-end
notifier,
to
notify
the
correct
back-end
that
we
will
need
some
information
of
your
bridge
and
add
a
command
needs
to
be
invoked.
The
correct
back-end
will
then
be
invoked
and
we
will
communicate
with
your
bridge
at
home
so
that
your
lights
turn
on
and
the
response
I
have
done.
A
This
is
passed
back
through
Redis
all
the
way
back
to
the
application
that
originally
made
this
request,
and
why
do
you
have
two
backends
you
might
ask
when
Philips
in
originally
in
2012,
released
Philips
you,
the
bridge
that
came
along
with
it,
was
not
that
powerful.
It
could
only
communicate
using
HTTP
requests
to
the
outside
world.
We
can
wake
it
up.
It
will
start
pulling
for
commands
to
our
cloud
and
after
some
time
was
code
to
standby
mode.
A
Again,
the
new
bridge,
as
I've
shown
on
this
slide,
is
way
more
powerful
and
is
able
to
make
a
persistent
WebSocket
connection.
This
is
much
more
faster
and
decreases
the
latency
of
the
overall
system.
Significantly,
the
five
services
combined
with
this
rebus
database
make
up
the
core
of
the
field
view
cloud.
A
We
are
running
one
cluster
in
production
only
in
Europe.
This
is
mainly
because
of
the
data
retention
and
privacy
concerns
that
we
have
for
our
European
customers
along
sides
with
the
five
services
that
I
just
mentioned.
We
have
a
few
others
making
up
grow
a
grand
total
of
ten,
and
when
I
made
this
slides,
we
were
using
40
nodes
and
having
an
120
pots,
and
now
we
are
almost
exceeding
200.
A
A
We
have
faced
many
challenges
while
starting
to
use
cuban
eddies
at
first
we
needed
to
build
trust
upon
the
product.
Many
times
have.
I
heard
my
colleagues
exclaim
what
is
kubernetes
doing
for
funky
stuff
to
my
code
to
find
out.
We
were
at
fault
gradually.
These
voices
diminished
I
want
to
share
with
you
some
of
the
lessons
that
we
have
learned
using
our
specific
type
of
workloads
on
Google
container
engine.
As
I
have
mentioned.
All
communication
with
our
cloud
API
is
currently
done
using
HTTP
requests
from
the
applications.
A
The
actual
response
to
these
requests
is
the
actual
response
from
your
home
at
any
given
time.
There
are
multiple
requests
waiting
for
these
responses
and
it
can
take
some
time
depending
on
how
fast
the
bridge
in
your
home
wakes
up
and
your
total
internet
connection.
Latency
is,
while
you
are
running
on
Google
App
Engine.
This
wasn't
was
no
problem.
A
We
had
separate
virtual
machines,
but
on
Cuban
Eddy's
your
nodes
will
share
basic
things
like
the
CPU,
the
memory,
the
network
bandwidth
and
also
the
underlying
socket
connections
during
a
particularly
dancey
intensive
evening,
one
of
our
pots
saturated
always
inbound
connections.
It
ended
outbound
connections
and
it
became
it
spilled
over
to
the
node.
A
So
it
caused
all
sorts
of
trouble
on
the
complete
note
that
post
mortem
was
no
fun
to
diagnose
to
prevent
these
kinds
of
outages
in
the
future,
we
have
now
implemented
circuit
breakers
on
all
our
external
calls,
using
hysterics
originally
developed
by
Netflix.
For
those
of
you
who
do
not
know
what
a
circuit
breaker
does,
whenever
the
error
rate
of
our
external
services
exceeds,
our
tress
hold
the
circuit
breaker
trips
and
we
stop
making
calls
to
our
external
service
once
a
probe
request
succeeds.
A
We
will
pick
up
our
pace
again
and
start
making
these
calls
again.
This
way,
we
prevent
a
spillover
from
failures
cascading
into
our
pot
and
influencing
the
whole
node.
Eventually,
we
are
also
we're
searching
for
other
ways
to
maximize
the
amount
of
sockets
that
we
can
utilize
on
our
nodes
and
we
found
out.
We
need
that
we
needed
to
modify
real-time
settings,
for
instance
like
the
nf
contract
or
the
TCP
socket
timeout
duration.
But
when
you
are
using
Google
container
engine
Google
manages
your
notes.
A
They
are
Ketel
and
this
is
a
great
thing,
because
less
operations
means
that
I
have
got
more
time
for
development.
So
how
do
you
manage
change
your
runtime
settings
of
your
nodes
and
we
ended
up
with
a
privileged
spot
running
in
a
daemon
set
on
every
node
and
because
it
is
privileged
as
powers,
it
may
change
settings
this
way
we
keep
Google
in
charge
of
our
nodes
and
we
can
overlay
our
own
settings
on
top.
A
A
It's
doing
nothing
mostly
during
the
day
that
makes
it
very
hard
to
scale
using
the
default
horizontal
port
autoscaler,
because
we
are
not
consuming
any
CPU
and
memory
because
we're
idling.
We
also
wanted
to
do
proactive
scaling
instead
of
retroactive
scaling.
We
want
to
be
ready
and
have
a
new
pot
ready
to
serve
new
traffic
before
we
saturate
all
the
connections
that
we
can
have.
So
we
built
a
custom,
horizontal
port,
autoscaler
and
also
again,
building
a
post
cost
building
a
custom.
Horizontal
autoscaler
is
not
that
hard.
A
A
It
then
predicts
the
amount
of
replicas
which
are
desired
and
if
it
is
more
than
we're
currently
having
scale
up
if
it
is
less
than
what
we
are
currently
having,
we
don't
scale
down
last
month
and
big
Internet
service
profile
in
the
Netherlands
ziggo
had
an
outage
and
we
lost
a
significant
amount
of
the
Dutch
traffic.
But
these
bridges
they
are
not
lost,
they
will
come
back
and
we
do
not
want
to
build
a
denial
of
service
attack
on
ourselves.
A
Even
though
that
we
have
a
running
gag
in
our
team
that
you're
not
on
the
team,
unless
you
have
brought
a
denial
of
service
attack
machine
as
our
user
base
grew
and
our
architecture
evolved,
we
as
a
team
also
needed
to
evolve.
We
had
to
ingrain
DevOps
into
our
culture,
you
develop
it,
you
ship
it
you
monitor
it.
We
also
had
to
adopt
the
measure
everything
philosophy
originally
preached
by
Etsy.
When
we
were
running
on
Google
App
Engine,
we
had
problems
answering
basic
metric
questions
like
what
95th
percentile
of
our
response
latency.
A
While
utilizing
stats
D,
we
could
omit
these
kinds
of
metrics
or
from
our
applications
to
gain
vastly
more
insights.
This
was
also
crucial
for
us
in
choosing
data
doc
as
our
monitoring
solution
on
kubernetes,
because
data
log
has
native
support
for
stats,
D
metrics,
the
cube,
CTL
command
line,
combined
with
cube
tail
for
logging
and
sprinkled
with
a
bit
of
shell.
A
Scripting
is
at
this
moment
enough
for
us
to
manage
our
production
environment
and
we
are
in
the
lookout
for
a
more
more
continuous
solution,
because
we
had
no
metrics
when
we
were
running
on
Google
App
Engine.
We
resorted
to
log
quite
for
boldly
what
our
applications
are.
Doing
and
I
see
this
everywhere
around
me,
and
this
is
fine
until
you
start
using
Google
container
engine
or
Cuba
natives
at
that
things
started
smoking
because
the
rate
at
which
our
applications
were
producing
locks
was
just
too
much
the
flu
and
the
pots
were
choking.
A
So
we
have
learned
a
lesson
over
there.
We
lost
quite
some
locks
and
since
then,
we
have
even
more
application
metrics
and
reduce
the
amount
of
locks
that
we
are
emitting
all
the
way
back
to
only
unexpected
application
errors.
These
ablation
application
errors
are
then
locked
using
structured,
JSON
and
always
enriched
with
our
port
and
node
name.
A
A
Looking
at
the
future,
we
see
that
there
are
already
a
lot
of
features
available
or
or
Indonesia
or
in
the
near
future,
become
available.
We
want
to
decrease
the
latency
of
our
US
and
Asian
users
by
using
cluster
Federation,
which
is
something
we
are
looking
into.
We
would
like
to
accumulate
anonymous
Diagnostics
data
in
batches
and
submit
those
to
bigquery
using
stateful
sets
combined
with
persistent
volume
claims,
and
it's
always
a
good
thing,
and
especially
for
us,
with
such
a
small
team
to
decrease
our
operations.
A
So
we
would
love
to
have
how-to
upgrades
of
our
own
out
so
that
we
can
scratch
that
from
the
list
and
the
disruption
budget
is
easy
because
it
allows
you
to
configure
a
relative
or
absolute
amount
of
pots
which
always
need
to
be
available,
and
this
is
important
for
voluntary
scaling
operations
of
your
notes.
This
feature
is
available
in
1.7
release.
It's
already
there,
but
it
it's
getting
extended.
A
We
have
experienced
that
kubernetes
is
great
at
resource
allocation,
lifecycle
management
and
container
Eisinger
app,
and
all
this
invites
you
to
build
true
micro
services.
I
am
a
developer
at
heart
and
I
don't
mind
to
do
operations
with
kubernetes
the
tooling
and
the
configuration
just
feels
very
native
to
me
and
as
I've
shown
you
with
a
few
examples.
The
defaults
of
cube
natives
were
great
out-of-the-box,
but
there
are
enough
possibilities
to
override
these
and
to
extend
them.
A
A
Let's
fail,
but
if
there
is
one
thing
that
I
would
like
you
to
remember
after
this
talk,
great
monitoring
is
paramount.
Only
then
will
be
able
to
keep
your
team
small
as
we
did
it
rate
fast,
break
things,
diagnose
those
and
fix
them
and
continue
delivering
great
software.
I
would
like
to
thank
you
all
for
listening
to
my
talk,
I'll
be
here
for
the
remainder
from
of
cube
con.
So
if
you've
got
any
questions
or
things
that
you
like
to
discuss,
you
can
find
me
here
or
reach
out
to
me.