►
From YouTube: Building a Secure, Multi-Protocol and Multi-Tenant Cluster for Internet-Facing Services [A]
Description
Building a Secure, Multi-Protocol and Multi-Tenant Cluster for Internet-Facing Services [A] - Bich Le, Platform9
Exposing internal HTTP-based services to the Internet is a well supported and documented feature of Kubernetes. What's less well understood is how to do it for thousands of services running on behalf of hundreds of possibly competing customers, in particular how to do it securely, protect the privacy of each customer, and support binary protocols other than HTTP. This is the problem that our company solved for our SaaS business which requires hosting and operating the control plane of popular infrastructure management software (e.g. Openstack, Big Data, and Kubernetes itself) as a service for our customers. Those control planes contain services exposing protocols as varied as MySQL and AMQP. This talk describes the challenges we faced and how we solved them using multiple technologies from the Kubernetes ecosystem. The solution includes a system that automatically creates namespaces, provisions certificate hierarchies, and manages ingress controllers for new customers, then wraps services with a set of side-car containers to handle tasks such as TLS termination. We describe how we employed Kubernetes native constructs such as Custom Resource Definitions to automate those tasks. For network communications, we discuss how to securely handle ingress, outgress, pod-to-pod, and cross-namespace traffic. To support both HTTP and TCP-based protocols, we describe a two-level network routing system consisting of both a "k8sniff" and an nginx ingress controller. For ensuring customer data privacy we compare these approaches: (1) Network Policy + Layer 2 virtualization; (2) TLS encryption of all pod-to-pod traffic; (3) a combination of the two. Finally, we debate whether the process isolation model of Linux containers is sufficient, and discuss our experience with stronger virtualization-based mechanisms such as Frakti / HyperContainer.
About Bich Le
Co-founder of Platform9 and veteran of VMware. Career in virtualization, cloud management and containerization.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
My
name
is
dick
Lee
I'm,
a
co-founder
and
chief
architect
at
platform
9
and
today,
I
want
to
talk
about
something
we
built
and
it's
a
lesson
about
how
to
use
kubernetes
building
blocks,
to
create
something
useful.
So
we
built
a
tool
to
make
it
really
simple
for
us
to
deploy
large
numbers
of
services
running
on
behalf
of
many
many
customers
and
services
that
are
exposed
on
the
Internet
and
how
to
do
this
in
a
secure
enough
way
to
isolate
each
customer
and
that's
what
we
mean
by
multi-tenant.
Ok,
so
to
motivate
today's
talk.
A
Let
me
briefly
discuss
our
a
use
case
at
platform
9,
so
we
run
open
source
software
as
a
service
for
our
enterprise
customers
and
we
specialized
in
infrastructure
management
software.
That
is
why
we
offer
an
integrated
product
that
uses
kubernetes
for
container
management
OpenStack
for
virtual
machines
and
fishing
for
server
list
functions.
Now.
What
makes
is
unique
is
for
these
stacks.
A
We
split
the
control
plane
and
the
data
plane
so
that
the
customers
can
run
the
data
plane
on
the
hardware
of
their
choice,
which
could
include
bare
metal
servers
or
their
public
cloud
instances,
and
then
we
host
the
control
plane
and
run
it
as
a
service,
and
then
the
two
planes
communicate
securely
over
the
Internet.
Ok.
So
what
does
a
control
plane
look
like
for
kubernetes?
It
would
be
the
master
nodes
that
most
of
you
are
familiar
with
for
OpenStack.
We
would
run
dozens
of
services
such
as
Nova
API.
A
So
we
are
currently
in
a
transition
period
and
moving
all
of
our
control
plane
services
from
VMs
to
containers.
And,
of
course,
we
want
to
use
kubernetes
and
take
advantage
of
it.
We
also
want
to
use
and
run
everything
on
a
shared
kubernetes
cluster
to
gain
high
utilization
and,
of
course,
all
the
benefits
you
get
from
kubernetes.
A
So
during
this
transition
we
wanted
to
build
a
set
of
tools
that
would
help
our
ops
team
automate
as
much
of
this
as
possible
right
something
that
lets
them
very
easily:
provision
new
customers,
new
regions
of
customers,
deploy
the
services,
make
sure
that
they're
exposed
on
the
internet
with
dns
names
and
secure
everything,
so
that
everything
can
be
well
isolated
and
support.
Multiple
protocols
such
as
HTTP,
and
also
some
of
the
TCP
binary
protocols,
and
since
our
ops
team
is
kind
of
ramping
up
on
kubernetes
and
kubernetes
can
be
pretty
complex.
A
We
also
want
our
tool
to
automate
as
much
as
possible.
You
know
automate
a
lot
of
the
repetitive
tasks,
maybe
abstract
away
some
of
the
complexities
of
kubernetes
without
really
hiding
its
true
power.
And
finally,
we
wanted
to
build
something
that
is
useful
not
to
us
but
also
to
the
community,
and
that
is
why
we
started
an
open-source
project
and
this
I'm
going
to
talk
about
code,
the
very
first
generation,
because
it's
a
work
in
progress.
A
Is
called
deployment,
cluster,
config
and
ops
also
known
as
deco,
and
this
is
a
program
that
follows
the
typical
kubernetes
controller
or
operator
pattern.
What
it
means
is
it
declares
new
custom
resources
that
you
as
an
admin
can
create
in
kubernetes
and
once
it
detects
those
resources.
It'll
just
kick
in
and
do
a
lot
of
automation
for
you
and
we'll
talk
a
lot
about
this.
So,
for
example,
it'll
manage
DNS
records
and
inside
of
the
cluster
it'll
provision
and
teardown
all
the
resources
that
you
need
to
coordinate
kubernetes
like
namespaces
deployment
services
and
aggressive
resources.
A
So,
let's
start
with
space
so
think
of
space
as
a
naming
and
security
boundary
for
a
set
of
internet
facing
services.
Ok
and
in
deco
we
implement
this
as
just
a
kubernetes
name
space
with
a
DNS
binding,
so
that
it
is
visible
on
the
internet.
At
a
well-known
name
to
illustrate
this,
let
me
show
you
a
little
animation
here.
So
when
you
run
deco,
it
automatically
creates
a
special
namespace
called
echo.
This
is
where
you're
going
to
upload
your
space
resources.
A
A
The
way
it
does
that
is,
it
first
looks
up
the
public,
IP
or
endpoint
for
the
main
ingress
service
of
the
cluster
okay
and
then
it's
going
to
just
communicate
with
a
DNS
provider
to
create
a
DNS
record
and
the
record
is
going
to
create
Maps
the
fully
qualified
domain
name.
So
in
this
case
example,
one
sorry
example:
space
one
platform:
nine
net,
okay,
so
now
things
inside
of
that
new
namespace
can
be
accessible
from
the
outside.
A
A
If
it's
HTTP
service-
and
you
want
to
use
a
reverse
proxy,
then
you
specify
a
property
called
HTTP
URL
path.
Okay,
what
this
means
is,
if
you
set
that
property,
then
your
service
is
going
to
be
visible
at
this
endpoint,
which
is
HTTP,
slash,
slash
the
fqdn
of
your
space,
slash
the
URL
path
that
you
specify
all
right.
A
So
here's
an
example
of
an
HTTP
service.
Keystone,
which
is
mapped
to
the
path,
slash
Keystone,
and
we
use
this
in
OpenStack-
here's
an
example
of
a
tcp
on
binary
protocol
service.
It
does
not
have
the
HTTP
URL
path
and
additionally,
it
specifies
that,
yes,
we
do
want
to
authenticate
the
client
TLS
certificate.
A
Next,
we
have
the
concept
of
a
project.
This
is
really
an
optional
feature
of
spaces.
Okay,
and
what
it
allows
you
to
do
is
restrict
internal
traffic
within
the
cluster.
So
normally,
when
you
create
a
new
space,
the
pods
in
that
space
will
accept
traffic
from
any
other
space
within
the
cluster.
But
you
can
restrict
that
by
declaring
this
field
here
called
Project.
So
here
what
I'm
saying
is
only
accept
incoming
traffic
from
other
spaces,
who
also
have
their
project
set
to
death.
All
right
internally.
A
Deco
uses
a
kubernetes
network
policy
to
enforce
this,
and
hopefully
I'll
have
time
during
the
demo
to
show
you
what
it
looks
like
inside.
Here's
an
example
of
how
you
could
use
projects
here.
We
have
two
customers,
foo
and
bar,
for
which
we've
provisioned
multiple
spaces
representing
different
regions.
So,
for
example,
we
have
food,
East
and
food
West
and
to
make
sure
that
within
the
customer,
the
spaces
can
communicate
with
themselves
but
disallow
communication
cross
customers.
A
We
also
have
a
global
space
here,
running
a
service
we
its
console
and
we
use
it
as
a
global
configuration
database.
So
each
pod,
when
it
starts
up
it
needs
to
talk
to
console
in
order
to
retrieve
its
configuration,
and
so
all
spaces
need
to
be
able
to
access
console.
So
what
we
did
here
is
for
the
global
space.
We
set
the
project
to
be
empty
and
what
that
means
is
it
can
accept
the
traffic
from
any
other
space.
A
A
Last
but
not
least,
let's
talk
a
little
bit
about
traffic
management,
so
this
is
probably
the
most
complex
topic.
This
is
about
network
routing
and
it's
also
about
TLS
termination,
which
are
related.
So
deco
is
built
to
accept
multiple
providers
of
traffic
management.
The
requirements
we've
already
discussed-
and
you
know
it
needs
to
support
HTTP,
reverse
proxy
from
from
the
internet
and
for
TCP
services
and
need
to
support
and
2n
TLS
authentication.
A
It
is
our
intent
to
integrate
with
service
service
meshes,
especially
sto,
that's
currently
in
progress.
We
are
currently
blocked
by
an
issue
in
sto.
It's
about
sni
support
for
multiple
server
certificates.
It
looks
like
they're
working
on
this,
so
I
expect
this
to
be
fixed
within
weeks,
but
luckily
deco
also
comes
with
its
out-of-the-box,
very
simple,
Traffic
Manager
and
which
I'm
going
to
describe
today
so
to
route
traffic.
The
out-of-the-box
manager
uses
off-the-shelf
components,
and
this
is
pretty
interesting.
I
think
it's
using
two
ingress
controllers.
A
One
of
them
is
called
k8
sniff.
This
is
a
pretty
neat
controller
that
was
first
presented
at
the
coupon
Berlin,
so
that
one's
going
to
look
at
TLS
traffic
and
if
it
finds
a
matching
destination
that
matches
the
SMI
headers,
it's
just
going
to
direct
traffic
to
that
service
if
it
doesn't
find
a
match,
is
just
going
to
punt
it
over
to
nginx,
which
is
going
to
handle
HTTP
and
reverse
proxying.
Okay,
in
order
to
handle
TLS
termination,
dekha
would
inject
s
tunnel,
sidecar
containers
into
pods
and
those
s
tunnel.
A
A
Okay,
so
here
we
have
the
Deco
namespace
that
I
referred
to
earlier,
and
this
is
the
two
ingress
controllers
that
I
talked
about
they're
configured
to
be
chained,
one
after
the
other,
so
I'm,
first,
going
to
upload
two
secrets
containing
certs
and
Cas
for
HTTP
and
TCP
style
services
and
then
I'm
ready
to
create
my
space.
Okay.
In
this
example,
it
is
a
space
called
foo
West
with
a
domain
of
platform,
nine
net.
A
So
the
first
thing
that
deco
does
is
create
a
namespace
called
foo
west
and
then
copy
over
the
secrets
so
that
they
can
be
used
within
the
namespace
and
then
it'll
create
an
HTTP
ingress
resource.
Okay,
with
a
virtual
host
set
to
the
fqdn
of
the
space,
which
is
foo
west
platform,
nine
net,
it
also
initializes
a
rules
table
for
the
paths
and
initially
it's
empty,
except
for
slash
which
goes
to
the
default
service.
A
A
But
since
this
is
a
TC
sorry,
this
is
a
TCP
service.
Decco
also
creates
a
TCP
ingress
resource,
which
is
just
an
ingress
resource,
but
with
an
annotation
that
says
that
it
is
meant
for
the
k8
sniff
controller
and
I'll
explain
how
that
works.
So
now,
a
client
on
the
outside
can
connect
to
my
sequel,
foo
s,
dot,
plot
from
nine
net
that'll
get
routed
through
the
external
load
balancer
and
into
K
eight
Smith,
which
is
them
going
to
look
for
a
matching
TCP
ingress
resource
with
the
same
host
name
in
this
case.
A
For
a
HTTP
app,
that's
a
keystone
here
we
specify
a
path
of
slash
Keystone
for
the
reverse
proxy
same
thing:
deco
is
going
to
create
the
deployment
it's
going
to
create
the
service.
It's
going
to
inject
the
sidecars,
but
since
this
is
for
HTTP,
it's
also
going
to
add
a
new
rule
to
the
path
array
of
the
ingress
source
for
HTTP.
A
So
now,
if
a
client
accesses
this
URL
HTTP
foo
West
platform,
9
net
slash
Keystone
again
it's
going
to
go
to
K
8
sniff,
but
this
time
it's
not
going
to
find
a
matching
TCP
ingress
resource,
so
we've
configured
KH
sniff
to
just
punt
over
any
traffic
that
it
doesn't
recognize
over
to
engine
X.
So
now,
nginx
will
look
for
a
matching
HTTP
ingress
resource.
It
finds
one
with
the
the
required
virtual
host
name.
A
A
So
the
conclusion
of
all
of
this
is
deco
treats
internal
traffic
just
like
external
traffic
right
everything
is
meant
or
assumed
to
be
insecure,
and
so
you
always
have
nth
and
TLS
encryption
and
customers
are
isolated
between
each
other
because
they
will
have
their
will
typically
have
different
route.
Cas,
ok,
different
hierarchies,
and
in
addition
to
this,
as
I
spoke
earlier,
we
use
network
policy
with
that
concept
of
a
project
to
add
an
additional
layer
of
security.
A
A
A
A
A
And
one
thing
I
forgot
to
show:
you
was
I've,
also
uploaded
some
secrets
for
MySpace,
so
this
one
is
we're
verifying
TCP
traffic
and
it
has
a
CA
in
it.
In
addition
to
the
cert,
this
one
is
a
wildcard
certificate
for
the
HTTP
proxy.
So
I
have
those
two
secrets
specify
here.
So
let's
go
ahead
and
create
this
dev
one
well
before
I
created
this.
A
A
Notice
and
something
new
here,
there's
something
called
dev
one
and
that
is
the
new
name,
space
corresponding
to
our
deco
space.
So
if
I
say
K
again,
you'll
see
dev
1.
Ok,
now,
let's
see
if
deco
has
created
the
DNS
binding
for
us,
it
has
ok.
So
now
our
space
is
accessible
to
the
internet,
using
a
well-known
name,
and
next
we
are
ready
to
create
our
first
application
and
I'm
going
to
use
something.
We
demo
a
lot.
It's
called
time
server,
it's
kind
of
stupid,
but
it's
it's
great
for
demos.
A
It's
just
a
web
server
that
returns
the
current
time
and
the
pretty
cute
UI-
and
this
is
just
a
regular
spots
back
with
some
added
metadata
and
here
we're
saying
that
it
will
be
mapped
to
the
time
server,
slash
time,
server
path.
Ok,
so
let's
create
this
guy
and
we
need
to
create
in
the
dev
one
namespace.
A
A
A
Okay,
now
we
get
three
colors,
okay,
pretty
simple,
but
it's
showing
that
there's
three
pods
backing
the
service.
Now
for
our
next
example,
we're
going
to
look
at
a
TCP
type
of
app
and
I'm
gonna
show
you,
let's
see
my
sequel.
Where
is
there?
It
is
okay,
so
we're
going
to
spawn
a
my
sequel
container
with
a
root
password
set
to
dummy
and
we
are
going
to
require
client
verification
for
TLS.
A
A
A
A
A
So
how
about
DNS
DNS
has
been
created
for
us,
and
now
we
can
connect
to
our
database
over
that
well
known
name
over
the
Internet,
except
that
it's
enforcing
strict
TLS.
So
we
can't
just
run
our
standard,
my
sequel
client
against
it.
So
what
I've
done
is
I
have
a
local
esto,
no
configuration
that
uses
the
correct
client
certificate
to
authenticate
itself.
A
It's
also
using
the
correct,
see
a
file
to
verify
the
server
so
I'm
going
to
run
s
tunnel,
and
now,
let's
see
if
we
can
connect
to
the
database,
there
we
go
so
database
is
ok.
So
we're
connected
to
my
sequel,
great,
the
last
thing
I
want
to
show.
You
is
you'll
notice
that
our
space
in
set
the
project
name
to
dev,
ok
and
as
explained
earlier,
we
use
Network
policies
to
enforce
this
inter
space
traffic
and
so
somewhere
in
this
list.
We
should
see
a
network
policy
there.
It
is
ok,
so
tip.
A
Okay,
so
what
you'll
see
here,
I,
don't
know
how
much
you
know
about
network
policies
but
there's
various
ways
you
can
specify.
Who
can
do
when?
Who
can
I
connect
to
what
here
we're
saying
only
allow
incoming
connections
from
the
namespaces
with
a
label
that
has
this
pattern?
Deco
project
equals
death?
A
Okay,
so
that's
what
we
expected,
but
also
notice
that
we've
inserted
another
one
here
for
a
mysterious
namespace
with
a
label
deco
project
ecosystem
and
the
reason
we
had
to
do
this
was
to
allow
traffic
from
the
ingress
controllers
right
because
remember,
@kh,
sniffing
nginx.
If
you
didn't
have
this,
then
the
space
would
not
be
able
to
get
any
traffic
from
the
internet.
A
In
terms
of
road
maps,
so
this
is
work
in
progress
and
from
a
security
perspective,
I
discussed
how
we
handle
Network,
but
we
want
to
go
one
level
further
and
isolate
customers
between
each
other,
even
deeper.
You've,
probably
seen
a
lot
of
projects
around
using
virtual
machines
as
a
better
way
to
isolate
containers.
So
this
is
kind
of
our
next
step
is
to
investigate
integration
with
may
or
some
of
these
technologies.
A
In
terms
of
other
things
we
want
to
integrate
with,
as
I
mentioned
earlier
service
meshes,
we
want
to
be
able
to
also
take
advantage
of
automatically
setting
up
pod,
auto
scaling
and
perhaps
note
auto
scaling,
and
then
we've
looked
at
single,
more
kubernetes
clusters,
but
we
would
like
to
expand
echo
to
support
multiple
availability
zones
and
also
kubernetes
Federation.
So
that's
some
of
the
things
we're
going
to
investigate
next
and
to
find
out
more
there's
a
github
repo.
We
need
to
add
some
documentation
and
a
lot
of
tests.
You
can
visit
our
company
website.
A
We
have
some
pretty
interesting
talks
this
week.
If
you
like
to
hear
about
server,
lists
and
functions
as
a
service,
we
have
a
talk
about
fishing
and
a
co
speaker
and
myself.
We're
also
going
to
be
talking
about
how
to
run
kubernetes
clusters
more
cheaply
and
that's
Friday
and
that's
it
and
thank
you
so
much
for
coming
here
and
thank
you
for
your
time.