►
Description
Managing a Multi-Tenanted Kubernetes Cluster in Production - Josh Bowen, Apigee
"Kubernetes clusters dedicated to a single organization are becoming common, either run by the organizations that use them or hosted by others. Less common is a multi-tenant use of a single cluster.
There are problems to be solved in managing a multi-tenanted Kubernetes cluster in production. At Apigee, we are building a new Kubernetes-based platform that hosts applications for our clients and ourselves on a single, shared cluster.
This talk will cover:
- Securely routing traffic to the correct tenant
- Isolating tenant network environments
- Authenticating and authorizing management API calls using our own and our customers' identity providers and access control policies
- Creating a multi-tenanted build and deploy flow"
About Noah Dietz
Noah Dietz is a software developer at Apigee and part of it's Microservices team. This team is dedicated to ideating and implementing new ways for Apigee to adopt a microservice architecture in its infrastructure. He has only been working with Kubernetes for a few months now, but he is the proud owner of a Kubernetes 1st Birthday party hat from the local k8s Meetup. Before hacking on Kubernetes, Noah worked mainly with Node.js on Edge Microgateway, Apigee's hybrid-cloud API management solution, and OpenAPI (f.k.a Swagger) tooling. He can be reached on Twitter at @no_d_here and occasionally blogs on Medium.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
I'm
Josh
I'm
a
software
engineer
at
Apogee
I'm,
giving
a
talk
on
managing
a
multi-tenant,
Akua
Nettie's
cluster
in
production,
some
notes,
multi-tenant
II,
is
still
a
work
in
progress
in
kubernetes.
I
do
not
have
the
magical
solution
that
are
gonna
solve
all
of
your
problems.
This
is
just
our
take
on
it
and
what
we've
done
thus
far
so
begin
so
multi-talented.
What
do
you
mean?
What
does
this
mean?
For
us?
A
This
means
multiple
actors
with
our
own
motivations
and
priorities,
operating
independently
of
each
other
in
isolation,
while
sharing
cluster
resources
kind
of
layman
I
mean
that
means
we
have
one
cluster
and
we
have
multiple
users
within
it.
Those
users
may
not
know
each
other.
They
may
not
trust
each
other
but
they're
within
the
same
cluster,
we're
not
giving
everyone
their
own
control
plane
when
they
are
a
different
user.
Some
example.
A
This
could
be
multiple
functional
groups
within
the
same
company,
multiple
applications
of
a
single
company
that
all
serve
distinct
clients,
or
it
could
be
multiple
clients
using
a
hosted
cluster,
which
is
one
of
our
use
cases.
There's
really
four
subjects
I'm
going
to
be
covering
here
on
the
cover,
authentication,
authorization
organization
and
communication
I'm
going
to
go
pretty
high-level
over
most
of
these
since
there's
a
bit
of
material
there's
a
lot
of
other
great
people
you
can
talk
to
about
these
specifically
I
talked
to
Eric
Chang.
A
If
you
want
to
talk
about
authentication
or
authorization,
he
just
gave
a
great
talk
like
an
hour
ago
about
this.
So
yeah.
Let's
start
with
authentication.
What
is
authentication
authentication
is
a
process
by
which
a
system
verifies
the
identity
of
a
user
who
wishes
to
access
in
for
this?
This
means
it
gives
us
a
user.
Once
someone
has
passed
authentication,
we
can
assume
that
they
are
who
our
Authenticator
said.
They
are,
and
we
need
this
because
we
need
to
know
a
user's
identity
to
determine
their
scope
as
a
tenant
in
this
multi-talented
cluster.
A
Kubernetes
does
authentication
in
a
couple
ways
again:
Eric
went
over
quite
a
few
more
ways:
I'm
going
to
cover
three
common
ways:
certificates
service
counts
and
then
some
of
the
work
we
did
with
Open
ID
Connect
certificates
are
probably
going
to
be
the
most
common
one.
People
have
probably
all
seen
a
similar
file
to
this
is
just
gonna,
be
a
cube.
Config
file.
You
can
see
you
have
your
certificate
authority.
There,
that's
going
to
be
hard-coded
into
your
API
server.
You're
gonna
have
your
client
certificate
in
your
client
key.
A
A
Kubernetes
does
not
actually
have
a
hard-coded
user
database,
so
there
is
no
idea
of
users,
they're
really
just
strings
once
they
get
past
the
authentication
stage
where
this
one
is
actually
coming
from
it's
going
to
come
from
the
common
name
of
your
client
certificate,
a
little
bit
of
trivia
the
documentation
there.
As
with
many
things,
it's
not
great,
but
that's
the
useful
thing
to
know.
The
next
way
is
going
to
be
service
account
tokens.
A
These
are
not
actually
going
to
be
from
users
in
the
traditional
sense,
but
more
from
pods
or
apps
running
in
your
cluster.
This
is
actually
going
to
be
had
they
authenticate
to
the
cluster
on
the
left.
There
you're
gonna
have
what
an
actual
service
account
looks
like
you,
don't
need
to
make
one
it's
gonna
be
by
default
in
every
namespace.
A
Every
pod
that
runs
in
that
namespace
is
going
to
use
that
default
service
account
and
on
the
right
it's
going
to
attach
to
that
secret
there,
and
that
secret
is
actually
gonna
contain
the
information
to
authenticate
to
the
cluster.
The
authentication
results
from
this
is
going
to
look
something
like
this.
You're
gonna
have
the
system
service
account
default
example.
Service
account
it's
worth
noting
that
that
default
is
actually
going
to
be
the
namespace
of
that
service
account.
So
that's
what
your
user
string
is
going
to
look
like
coming
out.
A
The
third
thing
about
authentication
I'm
gonna
discuss
is,
is
Open,
ID
Connect.
This
is
a
more
complicated
one
again.
Eric
covered
this
in
a
lot
of
detail
and
the
Dex
project
is
something
I
look
into
for
us.
We
used
our
Apogee
SSO
to
actually
connect
to
this,
and
what
this
means
is
you
have
an
authentication
provider
and
for
us
that's
at
logon
dot
e
to
e
Apogee
dotnet.
That's
our
e
to
e
authentication
provider.
A
The
way
you'd
actually
consume
that
at
the
client
is
again
in
your
cube.
Config
file
you're
actually
going
to
send
this
token,
and
this
is
just
going
to
be
a
JWT
that
you
generate
through
whatever
means
you
get
JW
T's
from
your
open,
ID
connected
provider,
you
pass
that,
through
on
the
right,
you're
gonna
see
all
this
stuff.
You
know
you're
going
to
have
the
actual
information
there
and
one
of
the
interesting
things
about
how
this
is
actually
going
to
come
out
is
by
default.
A
You're
gonna
get
this
thing,
which
is
going
to
be
the
issuer
concatenated
with
your
user
ID.
If
you
look
back
here
that
long
hex
is
going
to
actually
be
your
user
ID
and
your
issuer,
but
what
we
have
to
keep
in
mind
is
what
this
actually
ties
into
is.
This
is
going
to
give
me
my
user.
It's
gonna
give
my
FG
count
j-bone
at
Apogee
dot-com.
A
So
at
this
stage
we
can
assume
that
we
have
an
authenticated
user
used.
One
of
these
means-
and
now
you
have
a
user,
so
we
have
to
go
into
authorization.
What
does
authorization?
You
know
off?
Authentication
authorization
are
discussed
a
lot
in
the
same
context
because
they
usually
go
hand
in
hand
with
each
other,
an
authorization
once
we
know
who
user
is.
A
Authorization
tells
us
what
that
user
is
allowed
to
do
in
our
context
why
this
matters
is
because
we
need
to
strictly
determine
what
kubernetes
functions
a
user
is
allowed
to
do
in
a
namespace
or
in
the
cluster
as
a
whole.
Obviously,
if
we
have
a
multi-tenant
a
namespace,
we
can't
have
users
having
free
rein
to
access
the
API
server
because
again
authorization.
In
this
sense,
it's
limited
just
to
interactions
with
the
API
service.
So
this
may
be
your
pod
or
your
user.
A
Kubernetes
has
a
few
ways
to
do:
authorization
I'm,
just
gonna,
cover
three
of
them.
There's
a
trivia
based
access
control,
I'm,
assuming
a
lot
of
people
are
familiar
with
that.
That's
very
useful
for
bootstrapping
and
it's
kind
of
one
of
the
default:
more
authorization
modes,
there's
other
role-based
access
control,
that's
still
alpha
and,
unfortunately
still
going
to
be
alpha
in
one
five,
but
hopefully
we
can
move
to
beta
in
1/6
there
and
then
webhooks
so
for
attribute
based
access
control.
A
So
if
you
don't
have
this
policy,
no
one
is
going
to
be
allowed
to
look
at
anything
on
your
cluster
and
that
includes
components
of
the
actual
control
plane
like
the
cubelet
and
the
controller
manager
and
stuff
like
that
on
the
right,
you
might
have
a
more
restrictive
policy,
or
rather
a
higher
privilege
policy
that
you
give
to
an
admin
users.
It
doesn't
just
have
read-only
resources,
but
actually
can
see
the
whole
cluster.
This
might
be
something
used
for
bootstrapping
or,
as
you
are,
the
person
administering
a
cluster,
so
type
a
role
based
access
control.
A
In
this
case,
it's
going
to
be
Jane
is
allowed
to
have
these
permissions.
We
didn't
use
this
for
our
use
case.
One
of
the
reasons
was
its
alpha:
it's
relatively
new,
and
also
we
didn't
want
to
replicate
our
whole
permission
service
in
kubernetes,
because
a
large
portion
of
our
services
are
still
running
outside
of
kubernetes.
With
that
in
mind,
what
we
did
use
was
a
web
hook
server,
and
what
this
is
is
you,
after
we
have
an
external
server,
that
is
managing
your
authorization
for
you.
A
You
essentially
configure
the
API
server
that
when
a
request
comes
into
the
API
server,
it's
actually
going
to
make
a
call
to
that
server
and
it's
going
to
expect
a
response
from
that
server.
That
tells
it
whether
or
not
that
person
is
authorized,
and
this
allows
you
to
tie
into
a
permissions
back-end
in
whatever
database
or
existing
permission
service.
You
have
that's
going
to
look
a
little
bit
like
this.
You
know
we
have
a
web
hook
request
here,
and
this
is
the
actual
JSON
object.
A
We're
gonna
see
if
you
can
see
it's
a
standard,
kubernetes
object,
it
has
your
API
version,
your
kind
and
a
spec,
and
it
since
you
can
say
someone
in
this
namespace
is
trying
to
do
this
verb
action
in
this
group.
We're
group
is
going
to
be
like
the
API
group
and
on
this
resource
and
that's
gonna
give
their
user
it's
going
to
expect
a
response
that
looks
like
this
again
standard,
kubernetes
object
and
all
what's
really
looking
for.
A
It's
not
looking
for
a
spec
at
all,
always
looking
for
the
status,
and
it's
just
looking
for
that
allowed
true
or
that
allowed
false.
If
we
see
another
one
say
we
were
trying
to
use
the
user
Josh
again,
but
we're
trying
to
access
the
admin
namespace
this
time,
we're
gonna
get
an
allowed
false
and
then
we
do
additionally
get
to
have
a
reason
there
and
that'll
just
allow
us
to
say.
Okay.
This
person
was
not
allowed
to
do
this
because
user
Josh
is
not
an
admin
or
something
like
that.
A
The
next
thing
so
at
this
point
I'm
going
to
assume
that
we
have
a
user
of
authenticated
to
the
cluster,
a
user
is
now
authorized
within
the
cluster
to
do
what
we
say
they
want
to
do,
but
now
we
have
together,
you
have
what
is
it
that
we
actually
authorized
that
cluster
to
do
and
we're
doing
this
through
isolation
through
organizations?
So
we're
organizing
the
cluster
in
such
a
way
that
all
of
the
tenants
are
isolated
from
each
other
there's
a
few
concepts
within
this
namespacing
resource
quotas
and
network
security.
A
So
with
names
facing.
Let's
say
we
have
this
cluster.
We
have
the
world,
we
are
a
cluster
and
we
have
four
islands,
so
they're
all
our
tenants.
We
want
those
tenants
to
be
isolated
from
each
other
and
so
for
what
we're
doing
right
now
in
the
multi-tenancy-
and
this
is
obviously
a
point
of
contention.
If
you
talk
to
the
open
ship
guys
or
some
of
the
people
on
the
cig,
networking
channel
we're
using
a
namespace
as
the
boundary
for
what
a
tenant
is.
Obviously
there
are
some
downsides
to
that.
There
are
some
upsides
to
that.
A
We
thought
at
this
stage
that
the
upsides
of
saying
okay,
each
tenant
gets
one
namespace.
If
they
need
to
have
more
separation,
we
can
give
them
another
namespace,
but
that's
going
to
be
viewed
up
by
us
as
separate
user
and
we'll
manage
that
external
to
the
kubernetes
system.
But
this
just
allows
us
to
work
with
the
existing
kubernetes
primitives
a
lot
better
than
if
we
try
to
abstract
this
into
some
many-to-many
relation.
A
So
how
do
we
enforce
name
spacing
so
because,
as
we
discuss,
you
know,
there's
authorization,
it's
limited.
What
a
client
can
do.
Obviously
we're
not
going
to
give
a
client
cube,
CTL
and
say:
oh
just
go,
make
a
namespace,
because
we've
authorized
a
user
to
make
a
namespace,
now
they're
no
longer
isolated
on
that
namespace
boundary.
A
So
if
you
made
this
project,
we
call
it
in
Rober,
it's
open
source
on
github
I'm,
not
gonna
go
into
depth,
but
essentially
this
allows
us
to
use
some
existing
Apogee
concepts
that
users
are
familiar
with
and
use
that
to
create
namespaces
that
they
can
then
manage
themselves,
and
this
allows
us.
Doesn't
you
have
an
external
actor
that
is
authorized
to
talk
into
the
cluster
and
create
our
namespace
it's
there,
and
then
we
can
give
users,
those
namespaces
and
they're
isolated
within
that
the
next
currently
resource
quotas.
A
You
know
our
use
case
is
that
we
are
hosting
a
lot
of
tenants
on
one
cluster
they're
paying
us
to
host
on
our
cluster.
That
means
that
they
are
not
hosting
their
code.
We
are
paying
the
hosting
cost
of
having
their
code
running.
Obviously,
we
don't
want
to
be
losing
money.
We
are
still
businesses.
So
if
they're
using
more
resources
than
they're
paying
us
we're
gonna
lose
money,
that's
not
helpful,
so
we
want
to
have
resource
quotas.
Let's
say
we
have
this
total
pie.
A
You
know
we
have
four
tenants,
that's
fine,
we'll
split
them
up
like
that.
You
know
we'll
give
everyone
an
equal
share
of
the
cluster
or
really.
This
is
gonna,
be
more
of
an
enterprising
and
say
we
give
you
what
you
pay
for.
This
is
going
to
be
you
paid
for
this
many
API
requests.
Well,
this
is
going
to
be
a
similar
concept
of
you
paid
for
this
much
space
on
our
cluster,
and
then
we
can
split
the
fish
there,
there's
really
two
ways
or
two
main
resources
that
we
need
to
actually
quota
here.
A
We're
gonna
have
compute
resources.
Most
people
are
probably
going
to
be
a
bit
used
to
that.
There's
gonna
be
a
traditional
VM,
since
you
know
this
is
how
many
CPUs
you
get.
This
is
how
much
RAM
you
get.
This
could
also
relate
to
how
many
persistent
volumes
you
might
have
on
the
back
end,
but
it's
just
gonna
be
general
kind
of
think
of
it
as
the
AWS
sense
of
what
resources
you
are
allowed
to
use
for
us.
The
additional
one
is
going
to
be
the
object
quota.
A
This
is
going
to
be
more
of
a
kubernetes
native
concept
because
this
just
says
we're
gonna.
Let
you
do
these
many
kubernetes
resources,
because,
even
though
we
are
limiting
on
the
compute
resources
and
that's
gonna
actually
be
able
to
call
that
server
cost,
it's
you've
got
to
keep
in
mind
that
we're
still
paying
for
the
masternodes
on
that
kubernetes.
So
every
pod,
you
run
every
service.
You
run
every
persistent
volume
you
claim
you
make,
and
this
goes
on
to
load,
balancers
and
there's
a
lot
more
things
that
we
can
actually
cover
here.
A
That's
all
actually
going
to
create
something
and
we
want
to
limit
how
many
people
they
use.
One
of
the
big
examples
of
this
is
gonna
be
services.
You
know
if
you
know
how
kubernetes
cluster
works
on
the
networking,
and
there
is
not
an
infant
number
of
IPs.
We
can't
let
one
user
we're
on
a
pod
that
just
loops
and
takes
up
all
of
the
IPS
on
our
cluster.
Obviously,
that's
not
going
to
work
for
us.
So
that's
a
quick
thing.
A
The
more
important
component
of
this
whole
tendency
thing
I
think
the
most
interesting
part
of
this
is
network
security.
So
let's
say
we
have
you
know
this
world.
We
have
our
four
tenants.
Someone
may
try
to
talk
to
someone
else.
The
way
that
currently
coober
names
DNS
works
is
we're
not
hiding
someone
from
you
we're
just
not
allowing
you
to
talk
to
them,
but
that
means
someone
could
try
to
talk
from
ten
and
a
to
ten
and
be
something
like
that.
Obviously
we
don't
want
this
to
happen.
A
Calico
I'm,
not
gonna,
go
too
in
depth
into
calico,
KC,
Davenport
I'm
sure
is
wandering
around
the
conference
somewhere
go
talk
to
him
or
some
of
the
other
Tiger
guys
they
do
a
great
job
with
that
project,
but
not
to
give
them
all
the
fanfare.
There
are
other
enforcement
agents
for
these
network
policies.
We've
worked
is
right
out
there,
Romana
I'm,
not
sure
if
they're
here,
but
there's
lots
of
other
guys
so
look
into
that
more
before
making
decision,
but
going
down
into
the
depth
here.
A
Let's
look
at
these
network
policy
objects
and
what
chewie
allows
us
to
do
this
isolation
on
the
less?
It's
not
actually
network
policy
object,
but
because
of
the
state
of
network
policies
in
native
kubernetes
right
now
we
need
this
annotation
on
all
of
our
namespaces
and
what
this
says
it's
a
default
deny
it
you're
not
allowed
to
talk
to
this
namespace
from
outside
of
this
namespace
by
default
and
the
way
we
get
around
that
essentially,
we
choose
to
poke
holes
in
that
network
or
in
that
default
deny
using
network
policies.
A
But
this
is
really
important,
because
one
of
the
things
we
wanna
make
sure
is,
we
want
to
be
a
default
secure
thing
we
want
to
say
you
can
open
up
your
namespace
more
if
you
want,
but
by
default
no
one's
gonna
accidentally
get
in
because
you
failed
to
enforce
your
network
policies
the
right
way
on
the
right.
We
actually
have.
The
network
policy
object.
That's
actually
going
to
be
how
we
poke
the
holes.
It's
a
couple
ways
we
can
do
that.
You
can
essentially
allow
ingress
from
several
sources.
A
You
can
say
I'm
gonna,
allow
this
port
to
be
open,
and
people
can
talk
to
me
on
this
port.
You
can
say:
I'm
gonna,
allow
people
from
this
namespace
or
this
pod,
and
essentially
that's
gonna,
be
the
grouping
in
the
kubernetes
native
environment
that
allows
people
to
talk
to
you
similar
to
an
Rober.
How
did
we
manage
this
because,
obviously,
with
authorization
we
can't
let
people
manage
their
own
network
policies
or,
at
the
very
least,
we
need
to
have
our
own
way
of
managing
network
policies
for
them
to
bootstrap
them,
or
something
like
that.
A
So
what
we
made
is
a
product
that
we
call
Congress
again,
it's
open
source,
it's
on
github,
and
what
this
is?
It's
a
network
policy
controller.
It
actually
creates
these
network
policies
by
default,
manages
them
make
sure
they
continue
running
to
actually
make
sure
that
our
cluster
is
secure,
and
this
relates
all
into
communication
and
that's
a
big
part
of
this
cluster.
You
know
and
there's
really
two
parts
of
communication
here,
there's
gonna,
be
your
inter
cluster
communication
and
to
me
you're
in
truck
cluster
communication.
A
I
briefly
touched
on
it
intra
cluster
communication
with
our
network
isolation
I'll
go
into
it
a
little
more
in-depth
but
I'd
like
to
touch
on
inter
cluster,
because
it's
a
big
one,
that's
important.
It
actually
relates
to
how
we're
doing
some
of
our
inter
cluster
stuff.
You
know
it's
great.
If
you
have
this
nice
kubernetes
cluster,
you,
although
your
stuff
running
in
it,
but
obviously
any
enterprise,
it's
migrating
to
kubernetes
or
anyone
putting
some
stuff
on
kubernetes.
Not
all
of
the
things
that
are
talking
to
their
service
are
actually
gonna,
be
running
within
kubernetes.
A
There
have
a
lot
of
things.
They
may
be
exposing
that
api
publicly.
They
may
have
a
front-end
that
wants
to
talk
to
it.
They
may
want
some
of
their
existing
enterprise
services
to
be
able
to
talk
into
the
cluster,
and
so
that's
what
we
have
to
do.
We
have
to
come
from
essentially
the
public
internet
and
talk
into
our
cluster
and
the
way
we're
gonna
actually
accomplish.
That
is
with
an
ingress
controller
and
the
way
we
implemented
that
existing
is.
We
just
use
some
of
the
ingress
controller
stuff.
A
The
kubernetes
gives
us,
along
with
building
on
top
of
the
provide
example,
nginx
configuration,
and
essentially
this
allows
us
to
load
balance
across
all
of
our
existing
tenants
from
this
ingress.
So
the
ingress
controller
is
essentially
going
to
be
the
first
point
of
contact
from
the
internet
and
then
that
controller
is
going
to
be
able
to
talk
to
actually
all
the
clusters.
That's
gonna
look
something
like
this.
A
You
know
a
service
may
come
in
and
it's
actually
gonna
proxy
to
something
like
HP
food
service
or
bar
service,
where
those
could
just
be
service
names
in
kubernetes,
using
traditional
services
and
just
using
the
cluster
dns
from
that
using
ingress
resource
objects,
which
is
actually
a
kubernetes
native
object
and
a
default
ingress
controller
application
looks
at
the
API
server
pulls
in
these
ingress
resource
objects
and
that's
what
it
actually
uses
to
configure
its
ingress
rules.
So
on
the
left,
you
have
that
ingress
resource
and
what
we're
gonna
say
is
say
we
have
two
services.
A
We
want
to
expose
one
at
foo.com.
We
want
to
explode
another
at
Bardolph,
you
don't
come
and
the
backend
is
actually
going
to
be
a
kubernetes
back-end.
Where
do
we
actually
want
that
to
talk
to
so
it's
gonna.
Look
something
like
this
on
the
right.
You
know
fubar
comm,
it's
just
gonna
go
to
food
bar
comm.
It's
gonna
pass
that
host
through
and
talk
to
service,
one
same
thing
with
bar
dot,
foo
comm.
Obviously
you
might
want
to
do
pass
based
as
an
alternative.
Everything
comes
in
is
fubar
comm.
A
If
it's
/foo,
it's
gonna
go
to
service
one.
If
it's
slash
bar,
it's
gonna
go
to
service
two,
so
this
gives
you
just
another
layer
on
top
of
the
kubernetes
round-robin
load,
balancing
that
they
do.
In
addition
to
this,
so
now
that
we
have
a
way
to
talk
from
outside
to
inside,
we
need
to
look
more
clearly
an
inch
or
cluster
I'm
gonna
go
back
to
what
we
discussed
with
the
network
policies.
A
You
know
we
said
you
can
talk
with
in
your
namespace:
that's
fine,
that
ship
can
go
and
team
a
can
speak
within
team
a,
but
what?
If
you're
a
team
that
has
three
namespaces?
They
have
four
names
because
we
said
we're
picking
the
tenant
as
the
namespace
boundary,
but
you
may
be
an
organization,
that's
run
someone
like
Microsoft,
you
have
disparate
teams,
but
you
still
want
to
communicate
with
each
other.
So
this
is
intra
namespace.
A
This
is
fine,
but
this
is
inter
namespace,
and
this
is
not
fine,
because
we
have
this
default,
deny
policy
on
all
of
our
namespaces.
This
means
that
this
communication
won't
be
able
to
happen.
Well,
there
may
be
needs
to
make
this
happen.
So
what
we
did
is
we
looked
at
our
ingress
controller
and
say:
okay,
ingress
controllers,
doing
a
very
similar
thing:
it's
allowing
traffic
to
come
from
outside
and
then
it's
distributing
it
inside.
So
what
we
decided
to
do
is
essentially
make
a
modified
ingress
controller
and
place
it
inside
of
our
cluster.
A
We
made
this
using
nginx
a
little
bit
of
our
own
stuff
and
obviously
the
kubernetes
concept,
and
we
place
this
in
a
namespace
with
heightened
privileges
when
I
say
heightened
privileges,
I
mean
this
is
a
namespace
that
is
allowed
to
talk
to
all
other
namespaces.
So
the
network
policies
on
this
namespace
or
on
all
other
namespaces
managed
by
Congress
are
gonna
say
if
the
router
namespace,
because
that's
what
we've
been
calling
it
internally
if
the
router
namespace
is
trying
to
talk
to
you,
you
allow
all
talking
to
happen
and
that's
managed
by
us
another
key.
A
We
added
some
lightweight
routing
keys
to
this
just
attack
on
a
little
bit
more
security,
because
a
little
people
are
uncomfortable
at
the
idea
of
okay.
We
have
one
master
namespace,
so
you
get
an
extra
step
of
security.
You
essentially
get
to
put
a
password
on
any
of
your
services
that
run
in
this
and
then,
if
you
want
to
talk
to
it,
you
have
to
pass
that
password
through
and
our
router
is
gonna.
A
Do
the
job
of
passing
that
through
and
you
verify
that
on
your
side,
and
that
gives
us
something
like
this-
that
we
call
managed
communication.
So
this
gives
us
a
story
of
we
have
intro
cluster
or
intra
namespace
communication.
That's
all
managed,
that's
free
rein!
Anything
you
want
to
do
on
your
own,
put
it
in
your
tenant
namespace!
That's
just
gonna
work
out
of
the
box.
If
you
want
to
talk
from
external
the
cluster,
that's
gonna
go
through
an
ingress
controller.
A
If
you
want
to
talk
to
two
different
tenants,
maybe
you
are
an
organization
that
has
multiple
tenants
in
this
sense,
you
can
go
through
our
managed
community,
so
I
know,
I've
talked
a
little
bit
quickly
and
I
tried
to
cover
a
lot
of
things.
It
looks
like
we
have
quite
a
bit
of
time
left,
but
that's
the
end
of
my
slides
I'll
put
my
contact
information.
That's
just
Twitter
as
well
as
my
two
teammates
in
the
front
row
here
feel
free
to
ask
any
of
us
questions
again.
A
A
Yes,
so
the
question
is,
you
know
we
lock
down
traffic
between
the
namespaces.
What
do
we
do
about
DNS
and
actually
finding
right
now
we're
not
actually
doing
any
work
to
modify
sky
DNS,
which
is
actually
what
we're
running
internally
to
kubernetes,
since
that
so
people
are
discoverable
and
I
know.
That
is
a
thing
of
you
know.
That
is
information
by
telling
someone
they
exist.
There
are
modifications,
they've,
probably
done
to
that,
but
right
now
we're
not
actually
doing
anything.
So,
yes,
someone
could
do
that.
A
One
of
the
modification
we
have
to
be
done
is
we
don't
use
in
some
cases,
services
our
internal
router,
actually,
managers
on
pods
and
looks
for
pod
creation
events
and
does
its
own
load
balancing
rather
than
load
balancing
on
top
of
the
service
load
balancer.
So
then
those
pods
actually
won't
show
up
in
your
sky.
Dns
query.
So
if
you
try
to
you
know,
look
at
the
DNS
record,
you
won't
see
them
because
it
won't
be
a
service,
so
you
can
hide
them
by
not
relying
on
kubernetes
native
services.
A
So
it's
not
actually
an
ingress
controller,
it's
very
similar.
In
essence,
an
ingress
controller
is
a
load
balancer,
so
traditionally
the
HK,
proxy
or
nginx,
and
it
looks
at
ingress
events
in
the
kubernetes
api,
since
we
pulls
the
API
or
looks
at
the
event
watcher
and
works
on
ingress
resources.
The
difference
with
our
match
communication
is
we
pull
the
API
looking
at
pod
events,
and
then
we
have
custom
annotations
on
pods
that
are
created
for
us
if
they
want
to
be
routed
and
we
look
at
those
annotations
and
say:
okay
based
on
these
annotations.
A
A
So
one
of
the
other
reasons-
I
didn't
talk
to
that's
a
little
bit
specific
to
your
business
use
case
is
we
do
API
management,
so
we
actually
wanted
to
have
a
touch
point
in
inter
cluster
communication,
where
we
can
feed
we
can
pull
analytics
out
or
we
can
actually
do
some
custom
manipulation
of
hosts
averse
paths
and
stuff
like
that.
So
that's
one
of
the
big
reasons
that
we
went
with
this
solution.
A
A
A
A
A
Yes,
there's
actually
quite
a
bit
of
work
in
the
community
and
actually
I
think
we
just
sent
out
a
Google
Doc
on
the
Signet
working
group.
I.
Think
sig
networking
is
leading
this
effort
with
the
Google
Doc
trying
to
pull
interest
and
discuss
what
ideas
we
want
to
pull
from
multi-tenancy.
A
lot
of
the
starting
point
is
what
openshift
has
provided
in
their
solution
and
how
we
want
to
maybe
upstream
some
of
their
work,
or
do
that
so
I'd.
A
Look
at
the
Signet
working
group
they're
doing
some
work
with
the
overall
multi-tenancy
and
as
I
said,
this
is
not
what
I
would
consider
a
complete
solution.
This
is
not
necessarily
the
best
solution.
We
obviously
do
want
something
in
kubernetes
in
talking
with
some
of
the
more
kubernetes
core
people,
they're,
not
sure
if
they
want
multi-tenancy
to
be
a
core
part
of
kubernetes.
If
you
guys
were
at
the
meetup
on
Monday,
they
discuss
how
they
want
to
have
a
difference
between
what
is
core
kubernetes
and
what
is
a
functional
kubernetes.