►
From YouTube: Using a Kubernetes Operator to Manage Application Tenancy in a B2B SaaS App - Mike Arpaia, Kolide
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Using a Kubernetes Operator to Manage Application Tenancy in a B2B SaaS App - Mike Arpaia, Kolide
Companies that create products for other companies or teams often have to reason about how to deal with the application-level tenancy of each team. The two ends of the spectrum are to either deploy one monolithic application that handles multi-tenant data isolation via application logic or to deploy and proxy to many instances of isolated single-tenant applications. This presentation will discuss how Kolide has approached the problem of application tenancy by building a Kubernetes Operator to manage the complete lifecycle of each tenant as an isolated instance of a single-tenant application. We will analyze strategies for account management, observability, deployments, networking, and security. Finally, we will consider the efficacy of this strategy in general by analyzing the observed pros and cons after using this approach in production to serve thousands of customer instances.
To Learn More: https://sched.co/GrU9
A
All
right
cool,
so
I'm
gonna
go
ahead
and
get
started.
Thank
you
all
for
coming
to
my
session
about
using
a
kubernetes
operator
to
manage
tendency
in
a
b2b
Sassa.
So
on
the
Internet.
Most
people
call
me
Mar
paya,
except
if
you're
on
Twitter,
because
Mar
piyo
is
already
taken
a
mic.
Our
paya
that's
relevant
because
there's
a
bunch
of
code
snippets
and
my
deck
that
you
probably
won't
be
able
to
read.
Although
this
screen
is
really
large,
so
maybe
you
will
be
able
to
read
them,
but
I
tweeted
out
my
deck
earlier.
A
It's
also
on
the
sched.
For
my
talk,
so
you
know
don't
worry
about
taking
photos
or
anything
you
can
find
all
of
that
on
the
Internet.
So
a
little
bit
about
me,
I'm,
the
co-founder
and
CTO
of
an
infrastructure
analytics
startup
called
collide.
I
used
to
work
at
Facebook,
Etsy,
ISEC
partners,
a
few
other
security
companies
doing
security
infrastructure,
stuff
I
was
also
on
the
kubernetes
release
team
for
111,
112
and
113.
Doing
release
notes
stuff
and
that's
a
that
was
awesome,
love
the
release.
A
Team
I
also
was
the
creator
of
an
open
source
tool
called
OS
query.
When
I
worked
at
Facebook,
some
of
you
may
be
into
operating
system.
Monitoring
may
be
familiar
with
us.
Crew,
I
love,
go
I,
really
good
developers,
I
like
to
say
I've
been
writing,
go
stands,
go
one
and
I'm
all
about
it,
and
I
played
the
bass
and
I
loved
skiing
and
climbing
and
hiking
and
trailering,
and
all
that
good
stuff.
So
that's
all
you
need
to
know
about
me
really
about
collide
collide
is
an
infrastructure
analytics
company
b2b.
We
basically
sell.
A
We
have
a
web
application
that
does
operating
system
analytics
for
our
customers.
The
principle
thing
that
you
need
to
know
about
collide
for
this
presentation
is
that
we
have
a
SAS
product.
We
sell
to
other
businesses,
and
the
data
that
we
collect
from
our
customers
is
the
most
sensitive
data
that
they
have.
So
security
is
super
important
to
us,
and
that
motivates
a
lot
of
the
decisions
that
we
make
around
data,
isolation
and
architecture.
So
keep
that
in
mind.
I
also
want
to
give
a
shout
out
to
the
two
sres
that
work
at
collide.
A
The
left
there
is
Victor-
and
you
may
know
him
as
Group
on
the
internet
and
on
the
right
is
SEF
with
an
S,
not
a
C,
and
you
may
know
him
on
the
Internet
as
directionless,
and
they
have
been
super
instrumental
in
this
whole
process
of
both
coming
up
with
all
of
these
ideas
with
me,
as
well
as
putting
them
into
production.
So
couldn't
have
done
any
of
this
without
those
two
guys.
So
for
the
talk
today,
I
want
to
talk
to
you.
A
I
want
to
discuss
the
few
hockey
concepts
and
the
multi-tenancy
space
and
make
sure
that
we
all
agree
on
some
shared
definitions.
I
want
to
give
you
a
quick
rundown
of
how
we
chose
to
deploy
our
application
and
manage
tenancy
at
collide
and
then
I
want
to
dive
deep
into
the
kubernetes
operator
ecosystem
and
how
we've
used
the
operator
to
accomplish
various
aspects
of
operating
a
production
system
such
as
deploying
that
system.
The
networking
involved
security
aspects
of
it.
A
What
have
you
and
then
I
have
a
few
links
to
various
community
projects
that,
hopefully,
you
may
be
interested
in
as
well
as
some
enumerations
of
lessons
learnt
lessons
learned,
interesting,
gotchas
and
other
things
that
you
might
want
to
take
away
from
this
presentation
about
whether
or
not
you
should
or
should
not
do
this
based
on?
You
know
the
different
properties
of
your
business.
So
let's
hop
right
into
it.
Some
multi-tenancy
concepts,
so
this
talks
all
about
application
tenancy,
and
that
is
the
de.
You
know.
A
Companies
that
create
products
for
other
companies
have
to
reason
about
the
isolation
between
the
different
companies,
and
there
are
two
complete
ends
of
the
spectrum
on
how
you
could
go
about
managing
application.
Tenancy
one
is
to
create
you
know
a
kind
of
monolithic
application
and
which
handles
tenant
data
isolation
via
application
logic,
middleware
stuff
like
that,
and
the
other
end
of
the
spectrum
is
to
deploy
and
proxy
to
many
independent,
isolated,
single
tenant
applications,
and
this
tall
kind
of
you
know
it's
choose-your-own-adventure.
A
We
chose
the
second
path
and
we
chose
to
manage
that
with
kubernetes
operator
and
that's
you
know
what
I'm
gonna
explain.
So
I
just
want
to
explain
the
spectrum
and
that
you
know
you
may
or
may
not
choose
that
path,
but
if
you
do-
and
you
want
to
manage
it
with
kubernetes
operator,
this
talks
for
you
and
I'll
talk
about
some
of
the
at
the
end.
I'll
talk
about
some
of
the
properties
that
you
know.
A
It
may
be
a
better
idea
for
you
to
consider
one
way
or
the
other
if
your
organization
and
has
different
kinds
of
properties,
so
within
the
kubernetes
space,
people
often
talk
about
hard
and
soft
multi-tenancy,
as
it
turns
and
I
think
it's
useful
to
understand
these
two
terms
so
Jessie
Frisell.
It
has
a
ton
of
interesting
documents
that
she's
our
and
she
used
to
be
the
lead
of
the
multi-tenancy
working
group
and
we're
a
ton
of
really
interesting
documents
on
this
topic
and
I.
A
Like
her
definitions
of
these
two
topics,
so
I'll
read
them
to
you.
Kubernetes
is
new
kernel.
We
can
refer
to
it
as
a
cluster
kernel
versus
a
typical
operating
system
kernel.
This
means
a
lot
of
great
things
for
users
trying
to
play
applications.
It
also
leads
to
a
lot
of
the
same
challenges
we've
already
faced
with
operating
system
kernels,
one
of
which
being
privilege
isolation
in
kubernetes.
We
refer
to
this
as
multi-tenancy
or
the
dream
of
being
able
to
isolate
tentative
clusters.
A
So
hard
multi-tenancy
is
really
this
idea
of
you
want
to
you
know
a
common
use
case
for
hard
multi-tenancy
might
be
if
you're
selling
kubernetes
like
if
you're
a
kubernetes
provider-
and
you
want
to
give
someone
a
control
plan
that
they
only
they
have
access
to.
But
you
perhaps
want
to
run
all
of
these
like
kubernetes
clusters
that
you're
distributing
in
kubernetes.
This
would
be
hard
multi-tenancy,
there's
no
shared
infrastructure
between
the
tenants.
The
other
is
soft
multi-tenancy.
Now
what
jesse
wrote
about
soft
multi-tenancy
is
multiple
users
within
the
same
organization.
A
In
the
same
cluster,
soft
multi-tenancy
could
have
possible
bad
actors
such
as
people
leaving
the
company.
The
users
are
not
thought
to
be
actively
malicious,
since
they
are
within
the
same
organization,
but
potential
for
accidents
or
evil
leaving
employees.
A
large
focus
of
soft
multi-tenancy
is
to
prevent
accidents.
I
think
that
last
point
is
really
important,
so
the
common
use
case
for
soft
multi-tenancy
is,
if
you're
a
cluster
operator
at
a
company-
and
you
know,
there's
a
lot
of
different
teams
that
each
wants
space
on
your
kubernetes
cluster.
A
This
is
a
soft
multi-tenancy
where
you
can
kind
of
trust
people,
but
you,
you
know,
want
to
put
some
guardrails
up,
so
we
think
about
hard
versus
soft
multi-tenancy.
It's
a
big
spectrum
and
there's
a
few
key
points
of
isolation
that
you
can
trigger,
and
then
everything
between
those
points
is
kind
of.
You
know
a
little
more
ethereal,
but
the
SAS
application
multi-tenancy
problem
space
is
firmly
soft
multi-tenancy.
A
So
you
know
we're
not
really
interested
in
super
concrete
isolation
between
the
different
tents,
because
we
not
we-
you
know
we're
ones
where
a
set
of
operators
that
wrote
all
the
software
that
we're
deploying
right.
So
we
trust
the
software.
We
trust
that
you
know
it's
not
like.
We,
we
don't
even
have
to
deal
with
employees
being
disgruntled.
We
can
trying
to
you,
know,
circumvent
our
protections
because
we're
not
giving
our
employees
unfettered
access
to
deploy
whatever
they
want,
our
clustered.
A
So
multi-tenancy
for
us
is
more
of
this
logical
operation
and
you
know
that's
something
to
keep
in
mind.
Another
topic.
That's
often
talked
about
multi-tenancy
that
I'm
not
going
to
talk
about
at
all.
Today
is
metering,
and
this
is
really
popular
in
choruses
operator
framework,
and
this
is
the
idea
of
you
know
controlling
how
many
resources
a
single
tenant
can
use
compared
to
another.
We
don't
do
any
meter
metering,
because
that
would
imply
that
you
know
you
know
we
want
to
kind
of.
A
You
know
limit
the
computational
resources
that
one
of
our
own
customers
can
use
via
our
own
software.
You
know
wouldn't
really
make
sense
for
yeast
kiss,
so
we
don't
do
any
metering,
but
there's
a
lot
of
really
interesting
metering
stuff
out
there.
If
that's
interesting
to
you,
so
keep
that
in
mind.
Alright,
cool
app
architecture,
so
I
want
to
talk
to
you
a
little
bit
about
the
chosen
architecture
that
we
have
for
this.
A
That
I'll
go
over
for
this
talk
because
I
think
it's
useful
to
understand
the
kind
of
general
goal
of
the
deployment
model
and
then
we'll
go
into
how
we
accomplish
that.
So
generally,
a
single
application
instance
per
customer,
so
there
is
various
components
to
the
application.
You
know
this
contrived
example
where
there's
just
three
components.
A
You
know
one
on
the
top
left
one
in
the
top
right,
one
in
the
bottom
and
all
of
the
boxes
represent
various
infrastructure
components,
but
the
key
here
is
that
each
compute
and
data
requirement
of
a
tenant
is
completely
isolated
into
its
own
box
and
one
of
the
few
advantages
to
this
few
disadvantages.
To
this
you
know
this
optimizes
data
and
compute
for
very
sensitive
use.
Cases
like
I
talked
about.
A
We
do
operating
system
instrumentation
and
analytics
for
security
reasons
for
our
customers,
so
this
is
very
sensitive
data,
so
we
don't
want
any
risk
of
cross-pollination
of
that
data,
and
this
also
eliminates
the
need
for
product
developers
to
reason
about
data,
isolation
and
multi-tenancy,
which
are
historically
kind
of
complicated
topics,
and
you
know
if
you
find
yourself
at
an
organization
that
has
a
lot
of
infrastructure
folks,
but
not
a
lot
of
product
developers.
This
might
be
a
compromise
that
you
can
make
to
offload
your
product
developers
from
having
to
reason
about
multi-tenancy.
A
If
you
can
make
it
an
infrastructure
problem
and
also
you
know
we're
startup,
we
just
we
did
our
series
a
like
a
year
and
a
half
ago,
so
we're
pretty
early-stage
still,
and
this
allows
us
to
have
our
infrastructure
infrastructure
orchestration
and
account
management
deployment,
tooling,
all
that
kind
of
outlast
any
products
that
we
want
to
try
out
and
spike.
Someone
wants
to
write
a
new
version
of
the
app
and
deploy
it
to
a
few
tenants.
We
can
do
that
because
it's
all
rather
decoupled,
so
you
know
a
few
a
few
interesting
considerations.
A
So
once
you
have
these
little
clusters
of
applications,
the
general
strategy
we
have
is
that
we
deploy
all
of
them
independently,
and
then
we
have
a
networking
layer
that
exposes
you
single,
unified
ingress
and
that
ingress
takes
care
of
routing
traffic
to
customers
based
on
some
per
request.
Identifier
and
I
can
go
more
into
that.
What
those
are
later
there's
like
a
networking
section
so
we'll
get
into
that.
That's
the
goal!
You
know,
we've
got
these
little
boxes.
These
boxes
are
full
of
application
components.
Each
tenant
gets
their
own
application
components.
A
We
ingress
at
a
unified
point
and
then
route
to
all
of
them
independently
and
note
that
you
know
we'll
get
to
this,
but
this
doesn't
always
it's
not
always
in
one
cluster
we've
reached.
We
like
reach
the
limit
of
what
we
could
handle
with
one
cluster
and
had
to
reason
about
doing
this
kind
of
multi
tenant
ingress
for
many
customers.
So
that's
its
own
interesting
consideration.
So
this
talk
is
really
about
managing
tenants
with
an
operator.
So
once
we
decided
on
this
architecture,
it
was
pretty,
and
you
know,
we've
been
using
kubernetes
for
a
while.
A
I
was
pretty.
You
know
pretty
obvious
path
that
we
should
go
down
well,
so
each
tenant
gets
its
own
kubernetes
namespace
and
all
the
compute
resources
live
there.
We
also
have
a
higher
level
control
plan
for
things
like
sign
up
routing
that
ingress
proxy.
That
I
talked
to
you
for
account
management.
A
You
know
deployment
server
stuff
like
that.
So
let's
take
a
quick
aside
and
talk
about
the
operator
ecosystem
for
those
of
you
who
either
are
not
super
familiar
with
operators
or
not
super
familiar
with
the
software
development
ecosystem
around
it.
So
kubernetes
operators
are
like
operator
is
this
kind
of
marketing
term
that
core
OS
made
up
a
few
years
ago,
I
like
to
define
it
concretely
as
a
combination
of
a
custom
resource
definition
and
a
controller
and
your
custom
resource
definition
represents
the
schema
of
your
well
I'll,
get
to
the
definition
of
each
of
them.
A
You
know,
kubernetes
out
operations
are
awesome,
so
yeah.
So
for
this
talk,
I'm
not
really
going
to
use
the
term
operator
anymore.
I'll
use,
C,
Rd
and
CR
to
reference
the
customer
resource
definitions
and
the
customer
resources
to
turn
controller.
To
represent
these
like
deployed
thing
in
the
cluster
that
interacts
with
the
API
server
and
takes
action
based
on
the
instances
of
the
custom
resource.
Alright
awesome,
so
CR
DS
are
really
awesome.
Cut.
A
The
CR
dapi
is
kind
of
a
you
know,
popular
topic
at
the
conference
this
year,
which
is
great
and
I
love
it,
because
the
source
of
truth
is
an
annotated,
go
struct
which
works
directly
with
all
of
the
kubernetes
api
machinery,
and
then
source
of
truth
is
a
ghost
struct
that
always
makes
for
a
great
day
and
Michael,
and
that's
awesome
and
it's
a
neat
standard
to
adhere
to.
Since
you
know,
everyone
kind
of
has
the
yeah
mole
problem.
A
You've
got
too
much
e
amyl,
you
don't
know
what
to
do
with
it,
but
at
least
you
know
if
somebody
figures
out
a
better
way
to
manage
all
the
yeah
Mille.
Everyone
can
use
it.
So
I
don't
have
a
great
yeah
Mille
solution
for
you
yet,
but
you
know
when
one
comes
out:
I'm
sure
we'll
all
benefit
from
it.
Now
the
difference
between
custom
resource
definition
and
a
custom
resource
is
also
one
of
interesting
note.
So
custom
resource
definitions
are
like
the
schema,
the
specification.
A
What
fields
you're
allowed
to
set
on
this
object
and
a
custom
resource?
Is
you
know
it's
just
a
resource?
It's
one
instance
of
the
custom
resource
definition
and
I.
Think
it's
worth
being
explicit
about
using
CR,
D
and
C
are
appropriately
when
you
know
when
you're
talking
about
the
different
things,
whether
you're
talking
about
the
schema
of
the
resources
or
a
specific
resource.
A
So
this
is
great
because
you
know
we
call
our
custom
resource
definition
tenant
and
you
can
get
all
of
your
custom
resources
by
doing
things
like
ooh
cuddle,
get
tenants
and
stuff
like
that.
So
you
know
sometimes,
like
I
said
it's
parameterised
via
things
like
helm
in
case
on
it
and
I
use
helm
because
I
don't
like
the
tiller,
but
you
know
your
mileage
may
vary
so
this
is
kind
of
this
is
I.
A
Think
the
only
place
actually
in
the
talk
where
the
custom
research
is
shown,
but
this
is
a
you
know,
just
a
rough
example
of
the
kinds
of
things
we
have
in
here
for
each
tenant.
So
for
you
know
we
have
the
name
of
the
tenant,
which
are
just
these
like
random
strings.
We
have
a
few
to
bases.
So
if
you
want
like
to
my
sequel
databases
in
a
post-grad
database
for
one
tenant,
you
can
enumerate
those
there
there's
some
high-level
aspects
of
the
like
account
management
bits.
A
So
when
a
user
signs
up,
we
create
the
tenant.
We
include
these
account
management
bits
so
that
we
can
invite
them
or
send
them
a
email
saying
their
instance
is
up
and
running
when
when
it's
ready,
so
you
know
it's
useful
to
have
all
that
information
there.
And
then
we
have
some
random
information
like
how
large
the
PG
bouncer
pool
size
should
be.
And
you
know
these
are
things
that
we've
just
kind
of
iterated
on
and
then
the
main
bit
the
repos
bit
is
all
of
the
different
micro
services
that
make
up
your
are
tenant.
A
So
you
know
there's
only
two
listed
here,
but
in
practice
there's
several
more
on
each
of
our
tenants.
So
you
know
that's
that's
in
there
and
then
you
know
that
we
basically
specify
the
container
that
gets
deployed
and
then
some
parameterize
diamo,
which
we
have
like
kind
of
a
helm,
esque
templating
engine
that
we
use.
That's
like
helm,
except
without
the
tiller,
so
controllers,
so
controllers
are
the
other
side
of
customers
or
the
operator
pattern.
A
This
operator
off
of
the
sample
controller,
the
you
know,
github.com
such
kubernetes
slash
sample
controller,
which
is
produced
by
ad
API
machinery,
and
it's
a
little
more.
You
know.
There's
we
were
at
this
before
the
operator
framework
was
around.
If
I
could
go
back
and
do
it
and
if
the
operator
framework
was
around
when
we
started
the
project
probably
would
have
used
it,
but
I
don't
regret
not
using
it,
because
now
that
we've
used
sample
controller,
we
just
have
a
lot
more
control.
So
I
would
say
if
you're,
not
afraid
of
dealing
with
the
kubernetes.
A
The
API
is
very
deeply
then
go
for.
You
know:
go
for
the
metal
try
to
get
as
close
to
the
API
machinery
as
possible,
but
you
know
if
you're
a
new
do
to
kubernetes,
but
you,
like
writing,
go
the
operator
framework
is
definitely
more
user-friendly.
Api,
but
I
encourage
you
to
get
familiar
with
the
kubernetes
api
is
and
go
if
you're
a
good
developer,
because
once
you
learn
the
general
patterns,
they're
all
very
consistent
and
it's
really
easy
to
work
with.
So
you
know
go
for
it.
A
So
I'm
going
to
show
you
a
few
code,
examples
of
our
controller
just
to
show
you
how
we
do
so
just
to
like
show
you,
the
general
life
cycle
of
you
know,
controller
starts.
How
do
we
do
initialization
and
how
do
we
get
into
this
control
loop?
Where
tenants
are
managed
so
and
like
I
said
you
know,
you're,
probably
not
gonna,
be
able
to
read
most
of
this
from
your
seats.
That's
fine
I
just
want
to
exhibit.
Like
you
know,
the
smaller.
A
The
text
is
the
more
its
kind
of
on
purpose,
because
you
know
the
more
kind
of
unfortunate
initialization
you
have
to
do
so.
This
is
the
main.
So
when
you're
starting
up
when
you're,
creating
all
of
your
kubernetes
API
clients
and
stuff,
you
have
to
get
all
your
cube
configs
and
you
have
to
create
all
of
your
informer
factories
and
your
factory
factories
and
your
factory
factory
factories.
A
And
then,
at
the
end
of
the
day,
you
can
start
a
run
with
and
that's
at
the
bottom,
you
see,
C,
dot,
run
and
you're
running
your
controller,
so
basically
initialize
all
your
kubernetes
api
dependencies,
your
kubernetes
api
clients
for
all
of
the
core
types,
as
well
as
all
of
the
types
that
your
custom
resource
definition
is
adding.
Once
we
get
into
the
initialization
of
the
controller
itself,
when
we
initialize
new
controller,
you
know
it
accepts
to
all
of
those
initialized
bits
from
main
and
then
further
initializes
Informers.
A
You
know
you
can
watch
objects
as
well
as
have
an
in-memory
cache
of
objects
that
the
api
server
knows
about,
so
that
you
don't
have
to
hit
the
API
server
every
time
you
want
to
list
the
namespaces
in
your
cluster
or
something
like
that.
So
for
a
very,
very
slight
memory
hit,
you
can
have
all
of
this
stuff
cached
locally,
which
is
really
great.
So
if
you're
listing
out
deployments
or
secrets
or
namespaces
or
pod
disruption,
budgets
or
any
of
the
other
stuff
that
you
invariably
will
need
to
operate
on.
A
Creating
all
these
informers
is
really
nice,
so
you
can
see
there's
like
namespace
secret
deployment,
stateful
set
services,
pod
structured
budgets
and
then
towards
the
bottom.
There
there's
the
tenant
Informer.
So
that's
our
kind
of
API
object
for
the
tenant
and
it's
great
because
all
of
the
informer
factories
and
all
of
that
get
auto
generated
by
the
code
that
is
part
of
the
upstream
API
machinery,
so
there's
really
great
code
generation
for
that
now
you
know
this
is
where
we
set
up
the
event
handler.
A
But
this
is
just
to
exhibit
that
there
is
a
you
know:
function,
there's
a
function
on
the
controller
struct
called
in
queue
tenant,
which
will
add
the
tenant
to
the
work
queue
which
is
internally
managed
now
synchronization.
This
is
where
them.
This
is
where
the
magic
happens
right.
This
was
MTV
Cribs,
it
would
be
the
bedrooms,
but
so
there's
this
function.
We
have
called
resolve
tenant
state
and
this
basically
accepts
a
key
in
this
keys,
the
name
of
the
tenant.
A
So
then
we
use
the
tenant
and
former
factory
to
get
the
full
typed
odd,
the
full
type
struct.
For
that
tank
key
and
based
on
the
state
of
the
tenant
which
I'll
explain.
We
then
you
know,
resolve
it
in
a
certain
way.
So
when
we
first
created
this,
we
would
just
resolve
tenants
in
one,
but
then
we
started
to
get
a
lot
of
customers.
Some
customers
churn
out,
and
you
know
you
don't
want
to
keep
their
tenant
around
or
you
want
to
keep
it
in
kind
of
this
idle
state.
A
So
we
added
a
bunch
of
states
to
tenon.
So
that's
another
thing.
We
added
to
our
customer
resource
definition
and
we
have
different
kind
of
synchronization
loops
for
each
for
each
one.
So
I'm
not
going
to
go
into
each
of
the
different
resolutions
for
each
of
the
different
states,
but
you
can
see
the
states
there
are
frozen,
archived
purged
and
then
we
have
a
general.
You
know
catch-all
for
things
like
whether
they're
idle
or
new,
or
anything
like
that
they're
just
going
to
we're
just
gonna,
make
sure
that
they're
fully
deployed
and
healthy.
A
So
you
know
this
is
the
general
life
cycle.
We
instantiate
our
kubernetes
API
clients.
We
create
our
informers,
we
instantiate
it
controller.
We
have
a
little
run
loop,
that's
provided
by
the
API
machinery,
and
then
you
know,
based
on
the
state
of
the
tenant,
we'll
take
some
different
actions
to
ensure
that
the
tenant
is
deployed
properly
right.
A
So
that's
all
well
and
good.
Now,
I
want
to
talk
about
I
want
to
go
through
the
you
know,
aspects
of
production
system,
part
where
we
talk
about
deployments,
networking
security
stuff
like
that
so
deployments
so
to
salute
a
frequent
deployments.
We
like
everyone
else
like
to
deploy
as
quickly
as
possible.
You
know,
because
it's
2018
and
who
doesn't
like
to
deploy
several
times
a
day.
So
we
were
a
slack
bot
that
interacts
with
the
kubernetes
api
that
allows
you
to.
You
know,
make
certain
changes
to
tenon
so
like.
A
If
you
leave
it
off,
it
deploys
it
to
everyone.
So
the
slack
bot
here
is
kind
of
interacting
with
all
the
different.
These
API
servers
for
all
the
different
clusters
where
tenants
may
exist
and
basically
doing
like
a
poor-man's
apply,
which
is
interesting.
I'll
talk
about
I'll,
talk
more
about
a
server
side,
apply
and
kind
of
updating
in
the
perils
therein,
but
that's
really
fun,
that's
cool,
so
this
is
kind
of
again.
This
is
something
that
you
can
look
at
in
the
slides
for
reference
later
if
you
want
to,
but
this
is
basically
going
through.
A
A
Then
we
enumerate
through
those
tenants,
do
the
update
to
them,
based
on
whatever
hash,
we're
trying
to
deploy
and
then
do
a
an
update
operation
with
the
kubernetes
api,
so
that
now
the
tenant
is
changed
and
it
will
be
again
queued
in
the
controller.
So
that's
kind
of
the
flow
you
write
a
slack
command.
The
slack
bot
interacts
with
the
kubernetes
api
updates,
the
tenant
custom
resources.
The
controller
has
an
informer
factory.
It
observes
that
these
updates
have
happened
and
takes
the
action
to
converge,
actual
state
with
desired
state.
That's
the
place,
that's
it!
Yes,.
A
If
there,
the
question
is:
is
there
a
reason
why
we
do
this
and
go
instead
of
just
using
coop,
go
and
coop
cuttle
applying
yeah?
We
don't
actually
keep
any
of
the
tenant
animals
yeah
they're,
all
just
we
operate
on
weed
yeah.
We
don't
have
like
a
repository
of
the
animals
or
anything
we
just
reason
about
them
as
API
objects,
yeah,
networking
new
topic,
so
the
most
interesting
thing
that
we
do
with
networking
is
this
ingress
bit
so
for
any
kind
of
tenant?
A
In
sharding
scenario,
you
need
to
central
index
for
managing
external
ingress
traffic
and
there
are
a
ton
of
options
for
this
envoy.
You
can
script
with
nginx,
there's
a
really
great
talk
from
Shopify
des
or
Recon
about
this
for
our
needs.
We,
like
writing
everything
and
go.
We
were
at
a
whole
custom,
ingress
proxy
and
go
that
does
exactly
what
we
need
and
nothing
that
we
don't.
A
We
like
our
custom
proxy,
although
we
may
reevaluate
that
as
different
options
become
more
mature,
so
a
few
details
about
this,
this
proxy,
it's
all
l4,
so
it
makes
it
super
easy.
We
just
you
know,
terminate
you
know
we
l4
directly
to
the
pods
for
each
tenant,
so
each
application
is
a
ghost
service
that
is
totally
willing
and
capable
and
able
to
terminate
a
tail
force.
So
that
makes
things
a
lot
easier.
A
It's
called
shuffler
it's
similar
to
envoy
with
regards
to
the
fact
that
it
separates
the
control
plane
from
the
data
plan,
so
we
have
a
G
RPC
API,
where
we
can
set
mappings
where
we
can
say
hey
a
new
tenant
signed
up.
We
deployed
them
in
this
cluster,
so
create
a
mapping
so
that
when
you
see
traffic
for
this
tenant
name,
it
needs
to
go
to
this
kubernetes
namespace.
So
we
use
GCP
datastore
to
persist
that
GCP
datastore
effectively
scales
forever
for
this
use
case.
A
So
it's
really
great
and
then
the
data
plane
just
kind
of
gets
that
keeps
an
in-memory
cache
of
that
information,
and
you
know
just
on-the-fly
makes
a
decision
about
where
this
request
needs
to
go
based
on
the
tenant.
Id,
that's
in
an
HTTP
header,
and
this
adds
about
one
millisecond
of
latency,
which
you
know
isn't
awesome,
but
is
definitely
not
our
bottleneck.
So
so
we
don't
really
worry
too
much
about
that
in
training
space
traffic
traffic
within
the
namespace
is
interesting.
A
So
for
per
tenant
service,
the
service
traffic
we
just
used,
the
you
know
you
can
use
just
for
example,
herd
is
one
thing
that
is
in
each
tenant.
You
can
just
refer
to
it
via
the
DNS
herd.
This
is
actually
pretty
convenient
because
it
allows
you
to
communicate
intra
tenant
services
without
having
to
do
any
fancy
service
routing,
because
any
every
tenant
that
you
know
needs
to
talk
to
its
herd
or
its
K
store
or
whatever
it's
like.
You
know,
service
that
needs
to
talk
to
you
for
a
tenant.
A
It
can
talk
to
it
via
the
same
name
and
you
can
use
network
policies
which
are
a
part
of
court
to
lock
down
this
network
graph
depending
on
your
use
case,
and
we
do
for
ours.
So
that's
an
interesting
little
tidbit
now
security.
Now
this
is
an
interesting
one
for
sure
this
is
kind
of
interesting
from
a
few
perspectives.
A
One
of
the
things
that
I
think
is
the
most
interesting
and
the
thing
that
come
from
not
cause
us
the
most
pain,
but
that
we
spent
a
lot
of
time
kind
of
reasoning
about
and
discussing
and
optimizing
is
secret
distribution.
So
each
tenant
needs
a
variety
of
Secrets
API
keys
for
things
like
SendGrid
or
what-have-you,
as
well
as
the
database
credentials
for
the
per
tenant.
You
know
data
resources
that
it
has
and
what
have
you
and
you
know
if
you've
used
kubernetes
before
you
likely
know
that
kubernetes
secrets
cannot
be
shared
across
namespaces.
A
A
So
the
way
we
do
this
is
we
have
tooling
in
the
control
in
the
run
loop
and
the
synchronization
loop.
In
the
controller
that
when
we
go
to
deploy
a
tenant,
they
will
look
you've
like
enumerate
the
secrets
that
are
in
that
namespace
via
the
share
former
that
we
have
for
kubernetes
secrets,
observe.
What's
there
observed
what
should
be
there
and
make
any
changes
necessary
right?
So
this
we,
since
we
deploy
multiple
clusters,
we
can't
use
like
a
single,
shared
namespace
or
anything
as
a
source
of
truth.
A
So
we
have
to
have
some
EXO
cluster
secret
storage
mechanism,
where
we
can
pull
these
secrets
from
so
for
our
East
case.
We
have
this
little
interface
that
we
made
called
store,
which
just
is
get
list
put
delete
and
it
actually
operates
on
the
core
v1
secret
objects,
which
is
really
nice.
We
use
kubernetes
for
everything,
there's
already
a
typed
object
for
secrets,
so
we
have
this
very
minimal
interface
for
accessing
secrets
from
an
external
source,
and
you
know
this
is
what
the
controller
uses
to
get
secrets
from
that
source.
A
Now,
for
the
actual
implementation
of
this
interface,
we
use
KMS
and
GCS,
which
are
GCP
products.
Basically,
we
don't
use
vaults
because
auto
unseal
isn't
free,
so
we're
already
really
heavy
users
of
GCP,
so
KMS
is
basically
a
way
for
you
to
encrypt
very
small
data.
You
know
it's
just
an
API.
You
know
it
has
like
an
encrypt
and
decrypt
RPC
that
you
can
call,
and
then
we
store
the
encrypted
secrets
in
GCS.
A
So
whenever
we
need
to
get
a
secret
that
we'll
go,
get
the
encrypted
object
from
GCS
decrypted
from
with
kms,
and
then
you
know,
we
operate
in
code
with
the
core
v1
secret,
even
though
at
rest,
they're
encrypted
via
keys
that
we
can
rotate
and
manage
permissions
to
with
I
am
kms
is
pretty
cool
product.
It's
got.
Some
got
some
gripes
that,
though
we
can
talk
about
that.
A
After,
if
you
want,
if
you're
into
synchronization
loop
performance
is
really
interesting
when
it
comes
to
secrets
because
make
in
you
know
and
your
happy
path,
your
synchronization
loop
happens
really
quickly
like
if
you're
just
you
know,
maybe
updating
the
size
of
the
PG
bouncer
configuration
or
what-have-you.
Making
these
very
my
new
changes.
You
don't
want
the
run
Loup
for
that
tenant
to
take
very
long
right.
So
another
reason:
why
is
when
you
start
up
the
controller?
A
A
So
if
nothing's
changed,
you
want
it
to
be
able
to
just
work
through
all
of
the
you
know,
all
of
the
items
that
it
has
to
process
as
quickly
as
possible,
and
this
process
for
us
was
almost
completely
a
practice
of
seeing
you
know
how
efficiently
we
could
manage
the
dis,
acure
distribution
of
secrets.
So
a
lot
of
interesting
optimizations
here,
but
generally
this
you
know
we
really
had
to
optimize
out
making
any
calls
to
the
kubernetes
api
server
and
the
hot
path
of
synchronization.
A
So
originally,
when
we
started
doing
this
every
time
we
synchronized
a
tenant,
we
would
list
all
of
the
secrets
that
were
encrypted
in
GCS.
We
would
make
a
get
for
each
item
in
a
folder.
In
GCS
we
would
make
an
API
call
to
kms
to
decrypt
it.
We
would
list
all
the
secrets
in
the
namespace.
We
would
compare
all
of
them
and
then
you
know
make
the
changes
we
had.
So
we
made
like
you,
know:
40
RPC
calls
to
Google
products.
40
RPC
calls
so
the
kubernetes
api
for
every
10.
A
It
was
super
slow,
took
like
several
seconds
to
synchronize
110.
If
nothing
had
changed
and
at
the
end
of
all
of
the
optimizations
that
we
made,
we
basically
pre-staged
the
shared
secrets
into
a
kubernetes
namespace
and
used
shared
informers
to
enlist
that
namespace.
We
would
use
a
shared
Informer
to
in
memory
list.
The
tenant
namespace
compare
all
those
and
then
we
would
only
use
the
GCS
storage
for
per
tenant,
specific
secrets,
and
that
got
that
basically
made
this.
A
You
know
run
loop
performance
and
non-issue
so
you
know
be
warned
that
was
a
big
deal
and
it
was
a
pretty
easy
fix.
But
it's
thing
that
you
have
to
pay
attention
to
for
sure
when
you're
interacting
with
kubernetes
api-
and
you
know
something
that
could
be
running-
you
know
highly
concurrently
as
these
work
queues.
Often
can
occur
so
all
right,
I'm
gonna,
skip
to
that
stuff.
We've
learned
yeah,
so,
like
I
said
with
regards
to
performance.
A
You
know
it
was
very
much
a
practice
of
optimizing
the
the
speed
at
which
the
controller
could
work
through
one
tenant
when
minimal
things
had
changed.
When
we
reach
about
a
thousand
tenants,
the
cluster
started
falling
apart,
miraculously
in
a
number
of
different
ways,
so
we
kind
of
pointed
at
that
and
said:
alright,
that's
the
most.
That's
the
like
max
number
of
tenants
we're
going
to
deploy
in
a
cluster.
We
started
scaling
to
more
clusters,
so
you
know
keep
in
mind.
This
number
may
be
different
for
you,
but
that's
where
it
was
for
us.
A
A
Obviously
so,
in
order
to
so
I'll
take
that
we
started
creating,
we
added
a
new
state
of
tenants
which
were
unallocated
tenants,
so
we
would
have
kind
of
a
worker
keeping
a
pool
of
like
10
or
20
unallocated
tenants,
and
then,
when
someone
signed
up,
we
would
just
allocate
that
tenant
and
drop
them
instantly
into
it,
so
that
they
didn't
have
to
wait,
for
you
know
getting
an
email
and
having
to
go
back
into
the
application
so
that
they
could
convert.
They
would
just
sign
up
and
immediately
be
dropped
into
a
user
experience.
A
You
know
so
that's
possible
to
do
and
as
any
product
person
will
tell
you,
this
was
super
important.
We
did
it
really
early
and
I
encourage
you
to
as
well.
If
you
go
down
this
path,
another
thing
we
learned
is
health
checking.
So
initially,
when
we
were
doing
this,
we
would
go
through
the
in
table.
We'd
like
synchronize
a
tenant
and
make
all
of
the
we
would
like.
You
know,
create
a
deployment.
We
would
create
a
service.
A
We
would
create
all
the
API
resources
by
interacting
with
the
kubernetes
api,
and
then
we
would
just
be
like.
Ok,
we're
done
and
move
on
to
the
next
that
wound
up
overloading
the
kubernetes
api,
because
we
would
basically
go
through
all
of
the
tenants
make
all
of
these
operations
and
then
all
of
these
things
would
be
pending
and
the
kubernetes
api
server
would
be
trying
to
interact
with
all
of
the
different
cubelets
and
it
was.
It
was
a
pain.
A
So
you
know
we
can
adjust
the
parallelism
of
the
work
queue,
and
that
is
the
total
number
of
tenants
that
will
ever
be
operated
on
at
one
time.
That's
interesting!
So,
just
to
sum
things
up,
situations
where
this
architecture
works
really
well
is,
if
you
have
a
relatively
small
number
of
relatively
large
customers
that
have
relatively
unique
resource
requirements
where
you
can
really
take
advantage
of
your
ability
to
fine
tune
each
different
tenant.
A
A
That's
awesome
situations
where
the
starchitect
doesn't
really
work
out.
That
well,
is,
if
you
have
a
very
large
number
of
very
small
tenants,
because
the
thing
that
you're
gonna
optimize
is
the
minimal
cost,
like
you're
gonna,
be
able
to
compact
a
single
instance
of
your
application
as
small
as
possible,
and
that's
great
because
it
gives
you
a
single
point
where
you
can
be
like
okay.
Well,
we
know
one
tenant
costs
us
a
dollar
a
month
to
run
so
we
need
to
make
that
30
cents
and
you
know
then
we'll
be
mega
profitable.
A
But
if
you
have
just
a
ton
of
customers
that
are
so
small
that
you
need
to
just
run.
So
many
of
these
tenants
than
the
economics
of
that
probably
just
won't
really
work
out
for
you.
If
your
data
security
requirements
aren't
very
stringent,
the
juice
might
not
be
worth
the
school.
You
know
this
is
very
much
a
like
data,
isolation,
security
effort.
So
if
you
don't
really
care
about
that,
this
might
also
might
not
be
a
great
move
for
you
and,
if
you
don't
like
writing,
go
you
probably
won't
like
interacting
with
the
kubernetes
api.
A
So
keep
that
in
mind,
like
I,
said
general
benefits
of
this
approach
is
that
you
get
really
reliable.
10G
isolation.
Outages
in
a
component
of
a
tenant
are
generally
very
confined
to
one
tenant,
so
you
know
if
you're
you
know,
back-end
goes
down
for
because
of
some
random
thing
that's
happening.
It
generally
only
affects
that
one
customer
which
is
really
nice.
A
A
So
if
your
SRE
team
is
full
of
people
that
don't
like
writing,
go
or
and
don't
like
you
know,
everything
that
you
do
for
a
tenant
needs
to
be
scaled
across
all
of
your
tenants.
So
you
need
to
incorporate
all
of
these
kind
of
individual
tasks
into
some
mechanism
in
to
go
software.
So
if
your
SRE
team
is
really
down
to
completely
manage
operations
as
a
go
software
development
project,
then,
is
awesome.
If
that's
not
what
you're
looking
to
do.
A
Don't
do
this
for
sure,
because
that's
for
sure
what
you're
gonna
be
doing,
the
other
downside,
I'd
say
the
big
downside
of
this.
Is
most
people
don't
bend
kubernetes
this
way
they
don't
try
to
run
a
lot
of
really
small
tenants
or
a
lot
of
really
small
namespaces.
Rather,
you
know,
kubernetes
says
that
the
limit
of
pods
on
a
node
is
110.
If
you
try
and
pack
as
many
reach
the
pods
as
possible
onto
a
tenon,
and
you
start
approaching
that
number
things
start
to
hang
up
and
get
really
sketchy
in
really
interesting
ways.
A
Also,
once
you
scale
past
one
cluster
custom,
resource
definitions,
don't
really
work
well,
it
allowing
you,
articulate
multi
cluster
topics
or
ideas
that
span
clusters-
and
you
know,
obviously
observability
is
a
constant
challenge
and
the
fact
that
there
is
no
server-side
apply
yet
makes
a
lot
of
these
kinds
of
operations
somewhat
unsafe.
So
there's
a
multi-tenancy
working
group,
that's
doing
really
interesting
stuff,
that's
led
by
david
Oppenheimer,
who
was
one
of
the
people
who
was
wrote.
A
The
kubernetes
research
paper,
which
is
awesome
so
he's
written
a
lot
of
great
documents,
as
is
Jesse
on
the
concepts
with
multi-tenancy
that
are,
you
know,
super
worth
a
read:
there's
also
a
new
lead
of
that
working
group
named
Tasha
who
I
haven't
met
but
I'm
looking
forward
to
working
with.
You
know,
there's
a
lot
of
great
stuff
going
on
in
this
working
group,
but
they're
mostly
focused
on
establishing
definitions,
and
you
know
working
groups
aren't
long-lived
and
they
don't
produce
code.
A
So
you
know,
based
on
what
you
like
to
contribute
that
may
or
may
not
be
something
that
you
want
to
get
involved
with,
and
the
other
thing
is
lesser,
which
is
kind
of
a
shell.
I
started
working
on
a
version
of
this
that
is
open
source
and
it's
basically
just
a
shell
of
a
controller
for
me
to
work
on
open
source.
A
You
know
multi-tenancy
concepts.
So
if
you
are
interested
in
working
on
something
like
this
or
you
want
something
like
this
in
your
organization
and
you
want
to
work
together
on
it,
let
me
know-
and
maybe
we
can
make
something-
that's
a
little
more
generic
for
your
use
case.
I,
don't
think
I
have
time
for
questions,
but
that's
all
my
contact
information
and
thank
you
all
for
sticking
with
me.