►
Description
Evolving and Supporting Stateful, Multi-Tenant Decisioning Applications in Production [A] - Keith Gasser, Capital One
With our adoption of Kubernetes at Capital One, we have simultaneously reduced our application delivery time-to-market while providing a common platform for streaming pipelines. We leverage Kubernetes to manage stateful decisioning applications for multiple tenants and provide a host of analytical tools as platform services to help data scientists iteratively improve decision models. We will discuss the challenges in operating these pipelines which consist of Apache Nifi canvases/flows for data ingress/egress, Kafka as persistent stream backbone, Flink for decisioning, and a number of other popular open source data analytics packages such as Apache Drill and Zeppelin forming our “Analytical Environment.”
About Keith Gasser
Keith is a Software Engineer specializing in DevOps and Application Security at Capital One currently working on a team which has built a Kubernetes-based streaming and decisioning pipeline for Capital One Bank.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
And
thanks
for
sticking
through
this
with
us,
I,
don't
know,
I
know
my
head's
exploding
who's
having
a
who's
had
a
least
a
few
moments
of
clarity.
There
thanks
a
lot
for
coming
out
we're
going
to
talk
today
about
some
some
of
things
you
might
have
already
seen
in
some
of
the
other
tracks,
so
I've
peeled
some
things
out,
I'm
going
to
give
shout
outs
to
folks.
Who've
already
done
exemplary
jobs
in
this
space,
especially
in
the
ML
track.
A
We're
going
to
talk
about
applied
form
the
kubernetes
platform
that
we've
built
the
support,
decisioning
and
an
fi,
our
players,
capital,
one
I'm
Keith.
This
is
Bryce.
This
is
Gavin
and
we'll
carve
out
different
pieces.
Our
colleague,
Ravi
Dube,
unfortunately
couldn't
be
here
today,
he's
a
little
bit
allergic
to
the
cold,
so
Robbie
Robbie
does
not
like
the
cold.
We
really
enjoyed
seeing
snow
in
Austin.
Yesterday
I
was
so.
A
It
felt
like
a
bucket
list
item,
so
I'm
going
to
talk
a
little
bit
about
our
cluster,
that
we've
build
we're
sort
of
the
platform
and
the
site,
reliability,
engineers
for
the
platform
and
some
of
what
our
tenant
workloads
look
like.
The
kinds
of
support
we
provide,
we
really
come
from
the
software
engineering
side,
but
building
kind
of
cross
stack
capabilities
in
the
decisioning
and
m/l
space,
but
don't
hold
us
to
to
deep
capabilities
in
ml,
we're
just
getting
there
via
decisioning.
A
At
this
point,
so
I'm
going
to
talk
about
some
of
our
overall
mission
and
Bryce
is
going
to
talk
to
cluster
installation
and
an
ops,
some
of
the
nuances
of
of
our
platform,
engineering
and
site
s,
re-engineering
requirements
for
working
for
an
fi
and
Gavin's
going
to
talk
to
our
sort
of
encapsulation
and
provisioning.
Better
multi-tenancy
experiences
customer
experiences
for
what
we
view
as
a
pass
for
decisioning.
A
A
We
are
currently
deployed
production
to
single
region,
multi
AZ,
homogeneous
node
types.
We
use
em
for
ten
excels,
roughly
just
under
ten
of
them
to
support
the
workload
type
Sylas
below
one
interesting
thing
that
everybody
at
the
firm,
every
tech
team
platform
team,
app
team
at
the
firm
has
to
go
through,
is
a
sixty
day,
complete
rehydration
from
the
a
me
on
up
for
their
application.
A
Everyone
hates
it,
but
it
has
I,
have
to
say
it's
probably
instilled
quite
a
lot
of
discipline
for
resiliency
patching,
etc.
So
we,
you
know
you
you
get
very
quickly
to
a
point
where
you
can
rehydrate
your
entire
environment
and
then
that
supports
accelerated
upgrades
better
resiliency.
So
a
lot
of
a
lot
of
benefits
have
been
paid.
If
you
ever
want
to
beat
yourself
with
that.
Stick,
please
use
the
patterns.
We've
we've
outlined
here
we
as
much
as
we
hate
them.
We
we
we
also
it's
a
love-hate
thing.
A
So
the
types
of
workloads
we
run
are
two
types
that
are
domain-specific
workloads
on
the
cluster
roof,
one
for
real
time.
What's
real
time
are
kind
of
OLTP
stream,
decisioning
streams,
decisioning
for
transaction
streams
for
a
retail
bank,
and
that's
approximately
like
we
have
one
app
that
handles
around
600:
are
six
million
transactions
a
day
between
six
and
ten
starting
to
get
noisy?
A
Now
that
it's
the
retail
season
and
we
store
quite
a
bit
of
we
retain
180
days
of
data
which
you'll
see
as
we
get
in
the
fact
that
we're
streaming
with
kafka
is
a
little
bit
of
an
unusual
pattern
for
Kafka,
but
most
people
don't
retain
that
much
data
when
they
work
with
Kafka.
So
we've
had
that
requirement
and
that's
been
a
little
bit
unusual
for
for
our
installation.
A
We
run
batch
based
model
refits.
So,
as
I
said,
we're
decisioning
on
the
edge
of
ML,
we
do
run
model
POJO
models
that
I've
exported
from
h2o
and
you'll,
see
where
those
come
in
so
then
we'd.
You
use
also
use
pachyderm
for
refitting
those
models
and
to
be
able
and
using
batches
of
various
versions
of
data,
to
replay
those
models
where
you
test
those
models
to
improve,
mature
and
compete.
Those
models,
our
other
set
of
workloads,
are
ad
hoc.
A
Analytical
queries
for
data
analysts
and
data
scientists
and
our
final
set
of
workloads,
which
probably
everybody
has
our
telemetry
stacks
for
logging
monitoring,
cluster
services,
housekeeping
jobs,
the
various
Cron's
halves.
You
run
just
backup,
state
and
so
forth.
So
those
array
out
in
the
following
way
in
our
t1
and
t2,
that
is
all
stateful
and
all
multi-tenant.
A
We
were
sort
of
given
raison
d'être
from
the
fraud
budget,
but
we
have
expanded
the
decisioning
capability
and
offered
the
decision
capability
across
the
bank.
Our
knife
eye
workloads,
managed
ETL
into
and
out
of
the
platform.
Kafka
is
our
backbone
and
actually
has
become
the
kind
of
cross
service
bus
for
reefer
teams
who
are
refactoring
their
fling
caps.
A
All
kinds
of
interesting
mvw
sort
of
plat
pattern
work,
that's
going
on
in
the
flink
space
once
you
get
to
breaking
up
your
fling
cap
into
into
a
richer
ecosystem
of
sub
applications
for
determining
a
decision
and
then
visualizing
the
decision
querying
state
against
the
decision.
Interim
Kafka
topics
to
support
all
of
those.
Those
interim
states
pachyderm
as
dan
at
from
pachyderm
yesterday
talked
about,
is
a
tool
unique
to
kubernetes,
which
allows
for
state
versioning
of
data
and
then
triggering
there
of
triggering
model
pipelines,
model
refit
pipelines
and
invent
pipelines
using
batches
and
micro.
Batches.
A
The
analytical
environment,
Apache
and
drill
Apache
provides
a
notebook
capability,
just
like
jupiter
and
drill
provides
the
connectors
to
various
data
sources,
you'll
see
in
our
case
s3
Lake,
Aurora
Postgres
or
my
sequel,
various
stateful
sources
for
accessing
data
and
then
drill
bringing
the
data
together
and
then
Zeppelin
presenting
the
data.
And
then
we
we
have
the
standard
sort
of
standard
as
I've
seen
across
so
many
implementations,
logging
or
telemetry
stacks.
A
So
in
the
FK
share,
DFK
stack
across
our
environment
and
shared
metric
stack,
the
Higgs
stack
keeps
her
in
flux
and
Ravana,
and
then
we
run
because
we've
recognized
that
you
can't
just
run
streaming
services
without
running
some
kind
of
request,
response
services
and
sto.
The
service
do
service
has
been
so
helpful
in
that
regard.
We
began
running
sto
based
services.
A
So
this
the
orientation
a
little
weird
on
the
screen,
so
we
have
data
inbound
from
various
sources
to
knife
I
and
from
lambda
directly
into
our
Kafka
topics.
So
we
have
ingress
and
egress
topic.
Kafka
topics
was
established,
this
sort
of
black
box
for
the
platform,
the
decisioning
platform,
but
the
Kafka
topics
also
run
horizontally
across
fling
caps.
As
I
mentioned
for
interim.
A
Won't
spend
too
much
time
on
the
on
the
t2
types.
If
you
work
happen
to
catch
Nick's
presentation
yesterday
from
pachyderm,
it
was
excellent,
download
the
deck.
We
really
are
the
capabilities
that
we've
definitely
shout
out
to
this
community
they're.
Doing
such
interesting
work
for
event-driven
versioned
datasets
that
help
in
any
sort
of
you
don't
to
be
miss
doing
machine
learning.
You
don't
even
have
to
be
doing
decisioning
just
for
doing
various
kinds
of
data
analysis.
A
Even
an
are
you
know,
I,
don't
know
why
I
beat
up
on
our
but
I
like
being
on
our
so
our
analytical
environment-
and
some
of
this
is
you
know
this
is
a
I
think
we're
talking
through
a
series
of
patterns
that
kubernetes
has
been
very
given
us
a
lot
of
capabilities
very
quickly
to
launch
these
niche
this
these
these.
What
we
thought
were
much
more
niche
application
stacks
and
it
turns
out
to
be.
A
They
actually
turned
out
to
be
much
more
beneficial
or
beneficial
in
many
ways,
not
only
across
the
stack
but
for
many
other
kinds
of
business
needs,
so
general
business
aides,
around
decision
and
general
business
needs
around
analysis
are
sort
of
endemic
and
omnipresent.
So
these
patterns
are,
you
know,
very
useful
and
I
can
see
them
living
in
a
quite
a
long
time
and
just
being
blown
out
to
scale.
A
Our
telemetry
stack
right
now,
as
I
said,
is
sort
of
the
traditional
stack
that
you've
seen
probably
across
the
conference,
but
our
future
state
is
to
provide
separate
graph
on
ax
stacks
and
to
to
customers
to
tenants
and
then
to
provide,
keep
fluent
D
in
the
mix,
but
for
allow
teams
to
bring
their
own
log
aggregators
and
dashboards.
Since
we
see
a
mix
of
you
know,
I'd
like
to
have
Splunk
I'd
like
to
have
something
else,
I'd
like
to
do,
send
it
to
cloud
watch,
etc.
A
So
what
do
we?
Maybe
I
should
have
put
this
light
a
little
earlier.
What
our
stated
multi-tenancy
state
for
us
at
you
know,
going
from
one
six.
There
wasn't
as
good
support
before
state
and
one
seven
we
saw
the
stateful
sets
and
become
emerge
out
of
the
the
beta
where
of
pet
sets
and
they've
been
rock-solid
for
us
on
AWS
and
so
I
mean
that's.
That's
been
some
of
the
core
reason
to
build
the
kubernetes
cluster
is
it's
stateful
capabilities?
What
do
I
mean
by
multi-tenancy
I
mean
his
improvement,
he's
already
multi-tenant.
A
A
To
begin
with,
to
be
your
cloud
use
of
your
cloud
provider
more
efficient
and
everybody,
you
know
who
recognizes
that
that
some
need
for
Ella
elasticity,
because
they
might
have
been
coming
from
auto-scaling
groups
with
pure
UVM's
and
ec2
instances,
and
they
want
more
of
that
kind
of
thing
in
the
cluster.
So
that's
something
we
are
definitely
eager
to
leverage
with
kubernetes
and
all
the
great
work
that's
being
done
by
the
auto-scaling
sig.
A
A
You
know
three
times
out
of
the
day
when
you're
normally
connecting
six
million
times,
so
those
are
normal
types
of
blips,
just
retry
so
and
then
going
beyond
liveness
and
readiness
metrics
to
exporting
in
a
deeper
health
check
the
deeper
health
metrics.
So
how
many
folks
are
running
kubernetes
in
production
right
now
again,
many
about
at
least
half
the
crowd
and
how
many
run
with
state
like
how
many
running
safely
goal.
So
almost
the
same
amount.
Everybody
is
going
for
state
and
then
are
you
running
at
kind
of
for
your
own
teams?
A
B
Hello:
everyone,
my
name's
Brock,
so
the
name
spacing
tenant
isolation
namespaces.
What
are
they
essentially
a
virtual
cluster
within
kubernetes?
Soon
we
went
down
this
path.
Our
system
maps,
as
well
as
our
10
apps,
were
all
deployed
in
the
same
namespace,
so
we
had
tenants
stepping
on
each
other's
toes,
so
one
tenant
would
update
a
secret
or
config
map
unaware
that
another
tenant
was
using
it.
B
So
we
decided
to
break
out
the
deployments
as
well
as
namespaces
for
tenants,
so
we
locked
down
the
user
policies
via
decks
and
our
back
authentication
and
authorization
solutions
I
want
to
throw
it
out
there
that
we
do
not
we're
not
biased
towards
these.
It
was
particularly
for
our
process
in
our
culture
using
LDAP,
as
well
as
some
of
the
user
policy
based
you
can
get
with
our
Beck,
so
we're
we're
in
the
infancy
stage
of
migrating
from
flannel
to
calico,
and
with
this
we
can
get
enhanced
pod.
The
pod
communication
security.
B
All
right,
so
how
do
we
do
state
applications,
and
this
is
via
the
state,
full
I'm?
Sorry,
the
persistent
volumes
subsystem.
What
is
that
to
two
components?
Persistent
volumes,
which
is
a
piece
of
storage
somewhere,
persistent
volume
claimed
as
a
link
to
that
storage,
so
that
persistent
volume
is
married
with
a
pot.
Wherever
that
pot
moves
in
the
cluster
Christa's
volume
claim
will
move
with
it
and
that
way
any
application
can
always
access
its
data.
B
So
when
we're
talking
distributed,
applications
deployed
on
kubernetes
generally
say:
I'll
give
an
example
at
CD,
zookeeper
Kafka,
with
each
there's
generally
a
unique
ID
for
each
node
and
that
unique
ID
tells
that
node
what
subset
of
the
data
is
responsible
for.
So
we
can.
We
can
leverage
staple
sets
to
have
that
unique
ID
permeated
throughout
the
various
layers,
the
pod,
the
prime
claim
and
the
persistent
volume
we
want
to
throw
it
out
there,
storage
classes,
if
used
correctly,
are
sysadmin
best
friend.
B
So
we
are
a
bank,
we
do
have
compliance
requirements
and
one
of
which
is
the
rehydration.
So,
as
Keith
mentioned,
we
have
to
keep
our
nodes
refreshed
every
60
days,
and
this
is
to
ensure
the
latest
security
updates
and
patches.
We,
we
leverage
rehydration
via
a
commands
job
and
subsequently
we
can
also
leverage
it
for
upgrades.
B
What
it
does
is
it
scales
out
drains.
Each
node
ensures
that
persistence
volumes
are
reattached
the
persistence
volume
and
before
we
scale
in,
and
we
ensure
that
all
the
pods
are
healthy
when,
when
we
did
three
hydration
with
an
upgrade
months
back
and
it
took
about
eight
hours
manually-
and
this
was
a
cluster
with
a
static
Etsy,
demasters
I'm,
sorry
masters,
we,
the
CD
with
the
Masters
and
we
have
static
set
of
those
of
five
notes,
the
minions
or
about
seven
across
a
ZZZ
very
large
nodes.
B
So,
as
Keith
mentioned,
we
run
a
lot
of
Apache
based
JVM
apps,
and
these
JVM
apps
can
get
very
unruly,
especially
with
memory.
So
we've
encountered
applications
such
as
Zeppelin,
where,
if
they're
scheduled
on
the
same
node,
sometimes
they
take
too
much
memory
for
that
node
and
can
have
the
node
fall
over
subsequently
so
potting,
it's
high
affinity
ensure
that
at
most
one
pod
per
per
Zeppelin
is
on
one
node.
B
Some,
some
good
behaviors
are
good
safety
belts
or
I
guess
well,
anyway,
resource
limits,
limit
ranges
and
resource
quotas,
resource
limits
essentially
for
each
app-
and
this
is
you
know,
setting
limitation
on
CPU
memory
limit
ranges
if
the.
If
the
application
developer
does
not
set
them,
then
it'll
be
the
default
CPU
and
memory
for
that
deployment.
Replica
set
whatever
may
be.
Resource
quotas
are
setting
the
cap,
so
you
can
set
a
cap
on,
say
the
secrets.
B
Pods
the
number
of
services
for
a
given
namespace,
as
well
as
the
cap
on
the
number
of
CPUs
or
memory
cores
for
a
given
name,
space
and
okay.
It's
cubelet,
there's,
there's
a
very
small
nuance
to
this:
there's
a
property
called
system
reserved
for
the
cubelet
and
what
this
does
is
it
ensures
that
there's
always
at
least
sufficient
resources
for
CPU
and
memory
on
that
cubelet
to
always
ensure
that
it's
running,
if
you
have
unreal,
applauded
something,
went,
throw
it
out
there,
it
can
be
your
best
friend
so
for
the
future.
B
C
Model
of
the
platform,
is
I
read,
aflink
app
that
consumes
from
a
Kafka
topic
and
then
writes
to
a
copy
of
topic.
Also,
another
interesting
talk
was
Diane
marshes
from
Netflix.
Her
her
talk
about
how
culture
can
influence
the
tech-
and
that
was
really
resonated
with
us
as
well,
because
see
ICD
is
a
really
big
thing
at
Capital.
C
One
is
I
mean
we
really
have
been
pushing
the
ability
like
for
these
flink
developers
to
be
able
to
push
their
apps
quickly,
and
so
they
were
asking
for
tools
like
a
CLI
that
they
could
use
in
their
Jenkins
jobs,
and
so
that's
what
we
wanted
to
give
them.
The
idea
here
is
that
we
had
a
CLI
that
they
could
run
from
their
local
machine
from
when
they're
working
in
like
a
staging
or
QA,
but
then
also
the
same
CLI
could
be
installed
on
Jenkins.
C
You
know
it
was
go
a
p--
and
it
can
be
installed.
We
had
builds
for
both
Mac
and
Linux,
and
the
way
that
was
set
up
is
we
have
an
Al
B
and
a
set
of
micro
services
that
listen
for
those
various
CLI
commands
internally
or
in
grow
via
HTTPS,
but
the
communication
between
the
microservices
is
done
via
G
RPC.
C
So
what
we
try
to
do
is
simulate
the
job
manager
submission,
but
we
try
to
add
some
additional
functionality.
So
one
of
the
things
that
we
add
is
we
will
watch
the
deployment
for
a
fixed
amount
of
time
and
capture
all
of
the
states
that
that
flink
job
is
transitioned
into
so
is
it
an
example?
It
could
go
from
starting
to
running
and
if
it
stays
in,
that
state
were
in
good
shape.
C
C
One
of
the
things
that
Keith
alluded
to
is
our
sort
of
shared
model
and
some
of
the
learning
opportunities
that
came
from
this.
You
know
we
run
a
shared
fling
cluster
right
now,
but
we
realize
that
it's
kind
of
hard
to
put
some
guardrails
around
that
in
terms
of
like
maximum
number
of
task
slots
that
you
can
use.
C
The
other
thing
is
because
we're
not
using
something
like
yarn
one
job
can
it
can
have
a
disproportionate
and
like
negative
effect
on
the
overall
cluster,
and
you
know,
we've
we
talked
about
possibly
using
yarn
with
flink,
but
we
feel
like
kubernetes
can
provide
a
lot
of
that
capability.
You
know,
at
least
with
respect
to
CPU
and
memory
usage
for
us
and
the
other
thing
is
they
were
asking
for
their
own
clusters.
C
So,
where
we're
at
kind
of
in
our
journey
and
where
we're
starting
to
move,
is
this
idea
of
letting
more
of
a
self-service
model?
Let
the
customer
create
they're
clusters
like,
however,
it
is
they
want
to
mean
we
would
have
the
appropriate
guardrails
in
place
with
respect
to
resource
limits,
but
how
they
carve
it
out
is
up
to
them.
So
the
same
idea
is
to
use
Mike's,
AF
sling,
create
cluster
and
the
same
sort
of
ingress
provisioning,
but
now
we're
really
looking
at
CR
DS,
like
we've,
been
really
inspired
by
the
talks
we've
seen
about.
C
You
know
one
cluster
if
they
want
to
shared
one
but
say
they
have
one
that's
critical:
they
could
provision
just
one
flink
job
to
that
particular
cluster
and
then
do
their
other
operational
stuff
on
another
one.
We
really
want
to
put
it
in
their
hands.
The
other
thing
is
we're
looking
at
more
CN
CF
tools,
so
we
are
evaluating
jäger
right
now
to
see
as
a
replacement
for
Zipkin
and
I
went
to
the
Jager
talk
yesterday,
and
it
was
really
awesome
and
then
actually
with
that
Keith
is
gonna.
Take
us
home.
A
So
some
of
the
things
thank
you
both
that
was
awesome.
I
people
asked
me
what
I
do
and
with
the
dynamism
in
the
kubernetes
community
and
almost
and
also
to
you,
know
dodging
architectural
and
going
through
architectural
reviews.
Sometimes
we're
actually
doing
the
work
in
the
cluster.
You
know,
while
we're
proving
it
out
and
in
a
way
it's
made
us
pretty
fearless.
Sometimes
that
gets
a
little
scary,
because
when
you
come
from
more
compliance
based
environment,
they're,
usually
a
lot
more
all
the
process
gates
and
design.
A
But
kubernetes
has
really
give
us
a
lot
of
flexibility
to
test
things
out
more
quickly
than
we
would
normally
be
able
to
do
in
a
normal
system.
Delivery,
workflow
software
delivery,
workflow
as
well.
So
so
this
gives
I
tell
people,
you
know
what
do
you
do?
They
say
I,
say
I
feel
like
I'm,
an
in-flight
drone,
maintenance,
technician
and
so
we've
gone.
This
is
just
a
few
of
the
things
we've
gone
from.
A
A
We
kind
of
like
Kafka
connect
as
a
replacement
for
that,
but
going
from
influx
DB
and
heaps
tur
to
Prometheus,
given
the
awesome
work
in
the
by
the
Prometheus
community,
efk
to
sort
of
the
fluent
bit
plus
BYO
log
aggregation
ignite
for
heavyweight
cash
grid,
Redis
sky
DNS,
decor,
DNS,
s3,
direct
to
mania,
etc.
So
this
kind
of
gives
you
I
don't
know
if
your
journey
has
been
like
that,
but
we've
just
been
churning
through
things
evaluating
them
quickly
in
implementations
in
production.
Even
and
then
you
know
where
they
fail.
A
Ok,
we
move
on,
and
I
mean
this
is
so
much
of
in
some
way
we
are
encapsulated
in
kubernetes
from
our
from
our
end
users,
but
in
the
same
way
our
speed
our
time
to
market
for
the
platform
and
platform
services
is
entirely
almost
entirely
due
to
the
kubernetes
capabilities
conclusions
that
we
see.
So
you
know
if
you're
running,
kubernetes
I
would
encourage
you
to
think
about
what
you're
running
at
for
what
kind
of
manage
service
you
take
on
any
number
of
different
kinds
of
workloads.
You
could
get
into
trouble.
A
We've
carved
out
the
decisioning
space
for
our
cluster
folks
have
come
to
us
and
said
we'd
like
to
run
we'd
like
to
run
our
95
jobs.
There
we
were
I
could
run
put
all
our
Kafka
topics
there
for
us,
that's
we're
not
a
Kafka
as
a
service,
we're
not
a
knife
I
as
a
service,
we're
decisioning
platform
as
a
service,
so
deciding
what
kind
of
kubernetes
cluster
you're
going
to
run
is
really
putting
what
kind
of
service
you're
going
to
offer.
A
If
that's
the
kind,
if
you
are
a
managed
service
provider
in
that
regard,
love
the
the
DSL
of
domain-specific
language,
CLI
based
insulation,
because
you
get
that's
both
the
benefit
of
being
a
DSL
and
working
with
the
CI
and
CD
tools
State's
going
to
creep
in
it's
almost
impossible
to
avoid
it.
It's
even
especially
for
running
resilient
clusters
going
as
we
look
to
go
across
region
and
multi-cloud.
A
You've
got
state
to
manage
and
state
to
maintain,
that's
gonna,
that's
inevitable,
unchecked
and
especially
ad
hoc
workloads.
You've
got
to
really
look
at
we
splitting
resource
limits
on
them.
So
you
don't
get
kind
of
the
tragedy
of
the
Commons
you,
you
are
likely
almost
already
likely
multi-tenant.
You
may
not
even
realize
it
even
with
your
own
ops,
jobs,
we've
seen
and
I
think
it
was.
Is
it
Jevons
paradox?
Is
that
what
people
have
been
quoting
here?
A
We
see
it
all
the
time,
because
I
see
the
slope
of
the
of
our
of
the
line
for
the
storage
that
we
provision
for
stateful
workloads
just
continue
to
climb
almost
hockey
sticking
clusters
supporting
streaming
service.
We
thought
we
were.
Oh,
we
were
just
gonna,
be
a
streaming
service
cluster.
You
still
need
request
response
services
and
so
thinking
through
how
you're
gonna
host
rest
or
GP
RPC
facing
a
RPC
customer
facing
services.
Obviously
sto
is,
is
the
go-to
there
and
give
them
kubernetes
extensibility?
That's
all
working!
That's
so
many
great
things
work
in
progress.
A
We
see
more
and
more
specialized
clusters
coming
I've
seen
it
in
robotics
automation,
decisioning,
you've
seen
some
of
a
machine
learning
already
here
so
great
stuff.
That's
that's!
Coming
community
shout
out
one
of
the
say
thank
you
to
Sam
Brown
Sam,
raise
your
hand
for
organizing
the
Nova
kubernetes
Meetup.
If
you're
in
the
Northern
Virginia
area,
DC
area,
please
come
really
excited
to
grow
that
community
and
yeah.
Thank
you
all.