►
From YouTube: Taming Thousands of Knative Services for Thousands of Users - Martin Henke & Norman Böwing, IBM
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Taming Thousands of Knative Services for Thousands of Users - Martin Henke & Norman Böwing, IBM
Our challenge is to operate Knative in a high scale multi cluster, multi region, multi tenant environment, while the common Knative installation is usually used by a single tenant with a limited number of services. During this endeavor we are faced with numerous challenges to adopt Knative to that use case. We have to isolate Knative applications from different tenants, we have to scale the service mesh (Istio) and we need to configure Knative to be able to serve thousands of services relia
A
A
So
this
is
kind
of
a
glimpse
in
the
user
interface.
So
it's
the
easy
pass
is
just
you
type,
your
name
of
your
application.
You
type
your
container
press
create
and
your
service
is
running
and
you
can
access
it
secured
via
https
in
the
cloud
I
think
for
this
audience.
The
second
thing
is
probably
kind
of
interesting.
A
You
can
get
a
cube
config
using
the
command
shown
Above
So
to
can
select
your
project
you're,
getting
a
cube
config,
and
then
you
can
use
the
usual
tools
you're
using
so,
for
example,
you
can
then
KN
service
create,
and
you
can
run
your
application
in
the
cloud
easily,
and
for
that
you
don't
have
to
care
about
you
clusters
at
all.
You
don't
see
it.
A
A
and
I
think
we're
using
now
Connecticut
since
at
longer
times,
since
011
and
now
on
one
on
one
six.
So
we
made
a
long
journey
with
it,
and
same
is
for
istio,
as
you
can
see,
which
you
use
as
a
service
mesh
started,
also
with
one
five
an
hour
on
Feb
115
and
at
the
moment
we
are
also
working
with
some
here
with
the
community
on
getting
a
connective
conformance,
certification
running.
A
So
earthens
code
engine
is
running
globally,
I
think
now
in
nine
data
centers
it
runs
the
world
with
lots
of
users,
lots
of
services.
We
have
seen
that
this
use
case
is
kind
of
special.
So
you
don't
get
not
you,
don't
see
lots
of
blocks
or
documentation
how
to
run
it
in
a
kind
of
this
High
number
of
services
and
also
the
documentation
of
running
things.
A
And
since
though
istio
is
a
native
has
a
number
of
methods
you
can
choose
from
when
we
started,
we
had
to
make
a
decision,
because
we
decided
that
every
customer
project
has
its
own
certificate
and,
at
that
point
of
time
at
least
it
narrowed
down
or
narrowed
down
our
choice
of
service
measures,
I
think
to
one.
This
was
istio
supporting
that
configuration.
A
So
we
have
this
slightly
with
this
teacher
on.
This
is
kind
of
the
lessons
learned,
we're
not
on
the
way
and
we
might
want
to
share
so
this
is,
though,
there's
a
lot
of
service
measures
around
your
Tech.
Your
requirements
sometimes
limited
to
a
very
low
number,
in
our
case,
at
least
at
this
point
of
time
to
once
and
interestingly,
we
have
to
look
at
this
definition.
A
A
A
This
was
kind
of
okay
as
long
as
we
only
provided
public
accessible
services.
So
there
was
this
public
bus
anyhow,
you
can
assess
it,
but
we
also
provide
a
private
access
plot
to
services
from
the
customer's
VPC,
and
then
that
gets
a
real
problem
that
you
connect.
Something
you're
not
allowed
to
do
and
we
worked
around
that
I'll
show
you
actually
out
of
bands
of
connective
and
istio
by
putting
filters
into
Envoy,
which
is
it's
specifically
to
Simply
fight
code.
A
Here
adds
a
secret
to
which
outgoing
connective
local
call
and
when
we
receive
that
call
in
the
same
in
the
namespace,
we
check
that
this
added
Secret
is
fitting
the
namespace
that
it
originated
from
and,
of
course,
you
make
sure
that
this
is
not
handed
out
to
the
customer
and
we
overwrite
it.
So
it's
it's
a
question.
A
Last
thing
I
think
just
I
think
Skip
over
seeing
the
time.
Of
course,
we
have
to,
if
we
hand
out
resources,
we
have
to
make
sure
that
one
customer
cannot
take
the
whole
cluster.
So
we
limit
things
number
of
apps.
You
can
provide
the
number
of
revisions
because
revisions,
each
service
resources,
IP
addresses
in
the
end,
which
are
limited
so
yeah.
We
have
to
limit
things
and
make
sure
that
the
customers
play
well
with
each
other,
and
but
you
will
have
some
flexibility
there
to
react
to
customer
demands.
A
A
Good
thing
is
always
what
really
uses
these
connective
really
provides.
A,
fine
grain
control.
By
by
this
capability
settings
you
can
do
which
helps
us
a
lot
controlling
something
some
aspects
of
connective
and
with
that
I'm
through
the
multi-10
part
and
normal
will
take
the
scaling
section.
Thank
you,
Martin
yeah.
Quite
maybe
questions
through
that
spot.
Maybe.
B
Yeah,
first
of
all,
thank
you
for
the
insights
that
you
gave
us
the
one
question
more
and
yeah
I
love
a
question.
Would
would
this
be
also
possible
for
any
kubernetes
kinetic
installation
so,
like
the
the
recipe
how
to
set
up
Network
policies
and
also
this
extra
Android
filter,
there's
something
that
would
be
helpful
for
for
the
community,
because
I'm
asking
whether
it
might
make
sense
at
this
to
the
documentation
or
what
a
blog
post
around
that.
So.
B
C
C
C
C
Thank
you
Martin.
So
after
we've
secured
our
tenants
and
they
are
isolated
from
each
other,
the
next
thing
we
will
want
to
do
is
to
actually
scale
up
so
I
want
to
give
you
some
insights
on
what
we
do.
What
we
change
in
the
native
configuration
or
how
we
configure
K
native
to
scale
to
the
level
we
are
currently
at,
which
is
multiple
thousand
Services
per
per
cluster,
and
the
things
I
want
to
talk
about,
are
basically
three
the
cold
start
time.
C
So,
as
you've
seen,
we
are
using
istio
as
a
service
mesh
which
comes
with
overhead
on
top,
and
we
want
to
get
that
down
as
much
as
possible
and
also
some
particular
configuration
information
for
k-native
and
the
special
section
4K
native
plus
istio.
What
you
may
need
to
be
mindful
of
when
you
use
istio
with
K
native
and
how
to
get
the
cluster
to
a
sufficient
size.
C
So,
for
the
cold
start
time,
there
are
basically
two
things
with
istio
or
we
can
obviously
pre-pool
all
sidecar
images
out
of
band
to
get
basically
pre-warm
all
nodes.
We
have
so
that
the
image
does
not
the
sidecar
images
for
istio,
proxy
and
Q
proxy
proxy,
which
we
need
do
not
need
to
be
pulled
extra
on
each
node
when
scaling
up,
but
that's
actually
not
the
bigger
overhead.
C
We
also
experimented
with
istio
in
a
traditional
way,
so
that
the
istio
proxy,
which
runs
as
a
sidecar,
is
configuring,
the
IP
tables
as
a
networking
layer
and
basically
run
it
against
or
performance
tested
it
against
the
istiocni
plugin.
But
there
was
no
noticeable
difference
in
starter,
cold
start
time.
The
only
thing
that
mattered
in
the
end
was
the
mesh
tuning
with
getting
the
push
in
the
discovery
information
for
the
services
as
quickly
as
possible
to
the
activator
that
is
in
the
path
and
to
the
service
endpoint.
C
Yeah,
this
is
the
lesson
there
of
the
minor
overhead
of
the
actual
sidecar
image.
Pooling
both
images
are
pretty,
are
pulled
pretty
quickly
and
even
if
you
do
use
the
cni
plugin
and
configure
the
IP
tables,
not
in
the
actual
init
container
of
a
smart,
K
native
application
or
a
native
service,
it
does
not
matter
in
the
end.
For
the
code
start
time,
what
matters
is
the
istio
state
synchronization
that
needs
to
happen,
so
the
istio
control
plane
needs
to
get
the
information.
C
C
So,
from
a
pure
connective
perspective,
we
deactivate
the
HBA
for
the
activator,
why
we
saw
that
when
we
have
the
activator
HPA
enabled,
what
will
happen
is
dynamically
if
it's
necessary,
the
activator
instances
will
spin
up
or
down
depending
on
need,
and
what
will
then
happen
is
the
activator
service
will
be
changed
and
the
endpoints
for
all
services
in
the
cluster
will
be
changed,
which
means
istio
needs
to
pick
up
all
that
information
and
send
it
out
again,
which
is
a
huge
pain.
That's
we
need
to
avoid
at
all
costs.
C
The
second
thing
is,
you
see
there
we
are
using
proxy
mode,
so
we
have
the
activator
for
all
services
in
our
cluster
always
in
path.
It
will
never
be
taken
out
because
that
again
will
trigger
istio
to
need
to
push
more
information
and
the
one
thing
is
still
currently
only.
Does
it
only
pushes
state
of
the
world
information,
not
a
Delta,
so
even
if
one
service
in
your
cluster
changes,
it
will
not
send
one
service.
C
If
that
service
is
visible
by
other
services,
it
will
send
every
cluster
every
information
to
that
service,
which
is
a
lot
of
information
yeah.
Then
we
added
an
setup
for
all
the
K
native
components
so
that
we
have
run
multiple
and
different
availability
zones
to
BHA
with
the
that's
not
currently
in
the
current
native
defaults.
C
Installed
a
multiple
tenants
that
have
different
resource
quotas
and
can
adjust
those
or
requests
to
adjust.
This
might
have
an
impact
on
the
Q
side
car.
If
you
don't,
pin
it
to
something
specific
or
make
it
tied
to
the
size
of
the
service
itself,
then
the
kubernetes
API
calls
that
the
controller
does.
C
In
addition
to
that,
we
don't
want
to
only
Auto
scale
pots
or
Auto
scale
native
services
for
our
customers.
We
are
also
using
a
standard
cluster
autoscaler
with
a
provisioning
feature
on
top
of
it
to
basically
order
nodes
ahead
of
time
based
on
workload
demand.
We
anticipate
and
then
pre-warm
all
those
nodes
that
we
can
get
okay
native
services
on
that
app
as
quickly
as
possible.
C
Coming
to
the
the
istio
section,
I
think
what
was
it
K
native
1.4
that
supported
mesh
mode?
So
we
don't
want
to
or
we
don't
have
pot
addressability,
so
we
always
run
with
mesh
mode,
enable
to
basically
stop
from
the
defaulting
behavior
that
was
present,
where
first
can
native
try
to
go
directly
to
the
pot
and
then
default
it
back
to
the
cluster
IP.
So
we're
running
with
mesh
mode
and,
as
already
mentioned,
the
proxy
mode
we
run
in
so
the
activator
is
always
in
path
to
avoid
reasons
on
canadif
and
issue
size.
C
If
that's
not
present,
what
we
saw
when
scaling
out
further
and
further
is
basically
503s
a
lot
of
them.
If
you
take
the
activator
back
in
path
when
it
was
out
again,
the
activator
needs
to
have
service
information
for
that
service.
It's
going
to
be
in
path4
and
if
that's
not
present
at
that
time,
okay
native
switches,
the
endpoint
but
istio
does
not
have
did
not
send
the
correct
information.
Yet
the
user
will
see
503s
on
the
end,
and
we
want
to
avoid
that.
C
Another
thing
is:
this
is
just
the
the
standard
artifacts
when
we
have
a
service,
a
configuration
and
basically
a
route
and
the
route
traffic
to
different
revisions,
and
when
we
have
revisions,
what
will
be
created
by
k-native
R2
services
to
cube
Services,
a
public
one
and
a
private
one,
and
they
always
look
like
that.
It's
not
configurable,
and
this
is
for
an
active
revision
and
at
least
part
of
that
is
always
necessary.
C
C
There
are
defaults
where
they
are
retained
for
some
amount
of
time
or
you
can
pin
it
to
a
maximum,
but
for
a
non-active
revision,
K
native
keeps
all
artifacts
in
place.
There
is
a
deployment.
There
is
a
replica
set.
There's
a
port
outer
scaler,
there's
a
serverless
service
and
there
are
those
two
Cube
services.
C
C
So
to
come
to
the
last
part,
we
want
to
reduce
istio
mesh
synchronization
because
that
takes
up
a
huge
amount
of
CPU
and
causes
all
the
delays
in
cold
start
time,
and
for
that,
as
just
saw
the
Canadian
garbage
collection,
we
also
limited
it
to
maximum
one
non-active
revision
per
customer
so
or
pep.
So
the
app
all
the
connective
servers
can
only
have
one
non-active
revision.
C
This
gives
each
user
some
kind
of
safety
net
to
go
back
if
any
anything
goes
wrong,
but
it
prevents
us
from
having
all
those
Services
lying
around
that
are
not
actively
rooted,
because
we
don't
want
to
send
that
information
out
and
istio
supports.
Basically,
a
lot
of
features
I
want
to
concentrate
onto,
and
this
is
the
mesh
debounce
where
you
can
specify
amounts
of
time
on.
C
What
the
impact
of
having
non-active
revisions
will
be
and
to
keep
an
eye
on
istio
settings,
especially
for
d-bounds
d-bonds
after
means,
every
push
will
be
delayed
by
that
amount
of
time
that
will
have
a
direct
impact
on
provisioning
time
on
revision
creation
and
d-bonds
Maxes.
How
long
to
keep
aggregating
pushes
until
you
finally
send
it
out
and
as
a
future
istio
does
have
something
in
plan
to
support
Delta.
It
pushes
that
Envoy
already
supports
to
get
rid
of
all
this
unnecessary
synchronization
effort.
C
C
We
also
want
to
support
the
prostitute
proxy
protocol
together
with
istio
for
enhanced
audit
auditability,
and
we
want
to
adopt
the
Delta
pushes
once
available
and
with
that
I
want
to
thank
well
the
rest
of
our
team
and
especially
our
open
technology
and
developer
advocacy
team
that
is
doing
most
of
our
contributions
to
Canada.
If
you
all
know
Max,
Paul
and
Angelo,
we
are
mostly
from
the
operation
side
and
we
want
to
thank
the
rest
of
the
community
for
all
their
contributions
and
to
highlight
from
all
the
components
you've
seen
in
the
beginning.
C
That
Martin
showed
you
K
native
is
the
center
part,
and
you
saw
the
size
was
it's
basically,
the
centerpiece
of
everything
together
with
istio
and
the
yeah.
The
upgrade
experience
and
the
stability
for
K
native
is
is
good.
It's
better
than
most
of
the
other
components
we
use
and
I
want
to
thank
the
community
for
that
yeah
yeah.
Thank
you.
B
I
have
a
question:
have
you
experienced
any
size
limitations?
Besides
those
that
you
have
seen
like,
for
example,
the
maximum
number
of
services
that
you
can
deploy
on
a
single
cluster
I'm
asking
because,
for
example,
IP
tables
have
a
size
limit
and
if
you
say
that
a
lot
of
services
are
not
used
anymore,
they
all
still
populate
these.
This
kind
of
data
and
I
wonder
whether
you
have
seen
some
ceiling
in
with
respect
to
the
number
of
objects
that
you
can
deploy
on
a
single
cluster
actually.
A
Yeah
actually
I
think
we
didn't
hit
a
heart
limits
somewhere,
yet
I
think
we,
we
knew
I,
think
the
the
nearest
thing
actually
was
this
Gateway
configuration.
We
knew
that
at
some
point
of
time
will
at
least
Force
us
to
do
something
on
the
database
side
to
to
store
it
yeah,
but
I
think
our
experience
is
that
we
are
kind
of
we
have
a
kind
of
a
ongoing
fight
with
with
his
service
mesh
to
get
the
data
around
and
connected
to.
That
is
also
this
marshalling
and
marshalling
of
this.
A
F
I
was
just
going
to
say
great
great
presentation.
I
just
had
two
questions
so
with
the
limits
with
the
istio
there,
I
guess.
How
do
you
guys
handle
rollouts?
Were
you
doing
upgrades
to
istio,
potentially
still
running,
thousands
of
workloads
that
could
be
in
polite
when
you
have
to
do
the
rollout
of
those?
The
second
thing
is,
with
the
chatter
of
istio,
with
XDS,
so
obviously,
you've
had
to
fine-tune
that.
A
A
So
no
turn
time
for
that
ambient
meshes
is
the
thing
I
think
we
will
look
in
because
it's
exactly
I
think
not
for
this
string
problem,
but
for
the
for
the
resources
used
in
the
cluster.
So
if
much
less
sidecars
that
you
have
to
care
for,
and
so
it
would
be
in
the
end,
less
less
expensive
to
to
run
the
whole
system.
G
This
is
a
quick
question,
so
you
mentioned
that
a
lot
of
your
work
is
focused
on
getting
reducing
what
istio
is
pushing
out
and
reducing
the
number
of
activation
six.
What
sort
of
trade-off
are
you
making
by
doing
that?
Is
there
some
part
of
the
experience
of
using
K
native
that
you
lose
by
trying
to
like
prevent
istio
from
sending
out
all
that
data.
C
So
it
can
hit
the
cold
start
time
that
will
you
need
to
find
a
balance
between
cold
start
time,
provisioning
time
and
so
or
if
you're,
creating
new
revision
or
updating
your
service,
because
those
times
will
take
a
hit.
If
you
try
to
reduce
what
istio
needs
to
think
so,
depending
on
how
many
services
you
have
in
your
cluster,
what
rate
of
changes
or
how
many
services
are
provisioned
per
minute
per
second,
and
how
many
cold
starts
you
have
you
need
to
find
a
balance,
but
yeah
there
will
be
a
trade-off
made.
D
If
I
can
add
one
more
bit
to
that,
they
turned
off
a
bunch
of
the
auto
scaling
and
things
that
are
efficiency
gains
to
avoid
istio.
Stealing.
All
of
that
back
by
you
know
having
to
do
an
XDS
sync,
so
they
turned
off
auto
scaling
on
the
activator
and
they
turned
off
taking
the
activator
out
of
the
path.
When
you
get
a
big,
you
know,
when
you
get
a
service,
that's
getting
a
thousand
requests
per
second,
or
something
like
that.
E
A
Actually,
we
have
a
shotting
solution
behind
but
of
course
to
to
reduce
your
overhead
so
cluster
having
sorting
classes
overhead
of
the
having
a
control,
plane
and
management
component
on
that
we
try
to
use
the
Clusters
we
have
as
much
as
possible.
So
this
is
still
a
thing
to
to
get
as
most
as
possible,
effective
running
those
clusters,
but
we
have
a
sharding
concept
to
kind
of
adapt
to
growth
and
customer
numbers
that
we,
what
we
have.