►
From YouTube: IngressRoute: A Path Towards a Better Ingress Object for Kubernetes - David Cheney, VMware
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
IngressRoute: A Path Towards a Better Ingress Object for Kubernetes - David Cheney, VMware
The limitations of the Kubernetes Ingress object have been papered over with annotations by the community for several years. Contour, the Ingress controller my team are building, introduced a new CRD which addresses the limitations of the incumbent Ingress object and unlocks the ability for teams and operators to utilise modern web application deployment practices in multi team and multi tenant Kubernetes clusters. In this talk I’ll explain, from the point of view of an Ingress controller implementor, the limitations of the ingress object, and how our IngressRoute CRD addresses the current shortcomings while making it possible for multiple teams to collaborate and delegate responsibility in multi tenant scenarios. A slide deck from an earlier version of this talk is available here: https://dave.cheney.net/paste/ingress-is-dead-long-live-ingressroute.pdf
https://sched.co/Nrn1
A
A
So
the
part
of
kubernetes
that
I
spend
all
my
time
on
is
in
this
ingress
area.
But
perhaps
perhaps
you
don't?
Perhaps
you
don't
live
and
breathe
the
ingress
as
much
as
I
do
so?
I
want
to
go
over
a
little
bit
about
what
an
ingress
controller
does.
Ingress
controllers
are
responsible
for
bringing
traffic
from
the
outside
world
down
to
your
voice.
A
This
means
HTTP,
TLS,
load,
balancers,
reverse
proxies
all
of
that
good
stuff
at
layer,
seven,
and
just
to
be
clear,
even
though
we
do
use
envoy
as
our
data
plane,
we
are
not
a
service
mesh.
So
it's
a
common
misconception.
That's
an
old
news
envoy!
You
must
be
a
service.
Wish!
No
envoy
can
do
many,
many
many
things
we
use
it.
The
way
we
configure
it
is
as
a
load
balancer,
so
we're
focused
exclusively
on
bringing
remote
traffic
into
your
cluster
and
that's
quite
a
broad
topic.
A
So
the
way
that
I've
approached
this
design
space
is
to
say
a
good
ingress
controller
should
be
able
to
take
care
of
90%,
of
the
reasons
that,
maybe
previously
in
previous
lives
and
previous
automations,
you
would
have
had
to
use
nginx
or
Apache
or
squid
there's
some
kind
of
sidecar
or
it's
middleware
or
something
leaning.
The
request
flow
to
augment
your
application,
either
as
middleware
or
as
kind
of
glue
to
bring
it
all
together,
so
we're
never
going
to
get
to
a
hundred
percent.
A
hundred
percent
use
case
is
impossible.
A
A
The
big
one
is
traffic
consolidation.
So,
if
you
haven't
used
ingress
before
you
just
using
service
objects
with
type
load
balancer,
you
probably
know
that,
generally,
if
you're
inside
of
cloud
environments,
you
declare
a
service
type
load
balancer,
your
cloud
provider
will
going
to
spin
up
an
external
load.
Balancer
in
public
IP,
we
plumb
that
down
to
your
cost
track
me.
A
A
The
second
one
is
TLS
management
we're
halfway
to
through
2019.
You
need
to
be
talking
about
how
to
present
your
applications
behind
TLS
behind
edge
to
the
penis.
Now
the
good
news
is
on
kubernetes
side.
We
have
projects
like
cert
manager
and,
let's
encrypt
we're,
going
to
do
that,
work
of
getting
a
valid
public
certificate.
Renewing
it's
keeping
it
alive
that
kind
of
90
day
rolling
in
dough,
an
ingress
control
that
does
the
other
half
it's
going
to
make
sure
that
you
can
present
that
certificate
on
port
4/3.
A
So
you
can
secure
your
application
by
HTTPS.
So
there
really
shouldn't
be
any
reason
why
in
2019,
you
can't
have
a
secure
application
there,
no
structural
limitations
to
doing
this
anymore.
This
is
certainly
something
making
arrests
and
trials.
Hopefully
the
third
thing
that
I
think
our
big
part
of
the
ingress
patrol
promise
is
this
notion
of
abstract
configuration.
A
Now,
probably
given
your
backgrounds,
you
used
every
kind
of
load
balancer
from
f5
through
to
engine
X
through
the
EHA
proxy,
through
the
Apache
and
squids
all
who
have
different
configurations
to
do
the
same
thing
to
deploy
the
same
application
and
you
can
rewrite
the
configuration
in
whatever
language
they
are
users,
so
ingress
provides
us
a
way
of
having
an
abstract
way
you're
describing
the
properties
of
a
web
application.
The
different
routes
have
a
match
which
services
they
go
to
in
an
abstract
way.
A
Ingress
controller
takes
care
of
the
specific
configuration
for
the
specific
data
plane
that
you're
using
the
last
one
is
half
face
traffic
so
going
back
to
the
first,
the
first
point:
if
you're
using
a
service
IP
with
a
bustah,
you
get
TCP
at
layer,
3
down
to
your
pods,
now
HTTP
HTTPS,
just
T.
Simply
you
have
to
take
care
of
running
a
web
server
to
do
that,
handshake
to
their
TLS
handshake
to
to
do
pathways.
Wrapping
all
of
that
stuff.
You
have
to
do
that
in
your
application.
A
A
So,
with
that
background,
I
want
to
talk
about
some
of
the
larger
issues
with
in
graceful
folds,
not
specifically
one
that
I
work
on,
but
the
one
that
all
of
us
implement
is
work
on
dealing
with
the
kubernetes
in
this
object,
so
I
think
my
biggest
complaint
with
the
kun.
A
z/os
object.
Is
that
the
specification
for
it?
A
It's
just
a
bunch
of
text,
comments
in
the
API
documentation
there
being
higher
specification
port.
We
have
to
go
on
use
in
the
API
missionary
types.
It's
in
the
comments
on
those
parts.
So
Jason
is
not
a
specification,
Jason's,
barely
a
schema,
and
it's
not
sufficient
to
describe
how
to
implement
this.
With
this
specification,
you
need
to
make
this
concrete.
This
is
literally
the
specification
that
influences
like
myself
have
to
work
from
now.
A
This
is
perfectly
fine.
If
you
are
communicating
with
the
API
server
like
you're,
you
know
sending
a
message
to
the
API
server
or
using
their
kind
of
curl
command
line.
This
is
perfectly
fine,
but
as
an
implementer
someone
who's
building
tools.
From
top
of
this
API,
we
never
let
need
a
level
of
formality
in
this
specification,
but
is
simply
not
present.
A
What
I
think
that
we
need
is
something
akin
to
level
of
detail
of
an
internet
style
RFC,
because,
as
we'll
see
as
we
go
through
this
presentation,
there
are
a
lot
of
ambiguities
in
the
specification,
because
there
are
ambiguities
that
breaks
the
promise
of
the
of
the
ingress
object,
because
it's
ambiguous,
you
can't
abstract
a
figuration.
Is
it's
ambiguous.
Different
ingress
controllers
implement
features
in
different
ways
and
it
breaks
inside
your
nobility.
A
A
So
what
we
have
is
a
situation
where
each
ingress
document
can
have
the
notion
of
a
default
place
to
route
traffic
if
it
doesn't
match
anything
else,
rest
requests
that
don't
match
any
rule
go
to
the
default
back
engine.
It
seems
pretty
straightforward
if
you're
coming
for
a
programming
point
of
view
like
if
you
have.
A
A
Something
else
it's
something
akin
to
a
default
place
that
any
traffic
that
comes
to
your
cluster
for
any
virtual
host
could
potentially
go
for,
and
if
it
wasn't
complicated
enough,
you
can
mention
the
default
back-end
in
every
ingress
document
so
which
one
takes
precedence
and
it's
these
kind
of
ambiguities
which
are
left
to
the
interpretation
of
the
implementer
people.
Like
me,
the
nginx
ingress
controller
traffic
ambassador,
everyone.
If
Norton
yep
the
spec
we
can.
A
The
default
back-end
conflates
the
notion
of
a
virtual
host
like
Microsoft
comm,
which
is
the
host
header.
The
traffic
will
arrive
with
with
the
backend,
that's
going
to
serve
it
on,
and
that
doesn't
make
a
lot
of
sense,
because
for
some
some
cases
you
might
want
the
default
role
to
be
send
that
send
301
redirect
to
my
main
website.
So
you
come,
so
you
don't
want
to
have
dub
dub
dub
in
your
url.
But
if
requests
come
on
that,
you
want
to
redirect
them.
There's
no
way
to
do
that
in
the
ingre
spec.
A
A
Ingress
ceases
to
be
about
these:
are
the
routes
within
a
virtual
host
and
becomes
these
are
just
the
pool
of
routes
and,
as
I
said,
the
default
back-end
can
be
present
in
multiple
ingress
objects
as
well
as
this
very
loosely
specified.
They
are
a
global
default
back-end,
which
is
completely
not
defined
in
the
specification.
A
A
A
You
can
see
the
argument
for
why
it
works
like
this.
Ingress
objects
can
only
refer
to
the
services
in
the
namespace
they're
here.
So
if
you
want
to
combine
and
build
an
application
out
of
services
from
team,
a
and
Team
B
and
team,
C
put
them
all
on
routes
within
the
same
virtual
host,
you
need
to
have
an
ingress
object
in
each
of
those
namespaces
sharing
the
same
virtual
host.
A
So
if
they're
all
brought
together,
it
seems
pretty
logical
like
if
you
had
the
finance
department
in
one
in
one
group
in
one
namespace
in
the
ads
department
in
another
namespace,
that
they
all
collaborate
to
the
same
website.
You
would
have
an
ingress
document
in
the
same
website
across
all
of
them.
Minguk's
controls
stitch
them
together.
A
However,
this
means
that
if
someone
has
our
back
mission
to
write
an
ingress
object
anywhere
in
your
cluster,
they
can
add
a
route
onto
any
virtual
host
service
by
that
cluster,
there
is
no
way
to
namespace
to
use
our
back
to
say
you
can
only
write
ingress
objects
for
virtual
host
a
but
not
vote,
not
be
so
somebody.
Anyone
in
your
cluster,
with
the
Commission
to
write
an
ingress
report
can
add
a
route
to
your
virtual
host,
even
though
they
never
have
permission
to
write
into
your
namespace.
A
Just
a
case
you're
thinking
hey,
this
sounds
like
a
terrible
security
hole.
Maybe
I'd
like
to
not
do
that.
You
cannot
because,
in
a
perfect
example
of
Hiram's
law,
which
is
the
idea
that
programs
will
evolve
for
any
observable,
behavior
projects
like
cert
manager
will
lie
on
this
ability.
The
way
that
certain
managers
answers
the
HTTP
wonder
challenge.
Is
it
at
edits,
its
ingress
recorded
its
namespace
to
project
slash
dot
well-known
into
your
ingress?
A
A
What
this
boils
down
to
is
that
ingress
is
very
difficult
to
use
in
a
shared
cluster.
There
are
no
safeguards
to
prevent
anybody
that
has
the
permission
to
add,
or
edit
ingress
records
from
accidentally
or
perhaps
maliciously,
depending
on
what
kind
of
cluster
you're
running
messing
up
somebody
else's
ingress.
This
is
just
not
possible.
A
Let's
talk
about
a
few
other
problems
before
we
get
to
the
associates,
talk
about
a
few
other
problems,
and
this
is
more
focusing
on
a
modern
web
application
patterns.
Cuban
IT
services
are
mapped
onto
HTTP
routes
via
an
ingress
document.
However,
the
ingress
spec
only
lets
us
only
commits
us
to
have
one
service
program,
so
slash
prefix
goes
to
that
service
and
only
to
that
service.
Now
a
service
can
match
multiple
pods.
We
can
do
that
if
they
all
have
the
same
label,
so
we
can
meet
the
endpoint
control
of
all.
A
A
A
A
This
has
led
to
because
of
the
limitations
in
the
ingress
object,
because
there
are
so
few
fields.
We
have
to
put
everything
in
annotations,
I
common
things
like
do
you?
Do
you
allow
port
80,
or
do
you
put
a
301
upgrade
in
an
annotation
request,
timeouts
retry
parameters?
All
these
things
must
go
into
annotations.
A
A
So
arguably
the
annotations
apply
to
the
things
mentioned
in
that
document,
not
to
the
virtual
hosts,
but
this
argument
breaks
down
when
we
talk
about
properties
that
apply
to
the
virtual
host
itself,
like,
for
example,
what
is
the
minimum
TLS
version
that
this
website
will
accept,
say
your
company
has
a
policy.
You
only
accept
TLS,
1.1
or
higher,
so
that
means.
So.
A
How
are
we
supposed
to
interpret
this
annotation
if
you
split
your
hosts
across
several
documents
and
they
have
different
values
to
the
TLS
moon,
Protocol
version?
That
means
that
you've
committed
to
doing
a
different,
TLS
handshake
based
on
the
path
of
the
request
which
you
don't
know
at
that
point,
because
you
haven't
completed
the
TLS
handshake,
so
it
seems
that
some
annotations
apply
on
the
ingress
and
some
apply
globally
or
to
the
virtual
host.
A
A
Okay,
so
this
is
a
good
time
for
me
to
pause
to
say
that
while
I
have
a
bunch
of
gripes
about
arrests,
they
come
from
my
position
as
an
implementer
somebody
who
has
to
answer
all
these
questions
in
code,
like
I,
can't
I
can't
defer
the
documentation.
I
can't
ring
song
for
support.
I
have
to
answer
all
of
these
questions,
because
this
is
software.
It
must
know
absolutely
any
time,
but
I
don't
want
you
to
come
away
from
this
talk
and
saying.
Well,
this
guy,
whom
I
say
Australia
said
ingress,
is
no
good.
A
A
But
if
you
are
experiencing
these
problems,
like
you've,
you've
been
through
some
of
these
discussions.
You've
had
these
questions.
Let
me
tell
you
about
what
we're
doing
in
Toronto
about
a
year
ago
what
we're
still
called
FTO.
We
started
a
joint
development
agreement
with
Yahoo
Japan
to
build
a
large
load
balancing
solution
for
them
using
a
kubernetes
cluster,
effectively
like
in
the
clients.
A
Now
Yahoo
Japan
had
experienced
almost
all
of
these
problems,
not
through
malicious,
malicious
intent,
but
just
accidentally
messing
up
other
people's
routes
by
fat,
fingering
and
copy
pasting
things
in
error.
They
had
experienced
all
the
problems
in
trying
to
use
ingress
in
a
multi-tenant
solution,
so
we
went
away.
We
came
up
with
a
different
design
in
the
CID
for
the
ingress
route.
A
Now
the
first
thing
that
we
changed
was
that
each
ingress
wrap
document
refers
to
one
hostname
in
one
person
only
there
is
exactly
in
your
cluster.
There
is
exactly
only
one
document
that
describes
the
properties
of
the
host
here:
spec
virtual
host,
the
name
of
the
virtual
host
blog
heavier
calm,
and
because
now
we
can
define
one
place
to
talk
about
virtual
hosts
things,
not
routes.
We
can
put
the
tls
parameters
in
there.
A
We
can
put
other
information
about
the
virtual
host,
not
about
its
routes
and
there's
one
place
that
they
go
for
reasons
that
I'll
explain
in
a
little
bit.
We
call
this
a
route
ingress
route,
doc
use
because
because,
as
we'll
see,
they
are
at
the
root
the
root
of
the
tree,
and
so
anytime,
you
see
a
speck
with
a
virtual
host.
This
is
a
this
is
the
route
of
a
configuration
for
a
virtual
host
now,
because
we're
no
longer
limited
to
the
schema,
the
kubernetes
gives
us
an
ingress
object.
A
We
have
a
place
to
put
all
that
configuration
with
previously
stuffed
into
annotations,
for
example,
the
route
a
service
on
that
route.
We
can
control
things
like
the
load,
balancing
strategy
so
matching
route,
the
blogs
one
back
in
services
for
the
blog
service
and
the
strategy
we're
going
to
use
to
load
balance
across
the
different.
A
One
of
the
other
reasons
we
chose
envoy
is:
it
has
very,
very
good
configuration
reload
support
you.
There
is
no
kind
of
process
stopping
being
restarted.
Delays
like
reloading
the
process.
We
stream
configuration
changes
to
envoy
straightaway
and
they're
applied
straightaway.
This
means
that
we
can
do
things
like
long-running
WebSockets
without
dropping
connections,
and
so,
if
you
want
to
enable
WebSockets,
it's
not
an
annotation,
it's
just
a
thing
simply
there.
A
Now
in
general,
when
there
is
a
feature
that
should
be
available
for
all
users,
we
generally
like
to
turn
it
on
very
beau.
So,
for
example,
you
will
not
find
a
configuration
for
compression
because
we
just
turn
that
on
for
everybody,
I
can't
think
of
a
reason
why
you
would
not
want
to
improve
your
latency
and
save
bandwidth
by
compressing
the
results,
but
for
WebSockets
it's
a
little
bit
more
complicated,
because
if
we
allowed
any
route
to
be
upgraded,
attend
that
allows
a
malicious
attacker
to
send
a
WebSocket
upgrade
when
the
applications
and
expecting
it.
A
A
A
So
in
this
example,
we
have
both
a
timeout
policy,
which
is
one
second
in
the
start,
to
the
end
of
the
request
you
might
want
to
short,
you
might
want
them
all,
but
you
certainly
usually
want
to
have
some
kind
of
time
out,
so
that
connections
don't
stay
open
indefinitely
and
depending
on
the
stability
of
your
application
or
the
way
it
deals
with
overloads,
you
might
want
to
have
retry.
You
might
want
to
say,
try
three
times
giving
each
try
150
milliseconds.
A
In
this
example,
we
have
two
services,
so
it's
going
to
be
roughly
roughly
equal
weighting
between
those
so
rather
than
creating
a
service
paying
two
pods
with
matching
labels,
so
they
were
all
merged
into
the
same
service.
We
can
now
talk
specifically
about
two
different
services
that
are
served
that
serve
traffic.
The
slash.
A
A
So
in
this
example,
this
is
a
totally
made-up
product
that
he
had
some
mail
service.
Now
90%
of
the
traffic
he's
going
to
version
1.3
dot,
1
10%
of
the
traffic
is
going
to
version
2.
So
this
is
how
you
could
do
a
blue
and
green
style
deployment.
Maybe
you
started
1%
you
go
to
10%,
you
got
a
50%
and
so
over
time
the
amount
of
traffic
being
served
by
the
old
version.
A
Now,
traffic
moves
to
the
amount
of
traffic
served
by
the
new
version,
as
you
gain
confidence
that
things
working
out
and
you
can
change
these
numbers
on
the
fly
you
just
edit.
This
documents
who
control
edit
unless
changes
are
applied
immediately.
There's
no
restarting
there's
no
reloading
process,
there's
no
drop
connections,
while
the
configuration
is
reloaded.
This
happens
in
real
time.
A
So
there
are
two
things
to
go.
One
is
delegation.
Dell
is
delegation.
Is
our
answer
to
helping
multi
tenants
clusters
stay
manageable,
all
the
ingress
route
documents
you've
seen
so
far
and
what
we
call
routes,
and
that
is
because
they're
at
the
root
of
a
tree
of
delegation,
and
to
explain
why
delegation
I
think
is
important.
We'll
have
another
example.
A
So
again,
this
is
a
totally
made-up
company,
and
so
you
are
the
kubernetes
administrator.
These
are
the
problems
you've
been
asked
to
solve.
You
want
to
delegate
control
the
finance
route,
/
finance
to
the
Google
Finance
developers,
who
are
working
there
over
in
there
in
the
Google
Finance
namespace.
So
any
traffic
for
finance
and
below
should
be
going
to
the
applications
in
the
Google
Finance
namespace.
A
Importantly,
this
Google
Finance
team
shouldn't
be
able
to
alter
the
configuration
for
any
of
the
other
properties
on
this
website.
If
you
do
touch
ads,
touch,
search
touch
the
homepage,
they
should
only
be
able
to
work
in
their
little
area,
called
slash,
finance
and,
most
importantly,
none
of
the
teams
working
on
contributing
parts
of
this
application
should
actually
have
access
to
things
like
the
TLS
secret
for
serving
w-w-whoa.
A
So
how
would
you
do
this
with
the
the
kubernetes
ingress
object
itself?
You
could
certainly
split
your
routes
across
very
various
namespaces.
You
could
put
the
slash
Finance
match
over
an
ingress
document
to
the
Google
Finance
namespace,
but
as
I've
laid
out
today
you
it
is
difficult
to
do
that
securely.
It
is
difficult
to
do
that.
Knowing
that
you
have
probably
up
down
this
configuration
is
difficult
to
know
that
you
have
made
sure
that
nobody
else
can
overwrite
or
add
routes.
A
A
We
can
see
this
as
a
rodent.
Has
a
virtual
host
Anza,
it's
in
the
Google
Chrome
namespace,
there's
the
secret
in
there.
So
it's
it
lives
in
the
Google,
calm
namespace.
Unless
you
have
permission
to
read
that
namespace,
you
can't
read
that
secret
and
now
down
here
we
have
the
standard
matches
that
you've.
Seen
previously.
We
have
this
new
stanza,
rather
length
service,
you
say
delegate
and
you
can
think
of
delegation
at
this
level.
A
A
It
doesn't
have
a
virtual
host
stanza.
This
doesn't
mention
anything
about
the
host
it
applies
to.
It
is
what
we
call
it's,
not
a
roof.
It's
a
child
resource
it
can.
This
configuration
can
only
be
referenced
by
somebody
else,
so
you
can
write
whatever
documents
you
want,
but
unless
there's
a
reference
to
it
from
another,
it's
meaningless.
That
configuration
is
not
connected
to
anything,
so
one
of
the
ways
you
could
think
about
it
is
the
way
that
Delhi
that
DNS
works.
We
delegate
part
of
your
namespace
to
a
different
name
server.
A
A
In
the
final
piece
of
this
multi-tenant
puzzle
is
the
ability
to
restrict
where
we
will
look
for
these
root
namespaces.
As
an
opt-in
feature,
you
can
say
to
contour
on
startup
only
look
in
these
specific
namespaces
for
brute
ingress
documents,
so
we'll
only
start
building
this
tree
based
from
the
documents
that
you
permit.
A
Obviously
they
would
be
in
some
administrators
namespaces.
The
administrator,
just
like
the
Google
example,
would
set
up
the
configuration
for
Google
com.
They
would
control
difficut
and
they
would
delegate
the
actual
routes
like
the
the
routes
for
the
application
to
different
application
teams
in
different
namespaces.
A
So
that's
all
I
have
to
show
you
today
you
can
come
check
us
out
at
github,
become
pepto
contour.
If
you
find
us
at
hash
pond
contour
on
the
kubernetes
slack
and
of
course
you
can
email
me
anytime,
you
like
or
send
me
an
issue
on
github
yeah.
We
have
a
lot
more
information
on
there,
including
documentation
about
the
English
rat
CID,
how
it
works
all
the
different
options
and
many
more
than
I've
shown
you
and
I
think
I
have
maybe
two
three
minutes
for
questions.
A
B
B
A
A
We
have
customers
who
need
this
service
right
now,
so
we
have
built
this
for
our
customers
they're
using
it.
We,
you
can
come
to
VMware
and
ask
for
support
on
this.
This
is
a
fully
supported
product.
We
but,
as
you
say
each
at
the
moment,
each
of
the
ingress
controls
is
doing
something
different.
That's
terrible
for
the
community.
You'll
really
like
some
consolidation.
However,
I
have
I
think
those
discussions
of
consolidation
is
still
quite
early.
A
The
ingress
v2
stuff
that
Google
talked
about
a
month
ago,
I
still
haven't
seen
a
capful,
but
I
think
the
easiest
way
is
that
we're
on
cig
network.
We
were
really
looking
to
contribute
when
the
discussion
starts.
We
are
really
ready,
like
if
you're
watching
this
video
please
come
and
talk
to
us.
We've
really
run.
That's
all.
We
have
time
for
one
question
and
this
gentleman
here
then
we've
got
then
we
gotta
get
out
of
here.
C
A
A
In
terms
of
white,
why
would
you
choose
if
you
want
to
interested
in
English
art
stuff?
Why
would
you
choose
envoy
over
a
different
one
of
those
I
would
say
that
the
reason
it's
not
perhaps
for
performance,
so
any
of
those
things
I
think
almost
any
ingress
controller
can
saturate
and
network
these
days.
That's
not
hard.
The
benefit
that
we
have
is
envoy
has
a
very
good
reconfiguration,
APR
Ustream
configuration
by
G
RPC.
It's
not
writing
a
file
and
then
reloading
the
process.
A
We
can
stream
configuration
very
very
quickly,
so
we
can
make
those
changes
effectively
and
that,
certainly
from
my
my
history
as
an
operator
trying
to
debug
us
problems
is
a
huge
benefit
when
you
know
that
if
the
configuration
isn't
working,
how
you
thought
it
is
because
you
made
a
mistake,
not
you
didn't
wait,
long
enough,
okay,
so
I
I'm
out
of
time!
Thank
you
so
much.
Thank
you
for
that.
Thanks
speaker
for
your
patience.
Thank
you
very
much.