►
From YouTube: The Untapped Power of Services - L7 Load Balancing Without a Service Mesh - Damien Lespiau
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)
The Untapped Power of Services - L7 Load Balancing Without a Service Mesh - Damien Lespiau, Weaveworks (Advanced Skill Level)
At Weaveworks, we sometimes need more than the L3/L4 load balancing offered today with the Service abstraction. The Kubernetes Service & Endpoint objects have some extraordinary untapped powers: they can be used to build artisanal, high-level load balancing and session affinity schemes. This talk will go through a few examples: sharding across endpoints based on a layer 7 key, master endpoint election and demonstrate a tiny reverse proxy implementing service affinity using consistent hashing with bounded load.
About Damien
Damien has spent way too much time playing with Linux over the past 20 years. He has worked on all sorts of embedded products, is a GNOME contributor, has spend 4 years in the Linux kernel making Intel GPUs behave and is now working at Weaveworks, helping defining what the future of containers looks like.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hello,
I'm
Damien
and
today
I'm
going
to
talk
about
layer,
seven
load,
balancing
a
few
words
about
me.
I
wrote
my
first
line
of
C
20
years
ago.
23
years
ago,
even
I
worked
on
many
mg
devices.
I
am
an
own
contributor.
I've
worked
at
Intel
on
the
kernel,
Linux
kernel
I
spent
four
years
doing
graphics
driver.
So
if
you're
running
Linux
there's
a
good
chance,
you
have
some
of
my
code
and
I'm
sorry.
A
So
let's
talk
about
load
balancing
first,
the
ways
the
stuck
is
going
to
be
about:
where
do
we
need
layer?
Seven
advancing
especially
let's
we'll
break
down
the
different
parts
of
what
compose,
what
composite
is
load
balancing,
we'll
look
at
different
architecture?
How
to
implement
that
within
your
cluster
and
I
have
to
say
in
the
context
of
this
talk,
the
advancing
is
not
about
ingress.
It's
about
the
traffic
within
your
cluster.
A
A
So
first
things.
First,
I'm
sure
everyone
is
blessed
that,
but
where
is
a
community
service
on
Covanta
size
and
endpoints?
Let's
say
you
have
a
macro
service
in
your
cluster
and
you
decide
that
four
parts
would
be
will
be
fine
for
the
macro
service
that
it
will
be
able
to
handle
the
load
you
you
can
just
revert
it.
The
service
object
in
kubernetes
abstracts,
those
four
pods
into
something
front-facing
you
can
cope.
It
has
a
VIP
virtual
IP
address
and
you
connected
as
VIP
and
by
the
magic
of
Dannette.
A
Of
course
I'm
sure
everyone
knows
what
that
is,
but
it
it's
destination.
Nothing
you'll
be
rewriting
the
IP
header
to
rewriting
the
VIP
your
being
connected
to
to
one
of
the
IP
and
port
of
the
endpoint
demo.
It's
a
bit
soon
to
make
to
do
a
demo,
but
we'll
try
so
I
have
I
have
a
few
pods
running.
That's
the
full
post!
You
had
on
the
right.
A
That's
the
service
abstracting,
those
pods
and
I'm,
going
to
run
a
hundred
requests
to
that
service.
Vip
out
of
with
20
requests
a
second.
So
that's
going
so
you
get
some
answers
from
the
service.
That's
great,
and
now
we're
going
to
look
I've
instrumented
the
pot
to
tell
me
how
many
requests
they've
been
handling.
So
we
look
at
that
and
there's
maybe
a
bit
of
a
surprise-
is
the
whole
hundred
requests
ended
up
in
a
single
pod.
So
what
really
happened
here
so
yeah
just
in
case
the
demo
didn't
work.
A
A
Is
we
only
had
one
connection
between
the
client
and
the
service
VIP
and
because
HTTP
clients,
gp1
clients,
do
connection
pooling
and
they
use
keepalive
to
not
have
to
reconnect
every
time,
then
we
really
had
one
connection
for
the
Hollander
request
and
the
way
D
networks.
It
only
looks
at
the
first
packet
of
a
connection
to
decide
which
endpoint
they
they're
going
to
rewrite
the
VIP
arrest.
A
That
means
the
Hollander
requests
were
sent
to
using
the
same
connection
and
all
ended
up
in
the
same
endpoint
and
okay
in
http/1.
When
you
do
that
yeah,
if
your
request
rates
is
high
enough,
the
client
library
will
actually
have
to
create
new
connections
to
send
a
request,
because
the
connection
will
be
already
used
and
so
you'll
get
some
rebalancing
promise
with
HTTP
and
request.
Multiplexing
you'd
really
need
a
single
connection
to
send
request
with
a
high
GPS
request
per
second.
B
A
A
A
The
qulity
is
another
very
interesting
property
of
thread.
Balancing
two
examples
here:
if
your
part
you
want
to
connect
to,
if
the
service
you
want
to
connect
to
has
a
pod
running
on
the
same
node,
you
probably
want
to
connect
to
that
node
to
that
pod
on
the
same
node,
rather
than
trying
to
go
on
a
different
record
or
available
user
or
data
center.
So
there's
some
kind
of
cache
hierarchy
very
similar
to
cache
hierarchies.
Where
you
you
want
to
connect
to
the
closest
part,
whatever
the
closest
means.
A
A
A
With
unhealthy
parts
that
can
happen
from
time
to
time
and
when
you
do
have
an
alpha
pod,
you
want
to
remove
the
pot
from
the
pool
of
endpoints
you
you
can
reach
for
that
service
right.
The
kubernetes
provides
some
circuit
breakings
passive
circuit,
braking
by
with
a
la
Venus
and
readiness
probes.
If
communities
detects
your
pod
is
unhealthy,
it
will
remove
it
from
the
service
endpoints
the
list
of
service
endpoints.
A
Now,
let's
have
a
look
at
the
perspective
of
l4s
use.
Layer,
4
versus
layer,
7
and
4
is
all
about
connections.
Have
we
seen
elf
for
infinity
is
not
very
useful.
If
you
think
about
it,
the
only
thing
at
the
l4
layer
you
can
do
for
affinity
is
to
look
at
the
source
and
destination
IP
and
ports
and
try
to
make
some
decision
based
on
that,
but
in
especially
in
the
context
of
macro
statuses
talking
into
your
cluster.
Those
information
doesn't
really
give
you
anything.
A
On
the
other
hand,
as
layer
7
is
all
about
requests,
and
that
allows
you
very,
very
fine,
Granero
T
on
things
you
can
do
first
thing
is
actually
can
do
a
better
load
distribution.
Imagine
a
client
doing
one
request.
A
second
and
the
other
client
doing.
50
requests
a
second.
If
you
they
don't,
they
so
have
two
connections
right
each
clients.
As
one
connection,
if
you
do
that,
a
layer
4
you
end
up
with
one
pod
being
50
times
more
loaded
than
the
other
pod.
A
Affinity
now
can
use
a
lot
more
information
because
you're
passing
the
higher
level
protocol,
you
can
use
HTTP
header.
You
can
use
a
URL
part
of
the
URL
to
be
the
key
to
be
the
identifier
that
is
going
to
route
that
request
to
the
same
endpoint,
and
then
that
enables
you
to
have
a
more
fine-grained
sacred
breaking.
A
For
instance,
you
can
look
at
the
request
responses
and
if
you
see
that
one
endpoint
is
only
giving
500
errors
back,
you
may
want
to
remove
that
endpoint
from
the
set
of
n
points
of
the
service.
Let
it
cool
cool
off
a
bit
and
maybe
try
again
later,
but
even
really
want
that
endpoint
to
to
return
errors
forever.
Everyone:
okay,
let's,
let's
illustrate
the
diversity
that
layer,
seven
load
balancing,
enables
you
the
columns
here
are
the
different
load,
balancing
versions
you
can
find
the
wild.
A
The
rows
are
various
components
proxies
and
at
the
end,
oh
well,
the
service
mesh
resists
you
and
a
client
library
with
gokit.
The
first
two
columns
are
not
very
smart,
but
they
they
do
the
job.
You
have
to
imagine
a
request
coming
into
the
system
and
you
have
to
be.
You
have
to
choose
one
of
the
endpoints.
You
can
choose
that
endpoint
at
random
or
in
run
ribbon
fashion
and
I
will
already
answer
some
that
we
already
provide
some
benefits.
For
instance,
the
one
request
versus
50
requests,
a
second
for
each
of
those
requests.
A
You
can
just
pick
one
one
and
point
at
random
and
I
will
spread
the
load,
so
that's
really
something
the
next
two
columns
or
a
bit
smarter,
and
for
that
they
need
a
bit
more
state
for
endpoint.
In
this
case,
let's,
let's
see
we're
going
to
add
the
notion
of
load
and
the
load
is
the
number
of
requests
currently
being
handled
by
an
end
point.
So
if
the
load
is
12,
that
means
the
endpoint
is
currently
handling
five
requests.
A
What
you
can
do,
then,
is
you,
can
look
at
the
list
of
endpoints
and
pick
up
the
one
with
the
fewer
requests
and
give
the
new
request
arriving
to
this
endpoint?
So
you
have
bit
something
bits
most
much
smarter
than
random.
You
actually
have
more
information.
You
have
the
number
of
requests
days
handling.
So
that's
the
way
I
called
full
scan
list
loaded.
A
You,
you
look
at
the
full
list
of
endpoints
and
you
pick
the
listed
it
on
there's
a
very
interesting
result
that
is
actually
very,
very
broad
and
applies
to
more
than
load
balancing
if
you,
instead
of
looking
at
the
whole
list
of
endpoints,
if
you
just
pick
two
endpoints
at
random
and
choose
the
one
with
the
fewest
requests,
it's
nearly
as
nearly
as
good
as
the
full
scan.
Actually
it's
a
constant
like
the
Big
O
notation,
it's
just
so
constant
as
bad.
A
A
A
And
what
happens
next
is
that
the
proxy,
the
the
load
balancing
is
happening
within
the
proxy
and
the
proxy
will
connect
to
with
forward
the
request
to
another
proxy
somewhere
else
and
then
arrived
to
the
destination
pod
and
at
the
bottom
you
have
the
control
plane,
which
is
this
magical
thing
that
will
actually
go
and
parameter
all
the
proxies
depending
on
what
you
want
to
do-
and
this
is
quite
general,
of
course,
and
wendie's
to
do
lot
more
than
those
balancing
the
whole
traffic
measurement.
Suite.
A
So
this
approach
is
quite
nice,
it's
language
agnostic,
because
it's
a
different
process.
You
can
do
you
don't
have
to
do.
You
can
write
your
application
in
whatever
way
it
works.
The
clients
are
very
simple.
They
don't
have
to
know
anything
about
advancing
anything
about
traffic
management.
You
will
just
work,
that's
quite
a
nice
property,
but
you
have
the
proxy
in
the
database.
All
your
all.
Your
traffic
is
going
through
the
proxy.
It
has
a
bit
of
latency
overhead
and
a
lot
of
a
bit
of
CPU
usage
as
well.
A
The
next
way
you
can
do
this,
the
been
saying
is,
have
pretty
much
the
opposite:
everything
in
the
client,
so
the
client
becomes
quite
heavy.
They.
They
can
directly
connect
to
the
community's
API
server
to
get
the
list
of
endpoints
and
they
have
all
the
logics
within
themselves
to
choose
the
endpoint
using
one
of
the
many
algorithms
we
just
saw
and
I
give
you
quite
some
nice
control,
let's
see.
Well,
there
are
some
drawback.
C
A
No
extra
hope
in
the
data
path,
the
regular
plant,
it's
itself
doing
all
the
things
and
that
surprised
to
pay
to
have
full
control
of
what
you
doing.
If
you
look,
if
you
remember
that
table,
not
every
solution
had
all
the
algorithms.
So
if
you
want
something
very
precise,
then
maybe
you
can't
reach
it
to
be
the
current
solution.
If
you
do
something
in
the
client,
then
you
do
exactly
what
you
want
and
there's
interesting.
A
So,
though,
your
client
go
and
get
an
ask
this
service,
what
endpoint
can
I
use
please
and
that
the
advancing
link
is
all
there,
but
a
client
itself
will
do
the
request.
So
that
gives
you
some
abstraction,
so
we
don't
have
to
do
to
write
ten
libraries
and
remove
the
the
proxy
from
the
data
path.
That's
kind
of
an
interesting
one.
I
haven't
seen
many
many
solutions
with
that,
but
it's
something
I
want
to
explore
for
sure.
A
Okay,
let's
talk
about
a
bit
about
affinity
and
constant
hashing
with
body
loads
which
wasn't
the
abstract
at
least
so
this
is
the
hovering
guessing.
People
are
starting
to
be
familiar
with
that
constant
hashing
is
quite
prevalent
in
this
rigid
services,
it's
everywhere,
but
let's,
let's,
let's
go
through
step
by
step.
Let's
say
you
have
a
hash
function:
xxx
hash,
for
instance,
x-axis
right.
So
this
good
one,
this
32
bit
search
function
and
you
can
represent
those
the
whole
Brittles
of
the
hash
on
the
circle
at
the
top.
A
You
have
0
and
you
go
clockwise
to
the
power
30
with
2
power,
32
minus
1.
Well,
you
can
do
with
that.
Well,
you
have
the,
if
you
remember,
you
have
those
4,
endpoints
they're
identified
by
an
IP
address
and
a
port.
Probably
you
can
hash
these
strings
and
the
result
of
the
hash
ends
up
somewhere
in
that
rat-ass
ring.
So
that's
why
I
did
here
and
they
all
in
the
ring.
Now
we
still
want
to
answer
the
problem.
What
happens
when
a
request
arrived
in
the
system?
A
Where
should
it
go
which
endpoint
sheet
select
so
a
requester
eyes?
You
inspect
it
to
grab
a
key
that
you
want
to
do
you
affinity
with?
So
if
it's
a,
if
you
want
to
route
all
the
clients
requests
to
the
same
endpoint,
you
maybe
have
a
HTTP
header
doing
that,
and
because
you
have
that
key,
that
cares
a
value
and
you
put
it
somewhere
in
the
in
the
ring
and
the
way
it
works.
The
way
you
can
select
and
point
there's
going
to
handle
that
a
request
is
just
to
go.
A
Clockwise
on
the
hash
ring
and
choose
the
end
point
a
is
right
next
to
it,
but
that's
that's
not
really
load
balancing
and
for
the
very
simple
reason,
if
you
have
a
client
doing
the
50
request
with
the
same
key
and
acclaim
doing
one
request
for
the
same
key.
You
have
50
requests
and
either
ending
up
in
the
blue
one
and
maybe
the
one
request
and
ended
up
in
a.
A
Which
is
not
real
burn
thing,
is
it
so
in
arrives
custom
sashings
with
bandit
loads
and
the
idea
around
it
is
to
try
to
avoid
and
point
getting
smashed
by
hotspot
it
by
by
by
a
client
or
a
special
key.
If
you
want
and
the
way
to
define
that
or
the
way
they
define
it
is
to
have
an
invariant
saying,
no
endpoint
can
be
more
loaded
than
a
factor
of
the
average.
A
Maybe
that's
not
entirely
clear,
but
we
will
try
to
clear
it
up.
So
you
have
to
augment
your
ash
ring
with
theta
and
this
this
this
time
it's
the
load
for
each
end.
Point
you
keep
the
load,
that's
Nadia.
We
already
seen
so,
for
instance,
here
the
green
one
has
23
requests.
At
the
moment
there
your
requests
arise
being
hashed
and
up
somewhere
in
the
hash
ring,
and
then
you
have
your
question:
can
this
endpoint
handle
the
request
and
the
way
you
answer
it
is
okay.
A
I
have
100
requests
in
my
system,
which
means
the
average
load
of
each
end.
Point
is
a
hundred
fifty
by
425
and
my
invariant
is
I.
Don't
want
any
endpoint
to
have
more
load
than
25
times
my
constant
that
constant
I
chose
here
1.2
when
the
2000
T
5
is
30,
so
I
don't
want
any
endpoint
of
more
than
30
load.
When
the
request
arrives,
it
should
be
handled
by
the
blue
one,
but
if
the
blue
one
handles
it,
that
means
you
have
30
requests,
+
131
requests,
and
we
just
said
no.
A
A
If
you
can
imagine
here
that
you
have
a
mix
between
affinity,
you
try
to
stick
your
sessions.
If
you
want,
or
you
use
us
to
the
same
endpoint,
but
if
the
endpoint
is
too
loaded,
then
you
go
to
the
next
one
and
I
have
this?
Maybe
that's
better
explain
with
this
kind
of
affinity
scale
that
if
I've
imagined
you
have
described
between
no
affinity
and
persistence,
obviously
random
and
run
rabinin
and
power
of
two
choice
list
loaded
or
no
affinity.
They
didn't
even
look
at
the
data
this
level.
A
On
the
other
end
of
the
spectrum,
you
have
constant
hashing,
which
is
person
just
by
shorting,
where
you
always
end
up
in
the
same
end,
point
well
accepted
the
end.
Points
is
being
removed
from
the
heart
ring,
but
that's
a
common
case
in
the
middle.
You
have
question,
Envy,
bound
alert,
and
because
you
have
this
factor,
you
can
act
on
you
if,
as
a
sig
tensed
to
add
a
constant
tends
toward
infinity,
you
allow
the
system
to
do
you
allow
one
endpoint.
You
have
a
lot
more
load
than
the
average.
A
A
Alright
and
very
very
briefly,
I
want
to
touch
about
locality
and
there's
this
a
very
interesting
expressive:
abstraction
there,
which
is
the
nth
subset
of
endpoints.
A
service
we
said,
is
a
set
of
n
points,
but
what
if
we
could
define
subsets
of
that,
for
instance,
if
you
have
a
Z's
in
your
cluster,
you
can
subset
your
endpoints
by
by
easy.
A
You
want
to
connect
to
the
green
pods,
of
course,
and
you
can
load
balance
between
those
two
if
you
want
to,
and
you
can
think
about
a
lot
of
strategies
to
fall
back
if,
if,
if
the
green
AZ
isn't
isn't
available,
for
instance,
or
to
either
to
load
it
alright,
so
how
hard
can
this
really
be
and
I
mean
so
I've
played
a
bit
with
a
client-side
libraries
on
github
and
the
way
I
tested.
It
is
by
writing
a
small,
reverse
proxy
and
I'm
sure.
A
Everyone
knows
why
the
reverse
proxies,
but
in
any
case
that's
nothing
in
the
middle.
You
can
make
client
requests
to
it
and
we
just
forward
it
to
one
of
the
endpoints
as
a
extracted,
the
constant
hashing
and
Doni
loads
with
the
small
interface
the
first
and
the
first
method
is
really
to
tell
the
ring
waste
the
next.
What
is
the
set
of
endpoints
we
talking
about
and
the
second
two
are
in
the
two
last
beacon
request
and
a
request
or
Sentinel
calls
you
can
put
before
making
a
request
and
after
making
a
request?
A
Basically,
you
want
to
make
a
request.
You
ask
the
load-balancing
corazon.
Where
is
the
end
point?
I
can
actually
have
at
the
end
you
release
this
endpoint
and
that
that's
used
to
handle
to
to
tweak
the
load
for
each
endpoint.
Of
course,
now
the
service
discovery
part
is
just
a
sip,
is
just
a
commence
API
call
you
watch
the
endpoint
object,
you
receive
events
when
the
endpoint
objects
changes
first
and
if
you
scale
down
or
up
your
service,
this
will
change
the
set
of
end
points.
A
So,
and
based
on
those
events,
you
can
update
the
ring
with
the
set
of
n
points
that
you
currently
have,
and
then
the
reverse
proxy
proper
is
based
on
the
ghost
and
a
library
as
proxy,
and
you
can
see
that
first
he
looks
at
a
header
that
contains
the
key.
There
is
the
base
of
your
question
hashing.
It
gets
the
endpoint
for
that
key
it
reader
redirects.
The
request
to
the
right
hand,
point
at
the
end
called
and
request
to
to
decrease
the
load
bit
about
circuit.
A
Breaking
I
haven't
really
done
much
on
in
this
prototype.
It
just
relies
on
the
default
active
communities,
communities
circle
breaking
come.
If
your
pot
is
not
is
not
answering
the
liveness
probe
committees
we
restarted
and,
of
course
it
could
be
enhanced
as
a
tuile
with
the
various
secret
braking
algorithms,
you
can,
you
can
find
around
there's.
There
are
many
yeah.
We
can
definitely
talk
about
this
if
all
right
demo
so.
A
A
No,
where
am
I
so
I,
have
written
small
clients
its
energy
to
debrief
as
well,
that
does
I
can
actually
get
that
better
way
does
it.
He
has
to
go
routines
and
each
code
in
is
sending
a
100
requests
with
a
with
two
different
and
each
go
routine
uses
a
different
affinity.
Key
the
physic
is
are
called
mark
and
Sophie.
Why
not.
A
A
The
first
line
is
mark
ended
up
in
the
in
this
endpoint
17
times,
so
you
can
see
that
you
have
some
kind
of
strong
affinity
in
at
least
one
of
the
end
points
for
for
Marcus,
for
the
mark
is
the
second
one
and
the
rest
is
pretty
bit
among
the
other
end
points,
so
you
get
both
and
I
think
the
constant
here
was
125.
So
you
you
get
you
get
some
behavior
that
looks
like
it's
right.
A
A
Actually,
it
turns
out
that
making
a
clan
library
is
is
fairly
straightforward,
especially
if
you
use
the
cube
the
communities
API
directly
and
that's
actually
a
practical
solution
of
pragmatic,
even
solution
that
works
today,
and
you
can
do
exactly
what
you
want
there
I've
mentioned
using
the
copy
graciously
Green
CPI
is
library,
is
actually
quite
powerful
and
you're
using
your
assertion
engine
as
as
the
base
of
for
free
logic,
something
I
noticed.
That's
maybe
interesting.
A
There's
something
a
bit
weird
about
the
service
ip's,
like
all
of
these
completely
bypasses
a
service
IP,
and
that's
not
very
that's,
not
a
nice
feeling
to
bypass
part
of
the
platform,
and
you
have
solutions
that
are
that
or
building
other
abstractions
we
see.
Are
these,
for
instance,
the
question
is
how
much
of
this
can
be
folded
back
into
cavaties
itself,
to
maybe
try
to
limit
the
fragmentation
around
around
this
kind
of
problematic
it's
priced
on
the
endpoint
subset,
which
is
kind
of
interesting
and
expressive
object.
A
A
That
said,
I
have
promised
my
co-workers
that
I'm
going
to
solve
that
for
our
own
clusters,
so
I
will
have
something
at
some
point
will
be
open
source
if
anyone
is
interested,
I
will
announce
it
somewhere
in
on
our
blog
or
something
I've,
a
bunch
of
references
here.
The
topic
is
actually
very,
very
deep
and
interesting.
C
C
A
A
D
So,
first
of
all,
thank
you
so
much
for
the
speech.
That's
exactly
a
problem
that
we're
facing
right
now.
My
question
is
around
scaling.
So
in
a
scenario
where
load,
balancing
with
affinity,
you
have
a
number
of
endpoints
in
the
ring
and
then
you
scale
the
ring.
The
affinity
will
change
right.
So
have
you
do
you
have
any
suggestion
about
how
to
preserve
this
affinity?
In
case.
A
So
I
think
that
a
very
general
comment
is
you
don't
want
to
only
rely
on
affinity
as
in
the
state
you
build
inside
your
your
your
endpoint
has
to
be
rebuildable.
So
if,
if
your
endpoint
goes
away,
you
can
rebuild
that
state
in
another
endpoint,
that's
a
very
general!
You
know
comment
about
how
you
want
to
organize
your
state's
inside
a
very
highly
dynamic
environment.
A
So
that's
that's
really
the
answer
I
have,
if
you,
if
you
can
it
be
it's,
you
can
rebuild
your
state
if
you
can't,
let's
say
you've
a
service.
For
instance,
if
you
imagine
a
service
that
is
actually
gdb
call
watch
yeah,
you
know
you
have
core
files.
Anyway,
you
have
gdb
to
explore
your
qualified
the
service.
Well,
just
Weather
Service
would
do
it
with
4kg
DB
instance
download
the
co
file
loading
to
gdb,
and
you
give
you
say,
a
prompt
in
this
kind
of
environment.
The
context
is
massive
right.
A
The
context
is,
is
forking
gdb
and
loading
and
if
the
end
point
goes
away,
the
whole
session
goes
away
and
you
can't
rebuild
it
because
you're
GTB
in
that
case,
well,
there's
no
good
solution.
Really
you
can
rebuild
it
the
state
as
well.
That's
fine,
but
you
probably
want
to
have
the
user
to
tell
the
user
something
bad
happened
and
your
you
know
your
restarting
the
session.
So
that's
it
really.
Thank.
D
You
so
much
I'll
be
happy
to
follow
up
with
more
questions
later
about
this
sure
and
I
have
another
one
that
is
not
strictly
related,
it's
about
using
TLS
inside
a
cluster
mm-hmm.
So
in
the
case
where
the
connection
between
service
a
and
service
B
goes
over
TLS,
obviously
other
inspection
is
impossible.
So
you
cannot
use
the
mechanism
for
affinity.
I
mean
such
as
turnaround
that.
D
C
A
B
So
I
think
yeah.
First
of
all,
thanks
for
interest
in
token
presentation
and
I
think
what
previous
question
was
really
about
and
what
other
people
might
be
wondering
office
is
it
kind
of
a
project
you
are
just
starting
with
this
approach
or
any
other
products
exist,
or
maybe
planned
by
some
companies
or
walks
explaining
for
that
horse.
So.
A
There
are
some
macro
services,
libraries
and
talking
about
go,
especially
they
implement
some
of
the
limb
balancing
is
a
City
go
Kate,
go
kid,
has
random
and
run
ribbon,
so
this
could
be
part
of
go
kid.
There's
no!
But
I
have
no.
Yes,
it's
a
project
I'm
starting
and
I,
have
no
concrete
plans
at
this
point.