►
From YouTube: Using Envoy for data aware traffic routing in Azure Service Fabric - Vaclav Turecek, Microsoft
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Using Envoy for data aware traffic routing in Azure Service Fabric - Vaclav Turecek, Microsoft
Learn how Envoy is the key component in solving Service Fabric's unique data-aware traffic routing challenges. With co-located compute and state, service communication needs to be aware of data partitioning and replica placement in addition to the ingress, discovery, and networking challenges typically found in distributed applications and cloud-native architectures.
We'll show you how Envoy is being integrated into Service Fabric to abstract away the complexities of traffic routing and provide a language and platform agnostic service communication layer. As a bonus, we'll also show you how we made Envoy work on Windows.
B
A
So
all
of
the
Azure
infrastructure
as
a
software-defined
infrastructure.
Rather,
when
you
go
to
Azure
and
you
ask
for
a
network
or
you
ask
for
a
VM,
it's
applications
running
on
this
platform
that
answer
that
request
and
provision
those
things
a
lot
of
other
Microsoft
services
like
office
in
tune
dynamics.
All
these
things
kind
of
run
on
this
platform.
So
we
use
it
very
very
extensively
inside
of
the
company
and,
of
course,
like
with
a
lot
of
things
on
Microsoft
today.
That
platform
is
open
source.
So
the
interesting
thing
about
this
platform.
A
It
was
designed
with
States
in
mind,
so
it
came
out
of
the
sequel
org
about
a
decade
ago
when
we
were
developing
sequel
on
Azure,
and
what
we
found
is
that
we
have
a
lot
of
common,
a
lot
of
common
distributed
data
problems
that
we
need
to
solve
across
the
company
and
across
Asia,
and
so
we
built
this
platform
with
a
lot
of
these
data
primitives
built
in
from
distributed
applications.
So
there's,
of
course,
clustering
in
Federation.
This
is
just
a
layer
that
stitches
a
bunch
of
machines
or
VMs
together
and
took
cohesive
cluster.
A
But
the
interesting
things
are,
for
example,
leader
election
consensus,
algorithms,
configuration
of
placement,
all
that
stuff
is
built
and
as
primitives
into
this
platform,
as
well
as
replication
of
data,
quorum,
consistency
and
different
modes
of
consistency,
balancing
keeping
state
balanced
across
different
machines
and
then
finally
partitioning
which
is
just
a
fancy
word
for
data
sharding.
So
these
are
all
primitives
built
into
the
platform
that
we
use
to
build
out.
Other
data
aware
platforms
or
data
bases.
So
anytime
we
have
a
distributed
database
or
any
kind
of
a
data
ingress
service.
A
Like
the
you
know,
the
eventing
systems
are
taken
events
from
IOT
devices,
that
kind
of
stuff
that
all
runs
on
this
platform
makes
use
of
these
primitives.
So
let
me
show
you
what
this
actually
looks
like
when
the
platform
starts
up
this
kind
of
a
self
bootstrapping
platform.
It
starts
up.
You
take
a
set
of
nodes
which
are
just
VMs
or
machines,
and
that
forms
a
cluster
services
that
are
written
for
this
platform
that
get
deployed
to
it.
They
take
two
forms.
A
The
kind
of
simple
one,
which
is
something
that
we're
probably
used
to
in
most
places,
is
what
we
call
a
stateless
service
and
that
just
means
that
every
instance
of
your
code
is
running
the
exact
same
way.
Doing
the
exact
same
thing.
This
could
be
anything
like
a
node.js
application,
an
asp
net
application.
It
doesn't
really
matter.
This
is
kind
of
things
like
a
web
farm
or
something
like
that.
Everything
is
identical.
A
So
that
one
I'm
not
gonna,
focus
too
much
on
today,
because
that's
that's
sort
of
been
covered
in
a
lot
of
the
other
talks,
how
this
kind
of
thing
works.
This
is
this
is
what
I
want
to
talk
about
today.
This
is
the
the
staple
version
of
that.
So
when
you
run
your
code
here,
every
instance
that
runs
on
a
separate
node
isn't
actually
created
equal
because
it's
holding
the
data
in
inside.
A
So
in
this
case,
you'll
have
multiple
replicas,
but
only
one
of
those,
for
example,
will
be
a
primary
replica
that
can
take
write
operations,
whereas
the
other
than
secondary
replicas,
which
are
just
getting
States
replicated
to
them
and
those
don't
accept,
write
applications
or
write
requests.
Rather
so,
when
you
take
those
together
that
forms
a
partition
and
so
that
partition
is
kind
of
a
collection
of
replicas
or
a
collection
of
endpoints,
and
then
you
can
deploy
multiple
these
partitions
and
the
partitions
you
can
think
of
that
as
a
database
shard.
A
So
each
one
will
have
a
subset
of
the
total
amount
of
data
when
you
can't
fit
data
on
a
single
machine
kind
of
idiot,
and
so
you
take
that
whole
thing
put
it
all
together.
That's
what
makes
a
stateful
service,
so
you
can
kind
of
see
the
little
bit
of
a
difference
how
that
runs
versus
just
a
stateless
one
and
then
what
you
have
inside
each
one
of
those
replicas
is
the
application
code
or
the
service
code.
A
I'll,
probably
use
words,
application
service
interchangeably
here,
that's
running
next
to
the
data
that
its
operating
on,
so
that
state
is
kept
in
memory
optionally,
written
to
disk
all
that
is
and
replicated
out
to
multiple
machines
and
that
provides
high
availability.
So
another
way
to
think
about
this
is,
if
you
were
running
a
net
CD
cluster
or
a
Redis
cluster,
or
something
imagine
if
your
code
was
running
inside
of
that
cluster
rather
than
making
a
remote
calls
to
it,
you're
running
inside
and
next
to
the
data
that
it's
storing.
A
So
that's
that's
kind
of
the
idea.
Each
one
of
those
systems
obviously
has
to
provide
a
way
to
route
data
to
the
appropriate
place
where
the
data
is
located,
depending
on
the
request.
So
usually
a
client
library
will
do
something
like
that-
a
thick
client
library,
something
that
understands
how
to
find
those
endpoints
and,
of
course,
we
have
the
same
kind
of
thing
in
service
fabric.
But
what
we're
doing
now
is
replacing
a
lot
of
that
with
Envoy
and
using
that
as
the
main
mechanism
to
route
requests
and
socially
exactly
how
that
works.
A
So,
and
this
is
what
we
say-
or
this
is
what
I
mean
when
I
say
has
to
be
data
aware,
so
there
are
four
main
they're
four
main
parts
that
I
want
to
talk
about
today.
The
first
is
obviously
service
discovery
and
routing.
This
is
probably
the
main
piece.
How
do
you
find
where
to
write
where
to
route
data
to,
and
then
how
do
you
actually
do
the
routing
and
this
actually,
this
idea
of
being
data?
A
Where
affects
every
part
of
traffic
writing,
so
even
load
balancing
how
you
distribute
load
across
different
replicas
or
even
different
partitions
is
affected
by
this.
How
health
checking
is
done,
how
health
monitoring
is
done
and
even
down
to
versions
and
upgrades
has
to
be
handled
a
little
bit
differently
when
the
data
and
the
compute
or
co-located
in
the
same
process
or
in
the
same
container
space?
A
So
I'll
start
with
a
simple
example
just
looking
at
these
as
stateless
services.
So
imagine
these
are
web,
endpoints
or
web
applications
in
envoy
terminology.
These
would
each
be
a
cluster.
Each
replica
would
be
an
end
point
inside
that
cluster,
and
so,
when
you
deploy
something
like
this
on
service
fabric,
it's
really
no
different
than
what
you
would
have
on
any
other
platform.
There's
the
discovery,
the
XDS
discovery,
services
for
envoy
downstream
talks
through
envoy.
A
In
this
case
you
can
think
of
envoy
as
a
knowed
agent,
where
there's
one
instance
running
per
node,
you
can
do
the
same
thing
as
a
sidecar
model.
It
doesn't
really
make
a
whole
lot
of
difference
just
to
just
for
simplicity,
we'll
display
this
as
a
node
agent
for
now
the
the
little
orange
box,
then
at
the
bottom
there
that's
a
system
services
this.
This
is
a
specific
thing
to
service
fabric.
This
is
where
service
discovery
actually
happens.
A
The
system
services
themselves
are
just
a
set
of
stateful
services
that
run
on
the
platform
that
manage
the
cluster
and
they
manage
the
applications.
So
they
know
those
system
services
know
what
nodes
are
in
the
cluster.
What
applications
are
running
where
applications
should
be
placed
how
replicas
should
be
placed
they
detect
failure,
etc.
They
also
act
as
the
service
discovery
mechanism.
A
So
what
you
don't
see
here
is
zookeeper
@cd
or
anything
else,
because
the
again
the
platform
itself
is
meant
to
be
a
data
platform,
so
all
of
the
stages
kept
on
the
platform
yourself,
there's
no
external
databases
here
and
in
fact
there
can't
be
because
we
develop
our
databases
on
this,
so
you
wouldn't
have
a
sequel,
backed
by
a
sequel,
for
example.
So
so
this
is
what
it
looks
like
when
it's
stateless.
A
So,
typically,
the
way
this
works
there's
this
there's.
This
partition
resolution
pattern
that
we
follow
on
the
platform
where
you
take
some
data
from
the
request.
If
it's
an
HTTP
request,
you
take
something
from
the
path
or
from
the
header
or
a
query
string,
and
you
use
that
to
transform
it
into
a
partition
key,
which
is,
you
can
think
of
it
as
a
hash
table.
So
you
take
some
value,
you
hash
it.
That
tells
you
what
bucket
of
the
hash
table
it's
in
you
send
the
traffic
there.
A
Once
you
determine
that,
then
you
can
look
up
the
location.
Look
at
the
endpoint
where
that
data
is
so
that's
the
service
discovery,
part
of
it.
Then
you
ask
service,
carry
I,
have
service
am
going
to
with
partition.
Key
five
tell
me
where
the
end
point
for
that
is,
and
then
it'll
respond
with
that
and
then
finally,
you
forward
the
request.
A
So
again,
typically,
this
was
done
in
in
these
kind
of
heavy
clients
the
clients
that
would
do
all
this
work
on
the
client
side,
which
has
a
number
of
challenges
associated
with
it,
of
course,
which
I
think
we
heard
in
one
of
the
previous
talks.
You
have
to
write
it
in,
however
many
languages
that
you
want
to
support,
it's
difficult
to
keep
it
updated
the
clients
doing
a
lot
of
the
work
etc.
A
So
what
we
did
an
envoy
to
make
this
work
so
here's
an
example
of
a
of
a
partitioning
scheme
that
uses
a
range
partition
and
that
just
means
that
the
key
range
very
partition
is
just
some
some
integer
range.
So
for
that,
for
example,
users
as
a
zero
through
nine
will
go
to
partition
one
and
so
on
and
so
forth.
A
So
what
we
did
is
we
added
this
range
match
header
to
envoy.
This
is
this
is
one
of
our
small
contributions
that
we
did,
and
this
just
allows
you
to
match
a
header
to
a
range
of
integers.
So
rather
than
say,
a
header
equals
x
or
header
matches
some
reg
X.
You
say
if
a
header
falls
within
a
numerical
range,
that's
a
match
go
to
that
cluster,
and
so
this
allows
the
client
then
to
send
request.
A
Like
so
I
can
say,
go
to
the
application
with
this
domain
name
on
this
path
and
then
here's
the
partition
key
and
all
that
doesn't
envoy.
Is
it
just
matches
this
header
range
match
and
then
gets
routed
appropriately,
and
so
we're,
of
course
treating
each
partition
here
as
an
upstream
cluster
with
a
set
of
endpoints,
and
so
we're
just
trying
to
figure
out
a
way
to
send
a
request
to
the
appropriate
cluster,
so
this
works.
A
B
A
Know
even
more
intimate
details
about
it,
for
example,
it
has
to
know
the
upstream
partitioning
scheme,
so
the
range
match
is
just
one
way
that
you
can
do
partitioning.
You
can
do
named
partitions
that
are
an
exact
name
match.
For
example,
the
downstream
would
have
to
know
this
about
the
upstream.
The
downstream
has
to
know
if
it's
using
if
it
is
using
a
range
match,
it
has
to
know
the
range
of
keys
that
the
upstream
is
chosen.
A
So
the
upstream
can
choose
any
range
of
keys
here,
it's
0
to
29
and
it
could
be
1
to
10.
It
could
be
in
might
in
2
min
in
max.
The
downstream
then
has
to
know
all
this
about
the
upstream
and
the
worst
yet
about
this.
I
think
is
that
the
downstream
here
now
has
control
over
the
data
distribution
of
the
upstream.
So
if
I'm
sending
requests
to
an
upstream
and
I'm
deciding
what
the
hash
key
is,
that
means
I'm
also
deciding
how
data
is
distributed
in
the
upstream
service.
A
So
I
could
overload
partition
1
pretty
easily
by
saying
everyone
just
falls
into
partition,
1
everyone's
everyone's
0
and
that
kind
of
ruins.
The
upstream,
so
what
we
really
want
to
do
is
we
want
to
take
that
responsibility
out
of
the
downstream
transfer
that
over
where
the
upstream
has
control
of
it
by
putting
that
into
the
proxy
in
between
and
so
the
way
we
the
way
we
did.
This
initially
was
with
a
custom
HTTP
filter,
where
the
upstream
service
can
now
send
some
of
that
partition.
A
Config
information
through
the
system
services
and
into
the
discovery
services
for
Envoy
to
consume,
where
that
HTTP
filter
can
then
consume
that
information.
So
it's
not
it's.
It's
basically
kind
of
template
parameters,
so
the
upstream
will
say
something
like
my
is
my
partitioning
range
is
this:
my
partition
scheme
is
that
and
then,
more
importantly,
here's
the
value
I
want
you
to
take
from
the
HTTP
request
and
here's
the
hashing
algorithm
I
to
run
against
it
and
that
hashing
algorithms
usually
the
same
thing.
It's
usually
FN
V
ash
or
something.
A
It's
basically
rewriting
the
routing
rules.
On-The-Fly
is
kind
of
what
it
is.
You
can
do
that
you
have
to
clear
the
routing
cache
first.
Otherwise
it
won't
work.
But
as
long
as
you
do
that
it'll
reapply
the
rules
and
then
route
accordingly,
we
found
an
even
better
way
to
do.
This,
though,
is
to
just
use
the
built-in
Lua
script
filter
that
already
exists
that
allows
us
to
just
index
some
configuration
that
way.
Xvs
basically
writes
the
Lua
script
as
it
needs
to
be
when
it
gets
to
envoy
envoy,
execute
the
Lua
script.
A
Does
the
hash
grabs
value?
Does
the
hash
puts
in
the
header,
and
then
it
gets
routed
to
the
right
place?
So
that's
how
we
pick
that's
why
we
pick
what
cluster
to
go
to
what
upstream
cluster
to
go
to
the
partition.
Basically,
the
next
part
of
this
then
is
load
balancing
once
we've
once
we
decided
on
the
upstream
cluster,
so
load
balancing
in
the
stateful
world
is
kind
of
a
question
of
consistency
versus
availability,
because
again
not
every
endpoint
is
treated
equal,
and
so
this
is
this
little
picture
on
the
right.
A
I'll
keep
that
up
throughout
this,
so
that
you
can
kind
of
kind
of
see
how
this
works.
So
it's
really
only
the
primary
replicas
here,
that's
accepting,
read
or
write
operations.
The
secondary
replicas
are
just
there
to
keep
copies
of
the
data
for
high
availability,
so
for
a
full,
consistent
model.
You
would
only
ever
read
and
write
to
and
from
the
primary.
A
For
higher
availability,
you
can
read
from
secondaries,
but
at
the
cost
of
consistency,
because
the
secondary
is,
although
they
have
the
same
data,
it
is
a
quorum
Act
based
system.
So
when
you,
when
you
make
a
right
to
a
primary
replica
that
that
transaction
commits
when
a
majority
of
secondaries
have
acknowledged
that
transaction,
but
not
necessarily
all
of
them.
So
in
this
case,
where
I
have
three
replicas,
one
of
those
will
probably
be
stale
some
of
the
time
in
rare
cases
you
can
be
even
worse.
A
You
can
get
false
progress,
so
you're
reading
values
from
a
future.
That
hasn't
happened
yet
or
that
will
never
happen,
and
then
it
gets
rolled
back.
So
so
in
this
kind
of
in
this,
in
this
lower
consistency,
but
higher
availability
model,
you
can
send
reads
to
the
secondaries.
So
question
is:
how
do
you
do
this?
A
Basically,
what
you
want
is
I
have,
for
example,
an
HTTP
API
like
API
users
and
I
want
to
send
all
my
rights
to
just
that
one
primary
endpoint
in
this
cluster,
so,
for
example,
my
puts
post
deletes
are
going
to
go
there
and
then
my
gets
on
the
same.
Api
can
go
to
any
one
of
these
replicas.
So
there
is
a
pretty
cool
way
to
do
this
in
Envoy.
That
doesn't
really
require
us
to
do
any
additional.
You
know
additional
work
here.
This
is
the
load
balance
or
subset.
So
we
also
heard
about
earlier.
A
This
is
a
very
cool
thing
you
can
do
so.
Essentially,
what
we
do
is
we
set
up
these
subset
selectors,
which
are
just
a
set
of
keys
that
you
can
then
apply
in
your
endpoints,
so
I'm,
basically
tagging
each
endpoint
with
this
value
to
say
this
endpoint
is
tagged
as
a
primary.
These
other
endpoints
are
tagged
as
the
secondaries.
A
A
Unfortunately,
in
a
system
like
this,
where
only
one
of
those
endpoints
is
a
right
endpoint,
when
that
thing
goes
bad,
there
really
isn't
much.
You
can
do
at
the
proxy
level,
so
you
just
really
have
to
rely
on
the
platform
to
fail
that
thing
over
as
fast
as
you
possibly
can
again
it's
we
can
be.
We
can
be
as
good
as
we
possibly
can
about
failing
things
over
detecting
failure
and
failing
a
primary
over
to
another
note
as
fast
as
we
possibly
can,
but
it's
never
gonna
be
perfect.
A
Application
code
can
be
slow
to
shut
down.
It
can
be
slow
to
start
back
up.
So
again,
that's
what
that
availability
versus
consistency
trade-off
comes
in.
At
the
very
least,
in
this
situation
you
can
make
use
of
secondary
reads,
but
there's
nothing
you
can
really
do
about
writes.
Unfortunately,
you
can't
really
go
into
a
different
partition,
because
your
data
is
not
going
to
be
there.
A
So
the
only
thing
you
can
really
do,
then,
is
just
let
the
system
fail
it
over
to
a
different
node
and
hope
that
it
comes
back
up
same
deal
with
versioning
and
upgrades.
This
is
also
another
difficult
part
where
the
proxy
part
of
this
in
us
in
the
in
this
kind
of
stateful
world
is
only
limited
and
how
much
you
can
do
and
I'll
show
you
why
exactly
this
works.
A
So
if
you
wanted
to
do
a
side-by-side
test
of
a
service
like
I,
have
v1
stood
up,
I'm
gonna
stand
up,
v2
and
then
I'm
gonna
send
you
know
10%
of
my
traffic
to
v2
to
test
it
out.
Unfortunately,
it
doesn't
work,
because
all
your
data
is
still
in
that
v1
instance
remember,
they're,
all
co-located,
so
the
data
is
sitting
next
to
next,
the
next
to
the
compute.
So
you
can't
really
do
it
this
way,
you
can't
really
do
the
side-by-side
gradually
move
traffic
over.
A
So
the
only
way
this
really
works
is
with
in-place
rolling
upgrades
and
that's,
of
course,
built
into
the
platform
here.
One
of
the
places
where
the
proxy
can
Brown
boy
can
help
is,
since
this
is
a
very
cooperative
effort,
as
you
roll
through
upgrade
domains,
you're
taking
down
one
machine,
doing
the
upgrade
taken
out
of
the
machine
and
so
forth.
Every
time
you
do
that,
you
need
to
be
able
to
drain
all
those
requests
out
and
make
sure
all
those
transactions
finish.
A
So
that's
the
only
thing
we
can
do
there
for
testing
instead
of
instead
of
saying,
I'm
gonna,
move
10%
of
my
traffic
over
and
test
it
out.
Shadowing
is
actually
one
technique
that
we
can
use
here,
and
so,
basically,
what
this
means
is
that
a
hundred
percent
of
the
traffic
is
still
going
to
be
one
I
stood
up,
beat
you
next
to
it.
None
of
my
data
is
there,
but
I
can
copy
five
or
ten
percent
of
the
traffic
over.
A
So
all
my
users
are
still
sitting
on
v1,
but
I'm
just
shadowing
some
of
the
traffic
over
that
allows
me
to
run
some
integration
tests,
all
in
that
version
with
real
user
traffic,
but
without
actually
disrupting
the
services
that
they're
running
on
okay.
So
those
were
just
a
quick
crash
course
on
some
of
the
issues
that
we
have
with
data,
where
traffic
routing
and
how
envoy
helps
to
solve.
At
least
some
of
these
problems.
I
wanted
to
spend
just
a
little
bit
talking
about
the
work
we
did.
A
Porting
envoy
over
to
Windows,
just
in
full
disclosure
earnest.
This
wasn't
me
doing.
The
work
week
borrowed
permanently
a
developer
from
the
C++
compiler
team,
the
B
C++
compiler
team,
so
I'm
mostly
just
telling
you
what
but
Connie
the
things
that
you
went
through
and
it
turns
out
most
of
on
was
actually
he
actually
wasn't
a
whole
lot
of
work
at
C++.
So
a
lot
of
it
was
OS
agnostic
and
portable
over
for
the
parts
that
we
did
have
to
make
changes.
It
was
about
30%
of
it.
A
A
Luckily,
there
was
a
lot
of
help
from
the
community,
which
was
we're
very
thankful
for
that.
We
didn't
get
very
far
on
that
one.
So
so
just
a
couple
of
just
the
Cole
things
that
were
interesting
error
codes,
for
example.
There
are
some
error
codes.
It
just
didn't
exist
on
Windows
so
that
we
had
to
go
and
define
those.
There
were
a
whole
lot
of
error
codes
that
we
had
to
redefine
to
work
with
a
Windows,
socket
API
that
were
different.
That
switch
statement
on
for
quite
a
while.
A
This
one
was
really
fun
so
close
on
Linux.
You
have
this
close
method
that
exists
in
actually
both
places.
Only
some
windows.
It
takes
a
file
descriptor
on
Linux.
This
close
the
sockets
and
anything
else
is
scrap
five
by
a
file
which
is
pretty
much
everything
on
Windows.
We
don't
describe
everything
as
a
file,
so
this
doesn't
actually
close
sockets
on
Windows,
but
it
does
still
compile
and
still
runs
and
it
doesn't.
There
are
no
errors.
So
we
ran
this
the
first
time
and
we're
like
we
have
a
lot
of
like
just
sockets
hanging
around.
A
This
is
really
weird,
so
there's
actually
a
different
API
and
Windows
to
close
sockets.
It's
called
a
closed
socket
and
it
takes
the
socket.
The
other
thing
was
live
event
live
event,
doesn't
work
too
great
on
Windows,
the
the
problem
is,
there's
no
edge
triggering,
doesn't
know
what
it
is.
If
you're
not
familiar
with
edge,
triggering
it's,
this
idea,
where
just
real
quick
when,
when
the
socket
buffer
is
full,
for
example,
you
want
to
write
to
it.
You
want
to
get
notified
when
there's
space
in
the
buffer.
A
So
when
that
value
changes
now,
there's
buffer
space
available
hate
go
write,
something
to
the
socket.
If
you
don't
have
anything
to
write,
say
I,
don't
have
anything
to
write
and
then
and
then
live
event
leaves
you
alone
on
Windows.
It
just
keeps
bugging
you
it's
like!
No
there's,
there's,
there's
a
user
space
right,
some
that
there's
space
there
space,
so
that's
called
level
triggering,
and
so
just
keep
doing
that.
So
when
it's
actually,
when
the
when
envoy
is
idle,
it's
eating
up
a
ton
of
CPU.
A
So
when
we
initially
launched
this
ingress
gateway
component
that
uses
envoy
into
service
fabric,
we
ran
I.
Think
two
instance
of
it
and
immediately
a
CPU
is
just
pegged
up
to
hundred
percent,
and
it
was
because
of
this
issue.
So
we
we
put
in
a
workaround
that
kind
of
it
simulates.
What
it's
triggering
does
not
perfectly
the
real
long
term.
Solution,
of
course,
is
to
use
IO
completion
ports.
We
haven't
done
this
work
yet
because
this
is
a
much
much
much
bigger
task
to
take
on.
A
A
A
They're
hard
to
write,
we,
we
wrote
it
on
the
HTTP
sis
kernel
module,
which
is
what
runs
IAS
underneath
performance-wise
we
got
about
on
par,
but
there
was
a
lot
missing
there.
So
we
did
look
at
a
lot
of
a
lot
of
the
proxies
that
are
available
out
out
in
the
open
source
space
and
just
out
in
the
community
own
way
was
by
far
our
favorite.
Just
a
few
reasons
why
one
it's
c++
service
fabric
itself
is
about
two
million
lines
of
C++
code.
So
it's
we
like
C++
we're
pretty
familiar
with
it.
A
We
didn't
want,
have
to
go
out
and
learn
a
go
or
anything
else.
So
this
is
nice,
and
also
portability
across
across
operating
systems
is
helped.
Quite
a
bit
number
two
is
fast,
it's
very
very
fast.
This
is
very
important
for
us.
The
extend
ability
is
super
super
nice
being
a
little
write
filters
and
plug-in
custom
filters
was
a
huge,
huge
part
of
this,
and
we
obviously
need
a
little
bit
of
extend
ability,
although
we
didn't
have
to
extend
it
as
much
as
we
thought.
A
We
would
because
it's
very
feature-rich
so
there's
already
a
lot
in
there
that
we
can
use.
So
this
is
actually
very
good
and
last
but
definitely
not
least,
the
community
is
fantastic,
a
lot
a
lot
of
great
people
working
on
this,
and
it
was
super
fun
to
actually
do
even
just
a
little
bit
of
the
contributions
we
did
was
real
good
time.
So
that's
all
I
have
on
how
we
did
a
little
bit
of
envoy
work
in
service
fabric
over
on
Azure
time
for
questions,
good
time,
questions.
A
Yeah,
so
so
the
first
part
that
question
was
a
lot
of
the
recent
Windows
porting
work
that
has
been
pushed
back
upstream
has
been
done,
my
pivotal
they
from
what
I
understand.
We
have
to
confirm
them,
but
from
I
understand
they
picked
up
essentially
where
we
left
off.
So
we
forked
envoy
initially
over
an
Arg.
We
did
a
lot
of
the
kind
of
front
work
on
getting
things
just
to
run
Windows.
They
picked
it
up
from
there
and
made
the
build
system
work
mainly
and
then
I
think
they're
doing
some
additional
work
and
then
pushing.
A
Kind
of
picked
up
where
we,
where
we
stopped
initially
when
we
got
stuck
on
the
build
system,
yeah
I,
don't
think
we're
gonna
fork
and
continue
to
use
our
own.
We'll
definitely
want
to
have
just
one,
and
hopefully
we
can
get
back
to
helping
them
out
with
that
as
well.
Pretty
soon.
A
second
question
was:
if
we've
looked
at
using
libuv
instead
of
live
event,
we
did
briefly.
A
That
was
another
thing
that
we
could
do
so
we
actually
use
Lemieux
V
pretty
extensively
on
a
different
web
server
that
we
ran
a
Microsoft
called
kestrel,
which
is
the
web
server.
That
runs
an
asp
net
core,
so
we
do
have
quite
a
bit
of
experience
with
libuv
I
think.
Ultimately,
it's
probably
going
to
come
down
to
where
we
can
do.
I
Oh
completion,
ports,
the
easiest,
but
we
haven't
looked
too
extensively
until
the
movie
yet,
but
that
that's
them
that's
an
opportunity
as
well.
A
I
see
so
the
question
is:
how
do
we?
How
do
we
get?
How
do
we
do
right?
Read
your
own
rights.
So
if
there
are
requests
going
into
the
secondary
someone
up,
it's
the
primary,
how
do
we
make
sure
they
get
the
same
value?
There's
no
guarantee
there
if
you're
reading
from
secondaries.
So
again
it's
it's!
You
lose
out
on
that
that
level
of
consistency.
A
If
you're
reading
from
secondaries
generally,
we
discourage
it
for
the
most
part,
just
because
I
think
it
comes
with
a
lot
of
surprises
like
that,
so
we
usually
say
at
least
if
you're
talking,
if
you're
trying,
if
your
goal
is
to
scale
things
out
like
I'm
overloading
one
machine
and
I
need
to
read
four
more
machines.
The
way
to
do
that
is
that
partitioning
concept
break
your
data
up
into
smaller
chunks,
where,
as
reading
from
secondaries,
is
only
about
availability,
but
we
don't
have
guarantees
like
that.
A
A
Yes,
a
question
catch
yeah,
so
the
question
is:
does
the
if
the
primary
comes
back
up?
Does
that
the
same
endpoint
as
it
did
before?
Does
the
XDS
have
to
update
it?
It
depends
so
you
can
do
it
either
way,
so
you
can.
It
depends
on
if
you've
assigned
static,
IP
addresses,
which
isn't
something
that
most
of
the
time
is
done.
Usually
it's
a
completely
different
point.
It
comes
up.
A
Node,
it's
got
a
new
IP
address;
it
might
even
have
a
different
port
if
you're
assigning
ports
dynamically,
so
so
in
that
case,
yeah
that
what
happens
is
when
that
primary
comes
up
on
another
node
that
gets
reported
up
to
the
system,
services
system,
services,
update,
XTS
and
that
gets
pushed
back
down
on
where
then
it
updates
there's
a
little
window.
There,
though,
there's
a
small
window
where
there's
nothing
yeah
right
behind
the
pillar.
There.
B
A
Question
is
if
one
of
the,
if
one
of
the
partitions
is
overloaded,
so
too
many
users
end
up
in
one
partition,
they
end
up
overloading
a
node.
How
do
we
handle
that?
We
don't
have
a
good
way
to
handle
splitting
and
merging
partitions
today,
so
taking
taking
a
partition
splitting
it
up
into
other
places.
The
one
way
you
can
do
that
is
instead
of
using
a
range
partition,
you
use
a
name
to
partition,
so
the
partition
just
belongs
to
a
name.
So,
for
example,
you
have
50
States,
you
can
have
50
partitions.
A
Each
has
a
name
of
a
state.
You
could
do
that
way
and
then
you
can
add
more
partitions
and
then
kind
of
rearrange
how
the
routing
happens,
but
there's
no
way
to
then
merge
it
back
together
if,
if
the
hotspot
dies
down
or
you
lose
users,
so
that's
that's
a.
We
call
that
a
science
fiction
problem,
because
over
years
of
research,
we
haven't
been
able
to
figure
out
a
good
way
to
do
a
split
and
merge
that
those
partitions
win
when
it's
the
application,
that's
actually
responsible
for
determining
the
distribution
in
the
first
place.
A
So
if
it
was
a
completely
closed
system,
where
it
is
always
the
same
distribution,
we
could
probably
manage
that,
but
because
it's
totally
random
code
that
someone
wrote
that
does
the
distribution
and
the
traffic
routing
it's
harder
to
do
it
that
way,
I
think
with
when
envoy
is
actually
doing
the
work
of
routing
and
it's
programmable
by
the
upstream
service.
The
more
and
more
constraints
we
can
put
on
that
the
more
realistic
it
becomes.
So
this
actually
helps
out
with
that
a
little
bit,
but
we're
not
quite
there
yet.
A
Okay,
so
if
you
wanted
to
go
out
to
a
different
data
store
to
figure
out
which
partition
to
route
to
and
using
Lua
to
do
that
haven't
tried
it
yet,
but
I
believe
that
that
should
be
possible,
because
in
Lua
you
can
at
least
make
some
requests
out
before
you
make
a
decision.
So
we
did
actually
look
at
that
because
of
one
of
the
proxies
we
wrote
earlier.
Does
something
like
that?
A
It's
sort
of
on
demand
so,
rather
than
feeding
the
proxy
a
bunch
of
information
right,
here's
all
the
endpoints
and
your
setter
out
them
all.
It
would
happen.
A
request
would
arrive
of
the
proxy.
You
would
then
go
make
a
quick
call
out
to
resolve
the
endpoint
to
come
back
and
then
complete
the
request.
So
you
can
do
that
with
Lua,
but
we
haven't
tried
yet
so
I,
don't
know
what
the
performance
implications
that
are
should
be
possible
to
do,
though,
all
right,
I
think
that's
I,
think
we're
good
thanks.