►
From YouTube: Porting Envoy to Windows; A Progress Report - Yechiel Kalmenson & William A. Rowe, Jr., Pivotal
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Porting Envoy to Windows; A Progress Report - Yechiel Kalmenson & William A. Rowe, Jr., Pivotal
Envoy is a high-performance open source edge and service proxy that makes the network transparent to applications. As of now Envoy is only available on Linux, and that’s a big blocker for Windows teams who want to migrate their monolithic apps to more service-oriented architectures. Last year a team at Pivotal started working with Microsoft on making Envoy on Windows a reality. This talk will give a progress report on the work being done: * An overview of the history of the project. Starting with the work done by Microsoft, contributions to upstream so far, and what we have left. * Some of the challenges the team faced and how they overcame them. For example, the workarounds we employed to get a working Windows environment for Envoy, and some of the performance issues which still need to be solved. * What the team is currently working on and what the outlook for the future is like.
https://sched.co/UacB
A
A
There,
why
isn't
this
working
they're
just
out
for
the
brief
outline,
what
the
talk
was
gonna
cover,
we're
gonna
start
off
with
an
introduction
who
we
are
who's
pivotal
is
what
is
envoy
we're
gonna
go
on,
describe
the
issues
that
we've
encountered.
What
was
solved
already,
what
still
needs
to
be
solved?
Where
are
we
holding
and
then
we're
going
to
look
at
our
outlook
into
the
future?
A
So,
just
as
a
brief
introduction,
my
name
is
a
heel
I'm,
a
engineer
at
pivotal
and
the
windows
containers
team
were
part
of
that
includes
a
container
networking
I've
been
on
the
effort
to
port
on
both
windows
for
about
a
year.
Now
you
can
find
me
on
social
media
and
on
the
envoy
slack
at
UK,
okay
and
I'm.
B
Bill
otherwise
known
as
W
row
and
have
been
doing
porting
software
for
a
very
very
long
time
was
part
of
the
making
httpd
run
effectively
on
Windows
and
make
that
a
first-class
operating
system.
That's
kind
of
what
we
want
to
see
happen
here
with
the
Envoy
project
that
it's
equally
effective
when
it's
running
on
under
windows,
great.
A
So
what
is
envoy
come
with
the
assumption
that
most
people
here
are
familiar
with
it
somewhat,
but
at
the
most
basic
level
envoy
is
a
proxy
server.
It's
used
as
an
abstraction
that
allows
the
network
to
be
transparent.
The
end
users,
as
Matt
client
explained,
and
his
talk
in
this
instruction
last
year,
the
goal
of
on
ways
to
be
an
abstraction
that
all
has
higher
layer
systems
to
be
built.
That
makes
the
network
transparent
to
the
end
users
and
just
a
brief
timeline
of
what
we're
on.
A
We
came
from
Amelie
started
as
an
internal
project
that
lift
back
in
2015
Matt
Matt
Klein
started
working
on
a
proxy
server
which
he
originally
tried,
calling
lyft
proxy
lyft
proxy,
but
then
voice
sounded
cooler.
Then
in
2016
the
Amelie
project
became
open-source
people
start
contributing
and
then
in
July
2017
when
pivot
all
started.
Looking
for
a
solution
for
container
to
container
Arab
integrity
problem,
we
were
looking
towards
envoy
as
a
solution
and
I
always
had
perfectly
on
our
Linux
containers.
A
Unfortunately,
envoy
did
not
have
any
support
for
windows
still
doesn't,
and
we
started
looking
around
to
see
if
anyone's
discussing
it.
If
there's
any
thing
we
found,
there
was
an
issue
open
I'd
get
up
already,
it
seemed
the
Microsoft
had
started
doing
some
work
and
we
got
in
touch
with
the
avila
community
and
the
Microsoft
contributors
to
see
if
we
can
collaborate
and
help
things
along.
So
in
the
summer
2017
we
had
a
few
meetings
between
pivotal
convoy
and
Microsoft
to
try
to
hammer
out
the
approaches
divide.
A
A
So
we
start
working
on
the
issue,
though,
in
parallel
with
Microsoft,
eventually,
Microsoft
had
to
dedicate
the
resources
elsewhere
and
in
July
2018.
We
finally
had
a
patched
up
version
of
basil
that
was
able
to
work
on
Windows
and
we
were
able
to
compile
Microsoft's
branch
on
Windows
and
we
started
our
work
of
slowly
slowly
up
streaming.
That
patch
back
to
the
Envoy
community,
then
the
work
went
in
bursts
on
and
off
in
March
this
year,
Bill
joined
the
effort
and
started
moving
full
steam
ahead.
A
We've
built
up
a
lot
of
momentum
recently
Microsoft
offered
to
jump
back
in
and
really
help
us
out
with
some
of
the
thorny
issues
around
and
we're
really
hoping
that
within
a
few
months,
envoy
will
be
working
on
Windows
just
to
describe
what
pivotal
is
use
case.
What
we
needed
envoy
for,
in
short,
at
sharp,
where
we
need
to
solve
our
our
routing
integrity
problem
for
container
to
container
networking
where
the
issue
is
we're
a
we're
in
app
crashes
and
or
our
scheduler
schedules.
A
Another
app
on
the
same
container
before
our
router
gets
notified
traffic
will
still
go
to
the
old
container,
even
though
it's
on
a
different
app.
Now,
that
is
not
a
major
problem,
there's
only
last
for
a
few
seconds,
but
for
companies
that
have
higher
security
requirements.
Routing
integrity
is
very
important,
and
this
is
a
very
big
blocker
for
many
of
our
customers.
Envoy
would
work
by
getting
it
as
a
sidecar
into
the
container
and
make
sure
that
routing
in
and
out
of
the
container
and
to
other
containers
is
remains
integral.
A
So
whatever
happened
already,
what
have
we
done
already
on
Envoy?
So
there
are
a
few
problems
that
were
already
solved.
The
main
problem,
especially
around
getting
others
to
help
us
contribute,
was
to
get
Basile
working
on
Windows
and
the
main
problem
with
getting
Basile
working
on
Windows,
one
of
our
biggest
bugs
was
the
fact
that
Windows
has
much
shorter
limits
on
command
line
inputs
than
Linux
and
passing
those
long
Basel
passes
really
long
command
line
arguments
with
lots
of
Derek
lots
of
directories,
and
it
was
breaking
windows.
It
wasn't
working.
We
eventually
I
mentioned.
A
We
had
a
patched
up
version
of
basil.
We
saw
it.
We
got
around
it,
sort
of
a
band-aid
approach
by
first
of
all,
moving
the
entire
directory
structure
into
root
into
directory
in
the
root
and
shortening
that
directories
name
and
also
having
some
logic
to
make
sure
that
empty
directories
were
not
being
included.
A
A
Another
pleasant
surprise
was
that
many
linux
specific
socket
issues
that
we've
had
earlier
this
year.
We
discovered
that
they
were
actually
solved
by
the
work
around
getting
envoy,
compiling
gonto
sx
now
envoy
uses
bsd
sockets,
which
work
on
the
windows.
That
was
a
huge
help
for
us
not
having
to
do
that
work
ourselves.
A
Problem
we've
recently
solved
Windows
headers,
which
have
to
be
included
in
many
files,
especially
around
sockets,
and
things
like
that
windows
headers
define
many
common
keywords
that
would
break
the
build
break
anything
we
tried
to
build
this
code.
Snippet
was
literally
like
probably
40
new
faces.
A
The
codebase
literally
any
place
where
we
included
windows
headers.
We
have
to
undef
common
keywords.
True
delete
error,
get
message
things
like
that.
That
was
obviously
pretty
ugly
nah.
No,
no,
not
maintainable,
of
course,
and
really
ugly.
We
solved
that
by
putting
all
of
that
into
a
common
platform,
header,
a
platform
header
file
and
now,
instead
of
including
windows,
headers
just
include
source
Dom
blaze
source
include
envoy
platform
that
H
another
problem
was
that
they
see
a
lot
of
our
seeming
dependencies.
A
B
This
one
is
interesting
because
C
make
C
make,
has
its
own
rational
way
of
way
of
building.
Obviously,
that
is
not
Basel,
not
only
that
almost
every
consumer
of
C
make
a
lot
like
your
Auto
conf
Auto
make
projects
the
the
maintainer
's
have
their
own
particular
way
that
they'd
like
to
tweak
that
build.
So
basically
in
every
single
seam,
every
single
C
make
based
project
out.
There
has
its
own
quirks
that
make
a
unified
compilation
very
difficult
to
realize.
B
So,
basically,
the
Basel
community,
the
the
foreign
CC
rules
team,
takes
runs
through
the
C
make
detection
process
runs,
runs
through
that
whole
configuration
and
then
stops
and
says:
okay,
what
does
he
make?
What
decisions
did
C
make
make?
Let
me
see
a
ninja
build
file.
Okay,
now
we're
going
to
Basel
Phi
that
and
the
actual
compilation
is
all
handled
by
Basel.
Our
challenge
was
to
strip
out
all
of
these
rules
that
all
of
these
other
third-party
projects
had
introduced
and
that
that
was
a
challenge
and.
A
Finally,
the
last
one
that
we
saw
recently
was
where
we
had
an
issue.
Basically
all
the
external
dependencies.
There
was
no
unified
way
of
doing
it
somewhere
linked
in
dynamic
libraries
somewhere
static.
Now
he
got
everything
bundled
into
Envoy
static,
that
exe
all
the
external
dependencies.
That's
a
synopsis
of
stuff.
We
did
already
oh
yeah.
B
It's
a
nature
of
trying
to
work
on
any
project,
that's
going
to
be
cross-platform,
but
the
issues
that
can
be
reintroduced
by
other
contributors
right
off
the
bat
you
even
the
way
that
integers
and
Long's
get
handled
on
Windows,
verse
and
64-bit
windows
versus
64-bit
Linux,
pretty
radically
different
on
window
on
Linux.
It's
all
64
bits,
your
int,
your
long,
your
pointer,
your
pointer
type,
are
all
eight
octet
sits.
It's
simple
on
Windows.
This
is
actually
a
64
P
architecture.
B
So,
if
you're
in
this
room
because
you're
a
Windows
developer,
you
already
know
this,
probably,
and
if
you're
in
here
as
a
Linux
architect,
you
probably
know
that
they're
all
64
bytes,
so
it's
a
very
easy
thing
to
get
confused
by
envoy
is
a
great
I'm.
Going
is
solve
this
in
a
good
way
internally
for
their
own
api's,
and
they
basically
decided
to
build
everything
on
an
explicit
type,
whether
it's
unsigned
32
bits
sign,
64
bits,
whatever
a
boolean,
whatever
the
type
is
needed,
they
try
to
map
it
as
closely
as
possible.
B
The
problem
is
as
soon
as
we
start
using
absol
or
standard
C
interfaces
we're
right
back
to
the
same
problem
of
having
to
speak
in
terms
of
intz
and
Long's
and
how
these
differ
from
platform
to
platform
and
we're
not
the
only
ones,
because
now
we
have
the
Android
and
iOS
port
of
that
they
have
been
working
on
it,
lift
as
part
of
envoy.
So
these
are
things
that
we're
going
to
want
to
work
as
a
community
to
to
resolve
on
likewise
MSV
see
Microsoft
had
never.
They
began
POSIX
implementation.
B
This
sounded
great,
was
part
of
their
federal
contracts
that
they
had
to
implement
POSIX
and,
unfortunately,
as
requirements
got
removed.
That
effort
stalled.
So
a
lot
of
things
like
a
file
descriptor
is
not
an
int
on
Windows,
it
would
be
a
handle,
it
would
be
a
pointer
and
that's
not
even
the
same
number
of
bytes
when
it's
stored
and
passed
back
and
forth.
So
we
have
the
problem
of
having
a
half
of
a
POSIX
implementation
standard.
B
C++
does
some
great
things
for
us
in
this
respect,
but
things
like
the
networking
will
stack,
aren't
really
well
represented
by
the
C++
standard.
So
we
had
to
work
around
these
things.
Even
things
like
s,
size
t
if
you're
a
Linux
developer,
you
know
this
type,
you
used
it
and
you
probably
think
it's
standard
C
which
it's
not
it's
standard,
POSIX
iso
c++
twenty-five
years
later
finally
said
we
need
a
signed
size
or
relative
distance
between
two
pointers
and
they
introduced
a
pointer
death.
B
Also
the
compiler
behaves
differently.
Microsoft's
interpretation
of
C,
plus
M
C
C++
17,
is
not
identical
to
the
new
canoe
GCC
projects:
interpretation,
which
is
not
the
same
as
the
LLVM
si
Lang
interpretation
of
what
the
spec
says.
So
you
can
have
legitimate
differences
and
incomplete
implementations,
so
as
simple
as
the
mnemonic
in
where
your
keyboard
might
not
have
a
chill
D
character.
So
you
need
to
have
a
different
way
of
writing
compliment.
We
had
ternary
operators,
which
were
very
short,
simplified
ASCII
ways
to
map
those
well
sooner
or
later.
B
C++
said
that
the
standard
should
have
the
words
not
and
and
compliment
microsoft
still
has
not
implemented
those.
So
in
solving
those
issues
we
have
actually,
just
in
the
past
month,
been
dwelling
on
the
question
of
whether
we
are
going
to
rely
on
Microsoft,
Visual,
C
or
whether
we're
going
to
shift
over
to
using
ceiling,
and
it's
worth
pointing
out
here
that
on
Linux
on
OS
10,
the
implementation
of
envoy,
the
actual
binaries
you
download
from
them
are
built
based
on
C
lank,
and
it's
probably
familiar
to
you.
B
If
you
use
the
Chrome
browser
or
any
one
of
a
number
of
other
Google
products.
So
it's
very
much
their
favorite,
how
household,
favorite,
compiler
and
certainly
we're
going
to
have
the
easiest
transition
with
the
rest
of
the
community
if
we
agree
to
share
C
Langa's
R
as
our
interpretive
language,
but
we
do
hope
that
the
project
is
going
to
be
able
to
adopt
visuals
C
as
well
alongside
it,
so
that
we
can
see
where
something
peculiar
in
Si
Lang
doesn't
really
map
to
other
people's
interpretations
of
C++.
B
It
will
just
make
the
code
more
portable
if
this
gets
caught
by
the
CI
very
early
in
a
code,
submission
process
and
then
many
of
the
tests
that
are
that
right
now
this
is
a
prep.
This
is
still
a
work
in
progress.
It'll
continue
to
be
reintroduced,
many
of
the
actual
tests
are
built
are
invoked
by
invoking
the
tool
chain,
invoking
utilities
we
compile
and
with
the
presumption
that
we
have
a
bash
shell,
which
is
a
slightly
strange
thing
on
Windows.
Of
course,
we
have
services
for
UNIX,
but
we're
not
relying
on
that.
B
That
will
continue
to
trip
us
up,
and
so
some
of
these
tests
are
still
they
need
to
be
adjusted,
and
let
me
think
so
specific
bugs
that
we
are
fixing
right
now.
This
is
the
current
work
in
progress.
I
come
down
to
basically
three
things.
The
first
thing
that
remains
to
be
done
is
the
Windows
is
very
sever.
That
envoy
one
of
the
brilliant
aspects
of
envoy
is,
is
that
you
simply
alter
this
configuration.
You
push
a
different
certificate
file.
B
You
push
a
different
configuration
and
envoy
reacts
immediately
to
reread
the
configuration
on
the
fly
and
have
a
hot
restart
on
the
fly
it's
already
operating
with
the
new
endpoints
that
you've
defined
the
new
parameters
that
you've
defined
and
so
file
change.
Notifications,
work
differently
in
the
Windows
32
API
than
they
do
on
UNIX.
B
None
of
the
UNIX
compatibility
actually
helps
us
because
there
are
two
different
file
system
drivers
layers,
even
if
you're
running
in
services
for
UNIX.
So
that
still
needs
we
have
a
draft
implementation,
partly
done
by
Sam
Smith
his
work
at
pivotal,
partly
done
by
Microsoft,
and
so
we
we
need
to
polish
that
up
for
submission
second
issue:
the
process,
control
primitives.
This
should
be
pretty
obvious,
but
we
don't
have
a
kill
command
on
Linux.
The
way
that
signals,
travel
between
processes
and
threads
on
UNIX
there's
very
little
relationship
to
the
win32
API.
B
So
we
need
an
entirely
separate
implementation
of
the
process,
control
primitives.
To
pick
up
that
the
process
has
been
forcibly
shutdown
by
the
u.s.
to
know
whether
or
not
the
user
is
requesting
a
graceful,
shutdown
or
and
simply
to
persist
as
a
background
process
requires
some
additional
code
around
the
service,
Control
Manager
and
on
Windows.
Of
course,
you
expect
things
to
run
as
generally
as
a
service.
B
The
third
and
final
issue
that
we
are
in
the
process
of
resolving
and
we
have
been
working
our
way
through
them-
is
the
Windows
socket
and
handle
objects.
The
like
I
hinted
at
earlier
they're,
not
a
file
descriptor
they're,
not
a
simple
int
and
the
actual
primitives
in
the
windsock
api
differ
quite
a
bit
from
differe
attica,
Lee
from
the
better
parts
of
Linux
and
differ
in
pretty
substantial
ways,
even
from
a
bsd
socket.
So
in
terms
of
abstracting,
this
well
I
said
that
the
C
standard
doesn't
have
a
great
implementation.
B
B
Unfortunately,
that's
not
quite
enough,
because
the
windows
API
the
event
loop,
the
event
model
is
level
triggered.
Basically,
windows
poll
is
somewhat
crufty
and
very
frustrating
to
use.
There
are
alternatives,
there's
I/o
completion,
ports
and
other
models
of
writing
asynchronous
code
on
Windows,
but
that
actually
causes
us
to
have
to
rip
out
the
core
engine
and
replace
it
with
something
with
different
set
of
expectations.
B
So
this
code,
well
in
the
short
term,
is
going
to
look
exactly
like
Linux
and
behave
similar
to
lead
to
Linux
we're
going
to
have
to
do
additional
work
coming
in
next
year
and
we'll.
This
is
one
of
those
places
where
we're
hoping
for
Microsoft's
partnership
in
developing
and
substituting
those
small
bits,
those
components
of
envoy
which
give
us
the
event
loop
for
handling
10,000.
You
know
parallel
requests
on
20,
parallel
threads.
B
B
So
just
to
talk
a
minute
about
the
structure
of
envoy
envoy
is
built
of
just
a
pile
of
extensions.
It's
it's
a
mountain
of
different
components
that
is
assembled
together
to
achieve
all
the
things
that
you
would
get
to
enjoy
in
terms
of
the
logging
and
the
filtering,
the
roughly
we
express
so
many
different
protocols
in
envoy.
B
It
has
an
enormous
amount
of
flexibility
in
terms
of
what
filtering
you're
performing
and
how,
as
you
know,
that
touches
that
the
auth
stack
that
touches
at
the
protocol
stack
and
we
can
make
seamless
transformations
from
HTTP
1
to
quick,
which
will
soon
be
HTTP
3.
So
we're
going
to
be
able
to
manage
all
of
these
transitions,
and
we
do
that
by
having
individual
extensions
that
can
be
dropped
in
and
pulled
out.
B
B
That
doesn't
include
all
of
these
different
extensions,
so
they're
all
pluggable
and
during
the
porting
we've
basically
gone
ahead
and
set
this
issue
aside
in
order
to
focus
all
of
our
energy
on
the
core
on
the
main
state
engine
and
the
the
basic
protocols-
and
we
basically
comment
commented
out
left
aside
a
number
of
these
extensions.
So
what
we're
hoping
for?
B
What
we're
hoping
to
accomplish
is
to
find
additional
community
members
who
are
going
to
be
interested
in
seeing
these
different
extensions
now
populated
on
Windows
and
fully
supported,
and
then
optimized
enhanced
for
that
environment
as
well.
But
the
nice
thing
is
is
that
you
can
have
a
court.
You
can
have
a
fairly
basic
envoy
very
much
more
quickly
and
as
we
get
these
other
features
developed
we'll
be
able
to
go
one
step
further
on
that.
B
A
So,
where
do
we
go
from
here
and
what
does
the
future
look
like?
So
pivotal
is
very
committed
to
getting
seeing
this
Windows
port
through
we're
hoping
to
where
we've
built
up
a
lot
of
momentum
in
the
last
few
months,
the
Envoy
community
has
been
really
helpful.
You
know,
they've
been
really
welcoming
to
our
efforts.
A
Matt
and
lizand,
and
other
maintainer
--zz
were
all
very
responsive
to
us,
and
recently,
like
we
mentioned
Microsoft,
also
expressed
interest
in
helping
us
out,
and
getting
this
part
through
and
like
Bill
mentioned
envoy,
is
a
very
flexible
platform,
so
optimizing
and
contributing-
and
you
know,
help
tweaking
it
for
your
own
use
case.
All
that
remains
very
approachable.
The
community
is
very
helpful
and
it's
so
they're.
The
the
barrier
to
contribute
is
getting
lower.
A
If
you're
wondering
how
you
can
get
involved,
the
best
place
to
get
involved
in
the
Windows
efforts
is
in
Envoy
slack.
There's
a
Envoy
windows,
dev
bill
and
I
are
pretty
active
there.
The
envoy
community
is
pretty
active
and
Microsoft
recently
got
active
as
well.
If
you
want
to
see
work
currently
in
progress,
you
can
look
at
the
envoy
project
on
github,
especially
look
at
any
issues,
tagged
Help,
Wanted
and
we're
hoping
to
get
a
custom
tag
for
Windows
up
soon,
which
will
help
people
follows
Windows,
specific
issues
and
just
watch
the
building.
A
B
B
Now
that
doesn't
mean
that
the
Android
project
is
willing
to
adopt
that,
but
it
does
mean
that
they've
been
very
active
in
terms
of
providing
the
endpoints
providing
the
API.
Many
people
will
just
say
refer
to
envoy
as
one
big
mass
of
API
adding
those
API
extension
points
to
allow
vendors
to
come
in.
So
we
entirely
expect
that
not
only
will
Microsoft
be
interested
in
helping
make
sure
that
this
fundamentally
works
well,
but
they
may
be
bringing
in
plugins
and
hooks
into
a
specific
version
of
Envoy
dope
by
themselves
or
other
groups.
B
We
build
one
of
our
own
envoys
for
Linux
for
our
containers
projects,
so
we
expect
that
there's
going
to
be
some
customization
out
there,
but
these
are
just
different
channels
that
you
can
reach
out
and
find
out
not
only
how
grab
some
low-hanging
fruit
of
some
orc
that
could
be
contributed,
but
also
looking
at
the
bigger
picture
extending
envoy
further
on,
and
so
that's
what
the
envoy,
dev
and
envoy
windows.
Dev
channels
are
great
for.
B
B
So
the
effort
that
we've
been
conducting
we
as
the
hill
mentions,
we
started
in
earnest
in
March
April,
and
so
at
this
point
we
have
about
three
more
code
patches
to
commit
to
get
the
to
get
it
fundamentally
working.
So
we're
probably
looking
at
the
end
of
this
year,
then
going
into
next
year.
We
have
all
of
these
optimizations
that
as
I
mentioned,
so
when
will
it
be
ready
for
production
use?