►
From YouTube: Using Wasm to develop Envoy Filters for supporting Yahoo Headers - Mrunmayi Dhume, Michael Cieplak
Description
Using Web Assembly to develop Envoy Filters for supporting Yahoo Headers - Mrunmayi Dhume, Michael Cieplak
Today at Verizon Media (formerly Yahoo), the on-prem Kubernetes platform spans 35 clusters across multiple data centers serving ~2500 apps. There are 2 ingress layers - Apache Traffic Server (ATS) serving 2M peak RPS and Istio Ingress based Envoy Proxy with a peak of 220K RPS. One of the key plugins of ATS is the verification/generation of Yahoo Headers, used by apps to obtain downstream client information such as the remote address/port, and a signature generated using a combination of base64, MD5, and a private key to ensure header integrity. To migrate all ATS traffic to Envoy, it is necessary to port all plugins from ATS with minimal changes. To achieve this, an Envoy Web Assembly (Wasm) filter was implemented using the Proxy Wasm standard, which is able to process these headers. This talk will provide an overview of the filter implementation and the learnings achieved along the way.
A
B
A
So
today,
at
verizon
media,
formerly
yahoo,
we
have
over
30
kubernetes
clusters
across
multiple
regions.
Each
of
these
has
an
istio
service
mesh
deployed
on
it.
We
have
two
parallel
ingress
layers:
fronting
our
kubernetes
clusters.
One
is
powered
by
apache
traffic
server,
serving
two
million
peak
requests
per
second
and
the
other
by
envoy,
which
serves
230k
requests
per
second
by
the
beginning
of
2021.
A
A
A
A
So,
like
I
mentioned
previously
envoy,
is
a
rich
featured,
powerful,
l3,
l4
and
l7
proxy,
and
we
utilize
a
number
of
the
features
that
are
available
today.
However,
there
are
some
custom
features
that
we
would
like
to
add
and
extend.
Envoy
envoy
provides
a
number
of
extension
points,
such
as
through
custom
http
filters
network
filters,
access
loggers
among
others.
A
A
A
The
second
option
that
we
have
is
to
write
a
lua
based
filter
onward,
provides
a
lua
api
so
that
developers
can
write
their
custom
lua
script
using
it.
This
api
is
intentionally
small
so
that
the
lua
scripts
are
simple
and
safe
to
use
for
more
complex
use
cases
developers
are
encouraged
to
use
one
of
the
other
approaches.
A
A
A
A
A
So
if
you
take
a
look
at
this
diagram,
you
can
see
that
when
the
request
streams
come
in,
they
are
serviced
by
one
of
the
envoy
worker
threads.
Each
of
these
worker
threads
embeds
a
thread.
Local
vasom
vm,
so
if
you
write
a
vasim
filter
that
is
required
to
perform
some
per
request
based
manipulation,
that
filter
will
be
executed
in
one
of
these
thread:
local
vasim
vms.
A
In
addition,
you
may
also
have
a
use
case
where
you
want
to
perform
certain
non-request
specific
functionality.
For
example,
you
may
want
to
aggregate
the
metrics
across
multiple
requests
for
such
use
cases.
You
can
write
a
singleton
filter.
This
filter
then
gets
executed
in
the
base,
wasm
vm,
which
is
embedded
as
a
part
of
the
main
envoy
thread
in
general.
There
is
a
number
of
api
that
are
available
for
the
vason
module.
A
Vaso
modules
can
make
use
of
http
or
grpc
api
in
order
to
make
external,
http
or
grpc
calls
the
timer
api
can
be
used
in
case
there
is
some
functionality
that
needs
to
be
performed
at
regular
intervals.
There
is
also
provision
for
a
shared
data,
which
is
basically
a
key
value.
A
pair
store
and
a
message
queue
so
for
any
kind
of
operations
that
require
the
worker
threads
to
store
certain
data
which
can
then
later
on
maybe
be
consumed,
consumed
by
your
singleton
filter.
These
two
are
useful.
A
The
api
that
I
described
in
the
preceding
slide
is
defined
by
a
proxy
agnostic,
abi
standard,
which
is
called
proxy
vasim
so
by
proxy
agnostic.
I
mean
that
if,
once
you
write
a
vasiv
module,
it
can
be
executed
in
a
number
of
different
proxies,
such
as
envoy
apache
traffic
server
h.a
proxy.
As
long
as
these
proxies
implement
the
proxy
vessel
standard,
so
proxy
vasim
defines
a
number
of
event
driven
streaming
apis
which
govern
the
interaction
between
envoy
and
the
vassar
module
that
is
being
executed.
A
A
Additionally,
as
the
requests
come
in,
there
is
a
new
stream
context
that
gets
created
for
every
request
and
similarly,
a
new
stream
context
object.
So
this
stream
context
object
has
a
lifetime
which
is
equal
to
the
request
stream
in
which
it
gets
created.
Again
there
are
a
number
of
methods
that
can
be
overridden
here.
So,
for
example,
if
your
wasm
filter
needs
to
make
certain
request
header
manipulation,
it
would
overwrite
the
on
request,
headers
function
in
case
it
wants
to
append
certain
data
to
the
response
body.
A
A
So
what
are
yahoo
headers,
so
yahoo
headers
can
be
defined
as
a
set
of
custom
headers
that
are
used
to
securely
identify
a
request,
its
remote
source
and
the
path
that
the
request
takes
through
the
network.
So
there
are
a
couple
of
headers
that
we
were
considering
the
first
one
was
rid
or
request
identifier.
A
A
The
next
set
of
headers
that
we
were
concerned
with
were
ra,
rp
and
rs,
which
stand
for
remote
address,
remote
port
and
remote
signature
respectively.
So
ra
and
rp
are
used
to
are
used
for
the
address
and
port
of
where
this
request
actually
originated
from
rs.
In
turn,
is
com
is
generated
by
making
an
md5
signature
of
the
ra
and
rp
values
with
a
secret
and
then
base64
encoding
it.
A
The
primary
purpose
for
rs
is
to
make
sure
that
the
array
and
rp
values
do
not
get
tampered
with,
as
this
request
moves
through
the
network
sample
ra,
rp
and
rs
headers
are
shown
here
on
the
screen,
so
in
phase
one
of
our
filter
development,
we
focused
on
these
set
of
headers
in
phase
two.
We
have
a
couple
of
other
headers
that
we
wish
to
implement.
A
A
Some
of
the
per
hop
attributes
that
are
added
are
the
address
and
port
of
where
the
request
was
sent
from
the
address
and
port
of
where
the
request
landed
and
whether
that
hop
was
ssl
enabled
or
not.
In
addition,
we
also
add
a
signature
which
is
similar
to
the
rs
signature
that
we
defined
previously
again.
This
is
used
to
ensure
the
integrity
of
this
header.
A
A
A
In
our
use
case,
we
want
to
have
a
dynamic
value
for
the
via
header,
where
we
can
specify
attributes
such
as
the
protocol,
the
protocol
version
and
the
specific
proxy
through
which
this
request
was
passed.
So
if
you
take
a
look
at
the
example
over
here,
you
can
see
that
there
is
the
protocol
and
the
protocol
version.
That
is
the
actual
pod
through
ingress
pod,
through
which
this
particular
request
was
sent,
and
then
that's
followed
by
a
uuid.
A
A
B
B
So
we
actually
have
a
yahoo
connection
plugin
for
apache
traffic
server,
which
does
have
the
rs,
essentially
header,
validation
and
generation
flows
in
place.
So
the
main
idea
here
is:
we
wanted
to
reuse
that
functionality,
so
we
did
not
need
to
re-implement
it
and
also
to
stay
up
to
date
in
case
there
are
any
changes
there
and
also,
if
there
are
any
security
fixes
as
well.
B
B
So
the
problem
inside
of
our
web
assembly
filter
is
that
we
do
not
have
access
to
file
reads
here.
Okay,
another
challenge
that
we
ran
into
is
the
rs
secret
is
stored
in
the
yahoo
key
management
service.
So
here
we
ran
into
a
couple
of
problems
as
well,
so
the
first
one
being
is
the
existing
client
for
the
key
management
service
relies
on
the
curl
library,
which
requires
access
to
the
underlying
network
socket
which
we
do
not
have
inside
of
our
web
assembly
filter.
B
Also,
we
need
to
make
multiple
calls
to
this
key
management
service
to
get
the
rs
secret,
so
every
time
we're
waiting
for
a
response,
we
would
essentially
be
blocking
to
wait
for
that
response,
which
is
essentially
a
busy
weight.
This
blocks
the
envoy
threads
actually
from
executing
and
also
from
coming
up.
B
Another
issue
is
that
we
cannot
take
the
rs
secret
and
write
a
file
into
the
envoy
container
with
it,
because
once
again
as
part
of
our
webassembly
filter,
we
do
not
have
access
to
file
reads.
Okay.
So,
let's
take
a
look
at
the
architecture
that
we
came
up
with
to
get
around
some
of
these
challenges.
B
B
B
So
here
we
have
the
ingress
pod
with
the
envoy
container
when
it
is
coming
up.
It
has
a
singleton
vm,
which
is
running
as
part
of
the
main
envoy
thread
as
previously
described,
and
this
will
go
ahead
and
call
this
localhost
service,
get
the
rs
secret
and
then
write
it
to
the
shared
data,
which
will
then
be
available
to
be
read
by
other
vms
inside
of
the
envoy
container.
B
Okay.
So
now,
when
we
have
the
envoy
worker
threads,
which
are
coming
up
initially,
what
will
happen
is
they
will
use
their
start
time?
As
a
seed
for
the
rid
so
that
we
can
have
unique
rids
for
the
user
request
and
then
they
will
go
ahead
and
read
the
rs
secret
from
the
shared
data
and
load
it
into
its
own
memory.
B
B
B
So
here
inside
of
the
onstart,
we
read
the
secret
from
the
shared
data
populate
an
in-memory
variable
with
the
rs
secret
that
we
can
use
then
to
generate
and
validate
the
rs
header
which
is
coming
in
now
when
the
user
requests
are
coming,
the
stream
filter
chain
will
call
into
our
filter
where
a
context
object
will
be
created.
So
here
inside
of
the
constructor,
if
the
in-memory
variable
with
the
rs
secret
is
not
set,
we
will
actually
go
ahead
and
also
read
the
secret
from
the
shared
data.
B
This
is
used
as
a
backup,
just
in
case
the
ingress.
Utils
server
took
a
little
bit
longer
to
respond
and
for
the
main
functionality
we
have
the
on
request,
headers
implemented.
So
here
is
where
we
generate
the
rid
and
also
validate
and
generate
the
rate
rp
and
rs,
and
this
will
occur
for
every
user
request
which
is
coming
in
okay.
B
So
now
let
me
show
you
the
envoy
configuration
so
we
are
running
envoy
1.13.x
with
istio
1.5.x
inside
of
the
bootstrap
configuration
we
have
the
ingress,
util
server
cluster
defined
and
also
the
singleton
filter
in
the
filter
chain
configuration
we
have
the
yahoo
headers
filter
defined.
So
here
this
is
the
bootstrap
configuration.
B
As
you
can
see
here
we
have
a
static
cluster
resource
defined
for
the
ingress
util
server
pointing
to
localhost.
This
will
be
used
by
the
actual
singleton
vm
to
make
a
request
into
to
fetch
the
rs
secret
down.
Here
we
have
our
singleton
filter
definition
with
the
singleton
boolean
set
to
true
now
inside
of
the
filter
chain.
We
basically
have
our
yahoo
header
configuration.
B
B
B
Okay,
so
now
for
the
rid
flow,
it's
a
little
bit
simpler.
So
here,
if
the
rid
is
set,
we
will
just
continue
the
http
request.
If
it
is
not
set,
then
we
will
generate
a
new
one.
So
now
let
me
show
you
a
demo
of
how
this
functionality
looks
like.
So
I
have
an
echo
server,
which
is
behind
an
istio
ingress
pod.
B
B
B
Okay,
so
let's
talk
about
some
other
functionality
that
we
have
so
the
proxy
was
some
stats.
Api
allows
you
to
have
custom
metrics,
so
here
we
define
essentially
metrics
for
our
filter
to
see
how
it's
behaving
and
also
to
increment
in
case
there
are
any
failure
scenarios
so
that
we
can
alert
upon
them
as
well.
So,
as
you
can
see
here,
if
you
hit
the
envoy
stats
api,
you
can
access
the
metrics
which
you
are
setting.
B
Okay.
Now,
let's
talk
a
little
bit
about
some
of
the
performance
metrics.
So
here
we
wanted
to
run
a
load
test
to
see
how
our
filter
behaved
under
load.
So
we
have
one
istio
ingress,
pod
running
with
24
envoy
worker
threads,
meaning
24
cpu
with
our
filter.
We
run
4k
rps
for
10
minutes
and
we
have
vegetal
load
test
pods,
generating
the
client
load
with
an
echo
server
upstream.
B
So,
as
you
can
see
here,
we
have
multiple
scenarios
which
we
are
testing
which
is
similar
to
the
demo.
Essentially,
the
generation
flow
if
there
are
no
headers
coming
in
the
validation
flow,
if
headers
are
passed
in
and
the
validation
and
generation
flow,
where
the
incoming
headers
were
invalid
and
then
regenerating
new
ones.
B
B
B
So
if
there
are
any
issues
with
your
filter
and
it's
crashing,
it
will
not
bring
down
the
envoy
process
itself.
Also,
there's
the
security
aspect
here
as
well.
So
essentially,
the
vm
filter
cannot
access
anything
on
the
envoy
host
as
it
does
not
have
any
access
to
it.
Only
what
is
defined
through
the
proxy
was
some
apis.
B
B
So,
due
to
this,
we
you
had
to.
We
had
to
essentially
compile
them
down
into,
what's
known
as
llvm
bitcode,
which
then
can
be
used
to
import
and
compile
down
with
your
webassembly
module,
so
that
was
a
little
bit
of
a
learning
curve
and
wasn't
straightforward.
So
that
is
something
you
should
also
be
aware
of.
B
B
B
A
So
I
can
take
the
first
question,
which
was
regarding
the
performance,
so
we
did
see
some
amount
of
performance
improvement
in
general
in
terms
of
the
throughput
per
box,
but
it
does
require
a
little
bit
of
tuning
because
we've
been
using
apache
traffic
server
for
a
while
now
at
yahoo.
So
a
lot
of
those
have
already
been
tuned.
A
So
with
onboard
that
has
been
a
bit
of
a
learning
curve,
we're
learning
how
to
tune
some
of
the
aspects
like,
especially
if
there
are
large
payloads
we've
had
to
tune
some
of
the
settings
on
envoy.
So
there
has
been
some
performance
improvements,
but
there's
also
been
a
learning
curve
for
us
to
get
the
maximum
performance
out
of
envoy.
B
So
I
don't
think
I
can
share
too
many
details
about
that
and
then
another
thing
that
I
wanted
just
to
clarify,
because
I
know
in
one
of
the
last
slides
I
mentioned
that
basically,
the
web
assembly
functionality
is
not
fully
merged
yet
into
envoy.
So
actually,
as
of
a
few
days
ago,
it
did
get
merged
into
the
actual
envoy
upstream
code
base.
So
the
envoy
was
repo
was
basically
archived
and
then
everything
was
upstream
to
envoy
so
that
functionality
should
be
there
now.
So
just
wanted
to
clarify
that
as
well.
A
B
Yeah
and
yeah
I
mean
in
terms
of
compiling
with
open,
ssl
yeah.
It
was
definitely
a
little
bit
painful.
Basically,
I
we
had
to
compile
the
open
ssl
down
into
llvm
bitcode
first,
that
we
could
then
use
to
actually
compile
down
with
our
webassembly
filter,
so
that
was
definitely
of
a
learning
curve
for
us
basically
figuring
out
how
to
do
that
compilation
first
before
we
could
use
it
so.
A
B
Cool
yeah,
please
let
us
know
if
you
have
any
other
questions
in
the
chat,
yeah
and
then
also,
if
you
have
any
questions,
then
feel
free
to
reach
out
to
us
as
well.
So
basically,
our
handles
are
mduma
and
I'm
chet
black
on
the
on
voicetack
channel
and
also
on
github,
so
feel
free
to
reach
out
to
us,
as
well
with
any
questions
you
have.