►
Description
The Internet Research Task Force (IRTF) Open session, including Applied Networking Research Prize (ANRP) presentations, will be held during IETF 112 on 9 November 2021 at 16:00 UTC.
A
A
A
So
I
want
to
to
start
with
the
the
usual
note
well
and
reminder
of
the
intellectual
property
rules
for
the
irtf.
The
irtf
follows
the
the
same
sorts
of
intellectual
property
disclosure
rules
as
the
ietf.
You
know
if,
if
you're
aware
of
contributions,
if
you're
aware
of
patents
or
patents,
applications
that
cover
the
contributions
that
you
make
that
are
controlled
by
you
or
your
sponsor,
you
do
need
to
disclose
that
or
not
participate
in
the
discussion.
A
Perhaps
a
difference
from
the
ietf
disclosure
rules
is
that
the
irtf
expects
you
to
file
these
disclosures
in
in
a
timely
manner
in
a
period
of
days
or
weeks
rather
than
months,
and
we
expect
the
the
most
liberal
licensing
terms
possible
are
made
available
for
irtf
stream
documents,
and
if
you
need
more
details
about
this
process,
the
the
rfc
is
listed
on
the
slide.
Rfc
is
5743
78
the
8179
give
details
of
this
and
the
url
iot
irtf.org
policies
slash.
A
Also,
as
should
perhaps
not
be
a
surprise,
the
irtf
makes
recordings
of
online
and
in-person
meetings,
and
this
includes
audio
video
and
photographs,
and
so
on.
This
session
is
being
streamed
live.
I
believe
we're
going
out
on
on
youtube,
as
well
as
being
the
the
meat
echo
session
and
as
well
as
being
recorded
and
the
recording
will
be
be
on
youtube
after
the
session
as
well.
A
If
you
participate
online,
if
you
turn
on
your
camera
microphone
and
make
it
the
microphone,
then
you
are
in
the
recordings,
and
I'd
also
remind
you
that
the
chat
logs
in
all
of
these
meet
echo
sessions
are
recorded
and
they're
available
on
the
ietf
site.
So
what
you
put
into
the
chat
is
also
recorded.
A
I'd
also
like
to
remind
you
that
we
we
have
a
code
of
conduct
in
the
irtf
and
in
the
ietf,
and
you
know
as
a
participant
as
an
attendee
at
the
the
irtf
sessions.
You
agree
to
work
respectfully
with
the
other
participants
in
the
session.
You
agree
to
to
behave
in
in
a
professional,
respectful
and
polite
manner,
and
you
know
we.
We
have
the
code
of
conduct.
A
If
you
have
any
concerns
about
behavior
of
the
various
participants,
the
the
ombuds
team,
again
linked
from
the
slide,
is
available
to
to
investigate
those
and
and
to
act
as
a
point
of
contact.
If
you
have
any
any
issues
with
the
the
the
participants
in
the
session,
if
you
provide
any
personal
data
and
in
order
to
register,
you
would
have
had
to
provide
at
least
a
little
bit
of
personal
data,
that's
handled
in
accordance
with
the
its
privacy
policy
and
again
that's
linked
from
the
slide.
A
The
goal
of
the
irtf
is
very
much
to
conduct
research,
we're
not
a
standards,
development
organization
and
while
the
irtf
research
groups
can
publish
informational
or
experimental
documents
in
the
rf
series,
the
primary
goal
of
the
research
groups
is
not
publishing.
Rfcs
publishing
documents,
it's
to
promote
collaborations,
it's
to
promote
research,
to
promote
development
of
research,
ideas
that
relate
to
the
the
internet,
protocols,
applications,
architectures
and
technologies.
A
The
irtf
is
arranged
as
a
set
of
research
groups.
We
have
five
research
groups
which
are
highlighted
in
dark
blue
on
this
slide,
which
are
still
to
meet
later.
This
week,
the
crypto
forum
research
group
develops
cryptographic,
algorithms
looks
for
an
understanding
of
various
cryptographic,
algorithms
and
techniques,
and
many
of
those
are
used
in
ietf
standards.
A
The
pathway
networking
group
is
looking
at
the
the
interplay
between
the
end
points
and
path
elements
and
how
that
communication
should
happen.
The
privacy
enhancements
an
assessments
group
which
my
apologies
to
the
chairs.
I
noticed
this
was
listed
as
a
still
a
proposed
group,
rather
than
a
full
group
in
the
plenary.
A
We
also
have
a
number
of
groups
which
have
either
met
or
even
met
already
or
are
not
meeting
at
this
session,
including
the
the
map
rg,
which
had
a
bunch
of
interesting
measurement
papers.
Looking
at
the
network
architecture
in
in
the
previous
session
and
the
network
management,
global
access
and
congestion
control
groups
which
met
yesterday.
A
A
A
A
A
For
the
rfc
editor
function
and
this
program
was
created
after
the
departure
of
the
previous
rfc
series-
editor,
I
guess
a
a
year,
18
months
or
so
ago.
I
forget
the
exact
timing
back
in
the
times
when
we
all
used
to
meet
in
person
in
wonderland's
hotel
rooms,
so
that
I
guess
it
was
a
while
ago
and
that
that
program
has
been
developing
a
new
model
for
how
the
rfc
editor
function
is,
is
organized
and
arranged
and
arranged.
A
A
And
once
that
group
reaches
consensus
on
proposed
changes
to
the
operation
of
the
rfc
series.
There'll
be
the
the.
The
proposals
will
then
be
reviewed
by
an
rfc
series.
Approval
board,
which
will
comprise
the
the
rfc
series,
consulting
editor
and
representatives
of
the
various
streams
which
are
likely
to
be
the
irtf
chair,
the
ietf
chair
and
the
iab
chair.
Although
they
can
delegate
that
if
they
need
to.
A
So
I
I
would
encourage
you
as
participants
in
the
irtf
to
review
this
document
and
provide
feedback
to
the
rfc
editor
future
development
program.
They
have
a
meeting
tomorrow
in
the
1430
utc
slot.
They
also
have
a
mailing
list
by
which
you
can
provide
comments
and,
as
I
said,
please
do
review
this
document.
Please
do
send
feedback
on
whether
you
think
what's
being
proposed.
It
is
or
is
not
appropriate.
So,
as
you
see
fit,.
A
C
A
D
So
somebody
has
echo
I
this
morning.
Actually
it's
a
it's
a
new
thing
this
morning
at
11
a.m.
Paris
time
bangsang
received
his
award
is
now
a
chevalier
of
the
audre
du
merit
of
scientific
signs
for
scientific
work
of
the
french
government
and
while
the
award
was
given
to
him
for
this
contact
tracing
system
or
or
protocol
that
they
put
together
for
the
french
government,
it
was
actually
highly
mentioned
in
the
presentation
from
the
french
government
official.
D
That
veisa
was
a
major
contributor
to
the
ietf
with
14
rfcs,
which
I
think
is
about.
I
think
there's
two
french
people
have
tons
of
rfcs,
the
other
one
being
christian
utema
of
the
iab,
and
so
that
was
actually
mentioned
as
a
major
contribution.
Also,
his
work
in
the
irtf,
which
I
am
very
much
aware
of,
but
what
was
mentioned
was
also,
I
think,
what
we
all
came
to
appreciate
of
him.
D
It
was
funny
for
those
of
you
who
know
him
that
he
had
a
shirt
and
tie
and
a
jacket
which
is
like
first
for
me,
but
I
think
it
was
highly
like
it
was
highly
deserved,
and
I
think
we
always
forget
that
there
are
people
like
weinstein
who
are
not
like
the
most
vocal
and
who
are
a
little
bit
quiet.
But
in
the
end,
do
all
the
major
work-
and
I
felt
this
was
such
a
great
recognition
of
all
his
achievements
and
and
I'm
absolutely
proud
of,
having
been
his
co-chair.
Thank
you.
A
Yeah
think,
thank
you,
murray.
Congratulations
to
to
vancouver.
As
you
say,
this
is
an
incredibly
well-deserved
award.
That
panson
has
been
involved
in
the
ietf
and
the
irtf
for
very
many
years
and
has
made
some
some
really
tremendous
and
really
important
contributions.
So
I
think
that
this
is
astoundingly
good
news.
A
A
So
congratulations
very
much
to
the
these
three
speakers.
We
will
see
the
the
talks
in
in
a
few
minutes
if
you
do
see,
see
these
people
around,
and
I
see
that
they're
in
the
chat
here.
But
if
you
see
them
around
at
the
rest
of
the
itf
irtf
sessions
or
during
the
breaks,
please
do
talk
to
them.
Please
do
congratulate
them.
I
think
this
is
some
really
nice
work.
We've
got
here.
A
And
in
addition
to
that,
the
nominations
for
the
2022
applied
networking
research
prize
are
now
open.
The
the
deadline
for
these
nominations
is
the
19th
of
november,
so
you
have
10
days.
So
if
you
know
any
any
interesting
applied
networking
research,
if
you
know
any
good
papers
in
this
area,
if
you
know
any
any
people
who
you
think
will
make
interesting
contributions
to
the
community,
any
work
which
you
think
is
is
especially
valuable
and
isn't
getting
the
recognition
it
deserves.
A
A
And
with
that
we'll
move
on
to
the
the
talks,
we
have
three
talks,
we're
running
a
little
late,
but
I
think
not
all
of
the
talks
use
their
their
full
time.
So
the
first
talk
is
by
thomas
talking
about
xbgp
in
a
certain
amount
of
irony
that
the
talk
entitled
when
you
can't
wait
for
the
ietf
is
presented
in
the
last
of
the
three
award
talks
of
the
year.
So
I
guess
there's
something
to
be
said.
A
A
And
started
a
phd
looking
at
the
instability
of
routing
protocols
under
the
supervision
of
olivier
bonaventure
he's
currently
working
on
techniques
from
programmability
of
bgp,
which
led
to
the
prototype
xbgp
system
that
he'll
be
talking
about
in
this
talk
and
his
research
interests
include
distributed
routing
protocols,
programmable
networks
and
system.
Architects
he'll
be
talking
about
the
paper
xbgp
when
you
can't
wait
for
the
itf
and
the
vendors
which
is
originally
presented
in
the
acm
hardnet
symposium
in
2020,
and
I
think
thomas
is
in
the
chat.
A
C
C
You
can
see
the
standardization
delay
of
all
bdp
features,
so
we
measure
the
time
since
it
is
released
as
a
draft
to
the
robot.
It
is
released
as
an
lfc.
This
is
not
an
isolated
case.
All
the
routing
protocols
suffer
from
the
same
delay,
but
the
list
have
shown
that
this
is
also
the
case
for
other
lfcs.
C
Next,
you
need
to
implement
your
feature
to
the
operating
system
by
doors
and
finally,
the
rotors
must
be
updated.
But
if
you
are
a
small
network
such
as
ballet,
you
cannot
easily
influence
step
one
and
step
two,
because
you
are
you,
don't
have
enough
impact
to
actually
convince
a
routers
vendors
and
the
itf
from
your
new
feature
because
of
those
problems
we
introduce
xbgp
that
is
designed
to
bring
back
innovation
to
the
network
operator.
We
would
like
to
open
the
mdp
box
so
that
the
network
operator
can
actually
modify
the
protocol.
C
I
will
explain
you
how
hbdp
works
in
practice.
We
start
from
bgp,
which
is
considered
as
a
black
box.
You
can
configure
bgp
with
multiple
interface,
for
example,
cli,
netconf
or
snmp,
to
make
some
monitoring.
So
those
are
the
main
interfaces
to
configure,
but
you'll
not
have
any
way
to
modify
the
protocol.
You
as
being
a
network
operator.
This
recipe
will
open,
delete
and
will
export
the
protocol
in
the
protocol.
Member
inside
the
protocol,
you
can
add
a
new
state.
You
can
modify
existing
states
and
also
you
can
modify
the
transition
between
two
states.
C
C
Now,
actually,
xbgp
is
added
on
the
top
of
operating
system
vendors.
Now
the
routers
can
be
seen
as
a
kernel
where
you
can
actually
run
some
extension,
which
is
the
modification
to
your
pdp
protocol.
Now
I
will
explain
you
how
immediately
work
inside
bgp,
but
before
I
will
take
an
example
of
a
new
feature
which
is
not
standardized
by
the
itf
and
not
implemented
inside
any
operating
system.
So
what
we
would
like
is
to
add
the
geographical
localization
of
the
router.
We
see
the
vdp
update
to
make
some,
for
example,
filtering
or
some
analysis.
C
C
C
All
the
acceptable
route
inside
the
loan
carrying
will
be
taken
through
the
bgp
process
to
take
only
one
baseboard
that
will
be
pushed
to
the
forwarding
table
of
the
routers
and
then
putting
inside
the
export
heater
to
take
whether
or
not
the
route
can
be
exported
to
other
networks,
and
if
it
is
the
case,
it
will
be
stored
in
the
rj
board.
And
finally,
the
message
will
be
generated.
If
I
take
back
my
geographical
localization
extension,
you
need
to
modify
multiple
steps
of
the
vdp
protocol
and
those
are
represented
by
the
blue
arrows
with
ebgp.
C
It
is
possible
to
modify
the
protocol
thanks
to
the
introduction
of
insertion
point
insertion
points
are
a
way
to
execute
arbitrary
code,
and
so,
if
I
take
my
geographical
localization,
it
is
composed
of
multiple
subpart
and
those
apart
will
be
added
to
the
right
insertion
point,
but
that's
not
all
because
our
extension
must
talk
to
the
protocol
memory
to
retrieve
the
geographical
localization,
and
this
is
known
via
the
xmdp
api,
and
this
is
something
that
make
the
interaction
between
the
plugin
and
the
protocol
memory.
Now
I
will
explain
you
the
xp
api.
C
This
is
some
way
to
communicate
with
the
media
implementation
and
all
the
extension
code
or
plugin
must
use
this
api
to
receive
or
send
some
protocol
memory.
Why?
First
because
it
ensures
a
desolation
of
the
colleen
and
the
dp
implementation
and
second,
it
is
a
unified
way
to
access
to
the
memory.
Remember
that
xbgp
must
support
multiple
gdp
implementation
name.
Xbtp.
Api
contains
multiple
functions
such
as
function
to
send
and
read
the
gpp
messages,
also
the
function
to
get
or
set
some
memory
inside
the
protocol
memory,
for
example,
for
the
geographical
localization.
C
C
So
now
we
have
the
big
mixture
of
xbgp
and
I
will
show
you
some
new
skills
that
you
can
do
with
lbgp,
but
before
you
need
to
adapt
some
bgp
implementation
to
make
them
xbgp
compliant.
You
have
a
favoriting
and
board
in
your
case.
In
the
slide,
I've
put
an
array
with
all
the
lines
of
code
that
you
need
to
modify
your
ad
to
make
the
bgp
implementation
xbgp
compatible.
C
The
first
use
case
is
a
simple
one.
This
is
the
monitoring,
and
what
we
would
like
to
do
is
to
monitor
the
length
of
the
ice
path.
Actually,
it
is
difficult
to
achieve
it
with
traditional
interface,
but
within
gpa
you
have
an
interface
that
is
easier.
Why
would
you
like
to
monitor
the
is
pass?
For
example,
you
would
like
to
monitor
it
to
filter
out
large
as
pass
or
make
analysis
afterwards.
C
So
now
I
will
explain
you
a
complete
example
of
c
code
that
represents
the
xbgp
expansion.
First,
you
need
to
retrieve
the
data
from
the
host
implementation.
This
is
known,
true
get
arc
function,
so
this
is
a
function
of
the
api.
Then
you
will
do
your
computation.
So
here
you
will
pass
the
response
attribute
and
count
the
number
of
photon
of
the
system
content
inside
the
path,
and
then
you
will
use
another
function
to
actually
log
the
number
of
as
you
computed,
this
extension
code
will
be
added
to
the
import
filters.
C
The
second
new
skill
is
related
to
the
use
of
bgp
inside
data
center.
In
data
center.
You
use
bdp
to
make
your
internal
routine.
All
the
best
practice
concerning
the
use
of
bgp
in
a
data
center
is
stated
in
rse
7938
inside
this
document.
There
is
a
constraint:
you
don't
have
any
exact
path
so,
for
example,
to
go
from
level
two
to
level
zero.
C
This
extension
will
also
be
put
on
the
insertion
point
related
to
the
import
filters.
The
third
use
case
is
related
to
the
road
selection.
So
in
this
slide,
we
consider
a
stop
network
for
the
source
network
connected
to
several
transit
transit,
1n10
z2
and
to
reach
the
network
destination.
Bgp
must
advertise
a
route
to
the
source.
C
The
source
will
directly
receive
the
green
road
via
transit,
one
and
another
one
on
transit
2..
According
to
the
local
22
policy,
it
will
either
announce
the
blue
or
the
yellow
path,
maybe
transit,
to
choose
the
path
for
its
security
properties.
Not
nurses
prepare
the
past
with
a
longer
latencies,
and
so
the
ranking
of
the
source
and
the
transit
does
not
match.
C
C
C
P
is
no
longer
reachable,
so
the
rotors
will
send
a
result
to
the
neighbor,
but
for
whatever
reason,
due
to
a
software
bug
or
misconfiguration,
the
water
cannot
process
the
result,
and
so
it
fails
to
send
a
missile
to
the
other
level.
The
upstream
router
will
still
have
the
p
prefix
in
the
forwarding
table.
This
is
something
we
don't
want
because
it
may
lead
to
some
black
owning
or
traffic,
and
this
is
a
big
measurement.
C
C
There
are
some
mechanisms,
such
as
transformative
staff
that
can
actually
reload
all
the
entire
routine
table
of
the
neighbor.
But
since
bgp
zombie
is
concerning
a
small
fraction
of
prefix
according
some
measurement
inside
the
internet,
it
is
not
really
necessary
to
renew
all
the
entire
routing
table,
so
we
design
a
prototype
that
only
asks
for
those
of
these
that
we
detected.
C
Well,
we
need
to
have
an
automatic
way
to
verify
some
properties
before
injecting
it
to
the
routers.
With
with
vdp
we
make
the
following
approach.
First,
you
start
from
the
extension
source
code
that
will
annotate.
Then
it
will
be
passed
to
the
software
verification
tools
and
if
the
verification
tool
says
okay,
the
code
can
be
trusted.
It
will
be
compiled
to
the
by
code
and
then
injected
to
the
protocol.
Now
I
will
explain
you
some
properties
that
we
defined
that
all
the
plugin
most
satisfied
before
being
injected
to
the
routers.
C
C
We
also
restrict
some
api
function
to
some
extension
code,
for
example
for
the
monitoring
they
do
not
have
the
right
to
modify
the
protocol.
Finally,
we
use
crm
verification
tool
to
verify
properties
related
to
the
bdp
syntax.
As
an
example,
we
will
verify
the
syntax
of
the
geographic
coordinate
extension,
so
this
geographic
coordinate
extension
is
defined
on
the
graph
and
it
must
follow
the
format
specified
on
this
graph
and
those
properties.
I've
defined
on
the
on
the
slide
are
those
that
must
be
followed
by
the
plane.
C
C
C
A
A
E
Yeah
thanks
very
much
for
your
talk.
It
was
very
interesting
two
points.
The
first
is
a
question
is
in
terms
of
that
last
slide.
It
was.
It
was
your
desire
for
this
mechanism
to
always
have
the
transitive
bit
set
to
zero,
which
I
think
does
allow
for
a
certain
amount
of
prototyping,
for
instance,
right.
C
We
just
added
an
interface
to
allow
some
network
operators
to
implement
quickly
its
functionality
and
then
it
will
actually
test
in
another
network
and
finally,
it
will
be
maybe
if
it
works
well,
in
maybe
a
release
on
a
on
the
production
routing
network.
But
actually,
if
you
would
like
to
have
performance,
you
have
to
to
write
it
directly
inside
the
implementation
level
code.
So
yeah.
C
Maybe
when
you
introduce
new
code,
you
will
actually
break
some
some
rules
of
the
bgp
protocol,
but
we
designed
some
mechanism
to
guarantee
the
execution
of
the
plugin
locally,
but
the
execution
in
the
global
network
is
not
something
we've
done,
and
this
is
something
that
you
can
you
you
you
test
on
yourself.
So
you
you
make
some
tests
and
if
you
are
confident
about
that,
you
will
push
on
the
network.
C
So
there
is
no
really
a
way
to
test
the
global
safeties
of
the
the
implementation
you
you
put
on
the
on
the
router,
so
I
mean
this
is
something
we
can
actually
do
afterwards,
but
something
I
think
it
is
something
quite
complicated
to
to
verify
and
satisfy
the
security
property
of
the
global
network,
but
with
immediately.
We
have
also
some
just
local
verification.
C
Hi,
this
is
an
interesting
question,
because
when
xmgp,
you
can
do
pretty
pretty
much
what
you
can
do
well.
The
first
thing
is
that
you
have
to
choose
the
while
right
tool
to
actually
guarantee
the
security
property
of
the
underwater,
maybe
well
now
we
we
write
all
the
plugin
in
c
code.
This
is
not
something
which
is
really
secure,
but
you
know
in
another.
We
can
also
use
on
the
other
tools.
A
A
C
Oh
yeah,
actually,
what
you
have
to
do
to
put
xbgp
inside
an
implementation
is
to
first
follow
the
original
rfc
so
4271,
and
we
also
need
all
the
extensions
above
multiprotocol,
for
example,
in
the
in
the
in
the
paper.
We
reproduce
the
root
reflector
functionalities,
and
this
is
one
of
some
some
some
new
skills
that
prove
that
we
can
actually
implement
a
lot
of
complexity,
but
the
strong
basis
of
hbgp
relies
on
the
the
original
draft
and
multi-protocol
extension.
C
The
other
thing
you
can
you
can
do
you
can
actually
put
your
code
inside
the
extension
code
and
then
put
it
in
the
the
implementation
which
is
six
php
compatible.
But
if,
for
example,
if
you
want
to
to
modify
some
data
structure
that
you
just
acquired
inside
the
implementation,
you
cannot,
for
example,
if
you
take
the
the
functionality
that
enlarge
the
the
buffer
of
of
bgp
message,
so
4k
to
74
kilobytes
of
memory
to
one
message.
That's.
A
A
We
you
think
we
should
be
doing
differently
in
the
way
we
structure
the
protocols
that
will
help
this
sort
of
experimentation
once
the
rfc
has
been
published.
Other.
Are
there
any
general
lessons
that
that
can
be
learned
from
this,
or
is
this
very
specific
to
the
the
details
of
how
bgp
was
implemented.
C
When
when
we
look
at
xbgb,
the
the
protocol
was
pretty
well
established,
so
if
the
what
we
we
try
to
do
with
cbgp
is
to
understand
what
are
the
properties
of
the
protocol
and
then
we
establish
a
set
of
properties
to
actually
actually
model
a
general
actually
architecture
of
of
the
routing
protocol.
And
then,
when
you
have
the
general
model
of
this
routing
protocol,
then
you
will
look
inside
your
implementation.
C
A
Yeah
that
that
that
that
makes
sense,
so
I
guess
the
the
general
conclusion
is
the
modularity
in
the
in
the
the
way
the
specifications
are
written,
might
make
this
process
easier
for
developing
modular
apis
and
extensions
yeah.
Okay.
So
that's
maybe
some
something
that
for
the
standards
groups
to
think
about
all
right-
and
this
is
where
over
time-
and
I
I
don't
see
anyone
else
in
in
the
queue
so
thank
you
very
much.
I
think
that's
a
really
interesting
book.
Some
really
nice
answers
to
the
questions.
A
Be
able
to
to
be
around
in
the
break
afterwards
if
people
want
to
catch
up
with
him
in
in
the
gather,
space,
perhaps
or
of
course,
drop
him
an
email
all
right.
So
thank
you,
thomas
and
we'll
move
on
to
the
the
next
talk
of
the
the
session,
which
is
by
axa
kashaf
axa
is
a
phd
student
at
carnegie,
mellon
university,
she's
co-advised
by
eva
sekker
and.
A
And
I
apologize,
they
must
have
mangled.
Those
names
axis
focuses
on
network
security,
particularly
distributed
denial
of
service
attacks
and
currently
she's
working
on
building
reconnaissance
techniques
to
to
profile
the
capabilities
of
ddos
defense
systems.
Looking
at
understanding
and
improving
the
resilience
of
the
internet
against
ddos
attacks.
A
B
B
B
B
B
B
Are
there
other
hidden
dependencies
that
we
also
need
to
know
about?
There
might
be
inter-service
dependencies
which
indirectly
affect
the
website
and
finally,
since
stein
attack
had
such
a
huge
impact,
it
is
natural
to
think
that
websites
would
adopt
as
a
result.
How
has
the
world
changed
in
using
third
party
services
as
a
consequence
of
9
attack?
B
B
B
To
do
this,
we
look
at
the
life
cycle
of
a
web
request
when
a
user
makes
a
request
to
a
website.
It
first
goes
to
the
authoritative
dns
server,
which
provides
the
ip
resolution
of
the
website
after
the
ip
resolution
of
the
website.
User
starts
a
tcp
handshake,
followed
by
an
ssl
handshake.
If
the
website
supports
https,
if,
yes,
then
the
website
provides
a
certificate
which
is
verified
for
revocation
by
the
user
by
contacting
the
ocsp
server
or
crl
distribution
points
set
up
by
the
certifying
authority.
B
Finally,
if
the
certificate
is
valid,
the
user
requests
the
content
of
the
of
the
website
page.
If
the
website
is
using
a
cdn,
then
cdn
will
serve
this
content.
Based
on
this,
we
identify
three
services.
The
first
one
is
dns.
Second,
one
is
certificate,
validation
by
certified
authority,
and
the
third
one
is
cdn
note
that
these
by
no
means
are
the
only
services
that
a
web
request
encounters,
but
we
just
identify
these
three
services
and
define
the
scope
of
our
work.
B
B
We
are
also
interested
in
critical
dependencies
which
exist
if
a
service
provider
is
integral
for
a
given
service,
for
example.
Here,
if
cloudflare
is
down,
ocsp
servers
of
let's
encrypt
will
not
be
accessible.
Hence,
let's
encrypt
has
a
critical
dependency
on
cloudflare
in
case
of
dns
in
cdn
we
compute
critical
dependency
by
measuring.
If
the
website
is
redundantly
provisioned,
which
means
that
the
website
is
using
multiple
dns
or
cdn
providers
and
in
case
of
certificate
authorities,
we
compute
it
by
measuring.
If
the
website
has
enabled
ocsp
stapling.
B
B
I
would
go
into
details
for
this
in
the
next
slide,
that
how
do
we
actually
identify
if
it's
a
private
or
third
party
and
finally,
for
a
website
and
its
name
server?
We
want
to
identify
which
of
these
name.
Servers
belong
to
the
same
entity,
for
instance,
here
azure,
dns
and
o365
filtering
both
belong
to
microsoft.
B
B
To
identify
third
party
name
servers
prior
work
looks
at
second
level
and
top
level
domains
of
the
name
server
and
the
website.
If
they
match,
then
the
name
server
is
private,
otherwise.
Third
party,
for
example,
here
in
case
of
google,
its
name
server,
also
has
the
same
second
and
top
level
domain
google.com.
B
B
B
B
To
cope
with
these
issues,
we
add
a
few
more
heuristics
so
for
all
website
and
name
server
pairs.
We
classify
the
name
server
as
private
if
the
second
level
domain
and
top
level
domain
match
for
the
website
and
the
name
server
or
if
the
name
server
is
present
in
the
subject,
alternate
name
list
of
that
website.
B
B
In
general,
we
identify
approximately
10
000
third-party
dns
providers
in
our
data
for
measuring
cdns.
We
fetch
the
web
page
of
the
website
that
we
want
to
measure
the
cdn
of,
and
then
we
extract
all
the
resources
in
the
webpage
and
identify
the
resources
which
are
internal,
for
example,
for
reddit.com.
B
B
B
B
B
In
general,
we
identify
total
86
third-party
cdns.
In
our
data
for
certificate
authority
dependency,
we
just
fetch
the
certificate
of
a
website
and
from
that
extract,
the
links
for
the
ocsp
servers
and
the
crl
distribution
points,
and
we
use
the
same
techniques
which
are
tld,
matching
startup
authority,
non-matching
and
sandless
to
identify.
B
B
B
B
B
B
B
B
These
critical
dependencies
between
service
providers
give
rise
to
indirect
dependency,
but
why
should
we
care
about
indirect
dependencies?
I
mean
okay,
sure
they
exist,
but
what's
the
harm
turns
out
that,
as
a
result
of
indirect
dependencies,
we
see
huge
amplification
in
the
impact
of
a
given
provider,
for
example,
in
case
of
ca
to
dns
dependency,
cloudflare
now
critically
serves
37
of
the
top
100
000
websites
as
compared
to
24
before.
B
Hence,
because
of
these
indirect
dependencies,
we
see
amplification
of
provider.
Concentration
to
summarize
critical
dependencies
are
also
widespread
among
service
providers,
and
these
dependencies
can
amplify
the
concentration
of
service
providers,
which
means
that
the
effect
of
single
points
of
failures
in
the
internet
is
amplified
and
a
single
provider
can
now
impact
37
percent
of
the
top
100
000
websites.
B
This
is
what
we
observed
for
data
in
2020.
Now
we
see
how
the
world
changed
after
the
dyna
incident
by
comparing
2016
dependency
data
with
2020
data,
which
brings
us
to
our
third
question,
which
is
how
did
the
world
change
after
the
time
incident
did
websites
try
to
reduce
their
third-party
dependency
on
service
providers?
Did
the
concentration
among
providers
decrease.
B
B
B
B
B
For
websites
they
should
consider
redundancy
while
using
a
third
party
provider.
Moreover,
they
should
try
to
understand
their
indirect
dependency
arising
from
the
inter-service
dependencies
of
the
providers
that
they're
using
similarly
for
service
providers.
They
should
make
it
easy
for
websites
to
to
be
redundantly
provisioned,
as
we
observed
that
many
dns
providers
do
not
support
redundancy
and
also
service
providers
should
try
for
redundancy
in
their
inter-service
dependencies.
B
B
A
A
F
A
G
A
A
F
Okay,
can
you
hear
me
okay,
great,
so
thanks
a
lot
for
the
work.
So
it's
you
know,
any
studies
we
get
on
centralization
and
dependencies
is
fantastic
because
it's
definitely
a
trend,
as
you
have
proved
in
your
numbers.
You
made
one
statement
that
said
so
in
your
recommendations,
you
said
that
you
recommend
that
companies
be
more
transparent.
F
B
So,
actually,
I
have
a
multiple
sort
of
data
points
for
this,
for
instance,
currently,
let's
say
I'm
doing
research
in
the
redox
area
and
when
I
say
that
companies
should
be
more
transparent.
What
I
mean
is
that
they
should
be
transparent
about
what
attacks
they're
observing
so
that
we
can
sort
of
determine
the
skill
level
of
the
attackers
to
some
extent,
for
instance,
here
in
this,
in
this
attack
that
happened.
We
don't
really
know
for
for
the
dyna
tag.
B
Okay,
also
having
a
private
provider
rather
than
just
having
third
party
providers,
and
if
a
particular
provider
was
specifically
being
attached,
then
we
can
make
an
argument
that
okay,
maybe
we
should
be
redundantly
provisioned
and
not
in
just
this
attack,
but
there
are
many
other
attacks.
For
example,
currently
I'm
working
on
building
attackers
who
can
do
reconnaissance
on
on
different
service
providers
and
these
type
of
attacks
have
existed
for
so
long.
I
I
look
at
all
these
companies,
akamai
verizon
and
their
redox
reports,
and
I
see
that
they
talk
about.
E
Thank
you
again,
colin
alexa.
This
is
a
great
paper.
I
echo
what
what
wes
said.
It's
a
really
helpful
and
an
area
that
is,
I
think,
really
challenging.
You
know,
given
our
dependency
on
the
internet
today,
two
points.
First,
the
numbers
you
presented
in
terms
of
concentration
very
much
are
consistent
with
the
numbers
that
anna
maria
mandalari
at
the
imperial.
C
E
Showed
when
she
was
looking
at
iot
cloud
dependencies,
so
you
can
look
up
her
work
if
you'd
like,
I
think
she
presented
at
the
at
imc
as
well
at
some
point
about
this.
The
second
point
was
was
just
that
there's
an
activity
going
on
inside
the
ietf,
which
may
benefit
from
hearing
more
about
your
work,
which
is
benoit
clay's,
presented
something
called
sane.
Please
don't
ask
me
to
expand
the
acronym
I
couldn't
possibly,
but
it
essentially
is
a
dependency.
E
E
A
A
G
One:
hey
everyone:
my
name
is
kevin
bach
from
the
university
of
maryland.
Before
I
get
started,
I
wanted
to
give
a
huge
shout
out
to
all
the
ietf
organizers
for
making
this
event
possible
doing
all
this
remote
organization,
and
I
wanted
to
give
a
big
thank
you
to
all
the
collaborators
on
this
project.
It's
really
taken
a
village
to
get
this
going
now,
I'm
going
to
be
talking
about
censorship
today
and
specifically,
a
new
form
of
censorship
evasion.
We've
been
working
on
server
side
of
asia.
G
Before
I
get
into
that,
though,
I
wanted
to
give
you
all
a
brief
background
of
what
nation
state
censorship
looks
like
nowadays
to
motivate
some
of
the
approaches
we've
taken
now.
There
are
many
types
of
censorship
that
operate
around
the
world
today
and
today
I'm
going
to
be
talking
specifically
about
the
automated
in-network
censorship
that
operates
in
the
network
by
nation-states.
G
Now
some
nation-states
operate
censorship,
as
in-path
sensors
like
this
one,
with
the
sensor
physically
sitting
inside
the
network
path.
Other
sensors
operate
censorship
like
this,
instead
of
being
in
the
path
they're
on
the
path.
So
if
the
client
makes
a
forbidden
request,
you'd
see
this
request
moving
to
the
network.
Now
the
server
will
get
the
packet
and
the
sensor
will
too.
G
G
Specifically,
it
will
inject
spoofed,
tcp,
reset
or
tear
down
packets.
These
are
normal.
Packets
are
computers,
send
all
the
time
these
just
exist.
To
tell
the
other
side
stop
talking
to
me
immediately,
it's
going
to
send
one
of
these
packets
to
the
client,
pretending
to
be
the
server
and
one
of
these
packets
to
the
server
pretending
to
be
the
client
and
when
these
packets
arrive,
the
client
thinks
the
server
terminated
the
connection
and
the
server
thinks
the
client
terminated.
The
connection
immediately.
G
Both
of
these
sides
stop
talking
to
each
other
and
just
like
that
censorship
has
been
achieved
now
in
order
to
pull
off
this
attack
of
injecting
tear
down
packets.
The
sensor
needs
to
have
some
information
about
the
connection.
It
needs
to
know
the
port
numbers,
the
sequence
and
acknowledgment
numbers.
G
Now,
once
again,
our
client
is
about
to
generate
a
forbidden
request
to
this
resource,
but
this
time
we're
going
to
inject
a
tcp
reset
packet
of
our
own,
we're
going
to
set
it
in
such
a
way
that
the
ttl
or
time
to
live.
This
is
a
field
in
these
packets
that
tell
the
network
how
long
it
should
survive
in
the
network
and
it
decrements
once
per
hop.
G
We're
going
to
set
the
ttl
high
enough
such
that
we
reach
the
sensor,
but
not
so
high
that
we
reach
the
server
so
watch.
What
happens
when
we
send
this
packet
now,
just
like
before
the
sensor
will
get
a
copy
of
the
packet,
but
the
packet's
not
going
to
reach
the
server
it's
going
to
get
dropped
along
the
way.
G
Now
the
server
never
saw
this
packet,
but
the
sensor
has,
and
the
sensor
says
well,
it
looks
like
the
client
just
terminated
this
connection,
so
I
can
stop
tracking
the
connection
now
and
it
throws
away
the
state
it's
been
maintaining
about
the
connection
now
for
the
rest
of
this
flow,
the
client
and
server
are
free
to
communicate.
The
sensor
has
no
state
with
which
to
censor
us,
and
the
server
is
no
idea.
G
Now,
ideally,
servers
would
be
able
to
help
instead
of
deploying
software
at
the
client.
Instead,
we
would
deploy
it
at
the
server
and
if
such
a
thing
were
possible,
then
the
server
could
subvert
censorship
on
the
user's
behalf
without
clients
needing
to
deploy
anything
at
all
and
just
think
about
the
benefits
of
this.
This
would
immediately
broaden
reachability
and
accessibility
for
these
resources
without
clients
needing
to
do
anything.
Many
clients
connect
to
one
server.
Clients
no
longer
need
any
technical
expertise
or
to
download
anything.
G
Every
client
immediately
gets
plausible
deniability
and
it
helps
all
those
users
who
don't
have
technical
expertise
or
don't
know
they're
being
censored
in
the
first
place.
So
this
sounds
amazing.
The
problem
is
it
shouldn't
work
and
it
shouldn't
be
possible
and
to
see
what
this
is.
Let's
consider
the
waterfall
diagram
of
packets
that
are
exchanged
leading
up
to
some
censored
query
clients
going
to
send
a
sin
to
which
the
server
responds
with
the
synack.
G
The
client
completes
it
through
a
handshake
and
then
the
sensor
keyword
is
sent
from
the
server's
perspective,
though
there's
very
little
it
can
do
before
the
sensor
keyword
is
sent.
In
fact,
the
server
can't
influence
this
connection
past
the
synack
at
all,
underscoring
the
difficulty
in
this
space
there's
been
no
prior
work
on
evading
a
sensor,
evading
censorship
from
the
server.
G
This
tool
is
geneva.
Geneva
is
an
open
source,
originally
client-side
tool
that
my
team
built
a
few
years
ago
for
automatically
discovering
censorship
of
agent
strategies.
This
was
originally
published
by
my
team
in
2019
geneva
runs
strictly
one
side
of
the
connection
we
originally
designed
it
for
the
client
side
and
the
way
it
works,
is
it
manipulates
packets
as
they
enter
and
leave
the
system
now
geneva?
Is
a
genetic
algorithm,
so
in
fact
it
learns
how
it
should
manipulate
packets,
specifically
the
way
it
can
manipulate
packets
is
it
can
do
that
with
just
four
actions.
G
That's
duplicate!
You
take
one
packet
of
two
packets
tamper,
you
take
a
pack
and
you
change
it
in
some
way,
fragment.
Take
a
packet
break
it
in
half
or
drop
cause.
Take
a
packet.
You
drop
it
on
the
floor
now
I'll
call
out
two
things
specifically
here.
The
first
is
that
tamper
is
allowed
to
alter
or
corrupt
any
tcp
header
fields.
G
Importantly,
we
don't
give
it
any
semantic
understanding
of
what
these
fields
mean.
We
give
it
syntax,
but
not
semantics.
What
this
means
is
that
it
can
access
the
tcp
flags
fields,
it
can
change
the
flags
fields,
but
it
has
no
knowledge
that
if
I
set
the
flags
field
to
sin,
that
means
the
sort
of
a
connection.
G
G
Now
let
me
show
you
what
it
looks
like
when
geneva
puts
these
actions
together,
because
geneva
actually
composes
these
things
into
trees
and
these
trees
look
something
like
this.
At
the
top.
We
have
some
sort
of
match
or
trigger
okay
and
then
associated
with
that
match.
We
have
an
actions.
These
are
match
action,
pairs
and
a
strategy
is
comprised
of
some
combination
of
these
match
action
pairs.
G
G
G
This
talk
with
this
is
the
ttl
limited
reset
now
for
this
work,
specifically,
we
modified
geneva
so
that
it
could
run
on
the
server
side,
so
we
deployed
it
against
real
world
sensors
in
our
previous
work
and
a
lot
of
other
previous
work
in
the
space,
we've
tended
to
focus
on
http
in
this
work,
we've
brought
in
the
protocols
that
we
try
and
subvert
censorship.
For
so
specifically,
we
look
for
http,
https,
dns,
ftp
and
also
smtp
censorship.
G
G
G
So
this
is
one
successful
server-side
evasion
strategy
that
works
in
china.
Now
there's
a
lot
going
on
here.
So
let
me
break
this
down.
The
client
starts
off
just
like
normal
by
sending
a
send
packet.
Okay
now,
instead
of
the
server
responding
with
the
synack,
doesn't
respond
with
the
cynic
anymore.
Instead,
it
responds
with
two
syn
packets.
G
G
The
client
and
server
can
now
communicate
censorship
free
now
this
seems
crazy,
but
this
actually
works
with
varying
degrees
of
success
across
all
the
protocols
we
tested
and
for
context.
The
baseline
success
rate,
if
you
do
nothing
at
all,
for
evading
censorship
is
about
two
percent
now
this
is
one
successful
strategy,
but
it's
actually
just
one
of
many
altogether.
We
found
11
different
strategies
at
the
time.
We
did
this
work.
We
discovered
eight
in
china,
one
in
iran
and
india
and
another
three
in
kazakhstan.
G
G
G
Now
one
of
the
big
takeaways
from
this
is
that
it
really
has
helped
motivate
why
we
should
be
using
automated
techniques
to
discover
this.
This
is
likely
a
bug
in
this
sensor
that
it
may
have
been
difficult
for
a
human
to
discover
now.
Here's
another
example.
I
wanted
to
show
you
also
in
kazakhstan
now
in
this
one
during
the
during
the
three-way
handshake.
G
G
They
may
induce
some
behavior
from
the
client,
but
no
software
changes
as
needed,
and
we
tested
this
across
a
wide
range
of
diverse
clients
and
in
fact
we
found
only
one
case
of
a
strategy
that
relied
on
platform
independent
behavior.
We
were
able
to
very
quickly
write
that
strategy
to
make
it
platform
agnostic,
and
all
of
these
really
teach
us
more
about
how
sensors
work.
G
G
So,
specifically,
the
way
this
works
is
let's
say
the
sensor
misses
a
packet.
The
idea
is
simple:
it
can
just
re-synchronize
its
state
on
a
later
packet.
In
the
connection
now
the
exact
dynamics
of
the
resynchronization
state
are
constantly
evolving
over
time
and
understanding
how
the
resynchronization
state
operates,
allows
us
to
build
better
censorship
evasion
tools,
so
looking
at
the
resynchronization
state
gives
us
some
more
insight
as
to
why
that
first
strategy,
I
showed
you
actually
works
so
once
again,
here's
that
strategy.
G
G
G
The
next
packet
it
re-synchronizes
on
is
the
synack
packet
from
the
client,
but
because
we
force
the
tcp
simultaneous
open
when
you
do
a
tcp,
simultaneous
open.
The
mechanism
by
which
sequence
numbers
are
incremented
during
the
three-way
handshake
changes
and
the
great
firewall
is
not
properly
incrementing
its
initial
sequence
number,
while
it's
resynchronizing.
G
G
G
It
looks
something
like
this,
but
what's
going
on
here,
is
that
we're
finding
different
tcp
layer
bugs
for
each
protocol,
which
means
that
each
protocol
censorship
must
have
its
own
tcpip
stack,
and
this
strongly
suggests
that
the
great
firewall
is
actually
running
multiple
sensory
middle
boxes.
In
parallel,
it's
not
just
one
set
of
machines.
G
G
Now
you
may
look
at
this
be
like
well,
it's
obvious.
It's
just
got
to
be
using
the
port
number,
but
it's
actually
not
relying
on
port
number,
the
great
firewall
all
these
protocols
can
sensor
effectively
on
any
port.
If
you
try
and
make
an
http
request
for
some
forbidden
content
on
80
or
8000
or
a
random
port,
it
will
still
censor
you
the
same.
So
it's
not
using
port
number
we
think
is
going
on.
Is
that
how
does
it
know
which
middle
box
to
apply?
We
think
it
doesn't?
G
We
think
that
every
single
little
box
gets
a
copy
of
every
packet
independently
and
then
every
middle
box
applies
protocol,
fingerprinting,
every
middle
box
checks:
hey,
is
this
something
I
can
censor?
Is
this
belong
to
me?
Does
this
belong
to
me
and
then,
if
it
discovers
hey,
this
is
something
I
should
be
taking
action
against,
then
that
specific
middle
box
will
take
action.
G
This
raises
the
question:
where
are
these
guys
located
nearby?
Are
they
far
away?
So
we
did?
Was
we
used
ttl
limited
probes?
We
sent
forbidden
queries
with
different
for
different
protocols
with
different
ttl
limits,
and
we
tried
to
see:
are
these
located
in
the
same
spot
in
the
network
or
they
located
far
away?
G
We
find
is
that
largely
they're
co-located
at
the
network
level?
They
don't
seem
to
be
located
at
different
locations
now.
Next,
I
want
to
shift
gears
a
little
bit,
and
I
want
to
talk
about
where
we've
taken
these
ideas,
since
we
wrote
this
paper
now.
The
first
thing
I
want
to
talk
about
is
how
this
idea
of
service
activation,
and
this
idea
of
an
automated
approach
to
service
activation
has
allowed
us
to
be
highly
responsive
to
new
forms
of
censorship.
G
The
way
this
works
was
that
it
monitored
all
protocols
entering
and
exiting
the
country
on
certain
ports,
and
it
only
allowed
certain
of
those
protocols
to
happen
it
performed
vertical
fingerprinting
across
all
of
these
and
protocols
that
didn't
match
were
then
subject
to
censorship,
which
means,
if
you
tried
to
use
a
protocol
they
didn't
approve
of
regardless.
If
what
you
were
doing
as
innocuous,
they
would
take
that
connection
down.
G
So,
within
a
small
amount
of
time
of
this
censorship
system
being
deployed,
we
threw
geneva
at
this
problem
and
we
were
able
to
discover
four
strategies
to
evade
this
filter
four
from
the
server
side.
So
this
means
that
as
soon
as
iran
rolled
this
out,
we
were
able
to
defeat
this
thing
and
help
roll
out
those
evasion.
Tactics
to
those
in
iran,
another
example
of
new
censorship
events
that
occurred
is
last
summer.
China
began
censoring
the
use
of
encrypted
sni.
G
Now
the
sni
field
is
filled
in
tls
1.2,
where,
when
a
client
connects
to
a
server,
the
client
announces
in
plain
text
the
domain
it's
trying
to
get
to
sni
is
clearly
a
privacy
leak
and
the
tls
developers
are
working
on
fixing
that,
but
sni
is
how
sensors
have
been
censoring
https
for
quite
some
time
es,
and
I
people
have
been
looking
forward
to
is
the
mechanism
by
which
they
can
protect
themselves
from
smi
based
censorship.
And
unfortunately,
last
summer
china
decided
to
ban
the
use
of
esni
completely.
G
G
G
G
It's
good
because
this
has
opened
up
the
ability
to
do
censorship,
evasion
to
more
people,
even
those
who
don't
know
they're
being
censored
now.
This
is
made
possible,
though,
by
the
ugly
fact
that
middle
boxes
have
bugs
and
bad
assumptions
that
can
be
exploited,
and
unfortunately,
this
can
also
lead
to
some
very
bad
directions.
G
A
F
Well,
since
you
needed
questions
so
then
two
more
people
popped
up,
so
it
was
unclear
to
me.
Maybe
I
missed
it.
How
did
you
determine
ground
truth
for
if
you
were
doing
server
side,
you
know
studies?
How
do
you
decide
whether
something
is
successful
or
not
and
actually
got
transferred
to
and
from
the
client
properly.
G
Can
you
hear
me,
let
me
know
if
my
audio
is
not
great.
So
in
our
case
we
have
the.
We
have
the
good
fortune
of
being
able
to
control
both
the
client
and
the
server.
So
we
did
our
experiments.
We
actually
had
vantage
points
located
in
all
of
these
sensor
countries,
and
so
we
could
ground
truth
was
very
easy
to
determine.
We
could
try
and
obtain
some
forbidden
resource,
see
the
sensor,
take
action
and
then
try
and
do
that
again
with
the
server-side
strategy
running
and
see
it
succeed.
G
Now,
when
you're
talking
about
scaling
this
up
to
cases
which
you
don't
control
the
client
or
some
of
our
more
real-world
deployments
that
we've
been
working
on
recently,
it
starts
getting
a
little
trickier
when
you
start,
you
need
to
really
look
at
telemetry
and
look
very
carefully
at
what
the
if
the
packets
are
getting
exchanged.
If
the
connection
is
staying
alive,
that
sort
of
thing,
I
will
say
most
sensors
when
they
take
action
against
the
connection,
it's
fairly
obvious,
like
it
they're,
usually
not
quite
about
it.
I
For
the
talk,
can
you
hear
me
I
just
so
I
I
think
you
might
have
touched
upon
this,
but
when
it
comes
to
deploying
this
at
scale,
were
you
I
think
you
mentioned
like
tunnelbear
someone?
You
were
working
with
british
wondering?
Were
you
thinking
about
the
coming
up
the
strategies
in
a
centralized
way
and
then
distributing
those
strategies
to
every
deploy
to
every
server
running
geneva,
or
were
you
thinking
that
every
individual
deployment
server
deployment
would
come
up
with
its
own
strategies?
G
For
initial
rollout,
the
the
methodology
has
been
will
be
kind
of
a
strategy
oracle
and
provide
strategies.
I
didn't
get
to
talk
about
it
too
much
of
the
talk
but
geneva's
broadly
broken
into
two
halves.
There's
the
component
that
runs
the
strategy
on
the
network
and
the
genetic
algorithm
that
discovers
the
strategies
and
those
are
fairly
self-contained.
G
So
for
now,
we've
been
we've
had
success
with
having
third
parties
deploy
the
strategy
engine,
the
thing
that
takes
the
strategy
and
runs
it
on
the
network
and
we've
been
more
strategy,
oracle,
providing
strategies.
That
being
said,
there
are
more
and
more
groups
who
are
working
to
find
strategies
themselves,
and
so,
as
that
becomes
more
decentralized,
that'll
things
will
just
continue
to
improve.
H
Yeah
and
thanks
a
lot
for
this
talk,
it
was
very
interesting
to
learn
what
the
sensors
are
doing
there
and
I
probably
have
to
read
the
rest
of
the
paper
as
well
to
understand.
What's
happening
there.
We
talked
a
little
bit
in
the
chat
about
this.
Is
a
cat
and
mouse
game
right.
Can
you
just
tell
a
little
bit
about
like
how
long,
because
I
think
somebody
said
in
the
chat
that
some
of
these
bugs
have
been
fixed
so
like?
What's
the
timeline,
how
long
did
it
take?
G
Yes,
that's
a
really
good
question
and
that's
something
we're
still
studying
of
what
does
it
look
like
when
a
nation
state
rolls
out
a
bug
fix
like
how
long
does
that
take?
Where
does
that
go,
but
still
something
we're
studying,
and
I
don't
think
I
have
a
great
answer
for
you,
the
yeah.
I
think
I'll
put
a
tbd
stay
tuned
on
that.
It's
something!
That's
super
interesting
we're
going
to
keep
monitoring
and
we
do
test
these
things
fairly.
G
For
example,
a
country
like
china
may
have
may
have
many
instances
of
middle
boxes,
and
I
mean
perhaps
the
fix
get
rolled
out
somewhere
and
it
takes
time
to
traverse
or
there's
different
sets
of
middle
boxes
doing
things.
So
it
really
takes
quite
a
concerted
effort
to
measure
these
things
at
scale
and
get
a
good
understanding
of
how
the
fixes
move
through
the
network.
If
you
will
so
a
big
tbd
on
that,
but
it
is
a
super
interesting
question.
H
J
So
it's
an
arms
race
and
no
side
will
win.
I
think
by
definition,
possibly
china
might
because
they
got
more
resources
than
anybody
else.
But
notwithstanding
that,
my
concern
is
for
the
non-nation
states
affected
by
this.
So
if
you
like,
for
the
typical
end
user
versus
the
malicious
content
developer,
clearly
server-side
evasions
in
the
hands
of
malware
has
seriously
bad
implications
for
for
most
end
users.
J
G
It's
a
true
statement
generally
that
anything
like
this
can
be
used
for
good
and
bad
and
malicious.
I
think
dave
said
it
best.
Malicious
is
in
the
eye
of
the
beholder,
so
we've
intentionally
not
taken
this
work
and
applied
it
towards
the
firewalls,
like
you're,
describing
like
this
really
has
been
a
very
concertedly,
focused
effort
on
nation
states
to
try
and
at
least
angle
things
towards
the
good
side,
as
as
we
can
help
direct
it.
I
think
it's
difficult
to
quantify
if
the
who
who's
affected
the
majority.
G
Here
I
mean
there
are
billions
of
people
living
under
the
centered
regimes,
there's
also
billions
of
people
who
would
like
to
be
protected
by
firewalls.
So
it's
kind
of
a
tough
question
to
answer,
but
it
is
something
we've
been
thinking
about
and
it
was
a
serious
consideration
we
were
talking
about.
Should
we
release
this
tool?
Should
we
make
it
open
source
at
the
end
of
the
day,
we
we
thought
it
best
to
put
it
out
there
and.
J
A
Yeah,
it's
a
difficult
problem.
I
I'm
tempted
to
say
that
the
only
winning
move
is
not
to
play
just
following
on
a
little
bit
from
that.
Can
you
say
I
think
you've
touched
on
this
to
some
extent
already.
Can
you
say
something
about
how
you
did
the
tests
and
how
you
protected
the
the
users
who
are
running
the
tests
in
these
countries.
G
Are
we
good
okay,
we're
good?
In
these
cases
we
did
all
these
tests
ourselves,
so
there
were
no
end
users
directly
involved
and
the
no
end
users
were
harmed
in
the
making
of
this
paper.
If
you
will
and
no
end
users
were
we,
we
tried
very
hard
to
not
put
any
end
users
at
risk.
It
was
from
machines
directly
at
our
control
and
we
monitored
all
that
very
carefully
going
forward.
A
All
right
great,
thank
you
very
much.
We
are
basically
out
of
time
here.
I
think
we've
had
three
really
interesting
talks.
We've
had
a
bunch
of
really
interesting
discussion.
If
you
anyone
here
would
like
to
follow
up
with
the
the
rest
of
the
office,
please
do
reach
out
to
them.
I'm
sure
everyone
involved
will
be
happy
to
chat
more
the
you
know.
People
may
be
around
for
the
rest
of
the
meeting.