►
From YouTube: IETF99-IRTFOPEN-20170720-1550
Description
IRTFOPEN meeting session at IETF99
2017/07/20 1550
https://datatracker.ietf.org/meeting/99/proceedings/
A
B
C
A
B
A
A
You
know
shaking
hands
kind
of
things
so
tolerate
us
during
that,
and
then,
if
we
have
a
little
bit
more
time
at
the
end,
I
want
to
get
people
to
just
say
I'm
on
a
listening
expedition.
What
do
you
need
in
terms
of
publications
if
you're
working
in
the
IRT
F
so
be
thinking
about
that
and
come
up
to
the
mic
and
talk
about
that?
A
Okay,
so
the
for
those
who
don't
know
I,
think
you
do.
We
have
a
a
parallel
between
the
IR
Tia
and
the
ietf
PIR
camp
was
founded
to
kind
of
take
on
the
longer-term
work
and
to
not
have
some
of
the
process
strictures
and
and
timeframe
that
IETF
has
so
it
actually
goes
back
to
the
beginnings
of
the
IETF.
It's
not
a
new
development
by
any
means
and
we're
organized
into
a
bunch
of
research
groups.
This
is
there's
not
a
research
group.
A
This
is
the
area
of
meaning
and
then
also
there's
a
steering
group
with
the
research
chairs,
research
group
chairs
and
some
at-large
members,
so
because,
if
you
are
at
the
plenary,
you've
heard
me
say
this
last
night
too,
for
it's
very
unusual
that
everybody
meets
at
the
ITF.
But
this
time
everybody
did
so
I
think
everyone
must
like
Prague,
because
they
don't
have
to
meet
an
idea.
They
can
meet
in
different
ways,
so
that
this
is
the
list
of
the
groups
we
have,
and
we
also
had
a
proposed
research
group.
A
The
path
aware,
networking
research
group
pan
our
G,
which
is
our
our
our
G's,
are
formed
in
a
different
way
from
working
groups.
They
they
convince
the
chairs
with
some
consultation,
and
then
we
give
them
a
year
to
take
off
or
not
take
off
so
they're,
not
exactly
chartered
but
they're
in
the
proposed
state.
Now,
and
then
we
also
had
some
initial
organizing
for
a
meeting
a
group.
That's
currently
called
din
that
would
be
about
blockchain
and
crypto
Ledger's,
and
things
like
that
needs
to
organize
its
thoughts.
A
Okay
and
Saturday,
we
had
the
annual
applied
networking
research
workshop,
there's
where
the
proceedings
are,
and
you
could
also
see
the
recording
if
you
weren't
there.
So
one
thing
that's
confusing
is
we
have
the
AARP
and
the
and
our
W,
and
so
the
al
RW
is
this
kind
of
a
standard
research
workshop
emphasizing
applied
networking,
so
the
intersection
of
in
of
ITF
tight
style
things
that
we
do
and
research,
and
so
that's
going
to
happen
again
next
summer.
A
A
So
you
can
just
run
your
eyes
over
that
and
then
we,
my
cats,
don't
look
very
good
in
green
and
purple,
but
but
I
find
it
coming
to
see
them.
And
so
we
have
a
discuss
mailing
list
and
an
announce
mailing
list
that
you
can
join
and
I
RTF
dot.
Org
has
links
to
extensive
information
on
the
research
crews,
but
you
can
also
find
the
research
groups
through
the
data
tracker.
A
Okay,
so
we're
going
to
talk
about
we're
going
to
move
on
to
the
applied
networking
research
prize
talk
starting
with
Philip,
but
I
also
wanted
to
mention
that
this
prize
is
a
submission
of
papers
that
were
already
published.
So
it's
kind
of
like
a
best
paper
but
Pam
internet
best
paper
award.
So
we
ask
for
people
to
submit
what
they
think
of
as
the
best
papers
they've
they've,
seen
anywhere
and
to
nominate
a
speaker
who
would
benefit
from
experiencing
the
ITF
and
the
IRT
F.
A
So
the
winners
of
this
receive
a
trip
to
the
IETF,
and
some
hospitality
from
I
saw
involving
attending
the
the
the
ITF
fellows
dinner
I
believe,
and
you
should,
even
though
we're
meeting
this
this
late.
You
should
think
about
ways
that
you
might
encourage
these
speakers
to
to
participate
in
IETF
when
you
hear
what
they've
talked
about
so
with
that
I'm
going
to
introduce
Philip,
who
is
just
about
to
defend
dissertation.
Is
that
right,
yeah
and
then
move
to
a
position
with
Dave
Clark
at
MIT
and
has
a
great
study
of
a
multi
perspective.
D
E
F
E
Okay,
yes,
I'm,
going
to
present
our
work
on
carrier
grade
nat
deployment
in
the
internet.
This
is
a
paper
that
appeared
in
our
last
year's
IMC
conference
and,
if
you're
interested
in
the
PDF,
you
can
also
get
it
using
this
tiny
URL
link
here.
I
also
want
to
acknowledge
my
co-authors
here.
So
this
is
joint
work
with
Florian
now
CEO
mark
Randy,
Anja,
Christian,
Nick
and
burn
okay.
I
think
there
is
no
need
for
a
long
introduction
of
what
the
problem
here
is.
I
guess
everybody
in
the
room
is
aware
of
the
problem
of
ipv4.
E
Address
space
exhaustion
it.
In
fact
it
receives
a
lot
of
attention
more
recently,
also
in
the
media,
and
it
is
also
a
long-standing
problem.
So
I
just
found
this
quote
here
a
couple
of
days
ago
from
IAB
from
an
IAB
meeting
in
1992
and
already
back,
then
the
problem
of
too
few
ipv4
addresses
was
clearly
named
as
a
clear
and
present
danger
to
the
future
successful
growth
of
the
worldwide
Internet.
E
Now
that
was
in
1992,
if
you
know
fast
forward
to
2017
now
we
have
a
situation
in
for,
in
which
4
out
of
5
of
the
regional
internet
registries
that
manage
global
IP
address
allocations,
exhaust
Sedaris
pools.
Currently
we
have
only
about
1%
of
the
globally
routable
ipv4
address
space
that
is
still
under
located
and
usable.
E
And
of
course
the
question
is:
what
do
networks
now
do
to
mitigate
their
ipv4
shortage
issues,
and
the
first
thing
that
comes
to
mind,
of
course,
is
the
transition
to
ipv6,
which
will
present
a
long-term
solution
to
the
scarcity
problem
and
internet
also
receives
a
lot
of
attention
from
the
measurement
community.
So
when
it
comes
to
ipv6,
we
have
plenty
of
measurements
and
statistics
available.
I
will
just
show
you
two
very
common
ones.
You
might
have
seen
them
a
couple
of
times
already
in
the
last
days.
E
The
first
one
is
from
Google
and
it
basically
tells
us
that,
as
of
today,
about
20%
of
the
hosts
contact
in
Google
and
us
are
using
ipv6.
The
second
one
is
from
a
pinna
class
that
measures
ipv6
adoption
on
a
per
country
level,
and
here
we
can
see
that
we
are
doing
quite
well
in
some
countries,
but
still
also
there's
a
lot
of
red
color
in
this
in
this
map.
E
Now,
besides
transitioning
to
ipv6
is
peace
can
also
use
other
means
to
mitigate
address
shortage
issues
and
in
particular
they
have
two
other
ways
of
doing
that
now.
The
first
one
is
to
buy
ipv4
address
on
address
markets,
and
the
IRS,
like
the
registries,
recently
introduced
policies
that
allow
these
transverse
and
also
publish
statistics
regarding
these
transfers.
Also,
here
we
have
data
available
that
we
can
that
we
can
look
at
to
track
how
these
markets
are
evolving.
What
I
show
you
here
is
just
some
data
pulled
from
the
our
IRS.
E
That
shows
some
statistics
of
the
number
of
address
blocks
that
are
currently
or
where
are
bought
and
sold
and
on
address
markets
in
the
recent
years,
and
you
can
see
that
this
is
indeed
the
markets
are
indeed
taking
off.
But
then
there's
a
third
thing
that
network
operators
of
some
networks
can
do,
and
that
is
that
they
can
use
ipv4,
carry
a
great
net
to
to
multiplex
more
users
behind
a
fewer
public
ipv4
addresses,
and
when
we
started
looking
at
carry
a
great
net.
We
soon
realized
that
there
are
no
deployment
statistics
available.
E
So
it's
really
hard
to
keep
track
of
like
how
popular
this
way
of
combating
before
exhaustion
really
is.
You
know,
so
very
little
is
known
about
how
carrier-grade
nets
are
configured
in
the
wild,
and
this
is
the
focus
of
this
work
and
also
of
this
talk
all
right
now.
First,
we
wanted
to
get
a
bit
of
an
understanding
of
the
problems
like
that
ISPs
face
when
deploying
carrier-grade
NAT.
E
So
we
have
formed
a
little
survey,
so
we
asked
we
asked
ISPs
about
their
ipv4
scarcity
issues
and
whether
they
want
to
deploy
ipv4
carry
a
great
lap.
So
we
circulated
this,
and
eventually
we
received
some
75
answers.
Just
want
to
quickly
show
you
some
of
those
the
highlights
of
that.
So
probably
the
most
important
question
that
we
asked
is
like
did
you
or
do
you
plan
to
deploy
before
carry
a
great
nap,
and
we
were
quite
surprised,
I
mean
of
course,
75.
E
E
So
we
went
on
and
asked
him
also
a
little
bit
more
about
carrier
grade
nat
specifics,
so
we
asked
them
what
if
they
have
any
operational
concerns
when
when
introducing
he
for
carrier
grade
nat-
and
they
were
plenty
so
one
of
the
most
commonly
mentioned
one
was
that
subscribers
might
experience
problems
with
applications
that
just
don't
work
well
behind
carrier
grade
nat.
So
a
gaming
was
mentioned
several
times.
Another
issue
that
came
up
frequently
is
the
traceability
of
users
behind
carrier
grade
nat
and
also
issues
with
publicly
facing
gateway.
Ip
addresses
becoming
our
blacklisted
now.
E
We
also
asked
some
of
what
they
see
as
the
major
challenges
or
caveats
when
introducing
carrier
grade
nat
troubleshooting
connectivity
issues
was
was
frequently
mentioned,
but
also
more
generally,
how
to
how
to
dimension
carrier
grade
nodes,
meaning
how
to
do
the
resource
allocation
like
how
many
IP
addresses.
How
do
we
allocate
IP
addresses
and
ports
to
subscribers
and
how
do
I
set
the
right
quotas
for
individual
subscribers?
E
One
thing
that
summarized
piece
mentioned
I
and
I
want
to
mention
that
here,
because
we
will
also
see
some
effect
of
that
in
our
measurements
is
that
their
problem
would
be
for
carrier
grade.
Nat
is
a
shortage
of
internal
address
space
and
it
means
basically,
they
have
used
the
available
RFC
1918
ranges
already
in
their
management
network
and
not
have
problems.
Finding
address
ranges
that
they
can
use
behind
a
carry
great
nut.
E
Okay,
we
also
get
them
a
free
text
field
where
they
can
tell
us
more
about
what
they
think
about
carrier
grade
nat,
and
what
we
can
really
see
is
that
most
of
them
were
not
really
fond
of
deploying
before
carrier
grade
net.
As
you
can
see
also
from
the
first
comment,
but
a
surprisingly
large
number
of
networks
also
told
us
that
v6
deployment
seems
to
be
costly
and
difficult
for
them,
so
some
of
them
just
opted
to
first
go
with
carry
a
great
nap.
E
So
basically,
that
really
motivated
us
to
study
this
more
because
it
seems
to
be
widely
deployed
and
the
networks
also
voice
concerns
like
regarding
carrier
grade
nat
deployment.
At
the
same
time,
we
don't
have
broad
and
systematic
studies
available,
so
what
we
did
as
we
developed
our
methods
to
detect
carrier
grade
nat
presence
in
the
internet
and
also
methods
that
allow
us
to
extract
some
of
the
properties
of
commonly
deployed
cg
net
instances.
E
Before
I
talk
about
our
methods
just
want
to
quickly
introduce
some
terminology,
so
here
in
this
figure,
you
see
a
common
case
of
our
subscriber
is
connected
to
to
the
internet.
So
we
have
the
isp
in
the
center,
the
ISPs
some
public
ipv4
address
space,
yes
with
other
networks
in
the
internet,
towards
the
right
enhancer
or
the
public
IP
ad
for
address
to
its
subscribers.
In
turn,
the
subscriber
will
run
a
home
that
and
reconnect
these
are
her
devices
behind
that
mat.
So
that
would
be
the
canonical
page
here.
E
E
Okay,
I
mean
this
is
what
harry
great
net
deployment
looks
like
on
the
white
or,
of
course
like
in
practice.
There
are
more
complicated
scenarios
are
possible,
so
our
goal
was
to
find
ways
to
detect
the
presence
of
these
devices.
This
turned
out
to
be
really
difficult
because
gnats
operate
transparently
right,
meaning
that,
if
you
monitor
traffic,
if
you
monitor
packets,
there's
no
easy
way
to
tell
whether
it
is
packet
was
translated
at
some
point
on
the
path
before
we
can
see
it.
E
So
we
had
a
hard
time
with
that,
but
eventually
we
found
two
methods
that
allow
us
to
detect
carrier
gretna
presence,
and
I
will
now
introduce
them
to
you.
The
first
one.
The
first
method
that
we
found
to
be
working
is
what
you
call
detecting
cg
net
from
the
outside
using
BitTorrent.
We
call
this
one
from
the
outside,
because
with
this
method,
we
don't
need
any
probing
devices
within
the
networks
if
you
want
to
test,
but
we
can,
this
detection
works
by
crawling
the
BitTorrent
DHT.
Let
me
quickly
walk
you
through.
E
I
guess
everybody
here
knows
the
BitTorrent
protocol
and
in
the
in
in
the
classic
way
baton
would
work
the
way
that
you
want
to
download
a
specific
talent.
You
will
connect
to
a
tracker.
Tell
the
tracker.
I
want
to
download
torrent,
XYZ
and
turn
the
tracker
will
give
you
a
list
of
IP
addresses
in
port
numbers
of
other
peers.
You
connect
to
them.
E
So
in
order
to
do
that,
we
implemented
DHT
crawler,
which
is
basically
a
machine
that
continuously
asks
the
current
peers
across
the
Internet,
the
term
peers
participating
in
this
DHT,
and
we
asked
him
hey.
Please
give
us
a
list
of
IP
addresses
and
port
numbers
how
it
appears
that
you
recently
interacted
with
an
intern.
They
will
tell
us:
ok
can
reach
Pierre.
Each
beer
has
a
unique
ID.
I
can
reach
P,
X
Y
Z
using
IP
address
and
port
number.
Now
we
have
to
scroll
up,
we
implemented
as
:.
E
We
had
to
scroll
a
runny
by
the
way,
initially
not
for
the
purpose
of
detecting
nap,
but
it
turned
out
that
we
can.
We
can
do
it
partially
with
that,
so
we
had
a
scroller
running
and
after
a
while,
we
realized
that
in
some
instances
we
have
the
time
you
have
the
term
peers
like
this
peer
here
that
apparently
sit
behind
some
form
of
an
app
and
tell
us
that
they
can.
They
have
connectivity
to
another
peer
using
an
internal
IP
address.
So,
for
example,
sure
in
this
case
we
ask
a
specific
peer
to
peer.
E
Tells
us
I
can
reach
peer,
8
h2d,
using
an
internal
address,
so
they
leak
as
internal
IP
addresses
by
habit
or
DHT.
Now,
when
we
realize
that
this
is
happening,
we
tailor
the
crawler
to
extract
in
harvest
as
many
of
these
internal
end
points
from
the
baton
DHT,
and
we
can
do
that
quite
efficiently.
So,
within
one
week
of
crawling,
we
can
learn
private
IP
addresses
from
more
than
700
thousand
peers
in
five
thousand
guesses.
E
Okay,
so
to
better
understand
this,
and
also
to
be
able
to
resect
these
small
whole
nights
from
carrier-grade
nuts,
you
have
to
have
a
closer
look
at
the
leakage
relationships
and
what
we
basically
do
in
order
to.
What
we
basically
do
is
that
we
form
a
graph
and
whenever
we
have
a
peer,
the
term
PA
that
leaves
us
the
contact
information
for
a
peer,
be
like
this
one
and
PB
has
a
private
IP
address.
E
We
will
form
two
vertices
in
the
graph
where
the
leaking
PL
PA
is
a
big
blue
Burgess,
and
the
internal
peer
to
peer
with
the
internal
IP
address
is
a
small
red
verges.
Then
we
draw
this
arrow
to
show
hey
PA
litres
PB,
and
then
we
have
a
look
at
these
leaking
relationships
at
a
larger
scale.
So,
namely
we
form
a
graph
for
each
is
in
which
we
crawl
return
PS.
Now
let
me
what
these
graphs
look
like,
so
this
is
a
tiny
subset
of
our
data.
E
Of
course
we
recall
many
more
of
these
peers
and
it
shows
us
to
a
SS
and
in
the
first
a
s
we
did
not
detect,
carry
a
great
nap,
meaning
that
here
we
do
see
leakage.
We
see
patrol
MPs
leaking
us
internal
IP
addresses,
but
typically
the
only
League
is
one
or
two
and
also
on
that
is,
more
importantly,
all
of
these
leakage
relationships
are
isolated.
E
All
right,
of
course,
I
mean
there
is
much
more
thresholding
involved
and
we
have
to
pre
post
process
these
graphs.
We
have
more
details
on
that
in
the
paper,
but
here
I
just
wanted
to
give
an
impression
of
what
this
signal
looks
like
for
NAT
inveterate,
alright.
So
with
this
method,
we
can
test
some
roughly
at
2,700,
a
SS
in
which
we
crawl
a
sufficient
number
of
the
torrent
peers,
and
we
detected
carry
a
great
nap
meaning
these
clusters
on
in
one
or
two
hundred
fifty.
E
Now
the
benefits
are
clear
that
we
have
a
broad
coverage
and
if
we
can
just
do
that-
and
we
don't
need
to
put
any
probing
devices
within
within
these
networks,
the
problem
is
that
first,
we
need
a
lot
of
the
torrent
activity
in
a
SS
so
that
we
can
even
they
can
at
least
in
theory,
show
up
in
our
measurements
and
secondly,
is
even
if
we
have
significant
baton
activity
in
a
specific,
a
s
that
deploys
carrier-grade
net.
It
will
not
necessarily
show
up
here
because
it
kept.
E
The
net
must
be
configured
in
a
very
open
way,
namely
it
must
expose
internal
IP
addresses
of
the
torrent
peers
to
each
other.
So
with
this,
we
will
only
partially
capture
the
carrier
grade
nat
landscape.
So
we
complemented
that
with
a
second
approach,
which
is
what
we
termed
detecting
CG
net,
from
the
insight
using
using
metalizer.
We
call
this
approach
from
the
inside,
because
here
we
have
full
control
over
probing
devices
that
are
located
behind
the
carrier-grade
NAT,
so
I
want
to
quickly
introduce
Naturalizer.
E
Maybe
some
of
you
folks
know
that
knit
eliza
is
a
network
troubleshooting
suite.
It
is
developed
by
XE
in
Berkeley.
You
can
download
it
as
an
Android
app
or
as
a
Java
applet,
or
you
can
use
it
as
a
as
a
command
line
tool
and
when
you
download
metalize
done
executed,
it
will
run
a
series
of
tests
regarding
the
health
of
your
connectivity
and
if
we
also
show
you
a
detailed
report
showing
you
lots
of
properties
of
your
arm
internet
connection.
E
At
the
same
time,
when
you
run
metalizer,
you
will
support
our
research
because
we
use
that
in
measurement
studies
and
in
this
particular
study
we
use
data
captured
in
more
than
550,000
sessions.
Like
an
accession
just
means
a
person
executed
Naturalizer
and
they
cover
more
than
1500
guesses,
and
the
benefits
of
this
metalizer
data
is
that
it
gives
us
direct
access
to
the
IP
address
of
the
client
of
the
router
and
also
the
public
IP
address
in
advance
and
set
alignment
cellular
networks.
E
And
since
we
have
full
control
over
Naturalizer,
we
also
implemented
more
customized
tests.
Now
we'll
talk
more
about
them
in
a
second
just
a
quick
sketch
on
how
we
detect
carrier-grade
NAT
presence
with
metalizer.
Now
here
we
have
two
different
cases.
We
have
two
cellular
network
case
in
the
cellular
case.
Detection
is
straightforward,
because
in
a
cellular
network
the
IP
address
of
the
device
will
directly
be
assigned
by
the
ISP,
so
we
can
compare
the
device
IP
address
against
the
IP
address
that
we
see
at
our
measurement
servants.
Okay,
it
was
translation.
E
Now
the
benefits
here
are
clear:
we
have
much
more
control
over
this
measurement
point.
It
gives
us
direct
access
to
the
IP,
addressing
data
that
we
need
to
detect,
detect
area
great.
Not
the
problem
here
is
again
partial
visibility.
This
is
crowd-sourced
data
and
we
are
at
the
mercy
of
users
executing
this
for
us.
Okay,
now
let
me
show
you
some
key
regret,
not
deployment
statistics.
Now.
First,
when
we've
been
talking
about
like
what
percentage
of
networks
do
calls
carry
a
great
net.
E
Of
course
we
need
you
to
define
into
what
kind
of
networks
we
are
interested
in,
and
so
in
this
work
you
want
to
focus
on
eyeball
networks,
meaning
a
SS
that
primarily
connect
end
users
to
the
Internet.
So
the
first
thing
that
we
did
is
that
we
define
our
a
set
of
eyeball
asked
by
a
population
of
a
SS
that
we
consider
eyeball
a
SS,
and
we
do
that
using
two
different
data
sources.
The
first
is
to
Spamhaus
pv
ellis.
E
This
is
a
basically
a
list
where
you
can
register
residential
address
ranges
and
the
second
one
is
a
data
set
from
a
pina
gloves,
which
gives
you
an
estimation
of
the
number
of
subscribers
in
a
particular
areas.
So
we
use
these
two
to
refine
our
set
of
eyeball
asses,
which
is
about
3000
a
SS,
and
we
may
have
a
look
at
how
many
of
them
like
we
cover
with
our
measurements.
E
E
E
E
Now
both
our
methods
allow
us
to
study
that,
and
we
showed
it
here
for
again
for
the
non
cellular
and
for
the
cellular
case,
and
what
we
can
see
is
that
the
RFC
1918
ranges,
particular
least
10/8-
is
still
the
most
common
commonly
used
one
empty,
relatively
newly
allocated
164,
which
is
specifically
allocated
for
the
purpose
of
deploying
carrier.
Great
not
only
slowly
takes
off,
but
what
is
probably
more
intriguing
to
see
here
is
that
up
to
20
percent
of
the
SS
already
used,
multiple
internal
address
ranges
behind
their
carrier.
E
Great
not
now,
if
we
recall
also
from
our
survey
that
it
seems
to
be
an
increasing
problem
for
SS
to
find
usable
internal
address
ranges
to
use
behind
the
net
now
the
schools.
As
far
as
that,
in
the
case
of
some
major
cellular
networks,
they
seem
to
see
the
need
to
use
routable
IP
IP
addresses,
as
internal
IP
addresses
behind
their
carrier
grade
that
and
in
this
figure
here,
I
show
you
some
of
the
most
prominent
cases.
I
want
to
say
here:
you
don't
want
to
do
it
any
name
shaming.
E
We
see
this
behavior
form
or
ESS,
and
we
just
chose
to
show
these
six
because
they
are
like
really
big.
So
we
have
a
lot
of
data
and
can
be
confident
in
our
reasoning
here.
So
what
we
show
here
on
the
x-axis
is
the
corresponding
relative
address
block.
Let
me
here
we
show
the
network's
did,
use
them
as
internal
space.
Now.
E
What
we
want
to
highlight
here
is
that,
like,
for
example,
let's
have
a
look
at
the
25/8
address
block
I
recall
correctly,
it
is
UK
Ministry
of
Defense,
and
currently,
if
you
go
from
the
global
routing
table,
this
address
block
is
not
in
use,
and
but
what
we
see
is
that
there
are
at
least
four
major
networks
that
use
these
addresses
internal
internally.
There
might
even
be
more
as
such.
E
This
is
just
a
subset
of
them
and
we
would
argue
that
the
question
is
what
happens
if
somebody
wants
to
finally
use
this
address
block,
given
its
scarcity
will
increase.
We
can
imagine
that
at
some
point,
people
want
to
buy
and
use
this
address
block
and
we
can
likely
expect
it
a
lot
of
users
behind
these
users
of
these
of
these
networks
will
have
severe
connectivity
issues
here.
E
E
So
one
thing
that
we
do
is
that
whenever
you
run
metalize
or
net
Eliza
will
initiate
ten
subsequent
TCP
connections
from
the
device
to
our
measurement
server,
and
what
we
can
then
do
is
that
we
compare
local
IP
addresses
and
port
numbers
to
remote,
IP
addresses
and
port
numbers,
but
that
we
can
study
how
carrier
grade
nads,
allocate
ports
and
IP
addresses,
and
for
some
of
them
I
will
show
you
an
example.
We
can
also
estimate
like
how
many
products
in
an
individual
subscriber
receives.
E
We
also
implemented
a
custom
TTL
TTL
based
nap
test.
Basically,
what
this
test
does
is
that
it
sends
TTL
limited
probes
to
first
create
state
in
all
nets
on
the
path
and
then
to
retain
it
in
some
hops,
but
to
expired
in
other
hops,
and
with
this
test
we
can
pinpoint
the
location
of
nets
on
the
path,
including
carrier-grade
nuts,
and
also
extract
time
all
values
of
the
Nats.
E
And
lastly,
we
also
implemented
a
stun
test
which
allows
us
to
compare
the
kind
of
map
mappings
that
a
carrier
grade
map
would
give
you,
in
contrast
to
your
home,
router
Oki,
there's,
probably
not
enough
time
to
go
into
all
of
these.
All
of
these
results.
I
just
picked
some
of
them.
That
I
quickly
want
to
show
you.
One
thing
that
we
found
quite
surprising
is
that
up
to
20%
off
to
carry
a
great
map,
instances
that
we
detected
use
what's
called
arbitrary
pooling,
but
it
basically
means
these
are.
E
These
are
Nats
that
have
ranges
of
public
ipv4
addresses
and
if
they
use
arbitrary
pooling
it
means
that
you
as
a
user.
If
you
initiate
two
subsequent
connections,
your
public
IP
address
might
very
well
change
that
this
is
something,
of
course,
is
courage
in
the
in
several
RFC's,
and
we
were
surprised
that
we
see
it
to
that
large
extent,
because
just
think
of
what
this
lets
you
hosts
reputation
systems.
We
also
studied
in
detail
of
a
location.
E
Behavior
just
want
to
show
you
a
quick
sketch
here,
so
Nets
have
different
strategies
on
holiday,
allocate
oh,
it's
true
to
its
subscribers,
and
we
were
interested
in
studying
what
it
is.
Some
sort
of
a
common
uniform
behavior
you
within
a
specific
a
s,
and
it
turns
out
that
this
is
not
the
case
so
to
70
percent
of
the
ASA's
in
we
did
in
which
we
detected
carrier
grade
lab,
never
liked
a
mixed
port
allocation
strategy,
meaning
what
happens
to
subscriber
a
will
not
be
the
same
it.
What
happens
to
a
subscriber
B.
E
We
have
more
details
on
that,
of
course,
written
up,
but
generally,
when
studying
resource
allocation.
We
see
a
huge
diversity
here
and
we
also
see
that
actually,
the
majority
of
them
shows
like
non-uniform
behavior
and,
of
course,
just
think
of
what
this
can
do
to
your
applications.
To
most
reputation.
E
Of
course,
then
it
will
ran,
choose
port
numbers
from
this
shank
and
in
our
measurement
system,
nicely
closed
up,
because
for
each
metalizer
session
we
can
see
okay,
the
chunk
size
in
this
particular
ISP
is
like
five
hundred
twelve
ports
or
it
is
like
a
thousand
ports,
and
this
in
turn
allows
us,
then,
to
reason
how
many
subscribers
are
behind
a
sing.
An
IP
address
in
the
most
intense
case
that
we
found
here
in
our
measurements
is
512
ports
per
subscriber
or
128
subscribers
per
IP.
E
E
If
you
are
behind
a
symmetric
net,
that's
that's
it!
Basically,
and
in
the
case
of
CPU
devices,
we
see
that
this
is
quite
uncommon,
so
only
about
two
percent
of
them
do
that.
But
if
you
look
at
our
carrier-grade
Nets,
we
see
that
these
very
restrictive
nap
bindings
are
much
more
common.
In
fact,
it's
about
10%
of
the
non-cellular
deployments
and
even
40%
of
the
cellular
deployments
show
this
behavior,
meaning
that
the
mapping
that
a
carrier-grade
NAT
imposes
on
its
subscribers
is
often
more
is
more
restrictive
than
what
your
CPE
home
device
would
do.
E
Okay,
and
so
that
was
a
quick
overview
of
some
of
some
of
the
results.
What
we
can
generally
say,
we
see
that
carrier-grade
NAT
is
very
broadly
Paulette
and
it
is
like
a
everyday
reality
for
a
lot
of
users.
I
think.
In
fact,
it
is
even
reality
for
the
majority
of
Internet
users
I'm,
considering
that
most
of
them
are
in
cellular
networks
and
what
we
see
in
this
first
study.
You
see
a
stunning
variety
of
different
configurations.
E
Not
it
raises
some
challenges
so,
for
example,
one
regarding
regarding
measurement
and
I'd
be
very
curious
also
to
hear
your
opinion
on
that.
So,
for
example,
when
we
do
measurements
to
to
measure
end-user
performance,
it
should
have
performance.
I
mean
the
common
metrics
would
be.
Speed
are
less
and
latency
and
packet
loss.
Anything
with
these
measurement
with
these
metrics
are
that
they
don't
necessarily
capture
the
the
limitations
that
are
carry
a
great,
not
imposes
on
you,
so,
for
example,
to
capture
them
we
would
meet.
We
would
need
metrics
like,
for
example.
E
We
also
wanted
to
like
point
out
that
perhaps
give
given
how
widespread
carrier-grade
nets
are
deployed
and
given
their
measurable
effect
on
how
much
internet
you
receive
as
a
subscriber,
but
it
is
need
for
more
guidelines
or
possibly
like
best
practices
on
what
is
still
considered
an
okay
amount
of
internet
or
whether
it
is
also
something
that
could
be
interesting
or
subject
to
regulation
all
right.
What
I
conclude
and
I'm
happy
to
take
your
questions.
G
Judge
Nicola
our
one
comment
and
one
question:
when
you're
saying
about
20%
of
ISS
doing
arbitrary
marking
that
might
be
just
different
devices,
you
have
once
again
device
in
one
location
as
a
citizen
device.
Next
to
it,
you've
been
different,
has
different
pools
and
you
just
randomly
sheet2
an
awesome.
Yes,
yes
and
question:
did
you
compare
your
database,
detecting
v6,
only
networks
which
do
in
not
six
four,
because
it
looks
like
you're
connecting
to
before
destinations?
G
E
So
actually
most
of
that
is
like
currently
before
most
of
these
measurements.
The
thing
is
that
baturin,
the
public
DHT
of
the
torrent
is
not
it's
very,
sparsely
populated
in
ipv6.
So
currently
we
can't
really
compare
it
there
and
for
Naturalizer
we
do
have
a
v6
data
available,
but
we
didn't
include
it
yet
endless
because.
G
E
Just
won't
yeah
what
one
quick
thing
cuz
I
expect
it.
Of
course,
questions
regarding
ipv6
I
just
did
something
which
is
not
a
great
analysis
at
this
point,
but
just
did
it
yesterday,
so
I
took
the
a
SS
in
which
we
detect
carrier-grade,
NAT
and
I
just
checked
where
the
day
at
least
announced
an
ipv6
prefix,
and
here
we
see
it's
it's
about
half
of
them.
H
Hi
Alan
Jerome
first
to
follow
up
on
this
and
then
another
question.
There
are
two
cases
you
can
have
service
provider.
We
do
ipv6
and
ipv4
Calgary
not,
and
some
we
can
do
simply
ipv6
native
internally
in
that
six
four
yeah
was
that,
like
two
different
cases
that
you
may
want
to,
yes,
some
are
separate.
H
The
second
question
is
about
the
audible
addresses
that
you
found,
and
for
me
that
was
the
most
surprising
finding
in
your
study
and
in
very
very
interesting
because
you're
right
once
those
blocks
are
going
to
be
on
the
market
and
chopped
off
in
smaller
blogs.
When
we
may
see
some
interesting
behavior.
So
do
you
publish
a
list
of
actual
addresses
that
you
have
seen
something?
That's
a
finer
granularity
than
the
slash
eight,
because
I
look
in
your
paper?
Is
you
only
go
to
the
slash
eight?
H
E
H
I
E
E
E
E
J
Eric
not
Mike.
Thank
you.
This
very
interesting
study,
so
I
saw
one
thing.
You
had
two
things
you
had
this
sort
of
port
behaviors,
whether
it
was
you
know,
preserving
or
but
I
forget
the
term,
but,
and
you
also
had
the
restrictive
Mouse,
whether
it's
a
metric
or
something
else,
did
you
actually
look
at
potential
correlation
between
those
two
meaning
that
there's
sort
of?
If
you
group
them
together
that
things
fall
in
different
buckets?
No.
K
E
Saw
that
too
I
have
found
that
also
really
the
press.
If
that
this
can
happen,
I
currently
I
don't
see
a
way
of
how
to
how
to
measure
how
to
measure
that,
because,
in
the
end
like
for
one
session,
we
can
execute
these
ten
TCP
connections
and
we
can
run
the
stun
test
once
and
it's
not
that
the
same
user
will
be
nice
enough
to
run
metalizer
the
whole
day.
But
that
would
certainly
be
something
very
interesting
to
study.
A
How
else
to
make
it
I
have
a
question,
but
also,
if
there's
any
more,
we
have.
We
could
take
a
few
more
minutes
for
this,
so
this
is
one
of
the
nice
things
is
having
a
chance
to
discuss.
My
question
was
with
this
kind
of
behavior
that
you
observed.
Why
do
you
think
that
why
how
can
reputation,
systems
work
at
all?
Is
there
any
way
that
you
could
study
that
kind
of
impact
of
of
this
on
people's?
You
know
persistent
use
of
addresses
for
reputation.
L
E
Is
very
difficult
because
no
isp
writes
that
on
their
website,
or
at
least
very
very
few,
do
we
had
a
all
that
limited
set
off
like
a
couple
of
dozen
of
a
SS
for
which
we
have
like
ground
truth
on
whether
they
deploy
it
or
not.
We
did
not
find
false
positives
so
far
and
we
have
several
false
negatives.
I
mean
we
try
to
readjust
our
thresholds
to
be
as
conservative
as
possible
and
so
far
we
didn't
find
a
false
positive,
but
that
might
happen
at
some
point
sooner
or
later,
Thanks.
L
The
comment
was
just
I
think
you
made
a
comment
on
one
of
your
last
slides
about
regulation
and
I
thought
I
just
observed
that
I
was
just
looking
I
know.
There's
some
regulation
in
Belgium
related
to
this
now
I
was
looking
for
that,
and
the
first
thing
I
found
was
a
press
release
from
Europe
all
with
lots
of
text
in
it
about
how
alarm
how
alarmed
they
are
at
the
growing
use
of
cgn
and
the
impact
it's
going
to
have
on
their
on
their
law
enforcement
capabilities,
and
they
cite
your
paper
in
that
press
release.
L
So
I
thought
that
was
that
was
awesome
and
I've
also
heard
of
subscribers
of
ISPs
using
CG
and
having
problems.
Getting
two
major
pokemons
go
is
there's
another
thing
that
came
up
high
up
in
my
search
results
and
Xbox.
There's
there's
all
kinds
of
breakage.
This
is
causing
for
a
subscriber,
so
it
curves
to
me
that,
given
the
impending
regulation-
or
it
seems
likely,
regulation
and
the
effect
this
is
having
on
subscribers
continuing
to
measure
this
over
time.
You
might
start
to
see
this
stuff
goes
away
as
well.
M
Thanks
Philip
I,
you
and
I
have
talked
about
some
of
this
stuff
before
so
I'll
echo,
one
of
the
questions
that
I
have
for
you,
but
I
don't
expect
you
to
be
able
answer
it,
but
I'm
fascinated
by
Phelps
working
in
that
there
are
clearly
a
whole
bunch
of
kinds
of
commercial
devices
out
there
that
do
these
things
and
no
one
says:
hey:
that's
the
J
box
1,
that's
the
C
box,
1,
that's
the
Linux
one!
Somebody
must
know
so.
If
you
know
somebody
in
this
community
could
tell
us
about
what
they
are.
M
That
would
be
fascinating
to
know.
They
also
have
a
cost
in
the
capacity,
presumably
and
I,
don't
understand
why
this
market
place
isn't
described
in
the
open
right,
oh
and
they
just
have
a
four
color
glossy.
That
says
that
does
this
the
the
question
that
was
asked
before
about
reputation
and
it
cross,
for
instance,
the
world
wide
web
client
space,
not
speaking
for
my
employer,
because
I've
just
not
authoritative
about
what
exactly
we
do
in
our
reputation
product,
but
I
can
tell
you
some
of
the
things
that
you
can
do.
M
There
are
pieces
of
software
that
have
unique
identifiers
in
them
and
you
very
often
see
tens
or
hundreds
or
even
thousands
of
those
unique
identifiers
show
up
behind
one
IP
address
in
a
day
or
a
couple
of
days.
So
clearly
that's
an
added
address.
The
other
thing,
one
of
the
most
interesting
ones
that
were
pursuing
in
our
nascent
research
is
about
finding
v4
v6
address
pairings,
sort
of
the
way
that
a
APNIC
does
where
you
you
force
a
web
browser
to
connect
back
with
both
protocol
versions
and
and
over
a
very
short
amount
of
time.
M
You'll
see
1/24
associated
with
tens
or
hundreds
of
thousands
of
slash,
64's
and
v6.
So
it's
a
it's
an
interesting
way
and
that
the
unit
is
temporarily
unique
identifiers
in
terms
of
the
identities
of
sign
in
v6
multiplex
on
to
v4
addresses,
and
you
can
find
the
the
nets
that
way
and
then
the
last
thing,
following
to
what
Ellison's
I
mean,
she
said
like
I,
think
that
just
it
was
how
in
the
world
would
you
do
this?
It's
a
huge
problem.
M
It's
a
huge
like
record-keeping
problem,
because
you
have
to
continue
continually
monitor
this
and
the
other
place
we're
seeing
it
is
people
adding
through
cloud
infrastructure.
So
like
the
cloud
right,
they
have
a
nap.
That's
infinitely.
Scalable
and
people
are
putting
commercial
enterprises
behind
a
cloud,
and
so
we
see
a
cloud
address
one
day
and
it's
one
machine
doing
some
service,
but
the
next
day
it's
a
company
behind
it
with
a
thousand
employees.
It's
a
really
difficult
problem
and
I'm
glad.
That's
not
my
job.
A
N
Great
thanks.
So,
as
Alison
said,
my
name
is
Steve
chuckwei
I'm
at
the
University
of
Illinois
at
Chicago
and
I
want
to
talk
today
about
the
incident
involving
junipers
use
of
Julie
C.
So,
if
you
don't
have
any
idea
what
I'm
talking
about
I'll
jump,
I
did
so.
In
late,
2015
juniper
made
a
pretty
surprising
announcement.
They
had
an
out
of
cycle
Security
Bulletin
that
said
that
some
unauthorized
parties
had
gained
access
to
their
source
code
and
had
managed
to
introduce
two
backdoors.
N
So
the
first
was
an
unauthorized,
remote
administrative
access
and
the
second
was
the
somewhat
cryptic
message
that
a
knowledgeable
attacker
who
can
monitor
VPN
traffic
can
decrypt
that
traffic.
So
this
is
what
I
want
to
talk
about
it.
This
work
is
all
about.
How
did
this
actually
happen?
What
was
it
that
the
attackers
were
able
to
do
what
the
consequences
of
that
are,
and
then
I
want
to
end
with
a
couple
of
lessons
that
I
think
that
that
we've
learned
from
this
that
are
applicable
to
protocol
designers.
N
So
first,
these
are
the
devices
that
were
affected.
These
are
the
junipers,
secure
service,
gateway,
combination,
firewall
and
VPN
devices,
and
the
affected
versions
of
the
software
were
a
variety
of
screen,
OS,
6.2
and
6.3
versions.
So
let
me
start
with
the
first
one.
This
was
actually
from
my
point
of
view,
the
least
interesting
one,
although
it
was
a
little
bit
clever
here,
so
basically
what
the
attacker
did
was.
N
It
kind
of
looks,
like
some
other
format,
strings
that
you
would
be
using
for
logging
information,
and
it
turns
out
that
you
can
see
this
is
being
passed
right
here
to
store,
compare,
and
so,
if
you,
if
you
log
in
with
this
this
password
it
it
lets
you
log
in
that
with
administrator
access,
and
it
leaves
some
some
tell-tale
sign
and
the
log
which
one
could
could
actually
remove.
So
this
was
found
really
quickly
by
HD
more
and
was
able
to
test
it
and
and
sure
enough
it
worked.
N
So
the
next
one
was
more
interesting
from
my
point
of
view
because
it
was
really
cryptic.
It
just
said:
there's
this
knowledgeable
attacker
who
could
do
something,
and
so
once
this
security
bulletin
was
put
out,
researchers
around
the
world,
including
me
and
my
colleagues
decided.
We
were
not
going
to
spend
Christmas
with
our
families
and
instead
we
were
going
to
spend
a
bunch
of
time
reverse-engineering
stuff,
and
the
question
is
well:
where
do
we
start
with
this?
Well,
HD
more
again
was
very,
very
helpful
here.
N
He
he
ran
strings
over
one
of
the
one
of
the
the
versions
of
the
firmware
that
had
been
known
to
have
been
changed
and
also
ran
strings
over
junipers
fix
that
they
put
out
and
ran
diff
over
that,
and
this
is
almost
entirely
the
full
death
there's
a
few
other
things.
But
this
is
the
the
key
point
here,
and
so
this
is
kind
of
interesting.
If
you
are
a
big
crypto
nerd,
you
might
have
looked
at
this
and
said:
oh
I
know
what
these
are
immediately.
N
If
you're,
like
you
know
most
people,
you
put
them
into
Google
and
see
what
comes
up
and
the
first
five
of
these
long
hex
digits
long
hex
numbers
here
are
parameters
to
this
elliptic
curve,
P
256.
So
these
you,
you,
google
them
and
they
they
come
right
up.
And
so
the
question
is:
what
does
this
change
here?
What
what
change
from
9,
5,
8,
5,
etc,
to
2,
5
or
2,
C,
5,
5,
etc,
and
to
figure
that
out,
we
had
to
reverse
engineer
the
the
binaries,
and
it
turns
out
that
these
two
values
are.
N
Rather
this
one
value
that
was
changed
were
for
the
x-coordinate
of
a
non-standard
elliptic
curve
point
which
was
used
for
dual
EC
drbg.
If
you
haven't
heard
of
dual
EC
drbg,
it
was
a
or
it
is
a
pseudo-random
number
generator
that
was
designed
by
the
NSA
in
the
early
2000s,
and
then
they
made
a
push
towards
standardization,
so
it
appeared
as
part
of
an
ANSI
standard
I
think
it
appeared
in
some
other
standards,
eventually,
NIST
standardized
it
in
special
publication,
800-53.
N
C
their
default
random
number
generator.
It
doesn't
actually
do
anything
else
in
this,
for
this
particular
work.
But
it's
an
interesting
thing
to
note
here,
but
the
key
point
sorry
I
couldn't
resist.
The
the
key
point
here
is
that
in
2007's,
Microsoft
researchers,
sumo
and
Ferguson
were
able
to
demonstrate
at
a
crypto
Rumph
session,
a
theoretical
backdoor
attack
against
dual
EC
and
sort
of
after
this
people
thought
well,
okay.
We
know
this
is
bad.
It
has
some
biases
that
the
we
we
probably
shouldn't,
use
it
and
it
turns
out
it's
really
slow.
N
N
Some
of
my
colleagues
and
I
showed
that
in
fact
that
if
you
use
dual
EC
as
your
random
number
generator
for
TLS,
then
you
can
actually
break
TLS
so
fast
forward
to
you
know
2015,
and
we
see
that
once
again,
we
can
actually
do
this
with
with
I.
So
let
me
talk
about
what
it
means
to
have
a
backdoor
to
random
number
generator,
because
that's
kind
of
an
unusual
concept,
but
basically
the
way
a
normal
random
number
generator
works.
Is
you
start
with
a
seed?
N
N
Now,
Dulli
see
the
the
EC
stands
for
elliptic
curve,
and
so
just
do
a
quick
primer
on
elliptic
curve.
Math,
don't
need
to
understand
most
of
how
this
works,
although
it
is
a
pretty
neat
ear,
pretty
neat
area.
So,
basically,
a
point
on
an
elliptic
curve
is
a
pair
of
x
and
y
coordinates
that
satisfy
a
particular
equation.
It's
no
matter
what
the
equation
is
here:
the
for
our
purposes,
x
and
y
are
going
to
be
32
byte
integers,
although
there
are
larger
and
smaller
elliptic
curves,
that
one
could
deal
with
and
sort
of.
N
The
the
key
properties
here
are
that
you
can
add
two
points
on
the
curve
in
this
sort
of
geometric
fashion
and
arrive
at
another
point
on
the
curve,
and
you
can
repeat
this
process
over
and
over,
so
you
can
take
an
integer
in
and
a
point
P
and
you
can
compute
n
times
P,
which
is
just
adding
P
to
itself
at
times.
And
finally,
the
key
to
this
whole
scheme
is
that
given
a
point
P
and
a
point
in
P,
it's
hard
to
come
up
with
n.
So
this
is
the
elliptic
curve,
discrete
logarithm
problem.
N
So
let
me
describe
dual
EC
here,
so
this
is
sort
of
a
somewhat
simplified
version
of
how
this
works.
But
we
start
with
a
32
byte
state
s
naught
and
we
take
s
naught.
We
treat
it
as
a
big
integer.
We
multiply
it
by
some
fixed
elliptic
curve,
point
P,
which
is
the
generator
for
P
2
5
6,
and
then
we
take
the
x-coordinate
of
that,
and
that
gives
us
a
new
state
s.
1.
We
take
s
1.
N
We
multiply
that
by
a
second
fixed
elliptic
point,
Q
take
the
x-coordinate,
and
that
gives
us
R
1,
which
is
not
quite
our
output.
It's
almost
our
output.
So
what
we
do
is
we
take
the
least
significant
30
bytes
of
R
1
and
those
become
the
most
significant
30
bytes
of
our
output,
and
then
this
process
repeats
so
take
s
1
times
P,
you
take
the
x-coordinate
and
it
gives
us
s.
2
multiply
that
by
Q
take
the
x-coordinate
that
gives
us
R
2.
N
The
next
30
bytes
are
the
least
significant
30
bytes
of
that
are
the
next
output,
bytes
and
so
on.
So
the
really
cool
thing
that
Schumer
and
Ferguson
noted
about
this
is
that
if
an
attacker
knows
the
relationship
between
P
and
Q
in
particular,
if
the
attacker
knows
the
integer
D
such
that
P
is
equal
to
D
times
Q
and
it
turns
out
that
such
a
d--
exists
and
the
attacker
is
able
to
see
output
from
dual
EC.
Then
it's
actually
possible
to
recover
some
internal
state,
and
so
the
way
this
works
is
pretty
neat.
N
You
start
with
the
output
and
you
set
r1,
which
you
don't
know,
except
you
know
the
first
30
bytes
of
it
just
because
of
how
dual
EC
works.
Then
you
need
to
guess
the
two
most
significant
bytes
of
r1
or
in
reality,
what
you
do
is
you
just
try
all
65,000
possibilities,
so
that
doesn't
take
very
long
and
you
find
an
elliptic
curve
point
R
that
has
all
one
as
its
x-coordinate.
N
N
N
So
so
you
take
the
x-coordinate,
and
that
gives
us
2
and
if
it
was
not
immediately
obvious
how
that
works,
you
can
read
this
string
of
equations
here
or
you
can
just
take
my
word
for
it
all
right,
but
most
of
the
time
you're
you're
not
going
to
get
a
value
that
works.
So
how
do
you
figure
out
when
you've
actually
recovered
the
internal
state?
Well,
it's
pretty
simple.
You
just
walk
dual
EC
forward,
so
you
multiply
your
assumed
s2
by
Q.
N
Well,
obviously,
the
attacker
needs
to
have
this
secret
value
D,
but
the
attacker
also
needs
to
see
a
large
amount
of
output
from
a
single
block
of
Abdullah,
see
output,
so
say
at
least
28
bytes
and
the
fewer
bytes
you
have.
The
attack
becomes
exponentially
more
difficult,
so
if
you
have
30
bytes
or
more,
it's
it's
basically
free
to
do.
If
you
have
less
than
28
bytes,
you
actually
have
to
spend
a
lot
of
time
working
on
and
next
you
don't
actually
need
to
see
raw
output
to
do
the
comparison
step.
N
It's
sufficient
if
there
is
some
public
function
of
that.
So
let
me
give
you
an
example.
Imagine
a
network
protocol
that
has
a
28
or
larger
byte
nonce
that
gets
sent
in
the
clear
and
it
has
a
difficulty
that
also
gets
sent
in
the
clear.
So
here
we
don't
know
what
the
the
private
key
for
the
diffie-hellman
public
he
is,
but
that's,
okay.
We
can
use
the
Shuma
Fergusson
attack
on
the
nonce
to
recover
some
proposed
internal
state
of
the
generator.
N
Then
you
compute
what
X
would
be
just
by
following,
however,
X
would
be
normally
generated
in
this
protocol
and
then
you
compute
G
to
the
X
and
see
did
that
match
the
the
proposed
sorry
did
that
match
the
actual
public
he
or
not,
and
if
it
did,
then
you
know
you've
got
X
and
you
have
basically
broken
the
protocol.
So
you
might
wonder
how
can
one
learn?
D
and
I've
got
4
methods
that
one
could
use
to
learn
D
here?
The
first
is
you
could
solve
the
discrete
logarithm
problem?
N
The
second
is,
you
could
be
the
entity
in
charge
of
picking
the
official
Q
value
in
dual
EC,
and
so
either
way
you
do
this.
Is
you
just
pick
a
large
integer
e?
You
multiply
e
times
P,
you
set
that
to
be
Q,
and
then
D
is
just
the
inverse
of
e
modulo.
The
group
order,
so
some
simple,
math
and
you've
got
D
1/3.
You
can
use
a
non-standard
point.
N
Q,
you
don't
have
to
use
the
the
nsa's
Q
value
and
just
do
the
same
thing
as
in
step
2
or
you
can
gain
access
to
somebody
else's
source
code
and
substitute
your
own
Q
point,
just
as
you
would
have
done
in
step
2
and
it's
worth
pointing
out
here
that
solving
a
discrete
logarithm
problem
is
too
hard.
We
just
can't
do
that
picking.
The
official
point
was
done
by
the
NSA,
but
they're
not
saying
how
they
picked
it.
So
so
who
knows.
N
Screen
OS
that
was
used
in
these,
these
juniper
boxes
actually
use
a
non-standard
point.
Q
and
finally,
the
this
incident
that
I'm
talking
about
came
about
because
the
attackers
replaced
their
the
Juniper
skew
with
their
own
cue,
and
so
now
we
can
sort
of
answer
the
question:
what
was
it
that
the
knowledgeable
knowledgeable
attacker
knew?
N
Well,
the
attacker
knew
the
discrete
log
D,
so
this
was
actually
pretty
surprising
because
there's
a
2013
knowledgebase
article
that
says
that
although
some
of
these
screen,
OS
versions
use
dual
EC
it
it
doesn't
matter
because
dual
EC
is
not
the
primary
PRNG
and
in
fact
it
takes
the
output
of
dual
EC
feeds
that
through
an
ANSI
PRNG
X
931,
and
that's
what
gets
used
as
the
main
output.
And
so
in
light
of
this,
we
had
my
my
college
and
I
had
five
research
questions
that
we
really
wanted
to
answer.
N
So
why
doesn't
the
use
of
X
931
defend
against
some
compromise
Q
value?
Why
does
a
change
in
Q
actually
result
in
VPN
decryption,
because
this
is
just
a
random
number
generator?
Perhaps
other
things
should
be
able
to
defend
against
this?
What
is
the
history
of
the
screen,
OS
PRNG
code,
where
two
dual
ec
come
in
and
and
how
did
that
lead
to
being
able
to
decrypt
traffic
and
then
are
their
versions
of
screen
OS
that
are
using
juniper,
skew
that
are
vulnerable
to
the
same
passive
decryption?
N
N
To
sum
greater
or
lesser
extent,
in
order
to
answer
these
particular
questions,
so
we
can
answer
the
first
three
questions
letting
for
if
we
want
to
figure
out
the
answer
to
the
last
two
we're
going
to
need
some
other
body
of
material
that
we
just
don't
have
access
to.
Although
presumably
juniper
knows
the
answer
to
five,
which,
as
we'll
see,
will
answer
for
so,
let's
start
with
a
screen
os
6.2
s,
pseudo-random
number
generator-
and
let's
start
by
looking
at
the
the
PRNG
function
over
here
on
the
right
and
so
keep
in
mind.
N
This
was
decompiled
code.
So
any
of
the
names
that
you
see
here
are
names
that
I
came
up
with
not
names
that
appeared
in
the
firmware
there,
probably
very
different
in
junipers
actual
source
code,
and
so
this
function
has
a
couple
of
key
features
here.
The
first
thing
is:
you'll
notice
that
there
is
a
conditional
reseed,
so
it
seems
pretty
standard.
Some
condition
is
met.
The
receipt
happens.
Second,
we
can
see
that
it's
always
generating
32
bytes
of
output
at
a
time
using
the
ansi
x9
31
PRNG.
N
N
Sorry,
the
remaining
24
bytes
become
the
X
931
key.
So
at
first
glance
this
looks
like
exactly
what
Juniper
described.
We
have
dual
EC
here,
but
its
output
is
being
fed
into
X
9
to
31,
so
there
shouldn't
be
any
problems
with
this
particular
use.
But
let's
take
a
closer
look
here
because
it
turns
out.
That's
not
true.
So
the
first
thing
that
happens
over
in
this
PRNG
function
or
a
PNG
generate
function
here
is
that
we're
setting
this
index
global
variable
here
to
0.
N
Then
we
call
this
one
stage:
RNG
function,
which
it
turns
out
always
returns
false,
which
means
that
happens
every
single
time.
This
generator
is
called.
So
then,
if
we
look
over
at
the
the
receipt
function,
it's
generating
32
bytes
from
Julie
see
the
index.
Global
variable
is
set
to
32
for
some
reason
and
then
back
in
the
main
generate
function.
This
loop
never
execute
because
index
is
32,
so
the
loop
guard
is
always
false
and
thus,
at
the
end
of
the
function,
what
we
have
is
32
bytes
from
dual
EC
left
in
the
output
array.
N
This
global
output
array,
which
is
the
output
of
this
particular
function,
so
what's
going
on
here?
Well,
we've
got
these
global
output
and
index
variables
and
the
they're
being
reused
for
different
purposes,
so
the
output
variable
is
being
used
both
as
a
temporary
32,
byte
buffer
for
the
receipt
function,
and
it's
also
being
used
as
the
main
output
for
the
PRNG
and
this
index
variable
is
global
for
some
reason
that
doesn't
make
any
sense
and
will
actually
come
back
to
this
later
when
we
look
at
the
history
of
this
like.
N
Why
is
this
global
and
I
should
note
that
the
the
a
bunch
of
people
were
looking
at
this
code
and
we're
saying
well
I,
don't
understand
how
anything
could
go
wrong
with
this,
including
me
until
Willem
Pinker's
on
Twitter
pointed
out
that
that
when
the
receipt
happens,
the
the
ansi
generator
is,
in
fact
not
being
around,
so
he
was
the
one
who
figured
out
that
the
key
to
this
whole
thing,
and
so
from
this
we
can
answer
our
first
research
question.
Why
doesn't
the
use
of
X
9.31
defend
against
a
compromised,
Q?
N
N
So,
let's,
if
we,
if
we
take
a
look
at
at
one
of
the
ike
phase,
1
packets,
that
gets
transferred,
we
can
see
that
there's
a
header
and
a
series
of
payloads
that
follow
it.
So
there's
a
payload
that
says
which
crypto
algorithms
are
we
going
to
use,
there's
a
key
exchange
payload,
which
contains
a
diffie-hellman
public,
key
G
to
the
X
and
there's
a
nonce
which
the
standard
says
should
be
between
8
and
128
bytes
long
and
in
screen
OS.
N
The
private
key
is
this
20-byte
value
X,
which
is
generated
by
dual
EC,
and
the
nonce
is
always
a
32
byte
value,
which
is
also
generated
by
dual
EC.
Now
in
an
ideal
world
and
ideal
for
the
attacker
world,
I
should
say
the
way
the
attacker
would
attack.
This
is
by
applying
the
the
sumo
Ferguson
attack,
basically
directly.
So
if
the
nonce
is
generated
before
the
private
key,
then
the
it
looks
something
like
this.
So
we
have
to
run
dual
EC
a
few
times.
N
N
Unfortunately,
what
appears
to
happen?
If
you
look
at
the
code-
and
you
look
at
the
protocol-
is
that
the
nonce
appears
to
be
generated
after
the
diffie-hellman
private
key
is
generated,
and
so
the
sumo
Ferguson
attack
doesn't
actually
recover
X
right.
So
you
could
you
run
your
attack
over
here
on
r2,
you
get
s3,
but
you've
missed
on
X,
because
X
has
already
been
generated
and
you
can't
try
to
run
your
attack
on
X,
because
if
you
had
accident
there
wouldn't
be
anything
to
do
so
so
that
doesn't
work
so
initially.
N
I
was
a
little
disheartened
when
I
saw
this
because
it
seemed
like
you
would
have
to
deal
with
multiple
connections,
but
it
turns
out
that
the
reality
is
somewhere
between
the
ideal
case
and
the
apparent
case
screen.
Os
contains
cues
of
pre-generated,
Nantes
and
diffie-hellman
values
and
the
reason
it
does.
This
presumably
is
to
speed
up
VPN
connections.
These
boxes
are
not
super
powerful
and
you
know
computing
G
to
the
X
could
be
an
expensive
operation,
and
so
critically
the
the
nonce
q
is
always
filled
before
any
of
the
other
queues.
N
So
there's
a
timer
that
fires
and
once
a
second
it
will
say
I'm
going
to
generate
one
value
and
either
I'm
generated
nonce
or
I'll
generate
it.
If
you
Hellman
public
key
and
it
always
has
the
nonce
is
first
and
the
consequence
of
that
is
that
in
many
cases
the
ideal
attack
really
just
works.
The
nonce
was
generated
before
the
private
key,
and
so
you
look
at
one
transfer.
You
look
at
a
transcript
of
a
single
VPN
connection
and
what
do
you
see?
N
One
interesting
thing
about
ike
is
that
at
least
for
Ike
v1,
the
authentication
mode
that
is
chosen
and
for
I
could
be
one.
At
least
there
are
four
authentication
modes
actually
has
an
impact
on
the
the
traffic
keys
on
the
the
encryption
keys,
not
merely
on
the
authentication,
and
so,
if
you
have
authentication
via
digital
signatures,
then
the
attack
works
exactly
as
described.
There's
nothing
more
that
you
have
to
do.
If
you're
using
Ike
be
one
with
pre-shared
keys,
then
it
turns
out
that
the
pre-shared
key
is
actually
mixed
into
the
traffic
keys.
N
So
the
attacker
needs
to
know
the
pre-shared
key
in
order
for
this
attack
to
work
and
for
the
two
modes
involving
public
key
encryption.
Well,
the
nonces
are
encrypted
and
there's
just
no
way
to
run
the
attack
and
it
doesn't
work
at
all
and,
interestingly,
in
Ike
v2
the
authentication
modes
don't
affect
the
traffic
keys
at
all.
So
the
attack
just
works
and
that's
actually
that's
that's
kind
of
sad.
We
would
like
later
versions
of
a
protocol
to
be
more
secure
against
attack
than
earlier
ones,
but
in
this
case
that's
not
snatcher.
N
So
what
about
phase
two
well
phase?
Two,
some
more
nonces
are
exchanged
optionally,
another
diffie-hellman
exchange
happens
and
to
attack
this.
It's
actually
pretty
easy.
You
can
either
just
rerun
the
Schumer
Ferguson
attack
or
even
easier
is
you
could
run
the
dual
EC
generator
forward
to
just
get
the
the
new
values
you
care
about,
and
the
proof
of
concept
that
I'm
about
to
describe
here.
We
actually
just
ran
the
Schumer
Ferguson
attack
again
because
it
was
fast
enough
and
it
it
meant
we
didn't
have
to
bother
dealing
with
with
values
that
were
sort
of
generated.
N
Far
apart,
so
to
test
that
this
actually
works.
We
bought
a
net
screen,
SSG
550
m,
which
you
can
see
over
on
the
right,
and
we
replaced
the
Q
value
in
the
firmware
exactly
as
the
attackers
did
with
a
Q
that
I
generated
so
I
knew
the
discrete
log
D
and
then
we
set
it
up
to
do
to
have
VPN
connections
with
a
variety
of
different
configurations
and
sure
enough.
The
attacks
worked
as
described.
I
could
be
one
with
pre-shared
keys
required
the
pre-shared
keys
using
an
RSA
cert.
N
It
just
worked
fine
and
for
Ike
v2.
The
attack
just
worked
fine,
so
that
gives
us
the
answer
to
our
second
research
question:
why
does
a
change
in
Q
result
in
passive
VPN
decryption?
Well,
the
dual
EC
output
is
right
there
on
the
wire.
We
just
run
the
Shema
Ferguson
attack
and
at
least
for
some
VPN
configurations,
that's
sufficient.
N
Alright,
so
we
figured
out
what
the
attacker
did
and
we
figured
out
why
it
works.
But
now
we
have
this
question
of
of.
Why
does
screen
OS
do
this?
Is
this
something
that
was
there
from
the
beginning?
Probably
not,
and
in
fact
one
of
the
things
that
I
did
was
I
reverse-engineered
a
whole
bunch
of
other
versions
of
the
firmware
you
saw
on
the
table
and
we
can
see
that
there
are
a
number
of
differences
between
screen,
OS,
6.1
and
6.2
in
particular,
6.2
added
dual
ec
in
this
cascade
with
ansi
x9
31.
N
It
moved
from
reseeding
on
every
10,000
calls
to
the
generator
to
reseeding
on
every
call.
There
was
this
bug
with
the
index
variable
that
was
now
global,
so
that
it
it
exposed
the
dual
EC
output
even
more
interesting
than
Mike
Knotts
has
changed
from
20
bytes
to
32
bytes
and
6.2
added
a
nonce,
pre
generation
cue,
and
so
this
the
set
of
changes
actually
raises
a
number
of
questions.
Why?
Why
were
these
changes
made?
N
Standardization
point
of
view,
so
why
was
dual
EC
at
it?
I
don't
know.
This
is
a
this
was
kind
of
a
weird
thing,
I
can't
think
of
any
engineering
reason
that
would
require
this
and
in
fact,
in
6.2.
As
far
as
I
remember,
the
only
thing
that
the
elliptic
curve
math
was
doing
was
doing
dual
EC,
so
they
had
to
add
a
bunch
of
custom
code
to
their
OpenSSL
version
in
order
to
use
dual
EC.
So
it
seems
like
a
weird
trade-off
to
me
and
I
can't
think
of
any
standardization
reason
for
doing
this.
N
A
screen
OS
was
had
was
Phipps
certified
4x
931
already,
so
they
didn't
need
that
and
in
they
never
got
it
certified
for
dual
EC,
so
I'm
not
really
sure.
Next.
Why
does
screen
OS
6.2
reseed
on
every
call,
so
on
the
left?
You
can
see
6.1
and
on
the
right.
We
have
6.2
here
and
you'll
notice
that
this
reseed
counters
global
receipt
counter
was
retained
in
6.2,
but
it
doesn't
do
anything
it
gets
incremented
and
then,
when
X
931
reseed
gets
called,
it
gets
set
backs
to
0.
N
So
so
the
receipt
value
doesn't
do
anything
anymore.
So
why
does
this
happen?
Well,
I
can't
really
think
of
an
engineering
reason
to
reseed
every
time.
Maybe
there's
a
cryptographic
reason.
Maybe
they
were
trying
to
add
backtracking
resistance
to
X
9.31
I,
don't
know,
maybe
a
cryptographer
could
could
enlighten
me,
could
just
be
a
buck.
N
Next,
there's
the
receipt
bug.
So
in
6.1
we
can
see
that
output
was
initially
a
parameter
to
the
PRNG
generate
function
and
index
was
a
local
variable.
Then,
when
they
moved
26.2
both
of
these
become
global
variables.
For
some
reason,
I
I
don't
know
why
I
can't
think
of
any
good
engineering
reason
for
doing
this.
I
can
think
of
kind
of
a
bad
one,
but
it's
not
we're
talking
about.
N
Next,
we
have
the
ike,
not
size
increase,
so
6.2,
as
I
mentioned,
increase
the
nonce
size
from
20
bytes
to
32
bytes,
again,
I
can't
think
of
an
engineering
reason
to
require
this
I
can't
think
of
a
good
graphic
reason.
Although
I
will
note
that
the
Department
of
Defense
had
some
bizarre
to
me
line
about
how
you
wanted
the
public
randomness
to
be
at
least
so
big,
and
so
they
wanted
to
do
this
and
TLS.
N
Maybe
this
is
the
reason
here,
I'm
not
sure,
but
I
will
note
that
at
20
bytes
the
sumo
Fergusson
attack
would
take
about
2
to
the
96
scalar
multiplications,
which
is
completely
infeasible,
would
never
be
able
to
do
it,
we're
at
32
bytes.
It
takes
two
to
the
16
and
finally,
there
was
this
nonce
pre
generation
Q
that
was
added
to
the
diffie-hellman
pre
generation
Q's.
So
in
6.1
there
was
the
the
diffie-hellman
Jenner
pre
generation
Q
by
itself,
and
this
is
actually
reasonable.
N
N
Let's
just
generate
this
in
advance,
so
I
don't
know
that
at
our
adding
a
nonce
generation
Q
to
handle
the
fact
that
you're,
PR
and
G
is
slow,
is
the
right
move
to
make,
but
it
at
least
makes
sense
from
an
engineering
point
of
view.
And
so,
if
we
look
at
the
changes,
we
can
see
that
the
first
four
of
them
were
actually
required
for
VPN
for
passive
VPN
decryption.
N
If
you
didn't
have
these,
then
the
attack
just
doesn't
work
at
all,
and
the
last
change
enables
single
connection
attacks,
whereas
if
you
didn't
have
that
you
would
have
to
do
this
multi
connection
attack
all
right.
So
that's
the
the
history
of
the
VPN
and
that
sort
of
leaves
us
with
the
final
two
questions.
Are
there
versions
of
screen
or
less
that
are
vulnerable?
Well,
maybe
it
depends
on
how
Q
was
generated
if
Q
was
generated
in
a
safe
way
and
I
can
describe
one
way
of
doing
that.
N
Then
then
no,
the
answer
is
it's
not
vulnerable.
If
Q
is
generated
such
that
somebody
knows
this
integer
D?
Well,
then
yes,
it's
vulnerable
to
whoever
knows
D,
and
so
that
leaves
us
with
the
final
question
of
how
was
junipers
q
generated
and
I
just
can't
say:
I
I,
don't
I,
don't
know,
and
it's
impossible
to
tell
with
the
data
that
we
have
and
so
I
want
to
end
with,
with
a
couple
of
lessons
for
protocol
designers.
That
I
think
we've
learned
from
this
so
first
and
this
is
sort
of
banal.
N
Prng
is
extremely
important
for
cryptographic
protocols,
and
so
you
should
pay
a
lot
of
attention
to
what's
going
on
with
your
PRNG.
So
one
thing
you
can
consider
doing
is
hashing
the
output
of
it
before
you
put
it
on
the
wire.
If
you
have
a
good
PR
ng,
this
is
a
pointless
step.
If
you
have
a
bad
one,
maybe
this
will
help
you
I,
don't
know
it
would
have
in
this
case,
perhaps
more
important.
N
If
you
need
them
to
be
unpredictable,
then
maybe
this
doesn't
help
you
so
much
next
is
we
should
think
carefully
about
the
nonces
that
we're
using
in
our
protocols,
for
example,
we
should
not
allow
nonces
that
are
too
large
or
that
are
variable
length.
Anything
that's
too
large,
for
example,
Ike's
allowing
up
to
256
bytes
is,
is
unreasonable.
It's
just
too
long,
you
don't
need
that,
and
it
just
invites
implementations
to
expose
secrets,
either
intentionally
or
unintentionally,
by
using
bad
random
number
generators,
and
it's
worth
pointing
out
that
variable
length.
N
Implementations,
allow
you
to
fingerprint
implementations,
so
perhaps
don't
allow
that
one
thing
that
was
really
interesting
to
me
from
the
from
comparing
Ike,
v1
and
Ike
fee,
to,
as
I
mentioned,
that
I
QV
2
was
less
secured
against
a
bad
random
number
generator,
and
so
maybe
we
should
consider
whenever
we
have
some
low
entropy,
even
low
entropy
secrets,
we
should
consider
hashing
them
into
our
traffic
and
authentication
keys
and
finally,
I
just
want
to
end
with
talking
about
this,
this
concept
of
no
bus.
So
the
NSA
has
this
notion
of
nobody,
but
us.
O
Hi
Steve
Kenny
Patterson.
Let
me
just
start
by
saying
I
think
this
is
really
amazing
work
and
you
probably
didn't
tell
us
exactly
how
many
hours
you
spent
in
the
lab,
ruining
your
Christmas
to
do
all
the
reverse
engineering
and
congratulations.
That's
really
amazing.
Thank
you.
I
have
I,
also
have
a
question
and
what
public
statements
to
Juniper
make
about
the
penetration
of
their
software
development
entity
or
section.
N
That
led
to
this
right.
So
what
public
statements
did
they
make?
I
I
think
they
they
had
the
the
out
of
cycle
advisory
that
started
all
of
the
this
work,
and
then
they
eventually
decided
to
remove
dual
EC
from
their
products
altogether.
I
think
they
made
an
announcement
about
that
I'm,
not
sure
if
there's
been
any
other
public
statements,
so.
O
N
P
Stephen
Feil
yeah
I
totally
agree
to
ask,
but
I
thought
you
might
your
honor
I
think
this
is
a
fine
lesson,
a
demonstration
of
the
detail
of
exactly
the
crap
consequences.
If
we
did
go
so
stupid,
I
hope,
I,
hope
everybody
is
listening.
I'm
not
just
me
what
question
yeah
he
said
on
your
previous
slide.
You
kind
of
recommends.
P
Yes,
before
putting
on
the
wire,
so
there's
two
people
who
are
interested
in
the
quic
protocol
or
probably
on
here,
they're
meeting
that
but
they're
having
like
fun
debates
about
in
a
new
transfer
protocol
what
to
leave
in
the
public
header
that
might
be
visible
on
the
wire
versus
what
to
put
in
ciphertext,
including
things
like
packet
numbers,
which
may
not
be
sequential
and
may
hence
be
efforts
around
us
yeah.
So
I
guess,
that's
all
a
question
if
you
thought
about.
P
N
A
good
question
I'm
actually
not
sure
all
of
the
places
that
one
might
be
exposing
randomness
and
the
obvious
ones
are
nonsense
that
we
that
we
send
but
I
hadn't
thought
about
sequence,
numbers
I,
I,
guess,
I
haven't
looked
at
quick
in
quite
a
while,
but
I
would
have
assumed
a
sequence.
Number
was
sequential,
but
you're
saying
that's
not
true,
so
I
don't
know
so.
P
I
haven't
been
involved
with
details
quick,
but
there
was
one
male
earlier
today
on.
The
list
is
suggesting
that
you
could
randomly
pick
higher
bits
and
then
just
have
the
lower
bits
differential,
which
may
give
some
better
properties
in
some
way,
but
I
guess
just.
We
need
to
think
about
being
careful
about
that
kind
of
thing.
Yeah.
Q
Feel
how
big
yeah
I'm
just
amazed
that
anybody
would
want
to
do
this
in
the
first
place,
because
if
you
go
back
to
your
original
slide,
when
you
doing
the
Dooly
okay,
it's
not
like
they're
using
the
curve,
255
1
9
curve
or
whatever,
where
yeah
it's
pretty
close
to
the
power
of
2
I
mean
these.
Are
you
know
the
NIST,
curves
and
so
on
and
so
they're
gonna
wrap?
Is
it
even
an
embarrassed,
random
number
generator?
No.
N
Absolutely
not,
it
is
biased
and
one
of
the
reasons
that
that
it
only
uses
30
bytes
of
the
32
byte
output
is
precisely
to
deal
with
that.
There's
some
analysis
and
SP
800
90
about
about
why
that
is
a
that
was
the
right
trade-off
to
use
and
I,
admit.
I
didn't
totally
follow
why
that
was
the
case,
but
but
there
is,
there
is
a
bias.
N
Q
N
Free
sure,
yeah
I,
don't
know
why
anybody
would
use
Dooley,
see
it
in
that
nist
special
publication.
There
are
three
other
pseudo-random
number
generators,
one
of
which
was
designed
by
the
NSA,
the
other
two
by
I
think
john
kelsey,
at
nist
and
or
others
at
nist,
and
all
of
those
are
much
much
faster
than
dual
ec
there's.
There's
no
reason
to
to
use
it.
A
I'm
gonna
ask
you:
so:
are
there
any
folks
who
are
in
the
the
miracle
queue
who
wanted
to
ask
questions
so
either
of
our
speakers?
Now,
since
I
can't
see
it,
you
guys
will
have
to
tell
me
if
anyone
moves.
Okay,
all
righty,
that
it's
really
hot
in
here,
so
I
think
that
I'm
going
to
spare
you
I'm
going
to
have
a
really
long
discussion
now
so
use
up
every
last
bit
of
our
time
now,
I'm
going
to
spare
you
a
further
discussion,
but
I
will
ask
a
request.
A
Some
some
interaction
on
the
IRT
F
discuss
mailing
lists
about
some
of
the
directions
of
publish
out
the
the
overview
that
I
gave
to
the
EDD
team
and
asked
you
guys
to
give
your
your
views,
and
so
thank
you
very
much
for
being
at
this
IRT
F
open.
Maybe
you
can
go
get
some
drinks
be
set
early
on
and
deal
with
your
dehydration
and
thank
you
so
much
to
our
speakers.
You
guys
need
this
day,
so
we
can
photograph
you
getting
your
plaques.