►
From YouTube: IETF 110 IRTF Open
Description
The Internet Research Task Force (IRTF) Open session will be held at IETF 110 on 8 March 2021 at 12:00 UTC and include presentations of the Applied Network Research Prize (ANRP) presentations by Francis Y. Yan for his work on applying machine learning to video bit-rate adaptation ( “Learning in situ: a randomized experiment in video streaming”, Proceedings of USENIX NSDI 2020) and on the "Network topology design at 27,000 km/hour", Georgia Fragkouli on "MorphIT: Morphing Packet Reports for Internet Transparency", and Audrey Randall for her work on DNS caching and privacy (“Trufflehunter: Cache Snooping Rare Domains at Large Public DNS Resolvers”, Proceedings of ACM IMC 2020).
A
A
A
A
A
A
A
A
Works,
we
have
a
number
of
ways.
You
can
stay
informed
about
the
the
work
the
irtf
is
doing.
We
have
the
the
irtf
announce
mailing
list
and
you
can
see
that
the
url
for
joining
at
the
bottom
of
the
slide
here,
which
is
a
low
volume
up
list
for
announcements.
We
also
have
the
irtf
discuss
list
for
discussion
about
internet
research,
related
topics.
A
We
have
the
the
main
irtf.org
website,
which
has
information
about
the
organization
links
to
all
the
research
groups.
The
the
prize-winning
talks,
the
workshops
we
organize,
we're
also
on
social
media,
we're
on
twitter
we're
on
facebook.
We
we
have
a
linkedin
page,
which
unfortunately
isn't
linked
from
the
slide,
and
we
have.
We
have
presence
on
the
the
sitcoms
slack
channel
as
well,
so
to
look
out
for
us
on
the
various
social
media
forums.
A
The
irtf
is
is
organized
as
a
set
of
research
groups
and
there
are
14
research
groups
currently
and
of
which
13
of
them
are
all
of
the
research
groups,
apart
from
the
decentralized
infrastructure
group
a
meeting
later
this
week,
so
please
do
look
out
for
those
sessions
in
the
agenda.
I
I
believe
the
next
one
is
the
the
measurement
and
analysis
for
protocols
group
which
will
be
meeting
in
the
slot
immediately
after
this
one.
A
A
A
A
A
The
details
of
the
the
praise:
we
can
see
that
the
url,
the
rtf.org
a
and
you
can
find
links
to
all
the
the
past
prize,
winning
talks
there.
We
we
award
six
prizes
each
year,
two
for
each
meeting
typically,
and
we
we
we
always
get
a
large
number
of
nominations.
I
think
there
were
70
about
17
nominations
for
the
prices
this
time
now.
So
this
is
recently
selective
and
we've
got
some
some
really
nice
talks
coming
up
and
some
really
people
doing
some
really
nice
work.
A
The
talks
for
today
the
the
prize-winning
talks
for
today
I'm
very
pleased
to
announce
from
francis
yan
who
will
talk
about
his
work
on
machine
learning
for
video
bitrate
adaptation.
First
of
all,
and
that
will
be
followed
later
in
this
meeting
by
talk
from
audrey
randall,
we'll
be
talking
about
the
dns,
caching
and
privacy
and
and
using
dns
snooping
to
detect
malware.
A
And
let's
say,
we've
got
two.
These
two
really
nice
talks
coming
up
later
and
they're
also
archived
on
the
website,
and
you
see
the
url
on
the
slide
and
we'd
like
to
thank
the
the
internet
society
and
comcast
and
nbc
universal
for
helping
organize
and
for
sponsoring
this.
For
sponsoring
these,
these
these
prices.
A
The
paper
submission
deadline
will
be
the
21st
of
april,
and
you
know
this.
This
is
a
again
a
forum
for
applied
networking
research,
a
forum
for
the
internet,
research,
community
network
operators
and
the
standards
community
to
come
together
and
discuss
recent
results,
emerging
ideas
in
applied
networking
research.
A
And
that's
about
all,
I
have
to
say
the
that
the
remainder
of
this
meeting.
We
have
the
the
two
applied
networking
research
prize
winning
talks,
starting
with
francis
yan,
who
will
be
presenting
the
learning
in
situ
and
then
audrey,
who
will
be
presenting
on
truffle
hunter
but
be
before
we
get
to
that.
I
just
want
to
to
pass
over
to
matt
ford
from
the
internet
society
who,
who
sponsored
the
the
applied
networking
research
prize
to
say
a
few
few
words
matt's
over
to
you.
C
As
colin
mentioned,
we've
received
sponsorship
from
comcast
and
nbc
universal.
They
are
long-standing
supporters
of
the
anrp.
If
you
know
of
or
think
of
another
potential
sponsor
for
this,
do
please
get
in
touch
with
me.
My
email
address
is
pretty
easy
to
find
it's
ford.
Icesoc.Org
I'd,
be
very
keen
to
add
to
that
list
of
sponsors
for
the
anrp,
if
at
all
possible.
C
A
A
D
D
D
This
work
was
completed
back
at
stanford,
advised
by
my
former
phd
advisors,
keith
weinstein
and
phil
lavis.
Now,
let's
get
started
the
problem.
We're
tackling
here
is
adaptive
bit
rate
streaming
or
abr,
which
is
a
critical
algorithm
used
to
carry
a
large
portion
of
the
video
traffic
on
today's
internet
at
a
high
level.
D
D
This
problem
is
non-trivial
because,
let's
say
avr
believes
it's
okay
to
send
1080p
all
the
time
which
gives
the
highest
video
quality,
but
what?
If
the
network
capacity
suddenly
drops
to
a
level?
That's
unable
to
deliver
1080p
anymore
from
that
moment
on
the
playback
buffer
in
a
client's
player
will
be
drained
slowly
eventually
resulting
in
video
phrases.
D
This
talk
will
be
in
three
parts.
First
I'll
describe
puffer
a
live
streaming
platform
for
video
streaming
research,
then
I'll
show
a
surprising
finding
from
a
randomized
experiment.
We
performed
on
puffer.
That
is
the
confidence
intervals
on
the
performance
of
avr.
Algorithms
are
much
bigger
than
we
realized.
D
D
It's
a
live
tv
streaming
website
open
to
public
in
late
2018,
allowing
users
to
watch
six
tv
channels
for
free.
Our
goal
was
to
create
a
realistic
testbed
and
a
learning
environment
for
the
community
to
investigate
video
streaming
algorithms
and
we
operate
puffer
also
as
a
randomized
experiment
of
abr
schemes.
D
But
you
won't
be
aware
of
this
assignment,
while
you're
watching
tv
on
buffer,
our
server
will
record,
which
apr
algorithm
is
used
along
with
some
other
client
telemetry
on
video
quality
and
playback
buffer
for
analysis
purposes
to
recruit
users.
We
purchased
ads
on
google
and
reddit
for
keywords.
Like
live
tv,
the
other
users
we
attracted
came
from
the
press
articles
covering
puffer,
for
example,
new
york
times
recommended
puffer
to
those
who
need
free
tv
to
watch
at
home
during
the
pandemic.
D
D
D
D
Not
only
our
results
in
the
paper
are
reproducible.
All
the
user
data
collected
on
buffer
is
being
automatically
posted
to
the
website.
Every
day
after
anonymization,
you
could
select
any
data
and
view
the
algorithm
performance
we
plotted
in
figures,
but
you
could
also
download
the
data
and
do
the
analysis
using
our
scripts
by
yourself.
D
D
D
This
figure
shows
the
algorithm
performance
using
the
data
collected
on
a
single
day
in
january,
2019
puffer
streamed
more
than
17
days
of
video
to
about
600
users
watching
tv
on
that
day,
since
I'm
going
to
present
this
type
of
figure
several
times,
let's
take
a
closer
look.
First
on
the
y-axis,
it
shows
the
video
quality
measured
by
a
standard
metric
as
same
higher
is
better
on
the
x-axis.
D
D
D
D
So
we
left
the
experiment
running
for
a
week
and
collected
42
days
of
video.
Now
the
confidence
intervals
became
smaller,
but
not
actually
enough.
For
instance,
let's
look
at
the
scheme.
Mpchm
its
mean
style
ratio
is
about
0.4
percent,
but
the
confidence
interval
ranges
from
point
one
percent
to
point
nine
percent
twice
as
large
as
the
mean
value.
D
D
D
D
D
D
D
D
D
D
D
D
D
D
D
D
Let's
now
move
on
to
their
code
start
performance,
that
is
how
well
they
perform
in
new
sessions
which
they
hadn't
streamed.
Any
video
to
we
plot
the
average
video
quality
of
the
first
chunk
served
in
such
new
sessions
on
the
y-axis
and
the
startup
delay
on
the
reversed
x-axis
on
a
cold
start
to
a
new
session.
D
Firework
argues
that,
since
the
abr
algorithm
knows
nothing
about
the
network
conditions
of
the
new
session,
it
needs
some
session
clustering
algorithm
to
determine
the
nature
chunk
quality
based
on
other
similar
sessions.
Otherwise
apr
algorithms
will
have
to
choose
the
first
chunk
blindly,
which
could
be
too
conservative
or
aggressive.
We
don't
know
by
contrast.
D
Fugu
provides
an
alternative
option
recall
that
one
of
the
input
features
of
fugu
focus.
Ttp
is
tcp.
Statistics
such
as
rtt
measurements,
which
are
actually
available
as
soon
as
the
underlying
http
or
tls
or
tcp
connection,
is
established,
and
knowing
this
information
turns
out
to
allow
fugu
to
begin
safely
at
a
higher
first
chunk
quality
than
the
other
schemes,
while
maintaining
roughly
the
same
level
of
startup
delay.
D
D
A
D
D
Users
tend
to
stay
with,
but
other
than
that
we
really
don't
understand
the
reasons
and
what
other
factors
there
might
be
affecting
that
user,
behavior
and
other
question
is:
can
fugu
predict
whether
having
a
lower
base
quality
in
the
manifest
of
encoded
qualities?
Will
they
eliminate
more
stalls?
Okay,
yeah.
That's
a
good
point.
Thanks
david
for
the
question.
D
We
have
10
versions
for
each
video
track,
including
four
resolutions
with
different
crf
encoding
parameters
and
we
spread
out.
You
know
the
the
bit
rates
we
we
have
a
monit.
We
have
a
dashboard
to
monitor
whether
the
10
bit
rates
are
evenly
spread
out
in
terms
of
the
the
their
bit
rates,
the
sizes
and
their
ascent
values.
So
we
picked
10
levels
and
it's
the
lowest
base
quality.
I
believe
our
base
quality
is
already
low
enough
for
for
users,
yeah,
okay,.
A
D
D
D
E
Yeah
I
mean
I
read
some
papers
where
they
try
to
kind
of
understand
how
these
algorithms
unders
work
and
it's
usually
kind
of
a
black
box
thing.
But
the
one
thing
I
kind
of
got
from
is
that
these
algorithms
also
kind
of
change
very
frequently,
so
it's
probably
interesting
to
to
find
out
more
there,
but
also
hard
as
a
researcher.
Thank
you.
D
D
D
D
D
D
D
Usually
when
we
write
some
code,
it
can
work
99
of
time,
but
if
real
users
are
watching
it
all
the
time,
then
that
the
system
should
should
never
go
down
like
too
frequently
and
as
soon
as
buffer
stops
working,
I
would
receive
user
complaints
in
emails,
so
I
I'm
essentially
on
call
24
7..
I
have
a
so
that's
why
we
built
a
monitoring
system.
So
anytime
there's
a
bug
making
the
the
service
crash.
D
I
would
receive
a
receiver
alert
so
ahead
of
our
user
emails
thanks
our
users
for
letting
us
know,
of
course,
so
I
I
will
always
try
to
maybe
I'm
in
the
middle
of
nine.
I
would
still
get
up
and
fix
the
the
bug
immediately.
So
that's
the
biggest
challenge,
availability
and
especially
for
me,
the
almost
the
only
engineer
working
on
developing
the
the
platform.
D
B
D
A
Yeah,
it's
it's
a
challenge.
It's
a
challenge.
My
other
question
you
to
talk.
To
what
extent
is
I
mean
yeah?
Obviously,
the
specific
results
you've
got
relating
to
video.
But
to
what
extent
do
you
do
you
think
the
the
types
of
issues
you're
running
into
with
needing
to
do
very
large
scale,
very
long-running
measurement
studies
applied
to
other
types
of
network
measurement
research
and
do
you
think
you'd
find
the
same
issues
of
confidence
intervals
and
so
on?
If
you
repeated
other
types
of
network
measurement,
experiments.
D
I
I'm
not
sure
how
to
solve
this
issue.
Like
writing
research
code
on
a
production
system.
I
think
typically,
people
will
just
reproduce
where
fixed
potential
issues
potential
about
in
the
research
code,
but
in
our
case
we
just
we
want
to
faithfully
compare
with
other
algorithms
and
report
and
evaluate
their
performance.
D
For
our
algorithm
and
another
deeper
in
first
learning
based
algorithm,
yes,
we
have
to
take
into
account
the
compute
time
and
the
compute
resources
required,
but
fortunately
none
of
the
algorithms
consumes
too
much
computer
resource
and
it
takes
just
typically
several
milliseconds
to
to
compute
the
decisions
apr
decisions
online.
So
that's
not
a
bottleneck.
D
So
those
are
the
input
features
as
the
states
and
they're
also
the
updates
provided
to
the
controller.
Sorry,
so
those
are
the
inputs
to
to
ttp
to
the
transmission
time
predictor
and
for
the
model
based
controller.
The
input
is
the
current
playback
buffer
level
and
all
the
necessary
chunk
sizes,
because
it
needs
to
run
this
dynamic
programming,
also
known
as
value
iteration
algorithm
online.
D
Yeah,
that's
great,
so
does
it
leave
a
slow
start,
often
so,
first
of
all
the
pensive
paper
in
sitcom
2017.
If
I
remember
correctly,
they
disabled
this,
the
free
like
the
timeout,
the
tcp
time
timeout
such
that
it
never
or
it
hardly
leaves
the
slow
start.
Sorry,
it
hardly
leaves
the
congestion
avoidance
phase.
So
it
doesn't
return
to
the
slow
start
phase,
because
when,
when
you
send
a
video
chunk
every
two
to
four
seconds
we
don't
want
to.
D
A
All
right,
so,
thank
you,
francis
really,
nice
talk
some
really
good
discussion
there.
I
I
would
normally
say
that
francis
will
be
around
for
the
rest
of
the
week
and
you
should
grab
him
in
the
break.
If
you
have
any
questions
clearly,
this
is
a
little
bit
difficult,
but
hopefully
francis
will
be
able
to
join
that
the
gather
town
in
some
of
the
breaks
or
drop
him
an
email
or
on
the
chat.
If
you
want
to
talk
further
about
this,
this
work.
A
All
right,
so
at
this
point
we
will
move
on
to
the
the
second
of
the
prize.
Winning
talks
today,
which
is
by
audrey
randall
audrey
is
a
third
year
phd
student
studying
internet
measurement
and
security
at
the
university
of
california,
san
diego,
her
research
interests
center
around
measuring
and
understanding
harmful
behavior
on
the
internet,
from
underground
crime
to
stock,
aware
to
dns
hijacking.
B
B
If
you
could
observe
enough
dns
requests,
you
could
presumably
study
these
types
of
harm
in
more
detail.
You
could
figure
out
how
prevalent
they
are
where
they
occur,
how
frequently
they
occur,
but
to
do
that
you're
going
to
need
to
observe
a
lot
of
dns
requests
because
you're
looking
for
the
needle
in
the
haystack
you're
looking
for
a
very
small
amount
of
signal
in
the
large
amount
of
noise.
B
It
used
to
be
that
it
was
only
power
users
and
people
who
were
really
tech
savvy,
who
would
be
using
public
resolvers
but
we're
starting
to
see
them
get
hard-coded
by
default.
For
example,
google,
home
routers
all
use
google's
quad
8
service
by
default
and
firefox
routes,
all
their
dns
queries
to
cloudflare.
B
B
Well,
of
course
the
answer
is
yes,
there
is
a
well-known
technique.
That's
been
around
since
at
least
2004
called
dns
cash
snooping,
but
in
the
past
it's
been
presented
as
an
attack
and
it's
considered
a
privacy
threat
and
for
good
reason,
most
of
the
time
what
researchers
were
doing
when
they
did
cash
snooping
was,
they
would
scan
the
whole
internet
and
they
would
try
and
see
which
devices
would
answer
a
dns
request.
B
B
So
for
the
remainder
of
this
talk,
I'm
first
going
to
go
over
some
brief
background
on
cash
snooping
for
anyone
who
hasn't
seen
the
details
in
a
while,
and
then
I'm
going
to
talk
about
how
to
do
it
on
public
resolvers.
To
do
that,
you
need
to
understand
their
caching
strategies,
so
we
as
researchers
had
to
reverse
engineer
the
caching
strategies
of
four
large
public
resolvers.
B
B
What
you
can
do,
if
you
are
a
snooper,
is
make
a
request
for
example.com
but
set
a
flag
that
tells
the
resolver.
It
is
not
allowed
to
check
the
authoritative
name
server
that
way.
If
you
get
a
valid
response
back
with
a
valid
ip
and
a
valid
ttl
or
time
to
live
value,
then
you
know
that
the
domain
was
cached.
B
But
the
thing
about
cash
snooping
as
a
measurement
technique
is
that
it
only
provides
a
lower
bound
on
the
number
of
users
that
are
accessing
a
domain.
So
if
multiple
users
have
hit
the
same
cache
for
the
same
domain
before
that
ttl
expires
and
the
record
is
removed
from
cache,
you
won't
be
able
to
observe
them.
B
But
cash
snooping
on
a
single
resolver
is
actually
reasonably
straightforward
in
order
to
do
it
on
a
public
resolver
things
get
significantly
more
complicated.
So
let
me
talk
next
about
how
public
resolvers
work
in
broad
terms
and
then
how
they
work
in
more
specific
terms,
when
a
user
wants
to
send
a
request
to
a
public
resolver.
B
That
query
is
first
routed
using
ip
anycast
to
the
first
available
or
the
closest
point
of
presence
or
pop.
Once
there
it
can
be
routed
to
one
of
any
number
of
front-end
caches.
There
are
a
lot
of
these,
and
if
it
misses
in
those
front-end
caches,
it
will
be
sent
to
one
of
usually
several
back-end
resolvers.
B
B
B
B
At
this
point,
I
have
to
introduce
the
concept
of
a
ttl
line,
which
is
just
our
model
of
how
a
ttl
decreases
in
a
cache,
a
ttl
and
a
cache
ought
to
decrease
by
about
one
second
per
second.
So
if
you
plot
a
bunch
of
measurements
that
have
all
hit
the
same
cache,
you
ought
to
see
if
you
plot
their
timestamp
against
their
ttls,
that
that
decreases
by
one
second
per
second,
and
that's
this
green
line
in
the
figure
here.
B
B
B
B
We
assume
that
each
of
these
measurements
filled
a
new
cache
because
they
came
back
with
the
maximum
ttl
value,
and
we
were
able
to
confirm
this
because
we
controlled
the
authoritative
name
server
for
the
domain
that
we
were
querying.
So
we
confirmed
that
every
time
our
authoritative
name
server
got
a
new
request.
B
You'll
also
notice
that
all
of
the
measurements
that
are
not
one
of
the
top
row
of
circle
dots
lie
on
one
of
the
ttl
lines,
so
they
look
like
they
came
from
one
of
the
caches
that
we
observed
to
be
filled.
So
that's
great.
That
means
that
open,
dns
and
quadmine's
caching
architecture
is
reasonably
straightforward.
B
When
we
ran
this
experiment
on
cloudflare,
we
got
a
very
different
looking
graph,
so
we
do
get
a
a
first
measurement
which
came
back
with
the
maximum
ttl,
but
all
of
the
measurements.
After
that,
look
like
they
came
from
the
same
cache,
which
we
would
have
thought
would
be
unusual
in
a
resolver
of
cloudflare's
size.
B
You
can
also
see
that
for
a
while,
the
measurements
look
like
they're
exactly
on
the
ttl
line,
but
then
they
start
to
drift
over
time
and
we
notice
that
they
would
always
drift
upward.
So
what
we
think
is
happening
is
that
cloudflare
has
a
shared,
front-end
cache
shared
and
distributed
as
soon
as
a
measurement
arrives.
In
one
cache,
it
is
shared
with
all
of
the
others.
B
There
is
a
question
with
cloudflare
strategy
of
whether
or
not
it
is
completely
compliant
with
the
dns
rfc
for
ttls.
The
maximum
drift
that
we
saw
of
away
from
the
true
ttl
value
was
about
80
seconds.
So
we
were
using
a
domain
at
the
time
with
a
ttl
of
about
three
hours
and
we
saw
that
there
were
still
measurements
in
cash
whose
ttls
hadn't
yet
expired
for
about
80
seconds
after
they
should
have
expired.
B
Now,
it's
important
to
note
that
the
drift
scales
with
the
maximum
ttl,
so
probably
even
if
you
have
a
60
second
ttl
you're,
only
going
to
have
a
drift
of
a
few
seconds
and
that's
probably
not
going
to
be
an
issue
for
you.
Even
if
you
have
such
a
short
ttl
and
if
you
have
a
long,
ttl,
you're,
probably
tolerant
of
more
drift.
So
we
concluded
that
the
actual
problems
here
are
likely
to
be
very
small.
B
B
Champet
all
found
that
they
could
make
requests
and
and
get
a
accurate
ttl
back
on
the
original
requests,
but
then
they
would
keep
making
requests
and
they
would
find
subsequent
ttls
to
be
wrong
because
it
looked
like
those
ttls
were
coming
from
caches
that
had
never
been
filled
and
then
reprimando
at
all
noticed
the
same
effect
and
called
these
mystery
caches,
ghost
caches,
which
we
thought
was
a
great
name
for
them.
So
why
on
earth
are
these
caches
getting
filled
without
being
queried?
B
B
So
what
we
think
is
happening
is
this:
google
is
using
what
we
call
a
dynamic
caching
strategy
when
a
request
comes
into
google
and
it
misses
in
a
front-end
cache.
That's
its
light.
This
light
blue
cache.
Here
then
it's
going
to
get
forwarded
to
a
back
end
cache
and
let's
assume
that
that
back-end
cache
is
already
full
and
it
has
a
ttl
less
than
the
maximum
value.
B
So
you
can
think
of
it
like
this.
Every
request
that
comes
into
google
dns
has
a
chance
to
spawn
a
new
cache
that
is
visible
to
cash
snipping.
So
that's
great
news
for
us
as
researchers
running
a
measurement
study,
because
we
will
see
a
much
greater
percentage
of
unique
queries
on
google
than
we
will
anywhere
else.
B
B
They
did
expire
even
if
the
ttl
had
not
reached
zero.
Yet
when
the
original
back-end
cache
expired,
so
that's
good,
but
we
noticed
that
a
user
could
make
a
request
just
before
the
ttl
of
the
backend
cache
expires
and
get
a
cache
that
had
just
been
filled,
and
that
could
lead
to
extending
the
ttl
to
twice
as
long
as
it
should
be.
B
Whether
or
not
this
is
actually
a
problem
is
not
up
for
us
to
decide.
We
couldn't
think
of
a
use
case
where
it
would
be
super
problematic,
but
we
do
have
a
question
of.
Why
would
why
would
this
be
a
useful
strategy?
Why
store
the
maximum
ttl
in
the
front
end
caches
rather
than
just
copying
the
ttl
from
the
back
end
caches?
B
Now
it's
great
that
google
did
this
from
our
point
of
view,
because
it
really
enabled
our
measurement
study,
but
we
couldn't
come
up
with
a
reason
why
it
would
be
more
efficient
or
more
performant
to
do
that.
So
if
anyone
is
here
from
google
or
if
anybody
wants
to
weigh
in,
I
would
love
to
get
somebody's
thoughts
on
that
when
I'm
done
with
this
talk.
B
So
to
summarize,
open
dns
and
quad
9
appear
to
have
a
pretty
straightforward
caching
strategy,
and
we
don't
think
that
that
caching
strategy
ever
manipulates.
The
ttls
of
the
responses
at
all
cloudflare
has
this
shared
and
distributed
front-end
cache,
and
we
do
notice
that
the
ttls
are
affected
slightly
by
it.
B
But
we
don't
think
that's
likely
to
be
too
much
of
an
issue,
because
the
drift
is
so
small
compared
to
the
length
of
the
maximum
ttl
and
google
has
what
we
call
a
dynamic
caching
strategy
and
that
can
result
in
a
ttl
received
by
the
client
being
about
twice
as
long
as
it
should
be,
because
you
could
receive
a
maximum
ttl
right
before
the
back
end.
Caches
ttl
was
set
to
expire,
so
you
should
have
been
receiving
a
very
small
ttl
and
you
receive
one
that's
closer
to
the
maximum.
B
B
All
it
does
is
send
continuous
dns
queries
across
the
us
for
the
domains
that
we're
interested
in
when
it
gets
the
responses
back.
It
interprets
them
according
to
our
models,
to
try
and
figure
out
how
many
caches
were
filled
and
we
go
from
there
to
estimating
counts
of
users.
In
some
cases
we
have
three
months
of
data
or
did
at
the
time
we
wrote
this
paper
from
march
to
may
in
2020.
B
So
we
ran
an
experiment
where,
from
900
different
ripe
atlas
probes,
we
placed
a
domain
we
controlled
into
the
caches
of
public
resolvers.
The
idea
was
just
to
put
it
there,
as
if
people
across
the
u.s
had
done
it.
Naturally,
then
we
used
truffle
hunter
to
try
and
observe
it
in
those
public
resolver
caches,
because
it's
a
domain
we
control.
We
could
conclude
that
the
number
of
requests
that
came
into
our
authoritative
name
server
should
be
the
true
number
of
filled
caches,
except
in
the
case
of
google,
which
of
course
does
its
own
thing.
B
We
found
that
we
performed
best
on
opendns
and
cloudflare,
except
in
the
case
of
one
particular
cloudflare
pop,
where
we
think
there
was
some
routing
going
on
that
we
didn't
account
for
during
our
experiment
on
gp
dns.
It
turned
out
to
be
difficult
to
accurately
remove
all
the
front-end
caches
that
had
been
filled
by
our
own
probes,
so
we
conservatively
moved
removed
more
than
we
had
actually
created
in
order
to
ensure
we
never
over
counted.
B
So
if
it
turned
out
to
be
that
unbound
was
the
software
that
had
cached
our
domain.
That
record
essentially
became
invisible
to
us,
so
it
does
mean
we
can't
observe
about
half
of
the
filled
caches
at
any
quad
9
pop
the
takeaways
here
are:
first,
we
were
able
to
tune
our
algorithm
so
that
we
almost
always
underestimate,
which
is
good,
because
our
goal
is
to
provide
these
lower
bounded
estimates
of
prevalence
and
second,
even
on
the
resolvers,
where
we
have
high
error.
We
do
see
at
least
half
of
the
filled
caches.
B
B
B
B
If
you
haven't
heard
the
term
before
it's
kind
of
this
emerging
spyware
thread,
it's
this
software
that
can
be
installed
on
a
target's
device,
either
a
phone,
usually
a
phone
or
a
desktop
computer
and
it
tracks
them.
It
can
record
location.
It
often
has
key
loggers
to
record
text
social
media
browsing
history,
things
like
that
and
oftentimes.
It
can
record
ambient
sound
and
video
of
the
device
as
well,
and
it
can
hide
its
presence
on
the
device.
B
B
B
The
prevalence
of
overt
stalker
wear
is
hard
to
estimate
by
any
other
means,
because
it's
very
difficult
to
observe
it
in
the
wild
prior
work
in
this
space
has
mostly
been
conducted
in
clinical
settings,
so
researchers
will
conduct
individual
one-on-one
interviews
with
targets
and,
unfortunately
that
gives
them
a
low
sample
size.
During
these
interviews
they
found
few
to
zero
of
these
overt
apps
in
the
wild,
but
a
simple
google
query
will
turn
up
dozens
of
them
and
there
are
ads
all
over
the
place.
B
So
it
really
does
beg
the
question
of
how
much
of
this
overt
stock
aware
is
out
there.
Additionally,
by
the
time
a
target
has
come
in
to
talk
to
a
professional,
they
have
often
already
reset
their
devices.
So
it's
difficult
for
a
clinic
to
tell
which
apps
were
on
there
before
the
reset
and
finally,
clinics
often
lack
technical
expertise.
So
if
they
aren't
working
with
someone
who
does
have
expertise
in
this
space,
it
can
be
very
difficult
for
them
to
tell
if
a
device
has
soccerware
installed
up.
B
So
if
you
want
to
know
how
many
devices
have
stock
aware
installed,
all
you
have
to
do
is
measure
that
request
rate
and
then
divide
the
number
of
filled
caches.
You
see
by
the
request
rate
of
the
app
that's
the
technique
that
we
use
to
come
up
with
this
figure
here.
This
graph
shows
the
maximum
targets
that
we
ever
observed
with
stock
aware
installed
at
any
one
time
across
the
united
states.
B
B
The
two
most
interesting
apps
that
we
found
were
called
mobile,
tracker,
free
and
spy
to
mobile.
Those
are
the
two
most
frequent
that
you
can
see
at
the
top
of
this
chart.
Mobile
tracker,
free,
we
suspect,
is
so
popular
because
out
of
all
the
overt
apps
we
studied,
it
was
the
only
one
that
wasn't
subscription
based.
B
We
again
see
that
mobile
tracker
free
is
the
one
that
was
visited
most
frequently,
but
spike
mobile
has
fallen
down
in
the
rankings
a
little
bit
so
clearly,
it
is
the
case
that
the
popularity
of
the
app
does
not
necessarily
correspond
to
how
many
times
somebody
is
checking
the
dashboard
we
theorize.
That
might
be
because
of
differing
app
capabilities.
Mobile
tracker
free
has
a
lot
more
features
than
spy
to
mobile
spider.
Mobile
is
mostly
good
at
tracking
location.
B
It's
pretty
hard
to
detect
when
a
student
is
using
this
because
they're
not
actually
plagiarizing
they're,
not
copying,
work
that
exists
already
they're
hiring
someone
to
create
original
content
for
them.
Of
course,
your
mileage
may
vary.
Some
of
these
services
are
better
than
others,
but
there
are
a
few
that
are
good
enough
to
get
a's
in
most
cases,
even
including
college,
and
sometimes
graduate
classes.
B
So
it's
of
course
hard
to
observe
in
the
wild,
because
students
aren't
going
to
just
admit
that
they
have
done
cheating
even
on
anonymous
surveys,
which
is
how
a
lot
of
this
work
has
been
done
in
the
past.
So
we
had
truffle
hunter
look
for
it
and
we
observed
that.
Yes,
you
see
a
lot
of
requests
per
day
to
these
contract
cheating
websites.
B
Now,
of
course,
a
request
made
for
the
website
doesn't
necessarily
mean
that
a
student
bought
anything,
but
it's
still
an
interesting
number
to
observe.
We
saw
that
some
of
these
decrease
some
of
these
services,
which
we
measured
over
the
last
couple
weeks
of
may
were
decreasing
over
time,
which
we
thought
was
interesting.
It
might
indicate
that
schools
are
letting
out
for
summer
break
so
demand
for
cheating
is
going
down.
B
And
then,
finally,
because
we
had
some
of
these
domains,
we
look
for
typo
squatting.
These
domains
are
pretty
old.
We
don't
expect
that
they
are
being
used
to
fish.
Anybody
anymore
because
received
wisdom
in
prior
work
says
that
phishing
domains
and
type
of
squatting
domains
usually
roll
over
very
quickly.
They
get
blacklisted
and
then
the
miscreants
move
on
to
other
domains.
B
So
the
takeaway,
from
our
point
of
view,
is
that
cash
snooping
on
public
resolvers
shouldn't
actually
be
gotten
rid
of.
Yet
we
argue
that
there
are
minimal
privacy
concerns
when
you're
cash
snooping
on
public
resolvers,
because
there
are
too
many
users
to
figure
out
which
user
put
a
domain
into
cash.
B
And
if
you
allow
cash
snooping
on
these
resolvers,
then
you
can
measure
types
of
harm
that
are
otherwise
very
difficult
to
study,
in
particular
for
stalker
ware.
It's
very
difficult
to
figure
out
how
much
of
this
stuff
exists
in
the
wild
and
each
instance
of
stalkerware
represents
a
significant
amount
of
harm
being
done.
B
Furthermore,
contract
cheating
is
difficult
to
study,
because
students
are
just
not
honest
about
whether
or
not
they've
bought
cheating
software
and
then
there's
other
phenomena
which
we
didn't
get
a
chance
to
measure
very
well,
which
we
would
like
to
look
into
more
in
the
future,
such
as
these
new
hack
for
hire,
services
and
phishing,
which
by
all
accounts,
is
quite
common.
But
we
would
like
to
see
how
much
of
it
is
happening
in
various
places
around
the
world.
B
B
A
B
So
it's
it's
not
necessarily
a
good
thing
to
block
this
stuff,
so
most
threat
fees.
Don't
let's
see
jim
asked
where
we
got
the
domain
names
for
contract
cheating
services
they're
very
easy
to
find
if
you
google
them.
So
what
we're
measuring
is
hits
on
the
main
landing
page
for
each
of
these
services,
when
you
just
get
there
from
a
google
search.
B
G
B
The
particular
dual
apps
dual
use,
apps
that
we
measured
did
seem
to
be
less
common
than
the
most
common
types
of
overt
stalker
wear,
but
there
could
be
a
number
of
reasons
for
that.
So,
first
of
all,
we
didn't
measure
as
many
dual
use
apps
by
choice,
because
people
know
more
about
the
dual
use
ones
about
their
their
usage
and
their
prevalence
already,
and
what
we
really
wanted
to
know
was
the
overt
ones,
so
we
may
not
have
found
the
most
popular
ones.
B
H
H
B
B
Google
was
the
exception
to
this,
because
we
found
that
if
we
had
hit
a
back
end
resolver
that
had
the
query
cache
and
then
we
hit
a
front-end
resolver
that
did
not
have
the
query
cache
a
front-end
cache
that
did
not
have
the
query
cached,
then
the
the
record
would
be
copied
from
the
back-end
resolver
to
the
front-end
resolver.
So
our
own
probes
would
fill
the
caches
there
and
we
did
find
that
it
was
a
challenge
to
remove
the
poisoning
that
we
had
done
and
then,
as
someone
just
pointed
out
in
the
chat.
B
Yes,
so
unbound
will
return
a
refused
response
when
you
try
and
make
a
query
with
the
recursion
desired
flag,
which
was
why
we
could
only
measure
half
of
the
caches
at
quad9,
because
if
they
had
a
resolver
with
unbound
software
that
received
the
original
query,
then
it
became
invisible
to
truffle
hunter.
Does
that
answer
your
question.
A
B
Yeah,
so
I
think
the
the
issue
is
that
sorry,
the
the
issue
with
whether
or
not
to
enable
recursion
desired
is
you
want
to
enable
recursion
desired?
If
you
want
to
enable
these
types
of
measurement
studies
on
resolvers,
where
there
are
few
to
no
de-anonymization
risks
for
users
and
those
are
the
large
resolvers
if
you
are
running
dns
resolving
software-
and
you
know
that
it's
going
to
get
put
on
small
home
routers
which
can
be
used.
B
B
B
B
No,
it
would
actually
probably
be
reasonably
straightforward.
We
were
keeping
our
experiment
to
the
united
states
in
the
first
place
so
that
we
wouldn't
add
too
much
load
on
their
system,
and
then
it
just
ended
up
that.
That
was
the
data
that
we
had
to
to
publish
our
results.
We
are
at
the
moment
working
on
expanding
a
similar
tool
to
truffle
hunter,
but
instead
of
measuring
domain
usage,
we're
trying
to
measure
dns
hijacking
and
we
do
want
to
expand
that
to
worldwide
using
similar
techniques.
B
A
A
Okay,
so
I
had
a
question
I
mean
we're
seeing
obviously
people
making
increasing
use
of
you
know
different
types
of
dns
transports.
You
know
over
tls
or
over
https
and
we're
seeing
new
techniques
like
oblivious
dns
that
are
getting
proposed.
Do
any
of
these
make
a
difference
to
the
type
of
work
you're
doing.
B
A
A
B
Yes,
it
might
the
thing
about
stalkerware
apps
is
that
they
tend
to
be
incredibly
unsophisticated.
They
don't
try
and
obfuscate
their
code
at
all
half
of
them
crash
as
soon
as
they
get
installed
on
the
phone,
at
least
on
an
older
device,
and
they
they
seem
to
have
a
lot
of
bugs
and
problems
in
general.
So
I
would
be
surprised
to
see
them
adopting
any
sophisticated
techniques
like
that
in
the
near
future,
but
there
are
some
that
are
certainly
ahead
of
the
game
like
flexispy.
B
Okay,
jonathan
is
pointing
out
that
this
might
mess
with
our
geolocation.
Yes,
but
I
think,
as
we
realize
anycast
is,
is
not
a
great
way
to
do
geolocation
in
any
case,
so
you
know,
our
geolocation
is
a
little
bit
suspect.
We
can
say
that
users,
possibly
in
a
very
broad
region,
are
experiencing
more
stuck
aware
or
whatnot.
Then
then,
users
across
the
country,
but
anycast
is
not
a
great
way
to
figure
out
where
users
actually
are
to
the
best
of
my
knowledge.
In
any
case,.
A
A
A
A
A
If
you
have
any
any
questions
or
want
to
talk
about
any
of
this
work
further-
and
that's
that's
all
we
have
for
this
session
today,
look
out
for
the
applied
networking
research
workshop
call
for
papers
coming
up
and
look
out
for
the
rest
of
the
the
irtf
sessions
later
this
week
and
the
the
recordings
of
these
talks
and
the
links
to
the
papers
are
on
the
irtf
web
page.
If
you
want
to
look
into
them
in
more
detail
all
right
thanks.