►
Description
The Internet Research Task Force (IRTF) Open session, including Applied Networking Research Prize (ANRP) presentations, will be held during IETF 113 at 13:30-15:30 UTC on 22 March 2022. The session will feature ANRP presentations by Sangeetha Abdu Jyothi and Bruce Spang.
A
A
A
A
A
A
A
B
A
A
B
C
C
My
name
is
colin
perkins,
I'm
the
irtf
chair
and,
as
you'll
see,
I'm
I'm
remote
for
this
meeting.
Thank
you
to
brian
trammell,
who
is
standing
in
in
person
and
looks
like
he.
He
is
very
at
home
at
that
desk
at
the
front,
so
he
should
perhaps
be
a
little
careful.
That's
thank
you
for
standing
in
brian
and
yeah.
Let's,
let's
get
started
so
as
as
usual.
I'd
like
to
start
with
a
reminder
of
the
intellectual
property
rules.
C
C
C
Please
leave
your
audio
and
video
off
unless
you're,
sharing
or
presenting
or
asking
a
question
during
the
session
and
again,
there's
a
raise
hand
button
you
can
use
to
join
the
queue.
If
you
have
questions,
if
you're
having
issues
with
the
the
av
equipment,
the
the
url
on
the
slide
allows
you
to
report
those
or
mention
miteko
in
in
the
chat.
C
My
my
usual
reminder
that
the
iatif
is
a
research
organization.
Now
it's
not
a
standards
development
organization
in
the
irtf
we're
here
to
do
research
and
we're
here
to
provide
a
venue
where
the
academic
research
community,
the
industry,
research
community,
can
interact
with
the
the
engineers
and
the
standard
setters
in
the
ietf.
C
The
irtf
is
structured
as
a
number
of
research
groups.
The
the
slide
shows
the
groups
which
are
currently
active,
those
which
are
highlighted
in
dark
blue
on
the
slide,
the
crypto
forum
group,
the
pathway,
networking
group,
information,
centric,
networking
measurement
and
analysis
of
protocols,
network
management
and
human
rights
are
still
to
meet.
Those
which
are
highlighted
in
in
light
blue
quantum
internet
group
met
in
the
previous
session.
The
the
global
access
group
earlier
this
morning
and
the
privacy
enhancements
group
yesterday
and
the
others
are
not
meeting
this
this
week.
C
So
do
please
look
out
for
those
those
sessions
later
in
the
week
and
do
do
do
join
those
sessions
also
for
the
research
groups.
I'd
like
to
welcome
sofia
jelly
who
join
recently
has
joined
as
co-chair
of
the
human
rights
protocol
considerations
group,
which
is
meeting
tomorrow
morning
and
who's
replacing
every
doria
who
who
stepped
down
last
year.
C
One
of
these
is
the
applied
networking
research
prize,
which
is
something
we
we
organize
in
conjunction
with
the
internet
society
and
the
applied
networking
research
prizes
are
awarded
for
the
the
best
recent
results
in
applied
networking.
It's
awarded
for
interesting
new
results
that
are
potentially
of
future
relevance
to
the
standards
community
and
it's
awarded
to
try
and
recognize
upcoming
people
that
are
likely
to
have
an
impact
on
the
standards
and
technologies
in
the
future.
C
Now
the
2022
edition
of
that
workshop
will
co-locate
with
the
itf-114
meeting,
which
is
taking
place
in
july
this
year
and
is
expected
to
be
a
hybrid
meeting
with
the
in-person
components
in
philadelphia.
I
believe
the
program
chairs
for
that
workshop
this
year
are
tj
chung
from
virginia
tech,
marwan
fired
from
cloud
cloudflare.
C
C
Please
do
consider
submitting
your
research
results
to
the
applied
networking
research
workshop.
This
is
a
forum
for
for
the
researchers,
vendors
operators
and
the
standards
community
to
come
together
and
talk
about
their
their
latest
applied.
Networking
research
results.
We've
had
a
bunch
of
really
good
papers
in
previous
years,
so
so,
please
do
consider
submitting
this
year.
C
C
B
D
D
The
revenue
loss
per
hour
for
some
of
those
companies
are
estimated
to
be
tens
of
millions
of
dollars
and
at
the
scale
of
countries
it
can
be
hundreds
of
millions
per
hour
or
in
billions
per
day,
and
it's
not
just
the
economic
impact.
Many
of
the
other
critical
infrastructure,
such
as
health
care,
depend
on
the
that
depend
on
the
internet
and
will
be
severely
affected
now.
What
if
we
have
an
internet
outage
lasting
weeks
or
months
and
spanning
large
areas
across
the
globe?
D
D
D
Now
coronal
mass
addictions
involve
emission
of
electrically
charged
solar
matter
and
the
accompanying
magnetic
field
into
the
space,
and
these
cmes
can
take
anywhere
from
13
hours
to
five
days
to
reach
the
earth
based
on
the
speed
and
these
cmes
are
capable
of
causing
significant
damages
to
testicle
infrastructure.
So
cmes
are
the
focus
of
this
talk
and
we'll
discuss
their
impact
more
closely.
Soon.
D
The
gic
is
only
induced
in
cables
with
ground
connections
and
a
conductor
where
the
ground
connections
are
located.
Far
apart
on
the
earth's
surface
and
finally,
the
orientation
of
the
conductor
does
not
impact
the
risk
associated
with
gic,
so
the
north,
south
or
east
west
does
not
influence
the
induced
current.
D
Now,
let's
move
on
to
the
more
important
question:
when
will
a
large
event
that
affects
the
earth
happen
next,
so
solar
events
are
extremely
hard
to
predict,
just
like
earthquakes
or
any
other
natural
disasters.
Small
scale
solar
stalls
happen
all
the
time.
What
we
are
more
interested
in
are
solar
superstorms
that
can
have
a
significant
impact
on
our
lives,
so
the
largest
solar
events
on
record
occurred
in
1859
and
1921
long
before
the
advent
of
modern
technology.
D
The
1859
event
is
popularly
known
as
the
carrington
event,
and
both
these
events
triggered
extensive
power,
outages
and
caused
significant
damage
to
the
communication
network
of
that
time,
which
was
the
telegraph
network
and
the
carrington
scale
event
missed
the
earth
by
just
a
week
in
2012.
Very
recently,
now
the
estimates
of
estimates
for
probability
of
occurrence
of
such
extreme
space
weather
events
that
directly
impact
the
earth
ranges
from
1.6
percentage
to
12
per
decade.
D
D
So
the
peak
solar
activity
very
significantly
across
this
100
year
cycle
as
well.
So
this
causes
the
frequency
of
solar
events
to
vary
by
a
factor
of
four
across
horror
maxima
in
this
hundred
year
cycle,
and
if
we
zoom
in,
we
can
see
that
modern
technological
advancement
coincided
with
the
period
of
weak
solar
activity
in
the
past
three
decades.
D
D
D
So
long
distance
cables
are
in
short,
vulnerable
internet
routers
can
be
protected
from
direct
voltage
surges
using
voltage
suppressors,
so
most
of
our
localized
infrastructure
can
be
protected,
so
they
are
safe.
Now,
moving
on
to
wireless
infrastructure,
satellites
are
directly
exposed
to
solar
storms.
They
generally
have
radiation
shielding
for
protection
from
high
energy
electrons,
but
with
very
strong
stones.
D
So
very
recently,
there
was
news
on
40
starling
satellites
that
were
lost
to
geomagnetic
storm
when
a
small
scale
storm
hit,
but
the
these
satellites
were
just
being
deployed
and
they
were
at
lower
altitudes
where
this
risk
of
drag
is
much
higher
and
that's
how
they
were
lost.
So
satellites
in
general
are
vulnerable
and-
and
this
is
well
known,
cell
towers
are
protected
from
direct
exposure
because
they're
on
the
on
the
ground
and
protected
by
the
atmosphere.
D
D
Now,
analyzing
the
impact
so
to
understand
the
impact
on
internet
infrastructure.
I
looked
at
a
broad
set
of
data
sets
comprising
of
various
internet
components,
so
the
submarine
cable
map
consists
of
submarine
cables
connecting
various
continents
and
then
the
itu
cable
map
contains
land
cable
information
from
across
the
globe
collected
from
regional
entities.
D
And
here
we
look
at
the
distribution
of
summary
in
cable
endpoints,
so
we
know
that
higher
latitudes
are
more
vulnerable
to
induced
currents
from
solar
events,
particularly
latitudes
above
the
40
degree
threshold,
so
above
40
degree,
north
and
below
40
degree
south.
So
we
evaluate
the
distribution
of
infrastructure
components
in
this
range
region.
D
So
we
look
at
the
distribution
of
population
and
submarine
cable
endpoints,
and
we
see
that
the
population
is
more
concentrated
on
lower
latitudes,
while
submarine
cable
endpoints
are
concentrated
on
higher
latitudes,
especially
between
the
us
and
europe.
We
see
a
higher
concentration
of
submarine
cable
in
points
at
more
vulnerable
regions.
D
Now
next
we
look
at
distribution
of
various
internet
infrastructure
components
across
the
way
across
the
latitudes.
So
here
on
the
x-axis
we
have
the
latitude
threshold
and
on
the
y-axis
we
have
concentration
of
infrastructure
complex,
indeed
internet,
routers,
xps
or
dns
root
servers
above
that
threshold.
D
So
the
most
vulnerable
region
is
above
the
40
degree
threshold,
which
is
denoted
by
the
dotted
line
here,
and
here
we
can
see
that
only
about
16
percent
of
the
population
is
in
this
region,
but
35
to
45
percent
of
the
internet.
Infrastructure
is
in
the
vulnerable
region,
and
this
holds
for
other
components
as
well.
You
can
see
more
analysis
in
the
paper.
D
D
Here,
the
submarine
cable
data
set
is
complete.
It
comprises
of
almost
all
existing
submarine
cables.
The
lan
cable
data
set
is
not
complete,
but
if
this
was
the
largest
publicly
available
data
set,
so
here
shorter
cables
don't
need
repeaters.
Only
cables
have
a
longer
than
150
kilometer
need
repeaters
and
need
that
conductor
along
the
length
of
the
cable
which
is
susceptible
to
damages.
D
D
The
u.s
faces
a
higher
risk
of
losing
connectivity
to
europe
during
a
solar
superstar.
So
in
the
east
coast,
most
cables
between
the
us
and
europe
are
concentrated
between
the
northeast
and
the
northeast
of
the
us
and
the
uk,
which
is
most
likely
done
for
lower
latency,
and
there
are
no
connections
from
florida
to
southern
europe,
which
is
in
the
less
vulnerable
region.
D
D
D
And
here
we
have
locations
of
public
data
centers
across
the
globe.
Red
shows
a
higher
density,
followed
by
orange
and
then
blue.
So
we
observed
that
data
centers
are
concentrated
in
vulnerable,
high
latitudes,
and
this
observation
also
holds
for
hyperscale
providers
such
as
google,
facebook,
microsoft,
etc.
D
Understanding
the
impact
on
end-to-end
behavior
of
applications
remain
an
open
challenge.
This
needs
to
take
into
account
both
land
and
submarine
cables
and
also
better
failure
models
for
cable
failures.
So
currently
there
are
some
geophysicists
working
on
developing
failure
models
for
internet
cables.
D
D
So
this
gives
us
a
sharp
interval
to
prepare
for
impact.
So
how
can
we
use
this
lead
time
effectively?
We
can
develop
plans
that
allow
us
to
prepare
for
an
impact.
So
maybe
we
can
cache
data
across
data
centers
globally,
in
an
effective
way.
How
can
internet
service
providers
react?
Do
we
need
to
increase
cash
time
in
the
dns
system?
So
there
are
a
lot
of
interesting
questions
on
how
we
can
use
this
lead
time
effectively
and,
more
importantly,
we
also
need
to
rethink
our
models
of
failure
analysis.
D
D
D
Finally,
internet
and
power
grids
are
both
designated
as
uniquely
critical,
because
all
other
critical
systems
rely
on
them.
They
are
also
in
their
dependent
on
each
other
and
both
are
susceptible
to
failures
from
solar
superstorms.
Hence
we
need
to
study
the
joint
failure
characteristics
of
this
complex
in
the
dependent
system.
D
D
Now,
if
the
power
grid
in
the
east
face
it
only,
it
will
not
cause
any
significant
effects
or
overload
on
the
western
in
the
connection,
but
on
the
other
hand,
if
all
cables
connecting
to
the
east
coast
fail,
there'll
be
significant
shifts
in
bgp
paths
and
potential
overload
in
cables
on
the
west
coast,
so
internet
is
more
global
compared
to
power
grids
and
even
regional
failures
can
result
in
significant
consequences
for
the
broader
internet.
So
understanding
the
interdependency
is
an
interesting
question.
D
So
space
weather,
particularly
coronal,
mass
ejections
from
the
sun,
pose
a
significant
risk
to
our
internet
infrastructure
and
modern
technological
advancement
coincided
with
the
period
of
weak
solar
activity.
So
the
internet
infrastructure
has
not
been
stress,
tested
under
strong
solar
events,
and
the
complete
extent
of
this
threat
is
yet
to
be
estimated.
C
E
I
do
so
first
off
thank
you
for
the
excellent
work,
I'm
a
ham,
radio
operator
and
we'll
pass
this
on
to
some
friends
that
are
interested
in
it,
because
we
we
consider
that
type
of
stuff,
all
the
time
there's
a
number
of
ham.
Radio
operators
in
the
atf,
I'm
also
a
root
server
operator,
and
so
one
of
the
things
that
I'm
interested
in
is
is
it's
worth
noting?
First
off
that
the
root
instances
or
dns
instances,
you
know
may
be
that
their
particular
installation
does
not.
E
You
know,
pose
a
problem,
but
if
they
become
disconnected
from
the
rest
of
the
system,
then
they
will
fail
to
get
updates
at
which
point
dns
will
stop.
You
know
validating
after
you
know
a
while,
as
the
signatures
expire
in
the
regardless
of
the
zone.
Did
you
study
islands?
Did
you
find
you
know
isolated
islands
of
communication
where
they
would
be
functionally
cut
off?
I
mean
the
nice
thing
about
the
routing
system.
D
Yeah,
that's
a
that's
a
very
important
question,
so
I
don't
have
a
complete
answer
yet
so
my
my
initial
paper
only
looked
at
primary
regions
of
that
will
be
affected,
but
not
the
end-to-end
disconnection
on
the
complete
graph.
So
currently
I
have
students
that
are
working
on
this
topic,
so
I
don't
have
a
complete
answer
yet,
but
it
could
be
possible
that
there
are
very
big
islands,
but
not
and
not
a
lot
of
tiny
islands.
D
It
seems
like
so
that,
based
on
the
preliminary
analysis,
most
of
asia
and
so
the
asia
to
europe,
connection
will
probably
stay,
but
the
us
to
europe
might
be
more
affected,
so
complete
disconnection.
I
don't
know
at
this
point,
but
a
significant
reduction
in
capacity
between
the
bigger
land
masses
is
possible.
C
D
There
are
radio,
solar
storms
and,
apart
from
starting
accident
with
very
low
earth
orbit,
satellites
satellites
are
fine.
Do
you
have
pointers
to
justify
the
assumptions
on
satellites
to
be
honest,
yeah?
So
that's
a
very
important
question.
So,
as
I
mentioned
in
the
talk,
these
satellites
have
protective
covering
so
so
typically
they're
deployed.
D
Storms
happened,
so
the
solas
storms
are
arranged
the
the
the
intensity
of
these
stones.
They
are,
they
range
from
g1
to
g5,
where
g1
being
the
weakest
and
g5
being
the
strongest.
So
the
storms
that
hit
the
starting
satellites
were
g1,
which
were
really
weak,
but
it
took
down
the
satellites
because
the
satellites
were
at
a
lower
altitude
than
their
final
destination
altitude.
D
D
We
need
to
have
connection
with
the
earth
ground
stations
which
can
detect
the
current
altitude
and
correct
the
satellite's
position,
and
the
solar
storm
can
extend
for
hours,
maybe
10
to
12
hours
in
the
case
of
varying
storms.
So
if
we
lose
connectivity
to
a
satellite
for
such
extended
periods
and
they
fall
out
of
orbit,
it
is
possible
that
they
could
fall
even
though
there
are
thrusters.
If
we
cannot
communicate
with
the
satellites
they
could
get
damaged.
D
So
before
the
current
international
space
station
was
installed,
u.s
had
something
called
sky
lab,
which
was
the
previous
space
station
and
that
was
destroyed
in
the
1970s
during
a
large
solar
storm
and
they
couldn't
connect
with
it
to
push
push
it
back
up
into
the
orbit.
So
with
very
large
storms
that
extend
for
hours,
it
is
possible
that
satellites
could
be
destroyed,
but
we
don't.
We
have
not
experienced
something
that
large
very
recently.
D
Yeah,
so
there
is
a
question
on:
is
the
impact
focused
on
high
latitude
or
sun
facing
side
or
are
induced
currents
the
same
globally?
So
when
it
comes
to
the
direct
impact
on
satellites
satellites
that
are
facing,
the
sun
are
at
a
much
higher
risk,
but
when
we
look
at
the
induced
currents,
these
are
caused
by
interaction
of
these
magnetic
particles
with
the
earth's
magnetic
field.
So
in
the
case
of
induced
currents,
it's
not
just
the
sun
facing
side.
C
D
Yeah,
so
I
I
think
they
need
to
rethink
resilience
at
every
layer
of
the
stack
so
with
dns.
As
I
said
like,
we
know
that
root
servers
are
very,
very
well
distributed,
but
we
don't
know
how
the
entire
tree
would
hierarchical
tree
would
kind
of
fare
under
a
solar
storm.
Do
we
need
to
change
our
caching
intervals
or
change
how
we
manage
these
dns
records?
D
So
that's
another
analysis
that
we,
what
we
plan
to
do
like
how
how
would
bgp
affair
in
such
a
scenario
but
within
an
as
like
ospf
or
the
other
entire
routing
protocols,
will
fare
very
well
because
they
are
decentralized
and
they
can
use
whatever
parts
are
available
in
a
decentralized
fashion,
so
they
seem
mostly
safe.
It's
the
in
their
domain
protocol
that
needs
more
investigation.
C
Yeah,
okay,
that
makes
sense,
I
guess,
there's
a
whole
bunch
of
coordination
issues
and
management
issues
with
large-scale
circuit
click,
cloud
cloud
provider,
infrastructure
networks
and
so
on
as
well.
Yeah
yeah,
so
some
interesting
problems,
yeah,
okay,
other
yeah,
other
questions
for
senjita.
Should
we
move
on
to
the
other
talk.
C
F
F
So
we've
got
a
pretty
typical
way
that
we
do
this,
and
normally
we
come
up
with
some
intuition
about
why
the
new
album
is
better.
Maybe
we
run
some
lab
experiments
that
show
specific
instances
where
the
new
algorithm
is
better
and
then
finally
we'll
go
and
we'll
run
a
production,
a
b
test
so
like
a
randomized
experiment
on
actual
traffic,
where
we
try
out
our
new
algorithm
and
see
if
it
actually
works
well
in
practice.
F
So
what
is
an
av
test?
So
in
a
b
test
we
have
a
whole
bunch
of
users,
and
so
there's
normally
a
bunch
of
smiley
faces
on
this
slide.
So
imagine
a
bunch
of
smiley
faces
here.
These
smiley
faces
are
going
to
be
units,
and
these
units
are
something
like
users
they're,
like
video
sessions,
their
servers
and
what
we
do
is
we
randomly
assign
traffic
to
either
treatment
or
control.
F
F
So
this
is
a
generalization.
This
statement
that
an
algorithm
improves
performance,
we're
kind
of
making
a
statement
about
what
the
world
would
be
like
if
we
went
and
deployed
the
algorithm
based
on
the
results
of
an
a
b
test
and
typically
an
a
b
test
is
like
a
pretty
small
scale
experiment,
maybe
one
percent
or
less
than
one
percent
of
global
traffic.
F
So
this
is
a
fine
sort
of
generalization
to
make,
but
it's
a
generalization
that
requires
some
assumptions,
and
one
of
the
big
assumptions
is
that
the
outcome
of
one
unit
in
the
test
doesn't
depend
on
the
other
units
in
the
test.
So,
for
instance
like
what
happens
to
me
when
treatment
is
applied
to
me,
doesn't
matter
if
treatment
is
applied
to
you.
F
F
F
F
So
if
we
think
about
it
for
just
a
little
bit,
I
mean
this
is
the
ritf
we're
talking
about
networking.
It
should
become
pretty
clear
that
interference
exists
in
congested
networks,
so
we've
got
treatment,
traffic
and
control
traffic
in
an
a
b
test,
and
this
traffic
shares
the
internet.
It
shares
networks
in
the
internet,
it
shares
cues
in
the
networks
routers
and
the
networks
and
links
in
the
network,
and
there
are
things
that
we
know
from
say:
congestion,
control,
research
that
the
treatment
algorithm
can
do
to
affect
the
network.
F
F
So
interference
clearly
exists
in
congested
networks,
but
the
question
is
it
sort
of
raises
two
questions?
The
first
question
is
like
does
this
matter,
so
maybe
there's
a
little
bit
of
interference
a
little
bit
of
bias,
but
it
doesn't
really
change
the
outcome
of
the
test.
We
run
at
all,
or
maybe
it
does,
and
maybe
it
actually
would
sort
of
significantly
change
the
decisions,
even
we
would
make
as
a
result
of
these
tests.
F
F
F
So
this
experiment,
we
ran
in
cooperation
with
netflix
and
it
was
an
experiment
involving
bit
rate
capping.
So
first,
let
me
tell
you
about
what
bit
rate
capping
is
so
back
at
the
beginning
of
profit,
19
everyone
started,
staying
home
load
on
the
internet
went
way
up,
and
governments
around
the
world
worked
with
large
video
streaming
services
like
netflix
to
reduce
load
on
the
internet
and
the
way
they
did.
This
was
by
capping
bit
rates
and
overall,
this
reduced
the
load
that
netflix
was
sending
out
by
about
25.
F
F
So
an
algorithm
goes
and
it
can
pick
each
quality
level
at
each
segment
and
then
at
the
next
segment.
It
can
pick
a
different
quality
level
for
bitrate
capping.
What
netflix
did
was
just
stop
serving
the
highest
quality
levels,
so
they
just
got
rid
of
those
and
so
for
the
same
sort
of
seconds
of
video.
F
F
So,
let's
imagine,
we've
got
some
congested
link,
so
this
link
up
here
on
the
top
left
and
we're
thinking
about
a
congested
link
here,
because
the
whole
point
of
this
bit
rate
capping
thing
was
to
reduce
congestion
in
the
first
place.
So,
let's
imagine
we
took
all
the
traffic
to
this
link
and
capped
this
traffic.
F
F
The
cap
traffic
could
sort
of
reduce
the
data
that
it's
sending
the
control
traffic
could
be
not
effective
and
there
could
be
some
free
space
available
on
the
link,
and
in
this
case
the
link
would
be
not
congested
so
in
a
b
test.
The
results
that
we
would
see
is
that
the
cap
traffic
would
use
less
bandwidth
than
the
control
traffic.
In
fact
here
it
uses
the
correct
ratio
of
less
bandwidth,
so
it's
using
half
as
much
bandwidth
as
the
control
traffic.
F
But
what
we
wouldn't
see
is
the
impact
on
congestion,
because
we
would
be
comparing
the
cap
traffic
to
the
control
traffic
and
they're
using
the
same
link
which
is
not
congested,
and
so
we
would
see
similar
metrics
for
congestion.
We'd
see
similar
packet
loss.
We
would
see
similar
viewing
delay
and
so
on.
F
A
second
possibility
if
we
were
to
run
an
a
b
test,
is
that
the
control
algorithm
could
be
running
some
sort
of
conjunction
control
algorithm.
It
could
be
doing
some
sort
of
adaptive
bit-rate
algorithm,
and
I
could
notice
that
there's
some
free
space
available
on
the
link
and
it
could
start
sending
faster
or
higher
quality
data
and
fill
up
the
rest
of
the
link.
F
In
this
situation.
The
link
could
stay
congested
and,
in
the
result
of
an
a
b
test.
What
we
would
see
is
that
the
cap
traffic
is
using
less
bandwidth
sort
of
directionally.
We've
got
the
right
direction,
but
here
we
will
get
the
wrong
ratio
so
we'll
see
that
it's
using
about
a
third
as
much
bandwidth
instead
of
half
as
much
bandwidth
and
then.
F
So
in
this
setting
we
can
visualize
the
behavior
as
follows.
So
on
the
x-axis
here
is
the
fraction
of
cap
traffic
on
the
y-axis
is
a
measurement
of
per
session
throughput,
and
so
we've
got
a
bunch
of
different
video
sessions
here
that
are
sharing
one
link,
and
so
we
can
look
at
a
per
session
throughput
for
those
video
sessions.
F
This
line
here.
This
is
going
to
be
the
throughput
for
the
cap
sessions
and
it
doesn't
depend
on
the
fraction
of
traffic
which
is
kept
because
there's
some
sort
of
upper
limit
that
we
send
at
and
as
we
increase
the
fraction
of
traffic,
which
is
capped,
there'll,
be
more
space
available
on
the
link
and
so
we'll
continue
sending
at
this
upper
level.
F
F
So
what
we're
interested
in
measuring
here
is
what
happens
if
we
were
to
go
and
we
were
to
deploy
battery
capping.
So
that's
the
difference
between
100
of
traffic
is
capped,
which
is
over
here.
On
the
right
hand,
side
100
of
traffic
is
running
the
control
algorithm,
which
is
over
here
on
the
left
hand,
side
and
the
difference
between
these
two
points.
We
call
the
total
treatment
effect
and
this
is
sort
of
the
quantity
that
we're
interested
in
measuring.
F
If
we
were
to
go
run
an
av
test,
we
would
not
necessarily
measure
the
total
treatment
effect.
Instead,
what
we
would
do
is
we
would
pick
one
point
on
the
x-axis,
where
we
allocate
some
fraction
of
traffic
to
treatment
and
control,
and
then
we
would
compare
the
two
points
vertically
along
this
line,
so
we
could
run
this
50
a
b
test
and
we
could
look
at
the
difference
between
cap
and
control
along
this
line,
as
you
can
see
from
the
graph.
F
F
So
sort
of
the
problem
here
is
that
we're
just
looking
at
one
point
and
if
we
start
looking
at
multiple
points,
we
can
measure
capping,
effects
and
compare
this
to
the
biased,
ap
tests.
So
let's
say
we
did
the
following:
let's
say
we
could
somehow
run
a
five
percent,
a
b
test
and
a
95
a
b
test
simultaneously.
F
Then,
we've
got
four
points
on
this
graph
here
and
we
could
compare
this
like
point
on
the
far
left
over
here
when
five
percent
of
traffic
is
capped
or
95.
Percent
of
traffic
is
not
capped
to
this
point
over
here
on
the
right
where
95
of
traffic
is
kept.
So
this
is
sort
of
when
we're
switching
from
most
traffic,
not
cap
to
most
traffic
capped,
and
we
can
use
this
as
an
approximation
of
the
total
treatment
effect.
F
So
this
is
going
to
be
our
goal.
One
thing
that
is
tends
to
confuse
people
about
this
is
why
here
are
we
picking
a
five
percent
and
a
ninety-five
percent,
a
b
test,
instead
of
say
a
hundred
percent
and
a
zero
percent,
because
the
total
treatment
effect
is
the
difference
between
over
here
on
the
left,
zero
percent
and
over
here
on
the
right
100.
F
The
reason
we're
doing
this
here
is
because
we're
also
really
interested
in
measuring
the
bias
of
kp
tests,
and
if
we
were
to
pick
this
allocation
of
100
and
zero
percent,
we
wouldn't
have
a
b
test
to
compare
it
to
so.
This
we
thought
was
a
nice
compromise
between
coming
up
with
an
approximation
of
the
overall
effect
of
deploying
bit
rate
counting
and
then
also
still
having
a
b
test
that
we
could
compare
against.
F
So
this
is
going
to
be
our
goal.
The
real
crux
of
measuring
this,
though,
is
going
to
be
to
run
these
two
simultaneous
experiments,
and
the
problem
is
like
as
soon
as
we
start
running
this
five
percent
test,
like
it's
going
to
be
difficult
to
run
a
95
test
because
kind
of
we're
allocating
the
same
traffic.
F
F
These
two
links
were
very
very
similar,
so
we
had
the
same
set
of
servers
connected
to
the
links,
the
same
routers
here,
the
same
isp.
The
same
content
was
available
on
all
of
the
servers
and
when
we
learned
and
looked
at
historical
data,
we
saw
like
no
significant
differences
between
the
two
links
and
most
of
the
metrics
that
we
cared
about.
F
F
F
What
we
saw
was
in
each
a
b
test.
We
saw
about
a
five
percent
decrease
in
the
throughput,
so
capping
decreased
throughput
by
about
five
percent.
But
overall,
when
we
compared
these
two
treatments,
we
saw
an
improvement
of
about
12
percent
in
throughput.
So
when
we
switched
most
traffic
from
not
cap
to
cap,
throughput
went
up
by
about
12
percent.
F
We've
got
two
links
and,
as
you
can
see,
the
two
links
have
pretty
much
identical
throughput
over
time
over
the
course
of
the
entire
day
during
these
peak
hours,
which
we
shaded
here.
These
are
the
hours
of
the
evening
load
increases
at
some
point.
Congestion
sets
in
and
when
congestion
sets
in
throughput
decreases
pretty
significantly,
and
then
it
goes
back
up
to
normal.
F
So,
during
the
experiment,
we
again
have
these
two
links,
and
this
time
the
links
are
split
between
capped
and
uncapped
traffic.
Again
we
see
during
off-peak
hours,
the
links
are
both
pretty
similar,
but
during
peak
hours
we
start
to
see
a
difference
between
these
two
links.
So
these
top
two
lines
here.
This
is
the
first
link
where
95
of
traffic
is
kept,
and
the
bottom
two
are
the
other
link
where
five
percent
of
traffic
is
capped,
because
we're
doing
capping
here,
there's
sort
of
less
data
that
we're
sending
and.
F
For
the
link
to
be
congested,
we're
going
to
need
higher
load
and
that
higher
load
will
happen
later
in
the
day.
So
when
we
cap,
we
delay
the
onset
of
congestion
for
the
link,
which
is
count,
and
we
also
make
congestion
end
earlier,
and
so
this
means
that
between
the
two
links,
the
link,
which
is
mostly
kept,
is
going
to
have
sort
of
less
congestion
and
higher
throughput.
F
F
F
In
all
of
these
cases,
we
saw
improvements
in
the
total
treatment
effect
like
capping,
improved
these
metrics,
but
the
a
b
tests
sort
of
didn't
even
get
the
direction
of
improvement
wrong.
They
either
reported
that
the
metric
would
get
worse
or
that
the
metric
was
not
affected
by
the
treatment
and
the
the
a
b
test
was
like
extremely
confident
about
this.
Like
you
know,
you
compute
the
95
confidence
intervals,
they're
very
confident
that
it's
five
to
fifteen
percent
worse
in
an
a
b
test,
when
really
it's
25
percent,
better.
F
So
let
me
be
a
little
bit
more
specific
about
these
risks.
Here,
here's
a
pretty
common
development
process
you
might
use
if
you're
using
ap
tests,
it's
pretty
similar
to
the
process
that
the
team
we
work
with
pretty
closely
at
netflix
is
using.
So
maybe
you'll
come
up
with
some
idea.
You'll
implement
that
idea.
You'll
run
an
a
b
test
for
that
idea
and
you'll
get
a
sense
of
how
the
idea
performs
and
then
you'll
iterate.
These
steps,
one
through
three
until
you'll,
come
up
with
something
that
works
pretty
well
and
then.
F
You'll
deploy
that
idea,
so
the
risks
here
is
that
when
you
run
an
a
b
test,
you
don't
get
an
accurate
estimate
of
what
happens
when
you
deploy
the
idea,
and
this
can
have
a
couple
of
different
consequences
within
the
development
process.
It
means
you
could
give
up
too
early
on
an
idea,
that's
good!
F
F
So
so
those
are
the
risks,
but
another
thing
we
saw
is
that
we
can
run
experiments
that
help
reduce
and
remove
this
bias.
So
this
perdlink
experiment
we
described
is
just
one
example,
and
there
are
other
sorts
of
experiments
you
can
use
that
are
also
less
impacted
by
bias.
So
I'd
like
to
tell
you
about
two
of
these
experiments,
the
first
kind
of
experiment
design
is
called
an
event
study
in
an
event
study.
You
have
some
event
and
then
you
compare
what
happens
after
the
event
to
what
happens
before
the
event.
F
F
F
Another
good
thing
about
this
sort
of
experiment
design
is
it's
pretty
common.
Folks
might
already
be
doing
this
when
they
deploy
things.
So
if
you
go
and
you
deploy
a
new
feature,
it's
pretty
common
to
look
at
the
metrics
before
and
after
the
deployment
and
see
how
it
the
deployment
impacted
those
metrics.
F
One
way
we
can
deal
with
this
is
with
a
switchback
experiment.
This
is
another
sort
of
experiment
design
where,
instead
of
just
switching
once,
we
switch
back
and
forth
between
treatment
and
control.
So
maybe,
on
this
day
we
use
capping.
On
the
second
day,
we
use
no
capping,
and
then
we
switch
back
and
forth.
F
So
again,
this
gives
us
an
estimate
of
the
total
treatment
effect,
and
this
is
going
to
be
more
robust
to
seasonality
issues
because
sort
of
in
order
to
get
the
bias
here
from
other
events.
Those
events
need
to
line
up
with
most
of
the
switches
throughout
the
experiment
and
especially
if
you
randomize
the
days
on
which
you
are
using
treatment
or
using
control.
This
can
help
avoid
these
issues.
F
F
F
And
so
I'd
like
to
conclude
and
say,
there's
a
lot
more
that
we
can
do
here.
So
I
think
what
our
work
showed
is
that
if
you
have
an
a
b
test
and
this
a
b
test
uses
a
congested
network,
this
a
b
test
has
the
possibility
of
being
biased.
There's
this
pathway
for
treatment
and
control
to
interfere
with
each
other
via
the
use
of
their
shared
links.
F
F
G
G
G
In
particular,
one
thing
that
occurs
to
me
is,
I
don't
know
if
you've
looked
into
seeing
if
you
made
like,
for
example,
a
client
sticky
for
a
particular
a
b
test.
As
against.
I
don't
know
how
exactly
I
haven't
read
the
paper
in
detail,
but
how
exactly
the
choice
was
made
for
serving
particular
type
of
content,
but
I
imagine
that
I
guess
where
I'm
coming
from
is
that
it
depends
on
whether
bottleneck
really
lies.
G
Then
in
that
particular
case,
I
imagine
that
if
the
user
is
in
a
bucket
that
is
either
control
or
treatment,
then
the
user
would
basically
effectively
be
at
the
far
end
of
your
of
your
scale
right
like
for
you.
By
seeing
that
effect.
Have
you
looked
into
that?
Have
you
were
your
experiments,
sticky
to
users.
G
Yeah,
that's
very
helpful.
I
was
asking
this
because,
if
you
actually
did
the
experiment
by
trying
to
if
we
tried
to
it,
could
tell
you
something
about
where
users
end
up
sharing
bottlenecks
as
well.
If
you
actually
tried
to
do
it,
the
way
that
you
were
servicing
any
of
these
ways,
those
ought
to
make
things
better
when
the
bottleneck
is
closer
to
the
user
versus
not
and
those
experiments
can
actually
shed
some
pretty
interesting
light
on
exactly
where
bottlenecks
on
the
internet.
H
Hi
brian
trammell,
not
speaking
as
sort
of
I
guess,
a
solar
flare,
backup
chair.
So
there
was
something
that
you
said
earlier
that
that
was,
I
think,
kind
of
fundamental
and
really
interesting
that
I
want
to
repeat
back.
If
we
use
what
we
know
about
networks
to
design
experiments
about
networks
we
can
get.
H
You
know
we
can
get
a
lot
smarter
about
this,
and
then
I
was
looking
at
the
graph
that
you
had
of
the
switchback
experiment
and
what
it
ex,
what
that
reminded
me
of
more
than
nothing
else,
was
like
a
diagram
of
time.
Division,
multiple
access
from
sort
of
like
a
physical
layer
textbook
that
I
looked
at
in
college
right.
H
F
H
F
Yeah
I'm
happy
to
chat
about
it.
Definitely
I
can
share
a
little
bit
about
other
experiments
that
people
run,
and
maybe
this
will
be
interesting.
So
one
thing
people
do
in
social
networks
that
I
think
is
pretty
cool.
Is
you
would
sort
of
allocate
a
user
and
then
all
of
a
friends
users
to
a
particular
treatment
and
you
sort
of
think
about
this
graph
structure
and
how
you
can
allocate
based
on
this
graph
structure
in
order
to
get
optimal
behavior
and
so
maybe
there's
something
cool.
We
could
do
with
that
for
networks.