►
From YouTube: COVID-19 Network Impacts Workshop, 2020-11-11
Description
Recording from Day 2 of the Internet Architecture Board's COVID-19 Network Impacts Workshop.
Day 1 Video: https://youtu.be/RTJNaE7TnGA
Workshop home page: https://www.iab.org/activities/workshops/covid-19-network-impacts-workshop-2020/
B
I
do
not
want
to
why
it
muted
me
now
we're
recording,
thank
you
and
thank
you
for
letting
me
know
I
was
talking
about.
I
do
that
about
three
times
a
day,
so,
first
of
all
for
people
who
weren't
here
on
the
first
one,
this
is
an
itf
sessions
covered
under
notewell
the
agenda.
What
we
had
today
was
to
have
jano
is
going
to
produce
some
stuff,
I'm
not
sure.
B
If
he's
here
yet
he
said
he
was
going
to
be
a
few
minutes
late,
but
when
he
arrives,
we'll
probably
do
that
yari
completely
out
of
the
blue
without
telling
you
I
would
love
it.
If
you
could
present
your
your
slide
three
on
user
behavior,
because
it
sort
of
fits
right
in
here
I'll
talk
for
a
minute,
but
mostly
I
want
to
run
today,
I'm
just
having
discussion
around
initially.
B
What
sort
of
things
happened
with
between
the
people
trying
to
run
applications
on
the
internet
and
people
trying
to
run
the
internet
whatever
we
can
talk
about
with
that?
I
think
there's
a
ton
of
people
on
this
call
that
do
have
information
about
that
type
of
environment
and
then
move
into
sort
of
broader
things.
Issues
came
up
with
what
things
came
together.
B
I
prefer
to
have
this
much
more
of
a
discussion
today.
It
was
great
to
see
all
the
papers
and
presentations
on
monday.
It
was
super
information
and
but
we
didn't
get
enough
time.
We
didn't
get
the
time
to
talk
about
them,
so
I'm
hoping
some
of
that
can
come
up
and
happen
today.
Thoughts
on
this
agenda,
other
things
we
should
insert
remove
deal
with
so.
C
It's
I
was
just
wondering
if
we
towards
the
end,
I
think
we
should
either
tee
up
some
topics
for
friday
or
decide.
We
don't
need
friday,
which
is
either
could
be
possible,
and
there
is
one
topic
I
think
that
would
be
good
to
cover
somewhere,
which
is
the
other
part
of
this.
That
was
a
success
story.
C
F
B
B
E
I
I
B
Well,
let's
jump
into
that
when
we
get
to
the
discussion,
part
that
sounds
interesting
to
hear
about,
and
some
people
typing
not
on
mute,
so
do
music,
not
speaking.
I
do
not
see
donna
here
yet
it's
hard
for
me
to
see
all
of
this
same
time
so
yeah,
I
think
you're,
not
here
so
yori.
Would
you
be
willing
to
talk
about
your
slides
here
for
a
second
I'll
put
it
up
the
user
behavior,
one
of
just
what
you
guys
saw
and
how
things
were
shifting
on
the
applications.
A
D
Will
the
slides
three
then
directly,
so
I
mean
for
for
the
background
the
these
are
slides
that
I
have
made
just
to
sort
of
highlight
the
obvious
point
that
there's
more
to
measurements
than
than
just
a
sort
of
you
know
congestion
or
not
question
or
traffic
amount.
It's
also
about
the
qualitative
changes
and
what
people
actually
doing
and
what's
what's
happening
with
the
users
and
applications
and
so
forth,
and
this
was
research
and
not
done
by
me.
So
I'm
only
reporting
other
people's
research,
but
it
was
reasons
done
by
ericsson.
D
Some
of
those
things
I
believe
this
one
was
based
on
the
questionnaires
and
so
on.
The
x
axis
is
the
amount
of
user
change
or
number
of
users
being
changed
in
a
particular
category
of
things
and
on
the
y-axis
is
the
actual
usage
chain.
So,
and
you
would
see
some
obvious
things
not
unexpected,
like
ride.
Hailing
and
travel
applications
were
not
particularly
useful
this
year
and
then
remote
working
e-learning,
health
applications
copied
19,
apps
and
so
on
were
quite
popular.
D
So
I'm
not
really
sure,
there's
much
more
to
say
about
this.
This
is
sort
of
you
know,
one
one
view
of
things
like
what
what
did
people
actually
do
and,
of
course
there's
some
conclusions,
I'm
going
to
draw
maybe
colony.
If
you
briefly
show
the
next
slide,
two
more
things,
so
you
sort
of
theorize
like
what
what
actually
was
behind
the
like.
D
Your
call,
but
people
were
relatively
satisfied
with
their
support
and
felt
that
mobile
brokeband
was
at
least
as
good
as
it
was
before
the
crisis.
Now
you
know
I
wish
they
had
asked
exactly
the
same
questions
and
both
questions
on
both
groups.
They
didn't
this
may
have
to
do
with
you
know
previous
questions.
D
I
should
mention
that
there's
some
potential
long-term
impacts
based
on
what
people
started,
doing
like
new
user
groups
like
the
elderly,
used
new
applications
and
so
that
that's
probably
going
to
affect
the
future,
not
just
serious
behavior
and
then
lots
of
people
care
about
resiliency,
so
yeah.
I
think
that's
it
for
me.
B
Yeah,
so
I
you
know,
I
I
thought
that
you
know
when
sort
of
you
know
key
things
and,
as
we
just
saw
so
much
shift
to
you
know
a
ton
of
internet
usage
from
an
application
point
of
view,
and
we
you
know
you
saw
that
in
a
bunch
of
the
papers
that
were
presented
sort
of
talked
about.
You
know
those
types
of
points.
B
One
thing
I'll
say
from
the
the
area
that
I
was
working
in
was
was
around
as
webex,
but
all
of
the
web
conferencing
systems,
so
the
three
major
ones
are
our
three
largest
are
microsoft,
zoom
and
webex,
and
they
they
all
saw
incredible
growth
in
their
their
usage.
In
fact,
similar
incredible
growth.
On
that
there's,
there's
we
don't
have
the
data
out.
We
can't
show
the
data
yet,
but
synergy
as
a
research
firm
has
data
from
all
those
companies
will
show
how
they
all
compare
and
how
they
all
grew.
J
B
B
Here
we
had
a
a
15
or
20
percent,
improve
our
increase
in
network
traffic
is
sort
of
what
we
saw
different
several
presentations
over
the
last
couple
days
or
on
monday-
and
you
know
that's
is
certainly
true,
but
definitely
from
the
point
of
view
of
I
think
of
the
video
conferencing
vendors.
That
was
a
real
problem,
all
kinds
of
places.
We
had
incredible
numbers
of
systems
that
didn't
have
enough
bandwidth
areas
where
we're
having
really
network
performance
issues.
B
B
Of
course,
people
extended
the
capacity
of
their
network
scale,
their
networks,
we're
now
back
to
a
situation
where
those
those
systems
have
gone
back
up
to
the
full
bit
rates
they
were
before,
but
I
think
that's
one
of
the
sort
of
areas
that
I
I'd
like
to
talk
a
little
bit.
You
know
hope
we
can
talk
a
little
bit
more
about
when
we
get
to
the
discussion
part
of
this,
so
I
think
with
that.
Maybe
I
see
yana
is
here
now,
so
maybe
we
should
jump
into
your
slides.
B
K
K
K
Better
than
this,
at
any
rate,
I
thank
you
for
giving
me
the
opportunity
to
talk
about
this
with
this
data,
I'm
going
to
try
and
go
through
this
in
about
10
to
15
minutes.
So
at
a
high
level,
we
we
we
monitored,
and
we
were
obviously
so.
So
let
me
start
off
by
saying
who
I
am
I'm
jenna
enger,
I'm
a
distinguished
engineer
at
fastly,
which
is
an
edge
compute,
slash,
cdn
company.
K
K
We
serve
a
lot
of
traffic
for
customers
to
users
around
the
world,
so
obviously
we
were
sort
of
at
the
forefront
of
the
a
lot
of
the
the
the
effects
of
what
we
saw
happen
with
covet
in
terms
of
internet
traffic,
and
so
I'm
gonna
talk
about
that
in
these
slides
today
now
feel
free
to
stop
me
as
I'm
going
through
this
and
ask
questions.
I
may
not
be
able
to
spend
a
lot
of
time
answering
them,
but
I'll
be
very
happy
to
answer
them
quickly.
K
K
One
can
infer
what
the
the
natural
reasons
why
these
happen.
Obviously,
people
were
going
to
new
sites
and
various
things
a
lot
more
and
also
using
as
people
were,
children
who
driven
home
students
were
driven
home.
They
were
using
online
means
of
learning
such
as
khan
academy
and
others.
A
lot
do
you
mind
going
full
screen
on
the
slides.
Please.
K
We
investigated
a
number
of
things,
I'm
going
to
limit
this
to
just
a
few
things
that
we
saw
there.
So
we
wanted
to
look
at
the
the
sheer
change
in
traffic
volume
across
regions.
We
wanted
to
see
how
that
matched
up
with
different
events
that
were
happening.
As
you
remember
in
march.
Basically,
different
countries
and
within
the
u.s
different
states
were
basically
doing
things
at
different
times.
They
would
announce
different
things.
They
were
announced
as,
as
covet
cases
increased
or
changed
in
that
particular
region.
J
K
Send
people
home
lockdowns
various
other
events
that
happened.
We
were
trying
to
see
if
you
could
see
reflections
of
those
in
our
traffic
and
so,
like
I
said
at
first,
we
want
to
see
reflections
in
traffic
volume
and,
more
importantly,
we
want
to
find
out
if
the
internet
was
holding
up.
I
don't
know
if
you
remember,
but
around
that
time
again,
a
lot
of
articles
around
you
know
is
the
internet
running,
keep
up
with
all
of
these
people
going
home
and
doing
everything
from
home.
K
K
What
what
I
decided
to
do
was
to
use
tcp
delivery
rate
from
tcp
info
that
we
record
at
our
servers
as
a
reflection
of
download
speed.
So,
even
though
the
absolute
number
may
mean
something
slightly
different,
the
change
in
the
number
means
a
lot
more
and
we
expect
to
track
effectively
what
the
tcp
connections
see
as
download
speed
to
two
users.
K
K
So
at
a
high
level
again
we
saw
traffic
volume
change
dramatically,
pretty
much
went
up
everywhere
uniformly
and
some
countries
like
italy.
This
is
between
a
week
in
feb
and
a
week
in
march,
so
one
month
period,
the
the
the
the
change
was
just
dramatic.
This
was
the
period
when
also
italy
went
into
complete
lockdown
so
again
for
for
a
good
reason,
the
traffic
went
up.
We
could
explain
the
increase
here,
but
the
download
speed
didn't
uniformly
go
down
everywhere.
K
It
did
go
down
in
some
places,
but
it
actually
went
up
in
some
places
like
in
japan,
for
example,
it
went
up
by
10,
which
is
because
my
only
answer
is
it's
japan,
but
but
but
yeah
I
mean
the
infrastructure
definitely
did
held
up
in
many
places.
So
I'll
walk
through
these
in
just
a
little
bit
more
detail.
I
don't
have
a
lot
of
time
to
walk
through
all
of
them,
so
I'll
show
you
just
a
couple
of
graphs
and
move
on
to
the
next
topic.
K
So
here's
an
example
which
is
with
the
uk
just
a
few
things
here
before
I
start
off
the
x-axis
here
is
time
and
that's
going
from
february
27th
to
march
30th
about
to
be
a
month
long
period,
the
blue
curve.
If,
if
I'm
sorry,
if
you're
not
able
to
see
the
colors
very
clearly
here,
the
blue
color
is
the
one
that's
rising
up.
It's
the
one
that
has
a
positive,
roughly
positive
slope,
not
the
exponential
one,
but
the
roughly
positive
slope
is
basically
the
the
change
in
traffic
volume.
E
K
K
K
K
As
you
can
see
here,
there's
sort
of
a
rough
mirror
here
you
could
definitely
seem
to
suffer
a
fair
bit
from
from
from
the
increase
in
traffic
volume
in
terms
of
the
infrastructure
being
able
to
support
the
increase
there.
A
couple
of
things
I'll
note
here
you
can
see
the
the
weekly
patterns,
the
bumps,
are
weekends
and
the
flat
flatter
areas
of
the
weekdays
that
sort
of
disappears
when
the
trial
restrictions
happens.
So
I'll
come
back
to
that
in
a
moment.
K
That's
the
uk.
This
is
italy.
I
won't
go
to
the
details
here
again.
We
can
walk
down
and
sort
of
see
minor,
bumps
and
some
significant
ones,
but
the
biggest
ones
we
see
are
with
school
closures
and
non-smitten
school
versions.
We
see
a
bigger
bump
happen
there
dramatically
in
time
again,
quality
of
the
internet
did
go
down
but
again
not
to
the
same
extent
as
the
increase
in
traffic
volume.
That
tells
us
something
I've
got
france
here.
K
Of
course,
in
france,
you
don't
see
as
much
of
a
decrease
in
in
delivery
rate
and
that's
a
high
level.
Now
there
are
two
more
things
here.
I
just
want
to
note.
The
netflix
and
youtube
got
announced
that
they
were
they
were
in
europe.
At
least
they
announced
that
they
would
reduce
the
default
streaming
quality
to
a
lower
value
that,
in
our
in
our
numbers,
didn't
seem
to
make
much
of
a
difference.
K
I
mean
what
we
would
expected
that
to
do
is
to
have
increased
the
delivery
rate
of
our
traffic,
which
is
not
youtube
or
your
netflix,
but
it
didn't
really
seem
to
that's
true
across
various
countries,
as
we
see
them
in
europe.
That's
probably
just
just
what
it's
worth
again.
I
note
that,
because
I
think
that
one
of
one
of
the
things
I
take
away
from
that
is
that
abr
is
working
well,
that's
all
I
take
away
from
that.
I
don't
know
that
netflix
and
youtube
had
to
do
anything
special
there.
K
I
think
it
just
seemed
to
adapt
and
work.
Well,
I
won't
go
into
the
the
main
of
the
countries
here.
Japan
is,
japan
is
a
bit
of
an
anomaly.
It
does
see
a
similar
sort
of
a
loss
of
diurnal,
not
diagonal,
but
weekly
patterns
and
traffic
traffic
volumes
do
increase
to
weekend
levels
and
just
weekend
levels
remain
sustained.
K
We
lose
completely
that
signature
of
weekend
weekday
things,
but
I
couldn't
quite
put
my
finger
on
it.
I
couldn't
find
anything
there,
but
yeah
delivery.
It
actually
went
up,
which
is
you
know,
fantastic
to
see.
I
think
the
residential
ice
piece
that
really
stepped
up.
That's
all
I've
got
within
again
within
the
u.s.
It
was
regional,
so
we
had
to
do
state-by-state
analysis
and
you
see
similar
things
happen
in
the
u.s
as
well.
So
high-level
take
away
from
this
set
of
graphs
is
that
traffic
increases
were
triggered
by
public
policy
announcements.
K
We
were
thinking
that,
maybe
you
know,
one
of
the
things
we
were
hoping
to
see.
We're
not
hoping
to
see
but
expecting
to
see
was
that
as
covid
rates
increased,
we
would
see
people
automatically
shift
this,
but
it
wasn't
so
covet
cases
didn't
really
reflect
in
traffic.
What
if
it
didn't
traffic
were
the
public
policy
announcements,
which
kind
of
makes
sense
again,
because
public
policy
announcements
are
discrete
events
in
time,
and
so
those
are
the
ones
that
we
were
able
to
see.
K
Very
distinctly
quality
degradation,
however,
was
more
gradual,
as
people
shifted
homeward
people
increase
their
activity
at
home
and
try
to
figure
out
get
their
feet
on
the
ground,
and
so
on.
We
saw
track
quality
degradation
happen.
I'll
talk
a
little
bit
more
about
that
next
and
the
line
between
weekdays
and
weekends
disappeared.
This
probably
reflects
how
what
our
experience
has
been
since
march.
One
of
my
friends,
nick
harper,
created
a
website
called
march
2020..
K
Please
go
take
a
look
at
it.
It
tells
you
what
the
current
date
is
in
march.
I
think
today
is
march
256
2020.,
that's
how
people
feel
and
that's
basically,
what
there's
just
been
this
continuity
in
time
that
that
things
have
there's.
No,
that
are
normal
patterns
of
usage
or
normal
usage
or
normal
experience
of
time,
has
changed
quite
a
bit.
K
And
yeah
the
internet
seemed
to
keep
doing
well,
mostly
thanks
to
elasticity,
and
you
can
talk
about
that
more
later
as
well.
I'm
going
to
go
back
to
this
quality
degradation
question
very
briefly,
because
we
want
to
ask
the
question
of
there
is
some
degradation
we
do
see
it.
It's
we've
been
able
to
still
serve
traffic.
Things
are
still
doing
well,
but
clearly
there
is
some
degradation.
The
question
was:
how
do
we
is
this
degradation
equal?
K
K
So
this
is
five
digit
zip
and
I
was
able
to
get
the
data
from
irs
data
in
in
bigquery
and
and
other
data
that
allowed
me
to
basically
figure
out
what
the
what
the
incomes
were.
I
created
these
income,
which
were
informed
by
a
few
things,
but
basically
splitting
up
the
us
into
these
five
zones
so
to
speak.
K
K
That
should
be,
I
think,
medium,
that's
a
that's
a
typo
there,
and
then
we
looked
at
the
delivery
rate,
the
same
tcp
delivery
rate
or
download
speed
to
those
zip
codes
to
those
aggregates
and
here's
a
high
level
picture
of
median
download
speed
within
a
zone
or
a
an
income
level
across
time.
The
same
x
axis
is
time
and
y
axis
here
shows
you
five
different
curves.
Each
one
of
them
is
the
evolution
of
the
median
download,
speed
or
delivery
rate
across
time.
K
The
one
thing
that
really
struck
us-
I
mean
I
did
some
filtering
here
to
be
clear.
Some
filtering
here
to
avoid,
like
you
know,
large
players
like
traffic
going
to
other
cdn's
or
traffic
going
to
google
cloud,
or
things
like
that,
to
avoid
those
and
remove
this
anomalies
and
filter
this
out
to
only
residential
isps
and
so
on.
Mostly
the
one
thing.
That's
that
completely
was
was
was
huge
for
us
to
see
so
starkly
was
that
delivery
rates
actually
line
up
with
income
levels.
K
Exactly
and
that's
just
to
me
was
stunning,
because
if
these
income
levels
were
not
created
based
on
our
ability
to
differentiate
delivery
rates,
we
based
them
on
completely
different
things.
And
then
we
pulled
this
data
out
and
these
five
lines,
none
almost
practically
non-intersecting
five
lines,
and
you
can
see
the
bottom
two
lines
curve.
Sorry
start
to
intersect
towards
the
end
and
I'll
talk
about
that
in
the
next
slide
in
a
moment.
But
the
fact
that
these
five
or
at
least
four
regions
sort
of
are
very
clearly
stratified,
tells
us
something
quite
important.
K
The
fact
that
high
income
level
zip
codes
do
get
for
reasons
for
reasons
of
capacity
and
for
reasons
of
capability,
higher
delivery
rates
and
download
speeds.
That
tells
us
something
here
so,
however,
what
we
were,
what
we
were
looking
to
see
was
if
the
degradation
was
was
was
distributed
unevenly
and
we
don't
see
that
we
actually
see
the
bottom
download
speed
go
up
for
the
bottom
regions,
and
that
is
actually
interesting.
K
So
we
dug
in
a
bit
more
and-
and
we
found
this
so
this
is
comcast,
and
I
want
to
call
out
these
companies
here,
because
you
know,
operators
get
a
bad
rap
all
the
time
and
they
actually
did
something
really
good.
Here,
comcast
announced
at
some
point
that
they
would
increase
the
speed
of
the
lowest
year
of
internet
access,
called
internet
essential
specifically
for
low
income
families
and
so
on.
K
They
would
increase
the
speed
and
they
made
it
freely
available
for
at
least
three
months,
and
we
saw
corresponding
to
that.
We
saw
an
increase
in
download
speed
in
those
regions,
which
is,
you
know,
just
great
to
see
in
the
data
here.
So
the
bottom
two
curves
actually
rise
up
there
in
the
second
week
of
march
and
they're,
exactly
aligned
with
the
announcements
that
they
made
and
similarly
cox
communications
also
did
something
quite
similar,
and
you
can
see
that
there
they
basically
completely
eliminated
the
gap
between
these
income
regions.
K
So
I
will
end
on
this
note
that
we
we
we
we've
managed
to
where
we
managed
to
stay
afloat
like
the
internet
has
has
has,
for
a
number
of
reasons
and
patrick
mcmanus.
A
colleague
of
mine,
has
written
a
really
good
blog
post
about
what
are
the
key
reasons
why
we
think
we
survived
the
the
volume
increase
in
the
barrage
of
traffic
that
we
saw
through
this
these
times.
But
we've
we've
done
well,
and
I
think
I
think
that
we
were
expecting
to
see
not
so
many
signatures.
K
It
is
clear
that
that
that's
work
to
be
done,
that
we
can
make
more
equitable
distribution
possible,
and
it's
it's
also
clear
that
isps
have
a
fair
amount
of
power
and
ability
to
close
this
divide,
the
fact
that
they
were
able
to
gave
increased
the
the
the
quality
meant
that
they
had
the
the
fiber
at
the
base
level
available
in
those
regions
right.
The
capacity
was
available
in
those
regions.
They
were
simply
not
opened
up
and
the
fact
that
they
were
able
to
do
this
and
and
change
qualitatively.
K
B
C
M
K
I
did
not
in
this
case
I
did
not,
although
we
I
did
do
some
digging
not
in
these
graphs
they're,
not
distinguished.
I
did
look
around
a
little
bit
to
see
mobile
networks
were.
It
was
weird
I
mean
I'll,
be
honest
because
you
know
it's
like
suddenly
traffic
over
mobile
networks
in
manhattan,
the
quality
increased,
probably
because
there
are
very
few
people
using
it.
You
know
so
we
saw
some
interesting
signatures
there,
but
nothing
significant
enough
here
to
show,
I
think
pretty
much.
K
M
K
Yeah,
I
think
one
of
the
really
nice
things
that
one
of
the
things
that
I
would
have
loved
to
have
had
is,
as
numbers
with
network
type
yeah,
instead
of
going
around
guessing
and
digging
in
because
it
doesn't,
the
problem
of
classifying
asus
does
not
scale.
I
would
love
an
automated
way
of
doing
that.
K
K
Yeah,
I
think
the
data
in
the
in
the
last
two
graphs
that
I
showed
actually
demonstrates
exactly
that
they
had
to
turn
a
knob
somewhere
and,
and
things
got
better
right,
which
is
why
I
think
that
there's
there
is
there's
capacity
sitting
there,
whether
there's
a
cost
to
using
that
capacity.
That's
not
clear
to
me.
H
Andrew
yeah,
just
picking
up
on
that
point.
First,
obviously
it
depends
what
type
of
network
you're
talking
about.
That's
absolutely
true
for
a
cable
network
or
for
an
fttp
network
might
be
a
little
more
challenging
on
an
xdsl
network,
because
a
lot
of
the
isps
on
those
networks
don't
necessarily
have
different
speed
tiers.
H
H
I
completely
agree
with
the
point
that
you
know,
then
it's
discretionary
on
the
isp
and
jason
will
confirm
offline,
whether
there's
any
costs
involved
in
that
just
a
sort
of
second
point
just
to
which
I
think
covered,
saying
which
you
know
as
either
yary
sort
of
mentioned
you.
I
think
you
both
mentioned
in
fact
about
netflix
and
others-
maybe
degrading
their
sort
of
resolution
for
a
while.
H
So
on
some
networks
they
had,
they
were
experiencing
peak
load
anyway.
So
then
getting
game
downloads
on
the
same
evening
was
causing
them.
So
some
some
congestion
challenges
and
I
think
there
was
some
discussion
with
the
gaming
industry
to
try
and
have
a
better
coordination
between
isps
and
the
gaming
industry
to
avoid
stressing
networks
unnecessarily,
just
by
picking
different
times
or
different
days,
to
put
out
their
their
their
latest
versions
of
of
the
call
of
duty
or
whatever
don't
mention
that.
As
I
don't
think,
that's
come
up.
H
B
A
A
You
know
you
know,
though
you
know
the
consoles
basically
have
foreground
and
background
download
method.
So
it
really
depends
on
how
the
game
publisher,
you
know,
releases,
you
know,
releases
it
and
the
timing
of
that
and
that's
something
which
has
a
cdn.
You
actually
don't
have
a
lot
of
control
over,
because
you
know
they
release
it
whenever
they
release
it
and-
and
that
makes
it
incredibly
challenging
what
what
happened
earlier.
You
know.
A
A
To
do
many
of
the
down.
You
know
to
do
many
of
the
download
events
and
started
to
you,
know
nudge,
these
folks
to
release
thing
or
to
post
them
more
off
hours
such
that
the
load
was
shipped
basically
to
nighttime
hours.
When
most
people
were,
you
know,
were
sleeping
at
least
for
those.
So
so
that's
you
know.
That's
a
big
thing.
A
It's
you
know,
there's
always
new
gaming
event
downloads,
I'm
expecting
there
to
be
many
during
the
holiday
season,
especially
as
people
get
new
consoles
and
the
ps4
playstation
5
comes
out
as
well.
So
so
that's
gonna,
you
know!
That's
gonna
be
really
interesting
and
then
the
second
thing
is
because
of
the
failure
of
broadband
in
my
area,
so
I'll
switch
hats.
I
built
a
little
fiber
to
the
home
network
to
kind
of
service
myself
and
other
folks
around
here,
because
people
like
comcast
didn't
build
into
the
area.
A
Despite
having
infrastructure,
close
and
att,
you
know
att
offers
an
amazing
1.5
make
dsl
to
my
home.
You
know
from
their
service
system
so
so,
given
that
it's
been
really
interesting
for
me
to
observe
as
people
who
get
access
to
high
good
quality
internet
connectivity
is,
you
know
where
I'm
I'm
now
delivering
high
speeds
to
them
to
watch
that
traffic?
A
A
K
We
did
thanks
for
that
currently
mind
if
I
jump
in
here
or
what
yeah
yeah
so
so
yeah,
we
definitely
have
to
do
a
lot
of
dancing
around
those
things,
so
somebody
asked
the
question
of:
why
are
gaming
downloads
different
gaming
downloads
are
different
in
only
that
they
exercise
one
particular
muscle
of
a
cdn
extensively,
so
it's
basically
large
file
downloads.
It's
like
a
dos
attack.
Everybody
arrives
at
the
same
time
and
downloads
this
giant
file,
so
cdn's
traffic
is
not
like
that.
K
Normally,
you
know
it's
all
file
downloads
and
it's
all
you
know
web
pages,
it's
usually
short
connections,
because
web
pages
are
smaller
pages
and
so
on,
and
this
is
not
like
streaming
either,
because
the
client
isn't
adapting
condition.
Control
is
shining
here
to
the
extent
that
it
is
able
to
and
there's
a
lot
of
failures
there.
So
we
do
see
a
lot
of
basically
that
sort
of
play
out.
So
it's
a
very
particular
muscle
that
gets
exercised
here
and
you
have
to
prepare
for
it
differently.
K
B
So
I
want
to
jump
in
on
the
comment
about
the
about
the
the
people,
the
network's,
seeing
no
problem
so
a
lot
of
when
we
were.
We
were
seeing
a
lot
of
problems
in
a
lot
of
networks
before
youtube
and
netflix
drop
stuff,
and
by
problems
we
mean
we're
measuring
on
time
when
the
user
experience
of
the
video
and
audio
was
impaired
in
some
significant
user.
You
know
significant
way.
Like
you
know,
multiple
seconds,
you
know
it
per.
B
You
know
more
than
a
second
per
minute,
where
you're,
where
there's
like
significant
loss
or
quality
artifacts,
and
all
the
major
video
conferencing
providers
substantially
reduced
their
bandwidth
to
try
and
fight
against
that,
and
it
just
wasn't
enough.
It
was
getting
to
the
point
where
people
were
talking
about
turning
off
video
altogether,
which
would
have
really
impact
on
remote
learning.
Now,
when
we
went
and
talked
to,
the
network
operators
they're,
like
was
often
on
many
cases,
was
no
we're
not
having
any
problems.
B
Our
network
they're,
looking
pretty
good
we're
like
it's
looking
pretty
bad
to
us
and
they're
like
well.
Look
at
our
throughput
rates,
they're,
looking
fine
to
us
and
we're
like
well
look
at
what
happens
to
any
real-time
media
like
let's
set
up
a
voip
call,
and
it's
like
oh
yeah.
That
does
look
pretty
bad,
so
I
think
there
was
some
questions
of
measurement
there.
I
got
to
say
that
every
network
operator
we
contacted
or
or
talked
to
was
amazingly
awesome
about
working
with
us
trying
to
fix
the
problem.
B
Doing
everything
like
everyone
just
ignored
the
money
ignored
the
politics
just
was
like
yeah.
We
need
to
make
this
work
for
users.
What
can
we
do
to
make
things
better?
So
it
was
amazing
from
that
point
of
view,
and
I
think
a
lot
of
that
came
from
relationships
out
of
nanog
or
itf
or
other
things
where
you
could
directly
figure
out
a
technical
person
to
call
inside
of
a
company,
and
they
took
the
effort
to
get
you
to
the
right
person
who
could
solve
your
problem.
B
So
that
was
amazing,
but
I
I
think
we
do
need
to
it
made
me
think
that
you
know
we
need
better
ways
to
understand
for
the
network
operators
to
measure
their
impact
on
different
applications,
not
just
on
the
the
the
download
throughput,
and
I
think
a
lot
of
people
did
try
and
figure
out
what
they
needed
to
do
to
to
keep
app
keep
the
the
remote
learning
and
the
web
conferencing
applications
up.
So
some
of
the
traffic
did
split
separately.
K
If
I
could
jump
in
there
and
thank
you
for
that
comment,
I
think
that's
actually
I
don't
I
mean
I.
I
didn't
mean
to
suggest
that
our
viewport
is
limited
and
I
totally
appreciate
your
point
there.
We
don't,
we
don't
have
any
metrics
on
real-time
traffic
and
we
certainly
didn't
have
client-side
metrics
in
here.
So
we
couldn't
really
tell
what
was
happening
on
the
client
but
yeah.
That
point
is
actually
it's
a
very
good
point
for
the
longest
time,
at
least
with
the
operators.
K
Oftentimes
throughput
is
used
as
a
measurement
of
how
well
youtube,
for
example,
is
drawing-
and
that's
not
good
enough
and
I've
seen
this
time
and
again
with
operators
looking
at
what
I
think
of
as
micro,
metrics
or
micro
micro
benchmarks,
but
they
are
micrometrics.
Throughput
is
a
micrometric.
Your
hope
is
that
you're,
using
that
to
reflect
something
that
is
you're,
hoping
that
that
reflects
quality
of
experience
at
the
the
user.
But
in
a
lot
of
cases
it
doesn't.
B
And
look,
I
may
be
a
great
thing
if
we
came
out
of
this
going
like
the
hard
things
are.
You
know
classic
web
page
usage,
our
games
download
our
you
know,
video
conferencing
like
we
came
up
with
a
list
of
things
and
figured
out
different
types
of
metrics
for
them.
So
anyway,
I
want
to
stop.
I
want
to
power
hand
over
to
an
anon
here.
K
I
did
look
at
rdt
and-
and
I
I'd
love
to
look
at
your
data
at
a
high
level,
though,
to
your
point
of
what
can
we
do?
I
don't
know
that
we
can
really
condition.
Control
is
evolving
ultimately,
and
we
know
that
we
want
to.
We
want
to
reduce
these
variations
in
round
of
time,
but
at
times
of
times
of
deep
congestion
we
are
likely
to,
we
are
going
to
see
those
sorts
of
variations,
whether
our
buffers,
we
will
see
them
and
there
are
d
buffers.
N
Right
and
in
such
cases,
for
example
like
just
using
bbr
v1-
probably
not
the
best
approach
right
things
like
that,
and
I
think
it
might
be
hard
to
like
share
the
data
but
having
like
the
community
kind
of
like
guidelines
and
a
little
bit
thought
process
around,
just
like
how
video
providers
would
be
able
to
jump
in
and
reduce
their
bit
rates.
Arguably,
that
may
or
may
not
have
that
much,
because
abr
was
working
pretty
well.
B
B
L
K
B
K
P
P
But
I
was
also
wondering
if
the
same
analysis
would
be
interesting
for
the
uk
on
on
the
breakdown
of
revenues
for
different
people
in
different
regions,
because
we
actually
did
look
at
you
know
the
census
based
information
that
we
got.
I
mean
this
was
okay.
This
was
the
census
in
like
2011,
because
the
next
one
is
supposed
to
happen
next
year,
but
we
did
see
quite
interesting
dynamics
there.
So
a
lot
of
the
people
actually
moved
from
london.
They
went
to
different.
P
You
know,
to
areas
where,
like
marked,
you
know
suburban
or
areas
that
were
marked.
You
know
rural
or
actually
low
income,
what
it's
actually
a
group
where
it's
their
car,
it's
called
challenged.
I
don't
really
remember
the
actual
name
right
now,
but
yeah.
I
think
it
will
be
very
interesting
to
to
contrast
this
and
to
look
at
you
know
the
analysis
from
the
point
of
view
of
the
actual
operator,
with
the
analysis.
K
K
K
Like
I
said
for
the
u.s,
I
used
irs
data
that's
available
in
bigquery,
and
there
was
a
lot
easier
for
me
to,
even
though
it
was.
It's
actually
super
interesting.
If
you
ever
get
a
chance
to
look
at
the
data,
they've
anonymized
it
in
a
very
interesting
way,
but
but
I
couldn't
find
similar
data
even
in
terms
of
breakdown
of
a
down
to
smaller
enough
levels.
But
if
that
that
I
I
was
interested,
certainly
at
the
time.
K
I've
not
spent
a
lot
of
time
on
this
since
since
the
summer,
but
I
think
that'll
be
very
interesting
in
general,
I
think
there
is
like
I
said
there
is
a
story
here
right,
there's
a
separate
story
here
that
came
out
of
it,
for
me,
at
least,
is
that
there
is
a
correlation
between
quality
of
internet
of
the
bandwidth
at
a
high
level.
I
wouldn't
say
quality
generally,
but
certainly
bandwidth
and
and
and
economic
zones,
so
to
speak
right.
P
P
If
I
could
just
add
something,
because
we
actually
did
look
within
the
uk,
you
know
these
output
areas.
This
is
public
information
and
they
actually
tag
these
at
the
postcode
level
and
the
postcodes
you
know
in
the
uk
are
like
a
few
homes,
a
few
streets
very,
very
granular
information,
and
we
actually
did
check
whether
there
was
any
bias
within
the
way
that
also
deployed
their
network
and
we
couldn't
find
any
differences.
P
Actually,
we
couldn't
identify
any
new
hotspots
appearing
after
the
lockdown
immunity
after
the
lockdown,
but
I
think
it
was
a
very
interesting
point
that
was
made
before
you
know
whether
we
could
actually
contrast
this
with
user
experience.
And
this
is
something
that
we've
been
looking
at
and
we
were
trying
to
actually
understand
whether
we
can
compare
the
metrics
that
we
collect
from
from
the
network
side
with
the
actual
user
experience
and
create
basically
a
similarity
matrix.
That
is
able
to
tell
us
how
our
how
our
users
are
actually
experiencing
the
service.
K
Yeah
I'd
definitely
be
interested
in
looking
at
this
more
if
there
was
more
data.
I
honestly
I
didn't
dig
in
and
that's
all
I've
there,
but
I
think
that's
definitely
interesting
stuff
to
look
into
here,
especially
for
operators.
I
would
say
that
at
a
high
level,
the
the
things
that
one
one
should
be
careful
to
look
at
is
is
you
want
to
look
at
some
notion
of
per
capita?
I
think
that's
the
important
part
here.
D
Yeah
I
wanted
to
get
back
to
your
question
colin
or
observation
of
what
was
key
about
this
network's
feeling
that
things
are
fine
and
then
confronting
application.
People
are
feeling
that
now
this
is
not
working
very
well
and
that
that's
very
interesting.
Of
course,
I
think
we
have
some
sort
of
fundamental
issues
with
our
attempt
to
measure
things
actually,
because
you
typically
can
measure
only
usage
but
not
demand.
D
Of
course,
in
some
cases
you
can
see
that
well,
you
could
send
more
packets
clearly
into
this
unconsistent
network,
but
but,
for
instance,
like
people
who
aren't
connected,
obviously
aren't
going
to
show
up
as
users
of
packets
or
their
rtt
can't
be
necessary
and
so
on.
So
that
that's
an
issue,
but
maybe
that's
a
flip.
D
A
flip
comment
in
some
sense,
but
I
actually
wanted
to
ask
a
more
practical
question
for
you
colin,
that
when
you
work
with
these
conferencing
applications,
how
much
do
you
know
and
how
much
do
you
get
to
see
of
the
sort
of
the
big
picture
of
like
you
know,
problems
could
appear
in
many
places
and,
like
you
know,
my
connection
here
is
it
you
know?
Is
it
my
wi-fi?
Is
it
my
isp
access
network
connection?
Is
it
further
up
in
the
ixp?
D
B
You
have
full
pizza;
no,
no,
we
don't
have
the
tools
we
need
at
all.
It's
a
great
question.
I
think
this
is
a
great
lead
into
the
sort
of
next
session
of
what
we
would
making
it
better.
One
thing
I
will
and
look
I
I'm
speaking
from
point
of
view
of
webex,
but
I
I'm
pretty
confident
that
I
I
have
friends
that
working
at
the
other
major
conferencing
vendors
to
understand
they
have
very
similar
problems.
B
First
of
all,
it's
very
difficult
for
us
to
child
whether
something
went
wrong
on
the
wi-fi
network
or
after
that,
okay
and
being
able
to
get
almost
the
most
simple
metrics
from
the
from
the
access
point
you
were
dealing
with,
would
totally
change
our
ability
to
guess
where
the
problem
was
whether
it
was
wi-fi
or
or
on
the
network
beyond
that
for
the
network.
Beyond
that,
you
asked
about
sort
of
what
level
we're
tracking,
probably
a
lot
of
people
on
this
call,
have
used
thousand
eyes
or
a
product
like
that.
B
That
is
taking
tons
of
metric
points.
You
know
it's
sending
tons
of
probes
from
a
zillion
different
points,
we're
doing
it
from
you
know
from
all
the
webex
endpoints
retract
we
we
probe
multi.
If
you
look
at
the
webex
traffic
or
any
of
the
major
vendors,
they
probe
multiple
swans
of
their
data
centers
and
track
what
type
of
stats
they
have
and
then,
of
course,
we
correlate
those
and
try
and
backflow
through
the
network
to
understand.
Oh
look,
everybody
behind
this
particular
router
or
your
bgp
control
point
or
something
like
that.
We
we
can.
B
We
can
see
that
they're
having
problems
and
we
will
often
be
able
to
see
where
network
data
is,
you
know,
is
being
flopped
on
two
different
routes
or
all
the
different
types
of
traffic
routing
things
that
you
might
see.
So
we
can
correlate
that
we
can
bring
it
back
to
for
this
particular
service
provider.
We're
seeing
everybody
in
this
sort
of
network
block
is
having
a
similar
problem.
We
might
we
can
often
we
can
often
guess
how
a
you
know.
B
B
Often
we
can't
even
figure
out
who
we
would
contact
at
those
companies
to
figure
it
out,
and
that's
one
of
the
things
that
changed
during
coped
was
our
ability
to
go
like
we're,
seeing
a
problem
in
your
network
if
I've
done
that,
if
I've
seen
that
we're
having
a
real
problem
in
this
space
here,
it's
causing
issues
for
customers
a
year
ago.
If
I'd
had
that
problem,
they
would
have
been
like
yeah
log
a
ticket,
and
now
it's
like.
Okay.
Let
me
look
into
that
for
you.
B
It
was
like
a
real
real
shift
there,
so
anything
that
we
could
do
to
get
a
little
bit
more
visibility
into,
particularly
whether
the
problem
was
wi-fi
or
after
the
wi-fi
would
help,
because
no
one
has
any
the
applications.
Don't
have
any
clue
on
that.
Maybe
the
os
vendors
do
and
then
be
able
to
actually
talk
a
little
bit
more
realistically
about
the
network
measurements.
B
I
mean
we
were
thinking
about
the
things
we
do,
make
it
difficult
for
the
service
providers
to
actually
understand
whether
our
applications
are
working
well
or
not,
and
multiple
of
the
video
conferencing
providers
right
now
claim
they
can
do
fairly
high
resolution
video
conferencing
with
50
packet
loss.
I
mean
that
should
freak
us
all
out.
I
don't
think
it's
exactly
true.
Luckily,
but
the
fact
that
they're
claiming
this
is
really
disturbing
you
know
and-
and
everybody
is-
is
super
aggressive
about
the
amount
of
forward
error
correction
things
to
do.
B
B
K
O
That
so,
if
I
can,
is
that
I
mean
that
surprised
me
for
a
second.
When
I
heard
it
and
then
I
then
it
unsurprised
me
just
because,
like
the
the
tolerance
of
audio
failure
is
way
less
than
the
tolerance
of
video
failure
for
for
good
qoe
right
like
so,
if
you're
pixelated,
who
cares.
But
you
know,
if
you
said,
launch
the
missiles,
then
that's
a
problem.
So
the.
O
This
actually
lines
up
with
some
some
qoe.
You
know
issues
I've
had
on
various
conflicts
and
platforms,
not
yours
with
respect
to
like
actually
keeping
the
latency
in
sync
right,
like
so.
I've
seen
the
the
big
thing
that
I
ran
into
on
one
unnamed
provider.
As
soon
as
like
covid
went
bad
was
I
got
up
to
like
half
a
second
of
dsync
from
the
video
in
the
audio
and
I'm
guessing.
That
was
so.
Thank
you
for
explaining
what
caused
that
to
happen.
B
Yeah
I
mean
everyone's
gone
to
that,
and
and
look
I'm
not
arguing,
that's
a
good
plan
that
we're
doing
that
either.
I
disagree
with
it
in
many
ways,
but
when
when
people
are
like
well,
what,
when
you
have
a
set
of
engineers,
that
are
let's
say
engineers
are
not
deeply
involved
in
network
control
who
are
given
the
problem,
make
this
work
with
50
packet
loss,
guess
what
they
would
do,
yeah
that's
what
they
did.
J
K
And
now
the
challenge
was:
how
do
you
make
it
work
at
70
percent
loss
and
there
we
go
chasing
our
tail
again,
but,
but
I
I'm
surprised
just
very
quickly
not
to
I'm.
I'm
I'm
not
surprised
that
the
audio
stream
is
more
important,
obviously
yeah.
That
makes
a
ton
of
sense
that
that's
what
you
want
to
do.
What
I'm
surprised
still
is
that
the
bandwidth,
I
would
have
imagined
that
the
the
audio
stream
bandwidth
are
substantially
lower
than
the
video
stream.
K
B
Right
so
I
I'll
get
someone
else
on
the
queue
so
I'll
talk
a
little
I'll,
just
say
a
little
bit.
The
type
of
audio
quality
we're
having
right
now,
just
the
raw
audio
quality
is
about
30
kilobits
per
second
for
opus.
That's
about
what
the
codec
would
take
at
that
level.
This
level
of
video
that
you're
seeing
right
now
is
probably
around
for
each
one
of
these
thumbnails
around.
Maybe
I
don't
know
you
could
probably.
B
J
B
Turn
off
your
video,
it
improves
things,
usually
isn't
really
all
that
true,
because
most
the
services
they're
they're
transcoding
in
the
cloud
and
if,
if
one
person
on
this
calls
video
is
a
network,
is
really
crappy,
we'll
just
slow
down
the
video
to
that
person,
not
everyone.
So
it's
it's
video
on
or
off
makes
less
different
than
you
might
think.
A
Yeah,
I'm
yeah.
I've
been
watching
the
you
know.
If
you're
running
on
a
machine
you
go
to
like
you,
go
to
help
health
checker
and
see
like
the
webex
stats,
and
you
can
see
it's,
you
know,
ranging
anywhere
from
600
kilobits
to
900
kilobits.
It's
you
know
it's,
it's
pretty
good,
although
I'm
I'm
bursting
up
to
sending
a
megabit
and
a
half
video
right
now,.
B
You
what
I
mean
this
gets
really
questions
of
like
look
we're
going
to
run
out
of
time
here
soon.
You
know
what
are
some
of
the
big
issues
that
came
up
overall,
not
just
these
video
conferencing
programs.
I'm
happy
to
talk
about
my
problems
all
day,
and
you
know
what
are
some
things
that
we
could
do
to
improve
stuff.
I
mean,
let's,
you
know,
jump
on
the
queue
and
give
us
your
ten.
You
know,
let's
hear
from
some
people
we
haven't
heard
from
here.
E
Video
session
is
a
significant
number
of
other
other
other
users,
and
you
need
to
remember
that
companies
like
google
and
netflix.
We
also
have
other
our
own
users
doing
other
things
and
we
don't
want
to
interfere
with
our
own
traffic,
so
it
makes
perfect
sense.
The
other
cases
google
has
turned
down
youtube.
For
instance,
when
there
has
been
earthquakes,
separate
multiple
transoceany
cables
and
just
turn
it
off
is
it
makes
space.
The
other
thing
I
wanted
to
observe
is:
there's
a
there's,
a
wonderful
story.
E
E
B
Q
Q
So
a
bit
more
isolated
or
a
bit
more
eager
for
the
new,
so
just
kind
of
a
difference
really
in
in
terms
of
how
vulnerable
users
were
to
manipulative
techniques
like
scams,
fraud
and
that
kind
of
stuff.
So
I
suppose,
for
us,
the
network
impacts,
you
know.
Generally,
there
was
a
real.
We
talk
about
the
the
capacity
builds
and
their
speeds
and
the
volumes
and
the
internet's
ability
to
cope
with
that.
Q
I
think
we
just
saw
a
huge
spike
in
fraud
and
scams
related
to
covers
and
a
whole
new
set
of
attacker
behavior
to
combat,
including
new
targets
like
vaccine
laboratories
and
stuff
to
protect
and
as
well
as
this
like
big
shift
to
home,
working
and
vpn
and
spin-ups
and
stuff
like
this.
So
I
just,
I
suppose,
the
network
impact
like
from
for
covert
from
our
point
of
view,
was
just
really
this
huge
new
security
effort
that
was
needed
and
the
security
industry
is
always
quite
agile.
Q
We're
always
sharing
and
kind
of
thinking
about
the
greater
good,
and
you
know,
sharing
any
attack
techniques
that
we
find
and
ways
that
people
can
defend
through
indicates
compromises
are
common
techniques,
one
thing
and
that
the
ncsc
did
specifically
was
to
start
launch
our
suspicious
email
reporting
service.
There.
Q
A
lot
of
people
were
getting
an
initial
phish
through
email,
and
it
would
be
something
like
you
know:
tax
rebates
for
the
covet
or
whatever,
or
you
know,
all
those
sort
of
government
schemes
that
are
going
on
and
then
recognizing
the
way
that
users
were
more
vulnerable
and
generally
and
some
may
be
more
likely
to
click
than
in
a
normal
non-pandemic
time
and
this
service.
The
idea
is
that
people
can
report
it.
Q
So
more
advanced
users
can
report
it,
and
then
those
sites
can
be
taken
down
where
they
are
malicious,
so
that
it's
protecting
kind
of
the
herd.
It's
like
this
herd,
immunity,
kind
of
thing
and
so
yeah,
so
that
one
initiative
that
we
we
started,
but
that's
obviously
quite
a
low
level
manual
intervention,
and
so
I
suppose,
just
as
an
open
question
like
would
would
that
work
as
well.
Q
If,
if
there
was
another
kind
of
pandemic
or
another
kind
of
situation
like
like
we're
experiencing
now
as
people
shift
to
home
working,
you
know,
were
there
lessons
that
we
could
have
taken
from
the
security
impact.
I
know
that
a
lot
of
people
kind
of
rolled
out
their
their
enterprise
infrastructure
quickly
and
yeah.
It
was
just
just
a
whole
new
set
of
things
coming
out
that
maybe
we
could
just
look
at
a
bit
from
a
security
point
of
view,
it's
kind
of
what
I
wanted
to
start
a
discussion
on.
B
C
Q
Q
So
we
saw
a
spike
for
government
domains
being
used
like
yeah,
particularly
for
fraud
using
covid
as
a
cover,
and
part
of
that,
I
think,
is
that
we
already
have
taken
down
quite
a
lot
of
existing
fraud
sites
that
make
use
of
kind
of
government.
Brands
comes
under
copyright,
actually,
which
is
quite
amazing.
But
yes,
you
know
I
don't.
Q
Q
R
I
I
am,
I
was
going
to
ask
or
raise
a
different
topic
about
gaming,
but
just
to
go
back
to
kirsty's
point.
One
question
I
have
is
whether
there's
any
way
or
any
help
of
trying
to
solve
some
of
the
security
things
at
the
dns
level.
Let's
do,
for
example,
like
open,
dns,
now
cisco
umbrella,
where
it's
able
to
through
seeing
a
lot
of
the
dns
queries
spot,
potentially
new
domains
that
are
coming
on.
R
R
Attacks
and
things
effectively
to
be
able
to
do
some
sort
of
network
level
thing
where
you
get
that
same
sort
of
protection
through
various
people
saying
and
sharing
their
knowledge
that
this
is.
This
is
dubious
and
the
same
same
idea,
I
guess
with
gmail,
I
believe,
where
they
have
ability
to
filter
emails,
because
they
see
a
lot
of
the
spam
being
sent
to
lots
of
people
and
hence
they
have
extra
knowledge
about.
R
R
Q
Yeah
that
we
have
the
protective
dns
service
in
the
ncsc,
which
is
a
dns
level
filtering
system
that
covers
all
government
domains
so
where
we
do
have
indicators
of
compromise,
like
I
don't
know
like
fraudulentcoven.com
or
whatever,
and
we
can
just
deploy
that
and
it
it
protects
the
fleet
of
enterprise
devices
for
government.
So
we
found
dns
filtering
like
that
to
be
a
very,
very
powerful
tool.
It's
not
like
perfect,
and
it's
not
delicate,
it's
quite
coarse,
but
it
it
does
really
work
for
us.
B
B
I
I
As
everyone
knows
at
the
very
beginning
of
the
co-ed,
when
we
moved
to
online
teaching,
there
was
a
huge
surge
about
the
zoom
bombing.
You
know
this,
whatever
people
attacked
the
zone,
but
that
was
during
our
spring
quarter
teaching
so
the
one
when
we
started
the
fall
quarter
back
in
august
no
no
october.
I
Just
when
I
was
writing
the
submission
of
white
paper
to
this
workshop
on
that
same
day,
the
original
deadline,
we
got
another
pledge
from
the
campus
warning
us
that
there
is
a
huge
zoom
bombing
again
coming
in
so
at
this
time.
I
think
I
didn't
hear
from
others
as
a
serious
problem,
and
there
was
a
problem
at
the
ucla,
and
this
is
kind
of
a
anecdotal
plus
speculation.
I
Ucla.
As
you
know,
bigger
campus,
it's
kind
of
different
things
from
foreign
languages
to
the
performing
arts,
I'm
pretty
sure,
there's
some
faculties
that
are
not
really
know
the
the
use
of
safe
use
of
networking
that
well
so
that
for
those
people
they
might
not
have
good
precautions
or
practice
best
practices
handling
their
issues.
The
online
life,
but
previously
whatever
they
do,
is
only
affected
themselves,
but
now
everyone
actually
performing
their
job
online
and
their
lack
of
understanding
of
security
brought
up
the
big
issues.
I
I
That
is
my
first
comment.
There's
a
second
comment
I
just
want
to
mention
it.
Zoom
is
not
strange
to
us.
You
know
I've
been
using
zoom
for
online
master
program
for
six
years
or
five
years,
some
long
time
and
nobody
cared
about
attacking
it
until
it
became
the
first
line
of
tools
for
home
networking
remember
for
homeworking.
B
So
I
I
think,
that's
like
a
great
set,
I'm
going
to
put
myself
on
the
queue
here,
and
I
think
this
is
a
great
segue
in
time
to
move
into
the
sort
of
you
know
what
things
that
we
could
do
better.
So
I
mean
some
of
this
is:
is
users
changing
no
question
about
it,
learning
new
tools
and
new
things
adapting
to
things,
but
I
think
some
of
it
too,
like
to
answer
your
question
about
passwords
there
I
mean
you
know
from
a
technical
point
of
view.
B
One
of
the
things
that
we
could
do
is
is
build.
Our
app
is
figure
out
how
to
make
it
possible
to
not
have
passwords
and
look.
I
understand,
there's
a
lot
of
work
going
on
in
passwordless
right
now
so,
but
I
mean
I,
I
think
that
that's
the
things
the
type
of
things
I'd
like
to
talk
over
about
for
the
next
15-20
minutes
or
so
is
what
are.
K
K
We
all
have
our
own
little
lenses
through
which
to
see
the
internet,
but
we
don't
have
a
good
understanding
of
what
the
overall
thing
from
a
user's
perspective
is
we
all
have
again
only
those
who
run
client
applications
have
that,
but
they
don't
know
what's
happening
in
the
network.
For
instance,
this
has
always
been
true,
and
this
has
always
been
true,
because
when
people
run
experiments,
for
example,
if
an
operator
runs
an
experiment,
they
look
at
their
own
little
benchmarks
to
figure
out
whether
the
experiment
is
changing
things
or
not.
K
All
the
while
the
metrics
they
are
using
may
not
or
may
have
inverse
correlations
with
quality
of
service
from
the
quality
of
experience
of
the
the
the
user.
We've
seen
this
in
the
past.
This
particularly
has
been
true
with
with
middle
boxes
that
that
that
terminate
tcp
connections.
That's
just
one
example
that
often
times
are
bought
and
sold
because
they
seemingly
improve
usage
of
network
at
an
operator.
K
At
the
same
time,
I'm
familiar
with
some
experiments
that
actually
tried
that
demonstrated
that
that
video
streaming
quality
of
experience
increased
by
over
10
percent
when
they
bypassed
some
of
these
boxes
that
an
operator
had
deployed.
So
there's
there's,
there's
a
huge
there's,
a
huge
sort
of
sort
of
a
mismatch,
right
operators
and
and
endpoints
clients
servers.
There
are
at
least
three,
maybe
more
parties
that
have
different
viewports
into
this
world.
They
have
different
incentives.
K
Of
course,
I
don't
know
how
we
can
align
all
the
incentives
together,
but
at
a
minimum,
if
you
can
get
observability,
I
don't
know
how
to
do
how
to
solve
this
problem.
It's
and
it's
not
a
small
problem.
I
I,
I
don't
think
it's
a
small
problem
anyways,
but
if
there's
a
way
for
us
to
understand
how
end-to-end
things
are
affected
in
these
days
in
particular,
especially
through
code,
it
have
been
super
useful,
but
that's
something
we're
missing
we're
missing,
except
for
when
people
like
us,
some
of
us
get
together
and
sort
of
share.
K
I
That
is,
I
don't
see
a
clear
framework
on
how
this
is
internet
security
actually
is
a
coherent
system.
I
think.
Currently
we
have
peace,
meals,
there's
this
okay
security,
china,
but
then
secure
the
china
lead
to
this
unscalability,
because
no
one
can
we
cannot
allow
everyone
to
connect
to
the
servers.
So
that's
how
fastly
comes
into
fitness.
You
know
providing
middle
boxes
if
we
go
down
continuously
along
the
current
path.
I
O
Brian
me
stop
typing
for
a
minute,
so
I
would
endorse
jonah's
point:
it's.
You
know
a
super
difficult
problem
and
one
that
I
hope
to
actually
come
back
to
work
on
at
some
point
is
how
can
you
balance,
like
you
know,
the
end-to-end
observability,
where
end
and
end
are
basically
sort
of
semantic
things
on
either
side
of
the
network
that
the
network
is
going
to
affect
with
you
know,
privacy
and
security
right
because,
like
the
easiest
way
to
do
this
will
be
like
everybody
gets.
O
I
mean
you
open
everything
up
right
like
so,
tcp
was
nice
and
observable,
but
you
can't
build
a
you
can't
build
a
network,
that's
not
going
to
be
interfered
with
on
top
of
pcb,
as
we
found
from
the
middle
box
question
the
other
thing
I
wanted
to
go
back
to.
I
think
it
was
andrew's
question
at
the
very
beginning
of
today,
because
I
had
a
I
thought
about
this.
O
As
I
was
looking
at
some
of
the
data
and
looking
at
the
data
from
from
yesterday
is,
we
should
be
very
careful
about
how
we
measure
the
efficiency
of
the
network,
because
I
think
you
know
there's
a
there's
sort
of
rules
of
thumb
in
in
provisioning
the
backbone
that
says
well
you're
going
to
have
30
percent
you're
in
your
growth.
You
need
to
have
30
slack,
and
that
was
the
slack
that
kept
us
from
dying
in
those
days
when
everyone
was
asking.
O
O
These
overheads
in
the
network,
because
we'll
just
be
able
to
run
things
harder,
like
you
know,
taking
you
know,
taking
experience
from
data
center
networks
which
run
significantly
harder
than
the
public
internet
does
and
it
it.
You
know
we
can't
actually
say
how
much
of
a
safety
safety
margin
we
need
in
order
to
be
able
to
pull
off
something
like
oh
another
coven,
but
observability
would
get
us
the
numbers
where
we
know
like
what
the
difference
is
between
safe
efficiency
and
unsafe
efficiency.
O
Now
I
don't,
I
don't
see,
like
you
know,
even
as
the
the
co-chair
of
the
pathway
networking
research
group,
I
don't
actually
see
sort
of
these
technologies
being
deployed
the
internet
at
some
point
where
the
efficiency
would
significantly
threaten
sort
of
the
resiliency
of
the
network.
But
it's
something
to
keep
in
mind.
J
S
Sure,
thanks
so
a
few
comments,
and
maybe
some
areas
to
mark
for
improvement.
You
know
one
I
think,
as
maybe
it
was
stephen
joking
at
the
start.
You
know
if
this
had
happened
a
decade
ago
we
might
have
been
fairly
screwed,
but
you
know
this
was
a
remarkable
success.
The
fact
that
you
know
this
huge
shift
occurred
and
the
internet
basically
worked
and
supported
all
this
video
traffic
is
pretty
awesome.
S
So
I
think
that's
a
a
big
success
that
you
know
wasn't
mirrored
in
other
areas
of
the
economy,
like
you
know,
couldn't
get
paper
towels
for
three
months
or
something
like
this.
So
it's
it's
a
big
difference
in
terms
of
improvement,
though
I
want
to
echo
some
comments,
others
made
about
measurement
and
observability
on
some
of
all.
I
think
we
get
what
we
measure
it's
sort
of
like
you
know
what,
if
you
give
a
sales
person,
a
commission
for
selling
something
or
they're
going
to
sell
more
of
it
right.
S
So
if
you
measure
something
you're
going
to
get
people
to
focus
on
it
and
work
to
improve
it,
so
I
think,
having
some
new
measurements
here,
beginning
to
shift
focus
away
from
purely
sort
of
capacity
measures
is
really
important.
So
I
think
that's
one
area
that
we
can
improve
and
perhaps
building
in,
as
others
said
here,
observability
inherently
in
some
of
the
protocols
or
or
even
the
end-to-end
applications.
S
S
I
think
it
also
points
out,
maybe
going
to
john
the
slides.
You
know
that
there's
a
bit
of
a
gap
here
in
terms
of
policy
makers
and
regulators,
understanding
of
how
the
network
actually
works,
I'm
not
clear
that
they
really
understood
the
rise
of
adaptive,
bitrate
sorts
of
things
and
the
fact
that
you
know
maybe
their
their
actions
would
literally
have
no
effect.
S
You
know
they
were
thinking
of
a
network
in
a
different
era,
but
I
also
wonder-
and
I
think
this
is
maybe
understudied-
I
think-
maybe
more
sort
of
economic
analysis
has
to
occur
to
see.
You
know
why
wasn't
there
additional
capacity
freed
up
in
those
markets?
Is
there
a
structural
difference
in
the
way
that
capacity
investments
are
made
or
incented?
S
So
some
of
that
you
know,
is
probably
something
for
maybe
economists
to
study
and
then
the
last
comment
is.
I
do
think
that
they're
probably
calling
to
your
point
about
forward
error
correction,
some
what
might
seem
very
nuanced
points
of
feedback
that
need
to
occur,
but
but
very
important
ones.
So
I
feel
like
we're
missing
a
feedback
loop.
You
know
if
we,
if
you
thought
of
like
we're
all
one
global
devops
team
right.
We
still
have
this
split
between
some
architects,
someplace
and
people
operating,
and
it
seems
like
there's
this
feedback
loop.
S
That's
missing
back
to
some
of
the
the
app
developers
that
are
maybe
overusing
things,
and
it
reminds
me
a
little
bit
of
a
few
years
ago,
when
you
know
maybe
chip
said
vendors
were
building
in
huge
buffers
and
we
were
like
no.
Why
are
you
doing
that?
Like
stop
giving
me
three
second
or
five?
Second
buffers,
like
that's
counterproductive,
so
I
feel
like
we
have
to
find
those
feedback
loops
that
are
missing
and
help
close
that
loop
as
we
have
everybody
from
across
the
ecosystem.
Here,
thanks.
B
R
Well,
maybe
caesar
what
next
comment
so
on
the
gaming
I
said.
One
question
I
had
for
janna
was
was
whether
the
pressures
he
was
having
were
one
to
the
network
layer
or
on
the
servers
for
when
they're
handling
these
large
downloads
of
of
new
games
and
things
and
then
from
a
sort
of
practical
point
of
view.
B
Make
sense
so
look.
I
want
to
switch
to
a
different
thing
here.
I
want
to
see
up
stephen
stephen
for
a
second
to
talk
about
what
do
we
want
to
do
on
friday?
We
didn't
really
get
too
much
to
the
topic
today
of
wow.
It's
amazing.
The
internet
didn't
die
what
you
know.
This
went
great.
We
should
probably
talk
a
little
bit
more
about
that,
but
I
think
there's
probably
some
other
things
we
should
talk
about
on
friday.
G
I
was
sorry
in
the
middle
and
by
now
I
think
most
of
my
comments
were
made
by
other
people,
but
I
wanted
to
say
mainly
from
yesterday.
I
had
two
main
observations.
One
thing
is
that
we
kind
of
everybody
looks
at
their
isolated
metric
and
really
need
to
share
the
data
and
look
at
everything
together
and
then
also
that
we
actually
had
problems
to
to
measure
the
usual
experience.
Rather
than
like.
We
measure
what
we
can
measure,
but
we
it's
not
sure
we
measure
the
right
thing.
G
So
we
can
retrieve
the
data
when
we
need
the
data
and
then
also
have
some
kind
of
more
automated
way
to
share
the
data,
like
you,
don't
have
to
put
all
the
data
to
everybody,
but
you
have
to
collect
the
data
at
the
endpoints
or
whatever
and
then
make
them
available
when
you
need
them
and
make
it
an
automated
way,
because
coming
together
in
a
call
like
this
and
just
talk
about
it,
doesn't
work
all
the
time
and
it's
very
slow.
So
that's
where
we
could
do
some
protocol
work.
G
I
think-
and
one
minor
point
where
I
don't
have
a
solution
to
at
all-
is
also
somebody
made
the
comment
that
we
just
measure
what
how
the
network
is
used,
but
we
don't
have
a
way
to
measure
how
the
met
how
people
would
like
to
use
the
network.
So
you
know
what's
missing
like
what
what's
the
additional
capacity?
What's
the
additional
thing,
people
need
and
that's
more
a
social
question
than
a
technical
question.
Maybe,
but
it's
also
an
important
point.
I
I
So,
basically,
that's
my
my
my
observation
of
the
day
yeah,
how
packets
get
delivered
and
how
capacity
can
keep
up,
but
I
think,
there's
much
deeper
problems
we're
facing
as
the
security
talk
to
people
who
handle
ddos
today,
how
much
margin
they
have
holding
things
up
and
I
think
it's
a
much
bigger
problem
and
the
the
the
co-worker
that
made
the
system
more
kind
of
vulnerable
because
we
have
more
dependency,
I
have
to
say,
not
to
say,
made
the
system
more
vulnerable.
We
have
much
bigger
dependency
on
the
internet.
I
G
So
maybe
I
I
do
agree,
I
think,
there's
a
much
bigger
question
we
need
to
tackle,
I'm
not
sure
if
we
can
or
like.
I
think
this
was
a
problem
that
was
there
before
it
just
got
even
more
average
in
the
current
situation,
but
like
there
have
been
attacks
on
hospitals
and
these
kind
of
things
before
we
knew
before
that.
This
is
a
critical
infrastructure.
G
B
G
B
B
Point
there,
which
look
all
of
these
security
problems
that
we've
seen
in
covet
every
one
of
them
pretty
much
we've
seen
before
that
the
scams,
the
fraud,
everything
variants
of
them
right.
Your
point
was
that
I
think
the
most
important
point
is
that
the
internet
has
become
more
critical
than
ever
before
during
this
time
right
that
there
was
an
actual
shift
for
a
lot
of
the
population.
I
mean.
G
So
I
even
agreed
to
that
point
because,
like
people
notice
it
right
like
they,
they
they
have
been
using
it
every
day
right
but
like
if
we
would
have
had
an
internet
breakdown
like
a
year
ago,
that
would
have
been
like
equally
bad
for
our
economy
and
everything
right.
So
it
has
been
critical
before
I.
I
D
B
C
C
I'm
hearing
nothing,
okay,
so
let's
so,
let's
have
a
session
friday.
It
may
or
may
not
last
two
hours
I'll,
give
it
some
thoughts
and
have
a
look
through.
The
notes
see
if
there
are
topics
that
are
kind
of
obvious,
the
observability
one
and
seems
like
it
might
be,
at
least
if
anybody
has
other
topics,
then,
just
if
you
want
to
send
me
a
mail
I'll,
send
it
to
the
the
program
committee
list.
C
If
you
have
like
five
minutes
worth
of
slides,
we
could
do
a
couple
of
those
and
I'll
send
out
a
mail
or
with
you
know,
with
some
suggested
agenda
tomorrow
at
some
stage,
and
if
anybody
has
agenda
topics
that
haven't
come
up
yet
that
they
want
to
suggest
right
now,
go
right
ahead.
We
have
a
minute
or
two
left
or
if
they
have
a
better
plan.
B
One
agenda
topic
that
I
would
be
curious
in
any
getting
this
group's
feedback
on
if
there
is
time
for
it-
and
this
is
just
not
not
just
my
as
an
individual
contributor
here-
is
whether
we're
today
across
many
service
providers.
We
see
whiting
out
of
this
surf
class
markets
and
whether
there
might
be
an
incentive
to
not
wipe
them
all
out
or
allow
some
to
get
to
go
through
or
just
discuss
costs
a
little
bit.
C
K
Jenna
dip
serve
cage
fight,
put
color
in
jared
in
the
room.
I
was
gonna
ask
if
stephen,
I
I'm
assuming
maybe
you're
already
playing
this
as
well.
Very
quickly.
Ask
are
you
planning
to
have
some
sort
of
sort
of
action
items
coming
out
of
this
workshop?
I
missed
monday,
but
I
intend
to
be
there
on
friday.
C
Yeah
I
mean
I
guess
we
wanted
to
identify
possible
topics
to
work
on
any
conclusions
we
can
reach
you.
The
recording
for
monday
is
available,
so
it's
on
youtube,
so
you
can
watch
it
back.
There
was,
I
mean
you
know
there
was
a
bunch
of
other
measurement
kind
of
topics
yeah.
If
we
can
reach
conclusions.
That
would
be
a
good
thing.
I
think
somebody
earlier
said.
We
should
all
vote
that
the
internet
was
great
and
then
we'd,
probably
probably.
K
C
It's
actually
volunteering
to
look
over
the
notes
for
such
things
and
propose
them
yeah,
fair,
good,
excellent,
so
yeah
I
mean,
if
you
could.
If
you
look
over
the
notes
in
the
pad
and
see
I
mean
I
guess
the
one
that
strikes
me
from
today
was
this
observability
as
a
thing
to
design
in,
maybe
to
think
about
that,
so
how
to
do
it
in
a
privacy-friendly
way
in
an
efficient
way
and
so
on
useful
way.