►
From YouTube: Workshop on Analyzing IETF Data (AID), 2021-11-29
Description
Day 1 of the IAB's Workshop on Analyzing IETF Data (AID), 2021-11-29
Workshop page: https://www.iab.org/activities/workshops/aid/
Day 2: https://youtu.be/CMqTbZXF5G8
B
A
Introduction
and
the
kickoff
of
the
analyzing
iet
I
on
the
iab
analyzing
ietf
data
workshop,
show
me
the
numbers
it's
great
to
have
you
all
here
in
a
different
world.
I
would
be
welcome
you
all
now
to
the
slightly
cold
but
clear
and
sunny
amsterdam,
and
we
would
have
a
hackathon
in
the
building
of
the
university
of
amsterdam
and
we
would
probably
have
a
nice
dinner
on
a
canal
boat
to
see
amsterdam
from
its
prettiest
sites
from
the
old
canals.
A
A
Unfortunately,
we
currently
do
not
live
in
that
world
and,
as
you
might
have
seen
on
the
news,
the
netherlands
is
one
of
the
countries
with
the
highest
prevalence
of
the
code
covet
19
omegran
for
variant
right
now,
and
before
that,
the
covert
numbers
here
were
already
rising
at
an
alarming
rate.
So
in
that
sense
it's
probably
good.
A
At
the
same
time,
the
social
geographer
keller
easterling
writes
that
infrastructure
sets
the
invisible
rules
that
govern
the
spaces
of
our
everyday
lives
and
that
changes
to
the
globalizing
world
are
being
written
not
in
the
language
of
law
and
diplomacy,
but
rather
in
the
language
of
infrastructure
and
standards.
I'd
add
to
that
are
the
grammar
to
the
language
of
infrastructure,
so
standard
setting
has
become
one
of
the
main
norm.
Setting
processes
for
information
societies.
A
A
Pardon
me.
Trends
in
standardization
have
moved
from
government
standards
to
more
private
standards.
Bodies
that,
in
combination
with
technological
development,
have
contributed
to
the
speeding
up
of
standardization
processes,
which
has
formed
the
basis
for
a
lot
of
innovation.
On
top
of
these
standards,
whereas
there
seems
to
be
a
convergence
on
industry-led
private
standardization
practices,
not
all
private
standardization
privatization
efforts
function
in
the
same
way.
There
are
a
lot
of
different
processes
and
procedures,
intellectual
property
policies
and
access
policies
for
the
output
documents
and
participation,
whereas
most
standardization
now
revolves
around
consensus
building.
A
There
is
no
consensus
of
what
consensus
actually
means
and
how
it
is
achieved.
All
these
different
ways-
industry-led,
private
standardization
bodies
function-
affects
the
standardization
process,
the
outputs
and
also
its
outcomes.
That
means
that
how
sdos
work
impact
societies
in
ways
we
do
not
fully
comprehend
increasing.
This
comprehension
is
what
we
would
like
to
research
with
you,
especially
in
relation
to
the
internet
engineering
task
force.
A
When
I
talked
to
jay
daly
the
executive
director
of
the
ietf
administration
llc,
he
said
when
I
entered
the
ietf,
I
was
surprised
by
the
lack
of
the
use
of
data.
It's
like
people
prefer
to
argue
rather
than
use
the
data,
so
the
work
we
are
doing
here
potentially
can
inform
the
workings
of
the
ietf
its
participants,
its
leadership,
as
well
as
the
people
that
are
relying
on
rfcs
and
the
people
that
are
impacted
by
them,
and
these
are
not
just
aspirational
goals
in
the
ietf
administrative
strategy,
strategic
plan
from
2020.
A
A
The
questions
ahead
of
us
now
are
how
ietf
data
can
be
used
to
improve
the
ietf
and
its
processes
create
better
standards
that
see
more
implementation,
aid,
ietf
leadership,
authors
and
community
members
increase
ietf
legitimacy
in
a
time
where
standards
increasingly
gain
geopolitical
attention
and
scrutiny
and
to
aid
this
work.
How
can
we
create
a
diverse
and
sustained
community
around
people
that
use
if
itf
data
and
maybe
even
produce
ietf
data?
A
So
you
have
all
seen
the
agenda
and
the
schedule,
and
just
here
to
repeat
it
today
we
have
a
list
of
talks
where
we
get
more
insights
in
what
there
is
and
what
there
are
and
what
there
is
not
done,
tuesday
and
wednesday.
There
will
be
a
hackathon
and
thursday.
We
look
at
some
issues
that
weren't
discussed
today
and
look
at
the
results
from
the
hackathon,
because
we
have
people
from
diverse
backgrounds
and
experience
present
that
are
used
to
different
forms
of
debating
and
working
together.
A
We
would
really
like
to
emphasize
they'd
like
to
create
an
environment
in
which
everyone
can
and
will
actively
participate.
This
is
a
space
and
a
community
that
we
build
together.
It
would
be
great
that
we
all
take
that
into
account
in
the
discussions
after
one
and
a
half
year
of
online
meetings.
You
probably
all
know
the
video
conferencing
etiquette,
but
just
to
repeat
for
completeness
when
not
speaking,
please
mute
your
microphone.
A
If
you
want
to
speak,
please
raise
your
hand
in
webex
or
if
that
doesn't
work.
Just
visibly,
please
be
mindful
of
background
noise.
Try
to
keep
your
camera
on,
if
possible,
to
increase
interaction
during
the
workshop,
try
to
avoid
multitasking
and
also
feel
free
to
use
to
use
the
chat
for
discussion.
A
After
the
workshops,
member
of
the
program
committee
will
write
a
workshop
report
with
the
help
of
today's
voluntary
notetaker
kate
pundik
thanks
so
much
for
that
kate,
and
also
special
thanks
to
cindy
morgan
for
handling
many
of
the
logistics
to
get
us
all
together
with
the
danger
of
repeating
myself.
The
sessions
today
will
help
us
to
understand
what
data
there
is,
what
tools
there
are
to
analyze
the
data
understand
what
work
has
been
done
on
this
data
and
with
these
tools
and
what
methods
and
data
are
missing
to
answer
relevant
questions.
A
A
C
C
C
So
we
have
45
minutes.
It's
not
a
huge
amount
of
time.
There's
actually
quite
a
lot
of
information
that
we
could
dig
into
the
program
committee
chose
four
topics
to
be
sort
of
explicitly
discussed
or
or
sort
of
presented
briefly,
but
there's
more
in
terms
of
the
papers
so
go
go,
read
the
materials
that
it's
pointed
to
here
and
also
maybe
some
accent
in
the
other
papers
in
the
workshop.
So
we
have
robert
sebastian
stephen
and
greg
talking
about
their
topics,
there's
maybe
10
minutes
or
so
for
each
of
these
things.
C
So
let's
try
to
creep
it
keep
it
brief.
I
think
this
session
is
maybe
a
little
bit
more
about
information
than
discussion,
but
we
should
also
enable
some
some
time
for
for
discussion.
I
did
want
to
say
one
thing
on
this
first
slide,
and
that
is
that
at
least
my
mental
model
for
for
thinking
about
these
things
is
that
this
this
data,
the
actual
data
that
is
you
know,
exists
somewhere,
can
be
obtained
and
then
there's
ways
to
access
it.
Typically,
you
cannot
access
data
directly.
C
So
it's
kind
of
important
to
remember
that
there
is
this
user
there
somewhere.
That
has
a
burning
need
to
do
something,
whether
it's
a
itf
participant
or
the
idf
as
an
organization,
but
they
have
may
have
some
needs
and
getting
getting
sort
of
pull
from
those
parties
for
what
we
need
to
do
is
important.
C
The
other
thing
that's
that's
kind
of
interesting
that
we
often
seem
to
be
somewhere
between
the
formally
recorded
things
and
and
things
that
could
perhaps
be
recorded,
or
it
could
be
determined
somehow
sort
of
living
on
the
bleeding
edge
that
well.
We
didn't
record
this
information,
but
we
could
perhaps
determine
it
somehow
so
think
about
those
two
things.
While,
while
we're
going
through
the
the
rest
of
the
discussion
I'll
leave
it
there,
maybe
robert
you
could
start
talking
and
you'll
see
if
you
can
advance
the
next
slide.
E
E
E
So
during
the
the
next
few
minutes,
I'm
just
going
to
hit
a
few
highlights
if
we
could
go
to
the
next
slide,
please
so
the
best
way
to
get
access
to
the
data
tracker
data
to
get
all
of
it
and
to
do
anything
that
that
you
want
to
with
it
is
to
set
up
the
local
data
tracker
development
environment.
The
easiest
way
to
do
that
is
using
docker,
we've
recently
reworked
the
way
the
data
tracker
docker
environment
is
built.
E
Lars,
put
a
lot
of
effort
in
and
nick
put,
some
polishing
touches
on
it
to
give
you
an
environment
within
40
minutes
or
so
because
the
initial
build
is
quite
large
to
be
able
to
access
the
data.
There
is
a
developer
database,
that's
created
nightly,
it
has
all
of
the
data.
Very
very
few
things
are
scrubbed.
There's
a
footnote
on
the
page
I'll
link
to
earlier
that
details
this
grabbing,
but
it
basically
boils
down
to
people's
personal
passwords,
are
scrubbed
away.
E
E
E
E
When
you
are
using
the
api,
one
thing
that
we've
discovered
you
need
to
be
very
careful
with
is
you
are
retrieving
large
data
sets
you
want
to
order.
Your
data
set
the
tasty
pie.
Infrastructure
underneath
does
not
provide
its
own
in
the
and
the
objects
that
are
being
exposed,
don't
provide
their
own
natural
ordering.
E
So
if
results
are
paginated
and
you
haven't
provided
your
own
ordering,
you
will
get
chaotic
results,
go
ahead
to
the
next
slide,
please
so
a
couple
of
brief
previews
of
what
you
can
do
working
in
django's
shell.
On
top
of
the
database,
you
can
query
the
documents
that
I've
created
used
just
using
my
name
by
working
through
against
the
models
that
numbers
seemed
quite
large
to
me.
So
I
went
in
to
see
what
my
document
types
actually
were
and
discovered
that
the
data
tracker
thinks
I've,
contributed
67
drafts
and
191
reviews.
E
The
language
is
fairly
easy
to
pick
up
and
again
there
are
pointers
for
it
in.
In
that
note,
let's
go
ahead
and
go
move
to
the
next
slide
with
the
api.
You
can
feed
a
query
into
curl
and
tell
it
to
format
its
output
as
json
and
then
manipulate
that
with
command
line,
json
manipulation,
tools
to
do
things
like
very
quickly
and
easily
find
out
which
drafts
are
in
last
call.
At
least
these
were
the
sets
of
drafts
in
last
call.
E
When
I
prepared
this
slide
last
week,
we
can
move
to
the
next
slide.
Thank
you.
So
what
is
in
there?
The
data
tracker
is
huge.
The
the
database
is
that
the
reason
that
that
build
takes
nearly
40
minutes
is
is
mostly
in
loading
the
database.
It
has
the
numbers
that
you
see
on
the
slide
large
numbers
of
documents,
people
meetings,
information
about
the
interrelationships
between
these
things.
E
What's
not
in
there
is
interesting.
We
have
as
a
design
go
kept
minimal
personal,
identifying
information
for
people.
We
have
their
names.
We
have
what's
needed
for
them
for
us
to
interact
with
them
as
documents
go
through
the
development
and
publication
process,
but
we
have
very
carefully
not
gathered
a
lot
of
information
about
people
that
might
otherwise
be
interesting.
E
E
E
There's
an
example
on
the
slide
right
now
now,
because
of
the
way
we
capture
who's,
a
working
group
chair,
it
can
be
very
difficult
to
find
when
someone
quit
being
a
working
group
chair,
you
have
to
infer
that
from
the
data
instead
of
the
data
saying
it's
saying
explicitly
and
the
the
query
that
you
have
to
put
together
to
do
that.
Inference
is
complex.
C
E
F
F
So
this
talk
is
a
couple
parts.
First,
I
want
to
talk
briefly
about
big
bang,
which
is
a
scientific
software
toolkit
for
analyzing,
open
online
collaboration
and
deliberation
and
then
about
a
particular
research
project
that
is
sort
of
a
goal
of
mine
for
the
workshop.
So
big
bang
is
a
python
toolkit.
F
F
Basically,
though
it
it
helps
bring
the
power
of
the
scientific
python
stack
to
this
data,
which
has
been
made
available
through
these
various
open
standard
setting
sources,
we're
hoping
to
integrate
information
extraction
soon.
Originally
it
was
designed
to
look
across
various
different,
open,
open
sort
of
collaborative
communities.
This
is
an
early
plot
from
the
first
paper
on
this,
showing
the
interaction
between
the
scientific
python,
community,
the
wiki
wikimedia
community
and
the
open
street
map
community
and
showing
that
there's
some
bridge
nodes
between
those
groups
slide.
Please.
F
A
number
of
different
institutions
have
been
affiliated
with
it
or
contributed
to
it
in
the
past,
and
presently
it's
one
of
the
best
things
about
it.
In
my
opinion,
is
its
community
and
we
really
welcome
you
to
get
involved
next
slide,
please
so
an
example
of
what
might
be
done
with
this
sort
of
analysis.
F
A
question
that's
been
puzzling
me
for
some
time.
Is
this
question
of
the
difference
between
individual
and
organizational
behavior,
so
in
the
sort
of
the
ethos
of
the
ietf
there's
this
view
that
people
that
are
participating
are
participating,
participating
as
individuals,
but
naturally
they're
also
often
acting
on
behalf
of
organizations
like
companies.
F
So
there's
normative
questions
that
are
associated
with
this,
and
I
don't
have
a
I'm
not
going
to
weigh
in
on
the
answers
to
those
questions,
but
it
might
be
said
that
individuals,
individuals,
might
be
better
stewards
of
the
public
interest
than
a
commercial
organization,
but
there's
a
related,
descriptive
question,
which
is
how
to
determine
when
individuals
are
acting
independently
versus
as
part
of
an
organizational
action
and
what's
interesting
is
that's
empirical
work
that
spans
different
levels
of
abstraction,
which
is
quite
tricky
to
do
next
slide.
Please.
F
So
not
only
is
it
divided
between
many
different
working
groups,
but
it's
also
divided
between
every
email
address
has
a
domain
as
well
as
a
prefix,
and
you
can
group
emails
by
their
domain
and
then
see
how
domains
and
or
prefixes
participate
in
different
working
groups.
So
one
might
ask
just
looking
at
this
count
of
participation
in
different
working
groups.
Does
domain
x
have
more
or
less
of
a
stake
than
domain
y
in
the
first
working
group?
Are
they
more?
Are
they
more
coordinated
in
their
actions
or
not
next
slide?
F
So
a
random
distribution
of
of
agents
is
going
to
be
highly
weighted
in
terms
of
its
counts
and
its
complexity
in
the
lower
end
of
the
scale
and
not
show
a
lot
of
structure
at
a
higher
scale,
whereas,
as
these
components
become
more
organized,
the
distribution
shifts
to
the
right
next
slide.
Please
so,
with
some
preliminary
work
on
the
distribution
of
email
addresses
within
each
domain.
F
So
the
next
step
for
this
work
is
to
consider
organization
within
working
groups
and
see
whether
particular
working
groups
are
distinguished
based
on
individual
participation
or
domain
or
organizational
participation
and
the
kinds
of
domain
level
organization
and
participation
that
are
observable.
So,
just
as
an
example
for
htt
visa
gmail
has
this
very,
very
highly
skewed
sort
of
random-esque
distribution,
there's,
whereas
the
google.com,
who
are
very
involved
in
that
in
that
working
group,
there's
a
lot
of
area
under
that
curve,
which
indicates
perhaps
a
corporate
or
organizational
strategy.
G
F
G
F
F
C
F
Yes,
so
there
are
some
libraries
available
for
basically
guessing
a
gender
based
on
people's
first
names.
There's
certainly
limitations
to
that
method,
but
it's
a
quite
internationalized
database,
it's
better
than
you
might
expect.
It
has
lots
of
gaps,
though,
and
and
we've
used
that
to
to
see,
for
example,
which
working
groups
are
more
or
less
biased
in
terms
of
gender
participation
and
there's
work
on
affiliations,
partly
from
ingesting
from
the
ietf
data
tracker.
Partly
by
considering
this
mailing
list
analysis.
F
E
So
I
would
be
very
curious
to
see
what
tools
you
have
for
identifying.
What
part
of
a
name
is
a
first
name.
I
think
that
we
as
a
community
and
and
that
there
are
several
communities
that
have
some
interesting
times
in
front
of
them
on
challenging
the
assumptions
that
a
lot
of
our
software
currently
has
about
what
the
structure
of
a
name
is
and
if
you've,
if
you
think
you've
got
a
particularly
good,
handle
and
start
on
identifying
what
could
be
considered
a
name.
E
F
C
Yeah
there's
some
limits
to
the
process
when
you
use
these
libraries,
if
I
recall
correctly,
the
limit
is
around
15
failure
rate,
so
it
just
means
that
you
can't
classify
some
somebody
accurately
and
that's
fine.
It
actually
could
be
a
good
thing,
but
you,
you
still
get
some
some
data
as
a
aggregate
result.
F
J
Sure,
hi
everyone!
I
just
want
to
talk
about
briefly
the
icf
data
library
that
we've
been
developing.
So
if
you
go
to
the
next
slide,
so
the
library
that
we've
got
is
concerned
with
providing
access
to
sort
of
three
different
data
sources
of
idf
data,
so
the
mail
archives,
the
data
tracker
and
the
rfc
index.
If
we
go
to
the
next
slide,
we
can
see
that
those
three
different
data
sources
obviously
are
provided
by
three
different
methods
so
for
the
mail.
J
J
J
So
you
can
spin
up
a
mongodb
instance
and
our
library
can
talk
to
that
mongodb
instance
and
use
it
as
a
cache
for
serving
requests.
So,
instead
of
every
time
you
want
to
access
information
from
the
data
tracker
instead
of
making
a
request
to
the
data
tracker
directly,
that
request
can
be
served
from
the
cache
and
the
library
tries
to
be
as
smart
as
it
can
about
managing
that
cache.
J
Of
course
it's
got
to
balance
that
sort
of
performance
with
trying
to
be
as
consistent
as
possible.
So
we
want
to
make
sure
that
if
your
request
is
being
served
from
the
cache
that
you're
getting
the
same
results
as
you
would
get
accessing
the
data
tracker
directly.
So
all
of
that
sort
of
logic
is
in
there
basically
to
improve
performance
and
to
try
and
reduce
the
load
on
the
data
tracker
in
particular.
J
J
We've
also,
interestingly,
got
the
the
relationship
between
this
rfc
and
there
sees
that
it
perhaps
updates
or
obsoletes
or
rc's
that
update
or
obsolete
that
one.
If
we
go
to
the
next
slide
in
terms
of
mail
archives
via
imap,
we
have
access
to
all
of
the
itf
mailing
lists
and
mirrors
of
different
organizations.
Mailing
lists
from
around
1995.
J
The
library
exposes
an
interface
that
groups
those
messages
by
mailing
lists
and
also
provides
a
thread
abstraction.
So
you
can
see
the
replies
to
each
message
and
it
provides
an
interface
for
for
navigating
the
the
mailing
lists
by
thread
and
finally,
of
course,
we
provide
access
to
the
data
tracker
I'll,
not
repeat
everything
that
robert
said
about
what
is
available
via
the
api,
but
broadly
we've
got
information
about
documents
and
groups,
intellectual
property
disclosures
things
about
mailing
with
subscriptions
meetings,
people
and
reviews.
Those
are
the
main
groups
of
data
that
we
exposed.
J
So
just
I
want
to
just
finish
on
a
sort
of
brief
example
of
how
to
use
the
api
and
and
what's
available.
So
here
we
can
see
that
you
know
we
import
the
data
tracker
component
of
the
library.
We
then
instantiate
a
date.
Tracker
object,
that's
the
object
through
which
we
make
a
request
to
the
data
tracker
on
this
third
line.
Then
I
find
the
data
tracker
person
that
corresponds
to
this
email
address.
J
Next,
it's
going
to
try
and
find
all
of
the
meeting
registrations
that
correspond
that
have
been
made
by
that
person.
So
it's
finding
my
profile
and
then
it's
going
to
find
all
of
the
meeting
registrations
that
I've
made
and
then
it's
going
to
print
out
all
of
the
metadata
for
any
ietf
meeting
registrations
it
finds
against
that
have
been
made
by
me.
J
And
so,
if
we
go
to
the
next
slide,
we
can
see
the
output
of
this
example,
and
we
can
see
that
it's
found
a
whole
bunch
of
meeting
registrations
that
have
corresponded
to
my
data
tracker
profile
and
if
we
go
to
the
next
slide,
just
highlight
one
of
the
sort
of
interesting
things
about
this.
This
way
of
fetching
the
data.
J
Of
course
I
can
enter
whichever
email
address.
I
like
when
I,
when
I
register
for
the
meeting,
and
sometimes
I
enter
the
wrong
one
or
I
enter
my
university
email
address
and
because
we're
searching
by
data
tracker
profile,
because
the
data
tracker
knows
about
all
of
my
email
addresses
we
can
find
all
of
those
registrations
quite
easily.
J
J
The
library
itself
is
available
via
pi
pi,
so
you
can
just
go.
Pip
install
itf
data
and
you'll
get
the
latest
version.
There's
lots
of
examples
and
the
source
code
for
the
library
are
all
available
in
github
and
I'll
pop
the
link
into
the
chat
and
then
the
slides.
I
provided
some
more
examples
that
you
can
go
through
later
on
and
those
are
all
available
in
github
as
well.
K
J
So
we
have
code
to
detect
changes
in
the
data
tracker
interface,
and
so
we
will
we'll
know
if
it
changes
it
will
not
automatically
pick
it
up.
We
do
need
to
expand
the
api
to
include
the
new
field,
but
we
we
do
have
code
that
detects
it.
At
least
the
data
tracker
api
for
smaller
changes
exposes
a
version
number
and
we
invalidate
the
cache
if
that
version
number
changes
so
that
we're
consistent
with
the
data
tracker.
J
And
so
mark
is
asking
a
question
in
the
chat,
if
it's
possible
to
populate
the
cache
with
everything
and
then
work
disconnected.
So
it's
not
something.
We've
implemented
at
the
moment
the
ability
to
sort
of
pull
in
everything
the
cache
tries
to
fetch
from
the
data
tracker
and
switch
to
using
the
cache
sort
of
intelligently,
if
you're,
if
you're,
making
enough
requests
it'll
switch
to
using
the
cache,
but
it
should
certainly
be
possible
to
do
that.
It's
just
not
something
that
we've
implemented
yet.
L
But
thanks
stephen
for
that
presentation,
that's
super
useful.
I've
really
enjoyed
using
this
library.
I
I
was
curious
about
the
the
mailing
list
piece
that
that
piece
seemed
new,
or
at
least
I'm
not
up
to
date
with
what
what
sort
of
extra
interface
are
you
providing
as
opposed
to
getting
it
directly
from
django
that
robert
was
talking
about
before,
or
rather
than
just
parsing
it
into
parsing
the
actual
email
archives
themselves.
J
So
we're
not
providing
a
huge
sort
of
abstraction
on
top
of
the
accessing
accessing
it
directly
we're
just
providing
essentially
a
python
api
for
accessing
the
messages
by
mailing
list
or
by
thread
within
those
mailing
lists,
and
so
we're
not
doing
anything
and
the
way
that
big
bang
does,
for
example,
we're
not
adding
sort
of
that
level
of
analytical
code.
It's
purely
for
accessing
them.
C
Great
stuff,
thank
you
for
doing
this.
I
wonder
if
we
could
go
forward
and
then
do
some
more
discussion
in
the
end,
so
as
we're
trying
to
cover
different
kinds
of
pieces,
so
we
asked
greg
to
also
talk
about
the
ietf
website
and
what
information
is
actually
available
there
or
from
from
the
use
of
the
idf
website
greg.
M
M
Yeah
it'll
be
a
very
obvious
slide,
but
I
can
start
talking
just
real
quickly.
As
you
already
mentioned,
the
analytics
we
collect
on
www.itf.org,
pretty
straightforward
and
a
little
different
flavor
that
one
other
people
have
talked
about.
So
we
implemented
this
about
a
little
over
a
year
ago,
maybe
maybe
18
months
ago,
and
we
use
a
hosted
self-hosted
version
of
matomo
analytics
to
collect
information
about
visitors
to
www.itf.org.
M
For
example,
we
only
retain
individual
visit
information
for
five
days
and
aggregate
information,
I
think
for
12
months,
and
we
publish
summaries
of
the
information
that
we
collect
online
at
the
url
that
was
included
in
pr
one
of
yari's
first
slides,
so
is
the
slide
available.
I
just
wanted
to
know.
If
I
should.
C
M
So
yeah
sure
no
worries
there.
We
go.
Thank
you
niels
yeah.
So
two
other
quick
points.
First
of
all
that
the
content
again
as
as
I
mentioned
is
for
analytics,
is
only
only
that
which
is
covered
in
the
cms
that
we
use.
So
there
is
a
lot
of
content
on
www.itf.org
or
served
under
that.
That
is
not
in
the
cms,
for
example,
the
rfc
index
and
the
id
index
are
both
under
www.idf.org,
but
not
in
the
cms,
so
analytics
doesn't
cover
that.
M
But
and
again
the
content
is
in.
The
cms
is
focused
largely
towards
two
audiences
that
don't
exactly
cover
it.
Participants,
one
is
potential
participants
and
the
other
is
non-participants,
so
potential
participants
might
be
people
who
are
just
learning
about
the
itf
and
might
be
interested
in
getting
involved
in
developing
in
the
work.
The
itf
and
non-participants
might
be
managers
of
itf
participants
or
policy
makers.
M
M
C
C
M
D
M
C
Yeah
one
one
general
observation
on
of
this:
this
whole
thing
is
there's
quite
a
lot
of
tooling
available
and
quite
a
lot
of
data
available.
I've
been
working
on
this
space
for
a
long
time,
but
I
I
didn't
know
about
everything
before
this.
This
workshop,
so
so,
hopefully
reading
some
of
these
papers
and
seeing
this
presentation
has
made
people
realize
that
there's
some
some
data
over
there
that
I
could
go,
get
them
and
use
it
for
for
my
project.
But
of
course,
there's
also
a
question
of
like
what's
missing.
D
D
Like
one
very
interesting
set
of
information
that
I
don't
think
anybody
who
submitted
a
paper
to
this
workshop
utilizes,
for
example,
the
blue
sheet,
so
you
can
really
see
like
not
only
who
participated
in
a
meeting
but
who
participates
in
which
session
how
many
conflicts
we
have
and
so
on.
So
I
I
think
there
is
more
data
around
that.
We
could
utilize
in
interesting
ways.
I
B
One
thing
I've
been
wondering
is
sort
of
looking
outside
the
itf.
A
little
bit
is
to
see
what
kind
of
which
kind
of
pieces
of
software
and
other
items
are
actually
rc
compliant.
So
what's
what's
the
impact
of
of
the
work
of
the
itf
outside
the
itf,
I
don't
know
if
anybody's
ever
looked
into
that,
but
that'd
be
hugely
interesting
data,
especially
for
sort
of
management,
research,
economists
and
many
others.
Of
course,.
K
K
However,
various
individuals
have
have
done
it
for
various
pieces
of
technology,
so
the
tcp
guys
have
you
know,
kept
eyes
on
on
what
the
different
stacks,
especially
the
major
specs,
are
implementing
and
to
what
degree
they're
sort
of
conforming
to
the
rfcs.
But
I
think
other
sort
of
areas
of
technology
are
doing
similar
things.
K
This
there's
obviously
also
sort
of
interop
events,
hackathons
blackfests,
whatever
you
want
to
call
them
that
are
sort
of
more
or
less
formally
do
this,
and
some
of
them
are
paid
to
play
so
for
a
storage
company
nfs
for
decades
has
had
regular
plug
tests
with
you
know,
detailed
tracking
of
who
implements
what
and
and
how
it
interoperates.
So,
if,
if
you
know
you
can
convince
somebody
to
give
you
data
from
those
groups,
you
will
have
a
lot,
but
the
itf
itself
probably
doesn't
have
that.
C
Yeah,
the
question
of
conformance
is
like
a
very
hard
question.
Often
the
developers
themselves
don't
know
if
their
code
actually
conforms
to
the
spec
or
now
but
there's
a
weaker
property,
which
is
that
you
know
to
do
you
know
pieces
of
software
use
this
technology
specified
by
the
ietf
or
that
and
in
some
cases
you
you
might
be
able
to
trace
the
connection.
Somehow
I'm
always
very
interested
in
this,
like.
D
H
So
mirror
that
actually
that's
where
I
was
sort
of
going
was
even
though
like
by
the
way
I
didn't
know
you
were
that
the
ib
was
working
on
this.
I
would
say
that
that
you
have
little
chance
of
success
of
standardizing
it.
However,
going
back
to
yuri's
earlier
question
of
you
know:
are
these
things
being
collected
in
a
way
that
researchers
might
use?
I
have
stumbled
across
numerous
pages
that
are
either
interoperability,
charts
or
conformance
charts.
You
know
put
together
by
reasonable
groups
of
people
with
faces.
H
You
know
of
folks
who
we
know
it
would
be,
I
think,
valuable,
to
have
at
least
a
to
start
a
collection
of
those.
So
there's
there's
one
or
two
in
the
dns
world.
I
know
there's
certainly
some
in
the
http
world
I've
heard
that
there
are
certainly
ones
in
the
tcp
world,
but
to
at
least
have
a
collection
of
those
so
that
researchers
can
start
seeing
maybe
how
to
correlate
those
into
usability
data.
C
C
A
N
N
We've
actually
asked
these
speakers
to
prepare
sort
of
short
provocations
aimed
at
generating
discussion
amongst
the
speakers
and
all
of
us
here
today
and
specifically
we'll
talk
about
the
research
that's
being
done
on
industry
affiliation
and
control
and
focus
on
what
the
real
big
questions
are
within
the
space
and
why.
Those
are
interesting
outline
the
various
methods
that
the
authors
use
to
interrogate
the
data.
P
Yes,
thank
you
very
much.
Thank
you
very
much
once
again
to
the
organizers
for
having
us.
My
name
is
jesus
baron,
I'm
a
research
associate
at
northwestern
university
and
I've
worked
on
several
aspects
of
iatf
data
more
recently
and
this
particular
project
is
trying
to
work
with
all
your
kind
of
scale.
P
So
sebastian
has
already
mentioned
some
of
the
questions
that
we
are
trying
to
address
or
untangles.
Basically,
this
question
of
how
do
individuals
conduct?
How
is
individuals
conduct
determined?
Is
it
determined
by
individual
aspirations,
motivations
characteristics
or
is
it
by
affiliation
characteristics,
industry
control
and
to
to
get
to
the
bottom
of
these
questions?
So
this
is
gonna,
be
a
very
hands-on
presentation
and
really
focusing
on
the
very
manual
aspects
of
the
data.
P
So
it's
basically
we're
trying
to
find
out
who
do
individuals,
work
for
and
then
try
to
get
some
kind
of
causal
identification.
How
does
who
they
work
for
impact,
what
they
do
within
the
itf
and
then
specific?
In
particular,
we
look
at
attendees
at
the
ietf
meetings
as
the
population
and
we
look
at
being
appointed
to
a
working
group
chair
position
as
the
outcome
variable
and
yeah.
So
basically,
what
we
have
is
we.
P
We
look
at
affiliation
information
directly
in
the
attendance
data
on
top
of
the
domains
of
the
emails
as
two
different
sources:
organization
application,
and
then
we
clean
that
standardize
that
and
bring
it
up
to
the
level
of
the
global
ultimate
owner
or
parent
organization.
So
global
ultimate
owner
obviously
is
in
case
of
corporate
affiliations,
but
it
could
also
be
like
the
university
being
the
parent
of
a
faculty,
for
example.
P
So
you
can
see
that
if
you
could
go
on
site
background,
just
one
second,
so
what
we
basically
have
is
like
it's.
We
have
a
bit
more
than
100
000
attendance
records
where
we
see
that
in
the
raw
we
have
about
40,
000,
different
organization
records
or
different
organization
spellings,
which
we
standardize
up
to
about
13,
000,
different
parent
organizations.
P
No
sorry,
we
have
14
000
different
organizational
observations,
which
we
standardize
up
to
about
seven
thousand
parent
organizations
and
we
do
a
lot
of
interpolation.
So,
basically,
when
we
see
missing
records,
what
we
try
to
first
to
do
is
to
is
to
look
at
whether
we
can
see
some
kind
of
continuation
that
if
the
person
has
attended
the
previous
meeting
and
the
following
meeting
with
the
same
type
of
affiliation,
then
we
then
were
willing
to
interpolate
affiliation,
okay
and
next
slightly.
P
So
when
we
observe
people
jumping
around
between
what
we
call
membership
organizations
and
companies
or
public
administrations,
and
we
then
we
are
overriding
the
membership
organization
with
the
company
of
creation
information
and
then
what
we,
what
you
basically
want
to
do
with
that
is
our
next
slide.
Please
is
to
get
to
an
idea
of
okay.
How
does
who
they
work
for
determine
whether
they.
P
For
so
that
we
can
causally
distinguish
between
individual
characteristics
and
affiliation
characteristics
that
obviously
are
vastly
correlated,
and
so
we
have
hand
collected
information
on
260
merger
and
acquisition
events
affecting
affiations
of
individuals
participating
in
the
itf,
and
then
we
can
see
what
happens
to
individuals
after
the
affiliation
that
they
used
to
work
for
got,
acquired
merged
or
spinned
off
from
a
from
a
previous
company
structure
and
next
slide
beat,
which
is
going
to
be.
My
next
slide,
my
last
slide.
P
So,
basically,
if
a
smaller
company
gets
acquired
by
one
of
these,
then
we
see
a
statistically
significant
increase
in
individual's
likelihood
of
being
appointed
to
chair
positions,
whereas
in
case
of
spin-offs,
for
example,
where
we
in
a
smaller
organization
gets
spinned
off
from
a
large
organization.
We
see
a
mildly,
significant
decrease
in
individual's
likelihood
of
being
appointed
to
neutral
positions.
So
this
is
kind
of
our
way
of
trying
to
to
track
or
identify
in
what
ways
it
could
matter
who
you
work
for
in
terms
of
what
positions
you
reach
within
the
itf.
P
N
B
M
L
L
J
L
I'm
here
now
employed
by
the
center
for
democracy
and
technology,
where
I'm
advocating
for
certain
particular
human
rights
in
internet
and
web
architecture
proposals
and
presumably
we're
all
familiar
with
changes
in
affiliation
like
that.
That's
especially
common
at
ietf-
and
I
think
many
people
have
have
noted
it's
been
raised
here
already-
that
there
are
sort
of
like
interesting,
fundamental
questions
about
how
individuals
participate
mediated
through
their
organizational
affiliation,
even
as
that
changes
over
time.
L
So
my
so
in
in
each
of
the
questions
karen
has
put
out
for
us
I'll
try
to
talk
about
my
academic
side
and
and
now
my
civil
society
side.
So
my
academic
work,
my
dissertation
on
enacting
privacy
and
internet
standards
was
about
how
the
actual
multi-stakeholder
standard
setting
process
affects
resolution
of
those
sort
of
public
policy
disputes
over
things
like
privacy
and,
as
I
said
now,
cbt
is
sort
of
advocating
for
that
a
little
more
directly.
L
B
L
L
So
I'm
trying
to
find
ways
that
we
can
do
quantitative,
metrics
to
find
particular
gaps,
or
particular
trends
that
might
drive
how
we're
investing
public
interest
technologist
time,
along
with
maybe
some
sort
of
case
reports
of
what
is
working
or
what
isn't
working
to
sort
of
guide
those
investments.
I
think
the
big
questions
for
me
are
first
of
all
like
how
does
the
changing
affiliations
of
individuals?
How
does
that
spread
ideas
through
the
community?
So
I
I
don't
know
if
people
are
familiar
with
this
famous
paper
on
institutional
isomorphism,
maybe.
B
B
L
Moving
between
lots
of
organizations,
while
in
that
same
organizational
field,
might
create
some
sort
of
common
ideas
in
the
field
or
or
similar
organizational
structures
or
or
similar
concepts,
and
so
I
wonder
how
things
like?
Oh,
if
we
get,
you
know
strong
principles
of
privacy
at
one
person
when
they're
at
one
organization
do
they
take
that
to
their
subsequent
employers?
L
The
other
big
question
that
I
always
have
in
mind
is:
where
are
the
different
sectors
by
that
I
mean
sort
of
private
industry
or
civil
society,
non-profit
or
government
regulators
or
academics?
Where
are
those
different
sectors
represented
in
itf
or
other
technical
standards
setting
processes,
and
where
are
they
not
so
that
that
might
be
interesting
for
me?
L
B
L
A
difference
because
they're
participating
in
these
sorts
of
groups,
but
not
these
other
ones,
on
the
needs,
I'm
I'm
looking
forward
to
what
we
can
hack
on
this
week,
but
I
think
it
would
be
especially
useful
for
me
to
know
like
what's
a
good
data
set
for
types
of
organizations.
I
feel
like
we
often
have
this
like
little
casual
informal
people
like
look
up
the
you
know
the
tld
or
something
to
figure
out
if
it's
a
corporate
organization
or
are
we
trying
to
classify
it
really
quickly
into
oh?
B
L
B
N
Then
I
would
like
to
invite
don
to
talk
about
their
work
and
I
think
that
again
ties
in
really
nicely
to
the
work
of
nick
as
don
also
works
for
a
civil
society
organization,
which
means
the
kind
of
questions
that
you
ask
are
perhaps
a
little
bit
different
than
from
a
strict
research
perspective.
So
with
that
done,
the
floor
is
yours.
R
Thanks
corinne,
I
also
have
a
presentation,
but
just
going
to
share
a
few
words
with
everyone.
My
name
is
don
and
I'm
with
article
19
diving
into
the
intersection
of
human
rights
and
internet
infrastructure,
as
well
as
how
internet
infrastructure
can
impact
the
rights
of
users
and
communities
at
large.
R
So
what
we're
also
very
interested
in
is
also
related
to
affiliation
and
we're
curious
to
see
how
the
iecf's
diversity
efforts
have
also
been
reflected
within
ietf
discussions
and
the
development
of
standards.
So
I
think
that
would
also
really
connect
quite
well
with
upcoming
session
on
community
and
diversity.
R
This
is
also
fairly
similar
to
what
nick
had
just
mentioned
as
well,
in
order
to
be
able
to
find
what
is
shaping
what
is
actually
shaping
internet
protocols
and
determining
where
social
society
can
best
engage
and
then
further
to
this.
How
does
this
shape?
Who
authors
as
well
as
the
development
leading
up
to
the
rfc,
and
so
in
terms
of
that?
The
biggest
need
for
for
us
is
fairly
similar
to
what
nick
had
just
mentioned
in
terms
of
being
able
to
find
out
what
com?
R
N
Perfect,
thank
you.
So
much
again,
it
is
really
good
to
get
a
sense
of
not
just
what
can
be
done
with
the
data,
but
also
what
is
needed
from
the
data
from
the
perspective
of
a
different
kind
of
stakeholder
groups
that
are
participating
within
the
ietf,
and
with
that
I
would
like
to
move
to
our
last
provocation,
for
which
I
want
to
give
the
floor
to
elizabeth
and
thomas
and
their
work
on
iot
standards.
Please
go
ahead.
S
Hi
everybody.
For
some
reason,
I
cannot
open
the
presentation.
I
don't
know
thomas
if
you
can
try,
but
I
can
start
without
that.
So
we've
started
up
our
research
as
part
of
our
master's
dissertation
at
university
college
london
last
year
and
our
dissertation
was
focused
on
geopolitics,
of
emerging
digital
technologies
and
we
were
trying
to
understand
how
political
and
commercial
implications
affect
the
multi-stakeholder
procedures
within
the
sdos.
S
S
If
there
is,
there
are
any
alliances
between
different
types
of
different
stakeholders
within
one
mtm
standard,
and
we
try
to
understand
which
types
of
the
stakeholders
are
represented
at
the
discussion
and
which
are
excluded,
and
also
we
looked
at
the
national
representativeness
yeah.
I
think
news
national
original
within
one
of
term
standard,
and
what
we
found
was
that
there
is.
T
And
although
that
question
is
very
methodologically
like
interesting,
it
does
require
a
few
needs
which
we're
currently
in
the
process
of
exploring,
namely
methods.
For
example,
you
know
clear
information
about
affiliation
and
word
analysis.
Data
through
stated
affiliations,
mailing
lists
and
emails
and
technical
knowledge
for
us
encoding.
T
You
know
coming
from
a
public
policy
background,
we
have
used
r
and
tableau
for
one
m2m
data
that
we've
looked
at
and
we're
currently
starting
to
use
big
bang
to
look
at
ietf
data,
but
there
are
a
number
of
tools
out
there,
some
of
which
are
better
suited
to
certain
formats
of
data
than
others.
So
this
technical
knowledge
for
us
is
constantly
evolving.
So
thank
you.
N
P
Nick
you're
gonna
go
first
here,
so
yeah,
okay.
So
what
we've
been
using
as
basic
first
input
is
the
attendance
lists
of
itf
meetings
and
some
of
these
explicitly
state
affiliations
and
others
don't
for
those
that
don't
many
of
those
have
email
addresses
where
we
use
a
domain
name
as
affiliation
and
then
basically
it's
it's
just
a
matter
of
cleaning.
So
we
we
create
like
panels
where
we
can
like
get
the
series
and
we
try
to
standardize
and
harmonize
affiliation
names
as
much
as
possible.
P
P
So
that
would
what
we,
what
we
call
an
affiliation
change,
so
there's
a
little
bit
of
arbitrary
criteria
in
there,
but
just
to
to
get
like
a
cleaner,
consistent
data
set
and,
and
what
we
find
is
that
about
25
of
the
itf
attendees
in
our
data.
Have
one
of
these
aviation
changes
over
their
span
of
participation
in
the
itf
meetings
and
and
then
like
it's?
We,
we
categorize
these
into
different
categories
of
aviation
change,
so
there
are
some
that
we
call
internal,
which
is
basically
once
you
standardize
information
up
to
the
level
of
the
parent.
P
You
get
rid
of
an
affiliation
change,
so
it's
basically
between
different
sub-organizations
of
the
same
organization.
Then
there
are
these
changes
that
I'm
primarily
interested
in
because
they
provide
us
with
causal
identification,
which
is
changes
exogenously
induced
by
something
that
happened
to
the
organization,
not
by
something
that
happened
to
that
particular
individual.
P
So,
for
example,
a
merger
I
got
to
loosen
being
bought
by
nokia
is
induces
an
aviation
change
for
quite
a
large
number
of
individuals
in
the
data,
and
this
kind,
this
type
of
things
and
then
like
there's
this
naked
employment
change.
Where
one
individual
really
changes
aviation
from
one
organization
that
continues
to
exist
to
another
organization
that
already
existed
before,
so
that's
really
a
transition
of
the
individual
himself
from
one
organization
to
the
other.
So
these
are
kind
of
the
different
categories.
P
The
one
that
we
use
for
identification
in
our
paper
is
actually
quite
rare.
I
mean
that's
just
like
250
individuals.
I
think
that
that
actually
affected
by
that
in
our
data,
that's
what
we
use
as
source
of
identification,
because
we
are
really
concerned
about
this
correlation,
obviously,
that
certain
types
of
individuals
are
just
susceptible
of
being
hired
by
a
certain
type
of
organization,
and
that
would
completely
pollute
our
analyzers.
So
we
try
to
break
that
correlation
and
that's
what
we
use.
L
So
github
people
have
github
profiles
and
the
github
profile
can
include
your
company,
for
example.
So
I
tried
to
match
it
up
against
that
as
a
as
an
interesting
data
source
for
ietf.
That
was
not
very
successful.
It
didn't
it
didn't
work
very
much
okay.
Well,
we
also
have
meeting
attendance
records.
L
I
think
I
think
that's
what
you're
talking
about
primarily-
and
I
do
think
that's
extremely
valuable
if
you
are
a
regular
ietf,
attendee,
well
you're,
going
to
go
to
the
meeting
and
you're
probably
going
to
fill
in
like
an
important
affiliation
into
that
field,
and
that
can
be
very
useful.
The
other
thing
is
that
the
the
document
authorship
at
ietf,
in
particular,
includes
includes
some
information
or,
like
you
know,
some
actual
raw
data
about
who
you
said.
You
were
working
with
at
that
time
that
that
can
be
useful
data.
L
Even
if
it's
not
going
to
cover
the
you
know,
hundreds
of
thousands
of
people
that
have
ever
sent
an
email
to
an
ietf
mailing
list
and
also,
I
think
the
other
thing
is
that
it's
going
to
be
interesting
as
we
apply
to
other
standard
setting
bodies
that
might
have
different
sources
of
data
or
different
sort
of
conventions.
So
we've
had
some
colleagues
working
on
3gpp
for
them
it
seems
like
well.
L
So
it's
likely
to
vary
for
these
different
organizations,
and
I
think
we
should
just
be
aware
of
that
when
we're
looking
for
affiliation
data
so
that
certain
techniques
are
going
to
be
more
or
less
useful
in
in
different
contexts
and
and
yeah
and
and
we're
we're
just
gonna
have
to
explore
that
honestly.
The
other
thing
I
wanted
to
just
react
to
is
that
I
I
love
this
idea
about.
L
L
That
that's
going
to
be
messy
data,
but
it's
also
potentially
very
useful,
especially
for
this
sort
of
isomorphism
question
or
this
idea
of
ideas
being
spread.
Is
that
actually,
we
kind
of
want
that?
Maybe
maybe
people
who
went
through
when
a
university
and
still
have
some
connection
to
it
are
gonna
spread
this
kind
of
approach
into
different,
employing
organizations?
So,
let's,
let's
not
completely
give
up
on
that.
Even
if
it's
going
to
be
messy
data.
U
Yeah,
I
guess
I'll
jump
in
on
the
one
on
the
queue
right
now,
so
I
mean
I
I
think
you
set
me
up
really
well
corrine,
because
I
have
a
suggestion
for
nuance,
which
is
that
I
think
affiliation
by
stakeholder
group
might
be
more
useful
or
actually
a
proposed
framework
in
which
tracking
affiliation
changes
based
on
the
stakeholder
group.
So
maybe
it's
better.
U
If
I
explain
a
couple
of
examples,
so
one
is:
if
you
track
government
participation
across
sdos,
you
actually
are
going
to
find
there's
a
lot
of
private
sector
industry
representatives
that
are
also
representing
a
government,
so
that
can
get
rather
confusing.
It's
true
for
civil
society
as
well.
You
find
sometimes
governments
will
ask
civil
society
to
represent
them.
It's
just
not
as
common
another
example,
and
this
was
sparked.
U
This
thought
was
sparked
when
you
were
talking
nick
about
you
know
actually
tracking
people's
affiliation
over
time,
because
when
I
think
about
civil
society
and
I'm
used
to
working
with
human
rights
organizations
that
can
be
kind
of
dangerous,
you're
kind
of
providing
this
like
network
mapping,
not
network
in
the
internet
sense.
But
you
know
in
the
people
sense
that
can
actually
be
rather
vulnerable.
U
I
think
we're
a
little
bit
more
worried
about
industry
level
influence
than
we
are
like
with
you
know.
Human
rights
organizations
influence
like
I,
I
just
feel
like
there
might
be
a
different
approach.
So
that's
all
I
would
suggest
is
that,
as
we
dig
deeper
into
trying
to
come
up
with
a
framework
that
we
actually
not
treat
every
single
participant
exactly
the
same,
but
we
try
to
you
know
really
take
into
consideration
the
the
dynamics.
P
I
I
think
that
the
way
you
categorize
affiliations
is
really
subservient
to
the
question
that
you're
asking
I
mean
it's
I'm
fully
aware
or
like
it's.
I
I
really
struggled
with
these
fuzziness
and
the
affiliation
choices,
but
it's
I.
I
think
that
all
my
choices
or
all
our
choices
in
the
actual
research
design
were
then
guided
by
this
idea
that
we
were
interested
in
tracking,
where
they're
being
affiliated
with
a
large
industrial
affiliation,
has
an
impact
on
on
your
influence
in
the
sdo.
P
To
answer
my
question:
even
though
there
are
40
000
individuals
in
my
data,
because
these
are
the
individuals
that
provide
me
with
clean
identification,
I'm
not
pretending
that
I
I
have
full
comprehensive
coverage
of
affiliation
in
the
ihf,
I'm
just
very
consciously
selecting
a
very
small
group
of
individuals
that
provides
me
with
clean
observations
for
this
particular
question
that
I'm
asking.
So
I
think
that
if
the
goal
is
to
create
a
a
platform
where
different
people
working
on
affiliation,
data
can
collaborate
and
then
that
can
be
used
for
different
questions.
E
So
I
think
the
question
I
was
going
to
ask
is
mostly
answered,
but
I
will
frame
it
with
a
specific
instance.
Are
you
able
have
you
had
any
success?
It's
seen
through
consulting
organizations
or
just
individuals
that
are
consulting
that
present,
a
consistent
organization,
name
and
email
address,
but
are
attending
from
meeting
to
meeting
representing
different
large
other
organizations.
L
L
You
use
those
personal
email
addresses
or
personal
domains
across
affiliation,
and
I
do
think
that
is
something
that
we
can
measure
that
that
we
do
have
data
I'm
also
noticing.
Just
from
looking
at
some
early
graphs,
I
made
it
at
a
recent
tax
on
that
the
vbn
consortium
can
be
extremely
highly
present,
like
a
very
active
participant.
Well,
that's
that
might
be
one
person.
Some
of
you
know
that
person
who
who
is
working
on
behalf
of
the
of
the
larger
group
I
and-
and
I
don't
think
it's.
L
Q
N
P
I
I
think
that
we
we're
not
able
to
to
go
beyond
what
people
are
willingly
providing
on
their
registration.
For
the
meeting
I
mean,
what
we
do
is
that
we
have
four
different
sdos.
So
if
people
participate
in
different
sdos,
including
in
sdos
where
people
have
to
participate
as
representative
of
a
particular
member
organization,
then
we
would
have
that
information
from
another
organization,
if
they're
participating
with
exactly
the
same
name
and
contact
information,
for
example,
but
that's
quite
rare.
Actually
so
I
mean
yeah,
we
lose
a
lot
of
information.
P
We
lose
a
lot
of
individuals
for
which
we
can't
do
this
type
of
analyzer.
So
it's
it's
really.
Basically,
it's
not
comprehensive
population
data
in
that
sense
that
we
work
with
it's.
We
we
use
that
data
where,
where
we
are
lucky
that
we
have
the
information-
and
there
are
a
lot
of
things
that
would
bias
us
against
finding
something
because,
for
example,
people
already
had
some
kind
of
undisclosed
corporate
affiliation
with
that
new
organization.
So
if
that
was
the
case,
then
you
wouldn't
expect
that
this
aviation
change
has
any
causal
impact.
P
So
the
fact
that
we
still
are
able
to
find
some
causal
effects.
I
I
think
that
we're
we're
probably
understating
the
true
effect
like
all
these
things
should
bias
us
against
finding
anything.
So
it's
I'm
less
concerned,
given
that
we
do
find
something,
but
it
I
would
be
concerned.
If
we
wouldn't
be
finding
anything,
then
I
I
would
not
know
whether
it's
just
because
of
all
these
measurement
errors
or
because
aviation
really
does
not
matter,
but
it
it
does
seem
to
matter.
N
I
Great,
so
I
mean
there's
a
number
of
interesting
things
in
this
space
that
I
think
would
be
fascinating
to
study.
In
particular,
you
know,
affiliation
changes
lead
to
different
work
or
topic
shifts.
You
know
between
people
and
I
think,
that's
hard
to
detect
in
a
number
of
ways.
Speaking
personally,
I've
worked
for
research
organizations
where
our
work
has
been
tied
to
research
grants.
I
So
you
know
what
I
work
on
within
the
ietf
has
shifted
over
years,
based
on
the
research
grants
that
I've
been
working
on,
whereas
at
other
times
my
affiliation
has
changed.
I
think
five
times,
four
of
which
were
actually
resulted
in
buyouts
where
my
work
didn't
change,
because
the
actual
contract
moved
along
the
research
contract,
it
sort
of
actually
moved
along
with
my
affiliation
change,
which
led
me
to
the
final
question,
which
is:
could
we
actually
detect
opinion
shifts
after
an
affiliation
change
use?
I
I
H
It's
a
very
quick
question
of
whether
how
the
data
collection
looked
at
people
who
were
clearly
consultants,
working
for
organizations
or
governments
that
were
not
terribly
popular,
that
is
in
the
ietf,
and
I
know
this
happens
in
3gpp
as
well.
That
gov
some
governments
will
in
fact
only
send
consultants
and
the
consultants
will
identify
themselves
as
consultants,
not
as
working
for
the
government,
but
they're
clearly
doing
the
government
work.
H
And
if
you
look
at
what
they're
doing
so,
I
I
was
wondering
whether
that
kind
of
thing
you
know
in
the
ietf
most
most
of
us
know.
You
know,
for
example,
who
russ
housley
works
for
or
when
I
was
working
for
russ.
You
know
things
like
that.
Did
you
include
that,
as
as
you
know,
essentially
proxies
for
places
where
you
didn't
want
to
identify
yourself
as
the
as
the
primary
stakeholder.
P
P
For
a
large
company
and
at
the
same
time,
has
a
small
individual
consulting
company
in
his
own
name,
and
sometimes
it
tends
in
the
name
of
a
large
company
and
sometimes
in
the
name
of
the
small
consulting
company.
And
there
is
like
some
kind
of
switching
back
and
forth
between
these.
Then
we
would
what
we
what
I
called
interpolator.
We
would
override
the
small
consulting
company
with
the
name
of
the
large
company,
but
for
that
the
name
of
the
large
company
would
have
to
appear
in
at
least
one
registration
for
a
meeting.
P
Otherwise
that
would
just
stay
as
a
small
consulting
company
and
that's
just
measurement
error.
If
you
wish-
or
it's
it's,
it's
a
limitation
of
the
of
our
ability
to
actually
accurately
track
the
real
primary
affiliation
that
we're
interested
in
so
yeah.
We're
quite
aware
of
the
of
the
limitations
of
of
the
of
our
ability
to
to
actually
track
and
measure
affiliation.
So
it's,
but
my
defense
is
always
like
it's.
I
these
would
always
bias
us
against
finding
anything.
I
All
right,
thank
you
very
much,
so
this
next
section
will
be
on
community
and
diversity.
Just
as
a
reminder.
After
this,
this
is
45
minutes
long
and
there
will
be
a,
I
think,
a
30-minute
break.
So
you
will
get
a
break
here
shortly.
I'm
the
assassin
chair
and
we've
asked
three
people
priyanka
mallory
and
lars
to
sort
of
give
their
high
level
summary
presentations
in
five
minutes
or
so
per
person.
Q
But
I
also
work
full
time
at
tata,
consultancy
services
and
as
a
research
scientist,
and
before
that
I
was
in
my
past
life.
I
was
working
in
wireless
networking
studying
how
dns
can
work
in
mubarak,
ad
hoc
networks,
and
I
worked
at
redback
networks
on
the
routing
issues
which
was
which
was
acquired
by
erickson.
Q
So
because
of
my
own
interest.
I
got
involved
with
the
indian
internet
engineering
society
and
the
industry
network
technologies
council,
especially
with
their
ipv6,
webinars,
etc,
and
I
started
seeing
that
ita
as
an
organization.
It's
a
voluntary
global
organization
where
it's
communication
and
activities
are
recorded
and
they're
available
for
analysis
and
study,
as
we
learned
about
the
itfd
and
the
other
tools
that
are
available.
Q
So
I
could
see
that
I
could.
I
can
apply
some
of
the
my
findings
from
what
I've
applied
in
commercial
enterprises
or
other
open
source
communities
like
stack
exchange,
enron,
email
data
set
or
the
linux
kernel
mailing
list
to
the
idf,
and
my
understanding
till
now,
from
getting
involved
with
the
idea.
Q
If
it's
especially
with
the
pre-itf
meeting
that
was
held
in
this
part
of
the
world
a
few
years
ago,
is
that
the
diversity,
inclusion
and
representation
processes
of
the
idf
make
the
consensus
process
robust
and
what
I've
observed
actually
with
the
v6
ops
mailing
list
that
I
I've
been
going
through
a
little
bit.
Is
that
the
consensus
mechanisms
going
from
a
draft
to
an
rfc
and
to
the
final
internet
standard
they
depend?
Q
They
may
have
different
demographic
labels
attributed
to
them,
not
just
that
not
just
their
affiliation.
Labels
which
keep
changing
over
time,
but
also
their
gender
labels,
their
geographic
geographical
locations
and
other
things,
so
their
sense
of
community
may
not
be
appropriated
by
their
observable
demographic
indicators
as
to
what
they
would
advocate
from
what
they
would
influence
on
may
could
be
done
in
a
purely
data
driven
manner
to
understand
how
some
members
can
find
consensus
building
in
their
activity
next
slide
would
be
nice.
Q
I
don't
know
how
to
go
to
the
next
slide.
Okay,
so
some
of
the
questions
that
I
think
are
interesting
are
because
of
kobit
19,
a
lot
of
of
the
idf
and
all
across
the
world,
all
kinds
of
organizations.
They
have
moved
to
virtual
only
interactions,
but
has
that
increased
diversity
and
inclusion,
especially
in
the
idf?
How
do
you
measure
that?
Did
you
measure
that,
with
increased
participation,
increased
number
of
diversity
in
terms
of
drafts
going
to
rfcs
so
on?
Q
And
so
so
there
are
many
statistical
ways
to
identify
that,
based
on
the
data
from
data
tracker
and
mail
archives.
Now
that
itf
data
and
big
bang
and
other
tools
are
easily
able
to
get
this
now.
Also,
what
how
much
opinion
diversity
is
there
in
working
groups?
One
of
the
thing
that
I've
heard
is
that
certain
issues
become
long-standing
issues
and
people
keep
forgetting
about
what
was
the
discussion
that
happened
long
before
and
somebody
if
they
they're
not
repeating
those
concerns
and
they
get
lost.
Q
Q
Q
So
now
the
so,
I
was
because
there's
a
lot
of
understanding
here.
That
affiliation
is
important
and
you
already
see
that
the
rough
and
noisy
nature
of
affiliations
or
finding
how
affiliations
is
related
to
power
and
influence,
but
a
data
driven
manner
of
addressing
these
questions
without
considering
them
from
an
affiliation
perspective
or
from
a
directly
from
a
gender
perspective.
If
you're
taking
people
based
upon
how
they
are
interacting,
based
on
the
on
what
what
opinions
they
put
up
and
how
much
they
are
engaged
with
the
processes.
Q
How
they're
engaged
based
on
that
comes
from
the
participation
data
from
the
data
tracker.
The
minutes
of
the
meeting
as
to
who
is
saying
what?
Because
meetings
are
a
place
where
a
lot
of
decisions
are
getting
taking
place
and
then
mailing
lists
where
a
lot
of
opinions
are
flying
across
about
the
drafts,
which
are
full
of
technical
technical
domain,
specific
words,
as
they
say
in
nlp,
which
is
for
which
the
tooling
is
also
important.
Q
So
the
yes,
this
is
the
slide
thanks
wes.
So
this
is
all
the
work
that
I've
done.
One
of
those
which
will
be
interesting
for
us
to
look
at
on
the
idf
data.
This
is
a
published
thing,
is
to
identify
leaders
and
fine-grained
communities
who
have
coherent
topics
just
by
the
number
of
times
somebody
engages
with
the
idf
per
month.
Q
Q
So
this
is
an
interesting
technique
to
use
just
like
that.
There
is
a
graph
mining
technique
which
is
under
review
and
some
psycholinguistic
techniques
so
I'll
be
interested
to
know
from
the
this
meeting.
So
what
are
the
questions
that
are
important
to
the
iab
and
how
to
take
this
forward?
Thank
you.
I
U
Hey
everybody
so
as
you'll
note,
I'm
starting
out
with
a
rather
provocative
question,
because
I
think
when
we're
talking
about
diversity
of
participation,
we
could
just
extrapolate
to
the
to
the
total
endpoint
and
just
replace
all
of
us
engage
in
the
isf
for
feminists
and
then
try
to
see
if
that
has
any
consequence.
So
I
guess
I'm
couching
our
discussion
here
today
about
data
driven
research
in
a
larger
sort
of
approach
to
actually
determining
what
is
what
are
the
consequences?
So
next
slide,
please!
U
So
in
order
to
do
that,
I
apologize,
but
I'm
gonna
have
to
give
a
little
bit
of
context.
So
I'm
I'm
going
way
forward
in
my
sort
of
question.
I'm
also
going
to
go
back
a
bit
because
I
think
the
context
really
matters,
and
especially,
if
we're
trying
to
get
at
these
like
overarching,
deep
dive,
big
questions
about
diversity
and
community
inclusion.
U
U
That
question
was
posed
in
2014
and,
as
a
result,
it
was
both
a
sort
of
movement
building
activity
to
understand
what
are
some
of
the
issues
relevant
to
intersectional,
feminism
and
the
digital
age,
and
and
also
it
was
a
movement
building
activity
to
actually
build
a
movement
of
people
who
are
aware
of
that.
So
it
was
both
substance
and
process
and
you'll
understand
that
theme
as
we
move
forward.
So
next
slide,
please.
U
So
one
of
the
major
milestones
in
that
work
since
2014
was
something
called
the
feminist
principles
of
the
internet,
so
being
able
to
coalesce
like
an
uncovering
of
the
various
issues,
often
means
that
you
know
as
civil
society
or
folks
practicing
movement
building
that
you
actually
then
start
writing
them
down,
and
so
this
is
what
that
question.
You
know.
Imagine
a
feminist
internet
resulted
in,
so
you
can
go
to
feminist
internet.org
and
you
can
check
out
these
various
principles.
U
I
think
there
are
17
or
so
around
five
different
issue
areas
and
part
of
the
what
you
didn't
stop
there
part
of
the
point
of
then
articulating.
These
principles
was
to
actually
see
them,
then
in
action.
So
a
variety
of
different
movement
based
actions
from
all
over
the
world
then
resulted
from
this,
and
and
so
I'm
going
to
deep
dive
into
one
of
them.
That
actually
is
happening
with
the
ietf.
So
next
slide
please.
So
I
consider
this
draft.
U
We
consider
this
work,
actually
a
sort
of
branch
of
that
movement
work
to
take
those
principles
and
try
to
apply
them
to
to
figure
out.
You
know
what
are
the
relevant
overlap
overlapping
pieces
between
the
sort
of
principled
approach
to
how
the
internet
should
work,
if
it's
an
internet,
intersectional
feminist
analysis
and
how
it
actually
does
work
and
what
are
some
of
the
interesting
guidance?
Maybe
so
next
slide,
please
so
yeah.
U
I
think
we
had
our
first
draft,
maybe
in
2018,
if
I'm
not
wrong,
we
aim
to
do
similar,
because
this
is
also
happening
in
the
human
rights
protocol
considerations.
Research
group,
we
sort
of
modeled
it
after
how
8280
looked
at
the
universal
declaration
of
human
rights,
because
it's
a
set
of
principles,
you
can,
you
know,
sort
of
do
a
sort
of
same
thing
and
we
next
slide.
U
Please
went
ahead
and
thought
about
skipping
over
the
part
where
we
articulate
every
single,
different
piece
of
analysis
and
actually
just
do
a
sort
of
useful
document
as
sort
of
guidelines
for
folks
who
are,
you
know,
developing
internet
protocols.
What
are
the
high
level
points
you
might
want
to
keep
in
mind
and
hear
all
the
references
and
the
context
for
being
able
to
do
that,
and
so
that's
sort
of
the
current
trajectory
of
the
draft.
But
then
there
are
some
questions
that
have
made
us
pause
a
bit.
U
So
I
want
to
present
those
because
I
think
that
they
yeah
perfect
wes
that
they
sort
of
get
at
some
of
these
larger
questions
that
we're
all
trying
to
answer
right
now
today,
especially
in
this
section.
So
we
want
to
consider
if
we
mightn't
detangle
the
process
from
the
substance
a
little
bit,
and
I
cannot
decide
what
to
do.
I
am
honestly
torn
even
sitting
here
in
the
session
with
you.
U
One
is
that
if
we
actually
try
to
measure
diversity
of
participation,
can
big
bang
do
that?
Is
it
meaningful?
I
mean
we
had
the
same
conversation
with
affiliation
right
if
you
can
actually
say,
and
then
priyanka
also
troubles
it
right.
If
you
could
actually
just
say
like
you
know,
this
is
the
quality
that
we're
tracking
can.
Can
you
know,
can
it
does
it
have?
U
You
know,
does
it
have
a
consequence
and
also
how
do
you
get
to
that
point,
and
can
you
do
it
over
time
and
isn't
that
more
meaningful
than
a
snapshot
of
today
next
slide?
Please
another
open
question,
then.
Is
it
doesn't
actually
necessarily
talk
about
power
either
participation
doesn't
tell
you
whether
or
not
you
know
something
even
leadership.
Particip
participation
doesn't
give
you
at
the
end
of
the
day,
an
analysis
about
whether
or
not
it's
changed
outcomes.
U
So
can
basic
data
give
us
influence
or
the
other
side
of
it
is
marginalization.
If
it
can,
how
do
you
actually
achieve
that
measurement?
Do
you
continue
to
measure
it
and
then
also
at
the
end
of
the
day,
does
the
data
actually
give
us
something
of
consequence
that
we
can
action
and
do
something
about
I'm
not
totally
sure
so
then
the
final
slide,
I
think,
yeah.
I
think
that
you
know
like
I
said
this
is
the
one
that
I
really
can't
I
can't
decide
about.
U
You
know
we
could
also
do
something
more
interesting
with
a
separate
document
as
well.
So
we
could
think
about.
You
know
entangling
rather
the
process
question
and
the
substance
question,
because
that
seems
like
the
right
answer
from
a
political
perspective,
but
then
it
points
to
future
work
in
which
you
do
a
deeper
dive,
maybe
on
the
participation
or
process
questions,
that's
one
option
in
the
next
slide.
I
think
there
is
maybe
one
more
that's
just
pointers
yeah.
U
I
I
K
I
think
it's
very
important
and
it
matters
and
sort
of
again
the
ietf
quote:
unquote
leadership,
sort
of
really
pays
attention
and
we're
always
interested
when
people
sort
of
post
research
that
they've
done
using
various
data
and
basically
say
hey.
You
know,
look
at
this.
We,
you
know,
show
a
trend
here
that
we
didn't
know
existed
and
part
of
that
is
because,
as
niels
pointed
out
in
his
intro,
we
actually
have
shockingly
little
data
about
the
organization
right.
The
ietf
is,
you
know,
we're
not
wearing
an
erp
software
or
anything
like
that
right.
K
We
have
a
bunch
of
data
and
we
have
you
know
some
various
mere
few
statistics
that
we're
doing,
many
of
which
are
sort
of
you
know,
focused
around
you
know
who's
participating
and
how?
How
often
are
they
participating
and
so
on,
but
we
have
like
really
no
overview
of
who
our
participants
are
and
how
I'm
homogeneous
or
or
not
they
are
and
anything
anything
that
we
can
get
is
is
better
than
nothing.
K
K
So
one
example
is
this
is
taken
from
the
2021
community
survey.
That
jaded,
which
was
the
first
one
ever,
and
one
thing
that
was
sort
of
interesting
to
me
specifically,
was
that
you
know
33
of
people
that
have
said
they
use
their
own
personal
time
to
to
work
on
on
itf
things,
which
is
surprisingly
high,
or
at
least
to
some
of
us
surprisingly
high.
Given
that
next
slide,
you
have
other
viewpoints
being
expressed
in
the
past.
So,
for
example,
already
bush,
who
some
of
you
might
know,
wrote
this
position
paper.
K
Let's
call
it
in
2005,
basically
calling
us
the
internet,
vendor
task
force
and
ranting
about
how
nobody
could
possibly
participate,
because
it's
a
full-time
job
and
and
we're
all
just
here
to
sort
of
build
features
into
routers
right,
and
maybe
both
of
these
are
true
or
maybe
something
has
changed.
The
scary
thing
is,
we
don't
actually
know
right.
So
so
we
don't
know
if
the
itf
is
now
more
focused
on.
K
You
know,
developing
things
that
matter
and
I'm
not
saying
that
you
know
nerd
notes
on
routers,
don't
matter,
but
maybe
other
things
matter
more,
but
we
really
don't
know
right,
and
these
are
very
individual
viewpoints
here.
33
basically
say
you
know,
my
own
time
goes
to
the
ietf
or
some
of
it
and
randy
says
you
know
it's
all.
It's
all
just
vendors
and
then
colloquially.
We
also
know
there's
no
operators
around,
although
they
are
operators
around.
So
we
really
have
little
data
next
slide
and
sort
of.
Why
do
we
care
about
this?
K
We
we
we
would
like
noncom
elected
into
various
bodies
and
positions.
Right
sort
of
I
had
two
quotes
here
about
the
profile
and
division
and
you
can
find
that
like
under
itf.org
about
right,
so
we're
supposedly
a
large
open
international
community
of
of
people
with
various
backgrounds
and
we
sort
of
care
about
the
evolution
of
the
internet
and
that
smoothly
operates,
and
we
have
a
mission
right
in
the
missions.
We
want
to
make
that
internet
work
better
by
doing
various
things
right.
K
K
You
know
diversity
of
participation,
for
example,
or
how
can
we
strengthen
underrepresented
parts
of
the
community
because
we
don't
have
any
ground
truth
here
right
and
and
as
a
small
aside,
we
obviously
also
want
to
enable
sort
of
evolution
of
those
two
things,
the
profile
and
the
mission,
if
you
will,
but
but
let's
take
that
on
the
side,
so
data
really
matters
and
we
don't
have
it
right.
Then
there's
a
two
sort
of
you
know:
mba
style
quotes,
but
if
you
can't
measure
it,
you
can't
improve
it,
which
is
certainly
true.
K
We're
measuring
things
we're
just
not
analyzing
it
enough
right.
We're
gathering
data
which
is
good
but
but
we
need
to.
We
need
to
actually
continuously
look
at
it
before
we
can
do
something
with
it
and
we're
not-
and
you
guys
are
now
starting,
which
is
great
and
the
reason
we
in
leadership
want
to
do
this
right,
because
there's
management
and
there's
leadership
and
there's
a
difference,
and
this
is
sort
of
what
the
second
quality
is
right.
K
But
it
would
be
even
better
to
understand
it's
sort
of
all
the
same
thing
and
you
know
what
what's
the
size
of
the
trunk
versus
the
size
of
the
body
versus
the
size
of
the
tail
and
I'm
hopefully
we're
going
to
get
some
data
out
of
this.
If
we're
continuing
down
this
process
of
analyzing
different
gathering
new
data-
and
I
think
that's
the
last
slide.
Thank
you.
K
I
A
I
I
To
me,
one
of
the
things
that
I
think
stood
out
in
all
of
the
presentations
in
all
of
the
the
five
papers
was,
you
know
the
desire
to
to
build
consensus
by
diversity
right.
We
need
a
diverse
set
of
opinions
to
build
consensus
within
the
ietf,
and
that
really
means
a
diversity
of
opinions
and
and
there's
a
number
of
barriers
to
that,
and
one
of
which
is,
can
we
even
measure?
What's
missing
right?
How
do
you
measure?
I
How
do
you
measure
the
diversity
of
what
we
have,
but
it's
even
harder
to
measure
what
we
don't
have,
so
you
know
there's
a
lot
of
different
types
of
diversity
that
we
need
to
be
able
to
look
at
right.
We
need
to
be
able
to
look
at
different
cultures.
We
need
to
be
able
to
look
at
different
genders
different
regionality
and
different
organizations,
and
especially
in
some
topics.
I
I
A
A
You
might
not
get
exactly
the
same
actors,
but
perhaps
you
could
see
how
people
interact
and
whether
you
see
the
same
patterns
emerge
subsequently.
One
could
also
see
whether
there
are
other
modes
of
participation
possible
in
the
process
and
where
that
works
for
us,
one
could
think
of
the
work
of
kai
jacobs.
That
has
done
the
work
of
integrating
users
in
the
in
the
standardization
process,
so
those
might
might
be
ways
of
thinking
about
it.
K
It's
really
rare
to
have
an
organization
where
you
have
like
decade-long
participants
from
the
same
individuals
over
very
many
different
affiliations,
because
they're
like
personally
involved.
But
it's
certainly
you
know
it's
not
impossible
to
like
compare
ourselves
to
3gpp
or
I
triple
e,
or
somebody
like
that.
K
I
U
Yeah
I
mean
the
question
that
you
raised
wes.
You
know,
I
think
to
me.
I
just
keep
coming
back
to
the
larger
question
for
me,
which
is
diversity
for
what
and
that's
kind
of
why.
I
think
this
application
of
looking
at
intersectional
feminism
is
interesting
because
we
we're
trying
to
actually
not
just
we're
not
doing
this
as
an
exercise.
Diversity
for
diversity
sake,
and
I
think
that
there's
two
things
that
implies
when
you
ask
diversity
for
what
you
know
one
is
you
sort
of
answered
right?
U
We
make
better
protocols.
We
have
you
know
comp
diversity
is
competency
is
the
sort
of
way
you
you
put
the
first
bucket
of
answers
in
and
then
the
second
one,
I
think,
is
it
actually
forces
us
to
do
better
work.
So
I
think
sometimes
there's
not
a
lot
of
participation
from
africa,
for
example,
because
the
things
that
we're
working
on
in
the
ietf
are
not
relevant
to
network
operators
in
africa.
It's
they
don't
see
our
work
as
interesting
to
them,
and
so
it
forces
us
to
reassess
you
know.
V
Yeah
I
wanted
to
make
a
point
about
the
diversity
of
opinions,
because
when
when
it
gets
to
that,
then
I
think
it
needs
to
be
well.
We
need
to
be
able
to
identify
threats
of
communication,
and
that
was
something
that
we
dealt
with
in
this
master
thesis
that
I
guided
where
we
were
trying
to
get
that
bob.
I
mean
I
just
wanted
to
point
out
that
this
is
not
trivial.
V
You
can
have
threads
that
merge.
You
have
kind
of
multiple
people
answering
you
know
well.
Well,
if
you
structure
it
can
be,
it
can
be
a
tree,
but
you
can
have
multiple
parents
in
the
conversation
you
can
have
things
merging
splitting
and
I
think
that
needs
to
be
a
part
of
identifying
different
viewpoints.
V
G
W
I
Paul
excellent
paul,
so
I'm
going
to
throw
a
topic
at
you
too,
because
you
might
be
able
to
give
us
more
background
in
one
particular
aspect
of
trying
to
measure
the
opinions
of
people
that
aren't
directly
participating.
Icann,
in
particular,
has
a
at
large
community
that
tries
to
measure
you
know
people
that
don't
exist.
I
don't
know
if
you
can
speak
to
that
paul
you're
you're
part
of
I
can,
but
that
may
not
be
anywhere
near
your
realm.
H
So,
thank
you,
wes,
and
no,
I
can't
actually
even
before
I
was
at
it
at
I
can,
which
I've
been
there
six
and
a
half
years.
I
did
try
to
look
at
that,
especially
in
the
at-large
community,
which
would
be
one
of
the
ones
that
is
structured
most
like
the
ietf.
That
is
pretty
much.
Anyone
can
join
very
low
entry
to
barrier
to
entry
and
such
like
that
and
with
working
groups.
H
They
start
to
contribute
and
for
some
reason,
drop
out
after
one
attempt
either
going
to
one
meeting
participating
on
one
thread
and
such
and
it
would.
I
would
be
fascinated
to
see
if
there's
any
research
on
what
would
be
considered
outliers
because
they
only
participate
briefly,
but
to
find
out
why
they're
only
participating
briefly,
certainly
in
the
ietf,
we
have
seen
you
know
going
to
the
question
of
gender
diversity.
H
You
know,
I'm
not
the
only
person
who's
heard
from
a
woman
at
a
meeting
saying
I'm
not
coming
back
this
sucks
that
doesn't
prevent
participation,
because
there
are
mailing
lists
and
such
like
that,
but
certainly-
and
I
can,
where
there's
much
better
gender
diversity
and
such
like
that,
we
still
get
people
who
will
come
in
start
to
work
on
something
and
go.
You
know
what
this
isn't.
What
I
thought
this
isn't
as
interesting
as
I
thought
and
in
a
volunteer
organization.
I
Yeah,
that's
a
good
point
and
I
think
tailing
on
that
is
one
thing
that
we
haven't
really
talked
about
in
in
this
slot.
Yet,
which
is
the
ietf
is
interesting
because
it's
an
open
meeting
where
anybody
can
join
right.
We've
always
said
that
there's
no
membership,
which
is
fantastic.
I've
actually
heard
complaints
about
icann
that
the
only
thing
you're
going
to
join
in
icann
is
at
large
and
any
other
group
within
icann
requires
that
you
have
some
other
role
even
to
get
in
the
door
and
be
part
of
those
discussions.
I
But
one
interesting
thing
about
icann
is
that
they
actually
meet
around
the
world
and
they
force
themselves
to
move
to
all
the
continents
where
the
ietf
doesn't
do
that,
and
the
ietf
has
this
chicken
and
an
egg
problem
of.
We
really
only
go
to
three
continents
a
year
and
we've
made
an
exception
a
couple
of
times
where
we've
gone
to
australia
once
and
we've
gone
to,
which
is
oceana
and
we've
gone
to
south
america
once
and
how
can
we
measure
those
we've
never
been
to
africa?
As
far
as
I
know
how?
I
C
Yeah,
I
just
wanted
to
make
one
observation,
so
so
obviously
diversity
is
hugely
important
and
we
need
to
be
able
to
have
a
diverse
discussion
group
in
order
to
get
good
decisions
and
not
just
along
one
axis
but
multiple
axis
of
diversity,
but
but
it
still
feels
a
little
bit
like
there's
something
missing,
because
I
think
this
is
actually
a
sort
of
multi-level
problem.
We
talked
about
that
a
little
bit
in
the
context
of,
or
affiliation
and
priyanka,
make
this
comment
that
it's
it's.
It
goes
beyond
affiliate
affiliation.
C
The
same
thing
I
I
think
applies
to
some
extent
to
gender
and
location
and
so
forth.
So
you
might
actually
find
out
that
there
are
this
interest
that
people
have
or
or
they're
you
know
groups
behind
them
or
you
belong
to
us.
You
know
set
of
friends
who
believe
in
a
particular
way
for
some
topic.
Let's
say
you
know:
what's
your
orientation
towards
data
collection
in
the
internet
and
and
those
intro
diversity
in
those
interests
is,
is
also
hugely
important.
C
Of
course
it's
completely
impossible
to
measure,
but
but
I
feel
like
it's
not
that
we
just
get
like
a
representation
from
different
diversity
groups
and
then
we're
done
because
it's
it's
only
a
proxy
of
some
sort
and
the
real
beef
is
is
maybe
maybe
behind
what
you
know.
People
of
a
particular
gender
or
people
from
a
particular
location
or
continent
feel
is
the
problem.
But
if,
if
you
just
mix
people
in
in
the
wrong
way,
you
won't
actually
get
it.
So
I
don't
know
what
to
do
about
it.
N
The
things
that
we
haven't
spoken
about,
which
obviously
is
really
interesting
and
relevant,
especially
speaking
to
paul's,
point
around.
How
do
we
measure
who
stays
involved
right
and
like
what
part
of
organizational
culture
are
implicated
in
that?
I
think,
in
addition
to
the
kind
of
you
know,
data
and
analysis
that
or
the
kind
of
analysis
that
can
be
done
on
the
data
that
comes
from
the
mating
list
and
that
comes
from
sort
of
the
the
stuff
that
we
can
see
online
is
is
part
of
that
answer.
N
When
there
are
a
lot
of
options
for
doing
work
elsewhere.
Right,
like
the
itf,
obviously
is
still
a
hugely
important
organization.
But
there
are
many
other
places
where
you
can
take
your
your
stem
degree
and
have
an
interesting
career
that
are
perhaps
not
as
uncomfortable
to
work
in.
So
that's
just
one
of
the
things
that
I
wanted
to
raise
that,
in
addition
to
the
quantitative,
more
quantitatively
oriented
data
and
methods
that
we're
discussing
today,
I
think.
S
I
Something
you
point
out
korean
is
that
we're
we
might
be
able
to
measure
that
people
did
drop
off.
We
might
be
able
to
measure
that
there
are
these
short-term
spikes
of
people,
but
we
don't
have
the
ground
truth
as
to
why
so,
certainly
the
atf
surveys
that
you
know
go
out.
Every
single
time
are
helpful
in
that
regard,
but
we
don't
really
have
any
data
that
allows
us
to
mine.
You
know
why
people
aren't
staying.
I
We
might
be
able
to
mind
why
people
are
staying
if
we
send
out
new
survey
questions,
but
that
would
be
something
possibly
to
work
with
with
jay
on
for
future
questions
or
the
iesga
as
well
yary,
you
have
an
old
hand.
I
believe
anybody
else
have
thoughts
on
this
subject.
I
think
it's
one
of
the
more
challenging
ones
of
this
workshop
and
also,
coincidentally,
one
of
the
most
important.
O
O
However,
you
can
see,
for
example,
whether
people
participate
under
participation
results
in
further
engagement
or
you
know
I
send
an
email
to
the
mailing
list
and
just
get
silenced
by
response,
so
there
might
be
a
small
things
like
this.
That
can
help
you
to
get
some
inference
and,
of
course
it
won't
be
on
everybody.
O
But
since
the
mailings
are
quite
comprehensive
and
there
is
quite
a
few
number
of
people,
you
might
get
an
idea
of
what's
going
on
and
in
particular,
if
you
correlate
that
with
the
occurrence
of
meetings,
you
may
get
a
little
bit
extra.
Also,
you
could
get
maybe
participation
of
the
particular
people
in
the
physical
meetings
that
might
help
give
you
a
little
bit
of
light
on
it.
I
You
know
you
bring
up
so
you've
suffered
a
thought
in
my
mind
that
I
think
that
there
has
been
past
natural
language
processing
work
to
identify
emotions,
and
it
may
be
that
we
could
actually
do.
You
know
feed
all
the
mailing
lists
in
and
identify
the
really
emotional
messages
and
see
if,
if
some
of
the
participants
in
that
emotional
stream
suddenly
just
went
away,
I've
certainly
heard
of
cases
where
that
has
happened.
So
it'd
be
interesting
to
see
if
we
could
actually
extract
that.
Q
I
And
we
need
both
sides
of
of
such
a
data
mining
right,
not
just
the
negative,
but
also
the
the
ones
where
they
send
a
lot
of
heart
emojis,
because
they
just
really
love
the
atf
right.
Those
should
exist
too,
but
anybody
else
with
opinions
or
questions
things
we're
not
considering
groups
we're
not
considering.
That's
that's
the
hard
one
right
who
are
we
missing.
D
Yeah,
I
don't
want
to
stop
you
from
I'm
going
to
the
ranks,
but
I'm
recently
talk
more
often
to
policymakers
as
well,
and
this
is
even
harder
because
you
know
they
don't
have
the
time
and
the
willingness
to
come
and
engage,
but
they
still
would
like
to
have
a
bridge.
They
still
would
like
to
understand
what's
happening,
so
I
think
it's
also
not
about
participation.
Only
if
we
talk
about
diversity,
it's
also
how
we
get
other
communities
informed
and
how
we
can
keep
a
discussion
or
a
communication
going
between
other
communities,
but.
A
But
but
in
response
to
that
maria-
and
I
don't
wanna
want
to
reiterate
my
previous
point-
but
I'm
going
to
do
that
anyhow.
So
it
might
be
interesting
to
to
to
then
see,
because
we
know
that
there
are
other
people
and
groups
in
icann
right
to
to
simply
see
not
saying
that
it's
a
total
comparison
but
compare
of
three
gpp
ietf
and
icann
and
see
who
is
present
and
who
it
then
has
impact
and
engages
in
the
discussion
in
ways
and
how
those
networks
play
out
and
where
they
are
very
different.
A
So
and
then,
then
we
can
hone
in
with
more
qualitative
or
discursive
methods
to
analyze
where,
where
the
big
changes
are
or
maybe
see
that
it
are
the
same
six
people
in
working
groups
all
over
all
over
the
bodies
with
largely
the
same
corporations
that
are
the
same
dominant
corporations
in
that
sector.
That
that
actually
make
the
decisions
or
not
right.
H
H
H
Are
we
doing
interesting
work
enough,
interesting
work
that
people
would
come
and
volunteer,
and
I
can
because
we
are
trying
to
not
become
irrelevant
and
we
have
money
to
invest
in
not
becoming
irrelevant,
spend
money
on
trying
to
keep
people
interested.
It's
very
different.
I
don't
know
where
3gpp
is,
but
I
suspect,
they're
not
as
aggressive
as
even
as
I
can
is
on
on
that
sort
of
thing
and
other
folks
here
certainly
know
much
more
about
3gpp,
so
I'll
shut
up
here
now.
I
You
you
bring
up
good
points
paul,
and
so
thank
you
for
that.
You
know
if,
if
you
have
money
to
spend
becoming
more
diverse,
you're
likely
to
be
more
successful,
certainly-
and
you
know
we
do
have
the
guides
program
in
the
atf,
where
we
really
do
try
and
get
people
to
come
back,
but
we
don't.
You
know
it's
a
very
short-term
thing.
That's
spun
up
right
around
convention
time
then
doesn't
often
last
long.
I
I
will
close
today
with
with
one
one
of
my
earliest
memories
in
the
ietf
was
was
after
I
participated
and
did
a
few
things
and
made
some
comments.
Somebody
actually
came
up
to
me
and
said.
I
hope
that
you're
going
to
stick
around
and
I'd
really
like
to
see
you
as
a
working
group
chair
or
something
like
that
and
really
promoted
me
to
you
know
to
wanting
me
to
to
continue-
and
I
remember
that
and
even
though
it
was
probably
25
years
ago
or
something
that
memory
still
sticks
out
in
my
head.
B
A
Before
we
go
to
break
which
will
be
30
minutes,
I
propose
that
we're
going
to
merge
the
chat
in
here
and
in
slack,
so
we
don't
have
two
streams
of
communication,
so
people
seem
to
most
like
slack.
So
let's
have
let's
move
the
chat
there,
so
you
have
30
minutes
to
join
that.
If
that
doesn't
work,
just
point
it
into
the
chat
here.
I
D
Actually
know
what
he
wanted
to
say,
so
maybe
I
can
just
go
and
say
this,
so
we
have
prepared
a
little
google
doc
and
I
can
put
the
link
into
the
webex
chat
at
this
point
still,
and
this
is
starting
to
organize
the
hacking
groups.
So
I
just
randomly
put
some
groups
there
based
on
the
papers
based
on
the
discussion
and
so
on,
but
you're
welcome
to
add
additional
groups.
D
If
you
think
there's
something
missing
that
you
would
like
to
work
on
or
you
can
even
start
signing
up
yourself
to
one
of
those
groups,
so
have
a
look
at
the
document
think
about
it
and
we
still
have
one
more
session
to
go
before
we
talk
about
the
hacking
groups,
but
for
the
right
point
of
time
to
look
at
this
right
now.
Yeah.
U
K
D
So
we
have,
we
have
a
whole
session
on
that
right,
so
one
more
session
where
we
talk
about
decision
making,
I
think
and
then
we
have
a
whole
session
to
organize
ourselves
into
the
into
the
group.
So
you
don't
have
to
sign
up
right
now,
but
you
can
think
about
already
where
you
would
like
to
sign
up
or
you
can
add
new
topics
and
then
we
can
discuss
it
later.
I
D
W
W
W
Now
we
had
seven
submissions
that
fit
into
this
category,
and
I
I
wanted
to
leave
plenty
of
time
for
discussion
in
the
session
so,
rather
than
have
everyone
try
and
present
their
work.
What
I
did
this
time
was.
I
I
asked
michael
wessel
and
ignacio
castro
to
prepare
a
summary
of
of
what
they
saw
as
the
key
points
of
the
various
presentations
and
trying
to
ask
some
provocative
questions
to
get
things
started.
W
W
V
V
V
The
impact
of
okay
so
very
easy,
the
next
being
ours.
This
is
just
this
is
just
raising
a
question.
We
haven't
done
anything
on
that
yet
to
see
if
it
would
be
possible
to
better
understand
how
we
get
from
an
id
to
an
rfc,
as
you
would
have
seen
with
at
least
I
have
just
added
myself
to
group
five,
that's
connected
to
film
to
this
just
to
better
understand.
Why
would
some
ideas
fail
in
case
of
resistance?
You
know,
would
it
be
that
people
are
asking
for
more
inputs?
V
So
this
is
more
of
future
thoughts
and
all
related
to
nlp
characterizing,
the
iotf
through
the
lens
of
rfc
deployment.
Well,
that's
really
your
work.
What
should
I
say
about
that?
I
mean
it's
a
pretty
long
paper
giving
giving
a
pretty
detailed
analysis.
I
found
it
very
interesting
to
read,
but
I'm
not
sure
what
I
should
be
saying
about
it
here,
talks
about
trends
in
rfc
production
factors
of
rfc
success.
V
Maybe
one
thing
worth
pointing
out
for
people
here
that
they
probably
don't
know
if
they
haven't
read.
The
paper
is
that
this
paper
points
at
another
paper
which
has
done
a
very
long
and
thorough
manual
analysis
of
success,
of
rfcs
trying
to
understand
which
ones
are
deployed
and
not
deployed,
and
you
know
categorizing
this
there's
an
online
excel
sheet
from
that
paper
that
that
documents
all
this
effort
and
that
was
used
as
input
for
a
part
of
this
of
this
particular
paper
here.
V
V
Tools
for
email
analysis,
the
challenges
of
yeah
that
cross
document
core
reference
was
illusion
in
email
that
looked
to
me
like
an
nlp
paper,
looking
at
an
interesting
topic
in
lp,
maybe
a
bit
detached
from
what
the
other
works
here
have
been
doing,
just
showing
some
progress
on
how
how
well
there
is
this
general
problem
of
entity
detection.
I
think,
if
that's
in
the
right
name,
for
it
see
yeah
core
reference
resolution,
I
mean
just
understanding
whether
whether
there
is
an
entity
identified.
V
My
understanding
is
what
was
that
this
was
only
done
on
the
under
and
one
email
corpus,
which
is
a
standard
email
body
in
nlp,
but
hasn't
yet
been
applied
to
the
ietf.
So
that's
the
relevance
here.
It's
good
input
right
for
people
wanting
to
do
nlp
on
the
email,
archives
of
the
itf
and
then
the
document
called
research
proposal
right
that
describes
prior
work
on
from
iot
using.
V
V
V
So
next,
please,
okay,
the
first
one
here
describes
it's
just
a
description
of
the
upcoming
phd
thesis
that
we're
going
to
advise
when
that
person
is
in
place.
In
fact,
today
I
have
sent
an
email
to
somebody
saying
that
we
we
can
offer
you
the
job.
It's
been
a
long
process
with
many
many
administrative
hurdles.
V
So
if
everything
goes
well
in
a
few
months,
we're
gonna
have
somebody,
but
it's
just
a
futuristic.
I
mean
it's.
What
we
want
to
do
this
is
about
education,
so
the
idea
here
would
be
that
by
applying
nlp
on
emails,
we
might
be
able
to
get
better
educational
source
material
to
be
able
to
understand
why
certain
decisions
have
been
made.
That's
a
really
nlp,
not
so
much
about
the
procedure
more
about
the
content
and
then
there's
rfc's
change.
That
was
also
quite
interesting.
V
This
talks
about
the
tool
that
paul
hoffman
has
written
that
is
at
this
url
and
the
point
of
that
tool
is
to
well.
Essentially,
if
you
look
at
an
rfc,
then
the
rfc
header
is
going
to
tell
you
that
this
obsoletes
other
rcs
updates
other
rc's.
At
least
the
data
tracker
does
tells
you
that,
but
it's
not
so
easy
to
figure
out
which
future
drafts
or
drafts
that
are
out.
There
already
would,
for
instance,
obsolete
this
rfc
or
are
in
some
way
related
to
this
rfc.
V
W
I
mean,
and
and
thank
you
for
for
summarizing
so
before
igness
jumps
and
do
any
of
the
the
authors
of
the
other
presentations
want
one
to
jump
into
this
point
and
give
us
the
other
submissions
yeah.
My
apologies.
If
I
misrepresented
somebody
I
I
thought
it
was
a
really
nice
summer,
oh
jump
in
whenever
you
want
a.
H
Very
quick
note,
because
the
tools
I'm
talking
about
here,
I
don't
think,
are
as
relevant
to
the
workshop
as
some
of
the
others.
The
second
tool
that
I
discussed
is
actually
it
is
I'm
sorry.
The
second
topic
that
I
discussed
is
will
be
a
separate
tool,
so
the
cli
tool
already
exists.
Many
folks
here
use
it.
O
Then,
very
briefly
talk
one
sec
about
our
contribution,
so
that
paper,
which
we
have
posted
in
slack,
is
a
part
of
a
larger
project.
We
are
trying
to
understand
the
idf
and
it's
decision
making.
So
the
paper
is
just
a
teaser
of
what
we
are
doing
in
the
project
and
if
you
don't
mind
calling
I'm
going
to
try
to
share
my
screen,
because
I
think
that
maybe
I
sent
the
wrong
version
of
the
slides
and
I
have
a
more
recent
one.
Okay.
O
So
well,
thank
you,
michael
for
that
summary
steaming
from
that
summary.
We
were
thinking
what
are
the
overarching
topics
and
from
what
we
could
see
well,
one
of
them
is
what
are
the
questions
that
we
are
trying
to
answer
and
some
of
those
questions
seem
to
map
in
success
versus
failure.
What
is
a
good
rfc
or
a
successful
rfc?
Maybe
it's
not
the
same
thing
and
what
makes
the
process
of
rfc
making
successful
or
better
with
regards
to
decision
making,
which
is
not
necessarily
the
same
as
an
rfc
itself?
O
O
Also,
when
is
a
decision
made?
Can
we
infer
maybe
from
the
videos,
maybe
from
the
males
when
different
people
have
arrived
to
a
decision
that
gets
translated
into
a
into
a
change
in
in
the
draft
or
to
the
final
publication
or
that
option
of
it,
and
also
what
what
is
the
decision
that
has
been
made
and
related
to?
O
This
is
where
can
we
find
these
answers,
and
I
think
that
most
of
us
has
looked
pretty
much
at
the
mails,
probably
because
that's
one
of
the
most
griply
sources
of
information
on
one
of
those
sources
that
we
can
more
easily
tackle.
However,
there
are
also
fcs
with
the
drafts
and
all
the
versions
of
the
drafts
that
can
be
tackled.
O
They're
also
meeting
minutes
also
the
meeting
audio
video
information
in
the
git
repositories
and,
of
course,
this
information
is
not
as
easy
to
access
as
the
email
might
require
much
more
purpose,
pre-processing
or
might
be
less
representative
of
all
the
groups
like,
for
example,
some
groups
are
heavily
using
it,
but
not
all
of
them
are
using
information
from
the
meeting
and
the
in
terms
of
audio
video
might
be
quite
rich.
But,
of
course
pros
in
it
is
a
work
in
itself
and
there's
also
interesting
external
data.
O
That
again
is
also
sometimes
complicated
to
handle
but
or
obtained,
but
once
we
obtain
it,
we
could
all
benefit
from
it
as
a
community,
and
this
could
be
training
from
service
like,
for
example,
we're
planning
to
make
a
survey
to
understand
better
adoption
of
rfcs.
This
was
a
problem
that
we
had
in
our
paper.
O
O
People
to
manually
label
when
a
decision
has
been
made
might
be
useful,
also
for
understanding,
which
is
the
entity
that
is
being
referred
inside
of
the
mails,
which
was
also
mentioned
in
by
one
of
the
works
in
this
thread,
but
there
might
be
also
others
like,
for
example,
a
data
on
patents,
data
on
citations,
which
might
be
quite
relevant
and,
of
course,
one
of
the
things
that
I
think
that
we
are
all
converting
here
is.
How
can
we
manage
to
extract
that
information?
W
W
That's
something
while
we're
waiting
for
people
to
join.
I
thought
that
was
that
last
point
was
a
really
interesting
one
about
the
limits
of
what
we
can
measure
in
terms
of
deployment.
So
you
know
we
we
can
measure
what
what
the
itf
does,
but
once
something
is,
is
finished
whether
it
gets
used
or
not,
is
a
really
difficult
challenge
to
understand.
But
do
you
have
thoughts
on
how
to
go
that
takes
the
data
from
from
the
publication
process
to
what
happens
next.
O
So,
for
example,
with
regards
to
relevance
or
implementation
to
a
particular
rfc
or
technology
yeah.
That's
a
very
good
question
is
that's
something
very
hard
to
to
tackle.
First
of
all,
what
is
a
good
rfc
or
what
is
implementation?
Or
can
you
compare
implementation
of
different
things
like
something
might
be
implemented
across
the
whole
layer
and
something
something
else
might
be
constrained
to
a
particular
niche
where
it
might
be
part,
a
might
be
very
relevant.
O
I'm
not
quite
sure
I
mean
one
thing
that
you
can
do
is
we
can
you
can
use
that
data
set
from
the
paper
that
we
mentioned
in
ours
where
they
did
some
manual
labeling?
Maybe
you
can
look
for
whether
a
particular
draft
or
fc
is
mentioned
in
nano
or
other
network
operator
lists,
though
of
course
that
does
not
necessarily
mean
that
it's
implemented
just
means
that
it's
being
discussed
whether
there
are
articles
in
wikipedia
about
it.
A
O
Yeah,
that's
a
really
cool
idea.
I
guess
that
they
I
mean
I
I
can.
I
guess
that
you
could
do
part
of
trying
to
map
different
implementations
to
each
other.
Based
on
similarity
of
the
code.
I
guess
that
what
would
be
difficult
is
to
find
save
for
a
particular
implementation
or
of
a
given
protocol
how
to
map
one
to
the
other,
so
an
implementation
of
a
quick.
Let's
say
like
maybe
it's
well
described
in
the
github,
but
maybe
it's
not
so
clear.
O
A
G
Yeah,
so
I
just
wanted
to
kind
of
follow
up
on
this
point,
because
we
were
kind
of
very
interested
in
this
data
set,
because
we
think
the
the
natural
language
quite
complicated
and
one
of
the
questions
I
had
when
hearing
this
discussion
is
how
good
or
how
accurate
do.
We
need
the
extractions
to
be
to
start
making
conclusions
and
that's
something
that
I
I've
been
kind
of
listening
and
trying
to
understand
a
little
bit
through
the
course
of
our
conversation
is
like
to
what
level
of
quality
do.
G
O
I
would
guess
that
the
the
one
of
problems
that
you
have
is
that
they,
you
frequently
need
to
have
some
label
data,
so
when
you
make
those
inferences,
you
know
that
if
I
have
this
piece
of
data-
and
I
have
it
labeled
as
talking
about
x
or
referring
about
a
particular
entity
when
I
apply
nlp
algorithm
x
when
it's
beyond
this
level,
it's
a
pretty
good
inference
that
this
is
actually
the
case
and
frequently.
The
problem
is
that
they
we
just
don't
have
that
label
data.
G
O
Yeah,
to
be
honest,
I
don't
have.
I
don't
have
a
perfect
answer
for
that.
I
guess
that
at
the
very
least
you
can
observe
trends,
so
you
might
not
know
whether
the
mapping
is
precise
enough
to
know
whether
you
can
categorically
say
one
thing
or
the
other,
but
you
can
see
that
there
is
a
trend
and
that
might
be
helpful
enough
in
shedding
some
light
into
the
particular
question
that
you
are
trying
to
answer.
G
I
I
think
it's
clear
that
the
more
labeled
data
we
get
the
the
nicer
it
is
right
for
for
us
to
do
our
job,
but
I
also
think
like
one
of
the
actually
interesting
from
a
very
just
nlp
perspective
from
a
technical
perspective.
One
of
the
very
interesting
thing
about
this
domain
is
that
you
have
a
lot
of
change
and
it's
very
long
tail
right.
G
So
that's
why
it's
kind
of
exciting
about
ietf
data
is
the
fact
that
it's
very
messy
and
very
long
tail,
and
it's
not
like
what
you
see
in
other
domains.
So
in
some
sense,
annotated
data
is
great
if
you're
going
to
have
actual
answers
to
to
problems,
but
from
a
kind
of
a
lp
perspective,
it's
kind
of
fun
to
work
on
domains
that
are
actually
super
hard,
which
I
think
the
ietf
data
is.
Q
Yes,
as
an
nlp
expert,
I
have
analyzed
the
linux
kernel
mailing
list,
which
is
even
worse
because
you
have
patches
code
patches
in
messages,
so
that's
actually
much
harder.
The
itf
data
mailing
list
messages
are
probably
better
in
that
sense
and
that
a
lot
of
the
text
is
user
generated
human
generated.
Q
O
Okay,
so
one
of
the
very
cool
things
of
the
itf
is
that
you
have
many
layers
of
data,
so
you
can
see
the
conversation
in
the
mail
list
and
then
you
can
see
that
maybe
a
change
has
happened
in
the
draft.
So
ideally
you
can
map
that
conversation
in
the
change
in
the
draft.
So
that's
something
very,
very
cool
about
the
idf
and
I
guess
that
any
additional
layer
of
this
type
of
data
that
can
be
provided
is
super
helpful.
O
So,
for
example,
those
blue
sheets
with
participation
in
the
live
meetings
if
they
were
to
be
not
a
pdf
that
is
hard
to
scan
and
it
could
be
like
in
a
standard
format
or
something
like
this.
Probably
that
would
be
very
helpful,
because
sometimes
what
happens
is
that,
instead
of
addressing
the
cool
question,
you
spend
half
of
the
time
just
trying
to
gather
the
preliminary
data
source.
O
Another
thing
similar
to
that
like
at
the
moment
we
have
affiliations
of
draft
authors,
but
they
frequently
is
just
in
the
draft,
so
you
need
to
manually
parse
it
from
the
text.
So
if
that
was
in
the
data
tracker,
I
don't
know.
I
think
that
recently
it
is
that's
very
helpful
and
the
affiliation
of
the
people
attending
the
meeting
that
appears
in
the
blue.
If
that
was
in
the
tata
tracker
2.
That
would
be
also
quite
helpful
for
all
the
people
that
have
been
talking
before
about
the
interesting
things
that
stem
from
affiliation.
H
B
H
Some
of
us
are
familiar
where
an
rfc
is
published,
but
a
bunch
of
developers
think
that
it's
wrong,
so
they'll
actually
implement
something
close,
but
not
actually
it
because
they
didn't
like
the
way
the
rfc
came
out
and
then
interoperability,
which
is,
I
think,
what
many
of
us
are
really
hoping
for.
I
going
back
to
a
discussion
from
earlier
today
where
there
are
things
like
lars
put
up
the
quick
features
chart
you
know
who
who
has
implemented
various
features.
H
It's
not
sufficient,
though,
and
again
going
back
to
the
old
days
of
my
old
days
of
ipsec,
there
were
rfcs
that
were
published
that
were
explicitly
only
for
about
three
developers.
That
is
the
other
other
folks
were
not
interested
in
doing
this.
They
didn't
want
to
stop
it
from
being
done,
but
they
pulled
out
of
the
discussion
because
it
was
an
extension
to
in
these
case.
Most
cases
were
for
ike
v2.
They
were
extensions
that
they
didn't
want
to
offer
to
their
their
customers.
H
They
didn't,
they
didn't
see
the
purpose
to
it,
so
that
wouldn't
a
low
adoption
would
not
indicate
a
failure
if
it
was
in
fact
adopted
by
the
folks
who
had
participated
in
the
working
group
discussion
or
I'm
sorry,
the
developers,
especially
who
had
participated
in
the
work
group
discussion.
So
I
don't
know
I
I
again
I
said
earlier.
H
I
would
like
to
see
some
way
of
the
ietf,
or
at
least
you
know,
either
formally
or
informally,
corralling
those
those
feature
charts,
especially
the
ones
that
are
kept
up
to
date,
as
as
a
tool
for
researchers
to
look
at
success
for
some
of
the
things
and
just
this
morning,
actually
eric
vinky
who's
on
on
the
call
noted
that
even
when
a
thing
is,
has
an
rfc
and
basically
know
what
he
implements.
We
still
will
keep
it
around.
W
V
Yeah,
I
just
want
to
answer
on
that.
On
that
thing,
with
feature
charts,
I
mentioned
before
that
there
is
this
online
extra
sheet
that
some
people
have
constructed,
and
I
have
I
mean,
having
looked
at
it,
I
have
seen
that
they
have
specifically
looked
at
feature
charts
as
one
of
the
inputs
to
decide
that
this
has
been
deployed.
V
So
this
is
maybe
interesting.
What
I
can
do
is
I
can
dig
it
up
from
the
paper
because
actually
paper
from
colleen,
ignacio
and
all
these
others
that
has
made
me
look
at
the
other
paper
as
well,
and
that
has
to
link
to
this
extra
sheet
so
I'll
I'll,
dig
it
up
and
send
it
to
the
slack.
It's
really
interesting.
P
Regarding
this
discussion
about
measuring
implementation,
success,
failure
of
rfcs,
so
it
seems
that,
like
there
is
an
ambitious
and
difficult
way
of
doing,
it
is
actually
trying
to
measure
what's
done
in
implementations
and
then
there
might
be
a
cheaper
way
and
then
like
it's.
I
would
be
curious
about
whether
there's
any
value
and
that
what
your
thoughts
about
this
is
basically
looking
at
bibliometric
indications,
excitations
to
the
rfc,
from
scientific
literature,
references
from
other
rfcs
or
from
non-iatf
standards,
non-patent,
literature,
prior
art,
citations
from
new
patent
applications.
P
So
they
are
all
obviously
very
different
in
nature
but
like
if
you
correlate
these,
and
you
see
that
along
all
these
different
dimensions,
some
rfcs
like
really
stand
out
and
others
don't
like
it's.
That
probably
gives
you
an
indication
of
impact
right.
I
mean
like
it's
a
and
and
that
that
seems
to
be
data
that
is
systematically
available
for
all
rfcs
and
it's
less
context,
dependent
and
probably
easier
to
collect.
W
O
But
this
is
you:
were
you
were
spot
on,
so
we
have
actually
a
look
at
all
those
features
to
try
to
predict.
So
in
the
previous
paper
and
another
one
that
we
have
now
under
submission.
We
have
tried
to
predict
a
deployment,
a
publication
of
a
draft,
a
an
adoption
of
a
draft
by
a
given
working
group,
and
indeed
we
have
used
some
of
some
of
the
metrics
that
you
are
saying
in
particular
citations.
O
We
haven't
used
patents
just
because
it's
a
bit
more
complicated
than
we
would
have
first
to
crawl,
the
database
or
patents
to
identify
patterns
that
are
related
or
to
the
belong
to
the
affiliation
of
the
outhouse
road
on
by
those
authors
that
participate
in
a
given
draft.
But
that's
a
really
good
point.
There
is
some
work,
though,
from
the
economist
from
economics
trying
to
see
the
impact
of
patents
on
supporting
or
not
particular
drafts
in
the
idf.
W
All
right
so
so,
we've
obviously
said
a
lot
about
documents
and
outcomes,
and
so
on.
One
of
the
other
focuses
in
this
session
was
about
the
decision
making.
Are
there
any
things
which
make
it
especially
hard
to
model
the
type
of
decisions
that
the
itf
is
making
yeah?
What
are
the
challenges
in
this
space
and
what
types
of
decisions
are
people
interested
in
modeling.
O
Yeah,
I
might
have
some
yeah
well,
I
guess
it's
quite
useful
when
the
itf
has
various
standard
comments
through
the
mailies.
Like
a
does
people
support
this
rfc,
so
we
publish
it
is
how
we
adopt
it,
because
of
course
that
makes
easy
to
parse
the
process
of
the
decision
making.
So
anything
on
that
line
is
is
quite
helpful.
O
I
guess
the
main
decisions
that
at
least
from
my
side
are
particularly
interesting
in
is
whether
a
draft
is
adopted
by
a
given
working
group
and
whether
people
agree
or
disagree
on
particular
technical
features,
and
whether
people
want
further
explanations
from
them.
Please,
those
are
the
most
obvious
ones
that
come
to
mind.
O
So
I
would
be
also
curious
to
hear
from
the
side
of
the
people
from
the
itf.
What
are
the
problems
that
they
see
in
the
decision
making?
What
are
the
aspects
where
maybe
more
light
is
needed
or
more
help
is
needed?
I
know
colleen,
you
could
have
mentioned
sometimes
a
a
concerned
review.
So
maybe
these
are
also
things
where
people
from
on
this
side
of
the
arena
can
help
with.
W
E
So
I
will
ask
for
something
very
hard
and
it's
easy
to
to
look
well.
It
is
easier
to
look
at
the
kind
of
concrete
artifacts
that
you
mentioned
when
somebody
makes
a
call
for
adoption
or
something
you
know,
gets
a
a
publication
request
or
something
a
a
concrete
review
comes
in
through
the
review
tool.
Those
are
are
good,
easy
ways
to
see
at
least
what
point
in
time
a
a
decision
is
being
made.
E
K
With
my
isg
head
on
right,
so
some
of
the
things
that
that
I
think
would
be
useful
for
the
ifg,
so
one
thing
that
we
sort
of
anecdotally
hear
a
lot
is
that
you
know
the
itf
is
slow.
That
was
also
from
the
survey.
Lots
of
people
seem
to
believe
that
the
status
quo
is
taking
a
long
time
and
a
few
years
ago
or
many
years
ago
we
could
sort
of
confidently
more
or
less
point
to
the
rfc
editor
and
say
they're
slow,
but
that's
not
no
longer
the
case,
they're
actually
quite
fast.
K
The
on
the
rfc
editor
side,
the
slowest
thing
I
think
that
we're
seeing
is
the
authors
responding
during
auth
48,
so
which
is
good,
but
on
the
ietf
side,
it's
it's
much
less
clear,
sometimes
where
the
delays
are
right,
and
some
of
that
might
be
that
the
data
tracker
doesn't
necessarily
make
it
very
clear
who
holds
an
action
item
for
a
certain
document.
For
example:
is
it
with
the
chair?
Is
it
with
the
authors?
This
is
the
area
director
and
even
attributing
that
is
sort
of
difficult.
So
it's
so.
K
K
Some
groups
seem
to
be
very
clocked
by
the
meetings
three
times
a
year,
and
you
see
like
three
updates
pretty
much.
You
know
very
evenly
spaced
other
groups
are
completely
different
and
we
don't
really
know
how
many
of
the
former
group
source
the
letter
groups
we
have
or
if
there's
other
patterns,
sort
of
sort
of
insight
into
how
can
we
sort
of
streamline
the
process
so
that
the
the
community
can
work
better
and
faster?
I
think
we'll
be
able
to
look
at
the
isg.
O
I
think
that
that
was
a
very
good
point
like
who
has
the
accent
item,
because
I
guess
that
well,
I
see
it
myself
in
any
email.
Communications
very
often
is
not
clear.
I
was
wondering
if,
from
your
point
of
view
or
the
decision
and
the
clarity
on
who
has
the
action
item
has
improved
for
those
groups
relying
on
kit.
K
Or
it's
a
lot
of
work
on
the
chairs
and
the
authors
to
pull
email
back
into
the
issue
tracker.
So
I
don't
know
if
it
it
might
help.
But
I
think
the
working
groups
that
ca
use
for
git
are
sort
of
pretty
actively
tracking
things
anyway,
because
that's
what
I
use
git,
I
don't
know
if
it
really
helps
for
casual
groups,
so
much
so
for
gcbm,
for
example,
which
is
maybe
a
bit
slower
than
other
groups.
I
sort
of
use
git
for
for
my
documents,
but
it
doesn't
really
seem
to
speed
anybody
else
up.
E
E
K
Is
only
there
for
the
isp
stage
or
the
just
the
isg
stuff,
I
don't.
I
think
it
needs
to
show
up
in
more
places
or
something
right.
I
don't
think
it's
visible
enough
that
it
really
sort
of
seems
to
have
made
a
big
difference
yet
but
yeah,
but
I
mean
one
thing-
is
sort
of
improvements
to
the
data
tracker,
but
then
there's
also
simply
looking
at
the
data
we
have
and
trying
to
figure
out.
K
E
W
We've
had
a
lot
of
talk
about
particular
focus
and
obviously
there
were
a
broader
range
of
submissions
into
this
session,
and
I
I
think,
we've
probably
touched
on
the
most.
Is
there
anything
that
that
the
theaters
of
the
submissions,
which
has
been
a
lot
less
sort
of
discussed,
want
to
bring
up
very
quickly
before
we
finish.
K
D
D
I
made
some
notes,
but
I
didn't
update
these
slides
and
I
think
we
should
try
to
tackle
some
of
these
questions
in
the
next
couple
of
days
and
then
meet
on
thursday
again
and
see
if
we
could
reach
anything
or
if
we
have
just
more
questions,
which
might
also
be
a
valid
outcome
before
we
actually
try
to
think
about
how
to
organize
ourselves
in
groups
and
what
are
the
most
important
questions.
I
do
have
a
few
slides
here
and
I
will
quickly
run
through
them.
D
Basically,
and
a
lot
of
these
questions
you
will
find
on
these
slides
are
really
kind
of
a
one-to-one
copy
from
the
paper
because
they
were
phrased
like
ask
questions
in
the
in
the
papers,
and
sometimes
I
did
a
little
bit
of
writing
changes,
but
not
much
so
before
we
start
on
the
questions-
and
this
is
also
just
a
very
quick
summary
about
tools
and
data.
What
I
found
in
the
in
the
in
the
papers,
so
most
of
the
papers
really
looked
at
mailing
list
archives,
the
rfc
index.
D
D
There
are,
and
these
data
that
greg
presented
about
the
page
itself
like
who's
looking
at
the
page
and
these
kind
of
things
and
there's
also
things
like
survey
data
that
we
collect
about
the
itf.
Just
in
the
previous
session
github
was
mentioned,
we
might
be
able
to
get
some
metadata
from
the
video
conferencing
tools
we're
using
and
so
on.
So
there
might
even
be
more
data
than
we
think
about
should
think
about
that.
D
D
Talking
about
the
international
footprint
talking
about
gender,
a
lot
general
diversity,
questions,
inclusion
and
fairness,
transparency
as
well,
and
so
all
these
things
are
mentioned.
So
this
could
be
probably
a
group
of
people
who
might
be
interested
to
connect
each
other
and
work
together.
So
we
can
keep
this
in
mind
for
a
hacking
group.
Potentially,
then
we
had
another
whole
bunch
of
papers
which
were
connected
to
the
question
about
affiliation
and
industry
control
so
like
how
people
change
affiliation.
We
discussed
this
earlier
today
and
and
and
what
what
is
their
company
interest?
D
What's
their
own
interests?
How
to
separate
this,
and
how
does
it
impact
the
work
in
the
ietf
basically
and
then
the
last
session?
We
we
discussed
about
this
decision
making
and
process
aspects,
and
there
was
also
quite
interesting
to
see
in
the
paper
that
there
were
many
different
methods
about
how
to
do
content
analysis,
basically,
because
that's
a
big
challenge
here,
looking
at
keywords.
D
Looking
at
mailing,
this
content,
looking
at
metadata
as
well
to
understand
how
our
document
involves
looking
at
volume
analysis
over
time
and
looking
at
correlation
between
different
and
development.
And
so
there
was
like
a
whole
lot
of
methodology,
which
might
also
be
interesting
to
just
discuss
in
a
hacking
group.
But
then
there
were
again
like
a
whole
bunch
of
open
questions
on
the
methodology
as
well
right.
What
are
the
next
keywords
to
analyze?
D
D
D
Is
it
better
to
meet
like
what's
what's
the
right
thing
here,
looking
at
at
global
warming
and
co2
footprint,
so
these
are
topics
that
we
didn't
have
today,
but
we
will
have
time
for
these
topics
on
thursday
the
the
day
was
already
crowded
today.
So
we
took
this
out,
but
these
were
very
early
papers,
more
asking
questions
and
presenting
results.
So
we
actually
hope
this
that
these
papers
and
other
interesting
interested
people
can
work
together
and
maybe
even
present,
some
results
on.
D
Bottom,
which
is
the
number
of
zero
zero
drafts
published,
so
the
new
work
coming
to
the
ietf
basically,
and
when
we
look
at
the
curve
of
last
year,
the
yellow
curve
here
and
the
curve
of
this
year-
the
green
curve.
Here,
we
actually
see
quite
a
difference
to
the
previous
years.
So
it
does
look
like
that.
D
The
situation
we're
in
right
now
does
have
an
impact
on
productivity
or
at
least
an
impact
on
like
getting
new
ideas
into
the
ietf,
and
we
have
this
little
data
here
and
there
is
a
strong
hint
here
that
there
might
be
something,
but
it's
really
not
enough
to
understand.
What's
going
on
and
how
big
the
impact
is,
I'm
also
an
author
about
a
document
that
talks
about
organization
of
online
meetings,
and
I
just
realized
by
writing
this.
This
draft
that
there
are
so
many
questions
where
I
could
need
some
answers
about
online
meetings.
D
How
did
really
moving
to
this
fully
online
setup
impact
productivity?
That's,
like
you
know
what
I
talked
about
at
the
previous
slide,
but
also,
how
did
it
really
change
change?
How
does
this
change
from
fully
online
change?
How
we
work?
Maybe
we
use
more
interims.
Does
it
make
was
more
productive
or
less
productive?
Does
it
have
an
impact
on
cross-area
interaction?
Does
it
have
an
impact
on
inclusivity?
D
Does
it
have
an
impact
on
participation
in
general
like?
How
does
it
change?
Do
we
have
more
newcomers
and
do
they
actually
become
active
participants?
Are
people
less
active
people
speaking
up
less
often,
and
how
many
sessions
do
they
visit
you
know
is,
is
the
week
as
crowded
as
it
was
before,
or
are
people
visiting
itf
btx
more
selectively?
D
All
these
kind
of
questions?
I
don't
have
answers,
but
I
think
we
have
the
data
for
it
and
there's
a
whole
bunch
about
socializing,
which
we
know
there's
a
problem,
but
we
also
don't
know
how
to
solve
it,
but
maybe
there's
also
some
data
about
how
we
use
gather
and
other
tools
and
whatever.
So
that's
just
something
I
wanted
to
add
here
and
with
that
we
will
actually
go.
D
I
will
stop
sharing,
because
I
can't
see
anything
else
so,
but
we
will
go
to
this
google
document
and
I
already
started
putting
some
groups
there,
and
but
this
might
not
be
the
right
groups.
It's
just
like
what
came
to
my
mind
from
these
slides.
The
idea
is
that
we
organize
ourselves
into
smaller
groups.
D
This
could
be
like
two
or
three
people
who
are
really
interested
in
one
specific
topic,
or
it
could
be
a
little
bit
larger
group
of
whatever
seven
eight
people
who
all
work
on
the
same
area
of
of
questions,
and
then
these
groups
can
organize
yourself
freely
based
on
the
time
zones.
They're
in
you
don't
have
to
stick
to
the
workshop
time
zone.
You
can
also
find
other
slots
tomorrow
and
the
day
after
tomorrow
that
fits
well
to
your
group,
and
you
can
organize
yourself
freely.
D
You
can
use
slack,
you
can
use
gather
and
you
can
use
any
conferencing
tool
you
want.
If
you
need
support
with
the
conferencing
tool,
we
can
also
give
you
a
webex
session.
Just
let
us
know,
but
we
also
have
these
sunga
webweg
sessions
tomorrow
and
on
wednesday,
where
just
like
all
the
chairs
will
be
there
and
like
everybody
can
join
and
we
can
either
just
chat
or
if
there
are
any
problems
or
any
questions,
then
please
feel
feel
free
to
approach
us
there.
D
D
D
D
Okay,
yeah,
so
the
first
group
says
challenges
to
improve
tooling,
so
that
was
really
you
know
what
I
had
this
on.
This
one
slide
a
couple
of
questions
from
yari
and
and
paul,
but
you
know
also
you
know
what
are
the
challenges
with
the
current
toolings
that
we
have,
and
so
this
is
optional.
You
know
people
want
to
talk
about
it.
They
can
I've
seen
that
already
a
lot
of
people
signed
up
to
the
diversity
and
inclusivity
group.
So
I
hope
that
is-
and
these
topics
are
clear.
D
D
Yeah
questions
about
challenges
for
content
analysis.
This
is
also
probably
more
a
little
bit
about
tooling,
like
how
can
you
actually
apply
the
given
methodologies
to
to
email,
content
and
rfc's
content?
This
was
the
slide
about
you
know
things
like
keyword,
detection,
natural
language
processing
applied
and
so
on,
so
from
position
papers.
I
understood
there
were
a
lot
of
spoken
challenges.
Still
then
group
number
five.
D
This
is
the
decision
making
very
much
matching
to
what
we
discussed
at
the
previous
session
on
at
the
beginning,
like
how
how
our
decision
is
made,
how
to
how
what's
the
process
to
get
to
an
rfc
and
what
are
the
success,
success
factors
for
that,
and
there's
group
number
six
on
sustainability
and
climate
change.
This
is
also
the
two
papers
I
just
mentioned
about
the
questions,
how
the
meeting
and
how
our
technology
impact
climate
change
and
then
there's
the
group
number
seven
on
online
meeting
and
productivity.
D
That
was
like
my
question.
I
had
how
did
how
did
participation
change
with
the
move
to
fully
online
and
how
does
it
impact
productivity?
How
does
it
impact
impact
new
work?
How
does
it
impact
diversity?
Are
more
people
participating
in
these
kind
of
things,
and
then
there
was
group
number
eight
or
there
is
group
number
eight,
which
is
on
impact
and
implementation.
So
that's
also
something
we
discussed
in
the
earlier
sessions
about
like.
How
can
we
measure
if
the
products
we
have
in
the
ietf
rcs
are
actually
successful,
are
actually
implemented
and
used.
D
A
A
If
people
would
be
willing
to
have
a
bit
of
a
coordinating
role,
they
could
make
their
names
bold
under
the
group
so
as
to
ensure
that
that
people
meet
in
the
same
place
and
can
coordinate
and
that
we
have
slides
by
by
thursday.
So
if
people
would
be
so
generous
to
to
volunteer
for
that,
that
would
be
very
much
appreciated.
A
D
F
I
don't
I
you
know,
don't
have
a
strong
view
about
where
that
should
go,
but
it
seemed
like
something
I'd
be
happy
to
facilitate
that
that
the
main
thing
we're
missing
is
justice.
Baron
was
not
yet
on
the
slack,
so
just
as
I
guess
I'll
reach
you
out
to
you
by
email,
if
we
code
a
decision
about
how
to
where
the
runes
are.
P
F
P
D
I
D
D
V
A
I
Q
You
signed
up
for
this
one
yeah.
I
was
just
thinking
about
something
because
I'll
as
I
as
you
can
see,
I
am
doing
something
their
plan
for
the
group
two,
so
just
wondering
what
exactly
could
be
a
possible
project.
I
was
thinking
of
seeing
I
think,
a
question
that
whether
large
language
models
can
do
a
text
mining
nlp
on
the
content
that
is
there.
That
was
a
question.
Some
people
had
maybe
just
test
whether
that
that
is
happens.
N
Yeah,
if
I
can
just
hop
in
maybe
we
can
also
leave
the
larger
group
for
what
it
is
and
they
sell
if
they
figure
out
that
they
need
to
get
into
more
groups,
because
I'm
sure
that
when
the
discussion
hits
that
they
figure
out
like
these
are
the
different
issues
that
we
want
to
focus
on.
Instead
of
us
trying
to
do
that
before
we
know
what
the
what
the
questions
are.
The
group
was.
D
Sure
I
mean
this
is
possible
as
well
I'm
just
trying,
because
we
only
have.
We
only
have
two
days
right,
so
I
was
trying
if
we,
if
we
know
already
that
there
is
a
certain
split.
If
we
know
already,
people
have
different
interests,
might
be
more
efficient
to
do
it
up
front,
but
not
sure.
If
that
is
the
case,.
W
D
D
So
yeah,
if
I
remember
the
discussion
earlier
today
correctly,
then
I
think
there
were
at
least
two
sub
questions.
One
was
like
looking
at
trends:
how
people
changed
affiliation,
you
know,
is
there
trend
from
people
moving
to
one
company
to
other,
and
you
know
maybe
even
their
specific
events
when
these
changes
happening.
D
So
that
was
one
question
and
the
other
question
was
what
mallory
I
think
brought
up
was
about
stakeholder
groups
right
how
to
actually
assign
these
affiliations
to
our
people,
to
a
stakeholder
group,
and
how
does
the
stakeholder
group
act
or
is
represented?
So
maybe
that's
a
split.
We
could
consider.
P
So
there
were
also
these
questions
pertaining
to
these
indirect
affiliations,
like
these
consultants,
working
who
have
like
a
direct
affiliation
which
might
not
be
particularly
informative.
But
they
actually
are
controlled
by
this
indirect
affiliation
or
same
goes
for
academics,
who
might
work
on
a
research
project
funded
by
industry,
or
these
kind
of
things
which
are
more
challenging
for
you
to
measure
than
the
actual
employer,
but
just
as
relevant.
F
So
it
seems
like
some
kind
of
data
structure
for
the
topology
of
organizations
and
their
properties
is
an
order,
something
which
says
these
are
organizations
that
are
some
kind
of
stakeholder
group.
These
are
multiple
ways.
People
can
be
affiliated,
and
that
seems
like
a
pretty
broad
design
question
I
I
guess
the
reason
why
I
was
thinking
about
data
sets
is
that
doing
arguably
doing
that.
Modeling
well
is
sort
of
prior
to
a
good
empirical
analysis.
A
V
Happening
awesome,
so
we
have
climb
iab,
iid
climate
change
for
group
six.
We
have
for
group
five,
something
that
colin
has
created.
Ib
aid
process
decision
making
us
no
decision
making
was
the
one
that
I
made
that
you
should
leave.
I
can't
delete
it
anymore,
because
colin
has
one
I
made
one
in
parallel,
so
it's
ib
aid
process.
V
V
A
D
Yeah,
if
there
are
no
further
questions,
we
can
just
move
on
to
the
next
part.
I
also-
I
guess
karen
will
mention
this
as
well,
but
I
also
want
to
mention
that
we
do
with
that.
We
can
use
the
gather
town,
the
itf
gather
town
is
open
and
running.
As
always,
you
can
all
just
join
and
connect
there.
I
might
just
hang
out
there
a
little
bit
more
after
the
meeting.
People
have
questions
and
it's
there
24
hours
so
prefer
to
join
there.
N
Because
it's
always
such
a
hard,
hard
job
but
I'll,
try
and
give
it
a
go.
So
thanks
so
much
everyone
for
being
here,
the
participants,
the
program
committee,
as
well
as
kate,
for
taking
notes.
I
do
think
we
had
a
really
exciting
and
pretty
fast-paced
day.
We
did
a
lot
of
things
moving
from
the
tools
and
the
data
to
the
methods
to
how
to
use
these
for
a
variety
of
questions,
including
industry
control
who
the
community
is
and
how
the
process
by
which
decisions
get
made
actually
happen.
N
I
do
think
we
have
a
really
unique
opportunity
here
for
us
to
work
together
and
make
real
headway
and
answering.
For
example,
you
know
how
wearing
different
hats
can
influence
standard
outcomes
or
how
diversity
plays
into
ongoing
discussions,
or
what
questions
that
we
have
or
raise
cannot
be
answered
by
data,
but
need
a
different
kind
of
approach,
and
also
what
success
looks
like
for
the
its
process.
N
I
really
again
would
encourage
everyone
to
participate
in
the
hackathon
to
get
organized,
and
I
for
one
at
least
look
forward
to
the
hackathon
days
and
seeing
the
results
from
from
the
new
collaborations
that
I
that
I
hope
will
develop
this
week
and
seeing
all
of
you
again
on
thursday.
So
thank
you
again
so
much
and
if
there's
anything
I
have
forgotten,
please
feel
free
to
hop
in
and
if
not
see
you
tomorrow,
wednesday
and
thursday.