►
From YouTube: Torus Community Meeting September 2021
Description
Monthly meeting to discuss Open Learning Initiative's next generation platform. This meeting is about A/B testing functionality needed to support research.
A
Welcome
folks,
we'll
get
started
in
about
12
35
of
my
time
to
give
folks
some
time
to
get
get
on
and
you
know
grab
their
lunch.
Go
from
one
meeting
the
max
that
sort
of
thing.
A
A
The
topic
for
today
is
a
b
testing
and
it
won't
be
the
the
only
meeting
we
have
about
this,
so
we'll
collect
what
we
can
today
and
probably
have
a
part
two
at
some
point
in
the
future.
A
Okay,
let's
talk
baby
testing,
welcome
everyone
glad
you
were
able
to
join
right
before
a
holiday
weekend.
I
I
think
you
heard
me
earlier.
I
don't
think
anyone
knew
has
joined
since
I
said
that
this
will
be
the
first
of
probably
a
few
meetings
around
a
b
testing
it's
going
to,
since
it's
a
feature
that
we
don't
have
the
ability
to
do
at
the
moment.
I
can
imagine,
there's
a
lot
of
requirements
we
can
collect
and
then
it's
going
to
take
more
than
one
one
meeting
to
get
through
them.
A
So
what
we're
talking
about
when
we
talk
about
eb
testing,
at
least
from
my
perspective,
what
we're
talking
about
is
ability
to
perform
research
using
the
platform
and
research
on
courses,
research,
research
on
a
bunch
of
different
things,
that's
kind
of
what
I
want
to
talk
about
first,
so
I'm
going
to
start
from
the
beginning
and
really
think
about
what
aspects
of
courses
that
we
think
we
want
to
be
able
to
test
and
research.
A
What
mechanisms
need
to
be
in
place
to
reform
that
that
sort
of
testing-
and
I
threw
a
couple
ideas
out
there
just
to
help
brainstorm
and
you
know,
are
there
different
interface
needs
for
researchers
who
want
to
perform
this
type
of
a
b
testing
versus
instructors?
Who
just
want
to
kind
of
do
it,
maybe
in
a
more
lightweight
fashion.
A
So
when
I
I'd
like
to
discuss
that
a
little
bit
and
what
tools
exist
now
that
folks
might
use
that
we
might
want
to
integrate
with
and
really
just,
how
are
these
things
done
now
how's?
How
is
research
performed
now,
and
I
can
imagine
in
the
next
set
of
meetings
that
I
have
about
this-
to
invite
some
of
our
research
community
directly.
A
We
have
communities
that
talk
about
like
ai
and
machine
learning,
and
it
would
be
interesting
to
get
their
perspective
on
on
this
sort
of
thing
too
too,
and
the
kind
of
tools
they
they
use.
But
does
anyone
have
want
to
get
us
started
in
any
way
shape
or
form.
A
B
I'll
kick
in
hey.
I
think
that
that's
kind
of
the
irb
issues
and
to
what
extent
can
you
do
experiments
without
you
know,
making
it
easier
to
get
irb?
For
instance,
one
thing
one
style
that
seems
to
be
easier
to
get
through
irb
is
on
this
topic.
B
Like
I'm
testing
out
a
is
some
new
thing
and
b
is
current,
and
I
have
it
for
two
different
sections
of
the
course.
So
some
students
will
see
the
new
stuff
on
topic.
One
and
other
students
will
see
the
new
stuff
on
topic
too,
and
so
that
means
it
kind
of
they
both
are
exposed.
B
It's
fair
in
the
sense
that
every
student
gets
this
new
thing
once
in
the
course,
although
it's
on
different
topics
so
that
I
don't
know
yeah,
what
are
we
allowed
to
do
a
b
wise
because
it
sounds
great
like
I
make
something,
maybe
when
I'm
going
to
test
it
after
students
will
get
it,
but
what's
the
what's
the
ethical
way
to
do
that?
I
don't
really
know
that.
I
know.
A
Yeah
yeah,
I
can
say
that
I've
been
part
of
this
learning
engineering
group,
google
group,
and
they
what
got
into
a
pretty
lengthy
discussion
back
and
forth
in
email
about
the
ethical
considerations
of
a
b
testing,
and
you
know
some
some
places,
don't
even
don't
even
touch
it
like
won't,
even
consider
it
because
they
there's
so
many
implications
for
it.
But
what
could
the
platform
do
to
help
help
with
that
like?
Are
there
ib?
Is
there
other
templates
that
we
would
need
to
incorporate.
B
I
do
think
bringing
in
someone
who's
the
expert
in
this
for
a
lot
of
experiencing
it,
because
that's
the
one
I
could
think
of
that.
You
would
be
able
to
just
immediately
say
I'm
putting
this
new
thing
in
on
two
different
topics
and
do
that
kind
of
thing:
half
the
students
will
do
a
and
the
first
topic
and
the
other
half
will
ca
in
the
second
topic.
B
C
And
get
in
here
hi
the
irb
wouldn't
have
to
be
each
individual
institution
right.
You
could
get
it
done
once
at
carnegie
and
then
it
would
be
okay
for
anybody
who
adopts
the
course
or
would
it
need
to
be
reviewed
each
institution
that
decides
they
want
to
run
experiments.
A
Yeah,
that's
an
interesting
question,
I'm
not
sure
of
the
answer,
but
it
sounds
like
if
we
bake
in
the
ability
to
do
this
sort
of
testing.
We
also
have
to
bake
in
the
ability
to
provide
really
detailed
information
about
the
previous
studies
or
the
study
that's
going
on
and
be
able
to
track
those
conditions
through
no
matter
what
so
that
we
could
then
present
that
information
kind
of
back
to
anyone
else
who
wants
to
use
the
course
is
that
kind
of
what
you're
thinking
too.
C
Kind
of
it's
more
if
you
find
out
that
only
carnegie
mellon
has
to
run
it
through
irb,
and
you
find
that
carnegie
mellon's,
you
know,
review
board
is
going
to
be
amenable
to
this
I'd
call
it
automatically
paired
a
b
or
something
like
you're
only
allowed
to
fix
to
improve
two
things
and
give
one
improvement
to
each
half.
C
If
they're
willing
to
say
yes
to
all
that
in
all
conditions-
and
it
makes
sense
that
that
approval
be
allowed
everywhere
and
anywhere
because
it
comes
from
carnegie
mellon
right,
then
you
only
have
to
do
irb
once
at
carnegie
mellon
and
then
all
you
need
to
advertise
is
look
if
you
want
to
run
experiments.
This
is
the
only
kind
that
is
pre-approved
for
this
course.
A
A
C
To
be
doing
it
all
the
time,
because
that's
the
whole
point
of
having
a
local
copy,
isn't
it
either
that
or
I'm
just
going
to
not
run
an
experiment
or
I'm
going
to
pretend
that
the
data's
anonymous
and
it's
part
of
my
regular
classroom
behavior
so
and
it's
not
optional,
so
it
doesn't
matter
right.
So
then
you're
getting
a
little
around
the
edges
of
of
you
know:
you're
cutting
a
little
bit
into
ethics
out
of
ease.
A
Right,
what's
what's
cool
about?
This
is
what
I'm
imagining
is
its
own
little
its
own,
like
interface,
even
within
taurus.
To
have
this
whole,
like
you
know,
experimentation,
I
don't
know
what
we
call
it:
a
b
testing
panel,
almost
that
kind
of
helps,
walk
folks
through
what
they
can
or
can't
do
with
the
existing
setup
or
existing
set
of
experiments.
A
A
I
don't
know
what
to
call
it
protocols
like
kind
of
public
or
not
and
could
invite
others.
That's
some
interesting
stuff.
Gideon
thanks.
C
C
If
you
want
to
go
beyond
this
well,
the
system's
not
going
to,
let
you
do
it,
you
have
to
go
to
your
own
irb
and
you
know:
do
it
your
own
way,
so
it
makes
sure
that
nobody
takes
this
product
and
uses
uses
in
a
way
that
the
carnegie
mellon
irb
board
thinks
is
unethical,
but
it
allows
them
to
go
locally
and
convince
their
local
board.
Can
I
do
this
more
thing
right?
It
forces
ethicality.
A
So,
if
you
think
about
what
kinds
of
things
you
might
want
to
test
out,
what
so
there's
there's
these
we're
talking
kind
of
in
general
right
now,
like
these
two
interventions,
you
know
a
and
b,
but
would
you
imagine,
would
you
want
that
that
ability
to
be
very
granular
with
it
like
in
terms
of
even
like
text?
On
a
page
I
can
imagine,
we
probably
want
the
ability
to
do.
A
You
know,
text
activities,
assessment,
questions,
videos
versus
other
things,
videos
versus
videos,
so
I
guess
just
about
any
element
in
a
course
would
be
fair
game
for
a
b
testing.
Does
that
sound
about
right.
A
So
aside
from
irb
condition,
considerations
are
there.
What
are
things
that
you
wish
were
baked
into
the
system
now
that
would
allow
you
to
do
more,
not
just
experimentation
but
gather
more
information
about
the
context
in
which
the
course
is
being
used,
say:
yeah,
dave.
D
The
so
sometimes
when
you
do
conditions
you
want
to
have
striking,
I
feel
what
the
word
for
this
is,
but
you
want
to
make
sure
that
you
get
the
right
distribution
and
demographics
and
also
performance
levels.
D
The
learning
dashboard
to
estimate
the
student
where
they're
at
and
then
making
sure
that
the
conditions
are
balanced
by
those
various
criteria
and
that
that's
something
that's
hard.
It
just
adds
a
whole
burden
when
you're
trying
to
do
you,
know
random
assignment
but
making
sure
that
it's
done
on.
You
know
that
the
two
populations
are
as
similar
as
you
can
get
them.
D
Like
what
are
you
allowed
to
get
so
there
might
be
some
criteria
that
you
would
be
able
to
that's
a
whole
different
thing,
but
they
also
the
data
on
the
learning.
So
I
guess
there's
two
sources
of
information.
One
is
some
some
survey
you
gave
them.
You
don't
want
to
do
distributions
across
that
to
make
sure
it's
the
distribution
you
want
and
then
some
data
that's
coming
from
the
system.
D
A
Yeah,
that's
great
so
that
what
that
requires
then,
is
some
initial
set
of
criteria
you
collect
and
then
I'm
thinking
of
workflow
here
you
know
and
then
the
ability
to
use
that
information
to
then
do
assignment
of
either
condition.
C
If
you
want
to
look
up
the
language
and
the
standard,
you
know
for
research
board
purposes
if
you're
doing
something
else
like
population
proportional,
like
you're,
gonna,
pick
one
tenth
of
the
ones
in
the
top
third
and
one
tenth
of
the
ones
in
the
middle.
Third:
it's
not
called
stratified
sampling,
but
it's
the
same
idea,
you're
doing
some
kind
of
criteria
splitting
and
then
in
your
randomization
you're,
not
doing
simple
random
sampling
right.
B
C
A
Right,
but
what
you're,
what
you're
bringing
up
is
the
disability?
What
you
and
david
both
brought
up
is
this
ability
to
do
different
kinds
of
sampling
and
have
an
interface.
We
would
need
an
interface
to
be
able
to
to
kind
of
sort
out
the
students
to
and
assign
them
to
those
different
conditions.
So
right.
C
C
I
did
I
did
a
an
education
doctorate,
so
I've
dealt
with
this
stuff.
If
there's
any
sense
that
the
student
even
falsely,
feels
obligated
to
participate
because
you're
their
instructor,
the
irb
is
going
to
come
down
hard.
So
it's
often
easier
to
make
things
mandatory
and,
like
your
ab
balance,
design.
Okay,
it's
mandatory
right.
They
don't
have
a
choice,
they
kind
of
don't
even
know
which
they
get
the
right.
That's
not
a
big
deal,
but
then
slicing
by
surveys
that
you've
written
oh
right.
Could
somebody
abuse
that
or
could
students
feel
uncomfortable
about
that.
B
C
Maybe
not
if
it's
their
own
perform
if
it's
just
their
performance
within
the
system
showing
high
low
medium
on
the
first
test.
That's
fine
and
you
can
keep
their
anonymity
protected,
and
you
know,
however,
the
data
is
analyzed
or
whatever
right.
So
you
got
to
watch
out
and
I
think
it
really
is
something
that
that's
got
to
be
your
irb
board
and
how
how
far
out
they're
going
to
stick
their
necks
to
say
yes
for
everybody
using
it.
A
You
know
completely
different
than
than
what
the
instructors
see
if
it's
instructors
that
are
doing
these
experiments,
they
have
to
kind
of
have
this
almost
separation
from
the
experiment.
They
have
to
set
it
up
and
let
it
run
and
kind
of
not
meddle
with
it
or
review
it
until
the
very
end
so
that
they
can't
make.
C
Right,
I
would
expect
that's
what
your
irb
and
any
irb
would
require,
but
I
did
a
very
different
kind
of
qualitative
research
on
a
very
small
number
of
students
who
were
my
own
in
a
very
small
class.
So
it
was
a
different
scenario:
I'm
just
sort
of
anticipating
the
possibilities,
but
right-
and
it
will
be
asking
your
irb
to
approve
for
everyone.
So
I
expect
they'll
be
conservative,
but
it
certainly
seems
quite
possible
with
the
technological
tools
that
you're.
C
A
Right
so
it's
almost
like,
I
would
have
to
start
in
terms
of
gathering
requirements
for
these
type
of
interfaces
and
and
workflows.
I
would
probably
have
to
start
with
the
with
it,
with
some
irb
guidance
to
ensure
that
we're
setting
things
up
in
a
way
that
they
would
be
able
to
perform
investigations
easily,
or
you
know
the
fact
that
we'd
have
to
be
proving
that
the
whole
system
is
kind
of
set
up
in
a
way
to
do
no
harm
to
the
students
right.
A
C
E
In
the
past,
with
a
bunch
of
these
cmu
has
acted,
that's
the
honest
broker,
but
it's
still
up
to
your
irb
to
actually
approve
the
study.
D
It's
it's
a
little
there's
a
a
fairly
broad
irb
right
now
that
covers
collection
of
dating
sharing
the
fully
blinded
data
with
data
shop,
and
so
it
is
true
that
there
are
sites
that
are
participating
in
research
through
roi
that
that
institutions
irb
did
not
have
to
approve
it,
and
that
so
I
do
think
that's
one
part
of
the
discussion
is:
how
far
can
we
go
with
that
kind
of
broad
irb
that
covers
everything.
A
E
D
Would
I
what
would
be
ethical
to
ask
for
that
kind
of
coverage
for
it
so
very
interesting
thing,
but
I
do
think
it's
also
correct
to
say:
there's
a.
A
I
mean
you
could
probably
do
it
at
the
page
level.
Even
you
know
it
can
be
done
at
any
level.
I
I
really
want
to
pick
the
brains
you
guys
talking
make
me
think.
I
also
need
to
reach
out
to
our
partners
at
unh
because
they
develop
that
student
cognition
toolkit
and
they
they
do
experiments
all
the
time,
and
I
know
they
do
it
through
their
institution
uses.
I
r
irb,
I
believe
or
is
like
you
know,.
D
A
So
I
don't
know
just
of
any
other
places
that
do
regular
experimentation.
I
think
they
even
do
kind
of
some
baby
testing.
D
And
the
other,
how
is
there
a
reason
to
merge
this
with
the
discussion
of
adaptive
learning,
because
adaptive
learning
is
going
to
be
making
a
decision
on
what
to
show
the
student
based
on
a
bunch
of
stuff,
and
this
is
just
one
more
stuff
right?
What
condition
are
they
in
and
so
from
a
technical
perspective?
A
Well,
it's
kind
of
because
just
it
depends
on
what
kind
of
adaptive
right
you
can
do.
You
can
have
a
whole
bunch
of
different
types
of
adaptive
and
we're
only
I
can
imagine
you
know
like,
for
example,
when
I
was
at
acrobatic,
they
developed
a
the
ability
to
adaptive
based
on
they
would
deliver
scaffolded
versus
unscaffolded
questions
and
an
assessment
based
on
the
learner's
learning
estimate
for
that
learning
objective
at
the
time
that
they
entered
the
survey
or
the
assessment.
I'm
sorry,
that's
very
different
like
so.
In
that
case
it
was
all
built.
A
The
adaptivity
was
all
built
around
that
learner.
That
learning
estimate
for
that
objective,
that's
a
very
different
system
than
if
you're
basing
it
on
other
variables
right.
So
you
would
have
to
have
the
ability
to
say.
Okay,
you
can
experiment
in
this
system
on
these
things
that
also
promote
adaptivity.
A
So
then,
either
way
those
things
could
be
used
for
either
adaptivity
or
experimentation.
But
then
how
is
activativity
then
not
experimentation
right,
because
we
really
don't
know
if
that
if
that
approach
is
helping
the
learner
or
not
in
any
way
shape
or
form,
it's
kind
of
my
problem
with
with
that
activity.
To
begin
with,
it
sounds
like
a
good
idea
to
have
a
personalized
learning
path,
but
what's
going
on
here.
A
I
must
have
gotten
kicked
out
of
my
account
and.
A
Yeah,
so
there's
lots
of
different
ways
to
that.
That
brings
up
grain
size.
That
brings
up
the
mechanisms
which
you
can
that
can
control
other
mechanisms
for
experimentation.
A
A
And
it
can
be
done
in
so
many
different
ways
right,
but
a
good
point,
though,
if
we
have
those
mechanisms
already
built
in
then
they
will
be
used
for
experimentation.
I
love
to
see
what
we're
doing,
because
I
know
we're
building
this
whole
adaptive
engine
that
kind
of
bolts
on
to
taurus.
At
the
moment.
It's
it's
not
baked
in
it's
like
a
custom
activity.
A
Shim,
I
think,
is
how
I've
heard
it
referred
to,
and
I
you
know
that's
interesting
because
it
will
be
used
for
experimentation
then,
and
it'll
have
to
connect
up
to
whatever
interfaces
we
have
for
a
b
testing.
B
Yeah,
it
does
also
sound
like
you're,
going
to
need
some
way
of
either
guiding
the
adaption
for
experimental
parameters
or
ignoring
them
so
yeah,
if
you're
going
to
be
using
as
a
general
factor
in
the
course
and
you're
also
trying
to
use
it
as
an
experimental
factor,
you're
going
to
want
to
direct
it
for
some
people
direct
it
another
way
for
other
people.
So
we're
not
talking
about
a
simple
adaptation,
we're
talking,
multiple
adaptation
paths
right.
A
Yes,
exactly
and
that's
what
this
kind
of
custom
mechanism
will
allow.
I
imagine
different
customization
paths,
because
I
know
what
they're
basing
it
on
is:
smart
sparrows
ability
to
design
pathways
based
on
different
variables,
and
it's
I
know
for
me.
The
another
reason
why
I'm
not
a
fan
of
adaptation
and
adaptivity
and
in
these
types
of
systems
is
because
you
know
that's
a
lot
of
design
assumptions
you're,
making
when
you're
doing
that
when
you're
building
those
little
mini
pathways
for
each.
A
B
A
F
That's
an
interesting
question
established
forms
of
adaptive
testing.
I
believe,
there's
a
nursing
exam.
That's
that's
the
same.
It's
an
adaptive
test.
A
Well,
they
say
that
I
think
it's
been
so
long,
but
I
think
isn't
the
the
gre
built
a
little
bit
like
that
too,
when
it
went
on
online.
B
A
A
E
It
does
sound
like,
as
I've
been
percolating,
my
brain.
A
concrete
thing
here
is
the
for
setting
up
a
b
testing,
probably
going
to
have
to
have
an
anonymized
version
that
people
can
review
and
assign
to
the
various
conditions,
and
but
I
can
also
see
it
for
just
normal
classroom
rooms.
You'd
want
to
be
able
to
access
the
personal
information
outside
of
experiments.
E
A
You
know
if
it's
there,
if
we
were
able
to
collect
that
information
on
students,
but
that's
that's
information
that,
depending
on
what's
being
collected,
do
instructors
have
access
to
that
kind
of
information
now
on
their
students.
E
So
if
it
is
an
experiment
and
say,
oh,
we
can't
ever
use
the
student
names.
Students
names
never
show
up,
but
they
need
to
know
the
section
information
or
some.
You
know
just
a
checklist
of
what
information
is
collected
and
what
would
be
available
in
the
interface
on
the
review
of
the
distribution.
E
A
I
love
how
we
just
we
just
stick
a
pin
on
the
ethical
stuff,
but
and
I'm
glad
we
were
able
to
do
that
to
just
think
about
what
would?
How
would
we
want
to
see
this
play
out?
We're
going
to
say
something
more
dave
about
that.
D
As
you
say,
we
have
a
project
with
neboru
where
we're
doing
adaptive
learning
and
the
conditions
will
be
different.
Styles
of
adaptive
learning
so
we're
thinking
of
it
as
reinforcement,
learning
and
so
policy
and
deciding
what's
the
best
thing
to
show
the
student
next,
and
so
the
conditions
would
be
different.
Algorithms
for
making
that
choice.
D
A
A
A
D
F
The
thing
about
the
honors
class
actually
brings
up
an
interesting
thing,
which
is
basically
having
a
very
similar
functionality.
F
Basically
that
doesn't
actually
get
into
testing
anything
which
is
just
if
somebody
wants
to
like
run
one
course,
but
have
variations
in
it
for
some
reason,
which
is
like
somewhat
different
from
running
two
courses
like
I
know
when
I
was
an
undergrad,
I
was
in
some
courses
where
some
of
the
students
were
undergrad
students
and
some
of
them
were
grad
students
and
the
grad
students
were
like
had
like
extra
assignments,
and
so,
if
somebody
wants
to
do
that
kind
of
thing,
then
that's
basically
a
use
case
for
very
the
course
feature
that
doesn't
like,
even
if
they're,
not
testing
anything
which
is
kind
of
interesting.
A
Right
so
how
much
of
the
of
the
experimentation
mechanisms
can
be
exposed
to
just
not
do
experimentation,
but
do
other
things
with
it?
That's
yeah
because
it
will
be
used
that
way.
If
we
have
the
the
mechanism
to
do
that
kind
of
thing.
D
The
yeah
we
probably
have
to
do
it
between
each
section.
We
would
create
different,
different
sections
and
actually
we're
just
waiting
for
tourists
to
come
online,
because
how
we
do
to
really
do
this
correctly.
We
need
support
that's
outside
of
the
current
lli
system,
although
we're
working
with
john
stamford
to
try
and
get
something
he
has
we'll
be
able
to
put
a
little
thing.
I
hope
on
a
page
that
is
doing.
A
D
Adaptation
with
some
algorithms
and-
and
it
will
know
what
happened
on
previous
pages
because
it
will
actually
be
the
same-
learning
object.
That's
appearing
on
all
these
pages,
so
we're
kind
of
just
embedding
some
adaptive
tool
into
the
course
and
no
matter
what
page
it's
on
in
the
course
it's
going
to
be
showing
stuff,
but
that's
such
a
huge,
such
a
non-elegant
solution,
so
we're
ultimately
wanting
to
use
the
whatever
it
gets
built
in
front
activity
and
taurus
to
do
this
project.
A
Okay,
yeah
well
the
design
the
design,
for
that
is
what
takes
so
much
energy
and
thought
right
in
my
experience
anyway
yeah.
So
even
if
it
was
online
today,
it
probably
would
still
be
a
little
while
before
you
get
it
up
and
running
next
fall,
actually
is
not
bad,
not
bad
timing,
yeah
or
the
the
tool
development
and
the
design
then
piece
they
willing
to
do.
In
the
meantime,.
A
D
B
A
A
I'll
have
to
find
out
what
that
is,
it's
hard
to
keep
up
on
all
the
different
tools
that
are
developed
at
cmu
for
various
things
like
that,
but
it's
cool
to
find
out
about
them.
A
Okay,
so
we've
talked
about
assigning
students
to
different
conditions.
We've
talked
about
the
ability
to
track
certain
information
and
actually
make
that
information
available
or
or
not
to
different
populations,
meaning
instructors
or
administrators.
So
I
can
imagine
you
know
a
page
that
you
know
there's
there's
different
views
into
the
system
for
the
same
material
and
the
same
set
of
students
that
would
be
needed
for
experimentation
versus
not
and
being
able
to
kind
of
toggle
those
on
and
off,
depending
on
irb
protocols.
A
It's
in
the
works
already
to
be
able
to
communicate
back
and
forth
with
data
shop.
Can
you
think
of
any
other
like
any
wish
list
items
around
those
integrations
or
tools
like
I
said
that
don't
exist
in
any
of
those
that
you
need.
A
Right,
I
think
the
the
the
tricks
that
it's
a
great
great
thing
to
bring
up,
because
I
think
the
tricks
to
that
are
you
know?
What
can
you
know
about
a
student
when
it
when
it
when
they
hit
a
page
right?
You
know
they
hit
the
page.
A
A
E
A
Yeah
so,
regardless
of
our
of
our
say
in
the
matter,
we
want
to
be
able
to
provide
different
ways
of
measuring
those
things
and
even
maybe
bolting,
on
additional
tools
being
able
to
integrate.
With
other
tools
like
I
know,
I
think
eberle
created
a
tool
called
docuscope
that
I
think
it's
docuscope,
that
that
tracks
eye
movement
on
a
page
and
they're
testing
that
out
with
certain
courses.
E
Docuscope
is
for
completely
different.
Yeah
docuscope
is
for
analyzing
student
texts,
not
eye
tracking.
E
A
Got
mixed
up,
there
is
an
eye
tracking
tool
like
I
don't
know
what
it's
called
at
the
moment
yeah,
but
thanks
for
correcting
me
on
that.
But
we
do
have
something
like
that.
We
just
we're
not
using
it
readily
because
there's
so
much
testing
that
needs
to
be
done
on
that,
but
yeah
being
able
to
provide.
A
Even
if,
if,
if
we
think
activities
are
the
best
way
to
measure
engagement,
we
still
have
to
provide
different
ways
of
being
able
to
measure
that
and
and
maybe
even
new
ways
that
we
haven't
even
thought
of
yet
so
good
point
that
engagement
information
is
probably
one
of
the
most
requested
pieces
of
data
from
our
system
that
I've.
You
know
that
I
hear
a
lot
of.
F
I
do
want
to
bring
up
kind
of
on
the
opposite
side,
the
surveillance
level
of
that
which
is
pretty
intense,
especially
if
you're
getting
any
information
other
than
that
they
looked
at
it
like
when
they
looked
at
it.
That
is
pretty
intense
surveillance
and
that
accessibility
concerns
around
people
who
access
things
differently
like
if
somebody
needs
to
copy
it
into
another
application
to
adjust
the
text
somehow
or
if
they
need
to
print
it
or
if
they
like
need
to
download
it
for
online
access
and
how
that'll
affect
the
system.
A
Yeah,
I
wonder
how
much
guidance
we
have
to
provide
when
we
provide
those
tools
to
say
you
know
what
you're
actually
measuring
is
this
and
you
can
they're
actually
proxies
for
for
engagement.
F
Oh
yes,
and
also
if
we
have
that
functionality,
if
somebody
will
try
to
use
it
for
grading
and
then
that
will
become
an
accessibility
problem
or
or
something
like
that.
A
It's
so
I've
never
had
to
think
so
deeply
about.
A
F
Yeah,
and
also,
I
think
somebody
mentioned
this
somewhere
else,
where
I
was
that
something
these
things
can
take
kind
of
systemic
overhead.
That
can
be
a
problem
if
people
have
less
good
computer
systems
because
it
makes
it
harder
to
for
things
to
load.
If
there's
more
processes
running.
A
A
But
yeah
thanks,
michael
though,
for
bringing
up
the
engagement
thing
it
you
know
and
you
it's
almost
it's
almost
always
a
measure,
that's
being
that's
attempting
to
be
measured.
Even
if
that's
not
the
experiment,
right
is
how
do
we
know
when
students
are
actually
in
an
online
system
actually
reading,
actually
watching
actually
doing
what's
what's
running
or
if
it's
just
running
in
the
background
as
they're,
you
know
watching
tv
or
something
else
well,.
E
This
comes
with
one
of
these
summer
courses
I
participated
in.
We
had
a
short
course
probably
only
took
about
two
three
hours
to
get
through,
but
we
were
wondering
at
the
start
of
the
first
session,
how
far
people
had
gotten
through
it,
and
that
was
the
simple
question.
Is
we
and
we
couldn't
answer
that
question
at
that
point?
We
could
only
say
well,
they
got
through
this
activity,
but
we
also
when
we
pulled
the
people.
There's
like
oh
yeah.
We
read
all
the
way
through,
but
we
just
stopped
doing
the
activities.
A
Yes
right,
in
fact
this
just
came
up.
I
have
a.
I
have
a
to-do
item
in
my
long
list
that
I
might
actually
because
I
get
asked
this
question
all
the
time
I
get
asked,
how
many
hours
does
it
take
a
student
to
get
through
this
material?
You
know
we
built.
We
build
these
semester-long
courses,
semester-long
courses
that
vary
a
number
of
learning
objectives
from
you
know
I
I've
seen
semester-long
courses
be
anywhere
from
40
to
80.
A
You
know
just
trying
to
quantify
how
big
a
course
is
and
how
long
it
takes
any
given
student
to
work
through
it.
It's
funny
and
nobody
asked
that
of
a
textbook,
but
I
always
get
asked
about
it
in
terms
of
our
online
courses
and
I'm
to
the
point
where
I
might
just
have
a
cohort
of
student
workers
just
just
take
courses
so
that
I
can
get
timing
estimates
and
be
able
to
provide
more
clear
information
about.
A
The
denseness
of
it,
you
know
too
and
and
the
number
of
activities
I'm
sure
that
makes
a
huge
difference
too,
so
that
that
instructors
could
have
more
information
to
guide
students
and
actually
assign
realistic.
You
know
expectations.
A
But
yeah
so
that's
another
reason
I
would
want
to
know
kind
of
timing
just
just
to
know.
E
A
Aggregate
measures
yeah,
but
this
leads
into
you-
know
this
idea
of
maybe
at
some
point
a
marketplace
of
different.
You
can
imagine
a
marketplace
of
different
online
courses
around
the
same
topic
that
has
that
have
different
characteristics.
Right
one
could
be,
you
know,
amount
of
time
it
takes
to
get
through
it.
One
could
be
reading
level,
one
could
be
data
performance
data
in
the
aggregate.
You
know
this
course
performs.
A
It
would
be
nice
to
have
all
of
that
kind
of
information
around
any
given
course
be
able
to
contrast
and
compare
and
say:
okay,
I
have
this
particular
student
population,
so
it
looks
like
this
course
might
work
better
for
me
than
this
one.
Because
of
these
reasons.
F
Doing
something
like
this
by
aggregate,
I
agree,
does
have,
I
think,
like
a
lot
of
those
advantages
and
also
kind
of
parallels,
how
we
do
formative,
assessments,
right
and
they're,
also
aggregate
and
thereby
potentially
makes
it
less
likely
that
people
will
gain
the
system,
because
I
know
when
you
know
when
I
think
somebody
was
just
mentioning
that
you
know
people
said
they
they
read
things
I
mean
I'm
sure
everyone
also
has
the
ex
experience
where
people
say
they
read
something
and
they
didn't,
and
it's
just
that
they
know
that
it's
expected
that
you
say
that
you
do,
and
so.
F
A
Yeah
one
of
the
best
lessons
I
ever
learned
in
life
is
you
you
get
what
you
measure
right
like
if
you,
whatever
you
measure,
if
you're
measuring
productivity
well
by
golly
people,
people's
productivity
will
look
like
it's
getting
better,
but
at
the
expense
of
what
probably
quality
so
yeah
interesting
yeah
right
now,
like
you
said
sasha,
we
only
we
only.
This
is
a
complaint
we
have
in
our
system,
often
where
we
only
provide
the
formative
activities
in
the
learning
dashboard.
A
And
one
of
the
reasons
for
that
is
to
provide
what
what
norm
is
described
as
a
safe
space
for
practice
right
like
because
if
we,
if
we
start
allowing
grading
based
on
that
or
on
formative
practice,
you
know
students
just
they'll
either
gain
the
system
or
they
won't.
They
won't
use
it
the
way
we
want
them
to
use
it.
A
You
guys
have
all
given
me
good
stuff
to
think
about,
I'm
actually,
probably
gonna
wrap
up
in
about
five
minutes
and
so
you've
given
me,
like,
I
said
lots
of
things
to
think
about
in
terms
of
even
just
setting
up
experiments,
experiment,
setting
up
baby
testing,
irb
considerations
and
the
types
of
data,
even
in
there's
a
lot
on
the
types
of
data
that
you
all
are
looking
to
collect
or
know
about
any
given
set
of
courses,
or
course
this
is
really
going
to
help
me
figure
out
at
least
start
to
set
up
some
workflows,
four,
four
experimentation
for
a
b
testing
and
know
where
to
dig
in,
like
I,
I
said
before,
I
think
I'm
going
to
have
to
go
and
maybe
start
with
our
irb
office
to
say.
A
Okay,
if
we're
going
to
do
these
things,
what
how
involved
do
they
need
to
be?
What
are
things
that
we
haven't?
Thought
of
that
should
be
considered,
probably
starting.
There
is
a
good
good
place,
because
they'll
probably
give
me
more
to
think
about
than
any
anyone
else
and
again.
This
won't
be
the
the
last
the
last
of
discussions
about
a
b
testing.
A
I
will
say
in
the
next
tourist
community
meeting,
which
is
august
not
august
6th.
It
is
actually
I
didn't
change
this.
It's
october,
1st.
A
And
this
is
going
to
be
a
a
quick
demo
of
what
exists
in
tourists
at
the
moment,
and
I
would
love
to
hear
if
anybody's
used
any
of
the
any
tourists
yet
what
their,
what
their
feedback
and
and
experiences
have
been.
A
But
if
you
haven't,
then
I
would
really
want
to
just
see
your
reactions
to
the
interfaces
as
they
are
now
and
the
functionality
that
exists
so
far.
And
then
I
can
give
a
deep
dive
into
the
road
map
to
see
what's
coming
next
and
what
we're
working
on
next
so
yeah.
Please
join
us
in
that
meeting.
A
A
So
I
want
to
just
thank
you
all
for
joining
today
and
participating
and
giving
me
a
lot
to
think
about,
and
I
hope
you
all
have
a
really
lovely
long
weekend.
Yeah
you
too
aaron
thanks
mark.