►
From YouTube: Webinar | Innovative Evaluation in New Mexico
Description
he New Mexico Legislative Finance Committee is continually working to improve research methodologies to ensure staff are providing policy makers with the best and most up-to-date research available. The office is working with the Abdul Latif Jameel Poverty Action Lab (J-PAL) at the Massachusetts Institute of Technology on a randomized control trial on the effectiveness of early college high school curricula in New Mexico. Learn about New Mexico's innovative research methodology and J-PAL's State and Local Initiative during this one-hour webinar.
A
Name
is
Edward
Tyler
and
I
am
an
economist
at
the
Louisiana
legislative
auditor's
office
and
I
will
be
moderating
our
webinar
today,
I'd
like
to
thank
the
National
legislative
program,
evaluation,
Society
and
the
National
Conference
of
State
Legislators
for
sponsoring
this
webinar
NLP
es
is
the
NCSL
staff
Association
for
legislative
staff
involved
in
program,
evaluation
or
performance
auditing.
One
of
its
key
purposes
is
promoting
professional
development
opportunities
like
this
webinar
for
legislative
staff.
So
we
are
pleased
that
we
could
bring
you
this
program
today.
A
Over
the
next
60
minutes,
we
encourage
participation
through
our
chat
box,
so
feel
free
to
type
your
questions
and
answer
any
questions
in
the
chat
box,
which
is,
in
the
lower
left
hand
corner
of
your
screen
to
build
some
comfort
with
the
chat
function
and
also
to
learn
who
is
on
the
line
today.
Please
feel
free
to
type
the
state
from
which
you
are
calling
or
listening
in
from,
and
I
also
want
to
mention
the
resources
for
the
webinar.
Above
the
presentation,
you
will
see
a
couple
of
tabs.
A
B
C
B
So
our
office,
the
legislative
finance
for
me,
has
been
around
since
1957
were
full-time
nonpartisan
legislative
staff.
We
have
three
main
groups
in
our
office:
a
group
of
fiscal
analysts
that
develop
budget
recommendations
for
agencies
and
also
do
some
policy
analysis
around
agency
activities
and
performance.
We
also
have
economists
that
do
revenue
estimation
for
the
state's
finances,
as
well
as
a
team
at
program,
evaluators
that
ensure
accountability
and
government
by
performing
evaluations
of
various
state
agencies,
initiatives,
programs,
etc.
Our
program
evaluation
covers
pretty
much
all
functions
of
state
government.
B
We
typically
focus
on
k-12
education,
as
well
as
higher
education,
early
childhood
programs,
healthcare
and
public
safety,
but
also
cover
many
other
many
other
areas
and
our
projects
typically
take
between
three
to
six
months.
We
work
in
small
teams
to
perform
quantitative
and
qualitative
analysis
and
we
present
our
conclusions
and
recommendations
to
legislators
as
well
as
sometimes
other
stakeholders
state
entities,
occasionally
the
public
as
well.
B
They
started
working
internationally
and
now
they
also
work
domestically
and
we
received
an
award
from
their
state
and
local
innovation
initiative,
and
that
initiative
supports
local
and
state
leaders
and
offices
to
identify
you
know
which
policies
or
programs
work
and
which
work
best,
and
you
know
to
try
to
understand
why
they
work
and
how
we
can
possibly
improve
them.
The
word
that
we
received
the
loudest
access
and
technical
support
from
JPL,
so
we
worked
with
a
researcher
as
well
as
some
of
their
office
staff
to
help
develop
our
our
methodology.
C
B
It
an
Early,
College,
High
School,
is
a
dual
and
wrote
a
dual
enrollment
model
that
helps
students
graduate
from
high
school.
Well,
they
also
earn
college
credits
and
sometimes
career
certifications
through
a
higher
education
Institute.
So
typically
high
school
is
paired
with
a
college
university
or
Community
College
and
students
will
simultaneously
get
high
school
credit
and
college
credit.
B
The
early
college
high
school
differs
from
a
traditional
dual
credit
curriculum,
in
that
it
offers
a
cohesive,
integrated
curriculum
that
integrates
both
the
high
school
and
college
level
coursework.
So
typically,
students
start
out
with
high
school
coursework
and
then
we'll
move
into
college
coursework
during
the
second
and
third
year.
B
The
program
both
of
these
programs
are
three
years
long
early
college
high
schools
can
be
either
located
on
a
college
campus
or
at
a
high
school,
but
will
have
professors
from
the
college
typically
or
accredited
high
school
teachers
that
are
teaching
a
college
portion
of
the
program.
Thirdly,
College
High
School
model
is
really
designed
to
target
students
that
are
underrepresented
in
higher
education,
so
typically
that
would
be
low-income
students
or
students
who
would
be
the
first
in
their
family
to
go
to
college
the
models
and
evidence-based
practice.
B
B
One
of
the
more
comprehensive
studies
has
been
done
to
date
was
done
by
the
American
institutes
for
research,
and
you
can
see
that
say
is
a
difference
in
the
percentage
of
students
that
enrolled
in
college
between
early
college
high
school
attendees
and
then
a
control
group
that
did
not
attend
the
early
college
high
school.
So
why
did
we
decide
to
evaluate
Early
College
High
School's
in
New
Mexico
kind
of
two
main
reasons?
One
is
that
this
models
really
grown
a
lot
nationally.
It
launched
in
this
kind
of
a
wide
scale
way
around
2001.
B
The
Gates
Foundation
launched
an
early
college
high
school
initiative,
and
that
was
around
the
same
time
that
we
started
seeing
Early
College
High
School's
pop
up
in
New
Mexico.
So
today
there
are
about
3000
students
across
20
schools.
So
it's
grown
a
lot
and
I
guess
you
could
say
it's
gotten
a
lot
of
hype
both
nationally
and
here
in
New
Mexico.
So,
for
example,
our
public
education
department
has
started
a
certification
program,
which
is
great.
We
definitely
you
know,
want
to
get
more
kind
of
rigor
around
certification.
B
However,
we
haven't
done
any
research
in
our
state
about
if
this
is
actually
working
for
students,
so
we
felt
like
it
was
important
to
look
into
that
then.
The
second
reason
why
we
decided
to
this
evaluation
was
that
we
realized
that
the
the
way
that
Early,
College
High
School's,
admit
students
is
a
great
opportunity
for
experimental
or
quasi-experimental
analysis.
B
B
So
you
know
before
we
did
this
research,
we
already
knew.
We
already
had
some
correlational
research
around
early
college,
high
schools
in
the
state.
So,
for
example,
we
knew
that
on
some
measures
of
student
and
school
performance
they
definitely
perform
better.
For
example,
you
can
see
in
the
first
chart
that
Early
College
High
School's
get
better
school
grades
created
by
our
public
education
departments,
they're
a
lot
more
likely
to
get
A's
or
bees
than
public
schools
in
general
in
New
Mexico,
and
we
also
knew
that
they
had
higher
four-year
graduation
rates
than
their
districts
overall.
B
B
On
the
right
about
half
of
students
at
early
college,
high
school
they're
eligible
for
free
and
reduced
lunch,
that
number
is
over
two-thirds,
both
statewide
and
then
in
the
districts
where
the
early
college
high
schools
are
located
early
college
high
schools
also
have
lower
shares
of
Hispanic
and
Native
American
students
and,
in
addition,
students
and
early
college
high
schools
could
also
differ
on.
You
know
other
important
characteristics.
B
You
know
they
may
have
more
family
support.
They
may
have
differing
academic
performance
before
they
come
into
the
Early
College
High
School.
So
you
know
it's
given
that
it's
really
hard
without
doing
experimental
or
quasi-experimental
research
to
attribute
any
positive
outcomes
in
Early
College
High
School's
to
the
schools
themselves.
B
So
that
was
really
what
we
wanted
to
answer.
Does
attending
the
early
college.
High
school
lead
to
better
outcomes
versus
just
do
students
who
go
to
Early
College
High
School's
happen
to
have
better
outcomes,
so
I'm
going
to
turn
it
over
to
Sarah
to
talk
a
little
bit
more
about
specifically
what
we
did
and
what
we
found.
C
Thanks
Alison,
as
you
did
a
really
good
job
of
introducing,
we
really
wanted
to
examine
whether
a
Canadian,
Early,
College,
High
School,
would
lead
to
improved
post-secondary
outcomes
or,
like
college,
enrollment
and
college
degree
attainment
compared
to
students
who
did
not
attend
an
early
college
high
school
and,
as
you
also
set
up
one
of
the
reasons.
What
we
need
to
do.
A
posit
experimental
analysis
or
experimental
analysis
for
these
types
of
program
is.
A
C
The
groups
are
very
different
for
the
students
who
attend
these
types
of
schools
for
those
individuals
who
do
not,
and
so
initially
our
hope
was
to
use
a
randomized
control
trial,
because
many
of
these
schools
use
it
some
type
of
lottery
setup.
However,
as
we
started
digging
more
into
the
data,
we
learned
that
only
two
schools
randomize
their
lotteries
and
also
retain
their
records,
and
so
that
created
our
initial
hurdle
as
to
whether
we
could
do
a
two
randomized
control
trial
or
not.
C
If
you
look
at
the
figure
on
the
right
of
the
screen,
you'll
see
that
for
most
lottery,
steps,
you'll
have
an
applicant
pool,
and
then
these
applicants
will
go
into
some
type
of
admissions
lottery
and
then
will
be
selected
either.
As
an
early
college,
high
school
student
or
a
non
early
college
high
school
student
and
from
there
we
can
measure
our
outcomes,
which
was
which
were
degree,
attainment
and
college
enrollment.
However,
as
we
learn
more
about
these
programs,
we
found
out
that
a
number
of
the
schools
had
an
admissions
lottery
that
wasn't
truly
randomized.
C
This
was
a
key
challenge
within
the
evaluation,
because
initially
we
had
this
expectation
that
we
were
doing
a
randomized
control
trial,
and
that
is
what
we
had
applied
for
with
JPL
and
Alison
will
talk
a
little
bit
more
about
some
of
those
challenges
and
lessons
learned
as
we
move
forward.
So
within
the
study
we
used
records
from
existing
data
that
we
had
from
the
school
to
retain
the
records,
so
we
knew
who
had
applied
to
enroll
in
Early
College,
High
School,
as
well
as
who
did
enroll
in
Early
College
High
School.
C
Additionally,
we
collected
data
from
the
National
Student
Clearinghouse,
which
is
a
national
database
of
individuals,
college
entrance
and
degree
attainment,
and
we
were
able
to
access
this
data
for
a
nominal
fee
for
each
student.
Once
we
received
this
information,
we
analyzed
the
differences
in
between
these
two
groups
and
in
to
help
us
do
this.
We
worked
with
JPL,
as
Allison
has
already
briefly
mentioned.
C
Moving
forward.
What
we
saw
was
that
we
we
saw
that
there
was
a
significant
difference
in
any
degree
attainment,
so
individuals
who
enrolled
in
early
college
high
school
were
more
likely
to
have
any
type
of
degree
compared
to
our
control
group,
and
this
is
characterized
mostly
by
associates
and
bachelor's
degrees,
which
is
that
first
figure,
where
we
see
that
four
associate's
degrees
there
was
a
I
was
like
a
three
point.
Difference
yeah.
B
C
Three
or
four
point
difference
for
Sousa
streets
as
well
as
for
bachelor's
degrees,
and
then
there
was
a
non
significant
difference
for
matches
for
master's
degrees,
which
we
don't
get
into
here,
and
that's
simply
because
we
weren't
looking.
We
were
only
had
a
six
year,
longitudinal
or
framework,
and
because
we
didn't
have
a
large
variety
name,
we
couldn't
really
see
any
effects
in
terms
of
masters
degrees.
C
Additionally,
on
all
of
these
analyses,
we
controlled
for
our
free
and
reduced-price
lunch
and
English
learners
status
to
make
sure
that
there
weren't
any
significant
demographic
differences
in
these
groups
that
could
lead
to
differences
in
their
enrollment
or
degree
attainment.
We
also
see
that
students
who
were
in
an
early
college
high
school
for
the
full
treatment
effect,
which
was
three
years,
had
a
significantly
higher
rate
of
degree
attainment
than
those
individuals
who
were
only
in
the
program
for
one
year
or
those
who
were
not
enrolled
in
the
program.
C
Based
upon
this
information,
we
concluded
that
early
college
high
school
seemed
to
be
fairly
promising
and
that,
through
enrollment
in
early
college
high
schools
that
you
can
have
better
post-secondary
outcomes
specifically
if
you
are
enrolled
for
the
full
treatment
effect.
However,
we
need
to
have
more
rigorous,
statewide
evaluations
of
these
programs,
and
we
actually
suggested
this
to
our
public
education
department.
Hopefully
they
are
working
on
that.
C
B
Sarah
so
I'll
talk
about
a
few
things
that
we
learn
during
this
study,
which
may
be
relevant
to
other
legislative
offices,
are
thinking
about
doing
experimental
or
quasi-experimental
research.
So
you
know
one
thing:
that's
really
important.
To
think
about
is
what
the
the
feasibility
or
how
feasible
it
is
for
your
office
to
do
a
prospective
study
versus
a
retrospective
study.
So
you
know,
if
you
have
the
opportunity
doing,
a
prospective
study
definitely
allows
for
more
rigorous
research
because
you
can
design
the
study
you
can,
you
know
hopefully
have
some
influence
over
the
parameters.
B
You
can
kind
of
gather
the
data
that
you
need,
making
it
more
likely.
That
will
be
truly
randomized.
However,
as
was
the
case
for
our
office
and
probably
some
other
legislative
offices
timeline
scope
capacity
can
make
it
pretty
difficult
to
do
this.
Some
prospective
studies,
you
know,
could
take
years,
and
we
simply
do
not
have
the
time
to
do
that
on.
B
Hand,
retrospective
studies
are
definitely
a
little
easier
because
the
data
already
exists
or
you
know,
theory
exists,
but
the
trade-off
there
is
that
you
don't
have
a
lot
of
control
over
how
the
date.
Well,
you
have
no
control
over
how
the
data
was
collected,
so
you
might
not
always
have
you
know
enough
data
or
not
high
quality
data
right.
B
Study
so
you
know
we
hope
they
do,
that
I
think
that
would
definitely
allow
for
some
good,
rigorous
research
and
then
a
related
point
on
the
data
and
the
quality
of
the
data
is
just
getting
data
as
early
as
possible
and
being
able
to
verify
the
quality
and
availability
of
it
before
you
start.
So
you.
B
We
were
able
to
to
get
some
of
the
data
while
we
were
still
planning
but
weren't
able
to
get
data
from
it
from
as
many
schools
as
we
had
hoped,
so
that
did
somewhat
impact
what
we
were
able
to
do
with
the
study,
also
finding
that
if
the
lotteries,
you
know
possibly
weren't
completely
randomized
kind
of
made
us
change
this
from
a
truly
experimental
to
a
quasi
experimental
study,
and
then
it's
also
important
to
think
about
data
sharing
with
a
lot
of
experimental
or
quasi-experimental
research.
There's
a
lot
of
individual
level
data.
B
So,
for
example,
here
we
were
working
with
student
data,
you
know
which,
as
you
might
expect
it's
fairly
sensitive
to
the
public
education
department.
Understandably,
it's
pretty
hesitant
to
release
it,
so
it
took
us
several
months
to
be
able
to
get
the
right
agreements
and
kind
of
security
protocols
in
place,
especially
because
we
were
also
sharing
it
outside
researchers
that
add
another
layer
of
complexity.
B
So
just
important
to
you
know
think
about
that
before
you
get
started
in
terms
of
conducting
randomized
control
trials
or
experimental
research,
I
think
there's
a
lot
of
exciting
opportunities
for
legislative
offices.
So
some
of
the
approaches
that
were
thinking
about
looking
at
our
you
know,
programs
that
lend
themselves
to
experimental
research,
so
that
could
be
like,
in
this
case,
a
program
that's
oversubscribed
and
determined
admission
by
lottery
or
possibly
by
waitlist.
B
So
some
other
programs
that
could
meet
that
criteria
would
be
drug
courts
in
our
state
or
we
know
the
Health
Department
has
an
oversubscribed
program
that
provides
some
services
to
expectant
mothers.
That
also
relies
on
a
waiting
list,
also
programs
that
are
rolled
out
in
phases.
For
example,
there's
not
enough
funding
or
capacity
to
roll
them
out
all
at
once.
B
What
one
thing
worth
noting
is
is
just
that
it
we
found
it
was
important
to
clarify
project
expectations,
so
you
know,
for
example,
J
called
really
focuses
on
randomized
control
trials
and,
as
we
described,
we
had
to
sort
of
shift
the
scope
of
this
to
more
quasi
experimental
research
and
they
were
able
to
work
with
us,
and
you
know,
help
us
kind
of
rethink.
How
that
would
work,
but
yeah,
just
important
to
kind
of
know
what
your
partner's
goals
or
parameters
are
in
addition
to
what
your
office's
goals
and
parameters
are
there?
C
But
there's
one
thing
about
JPL
that
I
think
might
be
also
useful:
they
partner
you
with
a
researcher
so
that
can
help
with
other
technical
expertise.
So
if
JPL
themselves,
don't
actually
understand,
don't
have
the
answer
to
something
they'll
make
sure
that
you're
working
with
somebody
else
so
that
your
research
can
hopefully
end
up
being
published
as
well
as
you
have
that
depth
of
background
knowledge,
yeah.
B
A
All
right,
Sarah
and
Allison.
Thank
you
so
much
for
your
presentation.
It's
a
certainly
a
very
interesting
topic
and
exciting
to
see
you
know
so
much
research
on
with
randomization,
so
I
wanted
to
go
ahead
and
invite
members
of
our
audience
to
ask
questions.
Please.
If
you
haven't
done
so
already,
please
type
your
questions
in
the
chat
box
at
the
lower
left
of
your
screen
and
I
will
go
ahead
and
forward
them
to
the
panelists
we'll
try
to
get
to
as
many
questions
as
possible.
A
So
actually
I
have
just
to
start.
One
of
the
questions
I
have
I
mean
randomization
I.
Think
it's
actually
very,
very
timely
right
now,
because
I
think
earlier
I
think
it
was.
This
week
they
announced
that
the
winners
of
the
Nobel
Prize
in
Economics
for
this
year
were
three
researchers,
Esther
Duflo
I'm
not
going
to
try
to
pronounce
a
dr.
Bernard
G's
name
and
my
other
was
Kramer
with
the
three
who
won
all
three
of
whom
were
have
been
doing:
randomized
experiments
and
I.
Guess
one
of
the
things
I
wonder
is
in
New.
A
C
So
in
New
Mexico
we
have
a
process
called
legislating
for
results
and
we
focus
on
performance-based
budgeting
and
our
legislature
is
really
interested
in
making
sure
that
we're
evaluating
programs
and
making
sure
they
work
and
along
that
those
lines
they
are
interested
in
us
doing
randomized
trials.
But
their
main
focus
is
on
us
examining
programs
in
a
timely
manner.
So
there's
a
kind
of
a
give-and-take.
A
C
One
thing
that
our
state
just
passed
was
a
Senate
bill,
focused
on
performance
management
and
looking
at
program
inventories
of
different
agencies
to
ensure
that
we
as
a
state
are
making
our
using
evidence-based
programs.
But
in
terms
of
randomization,
is
something
that
is
an
ideal.
But
it
doesn't
stop
us
from
doing
other
projects
as
we
can't
randomize.
Something
and.
A
Thank
You
Sara
answer
Sara
Alison
for
answering
my
question.
It's
yeah.
It
is
exciting
to
hear
that
your
Legislature
has,
you
know,
really
supported
this
idea.
So
we
have
a
question.
I'll
just
go
down
the
list.
We
have
first
question
from
Karen
and
my
Gordy
and
Kieran
asks:
how
does
your
office
get
assigned
I've
evaluations?
Are
you
able
to
select
the
topics
that
lend
themselves
to
this
type
of
method.
B
Yeah,
that's
a
good
question,
so
we
we
get
assigned
evaluations,
kind
of
it's
a
bit
of
a
two-way
street
or
I.
Don't
know
probably
multiple
way
street.
Our
legislators,
you
know
sometimes
will
kind
of
submit
ideas,
requests
for
programs
I
want
us
to
look
at,
and
then
we
have
staff.
You
know
both
team
members
and
our
managers
also
submit
ideas
of
programs.
We
want
to
look
at
so
we
always
kind
of
have
a
running
list
and
then,
in
terms
of
you
know,
specifically
assigning
projects
to
teams
or
individuals.
B
Again,
it's
kind
of
based
on
individual
skill
sets
and
interests
combined
with
programs
that
are,
you
know
that
we
haven't
looked
at
in
a
while
or
that
have
kind
of
come
up
in
a
come
up
for
for
budget
budgeting,
and
we
want
to
you
know,
know
how
much
to
budget
or
we
have
concerns
about
performance.
So
it's
it's
a
mix
of
factors.
C
Yeah,
the
only
thing
I
would
add
to
that
is
we.
We
tend
to
follow
the
money.
So
it's
so
we
as
a
state
sent
along
on
Medicaid
and
education
is
probably
many
other
states
do,
and
so
because
of
that
we
usually
have
some
evaluation
going
on
that
focus
on
education
and
another
evaluation
usually
focus
on
some
type
of
Medicaid
or
health
care
health
care
piece.
A
B
A
B
And
that's
a
good
question,
because
that
was
really
what
we
were
dealing
with
here
and
that
was
ultimately
why
well.
That
was
one
of
the
main
reasons
why
we
decided
that
this
really
couldn't
be
a
truly
randomized
study,
and
that's
why
you
know
we're
saying
it's
quasi
experimental,
so
and
I
guess
just
to
give
a
little
bit
more
context
on
on
that.
What
happened
with
the
the
school
lotteries
that
we
were
looking
at.
You
know
we
knew
that
students
went
into
the
lottery
and
then
in
theory,
you
know
were
picked
out
at
random.
B
However,
there
was
a
waitlist
and
some
students
on
the
waitlist
ended
up
getting
in
because
other
students
had
had
dropped
out
or
had
gotten
in,
but
decides
not
to
go
and
that
process.
If
a
student
is
accepted
via
lottery,
besides
not
to
go,
and
then
if
a
student
on
the
waitlist
then
has
the
opportunity
to
again
go
or
not
go
that
process
isn't
random
because
there's
obviously
some
decision-making
going
on
there
and
that
could
be
affected
by
student
characteristics.
You
know
you
can
imagine
what
they
are.
B
They
could
be
anything,
you
know:
family
involvement
in
the
process
or
prior
academic
background,
or
something
like
that.
So
you
know
I
I,
guess
in
that
case,
because
this
is
a
retrospective
study
and
the
data
already
existed.
There
was,
unfortunately,
you
know
nothing.
We
could
really
do
to
to
kind
of
get
around
that,
so
we
did
have
to
shift
this
to
a
quasi-experimental
study.
Obviously
or
I
don't
know,
obviously,
but
if
this
were
a
prospective
study,
you
could
hopefully
work
with
the
schools
to
say:
hey,
you
know.
B
C
How
you
would
die
on
that?
If
there's
a
lot
of
issues
right,
because
usually
you
want
to
look
at
the
intent
to
treat
and
if
an
individual
was
supposed
to
receive
the
service,
but
didn't
they're
still
supposed
to
be
put
in
the
treatment
group,
and
so
this
can
create
a
lot
of
gray
areas.
When
you're
doing
the
analysis.
C
So
I
would
suggest
that
you
make
sure
that
you
can
document
all
of
the
differences
between
the
groups
as
possible,
and
then
you
can
do
some
different
statistical
analyses
as
well,
which
won't
make
your
study
into
a
randomized
control
trials,
but
will
allow
you
to
control
for
some
of
these
effects
and
therefore
give
to
give
you
more
reliable
results.
So.
A
A
B
A
Well,
thank
you
Sarah
and
Allison.
We
have
another
question
from
Eric
Beecroft
in
Virginia,
and
actually
this
is
similar
to
a
question
I
have
did
you
need
to
get
permission
from
comparison
group
members
to
access
their
data
from
the
National
Student,
Clearinghouse
and
I
have
a
piggyback
question.
Maybe
you
could
you
could
answer
with
that,
and
that
is
a
you
know.
How
did
that
work?
Did
you
did
you
go
through
one
of
your
New
Mexico
higher
ed
institutions
to
get
into
or
did
you
or
did
you
negotiate
that
with
them?
Specifically?
B
They
have
a
you
know:
collaboration
with
almost
every
higher
education
Institute
in
the
country,
not
every
single
one,
but
but
most
so
the
schools
are
sending
their
their
student
data
directly
to
National
Student
Clearinghouse.
So
in
that
sense,
no,
you
don't
have
to
get
permission
from
the
comparison
group
because
their
data
is
already
in
the
National
Student
Clearinghouse
I,
don't
know
exactly
how
it
works.
You
know
at
the
college
level
they
are
released,
they're
kind
of
management
of
student
data,
but
once
the
data
gets
the
National
Student
Clearinghouse,
you
can
access
it.
B
So
you
can
access,
you
know
full
names
and
date
of
birth
and
then
a
bunch
of
information
about
when
they
enrolled
what
their
major
was
degrees
obtain.
Graduation
things
like
that.
We
work
directly
with
National
Student
Clearinghouse.
They
have
a
service
called
student
tracker
that
you
know
different
entities
and
stakeholders
can
subscribe
to.
So
it's
based
on
a
per
student
cost
with
some
savings
for
requesting.
You
know
more
and
more
data,
so
you
know
definitely
I'd
encourage
people
to
reach
out
to
student
tracker.
A
B
A
C
So
we
classified
mrs.
Claus
experimental
because,
as
we
said,
there
was
this
non-random
nature,
and
so
we
analyzed
the
data
first
to
see
what
the
differences
were
between
the
two
groups.
We
looked
at
enrollment
and
some
of
some
of
the
other
demographic
information
that
we
had
and
then
we
analyzed
it.
A
All
right,
so
it
sounds
like
you
know,
the
general
principles
you're,
comparing
the
treatment
is
whether
they
went
to
the
one
of
these
early
college,
high
schools
and
so
you're.
Looking
at
the
post
treatment
outcomes
of
you
know
whether
they,
you
know
how
they
did
with
their
higher
educational
attainment
and
you
were
able
to
control
for
their
pretreatment
demographic
things
like
free
and
reduced
lunch
and
an
English
Learner,
saddest
things
like
that.
Yes,.
A
B
Add
to
that
you
know
in
theory,
we
could
have
controlled
for
as
many
variables
as
we
had
data
on,
so
the
records
from
our
public
education
department
did
indicate
students,
FRL
status,
English,
learner
status,
ethnicity,
disability
status,
gender,
so
we
could
have
controlled
for
more
factors.
We
thought
those
were
probably
the
two
most
significant.
A
Excellent.
Thank
you
for
your
answer.
We
have
a
question
from
Deanna
Ahmad
and
she
asks
how
are
your
findings
presented
to
the
state
legislature
in
New
Mexico?
What
kind
of
response
did
you
receive
and
do
you
know
about
any
plans
to
broaden
the
availability
of
early
college
high
school
in
the
state.
B
Yeah,
so
we
presented
our
findings
at
a
hearing
over
the
summer
to
our
committee
members,
legislative,
Finance,
Committee
and
I.
Think
some,
you
know
tangible
impacts
or
changes
that
we're
hopefully
going
to
see.
So
for
one
thing,
we
worked
a
little
bit
with
a
public
education
department
around
their
their
certification
process.
B
I
think
I
mentioned
they
had
even
before
we
started
this
had
already
been,
and
the
process
is
developing
a
certification
for
Early
College,
High
Schools,
which
is
really
good
because
previously
pretty
much
any
school
could
just
call
themselves
from
every
college
high
school,
and
this
will
put
some
really
clear
criteria
around.
You
know
what
it
means
that
you
have
to
have
a
certain
type
of
curriculum.
Your
teachers
have
to
be
accredited.
B
You
know
you
have
to
offer
X
Y
&
Z
programs,
so
you
know
we're
discussions
with
them
about
kind
of
using
some
more
outcomes,
based
criteria
or
evaluation
in
that
process,
and
then,
in
addition,
you
know,
as
I
said,
we've
also
talked
to
them
about
their
proposal.
Their
grant
proposal
to
do
a
prospective
study
on
Early
College
High
School's,
which
we
would
hopefully
maybe
be
involved
in
in
some
way
and
I.
Think
in
terms
of
the
the
response
from
legislators.
I
think
they
were.
A
You
well
thank
you,
Seren
allison,
for
your
answer
that
question
and
we
have
actually
I
think
a
related
question
from
Brenda,
but
I
think
Brenda's
asking
if
there
were
any
changes
in
state
law,
I
think
she's
asking
you
know
where
the
the
previous
question
was
maybe
dealing
with
education
policy.
Maybe
maybe
you
could
also
talk
a
bit
about
requirements
for
schools
to
maintain
data
or
anything
that
that
legislators
would
do
to.
You
know
bring
that
your
your
performance-based
budgeting
in
line
with
something
that
that
can
be
you
know,
looked
at
experimental
or
quasi-experimental
ii.
B
We
chose
not
to
focus
a
lot
on
at
this
point,
just
because
we
were.
We
were
working
collaborative
collaboratively
with
the
schools
and
asking
them
to
share
data
with
us,
and
we
felt
like
if
we
presented
this
as
some
kind
of
like
gotcha,
like
compliance
exercise
that
you
know
could
have
been
kind
of
counterproductive.
B
It
also
could
have
been
kind
of
counterproductive
in
the
way
we
were
working
with
the
public
education
department.
So
we
did
at
this
point,
I
want
to
keep
it
pretty
collaborative
and
just
focus
on
on
these.
You
know
outcomes
from
the
schools
that
we
were
able
to
evaluate,
but
that
did
come
up
as
a
concern,
and
so
you
know
we
do
hope
to
panel
look
into
that
a
little
bit
more.
A
That
we
had
an
audit
on
our
charter
school
lotteries.
One
of
the
one
of
the
subjects
was
that
they're
required
to
hold
lotteries
so
that
they
can
make
sure
that
they're
getting
an
equal
proportion
or
the
right
proportion
of
economically
disadvantaged
students,
I
think
one
one,
rural
parish.
They
their
lottery,
was
being
done
by
you
know,
lawyers,
you
just
you
know,
you
know
there
was
no
record
no
records
of
it.
It
sounded
like
something
you
know:
that'd
be
very
challenging
to
audit
something
like
that.
We
have
another
question.
A
This
is
a
very
technical
question
from
the
jail
Ark
and
they're
asking
there
have
a
specific
question
about
propensity
score.
Matching
I.
Think
you
all
mentioned
earlier
that
you
were
using
that.
What
are
your
thoughts
on
recent
literature
showing
that
propensity
score
matching
increases
imbalance
in
control
groups
and
is
that
something
that
y'all
have
have
looked
into
and
are
you
have
any
alternatives
to
using
propensity
score
matching?
What
are
your
thoughts
on
that
subject
so.
C
I'm
aware
of
this
literature,
but
I
haven't
had
a
chance
to
delve
into
it,
so
I
don't
want
to
talk
too
much
about
it.
Unfortunately,
I
think
what
we've
been
doing-
we've
kind
of
had
a
debate
in
our
office,
even
whether
we
should
be
doing
in
parental
stats
on
some
of
the
data
that
we
have
because
oftentimes
we're
looking
at
population
data
and
so
when
you're
looking
at
population
data,
sometimes
there's
an
argument
that
you
should
have
been
using
an
inferential
statistics.
C
Inferential
statistics
is
based
on
the
fact
that
you
are
using
a
sample,
and
so
we
haven't
kind
of
figured
out
yet
the
best
way
to
go
about
some
of
these
things
sometimes
will
do
HLN's,
rather
than
propensity
score
matching
to
try
to
deal
some
of
the
other.
Some
of
the
issues
that
way
and
I
don't
think
that
fully
answers
our
question
but
I
tell
every
the
literature
I
don't
want
to
kind
of
speak,
say
more
about
it.
A
Where
I
think
I
think
fully
answering
that
question,
we
would
would
require
a
pretty
you
know
innovative
new
idea,
and
it
certainly
something
that
I
think
a
lot
of
us
would
be
interested
in.
If
there's
a
you
know,
some
kind
of
improved
way
of
doing
treatment
effects
measurement
in
a
quasi-experimental
setting
like
this
I
had
a
question
also
that
I
wanted
to
ask.
There's
a
I
know
in
some
papers,
I've
seen
with
randomized
controlled
trials.
They
have
this.
A
B
I,
don't
know
that
would
be
cool.
I
know
what
they're
doing,
at
least
for
the
state
and
local
innovation
program
is
yeah.
Helping
state
and
local
governments
to
perform
RCTs
I
do
think
on
their
website.
They
have
some
kind
of
catalog
or
at
least
examples
of
the
projects
they've
worked
on,
so
you
know
certainly
may
be
a
good
resource
I
think
to
reach
out
to.
If
you
are
thinking
about
a
specific
type
of
type
of
research,
or
you
know
specific
policy
intervention
that
you
want
to
look
at.
A
Well,
then,
that's
good
to
know
and
one
other
question
I
have.
Of
course
you
know
coming
at
this
from
an
econometric
standpoint.
You
know
we're
almost
always
dealing
with
quasi
experimental
situations
and
there
are
a
lot
of
different
techniques
that
economists
try
to
use
to
to
do
a
natural
experiment.
You
know
you
could
use
difference
in
differences
or
instrumental
variables,
regression,
discontinuity,
bunching
and
other
techniques,
and
the
idea
of
all
these
is
that
you
use
some
of
you.
I
mean
they
the
term
they
use
is
exploit.
A
But
you
know
it
sounds
bad,
but
you
you
make
use
of
a
some
kind
of
quirk
in
the
way
that
the
program
is
set
up.
I
think
you
mentioned
this
even
where
you're
talking
about
you
know,
program
is
being
rolled
out
over
multiple
facilities
and
if
there's
some
way
to
you
know
randomize.
That
assignment
process
is
that
is
the
joint
poverty
action
lab.
You
know
a
proponent
of
using
these
kind
of
econometrics
techniques.
Are
they
really
more
focused
on
randomized
controlled
trials?
And
how
do
you
approach
that
distinction.
B
C
Any
discussions
with
them
on
that.
So
that
would
be
a
good
thing
to
talk
to
them
about.
I.
Think
that
we
might
we're
going
to
try
and
share
their
contact
information
as
well,
and
but
one
thing
I
would
say
so.
I
think
some
of
it
had
berries
on
who
you're
working
with
there,
because
they
have
a
very
extensive
team,
and
so
some
individuals
who
specialize
on
a
number
of
different
techniques,
and
so
potentially
they
would
have
some
one
or
two
individuals
who
know
more
about
Nam
econometrics.
B
A
And
are
they
is
it?
Is
it
a
particular
department
that
it's
most
closely
associated
with
you
know
or
is
it
you
know
going
to
be
the
I,
don't
know
what
their
if
they
have
a
public
administration
program
there?
But
today
is
it.
You
know
economics
or
sociology
psychology,
education
is,
it
is
associated
with
one
department
or
across
disciplinary
yeah.
A
Well,
terrific
yeah
I
would
just
ask
the
last
question:
we
wanted
to
get
that
contact
information
for
the
joint
poverty,
action
lab
and
I
see.
Allison
has
entered
it
in
the
chat
box
and
certainly,
if
yeah
there's
there's
a
the
email
address
for
the
contact
person.
So
certainly
if
it's
something
you're
interested
in
that
would
be
the
person
to
contact.
A
So
with
that
again,
I'd
like
to
thank
our
panelists
Sarah
and
Allison
thanks
to
n
LPS
for
setting
this
up
and
thanks
to
all
of
you
out,
there
who've
been
joining
us
today.
This
webinar
has
been
recorded
and
archived
and
it
will
be
available
on
the
NCSL
website
next
week.
So
this
webinar
is
now
concluded
thanks
again
for
listening
goodbye.