►
Description
Presenter: Karl Schmitt
Institution: Trinity Christian College
A
We're
going
kind
of
in
a
completely
different
direction,
so
this
is
actually
work
from
well
over
a
year
ago,
when
we
had
a
sort
of
sub-working
group
around
program
assessment,
and
so
we
have
a
report-
I,
don't
I
I,
don't
know
if
it
ever
got
posted
somewhere
where
people
can
actually
access
it.
But
I'm
gonna
give
you
the
the
vast
majority
of
it
in
this
presentation
and
then
maybe
Renata
or
someone
from
the
south
Hub
to
get
the
document
posted
somewhere.
A
So
the
subgroup
originally
was
tasked
with
this
idea
and
sort
of
came
out
of
my
struggle
with
recognizing
that
we
have
to
have
program
assessment
that
is
useful
for
developing
and
growing
programs,
but
then
also
the
fact
that
it
because
data
science
is
new
and
because
lots
of
places
are
sort
of
standing
up
these
programs
and
don't
have
the
space
to
build
out
a
productive
assessment
model.
Asking
questions
of.
A
Can
we
come
up
with
stuff
that
is
shareable
across
institutions
for
sort
of
a
successfully
modeling
and
successfully
putting
out
assessment
tools
and
or
rather
program
program,
level
assessment?
And
so
we
talked
as
a
committee
through
a
couple
of
different
topics
such
as
thinking
about
the
really
technical
outcomes,
things
around
professional
outcomes
and
then
things
around
dispositions.
A
No
hang
on
I,
don't
know
why
we're
not
getting
it.
Oh
there.
It
is
okay,
I,
just
didn't
click
enough
right.
So
to
start
with,
we
really
looked
at.
We
took
the
committee's
groups
and
looked
at
what
learning
outcomes
we
had
in
in
place
in
terms
of
programs
that
were
stood
up
in
some
other
places
and
and
sort
of
recognizing
the
fact
that
ACM
had
already
put
out
guidelines,
particularly
mostly
around
the
the
technical
outcomes,
but
also
sort
of
mentioning
in
for
things
some
of
the
other
stuff.
A
In
that
vein,
we
felt
that
the
communication,
at
least,
would
have
already
more
defined
things,
so
we
sort
of
narrowed
in
on
that,
and
what
you
see
here
now
is
the
four
learning
outcomes
that
we
had
pulled
from
existing
programs
that
we
were
all
part
of,
and
we
did
look
out
to
some
other
programs
Etc,
but
these
were
the
ones
that
were
focused
on
undergrad
programs
and
explicitly
articulated
communication.
So
I'll
give
you
about
a
second
to
sort
of
read
through
the
four
that
are
out
there.
A
So,
as
a
committee,
we
decided
to
try
to
write
something
that
we
thought
was
generic
and
sort
of
covered
everything.
And
if
you
read
that
learning
outcome,
you
will
say
wow,
that's
a
lot
and
you
can't
actually
assess
it
and
that's
true.
I've
recently
been
working
with
my
department
to
sort
of
revision,
our
outcomes
and
we
started
here
and
I
said
well.
We
need
to
sort
of
trim
this
down
a
little
bit
and
focus
it,
because
we
still
this
is
in
some
sense
too
encompassing.
A
A
B
A
Wanted
to
make
sure
that
we
were
providing
all
of
the
ways
that
we
could
approach
and
look
at
those
questions.
In
addition,
also
tackling
all
of
the
different.
You
know
modalities
of
communication,
and
so
we
had
to
ask
then
how
do
we
actually
assess
this?
And
how
do
we
go
about
building
something
that
might
show
that
students
are
achieving
this?
A
So
this
is
sort
of
the
one
slide
that
if
you
just
remember
one
slide
here,
it
is
here's
what
you
should
take
away
from
the
rest
of
the
the
rest
of
this
and
we're
going
to
sort
of
dive
into
more
details.
But
the
group
came
up
with
that
outcome,
so
communicate
about
data
models,
data
models,
methods
and
results
of
analysis,
including
use
of
appropriate
context,
limitations
and
biases.
The
audience
is
a
varying
technical
expertise
through
written
oral
individuals.
A
A
We
think
that's.
The
probably
the
best
way
to
show
and
have
an
accessible
actionable
item
is
to
have
students
be
building
out
a
portfolio
as
they
work
through
their
program,
around
communication
artifacts
and
the
various
different
modalities.
So
that
might
be
things
like
code.
That
might
be
a
thing
like
presentations.
A
It
might
be
things
like
written
reports
or
research
reports,
or
things
like
that
and
then
sort
of
following
a
sort
of
a
standard
assessment
process,
then,
is
that
these
portfolios
upon
graduation
or
as
you
work
through,
are
assessed
through
through
standardized
or
works
that
aren't
necessarily
tied
to
the
classes
themselves,
but
are
tied
to
looking
at
the
outcomes
that
you
want
for
your
program
and
I.
Don't
know
that
this
is
any
earth-shatteringly
shocking
recommendation
here.
A
I
think
that
the
thing
that
perhaps
is
a
little
different
from
when
we
think
about
the
technical
expertise
is
that
we're
not
thinking
here
of
giving
students
a
test
or
a
single
point
of
assessment.
We're
really
trying
to
articulate
that
we
want
to
be
collecting
assessment
items
throughout
a
different,
a
couple
different
ways
and
then
also
that
these
are
not
really
a
definitive
like.
Oh,
have
we
hit
a
bar
or
not
the
rubrics.
A
A
A
A
Think
about
working
directly
with
deployment
and
software
experts
that
are
our
their
goal
is
to
deploy
the
machine
learning
that
a
data
scientist
might
develop,
or
that
might
be
turning
that
into
something
that
is
posted
on
a
web
page
for
other
people
to
engage
with
Etc.
And
so
those
are
people
that
are
probably
not
coming
in.
A
With
a
lot
of
domain
expertise,
but
might
have
even
more
coding
or
data
expertise,
you
can
also
think
about
them,
working
with
perhaps
bosses
that
are
far
above
them
that
have
a
whole
bunch
of
domain
expertise
in
whatever
they're
working
in,
but
maybe
have
no
or
little
coding
or
expertise.
Thinking
of
that
as
sort
of
company
leaders
or
whatever,
which
is
not
say
that
leaders
don't
have
data
expertise,
but
very
broad,
sweeping
Strokes
of
examples.
A
This
is
kind
of
a
common
modality
that
we
see
at
a
lot
of
research
schools,
Etc,
where
they're,
bringing
together
diverse
group
and
they're
hiring
in
people
to
do
certain
components
that
they
have
in
those
people
cell
with
and
then
finally,
there's
the
general.
We
sort
of
label
this
as
general
public,
but
you
can
think
of
lots
of
other
people
that
might
fit
into
this
right.
People
that
aren't
living
in
this
technical
World,
either
technical
in
terms
of
coding
and
data
or
coding
and
in
terms
of
domain.
B
A
Yeah,
the
top
level
is
sort
of
like
specialist
versus
generalists
software
developers
is
meant
to
give
you
an
Insight
of
what
might
fall
into
that
category,
without
necessarily
being
explicitly
that
b
category
those
examples
that
are
meant
to
give
you
an
Insight,
not
necessarily
assume
that
that
is
exactly
what
we
would
be
talking
to
right.
So
there
I.
A
B
B
A
If
we
walk
through
these
right,
a
persuasive
essay
on
ethics
of
a
new
data
software
product
that
probably
is
falling
into
so
you
put
up
next
in
my
mind
that
falls
into
things
like
leaders
and
scientists
and
general
public,
so
I
have
sort
of
a
primary
and
a
secondary
category.
Up
here
and
again,
these
are
debatable.
There's
not
a
clear
answer
here.
The
point
is
to
highlight
that,
by
having
a
couple
of
items,
we
can
sort
of
hit
a
portfolio
a
portfolio
perspective
can
hit
different
categories
of
these
communication
requests,
something.
B
A
A
We
provided
a
couple
of
guidelines,
guidances
guidance,
questions
and
and
ideas
for
people
to
work
through
as
they
implement
this
in
their
own
program.
So
this
first,
the
first
two
sort
of
are
high
level
considerations.
Ask
and
this
sort
of
array
this
maybe
addresses
yours
I,
don't
know
it
might
be
an
offensive
way
to
stated
the
thing
and
no
domain
expertise.
Our
goal
here
right
is
to
be
as
clear
as
possible
without
providing
off
offense
and
I.
A
So,
if
you
I,
this
idea
of
working
with
a
two
by
two
grid
helps
us
clarify
the
components
that
we
want
to
focus
on.
The
committee
felt
that
sort
of
that
axis
of
coding
and
data
expertise
made
sense
and
domain
expertise
made
sense,
but
your
program
might
be
interested
and
focused
on
more
on
other
things.
I
think
equally
important
is
Raising
questions
around
do
elements
in
that
grid
need
difference,
different
weighting
or
importance.
A
So
thinking
about
the
fact
that
different
programs
and
places
of
implementation
might
want
to
have
more
items
in
a
portfolio
around
different
topics
or
you
might
want
to
consider
even
not
even
assessing
certain
components,
because
that's
not
what
their
actual
goal
is.
The
idea,
though,
is
to
help
provide
programs
a
way
of
thinking
around
where
those
communication
needs
might
fit
in
terms
of
direct
implementation.
So
I
showed
that
example
of
a
list
of
a
couple
of
different
portfolio
items
that
you
might
choose.
A
Here's
our
sort
of
recommended
idea
for
how
to
go
about
finding
what
those
might
fit
and
again
it's
not
rocket
science
here,
where
we're
sort
of
suggesting
look
at
your
classes.
Think
about
what
source
of
courses
particularly
required
courses
as
we're
building
on
an
assessment
product
occur
already
in
your
program
and
then
look
at
the
kinds
of
assignments
that
come
out
of
it.
A
Don't
don't
look
to
be
producing
new
assignments
in
it
unless
you
can't
find
things
and
then
look
for
artifacts
that
might
fit
into
the
different
grid
cells
so
that
you
can
then
collect
those
I.
Think
the
one
of
the
important
things
to
remind
people
that
aren't
used
to
doing
an
assessment
process
is
that
you
should
probably
be
developing
independent
rubrics
outside
of
the
classes
assessment
of
those
products
that
are
around
the
program,
ideas
and
questions
that
you
want
to
say,
because
you
don't
want
your
program
level
assessment
to
be
impacting
students
grades.
A
A
And
then
sort
of
a
reminder
in
this
process
is
to
take
that
step
back
and
make
sure
that
your
actual
plan
fits
the
needs
and,
and
has
buy-in
really
right
so
have
you
got
rubrics
that
go
with
your
goals?
Have
you
actually
picked
things
that
you
can
collect
for
for
a
program
to
assess
and
then
have
you
really
checked
in
with
your
stakeholders
and
stakeholders?
Here
is
very
broad.
A
Okay
from
a
high
level
from
a
higher
level,
we
wanted
to
sort
of
remind
people
developing
assessment
that
it's
not
just
about
assessing
a
program
level
outcome
and
seeing
whether
that
program
is
being
achieved
or
not.
But
a
good
program
design
goes
further
and
says:
well
are
the
outcomes
that
we're
getting
from
these
assessments
from
these
rubric
items,
providing
us
something
actionable
unchangeable
in
our
program
or
is
it
just
telling
us
whether
we
did
or
didn't
do
something
equally
valuable
in
that
sort
of
Education
assessment
process
is
asking?
A
A
Right
sort
of
wrapping
up
this
is
a
really
high
level
sort
of
like
hey
here's,
an
idea
for
starting
and
making
sure
that
people
that
don't
have
any
assessment
plans
or
haven't
figured
out
how
they
plan
to
assess,
can
sort
of
pick
it
up,
put
it
in
place
and
say
well.
At
least,
we've
got
something
to
start
from,
like
the
working
group
would
love
to
have
had
time
to
develop
out
sample,
rubrics
and
have
example,
items
and
things
like
that.
So
people
could
pick
those
up.
A
We
didn't
have
time,
but
I
think
that's
the
next
step
that
would
be
productive
for
either
a
working
group
or
some
other
research
process
to
do
I.
Think
in
hindsight,
if
we
can
simplify
or
figure
out
a
way
to
break
that,
that
generalized
program
learning
outcome
down
more,
it
would
be
really
helpful
for
other
programs
in
terms
of
replication.
There's
also
not
really
anything
for
comparing
across
programs.
A
If
we
build
out
these
sort
of
portfolio
things,
unless
we
get
standardized
rubrics
or
things
that
could
then
be
applied
at
multiple
programs,
there's
obviously
some
more
work
around
other
topics
for
building
out
sort
of
recommendations
for
assessments,
particularly
the
ethics
and
professional
practices,
sort
of
fits
into
this.
But
we
could
also
think
about
perhaps
generalized
problem
solving
or
applications.
A
All
right
and
we
are
at
11.
so
I'm
going
to
say
thank
you
for
listening
and
I,
don't
know
how
far
Renata
has
scheduled
us
time
out
I'll
throw
up
one
last
Quick
plug.
We
are
running
a
workshop
around
education
for
works
for
data
for
goods,
for
education
in
July.
What
more
I'm
happy
to
talk
about
that
later
and
the
answer.
C
C
So
we
were
looking
forward
to
this
group
being
able
to
present,
so
there
are
I
think
we
have
maybe
like
one
or
two
questions.
Please
put
yours
in
The
Ether
pad
too.
If
we
don't
get
to
them,
because
I
will
ask
the
speakers,
the
ethercron
will
stay
live
till
the
end
of
the
day
and
you
can
answer
questions
directly
there,
because
it's
you
can
directly
type
and
answer
anyone's
questions.
So
let
me
look
at
the
chat.
A
Oh,
you
know
what
Ervin
I
think
what
I
missed
and
where
we
missed
the
written
part
was
in
apparently
when
I
put
it
on
my
slide,
I
put
client-facing
presentation
in
our
text.
We
have
client-facing
presentation
or
report
thinking
about
the
idea
that
sort
of
those
reports
are
headed
out
to
clients
are,
could.
A
C
B
B
A
B
A
I
think
the
reason
we
wanted
I
get
yeah
the
the
there's,
no
specification
around
domain
there,
the
intent
I.
Think
being
that
whenever
you
start
articulating
what
the
domain
is,
then
you
start
having,
like
you
said,
the
general
public
has
a
specific
domain,
and
so
we
can't
articulate
what
domain
it
is.