►
Description
Dr. Julia Stoyanovich
"Teaching Responsible Data Science"
Julia Stoyanovich is an Assistant Professor at New York University in the Department of Computer Science and Engineering at the Tandon School of Engineering, and the Center for Data Science. Julia's research focuses on responsible data management and analysis practices: on operationalizing fairness, diversity, transparency, and data protection in all stages of the data acquisition and processing lifecycle. She established the Data, Responsibly consortium (https://dataresponsibly.github.io/), and served on the New York City Automated Decision Systems Task Force, by appointment from Mayor de Blasio. Julia developed and is teaching courses on Responsible Data Science at NYU (https://dataresponsibly.github.io/courses/spring20/). In addition to data ethics, Julia works on management and analysis of preference data, and on querying large evolving graphs. She holds M.S. and Ph.D. degrees in Computer Science from Columbia University, and a B.S. in Computer Science and in Mathematics and Statistics from the University of Massachusetts at Amherst. Julia's work has been funded by the NSF, BSF and by industry. She is a recipient of an NSF CAREER award and of an NSF/CRA CI Fellowship.
A
A
A
So
let
me
just
start
because
I
have
very
limited
time
today
with
a
quick
overview
of
the
courses
that
I'm
mentioning
here.
I've
been
teaching
responsible
data
science
at
nyu
for
three
years,
always
in
in
the
spring
semester,
and
the
first
offering
in
spring.
2019
was
a
special
topics
course,
where
I
had
18
masters
and
phd
students
in
the
data
science
and
computer
science
programs
and
three
people
auditing
in
2020.
A
We
already
had
46
students
with
four
people
auditing
and
just
this
year,
2021
the
course
was
also
rolled
out
to
the
undergraduate
students
at
nyu.
It's
a
requirement
of
our
new
bachelor's
degree,
undergraduate
bachelor's
degree
in
data
science,
and
so
I
caught
a
lot
of
this
course
with
a
colleague,
george
wood
concurrently
for
undergrads
and
for
graduate
students.
A
We
had
60
students
in
each
of
these
120
and
there
were
very
long
waiting
lists
before
enrollment,
so
there's
quite
a
bit
of
interest
in
this
material
and
the
course
is
seeing
substantial
growth.
The
audience,
as
I
mentioned,
are
undergraduate
and
graduate
students
in
technical
disciplines,
data
science
and
computer
science.
Primarily,
the
only
prerequisite
is
introduction
to
data
science
or
computer
science.
Machine
learning
is
not
a
prerequisite
for
this
course,
and
this
is
deliberate.
A
This
is
because
I
really
want
to
make
sure
that
we
catch
students
before
they
get
they
go
too
deeply
down.
The
machine,
learning
and
ai
pass,
and
the
challenge
here
primarily
again
is
to
reconcile
technical
training
with
interdisciplinary
ethical
concerns.
I
quoted
the
course
with
george
wood,
who
is
a
sociologist
this
past
year.
A
This
is
a
real
technical
course.
We
have
lectures
once
a
week
and
we
have
hands-on
labs
also
once
a
week
and
of
course,
this
past
spring,
we
taught
it
in
blended
mode,
meaning
that
most
students
were
remote,
although
somewhere
on
campus,
so
ny
you
have
in-person
instruction,
and
so
we
had
to
deal
with
those
challenges
and
ultimately,
across
the
two
courses
under
grad
and
grad,
we
had
two
lectures
scheduled
and
four
labs
again,
just
to
reinforce
that
this
is
a
technical
course.
A
We
had
several
homeworks,
we
had
a
team
project,
there
was
a
final
exam.
There
were
programming
labs,
the
students
would
hand
on
a
hand
in
after
the
the
lab
and
the
course
project
had
a
as
its
deliverable,
a
kind
of
a
nutritional
label
for
an
automated
decision
system
of
the
student's
choice,
and
I'm
happy
to
speak
about
that
later.
During
the
the
discussion.
Essentially,
a
nutritional
label
is
a
kind
of
an
audit
of
an
automated
decision
system
that
the
students
chose.
A
A
The
first
is
algorithmic
fairness,
and
here
I'm
giving
you
just
a
taste
of
what
the
course
web
page
looks
like.
Please
go
ahead
and
explore
it
yourself.
We
have
a
reader
that
accompanies
every
course
module
there
isn't
a
textbook.
Yet,
although
I'm
hoping
to
to
make
some
progress
towards
that
starting
this
summer
and
then
also
google
collab
notebooks
are
posted
here
for
the
labs
for
a
particular
week
in
the
first
module.
One
of
the
topics
we
cover
is
bias
in
predictive
analytics.
The
interpretation
of
that
term.
A
A
I'm
not
going
to
have
time
to
go
into
the
details
here,
but
this
is
just
to
give
you
the
flavor
of
some
of
the
presentation
materials
that
that
I
developed
and
that
the
course
uses
the
second
module
speaks
about
the
data
science
life
cycle
and
the
responsibility
concerns
ethics,
legal
compliance,
privacy,
data
protection,
data
quality,
the
permeate
the
data
science
life
cycle.
So
here
again
the
module
overview.
There
is
a
reader
that
that
we
compiled
george-
and
I-
and
here
we
return
again
to
a
conversation
on
bias
and
computer
systems.
A
Here,
I'm
showing
the
three
types
of
bias
that
are
due
to
buckley
friedman
and
helen
nisenbaum
from
their
1996
paper
depicted
as
a
dragon
here
with
three
heads,
one
head
for
each
type
of
bias,
and
we
focus
on
technical
and
emergent
bias
in
this
module,
specifically
reinforcing
that
to
fight
bias.
You
have
to
state
your
beliefs
and
assumptions
explicitly
you
cannot
simply
relax.
A
A
A
A
In
this
technical
course,
we
draw
content
from
fairness,
accountability
and
transparency,
from
data
engineering
and
from
security
and
privacy.
We
learn
algorithmic
techniques
for
data
analysis,
and
we
also
learn
about
recent
legal
and
regulatory
frameworks,
and
the
main
point
that
I
reinforce
throughout
the
course
is
that
the
problems
that
we
encounter
here
are
not
purely
technical.
They
are
socio-technical,
and
so
technology
alone
cannot
solve
these
problems.
A
That
all
being
said,
my
perspective
is
still
that
of
a
pragmatic
engineer,
not
a
technology
skeptic,
and
so
I'm
never
gonna
say
that
we
should
not
be
using
technology.
We
should
not
be
using
data
science
rather
that
we
should
be
careful
about
how
we
use
it
and
for
what
purpose
and
how
we
check
whether
in
fact
it
works
and
what
impacts
it
has
on
society.
A
The
last
thing
I'll
say
is
that
there
is
a
comic
book
series
that
I
have
been
developing
with
the
brilliant
fala
arif
khan.
Images
that
you
saw
today
are
also
due
to
fala,
and
this
is
an
ongoing
project
that
we
have.
We
have
so
far
released
two
comics
in
this
series
and
I
use
them
as
supplementary
reading
for
the
course
these
are
sort
of
at
the
level
of
survey
papers
for
a
technical
audience.
So
please
check
them
out.
A
The
first
one
is
called
mirror,
mirror
it's
available
in
english,
spanish
and
french
so
far,
and
we're
looking
to
translate
into
other
languages,
and
the
last
thing
I'll
mention
is
that
I'm
also
in
collaboration
with
the
queen's
public
library
and
p2pu
peer-to-peer
university
that
we've
developed
a
public
education
course
called.
We
are
ai
that
is
currently
being
piloted
at
the
queen's
public
library
in
new
york
city,
and
it
also
comes
with
a
comic
book
series.
These
are
much
shorter
and
generally
accessible,
so
please
check
them
out
and
thank
you
very
much.
B
C
So
I
guess
I
have
a
quick
question,
for
you
showed
us
a
slide
of
someone
that
tried
to
like
adopt
your
course
as
part
of
this
project
by
taking
your
course
and
then
teaching
it
a
week
later,
I'm
having
tried
to
adopt
other
people's
course
material
and
then
teach
it.
I'm
curious
how
how
that
went
for
her
and
whether
she
found
it
was
like
she
was
able
to
do
it
or
she
there
was
things
that
she
struggled
with
because,
like
this
is
super
exciting,
like
I'm.
C
This
was
a
great
looking
course
and
I
would
love
to
be
able
to
either
take
some
chunks
of
it
or
take
all
of
it
and
do
it
at
my
at
mining
or
college
as
well,
because
I
think
it
covers
a
lot
of
things
that
I
really
like
to
share
with
my
students.
But
I
also
want
to
make
sure
I
can
be
successful
with
it
right.
A
Right
great
question
and
thank
you
for
asking
that
question,
because
you
know
part
of
the
reason
that
I'm
excited
to
be
speaking
to
this
audience
today
is
because
I
do
want
people
to
take
the
course
and
to
adopt
it,
so
it
wasn't
just
that
debbie
came
to
class.
She
also
came
to
every
ta
meeting.
We
worked
very
closely
together.
She
and
I
to
make
sure
that
the
material
is
appropriate
for
her
students.
So
this
was,
of
course,
the
first
exercise
of
this
sort
right.
A
She
took
extensive
notes,
and
over
the
summer,
she
and
I
are
going
to
sit
down
together
to
actually
come
up
with
a
kind
of
a
guideline
for
folks
wanting
to
adopt
these
materials.
But
none
of
this
is
going
to
be
automatic.
I
don't
expect
it
to
be
that
you
can
just
take
the
stuff
and
run
with
it,
and
this
is
part
of
the
problem
with
tech.
B
A
A
It
limits
the
kinds
of
modules
we
can
use
also
to
teach,
because
not
all
software
runs
on
collab.
Not
all
libraries
do,
but
we
have
this
sort
of
canned
set
of
modules.
The
most
difficult
thing
is
coming
up
with
assignments,
because
we
can't
repeat
exams,
for
example
right
so
for
those
of
you
who
decide
to
adopt
portions
of
this
course.
A
B
A
I
know
the
session
yeah
I
put
this
all
into
etherpad
is
called
right,
but
I
can
also
paste
into
chat
I'll
paste
them
to
chat.
Thank
you,
of
course,
my
pleasure
and
if
the
comics
are
brilliant,
take
a
look.
I
can
say
it
because
I
am
not
the
sole
author
and
like
the
brilliance,
is
due
to
fallout
my
quota
they're,
just
really
funny
so
check
them
out.
A
B
C
So
I
I
guess
I
have
another
inner
question.
I
hope
that
might
be
interesting
for
people
to
hear
about,
because
I
went
to
a
workshop
earlier
this
year,
where
there
was
actually
an
ethicist
presenting
as
part
of
the
computer
science
workshop
at
a
computer
science
conference,
and
so
my
question,
you
know
you
sort
of
mentioned
this
challenge
with
impossible.
Imposter
syndrome.
C
Did
you
actually
work
with
having
an
ethicist
or
a
philosopher
someone
with
a
phd
in
philosophy
or
something
come
in
and
help
you
with
making
sure
that
you're
constructing
all
that
content
correctly
and
and
sharing
it?
It
sounded
like
you,
didn't,
have
one
in
the
classroom,
teaching
with
it,
but
like
there's
still
certain,
certainly
components
that
they're
experts
on
and
I'm
kind
of,
curious
to
see
whether
you
worked
with
them
or
whether
you
just
sort
of
tried
to
learn
it
yourself
and
come
up
with
it.
A
So
in
the
initial
iterations
I
I
learned
myself
and
still
for
the
most
part,
it's
essentially
just
you
know,
take
the
oxford
or
I
think
it's
the
cambridge
ontology
of
moral
philosophy
and
go
through
that
right.
So
that's
on
my
desk,
I
learned
from
sociologists
more
than
I
learned
from
philosophers,
so
I
got
a
lot
from
matt's
organics
book
bit
by
bit
and
my
initial
kind
of
round
of
teaching
consequentialism
versus
deontology.
For
example,
it's
almost
verbatim
from
that
book
this
time
around
I
caught
out,
as
I
mentioned,
with
george
wood.
A
He
has
a
phd
in
sociology
from
oxford
and
he
has
been
exposed
also
to
philosophy
concepts
this
past
year
I
have
been
working,
though,
very
closely
with
the
political
philosopher
who
is
the
academic
granddaughter
of
roles.
John
rolls
who
is
very
influential
in
this
field,
specifically
in
fairness
to
machine
learning,
and
so
the
second
volume
of
the
technical
comic
book
series
that
we
used.
A
I
also
used
in
the
course
is
based
on
this
paper
that
we
wrote
together.
Her
name
is
eleni
moniz,
so
eleni
mannis
fala
and
I
wrote
a
paper
called
fairness
and
friends
in
which
we
map
political
philosophy,
concepts
to
their
interpretations
and
peer
machine
learning,
and
we
contest
some
of
these
mappings
etc.
So
this
is
difficult
work
right,
but
we
have
started.
We
have
started
down.