►
Description
Presented by Frances Haugen and Patrick Phelps, Pinterest
How do Jupyter Notebooks change how data intensive teams work? How do we build heterogeneous data communities? This talk focuses on how the analytical teams at Pinterest adopted Jupyter Notebook and Jupyter Server and how it changed how analysis is undertaken and communicated. We'll discuss the social side of data science and why unifying data acquisition, analysis, and presentation fundamentally changes the way insights are generated. Tradeoffs between ease-of-use and reusable code production will also be covered along with security implications of adoption in an enterprise context.
About GitHub Universe:
GitHub Universe is a two-day conference dedicated to the creativity and curiosity of the largest software community in the world. Sessions cover topics from team culture to open source software across industries and technologies.
For more information on GitHub Universe, check the website:
https://githubuniverse.com
A
B
A
data
product
manager,
the
thing
I
focus
on
is
working
on:
building
user
oriented
algorithms,
so
historically,
we've
only
optimized
for
how
likely
you
are
to
save
a
pin,
we're
moving
towards
more
user
driven
metric
goals.
Like
variety
like
people
like
to
see
a
variety
of
different
things
is
how
they
don't
it's,
how
they
keep
from
getting
bored.
That's
the
kind
of
thing
that
a
data
product
manager
works,
something
cool.
A
B
Like
to
start
out
by
asking
a
question,
how
many
of
you
work
at
companies
where
there
is
an
isolated
analytical
function,
so
you
might
call
these
people
data
scientists
or
analysts
or
business
operations.
Analysts
like
we
definitely
do
now.
Most
people
who
work
in
software
work
at
places
where
analysis
and
quantitative
investigations
are
isolated
into
these
functions.
The
reason
for
that
is
many
fold
and
we're
going
to
go
into
that,
but
there's
a
lot
of
consequences
for
this
organizational
decision.
B
The
best
kind
and
most
impactful
kinds
of
data
science
insights
are
ones
that
join
both
rich
rigorous
statistical
tools
with
subject
matter
expertise.
The
best
question
that
you're
going
to
ask
the
best
investigation
you're
going
to
do
does
not
start
from
the
first
question
you
ask,
but
often
the
second
or
third
or
fourth
question
that
you
discover,
as
you
go
down
the
rabbit
hole
of
that
investigation
when
you
isolate
out
into
its
own
function,
the
analytical
investigations,
usually
those
investigations
end
with
that.
B
First
question:
you'll
ask
an
analyst
to
go,
find
you
an
answer
to
a
question
and
they'll
bring
you
back.
The
answer
to
that
question
really
good
analysts
will
often
do
another
hop
down
that
line,
but
they
very
rarely
will
go
with
it
with
a
level
of
persistence
or
curiosity
that
someone
who
is
actually
immediately
working
on
those
problems
would
do.
B
The
other
challenge
is
a
lot
of
companies
try
to
get
around
this
by
having
analyst
managers
data
science
managers
who
functionally
act
as
almost
like
anthropologists,
but
go
in
there
and
ask
you
all
these
questions
about
why
you
want
to
know
this?
And
what
are
you
going
to
do
with
this?
And
what
have
you
asked
before?
And
it's
because
when
you
isolate
out
this
function,
you
actually
have
to
care,
have
a
huge
amount
of
communication
overhead
to
actually
get
enough
context
to
get
good
answers.
B
So,
what's
the
alternative,
Patrick
and
I
are
very
big
advocates
for
what
we
call
heterogeneous
data
communities.
So
in
any
organization,
even
outside
of
software,
you
have
a
wide
array
of
stakeholders
who
could
be
effective
users
of
data,
not
just
consumers,
but
people
who
actually
go
and
ask
questions
and
solve
them
using
data.
A
heterogenous
data
community
doesn't
require
everyone
to
have
a
statistics
PhD.
It
has
organizational
practices
and
tooling
that
allows
people
to
go
and
be
these
equal
participants,
or
at
least
be
able
to
ask
informed
questions.
B
A
Why
doesn't
this
happen?
Why
do
we
have
analytics
teams
today
and
the
basic
answer
comes
down
to
the
data?
Science
is
hard
and
in
sort
of
the
course
of
our
time
doing,
data
science
we've
identified
sort
of
four
core
reasons:
data
science
is
hard
today,
so
the
first
is
that
most
data
sources
we
have
in
the
modern
world
are
incredibly
opaque.
A
classic
example
is
like
a
relational
data,
store,
sink
sequel
when
you
access,
sequel,
there's
some
tables.
These
tables
have
some
columns
that
have
names
at
the
simplest
level.
A
You
really
hope
those
names
describe
what's
in
the
columns,
because
otherwise
haven't
helped
you
at
a
more
complex
level.
We
often
have
opacity
because
really
subtle,
bugs
end
up
in
the
data
you
know.
Last
year
in
August
there
was
a
logging
drop
for
these
days
or
the
client
like
miss
logged.
This
kind
of
event
did
this
table,
give
back
failed
to
fix
that,
or
did
it
not?
What
about
this
other
table
that
I
want
to
join
on?
Are
they
actually
both
spam
filtered?
Are
they
spam
filtered
with
the
same
algorithm?
A
This
can
range
all
the
way
from
someone
who
has
been
doing
data
science
for
10
or
15
years,
you
know
and
kind
of
knows
all
the
statistical
methods
you
could
possibly
hope
to
care
about
down
to
your
sales
people
who
might
be
you
know,
looking
at
some
dashboard
and
trying
to
decide
like
in
this,
for
example,
this
plot,
like
is
this
going
up
over
time?
Is
this
good
for
our
advertiser?
Maybe
it's
just
noise.
Is
it
real?
A
All
of
these
sorts
of
things
are
different
skill
levels
in
an
organization,
and
we
think
it's
important
that,
like
all
of
these
skill
levels,
can
contribute
part
of
what
makes
that
skill
level
gap
so
hard
is
that
spreading
best
practices
today
in
data
science
is
a
really
really
complicated
affair.
What
tends
to
happen
is
that
you
know
there's
these
sort
of
experts,
they
sit
in
one
part
of
the
company,
they
work
on
one
specific
thing,
and
then
they
have
to
go
to
a
bunch
of
meetings
and
talk
to
people
and
what
happens?
A
Is
they
go
to
meetings?
Someone
else
has
done
some
analysis.
Sort
of
everyone
in
the
room
is
like
what
does
this
data
scientists
think
the
data
scientists
make
some
Proclamation
or
some
comments,
and
then
the
entire
thing
has
to
repeat,
and
your
data
scientists
end
up
sitting
in
one
our
meetings
to
our
meetings,
multiple
our
meetings,
it's
very
inefficient
for
their
time
and
the
way
companies
normally
try
to
get
around
this
is
they
want
to
just
develop
some
training
and
we've.
A
Actually,
you
know
been
doing
this
ourselves
and
we
can
tell
you
immediately
that
doing
meaningful
training
is
really
hard
in
order
to
gain
data
science,
skill.
You
often
need
many
hours
of
focus,
training
and
the
people
in
your
organization.
You
need
to
work
with
data
can't
spend
the
time
no
one
can
afford
to
send
all
of
their
salespeople
to
ten
or
a
hundred
hours
of
training,
and
no
one
wants
them
to
spend
that
time.
That
way,
even
worse,
it's
very
demotivating.
No
one
wants
to
like
be
like.
A
B
B
One
of
the
things
that
this
results
in
is
we
that
trust
is
guarded
very
preciously.
We
know
with
great
power
comes
great
responsibility
and
we
want
to
make
sure
we
pick
the
people
who
we
give
that
responsibility
to
intentionally
one
of
the
possibilities
for
how
you
could
spread
out.
This
functionality,
though,
is
by
just
having
some
skepticism
I
began
my
career
as
a
not
very
technical
product
manager,
I
had
an
engineering
degree,
but
I
worked
on
ranking
systems
at
Google
and
within
search.
B
B
There
were
so
many
ways
that
I
could
mess
it
up.
There
was
no
way
I
could
be
an
effective
contributor
in
that
way.
Part
of
the
reason
why
this
happens
when
you
have
the
wrong
tooling
is
people
feel
really
bad
feeling.
Skeptical
right,
they
don't
put
most
people
are
not
very
confrontational,
you're,
not
gonna,
go.
When
someone
brings
you
an
interesting
graph
and
say:
could
you
send
me
all
the
code
that
generated
this?
B
Can
I
see
your
work
processes
where,
especially
if
someone
used
a
spreadsheet
to
process
say
a
sequel
query,
even
if
you
get
the
sequel
query,
you
have
no
idea
what
transformations
that
data
undertook
to
result
in
the
final
graph,
because
people
are
conflict
averse
the
end,
because
the
conclusion
is
exactly
what
happened
to
me
when
I
was
22,
people
just
ignore
the
people
who
are
kind
of
outside
the
center
Center
circle
of
trust.
The
last
issue
on
the
human
side,
the
organizational
limitations
side,
is
there
I'm
documentation?
How
many
people
here
like
writing,
documentation?
B
Okay,
we
got
a
few.
My
my
personal
career
tip
is
documentation
is
amazing
because
you
can
learn
anything.
If
you
offer
to
write
someone
else's
documentation,
people
will
sit
and
explain
a
great
length
and
in
great
depth
any
topic
you
wanna,
learn
about.
If
you're
willing
to
write
for
them,
I
don't
mind
it.
It's
how
I've
learned
most
things.
At
the
same
time,
documentation
can't
exist
only
in
those
isolated
circumstances.
You
need
ongoing
processes
of
documentation
because
data
science
is
fundamentally
a
communication
activity
without
having
without
doing
information
capture
knowledge
capture
around.
B
What
are
the
caveat
of
an
analysis
and
analysis
can
become
very,
very
dangerous.
If
you
ever
have
a
conversation
with
a
data
scientist,
they
will
have
lots
of
limitations
on
the
thing
that
they've
just
said:
they'll
say
yes,
this
is
true.
This
could
be
true
right
or
it
appears
that
this
is
true
and
somehow,
as
you
write
that
email
and
it
propagates
out
three
or
four
times
in
a
forwarding
chain,
suddenly
thing
that's
gonna
save
the
company
is
doing.
B
X
like
this
is
true
not
it
could
be
true
in
these
circumstances
being
able
to
get
people
to
actually
write
rigorous
documentation
of
the
assumptions
they
undertook
and
have
that
move
along
with
an
analysis
is
a
very,
very
hard
problem.
As
a
result,
we
end
up
isolating
the
function
of
analysis
even
further
and
limiting
and
who
gets
to
participate.
The
people
who
wear
those
official
data
hats
so.
A
What's
the
alternative,
maybe
before
we
do
that,
there
I
meant,
you
know,
I'm
a
I'm,
a
manager
of
data
scientists
I
want
to
just
reiterate
what
Francis
just
said.
I
worry
extremely
much
about
my
data
scientists
becoming
isolated
and
the
reasons
are
twofold
one:
when
they're
isolated,
they
tend
to
feel
disconnected
from
the
work
they
do
and
they
don't
stay
at
the
company
and
two
when
they
leave
the
company.
All
of
that
history,
it
was
kept
in
their
minds
walks
out
the
door.
So
we
need
an
alternative.
A
Okay,
like
you,
can't
have
some
person
who's,
but
the
company's
six
years
be
your
backup
right,
and
so
what
we
want
to
talk
about
is
how
can
we
get
to
that
alternative?
So
we've
already
covered
why
data
science
is
hard
from
a
technical
standpoint,
and
we
actually
have
a
pretty
good
analogy
for
this.
If
we
think
about
what
data
science
is
today
and
the
tooling
that
exists
for
it,
it's
actually
very
similar
to
how
software
engineering
was
practiced
in
1980
in
the
early
90s
before
we
developed
such
tools.
A
As
you
know,
open
source
before
we
develop
things
like
github
and
in
fact
that
transition
was
so
powerful
for
software
engineering
and
we
need
that
transition
for
data
science,
so
github
was
really
powerful
because
it
made
code
bases
really
non
opaque.
You
could
go
look
at
any
open
source
piece
of
code
and
github
and
see
how
someone
did
something
you
didn't
have
to
trust
what
they
said
you
didn't
have
to
be.
Like
oh
I
heard
a
rumor
that
so-and-so
did
this
thing.
We
should
do
it,
you
could
just
go.
Look.
A
We
also
develop
practices
like
code
review
code
review
was
incredibly
important.
It
makes
software
consistent.
It
uphold
standards.
Today,
most
analytics
groups
don't
have
anything
equivalent
to
code
review.
We
also
from
you
know,
tools
that
we
have
developed
like
ideas,
we
get
extensibility.
We
can
search
what
others
have
done.
We
can
copy
Payson,
it's
most
simple
form,
but
we
can
also,
you
know,
reuse
it.
We
can
modify
it,
we
can
tweak
it.
A
We
can
understand
why
something
was
done
this
way
and
we
can
change
it
and
finally,
one
of
the
most
powerful
things
that
happens
with
code
review
and
with
these
tools
is
you
get
a
peer
feedback
process?
People
learn
by
talking
to
more
senior
software
encoder
reviews
from
them,
these
sorts
of
things,
but
there's
still
these
social
issues
right.
B
Brings
us
to
tooling
data
intensive
communities
are
incredibly
sensitive
to
the
tool
choices
they
make.
Most
of
us
have
been
told.
Most
of
most
companies
navigate
this
issue
of.
How
do
we
make
data
decentralized?
How
do
we
make
it
accessible
to
people
by
going
out
and
paying
a
vendor
a
truly
huge
amount
of
money
to
gain
some
kind
of
new
dashboarding
tool?
Maybe
it's
tableau,
maybe
it's
one
of
the
many
many
other
providers.
It
seems
like
there's
a
new
startup
at
any
given
moment
that
says
you
can
do
anything
with
our
dashboard.
B
The
problem
with
dashboards
is
many-fold,
so
the
first
is
any
dashboard
that
is,
has
a
low
enough
barrier
to
entry
that
people
in
general
can
use.
It
also
doesn't
have
a
wonderful
runway
that
people
can
become
independent.
The
second
one
is
that
any
dashboard
that
you
see
that
you
don't
actually
make
yourself
has
assumptions
baked
into
it
that
you
can't
actually
work
backwards
from
so
like
we
talked
about
before
about
how
part
of
why
data
is
dangerous.
Is
that
it
feels
objective.
B
B
The
secondary
thing
is
even
thinking
about
how
you're
going
to
think
through
a
problem
can
be
shifted
by
the
visualizations
that
you're
presented
with
when
someone
goes
through
and
makes
choices
about
how
to
walk
you
through
a
cognitive
process
on
the
dashboard
you're,
probably
not
aware
of
how
you've
already
had
biases
introduced
and
how
into
how
your
even
think
about
solving
these
problems.
The
second
danger
of
dashboards
is
that
we
tend.
We
have
trouble
understanding
the
genealogy
of
any
given
dashboard.
So
even
let's
say
it's
designed
just
right
for
your
use
case.
B
B
You
can
go,
look
it
up
and
maybe,
while
you're
looking
at
someone
will
come
by
and
say
hello,
interesting,
it's
not
the
dashboard
from
2008
like
no
one
uses
that
dashboard.
That
dashboard
is
totally
wrong
and
you're.
Even
me
like
well,
it's
on
the
wiki
like
for
it
I
heard
there
was
a
dashboard.
The
wiki
said
there
was.
It
was
a
dashboard
when
you
have
systems
that
make
it
very
easy
to
generate
dashboards.
B
It's
true
that
you
can
at
least
diffuse
information
more
effectively,
but
if
you
don't
have
a
way
of
actually
tracking
or
or
understanding
when
things
go
stale
or
when
they
expire,
you
end
up
hat,
introducing
some
problems
that
can
be
just
as
worse
as
people
not
having
an
access
to
that
data.
In
the
first
place,
the
last
one
is
spreadsheet
Zilla.
So
this
is
the
way
most
companies
do
analysis.
When
you
do
give
people
the
ability
to
do
ad
hoc
analysis,
people
go
and
write
an
SQL
query.
They
dump
maybe
a
file
of
data
somewhere.
B
Maybe
they
play
with
it
in
a
spreadsheet
and
then
they
generate
a
graph
and
maybe
there's
even
a
fourth
step.
Third
transformation
stuff
that
happens,
maybe
they
write
up
an
email
that
just
has
that
finding
maybe
the
graph
pasted
in
or
maybe
a
slide
the
problem
with
this
is
anytime.
You
start
spread
sending
a
spreadsheet
around.
Eventually
you
encounter
another
spreadsheet
that
says
something
else,
and
suddenly
you
might
find
yourself
in
the
situation
where
your
boss
is
asking
Jayne
had
this
spreadsheet
and
you
have
that
spreadsheet,
which
sped
your
spreadsheet
is
right.
B
If
you
don't
keep
together
and
you
don't
keep
a
way
of
tracking
the
genealogy
of
where
your
data
came
from,
how
it
was
processed
and
what
the
end
result
was,
you
have
no
way
to
go
and
work
backwards
and
mailed
a
debug
wire.
The
numbers
no
longer
dream.
The
thing
that
we
actually
want
people
to
have
is
what
we
call
data
independence.
A
So
what
is
an
alternative
at
Pinterest?
The
alternative,
we're
trying
is
the
Jupiter
notebook
now
I
hope
most
people
in
this
room
have
heard
about
the
Jupiter
notebooks
and
have
some
idea
what
they
are,
but
just
in
case,
what
Jupiter
notebooks
are
is
an
open-source
software
project
that
allows
you
to
combine
richly
formatted
text,
data
and
code
all
in
one
place
and
we'll
show
some
examples
of
what
they
look
like
in
a
few
seconds.
A
What's
most
important
to
understand
about
Jupiter.
Notebook,
though,
is
that
they
were
designed
from
research
groups.
They
were
designed
from
academia
and
that's
actually
a
really
weirdly
good
thing.
Most
things
we
don't
actually
want
what
academics
design
they're,
not
very
good
at
coding.
I
was
an
academic.
I
can
say
that,
but
it
turns
out
for
for
this.
This
is
the
culture
we
need,
and
this
interactional
environment
is
really
powerful
because
it
combines
all
these
things
and
the
reason
it
was
developed.
A
This
way
is
that
research
groups
today
operate
very
much
how
companies
need
to
operate
when
you
are
doing
some
sort
of
data
analysis
in
a
research
group
or
in
a
company
you're
going
to
have
wildly
different
skill
levels
and
in
research
groups
these
are
sort.
For
instance,
the
professor,
the
professor,
has
been
doing
this
research
for
4050
years
right,
I've
seen
all
of
the
evolve
like
all
of
the
evolution,
but
has
very
little
time.
Then
you
know
the
professor
has
postdocs.
A
They
have
grad
students
and
they
even
have
undergrads
and
undergrads
from
coming
in
the
door,
possibly
right
out
of
high
school
and
have
very
little
experience
with
working
with
data
understanding.
What
is
going
on
in
the
scientific
process
and
all
of
these
sorts
of
things-
and
this
is
very
similar
to
how
analytics
functions
can
operate.
You
need
to
have
a
tool
that
encourages
new
people
should
come
on
board
and
they
should
be
productive
from
day
one.
They
should
figure
out
how
to
use
the
tool
they
should
be
able
to
scale.
A
So
let's
talk
about
some
examples
right.
So
one
of
the
things
you
put
our
notebooks
allow
us
to
do
that's
really
powerful.
Is
they
allow
us
to
combine
these
various
pieces
of
information
so
in
this
case,
for
instance,
we're
seeing
a
viewer
of
a
3d
plot
right?
This
is
a
way
to
visualize
a
probability
distribution,
so
we
can
do
stats,
but
we
can
also
see
like
how
was
this
plot
generated
we
can
see.
Why
does
it
appear
the
way
it
does?
A
We
can
ask
like,
as
they
ask
them
this
notebook,
what
happens
when
we
rotate
this
sample?
All
of
these
sorts
of
questions
are
suddenly
possible.
You
cannot
do
this
if
this.
This
plot
is
in
a
spreadsheet
right
or
if
it's
in
a
slide
deck
you
might
be
like.
Well,
it's
a
two-dimensional
representation,
because
that's
what
our
tool
can
generate
and
I
hope
that
was
enough
information.
A
You
can
also
combine
many
forms
of
rich
information,
so
you
can
combine.
You
know
different
forms
of
media
from
outside
the
company
from
internal
and
you
can
remind
them
of
rich
text.
Finally,
one
of
the
things
we've
really
found
powerful
at
Pinterest
about
Jupiter
and
the
notebook
in
particular.
Is
you
can
have
plots
like
this?
You
can
have
plots
that
are
incredibly
informational,
II
dense,
they
are
not
histograms,
they
are
not
pie,
charts,
they
are
not
line
charts.
Instead,
they
are
designed
to
tell
a
story
around
the
data.
A
You
need
to
tell
the
story
of
okay,
so
this
is
incredibly
powerful
because,
like
this
plot
is
unreadable
to
the
people
who
probably
haven't
worked
in
this
field
before,
but
if
I
can
suddenly
have
a
whole
bunch
of
stuff
in
the
same
source,
that
tells
you
how
to
read
this
plot,
what
conclusions
I
drew
from
it
and
why
you
can
actually
start
to
learn.
How
do
we
use
these
kinds
of
plots?
Why
are
they
important
for
the
function?
B
Pinterest
uses
I
was
actually
before
we
dive
into
how
we
use
it
because
I
know
our
audience
it.
Our
taints
is
largely
male
and
in
the
United
States,
our
users
are
primarily
female
they're,
not
internationally.
I
will
spend
a
little
bit
of
time
explaining
what
Pinterest
is
so.
Pinterest
is
a
discovery
engine
the
people
you
use
to
plan
decisions
about
their
lives.
That
might
mean
what
they're
gonna
eat
for
dinner.
B
We
have
more
us
users
than
Twitter
does,
and
we
have
200
million
monthly
active
users
around
the
world,
our
court,
our
data
corpus,
which
is
why
I
joined
it's
a
beautiful
data
set
for
those
of
you
who
love
data,
has
over
30
billion
pins
that
have
been
curated
by
humans
into
hundreds
of
millions
of
boards
containing
billions
of
connections,
and
that
interconnected
graph
unleashes
all
sorts
of
interesting
possibilities
from
an
algorithms
standpoint
or
from
extracting
meaning.
It's
a
really
really
cool
data
problem.
B
That
means
we
also
do
a
lot
of
data
science.
The
way
we
integrate,
jupiter'
notebooks
into
our
development
flow
comes
at
every
single,
every
single
point
of
the
product
development
process.
At
the
exploratory
phase,
we
use
Jupiter
notebooks
to
be
able
to
begin
to
validate
initial
ideas
or
maybe
even
just
understand,
an
anomaly.
They
might
lead
to
a
thesis.
We
use
them
to
socialize
and
build
consensus.
We
use
them
to
analyze
and
monitor
experiments
and
then
after
launch,
we
use
them
to
also
do
ongoing
monitoring
all
in
each
of
these
phases.
B
These
kinds
of
analyses
largely
fall
into
three
kinds
of
buckets,
or
these
use
cases.
The
first
one
is
investigatory
analysis.
That's
those
asking
the
initial
questions.
Extending
dashboards
is
about
being
able
to
not
to
commit
before
baking
and
beginning
to
socialize
a
metric
and
realizing
and
recurring
analysis
is
about.
How
do
you
free
up
capacity
that
used
to
be
spent
on
analysts?
Doing
the
same
thing
over
and
over
again?
B
To
give
you
some
examples
of
how
we
use
investigate
or
analysis,
and
maybe
you
have
a
question-
that's
a
simple
ligand
discovery
of
how
many
impressions
should
our
top
content
get.
I
did
a
long
analysis
that
was
able
to
incorporate
not
just
the
initial
analysis
and
identification
of
which
pins
were
on
top,
but
because
we
can
render
rich
media
into
notebooks
directly.
I
was
also
able
to
build
a
visualizer
that
allowed
us
to
actually
see
which
pins
got
the
most
impressions
or
clickers,
not
the
best.
B
B
How
many
of
you
have
ever
wanted
a
metric
really
badly
on
your
dashboard
and
had
to
write
some
proposal
or
have
your
team
write
some
proposal
to
a
centralized
group
who
maintained
the
dashboard
justifying
why
this
thing
that
you
needed
was
worth
their
time
like
I've
done
that
before
one
of
the
things
that
Jupiter
notebooks
have
allowed
us
to
do?
Is
we
can
have
whole
analysis
suites
that
are
that?
Are
you
know?
B
For
a
long
time
we
can
have
notebooks
that
are
very
easy
to
run
and
very
rigorous
that
we
can
use
to
distribute
best
practices
and
then
the
last
one-
and
this
is
the
thing
that
has
probably
changed
the
lives
of
ourselves,
our
sales
team,
the
most
or
has
allowed
us
to
be
much
much
better.
Communicators
cross-functionally
is
when
we
do
major
experimental
changes
on
the
organic
side
or
on
the
advertiser
side.
Small
changes
to
how
we
do
ranking
can
have
a
dramatic
impact
from
our
partners.
B
We've
developed
a
very
rigorous
analysis,
suite
that
allows
us
to
poke
at
lots
of
potential
consequences
whenever
we
launch
an
experiment
and
be
able
to
even
identify
which
advertisers
we're
going
to
have
to
reach
out
to
and
do
coms
with
before
a
change
ships.
These
kinds
of
reports
have
made
it
so
that
we
can
have
a
much
much
more
rigorous
launch
processing
and
it
means
that
we
no
longer
have
to
have
like
have
a
team
of
analysts
who
are
dedicated.
B
Just
to
doing
this
thing,
we've
built
out
a
centralized
report,
server
that
allows
us
to
take
Jupiter,
notebooks
and
templatized
them,
so
you
can
do
just
change
a
few
fields
and
generate
these
axiom
analysis
again
and
the
thing
that
separates
this
Jupiter
server
as
we
call
it
or
dupit
a
report
server.
Is
that,
unlike
your
dashboard,
you
can
push
a
button
and
see
the
guts
that
actually
generated
this
amazing
report.
You
can
always
keep
the
context
tied
to
the
final
results,
no
matter
what
and
this
actually
has
consequences
beyond
you
know,
transparency
or
code
review.
B
One
of
the
you
may
look
at
that
and
say
well,
if
you're
you're
going
in
building
centralized
reports
are
like
doing
dashboards
again.
Isn't
that
kind
of
going
against
this
idea
that
people
should
be
going
and
asking
questions
on
their
own
and
the
way
we've
kind
of
reconciled?
That
is
the
way
we
think
of
all
users
of
data
at
Pinterest.
Is
they
kind
of
fall
into
three
buckets?
There
are
people
who
are
consumers.
B
There
are
people
who
are
producers
and
there
are
people
who
are
both
when
I
talked
about
the
idea
that
you
could
push
a
button
and
peek
behind
the
curtain.
That's
an
illustration
of
a
philosophy
that
we're
very
passionate
about,
which
is
that
we
always
want
to
make
sure
there's
a
path
to
mastery
available
to
users,
we're
right
now,
building
out
a
new
generation
of
our
dashboarding
software.
So
we
do
still
have
dashboards
internally.
B
We
will
always
have
dashboards
internally,
but
even
for
those
nice
polished
official
dashboards,
the
kind
of
things
that
do
our
financial
reporting,
one
of
the
things
that
we've
insisted
on,
including
is
a
button
that
actually
allows
you
to
convert
this
nice
polished
dashboard
back
into
Jupiter
notebooks.
So
you
could
have
a
notebook
that
can
reproduce
an
analysis
if
you
wanted
to
dig
further
yourself.
A
Okay,
so
there
is
still
an
important
point,
which
is
even
if
we
have
this
tool,
we're
still
gonna
need
data
education
for
our
workforce.
It's
still
gonna
be
important
and
we've
already
told
you
why
we
think
this
is
hard,
but
I
think
one
thing
we
want
to
talk
about
too
is
that
we
have
actually
now
been
running
a
data
science
course
at
Pinterest
and
I
have
been
surprised
how
effective
it's
been.
So
what
are
we
doing
and
why
so?
We
talked
about
education
and
we
going
in.
A
We
had
taught
workshops,
we
knew
like
these
were
serious
problems
and
we
decided
to
actually
try
and
build
a
course
from
the
ground
up
to
do
to
address
these
problems.
So
what
we
did
is
we
went
on
and
talked
to
people
who
teach
computer
science.
We
went
on
and
talked
to
people
who
teach
physics.
We
went
on
and
talked
to
people
who
teach
machine
learning
both
in
academia
and
professionally,
and
we
just
talked
to
them
about
like
what
a
students
learn.
A
How
do
they
learn
it,
and
what
we
came
up
with
was
sort
of
a
template,
I
system
that
tries
to
get
around
these
hard
points.
So
the
first
thing
is
this
is
all
taught
with
Jupiter.
Notebooks
in
an
effect
is
not
really
much
work
for
us.
We
in
fact
have
people
work
through
these
in
a
peer-reviewed
setting
and.
A
And
we
don't,
we
don't
really
have
to
do
anything
in
them,
so
people
get
this
interactive,
collaborative
and
sort
of
scaffolded
approach
from
the
beginning.
People
also
get
a
self-paced
approach,
so
we
don't
have
to
be
present
for
them.
We
don't
have
to
be
up
here,
giving
a
slide
deck
to
every
single
person.
Who
wants
to
do
this
I
honestly
admit
part
of
what
people
get
out
of
this.
Is
they
get
six
note
books
written
by
us,
so
they
get
all
that
code
and
they
can
go
and
use
that
code.
A
They
can
see
how
we
did
things
they
can
reuse
it
right
if
all
they
do
is
copy
and
paste.
That's
fine
and
finally,
what's
really
exciting
now
is
we're
actually
entering
a
new
generation
of
the
course
where
it's
entirely
taught
by
TAS,
so
we
don't
even
have
to
show
up
anymore,
which
is
you
know
a
great
bonus
for
me.
So
this
is
how
we
split
the
course.
A
We
have
foundational
skills
in
the
first
three
sessions
and
then
we
have
practical
data
science
in
the
next
three
sessions,
and
we
run
this
course,
ideally
every
two
weeks
on
a
cycle.
So
if
people
need
more
time,
maybe
they
get
stuck
on
lesson
three.
They
can
just
wait
and
in
a
few
weeks
another
course
will
come
around
for
them
in
the
same
session
and
they
can
do
it
again
with
different
partners
to
give
you
a
sense
of
sort
of
the
scale
we're
running
this
app.
This
is
one
of
our
recent
classes.
A
You'll
notice,
that's
one
of
our
TAS
talking
to
someone
all
of
this
is
sort
of
student
driven,
it's
an
experiential
experience.
They
have
to
help
each
other,
they
have
to
figure
it
out.
We
give
them
little
prompts
we
give
them.
You
know
they
can
highlight
to
reveal
answers.
We're
not
gonna
ask
them
to
get
stuck,
but
we
only
talk
for
about
five
minutes.
It's
kind
of
nice
I
go
get
a
coffee
most
of
the
time.
A
So,
what's
really
important
and
what's
really
made
this
work
is
that
all
of
the
lessons
are
jupiter'
notebooks,
every
single
one,
that's
all
they
are,
and
what
we've
done
is
we've
sort
of
created.
You
know
a
scaffolding
and
it
starts
out
really
strong
in
the
beginning,
like
we
asked
them
to
do
very
basic
things
and
to
discuss
why
they
did
them
and
by
the
end,
people
really
gain
skills.
A
So,
in
our
first
class
we
had
several
people
in
our
company
who
didn't
know
how
to
read
a
histogram
they'd,
always
felt
uncomfortable
asking
that
question
and
in
the
course,
for
the
first
time
they
were
able
to
ask
right.
We
had
a
prompt
that
was
like
what
is
this
plot
me
talk
to
your
partner
and
people
reveal
this
fact
and
by
the
third
class
people
could
read.
Empirical
cumulative
distribution
functions
with
error.
A
Envelopes
have
intelligent
conversation
about
their
differences
and
actually
use
them
in
their
real
jobs,
and
when
we
post
these
publicly,
we
always
use
github
to
do
it,
and
the
reason
we
do
is
it
renders
very
nicely,
and
it
is
important
when
you
especially
when
working
with
more
junior
people,
but
they
don't
come
in
and
stare
at
a
terminal.
They
don't
stare
at
you
know
sublime
they
don't
stare
at
an
IDE.
What
they
see
is
something
nice
that
looks
kind
of
like
a
story.
It
encourages
them
to
engage
and
by
rendering
Jupiter
notebooks.
B
So
one
of
the
major
thing,
one
of
the
reasons
why
we
host
our
curriculum
on
github
and
we
are
happy
to
work
with
other
people
who
want
to
work
on
data
education
as
well
as
we
really
believe
in
giving
back
to
the
larger
data
science.
Community
Pinterest
is
also
big,
is
a
big
believer
in
open
source
and
giving
back
to
the
community
at
large.
So
today
we
want
to
talk
a
little
bit
about
what
are
some
of
our
future
plans
for
how
we'll
be
sharing
our
data
science
resources
with
the
recipe
with
everyone
else.
B
One
of
the
things
that's
very
fundamental
to
Pinterest
philosophy
as
Pinterest
enables
people
to
take
action
to
do
and
the
way
we
do
that
is
by
providing
tools
that
are
extensible
our
effect.
One
of
our
founders
is
very
passionate
about
Legos
and
the
metaphor
of
Legos.
The
idea
that
you
have
these
very
small
building
blocks
that
can
be
made
into
incredibly
different
things.
We
want
to
make
sure
that
the
things
that
we
release
help
people
to
do
more
things
with
the
data
in
their
in
their
daily
jobs.
B
Later
this
year,
Patrick
and
I
will
be
open
sourcing.
Our
current
curriculum,
so
we've
in
the
past
had
done
one-off
workshops.
Ad
conferences
will
be
open
sourcing,
a
publicly
viewable
version
of
our
current
curriculum,
which
is
largely
based
on
Pinterest
data.
So
we
need
to
make
sure
we
have
a
data
source
available
for
you,
which
is
why
it's
not
available
today
and
later
the
spring
will
be
open
sourcing.
B
In
closing,
though,
if
I
get
my
clicker
to
work,
okay,
there
we
go
given
the
way
to
talk
to
you
for
about.
You
know:
30
35,
minutes
about
the
wonder
of
Jupiter
notebooks
I
have
the
thing
we
leave
you
guys
with
is
a
wish
list
item,
given
that
we
know
there
are
lots
of
programmers
in
the
audience.
One
of
the
thing
you
know
Jupiter
know:
Jupiter
notebooks
is
an
open
source
tool.