►
Description
NESRC Data Seminar Series: https://github.com/NERSC/data-seminars
Simulation-based and label-free deep learning for science
Abstract:
Precise scientific analysis in many areas of science is possible because of complex simulations that connect fundamental theories to observable quantities. These simulations have been paired with multivariate methods for many years in search of new fundamental and emergent structure in nature. Deep learning tools hold great promise to qualitatively change this paradigm by allowing for holistic analysis of data in its natural hyperdimensionality with thousands or millions of features instead of up to tens of features. These tools are not yet broadly used for all areas of data analysis because of the traditional dependence on simulations. In this talk, I will discuss how we can change this paradigm in order to exploit the new features of deep learning. In particular, I will show how neural networks can be used to (1) overcome the challenge of high-dimensional probability density modeling and (2) learn directly from (unlabeled) data to perform hypothesis tests that go beyond any existing analysis methods. The example for (1) will be full phase space unfolding (deconvolution) and the example for (2) will be anomaly detection. The talk will include a discussion of uncertainties associated with deep learning-based analyses. These ideas are starting to become a reality: the first deep learning weakly supervised anomaly detection search has recently been made public by the ATLAS Collaboration at the LHC. While my examples will primarily draw from collider physics, the techniques are more broadly applicable and I am happy to discuss extensions and applications to your science domain.
A
I'll
briefly
introduce
for
the
past
couple
of
minutes,
while
people
are
still
filtering
in
so
probably
many
of
you
know:
ben
nachman,
who's
in
physics,
division
here
well
here
at
berkeley
lab
for
those
of
you
who
don't
know
ben
he's
on
the
atlas
collaboration,
but
as
well
as
being
put
in
the
atlas,
vhc
collaboration.
That
is
a
particle
physicist,
but
as
well
as
making
considerable
contributions
to
particle
physics.
He's
also
made
considerable
contribution
to
deep
learning.
B
A
B
B
So
I
should
just
start
with
a
little
bit
of
science.
Here
we
go
so
the
goal
of
fundamental
physics
is
is
broad
basically
in
a
nutshell,
to
identify
new
length,
scale
structures
and
and
how
other
scales
emerge
and
their
variety
of
puzzles,
and
it's
even
it's
important
to
say
also
that
the
lhc
is
for
which
will
be
my
my
background
today.
B
Collider
physics
is
not
the
only
fundamental
physics
experiment
there,
of
course,
a
broad
range
of
them,
but
it
certainly
currently
hosts
the
the
biggest
data
experiments
in
the
field
and
the
prototypical
example
I'm
going
to
have
for
the
rest
of
this
talk
is
going
to
be.
Are
we
going
to
be
collision?
B
So
I
thought
I'd
start
by
going
over
how
we
do
data
analysis
and
energy
physics,
and
I
think
this
is
very
similar
in
other
fields
as
well,
but
in
sort
of
the
simulation
based
inference,
mindset,
there's
sort
of
two
parallel
tracks.
On
the
right
hand,
side
we
have
nature,
which
of
course,
is
ultimately
what
we'd
like
to
learn
something
about,
and
we
build
experiments
to
observe
certain
features
of
nature.
Those
experiments
produce
data.
We
then
do
some
dimensionality
reduction
and
we
have
a
sort
of
parallel
pipeline.
B
We
have
our
theory
of
everything
and
from
the
theory,
which
has
some
small
number
of
parameters.
We
can
simulate
everything,
basically,
so
all
the
way
from
the
the
subnuclear
length
scales,
the
particle
physics
to
the
nuclear
physics
to
the
detector
physics,
all
the
way
to
make
the
same
sort
of
high
fidelity,
complex
observables
that
we
can
see
in
our
detector,
we're
in
the
same
dimensionality
reduction
and
then
the
comparison
allows
us
to
infer
something
about
the
theory
of
everything
from
comparing
the
left
and
right
now.
Deep
learning
is
playing
a
huge
impact.
B
It's
starting
to
have
a
huge
impact
on
all
aspects
of
this
chain
from
the
very,
very
fast
processes,
so
online
processing
data
can
data
control,
quality
control
as
well
as
various
aspects
of
data
curation.
So
that
means
like
noise
mitigation
calibration
this
sort
of
thing,
where
it's
also
playing
a
role
on
the
left-hand
side.
So
in
the
these
up
and
down
arrows
so
going
from
the
output
of
physics
simulator
to
period,
everything
is
sort
of
parameter
estimation
and
going
forward.
B
Clearly
I
don't
have
time
to
talk
about
all
of
this,
so
today
I'm
going
to
focus
on
just
a
few
aspects
and
in
particular
I'll
start
with
simulation
and
deep
learning
for
inference
in
the
full
face
base
and
I'll
tell
you
what
I
mean
by
that.
This
is
an
exploration,
that's
empowered
by
deep
learning,
combined
with
slow,
stimulators
I'll,
then
pivot
a
little
bit
by
talking
about
uncertainties.
Briefly,
before
I
get
into
the
label-free
learning
and
the
example,
there
will
be
anomaly
detection.
B
So
the
context
is
that
in
we
want
to
do
some
inference,
so
we
have
some
theory
and
I
have
some
parameters.
We
have
some
some
face.
Space
x,
we'd
like
to
observe
so
there's
some
feature
space
x.
You
want
to
in
first
thing
out
the
parameters
given
the
feature
space
so
in
low
dimensions.
You
can
try
to
explicitly
model
p
of
x,
given
parameters
and
then
do
something
like
maximum
likelihood,
but
in
high
dimensions,
p
is
intractable.
B
You
know
if
x,
is
a
thousand
dimensions
or
even
worse
suppose
it's
a
variable
number
of
dimensions
that
can
be.
You
know,
order
a
thousand.
This
can
be
very
complicated,
and
so
you
need
likelihood
free
approaches,
approaches
where
we
can't
write
down
p
exactly
okay.
So
I
call
this
the
hyper
challenge
and
I
call
I
call
this
for
two
reasons.
So
one
the
feature
space
or
the
phase
space
that
we're
trying
to
learn
something
about
that
is
like
the
collision.
The
collision
debris
is
very
high
dimensional.
B
So
you
know
a
typical
collision
at
the
lhc
might
have
a
thousand
particles.
Each
of
those
particles
has,
you
know,
momenta
momentum
and
then
some
other
numbers
like
its
electric
charge.
So
I
can
easily
spend
a
variable
dimensional
huge
phase,
huge
face
space
and
those
particles
they
hit
our
detector
and
our
detector
is
basically
a
hyperspectral
camera,
so
it
has
many
many
layers.
B
You
know
millions
of
readout
channels,
and
so
those
data
are
then
represented
in
our
feature
space
that
we
actually
get
to
observe
by
some
other
high
dimensional
datum,
and
so
this
combination
of
hyper
varied
phase,
phase
and
hyperspectral
data
is
a
huge
challenge
and
the
bottom
right
schematic
diagram
is
just
to
illustrate
not
the
scale.
You
know
the
particles
hitting
our
detector
and
then
all
the
different
blobs
correspond
to
the
measurement
channels.
B
Okay,
so
unfolding
is
a
is
a
particular
challenge
where
we
want
to
not
in
for
some
particular
parameters,
but
we
want
to
learn
something
about
the
full
distribution
of
the
feature
space,
pre-detector,
and
so
this
in
some
fields
is
called
deconvolution
and
100
deficits.
We
call
it
unfolding,
so
you
want
to
learn
something
about
the
left-hand
side.
That
is
all
the
the
particles
things
like
the
green
blobs.
B
Now
we
could
do
is
if
we
knew
p
of
measured
given
true,
we
could
just
say:
okay,
I'm
going
to
declare
the
unfolded
to
be
the
maximum
likelihood
estimator,
and
this
p
of
measure
given
true
would
be
something
like
the
point
spread
function
in
height
dimensions,
where
we
call
bone
energy
physics,
the
response
matrix,
but
the
challenges
that
I
just
told
you
is
that
measured
is
hyperspectral.
True
is
hypervaries
high
dimensional
feature
space,
so
p
of
measure
given
true
is
intractable.
We
can't
write
it
down.
B
However.
What
comes
to
our?
What
comes
to
save
us
is
that
we
have
really
excellent
high
fidelity
simulators,
so
we
can
sample
from
p
of
measure
given
true,
even
though
we
can't
write
down
p
measure
given
true
exactly
and
even
though,
and
in
fact
we
can't
even
take
derivatives
of
p
measure
given
true
in
general.
B
So
imagine
I
have
two
data
sets
one
sampled
from
p
of
x
and
one
sampled
from
q
of
x,
I'd
like
to
create
weights
q
over
p,
so
that
when
I
have
a
weighted
data
set
one,
it
is
statistically
identical
to
data
set
two
meaning.
If
I
compute
any
statistic
on
the
weighted
data
set,
it
will
unlea
data
set
one.
It's
the
same
as
if
I
compute
that
statistic
on
the
on
with
the
data
set
too.
B
Okay,
that's
great.
However,
I
just
told
you
that
we
can't
easily
compute
p
and
q,
we
don't
know
the
densities,
and
so
here
the
solution
will
come
from
deep
learning
and
it's
based
on
the
the
simple
fact
that
neural
networks
basically
learn
to
approximate
the
likelihood
ratio
q
over
p,
and
so
the
solution
is
to
train
a
neural
network
simply
distinguish
the
two
data
sets
and
then
manipulate
the
output
in
a
clever
way
so
that
we
can
directly
learn
the
likelihood
ratio
without
ever
having
to
know
p
or
q
separately.
B
B
This
is
particularly
useful
for
particle
physics,
where
a
collision
may
produce
a
variable
number
of
particles
which
have
a
lot
of
symmetry
and
so
here's
a
particular
collision
event.
That's
been
simulated
where
you
have
electron
positrons
electron
and
a
positron
collide
and
outgo
some
collision
debris.
That's
all
these
lines
and
blobs
and
there
are.
B
There
have
been
major
advances
in
generative
models
recently
to
include
various
effects
like
these,
but
classification
is
well
ahead
of
generation
and
in
particular,
there
are
architectures
that
can
accommodate
this
kind
of
data,
and
so
the
the
results
I'm
going
to
show
are
going
to
be
based
on
the
deep
sets
architecture.
It
doesn't
matter
as
much
what
it
is,
but
it's
an
architecture
that's
set
based
that
allows
for
variable
length,
permutation
variant
operations,
okay,
so
I'm
going
to
show
an
example
based
on
those
simulated
electron
collisions.
B
So
the
way
it
works
is
there's
a
full
face.
Space
relating
happening
behind
the
scenes
so
full
face.
Space
here
means
all
the
particles
in
their
high
dimensional
glory,
and
I
can't
and
that
produces
a
new
data
set
better,
that's
weighted
and,
of
course,
it's
hard
to
visualize
that
it
was
effective.
So
we
can
compute
one-dimensional
statistics
and
then
make
a
histogram.
B
So
here
this
plot
this
this
slide
shows
two
histograms.
After
applying
this
re-wetting
technique,
and
so
we're
going
from
the
blue
to
the
black
and
the
weighted
blue
is
the
orange.
It
doesn't
matter
so
much
what
these
observables
are,
but
they're
sort
of
representative
of
some
approving
the
feature
space
in
some
complicated
way,
and
you
can
see
that
the
orange
and
the
black
dash
are
right
on
top
of
each
other.
So
we've
done
one
high
dimensional
rewaiting.
Then
we
can
compute
some
statistics
afterwards.
B
Okay,
but
our
goal
was
not
necessarily
just
to
re-weight
one
sample
into
another
sample.
I
wanted
to
do
unfolding,
so
I'm
now
going
to
introduce
a
new
technique
called
omnifold.
So
on
the
fold
is
a
deconvolution
approach
that
basically
proceeds
by
iteratively,
applying
this
rewriting
technique,
and
so
the
way
it
works
is
we
want
to.
We
want
to
learn
something
about
the
top
right
box.
B
We
have
both
the
simulated
part
and
the
generated
part
that
is,
the
pre
and
post
detector
simulation
okay,
so
the
procedure
works
iteratively.
We
start
by
taking
the
detector
simulated
events
and
we
re-weight
them
to
be
the
same
as
the
measured
data.
So
this
is
one
re-weighting
that's
high
dimensional
and
after
you
do
this,
the
weighted
simulation
should
be
basically
the
same
as
the
measured
data.
B
B
Basically,
the
the
pre-detector
part
inherits
these
weights,
so
these
weights
are
a
function
of
the
simulated
data
and
so
at
particle
level.
These
weights
are
not
a
function,
they're,
not
a
true
function,
because
you
could
have
two
different
points
that
have
the
same
features
but
different
weights,
and
so
in
order
to
get
around
this
to
make
a
true
weighting
function,
we
learn
a
second
rewaiting
step
where
we
we,
where
we
re-weight
the
unweighted
particle
level
spectrum
to
the
spectrum
that
has
the
pullback
weights.
B
Okay.
So
let
me
show
you
how
this
works
with
an
example,
so
here's
a
simulation
of
an
lhc
collision
and
to
illustrate
that
this
is
just
one
observable.
The
zero
is
called
subject,
but
doesn't
matter
what
it
is.
This
particular
observable
characterizes
how
two
clustered
one
cluster
like
an
event
is
collision
event
it's,
but
once
it
once
again
doesn't
really
matter
what
the
observable
is.
This
is
the
truth.
This
is
that
top
right
box
that
I
mentioned.
We
want
to
learn
something
about
this,
but
all
we
get
to
measure
is
the
black.
B
So
the
the
one
standard
approach
that
people
use
right
now
is
called
iterative,
bayesian
unfolding
or
ibu
for
short,
so
ibu
is
a
binned,
one-dimensional
approach.
That
is
also
iterative,
and
the
result
is
shown
here
in
gray,
and
you
can
look
at
the
ratio
plot.
You
want
it
to
be
at
one.
That
means
it's
unfolded
correctly
and
you
see
it's
pretty
close.
It's
within
15
across
the
whole
spectrum,
and
we
can
compare
this
with
omnifold.
B
So
let's
do
one
omnifold
and
that's
the
red,
and
you
see
that
the
red
is
also
very
close
to
the
green
and
the
ratio
is
much
closer
to
unity,
so
that
might
be
expected
because
the
red
has
access
to
more
information
than
the
gray
one
does,
because
the
gray
one
only
has
access
to
this
one
dimension
and
it's
binned,
whereas
the
red
is
unbinned,
but
also
has
access
to
the
full
face
space.
So
anything
that
would
determine
that
would
modify.
B
The
resolution
from
the
other
features
would
improve
the
resolution
of
this
observable
and
in
fact,
once
we've
done
one
on
the
fold,
we
can
compute
any
observable.
So
here's
a
paw
that
shows
six
observables.
It
really
doesn't
matter
what
the
insertables
are.
We
can
compute
anything,
but
the
the
main
point
is:
if
you
look
at
the
ratio,
plots
the
the
bottom
panel
of
each
of
the
six
blocks.
B
So
just
to
review
on
the
fold
is
unbend.
It's
also
maximum
likelihood.
I
didn't
prove
this,
but
actually
you
can
show
that
omnifold
asymptotically
achieves
the
maximum
fit
estimate
in
the
full
feature
space.
It's
full
face
space,
so
we
can
compute
intervals
after
the
fact.
So
traditionally
you
have
to
pick
or
observables
then
unfold
now
you
unfold
and
then
compute
observables
and
the
resolution
improves
from
all
the
other
features
that
are
going
to
the
unfolding.
So
that
is
to
say
that
the
ratio
plots
roll
closer
to
unity
all
right.
B
So
this
is
fantastic
and
I
think
pretty
has
the
potential
to
be
transformative.
However,
it's
still
early
days,
so
we
haven't,
we
have
we
don't
have
any
public
results
yet.
So
I
want
to
get
to
sort
of
to
pivot.
Now,
and
and
I'll
show
you
some
by
the
end,
some
some
actual
results
with
collision
data.
B
So
this
is
representative
for
many
analyses
at
the
large
hadron,
collider
and
elsewhere.
I
think
will
nicely
illustrate
the
kinds
of
uncertainties
that
come
into
an
analysis
that
use
the
simulation
okay.
So
how
does
the
search
work
so?
The
way
it
works
is
I
have
some
I
posit
some
new
physics
models,
some
new
particles
and
their
interactions.
B
I
simulate
those
particles.
I
simulate
the
known
physics.
That
is
the
background,
and
then
you
train
a
classifier
in
simulation
to
distinguish
the
signal
and
background
that
gives
me
a
function
and
using
that
function
we
can
define
two
regions,
so
one
region
which
is
at
high
values
of
the
classifier
output,
so
very
signal
like
that,
corresponds
to
what's
called
the
signal
region
and
then
the
low,
the
very
low
values
of
the
classifier
output.
So
it's
very
not
signal,
like
is
called
the
control
region.
B
So
this
plot
on
the
right
is
an
example
where
you
have
the
neural
network
trained
to
distinguish
two
signals.
Our
signal
from
background
and
the
x-axis
is
the
neural
network
output
and
to
the
right
corresponds
to
very
signal-like
and
to
the
left
corresponds
to
very
background-like,
and
basically
you
can
use
the
control
region
where
there's
not
that
much
signal
to
check
how.
Well
you
model
the
background,
and
then
you
can
compare
the
data
and
the
simulation
in
the
signal
region
and,
if
they're
very
different,
that's
great.
B
Section
of
you
know
whatever
new
particle
you
speculated
about,
okay,
so
now
I
want
to
think
about
how
uncertainties
play
a
role
in
in
such
a
setup,
and
I
like
to
classify
uncertainties
in
sort
of
two
two
ways,
so
uncertainties
on
the
precision
or
optimality
of
the
analysis
and
I'll
what
I
mean
by
that
in
a
second.
So
these
are
the
kind
of
uncertainties
that,
if
you,
if
you
have
if
these
are
large,
then
you
are
not
using
your
data
to
its
full
exit,
fullest
extent,
you're
wasting
time
and
money.
B
But
your
answer
is
not
wrong,
and
so
what
do
I
mean
by
optimal
by
optimal?
I
mean
that,
for
instance,
if
you're
trying
a
neural
network,
then
your
test
statistic
should
be
the
likelihood
ratio,
and
so
in
the
you
know,
the
binary
classification
case
that
I
mentioned
in
the
previous
slide.
If
you're
trying
to
distinguish
say
the
background
from
from
the
signal
plus
background,
then
the
there's
just
no
unknown
answer,
which
is
a
likelihood
ratio.
So
if
your
neural
network
is
not
that
or
monotonically
related
to
that,
then
your
procedure
will
be
suboptimal.
B
Okay,
in
contrast,
the
uncertainty
on
the
accuracy
or
bias
is
related
to
tail
probabilities,
so
fix
a
neural
network,
and
I
want
to
know
what's
the
probability
that
the
neural
network
value
is
above
some
threshold
and
that's
how
I'm
going
to
compute
p
values
and
if
that's
not
correct.
So
if
the
predicted
null
network
tail
probability
is
not
the
same
as
the
true
one,
then
my
result
will
be
incorrect
and
you
can
see
that
even
if
your
null
network
is
optimal,
you
could
still
get
the
wrong
p
values
and
vice
versa.
B
Okay,
so
it's
obviously
a
bit
more
complicated
because
in
addition
to
the
top
and
bottom,
so
the
precision,
optimality
and
accuracy
or
bias
you
also
have.
You
can
also
classify
uncertainties
as
being
statistical
in
nature
or
systematic
in
nature.
Now
other
words
are
used
in
the
machine
learning
literature.
Sometimes
you
might
hear
like
iliatoric
and
epistemic
uncertainty
and
they
roughly
map
onto
left
and
right
in
this
categorization.
B
But
the
categorization
shown
here,
I
think,
is-
is
very
helpful
and
also
representative
of
the
way
that
we
sort
of
approach
simulation
based,
imprints,
okay.
So
now
I
want
to
spend
a
little
a
little
time
going
through
these
boxes
and
saying
how
you
would
diagnose
each
of
these
uncertainties,
how
you
would
estimate
them
so
for
the
top
left
box.
B
Okay,
of
course,
in
general,
systematic
uncertainties
are
always
harder
than
statistical
ones
and
there's
one
key
component
due
to
the
fact
that
if
your
simulation
is
inaccurate,
then
the
neural
network
that
you
produce
could
be
suboptimal.
But
let's
come
back
to
that
in
a
second.
There
are
other
aspects
here,
so
I
mentioned
that
if
your
neural
network
is
not
the
likelihood
ratio,
then
it
will
be
so
optimal,
and
so
basically
that
can
happen
for
a
variety
of
reasons.
B
So,
for
instance,
if
your
network
is
not
doesn't
have
a
very
flexible
enough
architecture,
so
that
means
like
number
of
layers
nodes
per
layer.
That
kind
of
thing,
if
it
wasn't
initialized
well
or
the
training
procedure,
wasn't
flexible
enough,
and
so
these
are
things
where
we
kind
of
know
all
of
the
contributions.
B
But
it's
hard
to
quantify
exactly
what
your
uncertainty
is.
So
to
put
a
you
know,
a
density
on
some
nuisance
parameters
for
this.
For
these
uncertainties,
okay,
the
bottom
left
is,
this
is
less
painful.
This
is
the
physical
uncertainty
on
the
accuracy.
This
is
less
painful
than
the
top
in
the
top
left,
and
that's
because
here
the
neural
network
is
fixed,
so
you
don't
retrain,
but
you
still
have
to
you.
Might
you
can
still?
B
You
know,
estimate
with
the
bootstrapping,
and
I
would
say
the
most
important
and
most
difficult
uncertainty
is
the
bottom
right
one.
So
this
is
the
uncertainty,
the
systematic
uncertainty
on
the
bias-
and
this
is
tricky
because
it
basically
is
how
well
do
you
model
the
density
of
the
feature
space,
and
this
is
hard
and
challenging,
because
if,
if
the
feature
space
is
high
dimensional,
then
we
need
to
very
accurately
model
the
density
in
high
dimensions.
B
So
in
some
cases
the
uncertainties
will
factorize.
So
in
the
particle
physics
case,
imagine
you
have
a
final
state
with
like
a
number
of
particles,
and
you
have
an
uncertainty
matrix
per
particle.
Then
maybe
you
can
build
the
whole
uncertainty
matrix
by
having
decomposed
into
particles,
but
in
many
cases
we
don't
know
the
full
uncertainty
model,
and
so
that
means
we
don't
know
all
the
nuisance
parameters.
B
B
B
B
B
So
how
does
that?
How
could
that
work?
So
imagine
you
have
a
collision
event
j,
so
j
could
be.
You
know.
Some
high
dimensional
variable
length
feature
space
and
we
train
a
classifier
f
to
distinguish,
say
signal
versus
background
based
on
j,
and
I
want
to
learn
a
new
neural
network
g,
which
is
a
perturber
that
goes
from
j
to
j
plus
delta
j,
where
delta
j
is
some
high.
Dimensional
perturbation
I'd
like
to
perturb
this
node
network,
such
that
the
impact
on
the
classification
is
as
big
as
possible.
B
So
imagine
I
have
a
bunch
of
observables
o
of
j
and
I
can
validate
the
distribution
of
those
observables
in
the
control
region,
so
we
can
require
that
the
perturbation,
basically,
these
invariant,
these
observables
and
then
we
can
see
what's
like
the
biggest
impact
on
the
neural
network,
given
that
we
restrict
the
size
of
the
perturbation
in
a
region
that
we
are
allowed
to
validate
now
just
flash
one
plot
just
to
just
illustrate.
So
this
plot
is
for
some
classification
task
doesn't
matter
what
it
is
signal
versus
background.
B
The
x-axis
is
a
cut
on
the
classifier
threshold
classifier
and
the
y-axis
is
the
relative
discovery
significance.
So
if
it's
at
one
that
means
you
learn
nothing
from
the
classifier,
doesn't
add
anything
and
if
the,
if
it's
at
two,
that
means
that
if
you
had
a
you
know
an
excess
of
two
before
now.
Your
access
is
four:
it's
a
multiplier
on
the
on
the
number
of
sigmas
of
excess
that
you
have
so
just
focus
on
the
orange
for
a
second,
so
you
can
see
the
orange
solid
line.
B
This
is
a
classifier
trained
on
low
level
inputs.
So
that
means
like
all
the
particles
and
their
momenta,
and
you
see
the
peak
discovery.
Significant
is
a
2.5
ish,
so
that's
that's
great,
it's
very
effective
and
the
orange
dotted
line
which
is
hard
to
see
because
underneath
it
is
what
happens
if
you
apply
a
random
perturbation,
so
the
random
perturbation
has
almost
no
effect,
but
if
you
apply
this
evil
perturbation,
it
goes
from
the
red
cell
to
the
red
dashed,
and
so
you
see
it
drops
by
almost
0.5
units
of
relative
discovery
significance.
B
So
it's
a
really
big
effect,
and
so
this
basically
gives
you
like
what
is
the
worst
case.
Uncertainty
in
terms
of
you
know
an
imperceptible
change
that
that
would
have
a
big
impact
on
the
result.
So
this
doesn't
tell
you
how
to
make
the
uncertainty
smaller,
but
at
least
it
tells
you
how
you
could
diagnose
the
sensitivity
okay.
B
So
now
let
me
go
back
to
this
table
and
and
talk
for
a
minute
about
how
to
reduce
the
uncertainties.
So
in
the
in
the
statistical
case,
it's
pretty
clear.
The
way
to
reduce
the
injury
is
to
train
with
more
events.
Now
this
is
not
always
possible,
but
deep
learning
may
even
have
a
way
to
help
with
that
by
accelerating
or
augmenting
simulations-
and
this
is
related,
for
instance,
to
the
possibilities
of
using
generative
models
like
gans.
I'd
love
to
talk
about
that,
but
that's
for
a
different
time
for
the
systematic
uncertainties.
B
So
there's
the
ones
about
the
modeling
of
the
feature
space
I'll
come
back
to
that
in
a
second,
but
it
might
be
possible
to
reduce
uncertainties
or
at
least
alleviate
your
analysis.
Complexity
by
making
your
procedure,
for
instance,
your
neural
network,
independent
of
no
nuisance
parameters,
and
so
you
could,
for
instance,
use
borrow
ideas
from
fairness
to
make
your
independent
make
your
networks
independent
of
of
nuisance
parameters.
You
don't
know
very
well.
B
But
the
most
challenging
is
the
bottom
right,
and
the
bottom
right
is
yeah
the
case
where
you
have
these
high
dimensional
bias
uncertainties.
It's
the
modeling
of
the
density
in
the
high
dimensional
feature
space
and,
in
my
opinion
this
is
really
the
biggest
challenge
for
deploying
neural
networks
in
practice
and
solving
it
will
require
a
lot
of
work.
B
Okay.
So
one
of
the
challenges
in
particle
physics
and
other
areas
of
physics
and
beyond
is
that
it's
very
hard.
You
can't
acquire
labeled
data
from
the
collisions
themselves,
and
so
there
are
two
reasons
for
that:
one.
We
have
an
enormous
amount
of
data,
so
no
one
wants
to
sift
through
all
the
images
to
look
at
the
collision,
events
to
look
at
them,
but
also,
in
many
cases,
just
not
possible
to
know.
So.
B
So
at
best
we
have
mixed
samples,
so
we
might
have
some
set
of
collision
events
that
are
superposition
of
green
and
red,
and
the
signal
and
background
and
we'd
like
to
train
a
classifier
to
distinguish
red
and
green.
But
all
we
get
to
observe
are
all
the
properties
of
these
collisions
represented
by
balls,
except
for
the
color
and
the
label.
B
And
so
we
have
these
two
mixed
samples
with
a
different
composition
of
of
signal
and
background,
and
the
question
is:
can
we
use
these
mixed
samples
to
train
a
classifier
and
it
turns
out?
The
answer
is:
yes,
there's
a
wide
variety
of
techniques
from
a
set
of
tools
called
weak
supervision,
so
they're
sort
of
less
than
supervised
learning,
but
not
fully
unsupervised
and
one
one.
Such
technique
is
called
classification
without
labels,
and
we
call
this
koala.
B
The
idea
is
pretty
simple:
you
simply
label
everything
and
make
sample
one
as
a
zero
and
everything
in
mixed
sample.
2
is
a
1,
and
then
you
use
whatever
your
favorite,
fully
supervised.
Classifier
is
distinguish
the
zeros
from
the
ones,
and
actually
you
can
show
that
asymptotically,
the
classifier
used
to
distinguish
mixed
sample
1
from
example.
2
is
actually
optimal
for
distinguishing
signal
from
background
the
red
from
green,
even
though
I'm
not
telling
the
classifier
anything
about
the
signal
versus
background
labels.
B
So
this
is
pretty
amazing
you
can,
you
can
achieve.
You
know
a
really
good
classifier
without
having
the
labels
themselves,
but
now
I
want
to
take
this
even
one
more
step.
What,
if
I
don't
even
know
if
there
is
a
second
class,
and
so
that's
sort
of
the
context
of
anomaly
detection,
and
so
I
want
to
now
discuss
how
we
can
apply
koala
for
anomaly.
Detection-
and
so
the
key
thing
we
have
to
do
is
basically
form
the
mixed
samples
that
may
or
may
not
have
different
classes.
B
So
imagine
that
there's
some
high-dimensional
feature
space
and
in
one
dimension
I
know,
let's
call
it
emrez.
I
know
the
background
has
a
smooth
density
and
the
signal,
if
it
exists,
would
be
localized
somewhere.
This
will
be
the
main
assumption
of
the
method,
and
it
turns
out
that,
for
a
variety
of
very
fundamental
physics
reasons,
we
know
that
signals
would
be
localized
somewhere
and
we
often
know
the
dimension
with
which
they
would
be
localized.
B
So
we
don't
have
to
know
where
it
is
ahead
of
time
and
then
we
can
use
all
the
other
features
that
exist
in
the
feature
space
to
train
a
quality
classifier
to
distinguish
the
potential
signal
from
the
background,
and
this
has
to
be
done
in
a
smart
way
to
not
have
a
very
high
false
positive
rate.
But
there's
there
are
ways
of
doing
that
with
k,
folding.
B
Okay,
so
I
want
to
show
how
this
works
in
practice
and
the
example
I'm
going
to
use
still
all
in
simulation.
So
far,
is
this
same
event.
I
started
off
the
talk
with
so
this
is
a
proton
proton
collision
seen
by
the
atlas
experiment
at
the
large
hadron
collider,
and
these
two
collimated
sprays
of
particles
that
go
up
to
the
left
and
down
to
the
right
are
called
jets.
And
so
the
features
will
be
the
radiation
pattern
inside
each
jet
and
the
resonant
feature.
B
Okay.
So
let
me
now
show
you
how
this
works.
The
here's,
a
histogram,
the
x-axis,
is
the
invariant
mass
of
jets
simulated
with
some
simulated
collisions.
The
y
axis
is
the
number
of
events
per
bin,
and
the
blue
here
corresponds
to
the
number
of
simulated
events
in
each
of
these
bins
of
the
invariant
mass.
B
What
we're
going
to
do
is
we're
going
to
draw
some
vertical
lines
to
make
a
signal,
region
and
a
side
band
region,
and
then
we're
going
to
make
a
parametric
fit
we're
going
to
make
a
parametric
fit
and
excluding
the
signal
region.
Then
we
can
interpolate
that
fit
into
the
signal
region
and
compare
our
predicted
background
to
the
blue
points
and
see,
if
there's
a
difference
that
will
allow
us
to
compute
p
values
and,
if
they're
different.
B
B
So
we
can
scan
from
left
to
right
and
as
we
do
so,
you
can
see
that
the
p-values
they
fluctuate
around,
but
they're
all
consistent
with
boring.
So
if
there's
nothing,
you
expect
the
p-values
to
be
uniform
between
zero
and
one,
and
so
you
get
some
fluctuations
that
are
mostly
close
to
one
closely
one
sigma,
which
is
you
know,
p-values
that
are
that
are
on
order,
one
okay,
so
this
is
great.
This
is
this
is
necessary
that
if
there's
no
signal,
we
find
nothing,
but
it's
not
sufficient.
B
Obviously,
we'd
like
to
have
some
signal
sensitivity.
So
now,
let's
do
the
same
procedure.
What
if
I
inject
the
signal
into
the
data
stream,
and
so
in
this
case
I've
injected
a
signal?
It's
the
one
on
the
right.
It
doesn't
matter
what
it
is,
but
it's
some
new
particle
that
decays
into
some
jets
with
some
well
well-defined
structure
and
that
structure
is
different
than
the
background
structure
and
the
on
the
left
shows
the
signal.
It's
this
yellow
histogram
lurking
underneath
the
background,
so
you
can
compare
the
100
line
to
the
yellow
histogram.
B
You
see
that
the
background
is
many
orders
of
magnitude
above
the
signal,
and
so
you
wouldn't
find
this.
If
you
just
did
the
normal
bump
hunt
searching
for
a
bump
on
top
of
that,
the
smooth
background,
so
let's
apply
the
koala
procedure.
Let's
start
with
a
single
region
on
the
left
and
we're
gonna,
we're
gonna
scan
it
from
left
to
right
and
we're
going
to
keep
track
of
the
p
value
just
like
before
and
as
we
scan
the
p
values
start
the
same
way.
They
sort
of
start
boring.
B
You
know
consistent
with
uniform
and
then
once
the
signal
region
is
on
top
of
the
peak
of
the
signal.
If
you
just
look
at
the
blue
points
by
eye
on
the
left
plot,
you
see
that
you
can
just
see
the
peak
by
eye.
So
the
neural
network
is
very
efficient
on
the
signal
and
very
inefficient
on
the
background
and
the
p-value
drops
very
low
and
in
fact
we
injected
something
like
a
1.5.
B
Sigma
access
and
the
neural
network
enhances
that
to
something
like
a
seven
sigma
excess
and
as
we
continue
to
scan
then
once
we're
over
the
peak,
the
p-values
go
back
to
being
boring,
and
then
you
just
get
this
characteristic
dip.
Where
there's
a
signal,
that's
been
discovered
automatically
by
the
neural
network.
B
I
conclude
the
talk
so
for
this
first
round
the
feature
space
is
pretty
simple,
so
it's
this
two
jet
final
state,
but
we're
only
using
one
feature
from
each
jet.
So
the
feature
space
is
two-dimensional,
one
for
each
jet
and
it's
the
mass
of
the
jet.
It
doesn't
really
matter
what
the
mass
is,
but
it's
just
one
feature
that
characterizes
the
radiation
pattern
inside
the
jet
and
because
it's
two-dimensional,
you
can
simply
visualize
the
neural
network
as
an
image.
So
here
is
a
plot
of
the
neural
network
output
in
one
of
the
signal
regions.
B
So
it's
some
new
particle
decaying
into
other
particles
that
produce
jets
and
the
mass
of
those
jets
corresponds
to
the
masses
of
those
daughter
particles,
and
so
it's
basically
a
very
well
defined
localized
blip
in
this
plot,
and
so
the
green
arrow.
The
green
cross
corresponds
where
we
injected
the
signal,
and
then
you
can.
You
can
see
that
the
neural
network,
this
is
in
data,
has
automatically
learned
to
pick
up
that
there's
something
weird
going
on
at
400
100,
whether
we
injected
the
signal.
B
So
that's
fantastic
and
we
repeated
this
procedure.
You
know
over
all
the
signal
region
that
we
had.
We
found
that
the
the
data
was
that
was
consistent
with
this
movie
following
spectrum,
we
didn't
find
any
evidence
for
localized
excess,
and
so
we
can
proceed
to
set
limits
on
the
production
cross-section
of
of
new
particles
and
I'm
going
to
lead
up
I'm
going
to
build
up
to
this
last
plot
and
it's
kind
of
complicated,
so
I'm
going
to
take
a
few
slides.
B
B
So
in
general,
there
are
four
model
parameters:
there's
the
mass
of
the
a
particle,
the
mass
of
the
b
particle,
the
mass
of
the
c
particle
and
the
cross
section.
How
often
this
this
process
happens,
and
so
we
fix
the
masses
and
then
set
a
limit
on
the
cross
section:
okay,
so
yeah.
This
is
an
official.
Without
this
result,
that
came
out
actually
only
a
couple
of
weeks
ago,
it
uses
the
data
set.
B
The
full
data
set
collected
during
the
last
run
of
the
lhc
during
2015
and
2018
through
2018,
and
the
mass
of
the
a
for
this
particular
plot
is
going
to
be
fixed
and
the
neural
network
is
fixed.
Has
a
fixed
threshold
of
10
and
they're
going
to
be
limits?
Look
like
this,
so
this
is
like
the
480
point,
for
instance,
there's
a
point
and
then
some
error
bar
and
this
corresponds
to
the
95
percent
confidence
level
production
of
some
new
particle.
B
So
this
means
that
if
the,
if
the
particle
production
was
more
than
that,
we
would
have
seen
it
at
95
confidence.
Okay,
so
we
can
do
this
for
all
of
the
signals
that
are
shown
here.
There
are
six
different
models
and
for
each
one
of
them
we
can't
inject
the
signal
into
our
data
stream
and
see
if
you
would
have
seen
it
and
we
didn't
see
anything,
and
so
this
allows
us
to
set
these
limits
and
actually
kind
of
fun
fact.
B
So
we
can
compare
our
limits
to
limits
from
other
searches
that
are
sensitive
to
a
similar
final
state.
So
that's
the
red
and
the
cross,
and
so
the
red
is
an
inclusive
search
that
doesn't
use
any
classification
and
we're
very
competitive
in
many
cases,
much
better
than
that
search
and
the
dedicated
search
in
the
cross
searches,
basically
only
for
the
8080
point
and
it
actually
uses
more
features
than
we
used
so
that
we
expected
that
we
do
a
bit
worse,
but
also
it's
fully
supervised.
B
So
in
general
you
know
these
unsupervised
or
semi-supervised
approaches
are
very
broadly
sensitive,
but
for
a
particular
model
they
can
be
less
sensitive,
all
right,
and
so
I
want
to
just
the
last
slide.
Oh
yeah,
so
just
to
say
that
yeah.
This
is
really
deep.
Learning
supervision,
combined
with
anomaly
detection,
leading
to
a
real
physics
output.
I
think,
for
the
first
time
all
right,
so
I
want
to
have
a
lot.
One
last
slide
of
content
is
just
to
say
that
anomaly.
B
What
would
have
happened
in
hamburg
in
july,
and
this
schematic
diagram
on
the
left
is
just
illustrating
that
there's
a
variety
of
new
methods
that
are
trying
to
be
as
independent
on
the
single
hypothesis
as
possible
and
as
independent
on
the
simulation
of
background
as
possible,
so
up
and
to
the
right
means
independent
and
I
talked
about
koala,
but
the
variety
of
other
methods
here
that
are
based
on
autoencoders
and
neural
density
estimation
and
various
other
things
that
I
can
talk
about
later
if
you're
curious,
okay.
So
that
brings
me
to
my
summary.
B
So
today
I
started
off
by
talking
about
the
immense
potential
of
simulation
based
inference
for
accessing
our
full
face
space
and
the
example
I
gave
there
was
on
the
fold
for
unfolding
or
deconvolution.
Then
I
pivoted
to
talk
about
uncertainty
and
the
categorized
attorneys
as
being
either
optimality
or
accuracy.
Uncertainties,
and
I
discussed
a
new
possibility
for
setting
adversarial
upper
bounds
for
high-dimensional
bias
uncertainties.
B
I
then
talked
about
learning
directly
from
data
without
labels.
I
introduced
a
week's
revision
technique
called
classification
without
labels
or
koala,
and
I
showed
that
that
can
be
used
with
anomaly
detection.
Ending
with
some
results
from
real
collision
data
that
were
released
only
a
couple
weeks
ago,
and
now
my
my
last
slide,
so
I
don't
have
to
tell
you
anymore,
but
I
think
that
it's
clear
that
deep
learning
has
a
great
potential
to
enhance,
accelerate
and
empower
energy
physics,
as
well
as
many
other
areas
of
fundamental
physics
and
and
beyond.