►
From YouTube: 2021-01-12 - Ashesh Chattopadhyay - Deep Learning for Modeling Chaos and Geophysical Turbulence
Description
NERSC Data Seminars Series: https://github.com/NERSC/data-seminars
Title: Deep Learning Approaches for Modeling Multi-Scale Chaos and Geophysical Turbulence
Speaker: Ashesh Chattopadhyay (Rice University)
Abstract: Our atmosphere is a coupled, chaotic, and turbulent dynamical system with multiple physical processes interacting with each other, at continuously varying spatio-temporal scales. Building efficient and accurate weather/climate models that can predict the state of the atmosphere for the near and distant future requires us to resolve a broad range of spatio-temporal scales that often take up a daunt- ing amount of computational resources. Thus, current tractable climate models often have inaccurate and crude approximations of hard-to-resolve physical processes that drastically affect our ability to predict the dynamics of the system. Here, we propose alternative data-driven approaches that utilize deep learning algorithms trained on observations or high-resolution model outputs working in conjunction with numerical models to perform carefully constructed approximations that accurately capture the physics of these hard-to-resolve processes. This can reduce computational cost while bringing more insight into poorly understood physics that can dramatically improve our ability to predict the large-scale dynamics of the atmosphere.
Bio: Ashesh Chattopadhyay did his Bachelors from the department of Mechanical Engineering at Indian Institute of Technology, Patna where he worked primarily in optimization and computational geometry. He got his masters from the University of Texas, El Paso, from the Computational Science program where his research was focused on high performance computing. Since then, he has been a PhD student at Rice University in the department of Mechanical Engineering, where he works at the intersection of theoretical deep learning, dynamical systems and turbulence modeling for broad applications in atmospheric dynamics.
A
Hi
welcome
everyone
to
the
first
data
seminar
of
2021,
very
pleased
to
have
ashash
to
here
to
talk
about
this
project.
Ashash
did
his
bachelor's
from
the
department
of
mechanical
engineering
at
indian
institute
of
technology,
where
he
worked
primarily
in
optimization
and
computational
geometry.
He
got
his
master's
from
the
university
of
texas,
el
paso,
from
the
computational
science
program,
where
his
research
was
focused
on
high
performance
computing.
A
B
You
so
much
for
the
introduction
mustafa.
I
had
a
great
time
at
nurse.
Can
we
continue
to
work
even
after
that?
And
the
later
section
of
the
talk
is
about
the
work
that
I
did
at
nursk,
while
the
first
section
was
something
that
I
had
been
doing
in
my
phd
for
a
while,
something
that
probably
you
guys
and
some
other
folks
working
at
the
intersection
of
dynamical
systems.
B
You
know
climate
science,
flow
dynamics
and
deep
learning
are
sort
of
interested
or
you
know
could
be
potentially
interested,
and
this
was
done
in
collaboration
once
again
with
with
people
at
nurse
mustafa
karthik,
a
lot
of
help
from
jaideep
and
mark
and
then
with
my
advisor
and
an
undergrad
student
at
our
lab,
adam
schubel,
my
advisor
is
pedro
hassans.
Are
there
so
broadly
I'll
be
talking
about
deep
learning
approaches
for
modeling
multi-scale
chaotic
systems?
So
our
climate
system
is
a
multi-scale
chaotic
system
and
geophysical
turbulence.
B
So
usually,
if
it's
a
multi-scale
chaotic
system-
and
there
is
no
spatio-temporal
scale
separation
in
the
system,
then
we
can
call
it
turbulence.
I
I
guess
there
are
more
definitions
and
less
understanding
of
what
turbulence
is.
Generally
or
how
it
comes
about,
but
mostly
I'll,
be
talking
about
standard
turbulence,
models
or
or
climate
dynamics
models
which
are
turbulent
in
general
and
then
developing
theory
for
multi-scale
chaos,
which
is
one
component
to
almost
all
turbulence
models.
B
B
So
these
are
some
of
the
challenges
in
in
weather
and
climate
modeling
turbulence.
The
multi-scale
nature
of
this
of
turbulence,
high
dimensional
systems
and
the
fact
that
they're
so
chaotic,
which
means
that
a
very
small
error
in
in
your
in
your
initial
condition
can
very
quickly
blow
up
and
render
your
predictions
useless
in
in
a
few
time,
steps
of
prediction,
so
the
first
part
of
the
talk
I'll
I'll
make
this
distinction
I'll
call
x
as
the
last
scale
variables.
Whenever
I
talk
about
x,
you
would,
you
should
think
about
large-scale
circulation.
B
B
Now
generally,
what
happens
in
in
turbulent
flow
or
in
climatic
weather
models
is
that
you
have
these
subgrid
scale
processes,
intermediate
small
scale
processes.
Why?
And
you
don't
really
have
a
resolution
that
can
that
can
solve
the
equations
governing
the
processes
of
y.
So
you,
you
don't
have
resolution
that
can
actually
accurately
resolve
the
small
scale
processes,
so
what
people
tend
to
do,
and-
and
this
is
common
to
both
the
turbulence
and
and
and
the
climate
dynamics
community-
is
that
they
do
something
called
parameterization.
B
That
is
the
assume,
since
they
cannot
resolve
y.
I
cannot
run
climate
models
as
a
resolution
that
can
resolve
y.
They
assume
that
y
is
some
function
of
these
large
scale,
variables
x.
So,
for
example,
the
subgrid
scale
forcing
in
a
turbulence
model,
is
assumed
to
be
a
function
of
of
the
result
of
the
mean
flow
variables
and-
and
this
function
is
very
ad
hoc-
it's
it's.
B
So,
generally
these
physics
based
parameterization
of
a
diffusion
component
to
it
and
and
that
way
you
lose
out
an
accuracy,
but
the
same
time
you
make
sure
that
your
code
or
your
less
simulations,
which
are
large
any
simulations
which
mean
that
we
are
only
resolving
the
large
areas
they
don't
blow
up
now
in
in
the
climate,
science,
literature
and
a
little
bit
in
the
turbulence
community
as
well
from
work
by
andrew
meydanon,
but
mostly
in
the
climate
science
literature.
We
have
seen
in
recent
times
an
idea
of
super
parameterization.
B
What
people
tend
to
do
is
that
you
take
each
of
these
grid
points
or
great
cells
in
this
case,
and
then
you
solve
a
high
resolution
equation
for
the
subgrid
scale
processes
inside
each
of
these
grid
cells.
Now
this
is
this
becomes
really
expensive,
but
here
you
don't
actually
have
this
assumption.
You
don't
just
assume
that
y
is
a
function
of
x.
B
B
You
saw
some
approximate
version
of
it,
something
that's
more
tractable,
so,
for
example,
if
if
your
large
scale
grid
grid
cell
is
about
100
kilometers,
then
you
solve
the
smaller
equations
or
the
small-scale
equations
at
a
grid
resolution
of
one
to
ten
kilometers,
and
this
is
called
super
parameterization
and
it's
and
it's
much
more
accurate.
For
example,
it
can
capture
phenomena
such
as
the
madden
julian
oscillation,
something
that's
kind
of
hard
to
capture,
but
but
it
it
comes
with
a
comp
at
a
high
computational
cost.
B
So
ideally,
we
would
like
to
perform
climate
simulations
with
super
parameterization
instead
of
just
parameterizations,
but
this
comes
at
a
cost,
so
the
point
of
this
work
was
or
the
first
part
of
the
talk
was-
was
to
do
super
parameterization
in
a
computationally
tractable
fashion
and
and
and
and
and
of
course,
by
computational
track
tractable
fashion.
I
mean
leveraging
some
ideas
from
data-driven
methodologies,
so
deep
learning
and
then
trying
to
merge
it
or
integrate
it
with
traditional
numerical
solvers.
B
Now
before
jumping
into
that,
we
have
seen
a
whole
lot
of
work
in
the
last
few
years
on
the
first
part.
So
when,
when
people
have
done,
physics
based
parameterization
they've,
just
come
up
with
models
that
that
that
do
y
equal
to
px
so
they've
come
up
with
physics
based
models
for
p,
which
which
often
they
have
some
physical
intuition,
but
are
very
inaccurate.
B
What
a
whole
lot
of
literature
have
focused
on
is.
We
won't
impose
any
structure
on
p.
We
won't
assume
that
it's
diffusion.
We
won't
assume
there
is
some
constant
times
diffusion
so
that
we
can,
you
know,
take
care
of
energy
at
small
scales.
Instead
we're
going
to
throw
data
at
it,
so
we're
going
to
either
run
very
high
resolution
simulations
and
train
a
neural
network
to
give
you
p,
so
we
don't
have
any
structure
on
p.
B
Will
let
a
neural
network
decide
what
p
would
be
or
you
or
you
train
it
on
high
resolution
simulations
and
then
try
and
you
then
do
transfer
learning
on
observations
so
use
a
combination
of
high
resolution
simulations
and
observation
and
try
to
come
up
with
a
neural
network
based
parameterization,
and
it
has
worked
really
well
and
there's
been
a
lot
of
literature
that
have
looked
at
this.
For
example,
stephen
razz
had
had
a
very
famous
pnas
paper
that
that
did
physics-based
parameterization
by
training
it
on
super
parametric
simulation.
B
So
a
common
super
parameterized
model
is
called
sp
cam.
That's
basically
the
super
parameterized
version
of
cam
5
that
I
know
a
lot
of
people
are
not
actually
used.
I
know
mustafa
and
karthik
have
used
it
before
and
then
they've
been
worked
by,
yuval
and
and
and
and
then
some
work
by
chris
betherton
at
udop
and
there's
been
work
by
laura
zana
and
tom
barton
on
doing
a
parametrization
for
ocean.
Ladies
as
well
as
both
that
mostly
sciences
and
the
oceans
like
ocean
dynamics,
people
have
looked
at
physics-based
parametrization.
B
Now
when
we
when
we
wanted
to
explore
the
idea
of
super
parameterization,
it
was
obviously
because
we
thought
it
has
a
certain
advantage
over
just
doing
parameterization,
and
I
will
slowly
get
into
that.
But
this
was
mostly
based
on
work
that
we
did
previously,
where
we
we
saw
that
odes
and
pdes
can
be
directly
integrated
with
rnn
type
architectures,
and
this
was
sort
of
before
we
realized
that
attention
models
can
do
far
better
than
rnn
type
architectures.
B
But
there
are
certain
advantages
to
doing
ions
for
dynamical
systems
completely
from
theoretical
considerations,
but
the
idea
for
super
parametrization
is
pretty
simple.
It's
like
this,
so
you
have
a
dynamical
system
which
is
multi-scale
in
nature,
so
you
have
the
large
scale
x
and
the
small
scales
y
and
let's
just
consider
two
scales
for
the
time
being
so.
Dx
dt
is
some
function
of
x
y
and
a
parameter
p,
which
I
will
talk
about.
B
But
let's
say
it's
some
parameter
so
for
fluid
dynamicist.
This
p
could
be
reynolds
number
and
for
climate
dynamics,
people
this
p
could
be,
for
example,
radiative
forcing
or
you
know,
a
parameter
that
sort
of
controls,
the
effect
of
climate
change,
and
then
you
have
the
small-scale
equation.
Dydt,
which
is
some
other
function
of
most
importantly
x
and
y.
B
B
Remember,
super
parameterization
does
the
same
thing
except
for
this
capital
n,
which
is
a
renewable
network,
or
some
data
driven
model.
Super
parameterization
explicitly
integrates
this
numerically,
so
you
can
find
so
this
this
paper
is
published.
So
you
can.
You
can
see
more
details
about
this
on
on
online
it
it
got
recently
published
in
james,
but
this
is
the
general
idea
of
super
parameterization
super
parameterization.
B
So
to
test
out
this
idea,
we're
going
to
look
at
a
multi-scale
range
96
system.
So
this
is
a
standard
test
bed
to
to
transition
to
turbulence
or
transition
to
climate
models.
You
have
a
large-scale
terms,
a
large-scale
term
x,
and
this
is
a
little
different
from
the
lorenz
96
that
we
would
generally
think
of
when
we
think
of
weather
and
climate.
This
is
a
multi-scale
system,
so
you
have
the
large
scale
variables
x
and
there
are
eight
grid
points
in
x
and
there's
a
forcing
capital
f.
B
This
forcing
is
some
sort
of
radio
forcing
or
you
can
think
of
it
as
an
anthropogenic.
Forcing
this
forcing
determines
how
chaotic
this
system
becomes,
and
then
you
have
an
intermediate
scale
y.
It
has
64
grid
points.
Remember
these
are
representing
small
scale
terms,
so
it
has
to
be
resolved
on
a
higher
resolution
grid.
So
for
each
grid
point
in
x,
you
have
eight
grid
points
in
y,
so
in
total
you
have
64
grid
points
in
y.
B
If
you
have
eight
bit
points
index,
and
then
you
have
your
ultra
small
scale
terms
which
are
like,
if
you
want
to
think
about
physical
intuition,
this
is
similar
to
like
gravity
waves,
like
just
an
example
at
the
top
of
my
head.
These
are
gravity
waves,
super
small
scale,
stuff,
really
can't
resolve
them
until
you
have
very
high
resolution
models,
but
these
have
64
grid
points
for
each
grid
point
in
x,
so
a
total
of
512
grid
points.
So
this
is
a
fairly
high
dimensional
system.
B
It
has
three
spatial
scales
and
three
temporal
scales
as
well
and
there's
a
large
scale,
forcing
term
f,
which
is
sort
of
like
the
parameter
p
that
I
talked
about.
So
this
determines
how
chaotic
the
system
gets.
Now,
let's
look
at
the
time
evolution
of
a
grid
point
of
x,
y
and
z.
You
can
see
that
x
is
slow
moving,
it
has
it.
It
has.
B
B
Okay
and
the
system
has.
The
system
has
been
integrated,
numerically
with
a
fourth
order.
Angle
cut
the
solver
at
200
delta
t.
This
is
the
lyapunov
exponent
of
the
system,
and
this
is
related
to
the
decoration
time
of
the
system.
So
it's
it's
a
it's
a
fairly
chaotic
system.
It
has
three
special
temporal
scales,
they're
still
separated.
In
this
example.
The
the
scales
are
separated
in
this
example
and
turbulence
models.
Do
not
turbulence,
does
not
have
scale
separation
and
I'll
get
to
that.
B
B
Okay,
remember,
y
and
z
are
both
coupled
to
x
and
they're
all
contributing
to
x,
but
generally,
we
are
interested
in
forecasting
the
large
scale
under
the
large
scale
variables
in
atmospheric
dynamics.
So
we're
going
to
look
at
the
evolution
of
x
and
whenever
I
show
you
metric
on
error
or
rmse,
I'm
actually
looking
at
x
by
keeping
in
mind
that
y
and
z
both
contribute
to
the
evolution
of
x.
B
So
in
order
to
get
our
true
data
on
which
we
are
going
to
train
our
models,
we
use
direct
numerical
simulation.
This
is
where
x,
y
and
z
are
integrated
at
a
very
small
delta
t,
and
this
delta
t
is
dependent
is
a
characteristic
time
scale
that
is
dependent
on
z,
so
the
smallest
spatial
spatial
scale
has
to
be
consistent
with
the
delta
t
that
I
choose
so
there's
a
very
small
delta
t
and
you
integrate
the
whole
thing.
B
This
is
the
highest
resolution
simulation
you
can
afford,
for
example,
if
you
can
afford
in
this
example-
and
it
has
maximum
cost
so
I've
I've
normalized
the
cost,
the
cost
of
computing
this.
So
basically
by
cost
I
mean
the
number
of
equations
that
needs
to
be
solved
in
order
to
integrate
it
for
10
delta
t
so,
and
I
will
normalize
in
such
a
way
that
dns
becomes
thousand
and
everything
is
scaled.
With
respect
to
that
now,
a
high
resolution
simulation
would
be
something
like
this.
B
You
basically
ignore
z,
because
these
are
very
small
scale
structures,
so
it's
a
little
inaccurate
than
dns
less
less
accurate
than
dns,
but
you
still
integrate
at
a
delta
t
that
is
characteristic
of
zt.
So
this
is
still
a
very
intractable
simulation
for
all
for
most
client
models,
but
it
is
much
more
accurate
than
parameterization,
for
example,
and
we're
going
to
test
like
compare
all
these
baselines
to
see
where
we
stand
so.
B
Parameterized
models
are
low
resolution
models,
hence
lr,
and
this
is
basically
representative
of
all
common
classes
of
climate
models
that
exist
here
here
x,
which
is
the
large
scale
term,
is
integrated
at
10
delta
t,
while
the
the
small
scale
processes
are
essentially
parameterized
with
some
function.
F,
the
normalized
cost
is
much
lower
and
hence
a
tractable
climate
model.
Well,
super
parameterization
would
be
somewhere
where
x
is
integrated
at
10
delta.
T
y
is
integrated
at
delta
t
and
then
coupled
to
the
large
scale,
solver
every
10
delta
t.
B
B
So
this
is
the
true
data.
This
is
how
accurate
sp
would
be.
1.5,
mtu
or
2mtu
is
actually
a
a
pretty
good
lead
time
prediction
horizon
is
how
well
it
the
the
predictions
look
like.
Is
the
control
plot
of
x
and
t
on
t
on
the
x
axis
and
and
the
the
amplitude
of
the
variables
on
the
y
axis,
and
then,
if
you,
if
you
look
at
this
over
a
hundred
different
initial
conditions
and
you
look
at
prediction
horizon
so,
for
example,
this
is
prediction
horizon.
B
This
is
basically
where
the
relative
error
is
more
than
30.
If
you
look
at
prediction
horizon,
this
is
where
these
models
stand.
High
resolution
models
are
super
accurate,
very
costly.
Super
parameterized
models
are
less
accurate
than
hr,
but
more
accurate
than
low
resolution
parameterized
models,
but
they
come
at
a
huge
cost
as
well.
So
this
is
where
they
stand.
B
None
of
this
is
data-driven
so
far,
there's
all
numerical
models
at
different
resolutions,
so
data-driven
parameterization,
which
is
a
common
class
of
you,
know
common
sort
of
what
other
people
have
been
doing,
where
they
replaced
that
f
by
a
neural
network
trained
on
high-resolution
data.
But
apart
from
that,
it
does
the
same
principle.
This
is
this
is
what
we
will
call
ddp
data
driven
parameterization.
This
is
like
data
driven
subject,
scale,
model
or
turbulence
modules.
They
all
belong
to
this
class
if
they're
data
driven
if
they're
integrated
differently.
B
So
if
the,
if
this
function
is
actually
evaluate
actually
trained
by
a
neural
network,
instead
of
just
assuming
that
it's
some
diffusive
model,
while
data-driven
super
parameterization
has
the
same
principle
as
that
of
sp,
except
it's
integrated
drivenly.
So
you
don't
have
to
numerically
integrate
all
64
equations
of
y.
You
use
an
rnn
to
do
that,
but
then
you
couple
it
at
10
delta
t.
B
So
this
is
where
and
we'll
call
that
ddsp.
So
this
is
where
those
models
stand.
If
you
look
at
one
particular
initial
conditions,
you
have
ddsp
that
predicts
up
till
here
about
two
two
and
a
half
to
t
low
resolution.
Parameterized
models
they're
somewhere
here
much
smaller
than
ddsp,
while
ddp
is
maybe
slightly
better
than
low
resolution
models,
because
you
don't
assume
structure
on
that
function,
but
rather
let
neural
network
evaluate
that
structure.
B
So
when
you
again
predict
when
you
average
this
over
100
different
initial
conditions
and
look
at
prediction
horizon,
this
is
where
our
model
stands.
This
is
what
we
are
proposing.
Ddsp
is
almost
as
good
as
sp
remember.
It
is
almost
as
good
as
sp,
which
is
what
we
want,
but
is
100
times
cheaper.
That's
because
you're
not
actually
integrating
any
equations.
B
So,
when
I
showed
you
the
first
picture
for
x,
y
and
z,
I
showed
you
systems
that
were
in
quasi-equilibrium
with
each
other,
so
the
the
parameterization
assumption
that
says
that
y
is
equal
to
f
x.
That
is
the
small
scale
terms
can
be
represented
explicitly
as
a
function
of
the
large-scale
terms.
That
assumption
is
valid
when
there
is
temporal
scale
separation
between
the
spatial
variables.
So,
as
you
see
here
that
the
the
the
spatial
variables
their
temporal
scales
are
distinctly
separated,
this
is
much
more
intermittent
in
nature.
B
While
this
is
a
slow
moving
variable,
if
such
a
scenario
exists,
then
this
assumption
is
quite
justified.
That's
why
ddp-like
methods
work
so
well,
but
it
real
turbulence.
You
don't
actually
have
this
scale
separation.
If
you
see
here,
the
temporal
skills
are
actually
coupled
to
each
other,
and
this
is
a
more
realistic
situation,
and-
and
this
is
where
the
quasi-equilibrium
assumption
that
y
equal
to
f
x,
assumption
is
not
satisfied
and
in
climate
dynamics
or
in
turbulence.
This
assumption
is
not
satisfied.
It
belongs
to
this
class
of
phenomena,
so
ideally,
ddp
would
perform
very
poorly.
B
When
you
look
at
situations
like
these,
where
there
is
no
temporal
scale
separation,
so
we'll
call
this
case.
One
and
we'll
call
this
case
two
and
case
two
is
the
more
realistic
one,
and
as
soon
as
we
jump
to
the
results
of
case
two,
we
will
see
where
ddsp
has
the
advantage.
So
the
red
circles
are
case,
two,
which
is
the
non-equilibrium
case,
and
this
is
where
the
difference
between
ddspa
and
ddp
actually
stands
out.
When
you
don't
have
spatial
temporal
scale,
separation,
as
in
all
cases
of
turbulence,
models
or
or
climate
models.
B
B
So
the
other
thing
that
that
that
we
looked
at
in
this
problem
was
generalization
to
new
distributions.
So
we
all
know
the
deep
learning.
Architectures
don't
have
any
upper
bound
guarantees
on
generalization
accuracy,
and
while
you
know
it
may
work
for
ml
people
really
well,
because
they're
always
looking
at
test
and
training
data
coming
from
the
same
distribution
for
physical
systems.
This
is
super
important.
You
want
to
train
your
system
on
a
particular
parameter.
B
So,
for
example,
you
want
to
train
your
system
on
a
particular
reynolds
number,
and
then
you
want
your
model
to
generalize
to
other
reynolds
number.
I
mean
otherwise.
There's
really
no
point
it.
You
can't
just
go
and
train
every
time.
You
have
a
change
in
reynolds
number
or
a
change
in
forcing
in
the
atmosphere,
so
it
really
needs
to
generalize
and
really
things
don't
generalize
when
it
comes
to
machine
learning
models,
it
really
doesn't
generalize
when
you
move
away
from
the
distribution.
B
So
technically,
if
you
think
of
the
distribution
of
x
at
f
equal
to
20,
to
be
like
this,
your
system
is
not
going
to
generalize
at
f
equal
to
24
when
you
have
when
the
distribution
is
changed.
If
you
take
test
data
from
here
on
a
model
trained
from
here,
your
your
it
will
be
very
likely
that
your
models
don't
generalize
and
and
for
almost
all
applications
for
physical
system.
This
is
something
that
we
need,
otherwise,
it's
just
not
useful.
B
So
the
the
the
idea
that
we
wanted
to
explore
here
is
what
is
an
inexpensive
way
to
make
sure
these
models
that
we
are
training
generalize
easily
and,
to
be
very
honest,
deep
learning
people
have
sort
of
come
up
with
it
already.
So,
let's
just
take
an
example
to
make
sure
that
there
is
the
absence
of
generalization,
so
this
is
trained
and
tested
with
the
forcing
of
20.
Remember
the
as
the
forcing
increases
the
chaoticity
of
the
system
increases.
B
So
if
we
increase
the
forcing
the
prediction
horizons
for
all,
the
models
will
go
down
because
the
system
has
become
more
chaotic.
Its
inherent
limit
of
predictability
has
decreased
okay,
so
model
to
model,
it
may
differ
a
little
bit,
but
they
would
all
consistently
go
down
now,
if
you
train
a
model
on
f
equal
to
20
and
then
test
it
from
data
at
f
equal
to
24.
B
This
is
where
the
performance
lies.
I
forgot
to
mention
about
dd,
which
is
a
fully
data
driven
model,
which
means
there
is
no
hybrid
numerical
machine
learning
stuff
going
on
over
here.
You
just
take
x
and
you
train
an
r
and
n
to
predict
x,
t
plus
10
delta
t,
so
that's
a
purely
ml
model,
and
that
obviously
does
the
worse
amongst
all
of
them,
because
it
has
no
information
about
the
small
scale
dynamics
that
that
is
forcing
onto
the
large
scale
flow.
B
The
realistic
case.
This
generalization
gap
is
too
high
for
any
of
these
models
to
be
used,
so
what
we
did
was
well
transfer
learning,
obviously
so,
just
to
show
you
the
extent
of
transfer
learning
that
we
can
do
for
a
system
like
this
and
then-
and
you
know
we
have
tested
this
on
lorenz
96
on
2d
turbulence
flow
on.
You
know:
stochastic
burger,
starburst,
hierarchically,
complex,
partial
differential
equations.
B
So
this
idea
really
works
well,
so
you
train
on
one
million
samples
of
f
equal
to
20
and
and
then
you
and
your
training
for
for
parameterized
models,
the
gdp
model,
you
train
an
enn
for
the
ddsp
model.
You
train
an
rnn,
you
you,
you
find
a
set
of
weights,
you
initialize
your
model
for
f
equal
to
24
with
these
set
of
weights,
and
you
take
one
percent
of
the
data
and
you
retrain
the
model,
so
one
percent
of
the
data
this
can
be
done
in
online
fashion.
B
You
don't
actually
have
to
stop
your
production,
go
and
retrain
the
model.
You
can
do
it
on
the
fly
and,
and
then
you
deploy
this
transparent
model
and
as
soon
as
you
do
that
as
soon
as
you
have
transfer
learning
you,
you
see
that
you
get
back
the
generalization
accuracy
so
just
to
ensure
that
this
is
really
an
artifact
for
transfer
learning
and
not
just
a
fluke.
What
we
did
was
we.
B
B
A
B
So
so
so
yeah.
So
this
is
a
great
question
and-
and
let's
put
this
question
much
more
in
perspective,
so
what
mustafa
is
saying
is
that
you
have
you
have
data
from
f
equal
to
20
you
and
then
you
said
you
retrained
it
with
a
little
bit
of
f
equal
to
24..
That's
that's
what
I
said
and
what
mustafa
was
saying
that
you
have
f
equal
to
20,
f,
equal
to
24,
f,
equal
to
28,
let's
train
them
all
together.
B
Let's,
let's
take
data
from
all
these
three
different
probability
distributions
and
then
train
a
model
with
that.
So
from
a
dynamical
systems
perspective.
This
is
called
come
up
with
a
phase
space
that
is
representative
of
all
these
three
distributions
or
for
transfer
learning.
What
I'm
doing
is
I
have
a
phase
space
at
f,
equal
to
20
and
then
I'm
extending
or
stretching
the
phase
phase
space
to
f
equal
to
24.
B
B
B
Center
with
just
some
of
them,
I
see
yeah,
okay,
okay,
thank
you
with
the
same
amount
of
data
set.
Now
I
don't
know
whether
it's
an
artifact
or
the
generalization
error,
or
if
there
is
something
much
more
fundamental
from
a
dynamic
systems
perspective
that
that
makes
it
do
so,
not
that
I've
tried
to
come.
Look
at
you
know,
explanations
as
well,
but
experimentally.
Yes,
this
is
what
I
obtain.
I.
A
B
Okay,
so
yeah
okay,
so
this
was
pretty
much
the
end
of
the
dsp
sort
of
work
that
we
did.
You
can
find
more
results
and
more
and
so,
and
I'm
looking
at
short-term
prediction
horizon
here,
we've
also
looked
at
long-term
predictions,
such
as
the
probability
distribution
for,
say
you
know
a
hundred
year
runs
or
very
long,
runs,
etc,
and
all
of
these
results
hold
up
there
as
well,
and
you
can
find
all
of
them
in
the
paper.
B
But
this
is
where
I'm
going
to
end
my
ddsp
the
work
on
the
ddsp
and
quickly
jump
to
the
work
that
I
was
doing
with
nursk
and
the
the
work
that
I
was
doing
with
mustafa
and
karthik
at
nisk
was
on
data-driven
weather
forecasting.
So
all
of
this
that
you
saw
were
parameterization
or
super
parameterization
efforts
where
you
have
a
numerical
model
working
in
conjunction
with
the
driven
model.
B
At
nurse,
I
was
doing
work
where,
with
fully
data-driven
models,
of
course,
we
were
looking
at
more
complex
architectures.
Deeper
architecture
is
something
that
the
architectures
that
are
more
reasonable
for
forecasting
and
we're
looking
we're
looking
at
data
sets
that
were
more
complex
as
well.
For
example,
the
era
5
data
set,
which
is
actual
observation,
data,
not
actual
observation.
These
are
re-analysis
data,
but
this
is
as
close
to
actual
observation
as
we
could
get
and
we're
not
the
first
people
who
are
doing
their
own
weather
forecasting.
B
A
lot
of
people
before
us
have
done
it
and
the
go
to
architecture
or
go
to
type
of
architecture
has
always
been
encoder
decoders
or
some
sort
of
encoder
decoder
or
unit,
for
example,
and
ideally,
the
way
these
problems
are
set
up
are
not
in
the
form
of
rnns,
because
iron
and
some
inherent
you
know,
stability
issues
with
physical
systems.
The
way
they
set
up
these
problems
is
that
they
show
you
fields
at
time.
T
this.
B
Convolutional
networks
like
whatever
it
may
be,
whatever
fancy
architecture
they've
gone
for
inherently
these
are
convolutional,
so
that
means
they
have
convolutions
they're
pooling
layers
and
a
cascade
of
convolutions
and
cooling
layers.
John
wayne,
then
doraleous
missouri
and
they've
they've
shown
these
across
different
data
sets,
like
jbl,
drive
models
at
different
resolutions,
they're
showing
it
on
era
5
at
different
resolutions
and
they've,
always
shown
while
they
could
get
comparable
performance.
So
a
pretty
good
performance.
B
You
could
never
actually
beat
numerical
weather
forecasts
like
not
even
close
to
numerical
weather
forecasts
and
way
behind
operational
forecasts.
So
as
much
as
their
weather
forecasting
is
super
exciting.
We
are
lagging
behind
in
terms
of
accuracy
or
performance,
so
one
of
the
things
that
we
wanted
to
address
and
we
are
we're
not
competing
to
build
models
that
that
have
incrementally
better
performance
on
this
very
standard
test
bench
of
observational
data.
B
I
did
during
my
phd
that
was
on
capsule
based
architectures,
and
the
idea
is
as
follows:
convolutional
architectures
are
in
translational
invariance
right,
so
so,
for
example,
if
you
give
it
an
image
of
a
boat
upside
down
by
construction,
you
should
not
be
able
to
say
that
it
should
not
be
able
to
say
whether
it's
the
boat
or
not,
whether
it
will
be
accurate
to
predict
some
of
the
inverted
boats
or
not
that's
a
different
question
altogether.
These
are
very
expressive
networks.
B
If
you
train
them
from
enough
number
of
augmented
samples,
it
can
do
anything,
but
by
construction
they
don't
have
any
property
that
allows
them
to
predict
like
predict
on
on
features
that
are
rotated
or
inverted,
and
this
is
super
important
in
physical
systems,
because
a
high
pressure
region
over
a
low
pressure
region,
for
example
in
in
atmosphere,
can
lead
to
very
different
dynamics
than
a
low
pressure
region
over
a
high
pressure
region.
So
these
two
are
different
features.
These
two
are
distinctly
different
features.
B
They
should
not
have
the
same
likelihood
associated
with
it
in
terms
of
predictions,
but
convolutional
architectures
don't
have
a
way
to
to
explicitly
differentiate
them
by
construction.
Once
again,
they
can
differentiate
them
if
they're
trained
on
enough
number
of
augmented
samples,
but
by
construction.
There
is
nothing
there
that
allows
them
to
do
that,
and-
and
this
was
nicely
shown
in
a
saber
at
all
paper
in
2017
eurips,
where
they
showed
that
capsule
based
architectures
have
the
sort
of
ability
to
do
what
combinations
cannot.
B
So
it's
the
go-to
architecture
for
doing
this
should
be
capsules
which
have
a
property
called
equivalence.
Preserving
and
that's
what
I'll
talk
about
in
in
the
in
the
next
few
minutes,
so
this
work
was
the
most
of
the
work
is
about
this
paper
that
we
are
submitting
and
and
and
parts
of
workshop
paper
that
we
presented
at
new
reps
2020
this
year
and
the
idea
is
revolving
around
capturing
rotational
features.
Okay.
B
So
this
is
what
I
mean
by
capturing
rotational
features.
If
you
have
an
input
that
is
rotating,
your
architecture
should
be
by
construct.
We
should,
by
construction,
be
able
to
detect
that
rotation
in
feature
space
and
the
reason
why
convolutional
architectures
cannot
do.
That
is
because
they
have
pooling
layers
and
pooling
layers
cannot
capture
the
large
distortions
in
feature
space.
B
So,
ideally,
what
we
wanted
was
have
a
an
architecture
that
by
construction
or
that
is
sort
of
embedded
in
it,
the
the
property
that
is
embedded
in
it,
which
is
which
would
allow
it
to
capture
rotational
variances
in
the
deeper
features,
and
we
wanted
to
incorporate
this
as
a
at
least
a
sub
differentiable
model
so
that
you
can
do
back
propagation
on
it.
So
the
way
we
did,
that
was
with
spatial
transformer
networks
which
are
equivalence
preserving
in
the
latent
space.
B
So
when
we
set
up
this
weather
forecasting
problem,
we
went
to
an
unit
which
is
a
standard
architecture
that
people
would
go
to
and
you
you
show
it
z,
500
at
time
t
and
you
want
to
predict
z,
z,
500
at
time,
t
plus
delta
t.
This
is
the
sort
of
architecture
you
use
the
only
what
and
we
ensure
that
there
is
zonal,
periodicity
enforced
by
doing
circular,
convolutions
and
whatnot.
B
So
the
the
results
that
I'm
showing
you
here
is
where
we
are
trained
on
12
hourly
z500
patterns
from
the
error,
5v
analysis
training
was
done
in
1979,
2015
validation
between
2015
and
2017
and
testing
was
done
on
data
set
from
2018.
We
have
separated
out
the
year
so
that
there
is
no
correlation
between
the
patents
and
we
see
the
ustn,
which
is
the
equivalence
preserving
unit,
at
least
for
this
specific
example
does
better
than
the
than
unit
architecture.
12
means
it.
B
B
The
other
thing
that
we
addressed
here
was
adding
data
assimilation
to
this
whole
whole
weather
forecasting
model.
So
how
does
that
work?
Well,
data
assimilation
is
as
follows:
you
have
a
model
that
forecasts
and
that
keeps
on
forecasting
from
an
initial
condition,
because
it's
a
chaotic
system
you'd
soon
lose
out
your
prediction,
because
it's
because
the
errors
add
up
and
you'd
soon
diverse
from
your.
B
B
If
you
just
replaced
it
with
that
noisy
observation,
so
it's
a
way
of
reducing
uncertainty
in
the
updated
state.
Every
time
you
have
an
observation
now
forecast
in
general
is
done
with
a
very
expensive
numerical
model,
but
over
here
we
have
a
data
driven
model,
which
is
our
ustn
model
and
we
are
doing
it
every
hour.
So
we
have
an
hourly
usd
and
trained
that
predicts,
z,
t
z,
t
plus
delta
t
z,
t
plus
two
delta
t,
so
on
and
so
forth.
At
every
24th
hour
it
receives
a
noisy
observation.
B
B
These
are
some
of
the
details
of
how
the
en
kf
is
implemented,
and
if
there
are
questions
I
can
go
into
the
details
of
them.
So
you
essentially,
what
you
do
you
get?
You
evaluate
a
kalman
gain
based
on
a
large
number
of
ensembles,
and
because
you
have
a
data
driven
model,
you
can
actually
do
a
large
number
of
ensembles.
B
By
large
I
mean
4
000
ensembles,
which
is
typically
intractable,
with
a
numerical
model
where
people
do
50
ensembles
at
best,
but
because
it's
a
data
driven
model,
I
can
do
as
much
as
I
want,
because
they're
super
inexpensive
during
inference,
and
then
you
use
this
kalman
gain
which
you
have
identified
from
the
ensembles
and
you
have
an
updated
state
and
acts
as
an
initial
condition
as
you
move
forward
further
along
in
time
with.
As
you
look
at
the
results,
this
is
with
a
smaller
noise.
The
blue
one
is
with
smaller
noise.
B
So,
every
time
you
have
a
simulation,
your
correlation
gets
better,
which
means
your
performance
improves,
because
your
initial
condition
is
improved,
so
this
state
is
updated
because
it
receives
new
information
and
it
acts
as
a
new
initial
condition
and
again
every
24
hours
have
you
have
improvement
in
performance
and
and
how
how
how
better
you
do
is
completely
dependent
on
on
the
the
level
of
noise
that
you
have.
So
if
you
have
more
noise
in
your
observations,
you
will
always
be
lower
than
if
you
have
less
noise
in
your
observations.
B
B
Now
here's
something
interesting
that
you
that
that
we
found
recently
and
and
initially
we
thought
it
was
a
bug
and
I'm
happy
to
say
that
we
have
tested
this
out
on
newer
systems.
Now
that
mustafa,
you
probably
don't
know
about,
I
got
some
new
results
today
on
a
different
system
and-
and
this
really
works
out-
and
we
call
this
a
curious
case
of
temporal
sampling.
So
the
idea
is
like
this.
If
I
train
a
model
to
predict
every
hour,
I
can
plot
my
correlation
with
hours
and
rmse
with
hours.
I
would
get
a
graph.
B
I
can
also
train
a
model
that
predicts
every
six
hours.
I
can
also
train
a
model
that
predicts
every
12
hours.
If
you're
a
numerical
analyst,
you
would
think
a
model
that
predicts
every
hour,
which
is
high
resolution,
should
be
better
than
a
model
that
predicts
every
12
hours
for
a
deep
learning
model.
This
the
relation
is
exactly
opposite.
A
model
that
predicts
every
12
hours
is
much
better
than
a
model
that
predicts
every
six
hours
and
and
and
hierarchically
as
a
as
compared
to
a
model
that
predicts
every
hour.
B
The
blue
line
has
the
worst
performance.
It's
an
hourly
model.
The
black
line
is
a
12
hourly
model,
it
has
the
best
performance,
so
the
relationship
is
entirely
inverted
and
that's
because
this
is
an
autoregressive
model.
Every
time
you
call
this
model,
you
are
incorporating
generalization
error,
so
you're
adding
error
every
time
you
call
a
deep
learning
model
because
it
has
a
generalization
error
associated
with
it,
the
less
number
of
times
you
call
it
the
better
performance.
You
should
expect.
B
B
D
B
Right
so
so
there
are
certain
artifacts
here
that
I
I
I
could
not
explain.
For
example,
this.
This
drop
that
you
see
is
related
to
this
weird
plateauing
in
the
rmsc
as
well.
So
to
be
very
honest,
I
I
don't
have
a
very
clear
explanation
as
to
where
why
this
drop
happens
or
what
is
contributing
to
this
drop
but
to
to
to
put
things
in
perspective.
B
B
If
you
have
a
noise
initial
condition
and
you
have
a
generalization
error
and
there's
a
third
thing,
there's
an
error
associated
with
the
non-linearity
of
the
system,
so
there's
an
error
associated
with
the
lyapunov
exponent
of
the
system,
because
chaotic
systems
have
non-linear
error
transitions,
so
there
are
three
errors
contributing
to
to
the
final
value
of
the
error
that
you
see
at
every
time
step
and
all
of
them
interact
non-linearly,
so
there
none
of
them
are
linear
with
time.
So
it's
sort
of
hard
to
analyze.
B
What
is
which
one
is
you
know
causing
which
artifact
in
this
curve?
So
to
be
very
honest,
I
don't
have
a
very
good
answer
to
that.
So,
for
example,
if
I
change
the
system
to
qg,
I
see
a
very
linear
line
in
terms
of
like,
in
terms
of
you
know,
drop
in
error
and,
moreover,
I
see
a
very
linear
relationship
between
1
6
12
like
that.
B
So
if
you
cut
off
say,
for
example,
0.6
and
you
look
at
the
prediction
horizon,
I
see
that
the
prediction
horizon
is
linear
function
of
the
sampling
time
period
or
the
inverse
of
the
sample
of
the
sampling
frequency,
but
I
don't
have
a
very
good
explanation
as
to
what
which
of
these
artifacts
is
associated
associated
with
each
of
these
errors.
I
want
to
point
out
the
three
very
non-linearly,
interacting
error
terms,
propagating
every
time
you
call
this
model
and
one
of
these
errors,
the
generalization
error.
B
E
B
That's
an
interesting
question
by
construction.
No,
at
the
same
time,
this
model
is
predicting
it's
just
taking
z
and
predicting
z.
Okay.
So
if
you're
talking
about
kinetic
energy,
so
sure
you
can
take
the
derivative
of
z-
and
that
would-
although
that
is
not
explicitly
velocity,
because
it's
your
potential,
but
it's
somehow
related
to
velocity
and
then
you
can
think
of
deriving
the
kinetic
energy
from
there.
B
But
for
real
data
thinking
about
conserving
energy
is
very
difficult,
but
for
toy
systems
you're
right
you
could,
you
could
sort
of
do
a
hand
calculation
from
your
predicted
variable
or
do
a
numerical
calculation
from
your
predicted
variable
and
see
whether
you're,
conserving
energy
or
not
and
by
construction?
No,
I'm
not
I'm
not
conserving
energy
by
construction,
whether
it's
ultimately
conserving
it
or
not.
I
don't
know
for
the
qg
system.
Yes,
it
does
conserve
energy,
but
for
a
different
system.
I
wouldn't
know
because
it's
not
there
by
construction
by
construction,
there's
only
equivalence.
E
B
Yes,
yes,
so
since
there
are
great
questions,
I
could
just
I'll
quickly
go
through
this
and
we
can
take
other
questions.
So
basically,
I
use
this
observation.
This
thing
that
12
hourly
model
is
more
accurate
than
an
hourly
model
and
use
them
as
virtual
observations,
and
I
do
data
assimilation
every
12
hours
as
well.
Remember.
This
is
not
real
data.
This
is
model
forecasts
which
are
more
accurate
than
this
model
forecast,
which
is
an
hourly
model,
so
use
this
as
virtual
observation
and
update
the
state
every
12
hours.
B
I
could
have
done
this
at
every
six
hours
as
well,
then
this
could
be,
and
this
could
really
become
a
very
sophisticated
sort
of
forecasting
pipeline.
But
this
is
just
what
I
tested
and
then
I
do
again
observation
assimilation,
every
24
hours
and
when
you
do
that,
you
see
that
every
12
hours
there's
a
jump
in
performance
and
there's
a
slightly
less
jump
in
performance,
every
24
hours
and
this
sort
of
dies
down
as
with
time,
even
your
12
hourly
model
loses
out
its
accuracy,
at
which
point
of
time.
B
So
this
is
generally
the
idea
of
doing
ml
and
da
with
virtual
observation.
So
the
goal
of
this
project
was
to
do
weather
forecasting.
The
way
weather
forecasting
should
be
done.
That
is
not
free
forecasting,
but
with
data
assimilation
and
then
use
these
deep
learning
tricks
or
these
weird
temporal
sampling
ideas
or
observations
that
we
had
and
integrate
them
in
this
data
simulation
framework
with
machine
learning
make
sure
we
have
a
good,
or,
you
know,
potentially
say
optimistic
forecasting
pipeline.
B
That
is
fully
data
driven
okay,
so
there's
no
numerical
model
working
here
and
correlations
are
above
0.9.
So
this
is
by
no
means
way
behind
numerical
weather
forecasts
either.
So
this
is
sort
of
promising,
and
we
are
looking
more
into
this
and
I
think
that's
it.
Thank
you
and
I'm
happy
to
take
questions
or
go
through
some
of
the
previous
slides.
If,
if
you
have
one,
if
you
want.
A
F
F
B
This
is
about
five
and
a
half
degrees,
5.65
degrees,
while
the
higher
the
you
know,
the
numerical
weather
forecasts
are
at
the
high
resolution.
So
that's
why
I
couldn't
compare
with
nwp
forecasts,
but
if
you
simply
interpolate
the
high
resolution
forecast
onto
this
grid,
then
we
are
at
comparable
accuracy.
Of
course
these
are
not
free
predictions.
B
A
We
know
that
like
if
we,
if
we
you,
know,
train,
bigger
and
bigger
models,
and
and
all
of
that
we
can
start
reducing
the
the
forecasting
error
further
right
and
then
right
should
be
some
as
you
started
with.
There
is
some
some
a
lot
of
features
for
these
chaotic
systems
that
are
unpredictable
by
by
their
intrinsically
unpredictable
right
right
right.
You
think
that
the
the
irreducible
error
should
be
the
same
error
that
you
would
get
from
a
full
physics
simulator
or
there
should
like
it
would
be
something
much
higher
than
that.
A
A
B
I
think
the
irreducible
error
that
is
coming
from
the
intrinsic
property
of
the
chaotic
system,
that
is
the
irreducibility
of
the
error
coming
from
the
lyapunov
exponent
because
of
error
propagation
through
the
lyapunov
exponent.
They
should
be
the
same
if
you
have
the
best
deep
learning
emulator
out
there,
but
the
generalization
error
for
the
deep
learning
emulator,
that's
something
that
we
don't
have
a
control
over
or
you
know,
even
an
any,
even
a
weak
bound
over.
You
don't
have
a
big
o
bound
or
a
big
omega
bound
any
sort
of
bound
over
that
error.
B
C
A
B
It's
I
guess
jedi
would
be
best
to
answer
this
question,
but
for
real
data
it
is
still
empirical
for
systems
where
you
can
actually
go
and
calculate
the
jacobians
and
so
on.
You
could
still
have
a
pretty
good
estimate
of
how
this
error
propagates
through
the
lyapunov
exponent,
but
for
era5
the
model
is
so
complicated.
The
underlying
model
is
so
complicated
with
the
data
assimilation
and
everything.
I
think
it's
a
very
empirical
calculation.
B
A
B
You
can
so
two
problems
for
the
for
the
lorenz
96
system.
You
can
actually
go
and
calculate
the
lyapunov
exponent
of
xyz
separately
by
you
know,
calculating
the
jacobians
and
all
so
that
is
a
very
good
estimate.
The
value
that
I
gave
you
4.5
for
the
x
variable
alone.
That
is
a
very
good
estimate
for
error.
5.
B
I
never
reported
liable
of
exponent
because
one
is,
I
only
have
access
to
z
and
there
is
no
point
calculating
level
of
exponent
with
z,
but
calculating
layout
of
exponent
from
data
is
much
more
finicky
than
any
machine
learning
model
out
there.
So
there's
there's
really
no
point
in
calculating
element
of
exponent
from
any
of
the
variables
in
error,
five
using
just
data
alone,
but
for
the
lorentz
system.
It's
a
very
robust
value
that
I
showed
you
for
4.5,
that's
from
the
equations,
not
from
data.
C
Sorry,
I
know
we're
going
over
time,
but
just
like
a
quick
question
is
that
you
know
there
is
some
subtlety
to
like
you
know
when
you
have
these
multi-scale
models,
I
mean
like,
what's
the
right,
lyapunov
exponent
to
sort
of
report
which
which
actually
like
is
meaningful
in
terms
of
the
error
propagation
right
and
did
you
ever
like?
Do
you
try
to
like
make
sense
of
that?
Because
you
know
if
the?
If
you
try
to
look
at
the
fast
scale,
the
open
of
exponent,
then
it'll
give
you
like
yeah.
C
B
So
because
I
am
looking
at
the
large
scale
flow
you're,
absolutely
right.
The
the
fast
scale
year
point
of
exponents
will
also
contribute
to
that
error.
But
because
I
was
looking
at
just
the
large
scale
flow,
I
calculated
the
lyapunov
exponent
using
using
the
equations
from
just
x
now,
given
that
there
is
an
expression
of
y
in
the
x
equations
as
well,
so
when
you're
finally
calculating
jacobian.
B
That
effect
would
come
in,
but
I
did
not
quite
combine
the
three
exponents
as
such,
but
I
calculated
the
exponent
just
from
x,
but
there
is
a
term
involving
y
in
that
equation,
so
that
part
is
accounted
for.
So
it's
not
the
best
measure
of
liability
of
exponent,
but
it
is
sort
of
the
measure
that
people
generally
use.
So
this
thorns
at
all
paper
at
royal
met
society
2017,
that's
pretty
much
how
they
calculated
the
layupon
of
exponent
as
well,
and
they
showed
it
in.