►
From YouTube: 2022-06-28 - S. Subramanian - FourCastNet: Data driven high resolution atmosphere modeling at scale
Description
NERSC Data Seminars Series: https://github.com/NERSC/data-seminars
Title:
FourCastNet: Data-driven, high-resolution atmosphere modeling at scale
Speaker:
Shashank Subramanian, Data & Analytics Services Group, National Energy Research Scientific Computing Center (NERSC), Lawrence Berkeley National Laboratory
Abstract:
We present FourCastNet, short for Fourier Forecasting Neural Network, a global data-driven weather forecasting model that provides accurate short to medium-range global predictions at 25km resolution. FourCastNet accurately forecasts high-resolution, fast-timescale variables such as the surface wind speed, total precipitation, and atmospheric water vapor with important implications for wind energy resource planning, predicting extreme weather events such as tropical cyclones and atmospheric rivers, as well as extreme precipitation. We compare the forecast skill of FourCastNet with archived operational IFS model forecasts and find that the forecast skill of our purely data-driven model is remarkably close to that of the IFS model for short to medium-range forecasts. FourCastNet generates a week-long forecast in less than 2 seconds, orders of magnitude faster than IFS, enabling the creation of inexpensive large-ensemble forecasts for improved probabilistic forecasting. Finally, our implementation is optimized and we present efficient scaling results on different supercomputing systems up to 3808 NVIDIA A100 GPUs, resulting in 80000 times faster time-to-solution relative to IFS, in inference.
Bio:
Shashank Subramanian is a NESAP for learning postdoctoral fellow with research interests in the intersection of high-performance scientific computing, deep learning, and physical sciences.
Host of Seminar:
Peter Harrington, Data & Analytics Services Group
National Energy Research Scientific Computing Center (NERSC)
Lawrence Berkeley National Laboratory
A
B
We're
going
I'm
going
to
be
presenting
this
work
on
forecast
net,
which
is
a
gear
driven
high
resolution,
weather
model
surrogate.
So
before
I
get
into
the
presentation,
I
do
want
to
acknowledge
the
team
behind
this
work,
that
comprises
of
several
researchers
from
various
institutions
from
nvidia
nurse
purdue,
rice
and
caltech.
B
Okay,
just
to
motivate
the
problem
a
little
bit,
the
challenge
is
probably
clear
to
a
lot
of
people.
Today,
there
has
been
a
dramatic
rise
in
extreme
weather
events
across
the
globe
that
is
attributed
to
climate
change,
that
is
happening
here
and
today
and
is
amplifying
the
changing
weather
patterns.
B
And
basically
there
is
this
large
need
to
be
able
to
simulate
global
weather
patterns
and
be
able
to
predict
the
future
of
the
atmospheric
state
in
order
to
make
well-informed
decisions
and
climate
science
has
been
progressing
in
the
past
several
decades
in
terms
of
modeling
very
steadily,
whether
we're
being
able
to
model
higher
and
higher
resolutions.
In
order
to
now,
there's
all
finer
scale
structures-
and
this
is
a
figure-
that's
taken
out
of
a
nature
paper,
a
few
years
back
and
adapted
by
nvidia.
B
Structures
of
storms
and
hurricanes,
and
that
will
require
a
10,
000
x,
compute,
because
compute
roughly
scales
as
a
resolution,
to
the
fourth
and
in
order
to
get
to
this
holy
grail
scale
of
a
meter
resolution
which
resolves
these
low
cloud
parameterization
processes
and
that
actually
represents
the
greatest
uncertainty
in
climate
models.
That
would
require
like
a
billion
x,
compute
and
it
would
based
on
current
plans.
It
would
take
up
to
2060
to
reach
that
resolution
and
by
which
time
that
would
be
too
late.
B
B
So
today
we
have
orders
of
tens
of
ensembles
that
is
run
operationally,
but
we
would
need
about
thousands
of
ensembles
to
adequately
characterize
the
distribution
of
possible
outcomes,
and
this
can
really
help
us
in
being
able
to
quantify
extreme
events
and
finally
related
to.
The
second
challenge
is
the
scalability
and
performance
of
such
models,
so
current
models
are
not
designed
to
exploit
modern,
supercomputing,
substrates
and
gpus,
etc,
and
high
performance
gains
that
you
get
from
a
lower
mixed
position.
C
B
These
models
influence
are
blazingly
fast
and
can
run
on
just
orders
of
artisan
one
gpus
and
that
can
really
help
us
dramatically,
reduce
energy
costs
and,
finally,
today,
a
lot
of
optimization,
a
lot
of
optimized
implementations
with
really
great
software
frameworks
to
enable
scalable
ai
to
different
parallelism
strategies
and
towards
this
end,
we
introduced
this
model
for
gas
net,
which
is
a
scalable
data-driven,
global
weather
forecasting,
surrogate
model
and
to
the
best
of
our
knowledge.
This
is
the
color.
B
B
This
is
like
a
operational
research
facility
that
gives
you
24
7
weather
forecasts
of
important
atmospheric
variables,
so
the
data
is
at
40
years
is
for
40
years,
at
hourly
intervals
for
several
variables
at
a
25
kilometer
grid.
That
roughly
translates
to
an
image,
that's
of
size
720
by
1440
pixels.
So
these
are
like
some
example:
images
taken
at
some
specific
time
stamp.
B
This
is
a
slower
moving
field,
that's
your
potential
height,
and
this
is
the
global
two
meter
temperature
pattern,
and
we
can
immediately
see
that
there's
a
lot
of
fine
scale
structures
that
we
that
we'll
have
to
model
really
well
and
in
general,
the
ef5
of
the
analysis.
Data
set
represents
the
best
available
estimate
of
the
earth's
atmospheric
state
because
it
performs
a
data
assimilation
step
by
incorporates
observations,
already,
observations
from
the
earth's
atmosphere
into
a
numerical
weather
prediction
model
and
the
way
it
works
is
this
real
analysis.
B
State
is
taken
as
an
initial
condition.
That's
fed
into
a
numerical
weather
prediction
model
that
essentially
forwards
steps
that
in
time
to
get
the
future
of
any
of
the
atmospheric
states
and
what
we
want
to
do
is
we
want
to
replace
that
numerical
prediction
model
with
a
data
driven
model.
So
our
basic
strategy
here
is
to
create
forecasts
by
recursively
stepping
forward
in
time.
B
So
this
field
is
pretty
amazing.
So,
as
a
starting
point,
most
most
works
use
a
subset
of
the
full
state
variable
to
characterize
the
atmospheric
state,
so
forecast
net
uses
about
uses
21
variables-
that's
highlighted
in
this
table
here,
so
we
have
variables
related
to
our
surface
surface
wind,
velocities
and
temperature
and
the
same
states
at
various
different
pleasure
levels
and
also
variables
that
will
call
them
water,
vapor
and
total
precipitation.
B
That's
not
included
in
this
table
because
we
deal
with
that
separately
and
I'll
get
to
that
in
a
few
slides
in
the
future.
We
do
look
at
using
all
the
variables
like
numerical
weather
prediction
models,
but
it's
possibly
unnecessary
to
use
all
the
variables
to
characterize
the
state
and
maybe
smart
dimension.
B
Dimensionality
reduction
techniques
can
help
us
here,
but
from
a
high
level
input
that
to
our
model
is
a
21
by
720
by
1440
tensor,
so
21
channels
represents
21
variables
and
that's
the
height
and
width
of
the
image
and
during
training
we
essentially
predict
a
tensor
of
the
same
size.
At
time.
T
plus
delta
t
from
x
of
t
and
influence
is
auto.
Regressively
forward
stepped
in
time,
so
the
output
from
the
model
is
once
again
fed
back
as
input,
and
this
is
repeated
to
get
more
and
more
forecasts.
B
A
B
Yeah
I
mean
we
haven't,
looked
specifically
at
training
with
vsan.
This
is
all
there's.
Definitely
this
climate
shift
that
is
probably
going
to
affect
us,
but
here
we
actually
use
the
full
40
years
for
training
like
like.
I
think
we
use
1979
too
20
2015
2016
to
play
in
the
full
model,
but
it's
something
that
it's
a
it's
a
very
good
point
and
we
are
looking
at
how
there's
a
drift
to
the
climate
change
and
how
that's
going
to
affect
our
predictions.
B
Okay,
so
so
our
model
is
based
on
a
vision,
transformer
backbone,
so
vision
transformers
are
one
of
the
state-of-the-art
networks
for
now
vision,
tasks
that
are
an
alternative
to
convolutional
neural
networks
and
the
way
they
work
is
they
have
this
tokenization
of
input
using
a
patch
embedding
layer.
So
if
you
have
an
image,
that's
it's
first
split
into
patches
and
each
patch
is
essentially
flattened
and
projected
onto
a
high
dimensional
space.
B
So
we
burst
at
all
came
up
with
adaptive
free
annual
operators
this
year,
and
I
see
a
lot
and
I
do
invite
you
to
check
out
that
paper
for
more
details,
but
from
a
high
level.
They
wanted
to
control
the
complexity
of
the
operations
further,
and
so
they
introduced
like
a
block
diagonal
structure
on
the
channel
mixing
weights.
That's
in
the
frequency
domain,
so
essentially
a
d
by
d.
Kernel
is
now
split
into
k.
Blocks
of
size
d
by
k
by
d
by
k
and
the
weights
are
also
shared
across
every
single
input.
B
So
you
have
an
input
that
goes
through
the
embedding
and
goes
to
the
spatial
mixing
that
happens
in
the
fourier
space
and
then
there's
also
this
channel
mixing
layer
that
comes
out
which
essentially
just
mixes
the
inputs
in
the
channel
dimension
and
that's
just
a
multi-layer
perceptron
that
goes
through
a
linear
decoder
that
reshapes
it
back
into
a
tensor
of
the
same
shape
as
the
input
at
time.
T
plus
delta
t.
B
We
use
the
full
data
set
for
training
and
validation
is
on
2016
and
2017
for
hyper
parameter
tuning
and
getting
the
best
model
configuration
for
testing.
We
test
on
audios
from
2018
and
just
to
give
you
an
idea
of
the
data
set
size
is
about
five
terabytes
and
we
have
approximately
fifty
thousand
images
that
has
two
thousand
samples
that
need
to
get
to
entertaining
in
a
training
iteration.
B
B
So
we
do
treat
total
precipitation
as
a
diagnostic
variable
and
don't
include
it
in
the
original
list
of
variables,
because
this
is
typically
what's
done
due
to
the
fact
that
this
variable
is
extremely
challenging
to
predict.
It
has
highly
exposed
features
and
a
distribution,
that's
very
different
from
the
other
variables
and
in
general
it
doesn't.
It
doesn't
affect
the
evolution
of
like
the
other
prognostic
variables,
so
so
for
precipitation.
We
train
a
separate
model,
that's
completely
diagnostic.
B
B
B
So
here
the
red
curve
denotes
the
model
prediction,
the
the
acc
anonymous
value
from
forecastnet,
whereas
the
blue
is
from
the
ifs
and
and
we
see
that
our
values
are
really
compatible,
and
I
will
get
to
that
for
the
other
variables
in
a
few
slides
related
to
the
previous
slide.
We
are
able
to
predict
hurricane
pats
and
intensities
quite
well.
So
this
is
the
track
of
hurricane
michael
that
had
devastating
impacts
in
places
like
florida
in
2018.
B
Pretty
well
so
these
are
surface
wind
speeds
and
wind
speeds
at
a
different
level
up
to
up
to
three
days,
and
you
see
that
compared
to
the
ground,
fluid
a
model
is
able
to
predict
the
intensification
of
the
hurricane
pretty
well.
We
do
have
a
gap
to
close
in
terms
of
the
intensity
of
this,
and
that's
probably
because
we're
still
not
a
resolution
that
can
capture
fine
scale
structures
of
storms.
Well,
but
we
are
hopeful
that,
as
we
go
to
higher
resolution,
we're
able
to
close
this
gap.
B
Another
important
thing
is
that
we
can
capture
near
surface
wind
predictions
or
land
pretty
well.
This
has
important
implications
to
things
like
wind
resource
planning
for
wind
farms.
This
is,
in
general,
a
harder
field
to
predict
because
of
different
oil
grafted
features
over
land,
but,
as
we
can
see
compared
to
the
ground
fluid
we
are
able
to
capture
some
of
these
high
wind
speed
patterns
over
the
united
states.
Pretty
well.
B
So
coming
to
precipitation,
this
is
one
of
the
most
challenging
fields
to
predict
in
weather
forecasting.
We
do
show
a
pretty
good
skill
compared
to
ifs
on
this
variable
too.
So
this
is
once
again
the
spatial
temporal
prediction:
that's
the
initial
condition
and
the
predicted
value
at
36
hours,
and
this
is
the
ground
fluid
from
the
analysis
data
set.
B
And
one
of
the
reasons
why
we
do
well
on
total
precipitation
is
because
we
do
have
moisture
available
dynamics
that
are
captured
well
in
our
backbone
model.
So
this
is
one
of
those
fields.
This
is
just
a
video
or
96
hours.
The
top
row
is
forecast
net
predictions
on
the
bottom
is
the
ground
fluid,
and
this
is
just
a
zoomed
in
region
of
the
atmospheric
river.
B
B
B
If
you
go
beyond
the
98
percentile,
we
do
see
that
forecast
net
lags
behind
the
ifs
that
also
lags
behind
the
actual
ground
smooth
another
way
to
visualize.
This
is
to
plot
the
relative
quantile
error,
which
is
the
error
between
the
top
quantiles
between
the
ground
fluid
and
the
predicted
and
the
predicted
field,
and
once
again
we
will
see
that
we,
we
are
lagging
behind
ifs
on
extremes.
B
B
But
for
the
sake
of
comparison,
we
downsample
our
prediction
and
then
compute,
acc
and
rmc
values
for
two
fields
that
both
our
models
predict.
So
here
the
black
line
is
the
acc
embassy
value
from
dlwp
and
as
before,
the
red
is
from
forecast.
Net
and
ifs
is
the
blue
line,
and-
and
we
do
see
that
we
have
a
much
higher
scale
on
weather
time
scales.
B
Okay,
so
I'm
moving
into
some
of
the
computational
aspects,
which
is
the
second
setting
part
of
the
talk
forecast
net,
is
faster
and
more
energy
efficient
than
numerical
weather
prediction
models.
So
in
this
table
we
are
summarizing
the
latency
and
energy
consumption
for
the
24
hour,
the
100
member
ensemble
forecast.
B
B
B
But
there
are
actually
several
caveats
here
that
you
want
to
make
note
of.
First,
this
forecast
net
is
not
physics
constrained.
It
is
really
important
to
have
physically
consistent
solutions
because
well,
when
you
want
to
extrapolate
forward
in
time,
we
don't
want
to
be
making.
You
know
we
don't
want
to
be
making
on
physical
predictions.
However,
there's
a
wealth
of
research
that's
happening
on
how
to
introduce
physics
constraints
into
deep
learning
models,
and
that
is
something
that
we're
looking
at
moving
forward.
B
B
So
in
terms
of
the
model,
the
main
computational
engines
are
ffts
and
dense,
matrix
multipliers.
So
currently
we
had
a
25
kilometer
resolution
sequence.
Land
is
about
a
million,
that's
essentially
720
by
1440,
and
you
would
have
to
perform
d
ffts
in
every
in
every
block.
So
d,
once
again
is
the
embedding
dimension
and
that's
typically
about
10,
24
or
20
48..
B
Moving
to
higher
resolutions,
the
image
size
would
become
3600
by
7200
and
eventually,
at
the
largest
scale.
18
000
by
36
000
is
going
to
be
the
image
dimension,
so
we
would
really
need
to
have
distributed
ffts
to
tackle
this
problem,
but
there's
a
lot
of
work
happening
today
in
qfft
to
get
data
efficient,
distributed
ffts
to
scale
out
to
multiple
nodes.
So
this
is
a
challenge
that
we
can
definitely
tackle.
B
We
have
dense
matrix
multiplies.
This
is
specifically
in
the
channel
mixing
layer
in
the
in
the
input
space,
so
those
matrix
matrices,
look
like
4d
by
d
times,
t
plus
n,
but
once
again
d
is
the
embedding
dimension
and
in
general,
as
you
go
to
higher
and
higher
resolution,
not
only
does
the
compute
go
really
high,
but
our
memory
footprint
also
increases
so
even
at
intermediate
resolutions.
B
Finally,
we
are
now
going
from
one
time
step
to
just
the
next
time
step,
but
we
could,
in
principle,
have
a
history
of
time,
steps
and
increasing
that
temporal
resolution
once
again
increases
the
scaling
challenges.
The
big
point
here
is
that,
as
we
move
forward,
we
would
have
to
think
of
different
parallelization
strategies.
B
So
today,
so
I
mean
before
that.
I
want
to
talk
about
what
what
parallelization
strategies
exist
for
for
deep
learning
models.
So
the
most
standard
thing
is
data
parallelism
where
we
have
a
batch
of
images,
that's
passed
into
the
model
and
the
batch.
The
global
batch
is
now
split
across
different
processors
or
gpus.
In
this
case,
and
essentially
every
process,
every
processor
has
a
copy
of
the
model
rates,
does
its
computations
independently,
and
only
when
you're
updating
the
weights
is
an
all-reduced
operation
to
get
the
latest
gradients
onto
every
single
processor.
B
So
that
is
split
into
different
gpus
and
these
dense
layer
operations
now
become
disputed,
matrix
matrix,
matrix
multiplies
so,
for
instance,
the
hidden
size
is
distributed.
We're
now
looking
at
like
a
low
parallel
matrix
multiply
in
the
forward
pass
and
it's
a
different
operation
in
the
backward
pass
and
I'll
get
to
that
soon.
B
B
B
If
you
have
four
gpus
in
a
single
node
at
most,
you
would
have
a
four-way
model
parallelism
and
those
four
gpus
would
comprise
a
single
model
instance,
and
this
is
done
so
that
we
can
exploit
the
low
latency
high
bandwidth,
envy,
link,
interconnect
between
them
and
the
data
parallelism
now
happens
across
different
model
instances.
So
now
we
have
to
be
careful
about
gradient
reduction,
so
reducing
gradients
needs
to
happen
only
across
model
instances
and
not
across
every
single
parameter.
B
Okay,
so
getting
into
some
a
little
bit
more
detail,
so
in
the
channel
mixing
mlps
in
the
input
space,
we
would
need
to
think
about
how
the
forward
and
backward
paths
work.
So
operations
need
different
communication
patterns
in
forward
and
backward
passes.
So,
for
instance,
a
low
parallel
matrix
multiply
will
not
have
communication
in
the
forward
pass,
but
the
backward
pass
where
we're
now
computing
its
transpose
matrix,
multiply
with
with
a
gradient
that
would
require
an
all
reduce
operation.
B
So
this
is
like
an
example
called
snippet
of
the
mlp
from
our
model.
So
here
our
hidden
dimension
for
the
mlp
is
typically
four
times
the
channels.
So
we
split
that
into
different
gpus,
and
so
the
input
gets
in
the
the
first
function.
Does
nothing
in
the
forward
pass
it
just
copies
the
copies,
the
tensor
and
then,
and
then
the
matrix
multiply.
It
is
computed
using
a
convolution.
B
This
is
now
a
column
parallel
matrix
multiply,
and
so
we
would
have
to
have
an
or
reduce
operation
and
then
finally,
we
return
the
tensor.
But
when
we're
doing
the
backward
pass,
we'll
have
to
be
careful
because,
when
you're
going
back,
this
operation
will
just
pass
the
tensile
through
because
it's
a
low
parallel
matrix
multiply.
But
when
we
come
all
the
way
to
the
top,
we
would
have
to
implement
normal,
reduce
because
now
that
is
a
column
parallel
matrix
multiply.
A
B
So
the
spectral
convolutions
are
an
important
part,
but
these
are
embarrassingly
parallel.
As
I
explained,
there
is
no
specialized
pi
touch
layer.
We
just
use
yield
complex
and
complex
studio,
dodge
fft
functions,
so
the
channels
are
split
across
different
gpus,
so
each
ffts
can
operate
in
a
completely
disjoint
fashion.
There's
also
this
block
diagonal
matrix
multiplied
for
the
mixing
in
the
frequency
space
in
the
channel
dimension,
so
each
block
here
is
once
again
distributed
to
different
gpus.
So
now
you
can
think
of
this
cartoon.
B
B
I
o
that
wants
concurrent
gpu
computation
and
also
to
make
sure
that
the
pre-processing
steps
that
for
us
is
normalization
of
the
images
and
adding
noise
and
stability
that
does
take
place
on
the
gpu.
So
dali
is
an
open
cell
framework.
That
is,
it
actually
is
pretty
flexible
to
turn
on
different
file
types.
So
we
use
hdfi
here,
which
is
a
pretty
common
file
type
for
scientific
applications.
B
B
Secondly,
something
that
really
affected
our
model
panel
scaling
was
too
much
interference
from
the
cpu,
because
we
also
had
this
orthogonal
data
power
scaling,
that's
happening
so
since
cpu
determines
what
kernels
launch
next,
we
do
find
that
sometimes
because
of
context
which
it
can
install
the
gpu
execution.
Kuda
graphs
are
a
common
way
to
take
the
cpu
out
of
the
loop
as
much
as
possible.
B
So
these
graphs,
they
record
a
sequence
of
kernel
launches
and
then
you
replay
it
as
often
as
needed,
and
our
graphs
are
static
because
in
training
is
essentially
the
same
option.
The
same
operation
that's
happening
again
and
again,
so
we
record
the
full
training
process
forward
and
backward
pass
as
a
graph.
B
We
don't
do
it
for
I
o
or
validation,
because
we
we
we,
we
observed
that
it
doesn't
really
cause
a
lot
of
performance
globs,
so
metrics
like
acc,
computation
or
messy
computation
or
certain
model.
Specific
operations
are
a
bunch
of
mathematical
operations.
These
are
all
lightweight
ops,
but
each
of
them
generates
separate
kernel
call.
So
we
just
use
dodge.j
dot
script
to
get
a
slip.
Function
diffuse
all
these
operations
into
larger
kernels.
B
To
reduce
launch
latency,
finally,
we
use
nvidia
inside
compute
to
measure
our
floating
point
operations
per
second,
so
we
have
a
script
that
gives
gives
gives
us
a
list
of
flux,
generating
instructions
that
are
each
multiplied
by
specific
weighting
factors.
It's
like
one
for
multiplier
or
add,
use,
multiply
and
add,
has
a
weight
of
two
cancel
operations,
have
higher
weights,
etc?
B
So
we
did
scaling,
runs
on
three
different
supercomputers
joules
booster
at
unix
super
computing
facility
per
module
at
nurse
and
selena
nvidia,
and
here
we're
measuring
on
the
y-axis.
The
big
flops
per
second
range
and
x-axis
and
number
of
gpus,
and
the
different
colors
represents
different
model.
Parallel
instance
sizes,
so
instant
size
for
mainstage
model
parallelism
is
across
four
gpus.
B
We
observe
really
good
scaling
for
pearl
model
and
saline
all
the
way
to
almost
3000
gpus,
whereas
dual
splitster
tapers
off
at
about
1024
gpus,
possibly
due
to
some
file
system
performance,
stops
in
terms
of
data
parallel
scaling.
We
do
really
well.
So
we
think
of
going
from
two
to
seven
to
two
to
the
gpus
and
focus
on
a
single
model.
Power
instance
that
effectively
boils
down
to
doubling
the
patch
size,
and
so
we
can
see
the
scaling
is
really
good
across
in
all
systems.
B
However,
the
balance
between
model
and
data
parallelism
is
pretty
different
and
we
do
observe
system
dependent
performance
so,
for
instance,
on
pole
model
and
joules.
It
is
we.
We
observe
that
a
model
instance
size,
four
performs
better
than
two,
for
example,
a
focus
on
instant
size.
Two
with
two
to
the
seven
gpus.
B
I
want
to
scale
to
two
to
the
eight
gpus.
I
can
do
that
as
a
data
panel
scaling
by
doubling
the
patch
size,
the
global
batch
size,
or
I
can
keep
that
the
same
and
increase
the
model-
parallelism
to
instance,
size
four,
and
we
observe
that
monopolism
of
four
has
better
performance
than
two.
The
same
is
for
pole,
mother,
but
the
story
is
the
worst
in
saline.
B
Well,
if
you
want
to
go,
for
example,
from
two
to
seven
to
eight
again,
it's
this
size,
four
and
eight
actually
do
worse
than
instant
size
two.
So
there
is
a
higher
performance
gain
when
data
parallelism
is
increased
on
celine,
whereas
it's
the
opposite
for
poor,
modern
jewels,
and
this
is
because
celine
has.
This
is
probably
because
celine
has
balanced
into
a
node
and
internet
bandwidth
and
uses
like
hybrid
envelope
and
mini
band
algorithms,
and
hence
does
not
suffer
as
much
from
large-scale
collectives.
B
B
There's
a
caveat
here
in
that
hybrid
parameters,
we're
dealing
only
at
a
single
scale
and
then
extrapolated,
but
in
principle
models
can
be
fine
to
you
and
to
get
best
performance
at
higher
scales
too.
And
the
bigger
message
here
is
that
faster
time
distributions
can
be
afforded
with
greater
parallelism
and
that
allows
us
to
explore
the
space
of
hyper
parameters
during
much
faster.
B
Okay,
so
with
that,
I
would
want
to
quickly
conclude
this
talk,
so
this
is
just
our
contribution.
Slides
flashed
again,
so
I
talked
about
three
main
challenges
and
some
of
our
contributions
to
overcome
them
so
forecast
net
is
fully
data.
Driven,
shows
really
good
skill,
comparable
to
the
ifs
on
important
variables.
B
And
finally,
I
talked
a
little
bit
about
hybrid
model
and
data,
parallel
data
parallelism
strategies
to
help
scale
out
forecast
net
to
about
4,
000,
gpus
and
moving
forward.
We
are
thinking
about
also
things
like
spatial
parallelism
or
pipeline
parallelism
that
I
didn't
talk
about
today,
interest
of
time,
and
this
is
ultimately
needed
to
enable
this
exascale
weather
and
climate
computing.
B
So
our
main
contribution
here
is
that
this
is
the
current
state
of
the
deep
learning
mates
based
by
the
model,
that
is,
a
scalable
data,
driven,
better
model
surrogate
that
enables
interactivity
at
scale
and
because
of
the
computational
gains.
We
can
now
simulate
a
wider
variety
of
diversity
of
scenarios
that
can
enable
better
decision
making.
B
B
We
are
looking
to
incorporate
the
full
state
vector-
and
this
helps
us
add
physics
constraints
to
a
model
in
order
to
get
physically
consistent
solutions.
Something
that's
extremely
important
that
we
haven't
looked
at
yet
is
observational
data.
We
need
to
see
how
a
model
actually
performs
with
observations.
B
I
introduce
diagnostics
that
show
our
performance
here
and
finally,
some
of
the
other
things
are
like
better
ensembling
strategies.
This
is
also
a
pretty
new
area
of
research
and
currently
forecast
net
is
at
weather
time
scales.
We
would
want
to
also
move
to
climate
scales
and,
what's
very
closely
related
to
observational
data
as
data
assimilation,
so
we
are
reliant
on
models
like
numerical
weather
prediction
models
right
now
for
initial
conditions,
but
we
hope
that
in
the
future
such
models
can
also
be
introduced
into
the
data
assimilation
pipeline.
B
So
with
that,
I
would
like
to
thank
everybody
for
their
attention.
I
do
invite
you
to
check
out
the
preprint
we
have
online
regarding
this
model.
The
scaling
ones
are
submitted
to
garden
bell,
but
due
to
embargo
policies,
we
don't
have
it
up
yet,
but
we
hope
to
have
that
up
to
up
soon
and
we're
also
hoping
to
open
source
the
code
in
the
next
couple
of
weeks.
There
are
some
legal
technicalities
that
we
have
to
get
over.
B
B
C
B
B
What
was
the
motivation
for
choosing
if
architecture,
okay?
So
this
is
actually
a
really
good
question.
There
are
a
lot
of
reasons
to.
There
are
only
intuitive
reasons
to
use
this
architecture,
so
it's
based
on
a
vision
transformers.
So
that
is
the
state-of-the-art
model
for
vision,
tasks
and
the
fully
annual
operator
has
shown
a
lot
of
success
in
simulating
differential
equations,
essentially
learning
continuous
mappings
from
input
to
output
space,
so
it
did
make
sense
to
use
they
have
no
architecture
which
combines
both,
because
this
is
a
vision
task.
B
B
There
are
other
architectures
online
because
before
this
there
was
a
state-of-the-art
model
which
was
a
resnet
which
did
which
had
like
no
down
sampling,
but
none
of
these
approaches
were
also
scalable,
so
from
a
practical
point
of
view,
using
if
no
means
computational
engines
or
ffps
and
matrix
multipliers
and
that
optimized
libraries
to
help
scale
out
this
model
table.
So
I
hope.
A
B
B
B
Do
you
find
that
grid
projection
impacts,
the
accuracy
we
haven't
looked
at
that
yet
so
you
are
like
that
this
is.
We
are
not
looking
at
a
rectangular
projection
of
our
for
inputs,
which
actually
lie
on
a
sphere,
and
we
did
this
mainly
for
computational
reasons.
It
is
just
more
efficient
to
do
ffts.
B
B
What
do
you
mean
by
physics
constraints?
Oh,
so
this
model
is
fully
data-driven,
so
it
takes
inputs
as
data
and
then
learns
to
map
it
to
an
output
space.
That's
the
data
at
the
start
as
a
different
time
step,
but
all
weather
models
have
differential
equations
that
describe
the
underlying
physics,
this
start
off
as
mars
and
energy,
mass
momentum
and
energy
conservation,
and
also
like
hundreds
of
parameterizations
of
different
physical
processes.
B
There's
a
lot
of
work
on
this.
It's
it's
unclear
if
how
they
work
on
real
systems
like
weather,
which
is
a
very
high
resolution
and
high
dimensionality
system,
but
moving
forward.
I
think
we
are
very
interested
in
seeing
how
how
much
gains
we
can
get
by
introducing
these
physical
laws
as
constraints.
C
B
B
Well,
we
are
doing
several
things,
so
first
is
obviously
going
to
higher
resolutions
to
dissolve
the
scales
of
extreme
events
like
storms.
Having
better
ensembles
means
that
we
are
now
capturing
the
tail
end
of
the
pdf
more
accurately
and
then
so
that
could
help
us
in
capturing
extreme
events
better.
B
These
are
all
works
in
the
pipeline,
but,
like
I
said,
this
field
is
pretty
nascent
and
there's
a
lot
of
stuff
to
be
done,
and
I
hope
that
that
sort
of
answers
the
question:
what's
the
throughput
bottleneck,
is
it
communication
bound
yeah
yeah?
So
a
major
complication
here
is
ffts
and,
as
we
scale
out,
it
is
going
to
be
a
the
biggest
drop
in
performance
is
going
to
be
due
to
communication.