►
Description
Presented by:
Jacob Tomlinson
Lead Engineer, Met Office Informatics Lab
The Met Office is a world leading weather and climate centre with the largest operational supercomputer in Europe. With their data growing rapidly every year, they've realized a need to change their approach to analysing it. This talk covers how they are working with a suite of open source tools to solve this problem. They have mixed popular projects, like Jupyter Notebooks, with new data processing libraries (like Dask) and their own bespoke weather analysis tools (like Iris). They are also opening large volumes of data to the public, which through these tools can be manipulated quickly and easily from the comfort of a web browser.
At Satellite 2017, attendees came to exchange cutting edge practices on how to bring together the people and tools you need to build great software. Watch the recordings to find out what GitHub's been working on, and hear how other teams use the GitHub ecosystem to customize the way they work.
We'll be getting together again in San Francisco on October 10th-12th for GitHub Universe. We hope you'll join us for three days filled with the creativity and curiosity of the largest software community in the world.
For more information, go to githubuniverse.com.
A
Right,
great
hello,
everybody
I
think
we
get
started
now.
So
welcome
to
my
talk.
Hopefully
this
is
going
to
be
just
kind
of
a
couple
of
stories
about
Who
I
am
what
I
do,
what
we
do
and
how
we
use
kind
of
open
tools
and
open
data
to
kind
of
make
science
better
in
general
right.
So
my
name
is
Jacob
Tomlinson
I'm
the
lead
engineer
for
the
metals
informatics,
lab
I
like
stuff
like
this,
so
to
kind
of
process.
A
My
whole
talk,
I'm,
not
a
scientist
I'm,
an
engineer
in
a
team
of
scientists,
okay,
so
I'm
hoping
to
get
the
science
right.
But
this
is
this
is
more
of
a
technical
talk.
So
for
those
of
you
who
may
not
know
what
the
Met
Office
is,
the
Met
Office
is
a
UK
government
department.
We
do
weather
forecasting
and
climate
research
for
the
whole
globe
and
the
UK
it's
a
very
old
organization
has
been
around
since
1800s
used
to
be
military
organization.
A
Kind
of
came
out
of
the
Navy
is
now
a
separate
trading
fund
under
UK
government
and
we're
based
on
in
the
southwest
an
annex.
So
a
lot
of
people
know
us
for
stuff
like
this
right.
You
see
the
weather
on
the
TV
and
on
the
internet
and
on
the
apps
and
all
those
all
those
different
things,
but
we
also
you
know,
that's
just
kind
of
like
the
very
surface
of
what
we
do.
A
We
do
all
sorts
of
other
things
in
all
sorts
of
different
industries
from
everything
from
kind
of
telling
wind
farms
which
directions
point
their
windmills
to
telling
airports
how
much
the
icer
to
spray
on
an
aeroplane
so
that
they
you
completely
dice
the
plane
but
use
as
little
de-icer
as
possible
because
aviation,
the
ice
is
very
expensive,
so
you've
got
loads
and
loads
of
products
that
kind
of
huge
range
of
stuff.
But
it's
all
built
off.
A
very
large
data
set
that
we
generate
ourselves
using
a
physical
weather
model.
A
I
specifically
work
in
a
team
called
the
informatics
labs.
So
we
are
we're
kind
of
like
the
R&D
team,
like
the
innovation
area,
we're
a
small
mixed
discipline
team
within
this
organization
of
2,000
people,
there's
10
of
us
that
sit
in
a
room
where
a
mix
of
scientists,
engineers,
designers
and
we're
looking
at
how
we
can
progress
the
organization,
how
we
can
use
different
technology,
different
techniques,
change
our
processes
to
make
faster,
going
forward
and
help
us
kind
of
our
I
mentioned
a
bit
better.
So
we
have
a
very
large
IT
infrastructure
within
the
organization.
A
As
you
can
imagine,
we
stay
away
from
that
completely.
We
act
as
an
external
body,
almost
so
everything
we
deploy
within
our
team
we
put
on
AWS
or
Azure
in
the
cloud
somewhere.
We
then
access
our
data
through
the
kind
of
standard
data
API
that
we
make
available
to
everybody
else.
We
try
and
dogfood
our
own
our
own
systems
as
much
as
we
can
and
anytime
that
we
can't
access
something
that
we
need.
A
A
A
There
is
not
necessarily
useful
and
so
we're
putting
a
lot
of
effort
into
trying
to
find
what
people
find
interesting
and
what
we
can
take
from
that
and
build
a
community
around
I'll
talk
a
bit
more
about
other
other
stuff
that
we
do
that's
a
bit
more,
actually
open
source
rather
than
just
coding
the
open
in
a
minute.
So
I
was
just
going
to
spend
a
bit
of
time
telling
you
about
what
kind
of
data
we
have
and
how
we
generate
it
and
what
we're
trying
to
do
with
it.
A
So
the
kind
of
thing
that
we
do
is
we
run
physical
weather
simulations
on
some
supercomputers
that
we
have
in
our
basement
right.
We've
just
spent
a
decent
chunk
of
money
on
a
next-generation
supercomputer
we've
got
another
building
and
installed
this
one.
It's
a
really
powerful
machine
and
we
use
this
in
a
process
for
predicting
the
future
right.
It's
quite
a
there's,
not
many
businesses
that
can
say
that
their
whole
business
model
is
based
around
predicting
the
future.
So
it's
quite
an
interesting
space
for
us
to
be
in,
but
we
do
this
scientifically
right.
A
So
the
first
thing
we
have
to
do
in
order
to
predict
the
future
of
the
atmosphere.
We
have
to
know
exactly
what
the
atmosphere
looks
like
right
now,
so
we
have
weather
stations
all
over
the
country,
thermometers,
our
radars
and
all
that
kind
of
stuff.
We
collect
everything
for
the
UK,
we've
got
plant
organizations
all
around
the
world.
Every
country
has
some
kind
of
maths
office
that
that
does
weather
forecasting
and
they
collect
all
their
measurements,
and
then
we
all
share.
A
This
is
a
really
nice
collaboration,
there's
a
global
thing
where
everybody
shares
their
information
and
one
of
the
reasons
that
we
have
to
do.
That
is
because
you
can't
actually
run
a
physical
weather
simulation
without
entire
global
coverage,
because
you
have
to
know
if
you
want
to
run
a
local
weather
simulation.
You
have
to
know
what
the
edges
look
like
in
order
to
see
if
there's
any
weather
system
coming
in
from
the
outside.
A
So
if
you
want
to
start
from
scratch,
you
have
to
have
continuous
circuit,
you
can't
have
any
edges,
so
you
have
to
do
a
global
forecast.
This
is
fear.
There
are
no
edges,
that's
great,
but
in
order
to
be
able
to
do
that
global
forecast,
we
have
to
share
that
information
with
everybody
else,
so
we're
ingesting
millions
of
measurements.
You
know
per
day
into
a
big
data
store,
and
these
are
all
point
measurements.
A
These
are
all
little
bits
of
data
temperatures
and
wind
speeds
and
all
kind
of
stuff
that
has
come
from
our
equipment
and
our
partners
equipment
and
that
will
get
loaded
in.
But,
as
you
can
imagine,
there
are
not
evenly
spaced,
there's
often
more
weather
stations
in
highly
populated
areas.
You
know
you
can
imagine
this
is.
We
know
we're
currently
on
an
island,
so
there's
less
coverage
over
the
ocean
areas.
We
get
loads
of
measurements
off
aeroplanes
as
they're
moving
along
and
boats
and
all
that
kind
of
stuff.
A
But
we
have
to
run
this
on
a
supercomputer
right.
We
have
to
run
a
physical
weather
model
that
requires
a
grid,
so
we
have
to
do
a
big
first
step
where
we
take
all
of
those
point
observations
we
interpolate
between
them.
We
do
a
labor
smoothing
and
we
convert
that
into
a
gridded
data
set.
That
explains
what
the
atmosphere
looks
like
right
now.
Once
we've
got
that
model,
we
once
you've
got
that
that
sorry,
these
initial
conditions
for
the
model.
We
then
run
something
that
we
look
after
in-house
with
the
unified
model.
A
It's
a
physical
weather
model
that
predicts
how
that
environmental
system
is
going
to
change
go
into
the
future.
It's
been
a
little
bit
more
about
kind
of
how
it
does
that
in
a
minute,
but
once
we've
run
the
model,
we
get
lots
of
data
coming
out
the
vast
vastly
more
data
coming
out
the
other
end,
and
then
we
do
lots
of
post-processing.
We
have
lots
of
downstream
system
that
pull
the
data
out
of
the
streamers
coming
out
of
the
machine
and
generating
all
the
downstream
product.
A
So,
like
the
weather
map,
I
showed
you
before
that's
one
of
the
products
kind
of
web
interfaces
and
apps
and
things
that
we
provide
to
our
customers.
Those
are
the
products,
those
are
kind
of
more.
You
know.
Smaller
data
sets
that
are
being
derived
from
this
larger
data
set
and
we
end
up
doing
a
lot
of
kind
of
web
development
and
app
development.
That
kind
of
thing
to
visualize
the
data
and
try
and
communicate
it
to
people,
because
you
know
we
don't
want
to
just
dumped
loads
of
raw
data
at
you.
A
We
want
to
you
know
if
you're,
a
business
and
you're
trying
to
do
a
thing.
We
want
to
tell
you
how
to
do
your
thing,
if
you're
a
supermarket
and
you
trying
to
figure
out
what
to
order
look,
you
know
the
weather
is
a
big
factor
in
what
supermarkets
needs
water.
If
it's
going
to
be
hot,
then
they
probably
need
to
order
barbecue
stuff.
A
So
if
it's
on
a
weekend
you
know
that's
their
business,
need
they
don't
actually
care
about
the
weather
they
care
about
what
their
stock
change
is
going
to
look
like,
so
we
try
and
build
products
that
really
help
them
to
do
that.
The
actual
machine
itself,
it's
a
nice-nice
machine.
We
technically
we
have
three
machines.
A
Sometimes
we
group
them
together
because
it
makes
the
stats
look
better
right,
but
we
have
three
discrete
machines
within
our
data
centers
which,
according
to
the
notes
on
my
slide,
I
can't
run
behind
they
are,
but
they
are
the
11th
38th
and
39th
fastest
machines
in
the
world.
As
of
whenever
that
was
these
numbers,
the
front
I
think
up
some
around
January,
so
we
probably
be
lower
down
that
list
right
now:
everybody's
building,
new
supercomputers,
all
the
time,
the
actual
machine
weighs
140
tons.
A
It
can
calculate
at
23,
platters
Lots
and
it
uses
13
megawatts
of
electricity
right.
That's
a
vast
amount
of
electricity,
so
the
data
that
comes
out
of
that
is
big
right.
We
have
lots
and
lots
of
data
coming
out
of
it
and
it's
hard
for
us
to
move
that
data
around
once.
It
comes
out
of
the
machine,
we
put
it
into
archives
and
that's
where
it
sits
as
it's
coming
out
of
the
machine
into
the
archive
we're
deriving
all
of
our
downstream
products.
A
But
then
we've
got
this
big
archive
full
of
stuff
and
because
we
are
part
of
the
government,
because
we
have
this
kind
of
our
climbing
responsibilities
and
this
historical
responsibility
we
try
and
keep
as
much
as
we
can.
We
keep
a
ridiculous
amount
of
data,
so
I
was
going
to
do
a
quick
hands
up
thing.
Who
here
is
worrying
about
dealing
with
data
in
your
organization?
Who
here
has
kind
of
data
issue?
A
Okay,
see
a
few
hands
who
here
is
dealing
with
kind
of
more
than
about
a
terabyte
of
data
within
your
data
set,
okay
can
see
a
few
hands
anybody
dealing
with
more
than
a
pester
byte
of
data.
Yeah
another
hand
cool
excellent.
So
hopefully
you
understand
that
in
the
realms
of
working
with
these
amounts
of
data,
we
have
these
big
issues.
You've
got
all
these
massive
organizations
like
Google
and
Facebook
that
are,
at
the
kind
of
exabyte
scale,
they're
dealing
with
ridiculous
vast
amounts
of
things,
but
we're
dealing
a
petabyte
scale
situation.
That's
exciting,.
A
Wow
I
keep
going
for
a
minute
until
they
reappear,
but
we
have
currently
in
our
archives.
As
of
January
of
some
some
figures
from
January,
we've
got
about
40
petabytes
of
data,
just
sitting
in
our
archive
at
the
moment
that
the
close
of
the
the
rate
that
we've
run
the
machines-
that's
increasing
it
around
200
terabytes
per
day.
So
we
expect
by
2020,
we'll
be
close
to
300
petabytes
of
data.
I
can
see.
Oh
there
we
go
yeah
cool,
so
we're
going
around
300
pets.
We
will
have
around
300
petabytes
of
data
in
2020.
A
You
know
we're
generating
all
these
products
with
people
as
we're
as
we're
building
all
of
this
data,
but
there's
so
much
more
value
in
there
that
I'm
sure
we
can
be
be
deriving,
but
we
also
have
a
responsibility
to
make
this
data
open
right.
We're
publicly
funded.
This
machine
is
the
property
of
the
taxpayer.
We
have
a
responsibility
to
make
as
much
of
our
data
open
as
physically
possible,
but
I
mean.
Does
anybody
here
want
want
to
just
download
the
archive
right
if
you've
got
anywhere?
A
To
put
it
like,
you
know,
we
can
try
and
give
it
to
you,
but
the
actual
bandwidth
cost
for
us
is
going
to
be
enormous.
The
storage
costs
we
use
can
be
enormous,
so
it's
kind
of
like
it
we're
in
a
weird
situation
where
we
want
to
give
you
the
data,
if
you
want
it,
but
there's
just
so
much
of
it.
So
we
are
part
of
the
open
data
government
plan
we
have
loads
of
data,
sets
that
we
make
available.
A
Some
of
them
are
REST
API,
where
you
can
consume
bits
and
pieces,
it's
generally
small
amounts
of
information.
It's
usually
some
of
the
downstream
products
that
we've
derived.
So
one
of
our
more
popular
data
sets
is
the
kind
of
a
five
to
seven
day
weather
forecast,
the
same
kind
of
thing
that
you
would
see
in
the
app
whatever
so
you
can
put
in
a
location,
and
it
gives
you
back
temperatures
every
hour
going
forward
to
the
future.
That
kind
of
data
set.
A
We've
also
got
a
load
of
kind
of
static
data
sets
to
do
with
climate
and
to
do
with
kind
of
historical
weather.
We've
got
nice
data
set
which
will
give
you
information
going
back
30
years,
but
it's
very,
very
small
in
you
know
it's
been
compressed
massively.
It's
been
reduced
massively
think
our
dataset
is
kind
of
in
the
tens
of
gigabytes.
So
we've
had
to
take
all
of
this
data
and
pull
out
what
we
think
will
be
useful
to
people,
but
it's
not
always
easy
for
us
to
figure
out
what
that
is.
A
A
A
lot
of
organizations
are
collecting
vast
amounts
of
data,
if
you
think
about
Facebook
or
something
they're
collecting
data
from
all
of
their
users,
whereas
we're
taking,
in
a
relatively
small
amount
of
data,
blasting
out
loads
and
loads
of
compute
time
to
generate
massive
amounts
of
data
and
then
we're
trying
to
consume
that
back
down
into
a
product
against
we've
got
this
big
kind
of
expansion
phase,
so
that
does
make
us
a
bit
unusual.
In
that
sense,
one
of
the
things
about
that
data
that
makes
us
unusual
is
what
that
data
looks
like
right.
A
We
are
trying
to
deal
with
all
sorts
of
big
data
processing
technologies.
You
know,
big
data
is
one
of
those
funky
buzzwords
at
the
moment.
We're
like
it's
a
bit
a
bit
too
hand
wavy,
but
we
do
need
to
deal
with
things
on
an
on
a
large
scale,
but
a
lot
of
data
sets
that
people
seem
to
be
dealing
with
our
tables
right.
Your
data
will
fit
into
a
table
somehow
you
can.
You
know
whether
there's
some
joint
in
there
and
you've
got
a
few
kinds
of
tables
linked
together.
A
So
I
just
show
you
a
few
dimensions
here
right,
if
you
imagine
you
just
imagine
the
UK
or
Europe
or
something
you've
got
last
food
and
longitude
right,
we've
broken
it
up
into
a
grid,
so
you
can
see
this
latitude
and
longitude,
but
the
weather
model
that
we
run
is
three
dimensional,
so
you've
got
altitude
as
well.
So
you've
got
these
three
kinds
of
dimensions.
You
could
probably
store
that
in
some
kind
of
relational
database,
but
then
we're
going
forward
in
time.
A
So,
as
we
run
the
model
we
start
at
now
and
we're
going
to
go
an
hour
to
the
future
and
an
hour
into
the
future
and
an
hour
into
the
future,
so
we've
got
these
kind
of
cubes
of
weather
for
each
hour
going
into
the
future
for
that
time.
So
within
that
cube,
there's
lots
of
different
variables
as
all
the
things
you
care
about
wind
speed,
humidity,
pressure,
probability
of
rain,
all
those
different
bits
and
pieces,
that's
a
whole
nother
dimension.
A
Added
to
that,
then
we
do
this
thing
called
ensemble
forecasting
so
with
we
know
that
within
what
we
do,
there
are
margins
of
error
in
the
physical
calculations
that
we
know
right
running.
You
know
our
model
literally
runs
physics,
equations,
their
physics,
equations
that
you
can
kind
of
test
out
on
a
you
know
in
a
bench
at
school
kind
of
physics,
equations
when
you
couple
them
all
together,
you
start
getting
complex
weather
systems
coming
out
of
it,
but
there's
such
complex
systems
that
they
kind
of
follow
the
laws
of
chaos.
Theory.
A
So
within
clear
theory,
if
you
tweak
something
very
small,
you
can
get
a
wildly
different
output
within
that
system,
and
we
know
that
our
margins
of
error
is
a
big
enough,
that
if
we
move
slightly
within
a
margin
of
error,
we'll
get
a
completely
different
forecast
or
a
slightly
different
forecast
to
what's
going
on
now.
So
what
we
do
is
when
we
initialize
our
model.
A
We
give
a
little
bit
of
flutter
within
those
these
margins
of
error
that
we're
comfortable
with,
and
then
we
get
this
spread
of
uncertainty
into
the
future,
and
then
we
can
see
whether
there's
any
clusters,
whether
there's
lots
of
them
agreeing
with
each
other.
Then
we
can
give
strong
confident
that
something
is
going
to
happen.
A
We
can
give
much
better
probabilities,
but
it
means
that
for
every
single
one
of
these
forecasts,
we're
then
running
at
thirty
times,
we've
got
all
of
these
different
you've
got
each
our
future
going
out
with,
and
we've
got
thirty
versions
of
that
hour
as
well.
So
you've
got
this
vast
spread
and
then
you
imagine,
we
update
this
data
set
every
hour
right.
We
run
this
model
every
hour,
so
some
of
our
models,
some
of
them
we've-
run
every
six
hours
because
they're
just
massive,
and
we
come
from
them
any
more
frequently
in
that.
A
But
it
means
that
our
data
has
got
all
these
funky
dimensions
and
there's
different
ways
of
storing
that
and
different
ways
of
handling
that.
But
one
of
the
things
that
we
struggle
with
is
because
of
the
fact
that
we've
run
the
model
every
hour
most
of
our
customers,
most
of
the
people
that
are
consuming
our
data.
They
only
want
the
most
up-to-date
forecast.
If
you're
looking
at
something
that's
going
to
happen
in
three
or
four
days
time,
you
want
the
very
latest
run.
You
don't
want
to
run
that
we
did
yesterday.
A
A
That
starts
and
that's
go,
go
okay.
So
as
an
organization,
we've
spent
a
long
time
thinking
about
how
to
store
this
data-
and
this
is
just
one
of
the
things
that
we
take
into
account
when
we
store
the
data,
is
ways
of
storing
two-dimensional
three-dimensional
data,
and
this
pattern
is
called
a
dead
order
car.
A
So
that's
called
a
quadtree
and
for
three
dimensions
you
can
get
art
trees
as
well,
but
there
is
a
storing
your
data
like
this
makes
it
easier
to
access
that,
especially
when
you're
kind
of
seeking
on
desk
you're
going
left
and
right
because
of
the
Zed's
within
the
data.
If
you
want
to
find
data,
that's
close
to
it.
It's
stored
close
to
it
on
the
disk
as
well.
A
A
Where
am
I
oh,
that's
better?
Okay,
so
it
does
sound
like
we're
unusual.
We
store
things
in
an
unusual
way.
We
process
things
in
an
unusual
way,
but
we
found
increasingly
that
we're
not
unusual.
You
know
lots
of
people
who
look
at
our
data
and
think
wow
you're
weird.
You
know
I
speak
to
a
lot
of
people
at
conferences
and
I
start
talking
about
our
problem
and
they're
like
oh.
A
That
sounds
really
interesting,
but
I
have
no
idea
how
I
can
help,
but
we
found
lots
and
lots
of
people
that
are
working
on
on
similar
things,
not
necessarily
to
do
with
weather
forecasting
or
environmental
science
or
meteorology,
or
anything
like
that,
but
people
that
are
dealing
with
these
large
multi
dimensional
data
sets
because
at
a
generating
data
with
supercomputers,
you
know
quite
a
lot
of
product
design.
These
days
is
done
with
a
supercomputer.
A
You
can
do
physical
simulations
on
the
supercomputer
medical
imaging
is
also
a
really
interesting
area
for
us,
because
the
developers
in
that
sphere
of
technology
are
thinking
a
lot
about.
Visualizing
three-dimensional
data
how
they
can
visualize
different
data
sets,
and
so
one
of
the
things
were.
The
guys
in
our
team
did
recently
was
to
oh,
no,
that's
not
done
again
yeah.
So
one
of
the
guys
in
our
team
took
a
medical
scanning
source
project
that
was
for
looking
at
brain
scans
and
put
a
hurricane
into
it.
A
So
we
just
been
a
little
bit
of
time,
hacking
around
getting
our
data
into
a
format
that
their
their
system
would
like,
and
they
managed
to.
You
know,
draw
up
some
quite
nice.
This
was
just
a
screenshot
of
the
web.
Browser
I
took
the
browser
right,
and
this
was
a
browser.
You
could
manipulate
the
hurricane.
A
So
we
are
generating
all
of
this
data
we're
trying
to
collaborate
with
people
on
how
we
can
win
work
on
this
data,
but,
as
I
said
at
the
beginning,
with
deriving
lots
of
things
as
its
coming
out
of
the
machine
right,
we're
deriving
all
these
products-
and
we
know
that
we're
looking
for
these
products.
We've
kind
of
spent
a
lot
of
time,
thinking
about
what
the
customers
going
to
want.
A
When
we
work
out
what
data
we
need
to
get
that
thing,
and
so
as
it
as
the
feed
is
coming
out
of
a
supercomputer,
we
can
pull
out
the
bits
of
data
that
we
need
and
generate
the
product,
but
that
research
that
insight
into
what
the
customers
needs
has
come
out
of
kind
of
people,
thinking
about
a
problem
and
then
working
through
it
and
then
doing
some
exploratory
research
to
figure
out
whether
they
can
get
that
out
of
the
data.
But
doing
that
kind
of
thing
is
a
bit
like
creating
a
train
right.
A
You
know
where
you're
going
to
start.
You
know
where
you
going
to
finish.
You
want
to
do
it
reliably.
You
want
to
do
it
every
hour,
so
you
spend
a
lot
of
time,
making
something
that's
very
robust,
very
resilient,
but
extremely
inflexible.
You
can't
just
say:
oh
I
want
to
go
over
here
now
right,
a
train
goes
from
one
place
to
another
place,
and
you
can't
you
can't
change
that
and
for
exploring
these
new
these
new
products
for
exploring
these
new
things
and
these
new
advances
to
science
within
our
data.
A
We
need
something
a
bit
more.
Like
an
auditorium
vehicle,
something
where
we
could
go
off-road,
something
that
we
can
still
go
100
miles
an
hour
if
we've
got
an
open
straight
in
front
of
us,
but
something
that
we
can
slow,
write
down
and
go
over
all
the
rocky
ground.
I
appreciate
us
a
bit
for
tortured
analogy,
so
apologies
for
that,
but
the
only
way
that
we
can
do
that
is
by
collaborating
with
other
people
and
figuring
out
what
other
tools
we
can
use.
So
this
is
just
a
range
of
stuff
that
we
use
for
analyzing.
A
A
So
a
common
workflow
for
our
staff
is
to
get
some
data
of
our
archives.
Put
it
onto
some
kind
of
very
high-powered
workstation
write
a
bit
of
Python,
run
the
place
and
that
script
might
take
24
hours
right,
so
they
spend
a
bit
of
time
figuring
out
what
that
scripture
look
like
they
run
it.
It
starts
chugging
through
the
data
eventually
runs
out
in
memory.
It
crashes,
they
would've
gone
home
right
because
it's
like
been
four
hours.
They
come
back
in
the
next
day.
A
They
found
the
script,
that's
crashed,
and
so
they
start
it
up
again.
They
fix
the
bug
and
they
go
again
in
the
iterate
a
few
times.
You
know
now
might
be
a
week
later.
They've
got
their
script
working
perfectly,
but
they
potentially
have
lost
the
train
of
thought.
It's.
You
know
they're
trying
to
figure
out
a
thing
with
this
with
this
bit
of
Python
and
they
spent
a
lot
of
time
doing
computer
science
and
we
we
hire
them
to
do
physicists,
an
environmental
scientist.
A
So
we
can't
you
know
we
can't
expect
them
to
work
in
this
way.
Really.
This
is
kind
of
akin
to
doing
a
batch
job
right.
There
they're
working
in
this
very
batch
paradigm,
and
it
also
involves
a
lot
of
copying
data
around
right.
If
they've
all
got
individual
workstations,
they
might
have
really
high-powered
UNIX
workstation
under
their
desks,
but
it
means
they've
got
a
coffee
loads
of
data
at
the
archive
put
loads
of
spinning
disks
in
these
machines.
It
slows
everything
way
down.
A
So
this
is
I
find
this
quite
a
common
term
within
within
software
development,
especially
you
know,
I'm
sure,
you've
all
heard
the
adage
of
developers
are
lazy
right.
You
write
code
so
that
you
don't
have
to
do
a
thing,
but
we
use
this
to
describe
some
of
the
ways
that
we
handle
the
data
internally.
A
So
I
want
to
talk
about
a
few
different,
a
few
different
things
that
allow
us
to
work
lazily
and
what
I
mean
by
lazily.
In
this
context,
is
we
want
to
leave
the
data
where
it
is
as
long
as
we
can
and
we
don't
want
to
do
any
calculations
until
the
very
last
minute
we
want
to
know
exactly
what
we're
going
to
do
with
it,
but
we
almost
want
to
appear
is
if
we're
doing
lots,
but
but
do
it
as
little
as
possible.
A
So
I'm
going
to
just
tell
you
about
a
few
libraries
that
really
really
help
us
with
this.
It's
not
gone
connects
again.
There
we
go
so
Jupiter.
If
you
haven't
seen,
Jupiter
juice
is
awesome
right.
We
use
this
as
a
really
really
good
kind
of
first
stab
at
giving
our
scientists,
particularly
both
our
engineers,
a
good
environment,
start
working
from
so
jupiter
notebooks
are
a
web-based
application.
You
have
usually
a
kernel,
a
Python
kernel,
running
somewhere,
but
it's
pretty
language
agnostic.
A
These
days
you
can
run
pretty
much
any
any
kind
of
interpreter
kernel
somewhere,
and
you
have
this
kind
of
look
a
bit
like
a
Google
Doc
right.
It's
a
web
page.
You
have
this
kind
of
concepts
of
cells,
so
you
can
have
code
cells
where
you
write
Python
and
when
you
hold
shift
and
press
ENTER,
it
executes
the
placing
on
the
remote
kernel
and
it
brings
you
back
whatever
the
response
from
that
would
be.
A
So
if
you
printed
something
it
would
print
underneath
to
sell
if
you
use
MATLAB
or
some
kind
of
library
to
plot
an
image,
it
puts
that
image
underneath
the
cell.
Some
of
that
you
can
change
the
cell
type
if
you
want
to
contribute
to
markdown.
If
I
want
to
make
some
notes,
if
I
want
to
write
about
what
I'm
doing
I
don't
have
to
do
that
in
code
comments,
I
can
write
that
in
markdown
you
get
some
kind
of
inline
documentation
and
it's
got
lots
of
nice
fancy
widgets
and
things
like
that.
So
you
can.
A
If
you
don't
know
what
you're,
what
you're
kind
of
wanting
to
explore?
You
could
just
not
specify
how
to
plot
something
and
I'll
give
you
a
slider,
which
is
great
so
in
our
situation
of
ensemble
members
right
if
we've
got
like
30
different
cubes,
and
we
want
to
kind
of
have
a
look
at
all
of
them.
We
could
just
say
plus
a
cube
but
not
tell
it
which
one
and
it
gives
us
a
slider.
It
gives
us
a
plot
and
a
slider.
A
The
other
neat
thing
about
it
is
when
you
save
the
notebook
all
of
the
output
of
the
code,
all
of
the
images
you've
generated
go
into
the
file
and
then,
if
you
commit
the
Clapp
file
and
push
it
to
github,
github
give
you
a
nice
preview
version
of
that
as
well.
So
it's
quite
nice
for
us
sharing
things
with
each
other
within
our
workflow,
so
one
of
the
libraries
I
want
to
talk
about
is
IRS.
This
is
something
that
we
maintain
internally.
A
This
is
an
open-source
library,
so
you
can
find
it
on
github
site
or
various
titles
like
one
of
our
internal
organizations
and
this
as
I
said
before,
we
store
our
data
in
weird
file
formats,
using
kind
of
dead
alder
cards
and
different
things
like
that.
To
get
really
good
compression
on
our
data,
make
it
quite
possible
for
us
and
this
library
helps.
You
have
manage
that
right.
You
can
imagine
not
that
many
people
use
this
file
format.
It's
quite
a
specific
bespoke
scientific
file
format.
A
It's
just
going
to
now
go
okay,
I
only
care
about
these
bits
of
these
files
and
I
can
do
some
operations
on
that.
Do.
Some
averages
means
and
different
things
like
that
that
society
people
do
and
because
I've
picked
out
my
bit
of
London
when
it
runs
the
calculation
on
those
it
only
loads
the
chunks
in
from
the
files
that
cover
London.
So
it
allows
it
to
be
lazy
in
the
fact
that
it's
not
loading
loads
of
data
and
they're
discarding
loads
of
data.
A
A
A
It
allows
you
to
be
quite
iterative
with
your
Python
when
you're
developing
something
like
that,
but
that's
that's
great,
but
when
I
run
that
on
my
desktop,
if
I
want
to
do
a
calculation
and
all
that
data
once
I've
loaded
it
into
iris
and
I,
do
do
a
calculation,
it
will
pull
all
those
different
bits
of
data
into
my
machine
and
run
it
in
a
single
thread
within
Python
right,
which
is
not
ideal
in
terms
of
compute.
You
know,
I
might
have
a
really
powerful
me.
Workstation
I
might
be
running
this
on
a
mainframe.
A
I
might
be
doing
it
on
something
big
and
bulky,
but
I'm
still
stuck
in
this
kind
of
serial
serial
world,
where,
although
I'm
lazy
about
what
data
I
load,
I
do
have
to
load
it
all
sequentially.
So
another
library
that
we're
really
into
at
the
moment
is
called
tasks.
If
any
of
you
have
heard
of
Sparkle
tools
like
that,
it's
kind
of
a
similar
idea,
this
one's
written
in
Python,
which
we
have
some
interesting,
pragmatic
issues
around
the
fact
that
we
have
lots
of
science
staff
who
they
write,
Python
as
a
by-product
of
their
job.
A
Right
they're
here
to
do
physics
they're
here
to
do
environmental
science.
They
write
Python
in
order
to
do
that,
but
they
don't
care
that
much
about
Python
they're,
not
into
it.
You
know,
they're,
not
software
developers
they're
not
really
interested
in
that
kind
of,
and
so
they
need
to
know
the
advantages
of
doing
a
thing
before
they're
actually
going
to
do
it.
They're
gonna
do
something
because
it
sounds
cool
they'll
do
something
because
it's
useful
to
them.
A
So
if
we
give
them
spark
and
they
do
something
wrong
and
it
gives
them
a
javis
at
race,
they'll
go
we'll
Java
I'm
a
pleasant
person
I'm,
not
even
really
a
real
Python
person,
I'm
a
kind
of
Python
person.
I.
Definitely
don't
look
at
a
Java
spectra,
so
it's
quite
a
big
parent
to
them
using
that.
So
one
of
the
the
factors
that
went
into
us
picking
tasks
is
that
if
it
goes
wrong,
it
gives
them
a
Python
stacktrace,
at
the
very
least
at
least
they're
slightly
more
than
their
comfort
zone.
A
But
what
desk
allows
you
to
do?
Is
map
out
your
computer.
You
installed
ask
when
loads
of
machines
in
a
big
cluster
and
allows
you
to
do
kind
of
standard.
Mapreduce
jobs
allows
you
to
do
quite
small,
microbats,
interactive
jobs.
So
if
I'm
going
to
load
in
a
cube
of
London,
say
and
I
want
the
mean
over
London
I
can
distribute
all
of
that
data
until
those
machines
that
can
load
little.
Each
machine
can
load
a
little
chunk
of
that
into
memory.
A
It
can
do
some
means
locally,
and
then
one
of
the
one
of
the
machines
can
do
an
aggregation.
You
know
map
produce
that
kind
of
standard
thing,
and
it's
great
that's
really
cool
for
doing
that
kind
of
thing,
but
not
a
lot
of
our
workloads
work
very
well
in
a
MapReduce
paradigm,
because
there's
lots
of
cross
communication
between
different
things.
Different
cells
need
to
talk
to
each
other
different
things
need
to
communicate.
So
that's
one!
That's
why
we
supercomputers
to
run
a
lot
of
things.
A
You've
got
that
really
good
good
bandwidth
within
the
within
the
machines
and
low
latency,
but
we
might
have
graphs
that
look
more
like
this
right,
so
you've
got
some
steps
that
can
be
paralyzed
quite
well
and
then
it
needs
to
come
together
again
and
then
I
like
split
out
again
and
then
come
back
together.
The
desk
allows
us
to
to
run
this
across
a
big
cluster
machines
and
optimize
where
we
can-
and
it
does
some
very
clever
optimizations
in
terms
of
being
lazy
again.
A
So
if
I
keep
this
graph,
if
you
look
at
each
of
the
circles
like
that
is
a
discrete
operation
that
might
be
a
line
of
Python
or
something,
and
so
it's
queued
up
this
graph.
But
when
I'm
at
the
point
of
seeing
this,
it
hasn't
actually
run.
Anything
I've
got
my
end
value
at
the
top
and
it
won't
actually
do
anything
with
that
in
value
until
I
find
print
it
until
I
try
and
actually
look
at
it.
A
So
I
try
and
realize
what
that
value
is
and
then
all
of
a
sudden
of
the
computes
praise
off
and
it
try
to
run
through
this
whole
graph,
and
it
does
all
the
calculations
and
one
of
the
reasons
it
does.
That
is
because,
if
I'm
at
this
number
at
the
top
I
might
then
say
actually
I
don't
care
about
London
I
only
about
shortage
right
and
I'm
going
to
cut
down
into
its
much
smaller
area,
and
then
the
compute
graph
can
optimize
by
going
well.
Actually
that
means
I,
don't
need
just
the
graphs
I.
A
Don't
need
this
bit
of
the
graph
I.
Don't
need
to
run
up
through
these
different
columns
and
I
can
paralyze
these
better
and
it's
much
more
intelligent
about
how
it
can
run
those
things.
So
it
doesn't
realize
until
the
very
very
end,
because
it's
trying
to
reduce
the
amount
of
calculation
and,
if
I'm,
just
trying
to
plot
like
a
PNG
to
send
to
somebody
and
it's
small,
and
it
knows
that
it
only
needs
low
resolution
data.
It
can
do
some
averaging
and
down
scaling
and
that
kind
of
stuff
that's
kind
of
the
future.
A
Where
we're
going
with
this,
so
this
is
a
bit
of
an
alley
diagram.
This
is
something
that
we've
thrown
together
recently.
This
is
a
project
we've
been
working
on.
It's
called
Jadis,
it's
a
collection
of
this
stuff
right.
This
is
how
we've
deployed
this
stuff
for
ourselves.
As
I
said,
we
deploy
lots
of
stuff
on
AWS.
So
recently
we
took
away
the
data,
put
it
into
s3,
which
is
really
cool.
We
are
looking
at
making
our
data
open
as
well.
A
So
if
you
are
interested
come
give
me
a
shout,
but
you
can
see
on
this
diagram
that
we've
got
a
couple
of
auto
scaling
groups
like
there.
The
interesting
bits
right
one
here
and
one
here
and
the
top
auto
scaling
group
gives
each
developer
each
scientist
their
own
notebook
environment
right.
So
you
log
in
through
the
Oh
off
at
the
top,
it
spins
up
a
box
or
it
starts
container
somewhere
or
pack
you
in
some
into
some
place
within
that
stack,
and
you
get
your
own
Network
environment.
A
I
can
make
these
decisions
and
that's
awesome.
That
means
I
can
get.
My
work
done
a
thousand
times
faster,
but
it
also
means
that
I
can
spend
a
thousand
times
more
money.
So
the
organization,
if
I'm,
massively
scaling
this
crushes
all
the
time
it
builds
up
a
big
bill.
So
that's
quite
an
interesting
area
for
us
is
looking
at
how
we
can
kind
of
keep
on
top
of
that
and
maybe
give
people
different
budget
things
so
kind
of
just
to
close
up
what
I've
been
talking
about.
A
I
hope
this
has
been
interesting
for
you,
but
I
wanted
to
share
that.
We
generate
lots
of
data
and
we
try
and
make
it
as
available
as
possible.
You
lots
of
stuff
on
that
on
the
data,
but
the
UK
government
data
website.
So
please
have
a
look
there.
If
you're
interested
there's
a
service
on
our
website,
the
data
point
we
keep
adding
stuff
to
that.
So
please
have
a
look
and
we
use
and
contribute
to
as
many
open-source
tools
as
possible,
because
it's
only
by
collaborating
with
other
people
that
we
can.