►
Description
In this talk, we will present how we are implementing the OS-Climate data commons platform based on a data mesh architecture to make data accessible, available, discoverable, and interoperable across various development streams, while supporting strict compliance requirements around data access and regulatory disclosures. We will explain how the OS-Climate platform leverages open source tools including KubeFlow, Trino, Jupyter, Elyra pipelines and OpenShift to build maintainable and collaborative pipelines, and federate heterogeneous data sources into a controlled, common data resource for our community of climate data scientists and financial sector stakeholders.
In this talk, we will present how we are implementing the OS-Climate data commons platform based on a data mesh architecture to make data accessible, available, discoverable, and interoperable across various development streams while supporting strict compliance requirements around data access and regulatory disclosures. We will explain how the OS-Climate platform leverages open source tools including KubeFlow, Trino, Jupyter, Elyra pipelines and OpenShift to build maintainable and collaborative pipelines, and federate heterogeneous data sources into a controlled, common data resource for our community of climate data scientists and financial sector stakeholders.
A
A
And,
of
course
the
the
focus
of
the
presentation
is
going
to
be
on
the
platform
architecture,
try
to
explain
how
we
are
approaching
the
climate
data
problem
and
why
it's
a
novel
approach
to
do
so
and
eric
will
spend
a
fair
bit
of
time.
You
know
showing,
with
some
demo
some
of
the
platform
capabilities.
A
So
first
I
actually
work
in
the
financial
services
team
and
quite
often,
I'm
asked
the
question:
why
is
financial
services
critical
to
the
climate
issue,
I'm
just
back
from
glasgow
where
the
26th
climate
conference
happened?
You
know
in
the
past
two
weeks
and
actually
one
big
change,
an
important
change
in
the
way
the
climate
problem
is
now
being
approached
is
the
involvement
of
the
corporate
sector
into
the
resolution
of
the
climate
problem.
A
So
what's
interesting
is
for
financial
institutions.
Climate
in
itself
is
actually
a
challenge,
an
important
challenge,
because
investment
that
these
institutions
are
making
translate
to
potentially
climate
risk.
Let's
say
you
are
financing.
You
know
energy
production
and
you
start
to
lend
money
for
the
next
25
to
30
years.
It
becomes
extremely
important
to
understand
how
climate
change
could
affect
the
business
that
you
are
investing
in
and
obviously
their
ability
to
service
the
loan
that
you
are
making
and
that
means
being
able
to
understand.
A
A
A
The
first
one
is
in
the
past,
a
lot
of
financial
institutions
or
any
sort
of
ngo
or
non-for-profit
that
work
in
the
field
of
climate
data
tend
to
try
to
solve
problems
on
their
own.
We
believe
that
using
an
open
source
approach
and
an
open
source
community
to
actually
collaborate
to
have
multiple
stakeholders,
invest
their
intellectual
property
and
share
the
cost
and
burden
is
actually
a
more
efficient
alternative,
and,
with
this
open
source
approach,
we
basically
look
at
building
mainly
two
capabilities.
A
A
Why
is
it
important
to
adopt
a
community
based
open
source
approach
is
mostly
because
we
want
to
be
efficient
at
solving
problems
and
share
the
solution
and
obviously
try
to
solve
the
climate
data
problem
faster.
If
everybody
is
kind
of
pulling
the
rope
in
the
same
direction
running
this
open
source
means
we
have
a
shared
and
structured
governance
around
the
project,
so
the
key
industry
player
can
agree
on
what
main
data
challenge
we
have
and
what
is
the
focus
of
this
engagement?
We
also
create
a
very
high
trust
collaboration
structure
between
player.
A
This
includes
some
commercial
data
player
who
are
actually
sharing
their
data
on
the
kind
of
what
we
call
pre-competitive
basis,
which
means
that
they
share
the
data
with
us
to
be
able
to
train
a
new.
You
know
data
models
and
analytics
model
and
then
spread
them
across
the
industry
and,
of
course,
from
a
licensing
point
of
view,
open
source.
A
Now,
let's
focus
on
the
climate
data
challenges.
There
are
mainly
three
main
challenges
that
we
are
trying
to
solve.
The
first
one
is
about
data
availability.
Interestingly,
there
are
really
two
challenges
here.
One
is
there's
actually
a
huge
volume
of
data
that
is
generated
by
the
industry
and
the
data
is
extremely
distributed,
so
it's
very
hard
to
actually
connect
data
to
each
other.
At
the
same
time,
although
data
volume
is
high,
certain
region
or
certain
industry
have
a
lot
of
either
insufficient
granularity
or
limited
coverage,
so
there
can
be
missing
data.
A
So
one
example
is:
you
could
have
public
utility,
disclosing
energy
production
and
co2
emission
for
energy
providers
in
united
states,
but
there
is
no
such
data
available
for
china,
so
you
may
be
missing
some
geographic
or
industry
data.
Now,
even
if
you
have
the
data,
another
problem
is
because
it's
distributed
it's
extremely
hard
to
compare
across
different
sorts
because
they
may
not
follow
the
same
disclosure
standard
or
they
may
not
follow
the
same
data
format.
A
And
last
but
not
least,
another
problem
is
data
reliability,
as
you
feed
the
data
for
this
model.
You
want
to
make
sure
that
you
understand
how
the
data
has
been
produced.
So
one
issue
typically,
is:
there
are
a
lot
of
community
out
there
that
build
their
own
model
generate
data,
but
nobody
is
able
to
understand
transparently
how
the
data
that
has
been
generated
could
be
trusted.
A
Our
approach
is
to
use
a
self-service
data
management
platform,
so
what
we
are
doing
really
is
to
make
a
collection
of
well
integrated,
standardized,
open
source
tools
for
the
community
to
ingest
data
process
data
and
distribute
data.
So
the
idea
here
is
to
standardize
and
make
this
tooling
easily
available
for
data
scientists,
so
typically
people
who
don't
have
necessarily
the
knowledge
of
how
to
install
the
infrastructure
behind
it,
and
then
they
can
start
to
build
those
data
integration
on
their
own.
A
The
last
dimension
is
reliability,
and
so
here
this
is
interesting
because,
typically
in
distributed
system,
you
get
a
lot
of
agility,
but
then
the
problem
can
be.
How
do
you
ensure
that
people
follow
a
certain
set
of
standards
and
also
respect
certain
governance
and
security
around
data?
So
here
we
use
a
federated
governance
approach,
which
pretty
much
is
twofold.
A
The
first
is
we
treat
a
data
very
much
like
code,
so
we
use
a
processes
that
have
been
used
in
the
open
source
community
for
years,
so
they
are
extremely
well
mature
process,
well
proven
to
manage
and
version
code.
We
are
extending
this
process
to
the
data
itself,
so
data
sets
are
managed
like
code.
They
are
transparently
generated.
A
So
this
helps
us
to
turn
what
I
call
a
data
mess
into
a
data
mesh.
We
have
an
extreme
number
of
heterogeneous
data
source
and
data
contributor
and
instead
of
building
yet
another
source
or
another
database.
What
we
do
is
we
are
actually
standardizing
the
way
that
people
now
consume
the
data.
We
are
federating
across
data
source
and
we
are
standardizing
the
tooling
and
the
approach
to
ingest
process
and
distribute
data.
A
So
I
will
now
leave
the
next
section
to
eric
to
present
on
how
we
are
actually
using
open
source
tooling.
To
do
this
and
from
the
red
hat
perspective,
we
are
really
building
on
an
engagement
which
is
actually
an
open
source
project,
but
it's
called
open
data
hub
and
open
data
hub
is
really
an
ai
platform
powered
by
open
source.
So
we
are
literally
selecting
multiple
upstream
projects
in
the
open
source
community
and
putting
together
a
platform
that
can
easily
be
deployed
by
data
scientists
across
the
globe.
B
Sorry
about
that,
I
was
expecting
vincent
to
drive
so
riffing
off
what
vincent
said.
There's
a
couple
things
to
say
about
open
data
hub.
One
important
thing
is
it's
an
open
source
downstream
and
a
very
typical
red,
hattish,
open
source
downstream
way,
in
this
case
it's
the
downstream
of
kubeflow
and
we're
using
it
internally
and
externally,
as
a
reference
architecture
for
machine
learning,
workflows
on
kubernetes
and,
lastly,
it's
federated
which
to
say
that
the
the
actual
components
are
selectable
and
mixable,
and
also
just
by
virtue
of
kubernetes
and
openshift
platform.
B
It's
very
easy
to
actually
add
other
components
to
the
system.
If
you
want
to
the
coverage
of
the
available
tools
is
actually
quite
good.
You
know
in
both
dimensions
in
terms
of
process
all
the
way
from
business
goals
through
data
preparation
model,
development
deployment
and
then
finally
monitoring
and
then
along
the
axis
of
personas.
It
similarly
covers
use
cases
for
people.
You
know
business
leaders,
data
engineering,
data
science,
all
the
way
through,
like
app
development
into
operations.
B
I
emphasize
in
this
diagram
a
few
of
the
tools
you're
going
to
see
later
today,
a
superset
for
dashboarding
and
jupiter
for
both
exploratory
data
science
and
also
pipelining,
and
then
trino,
which
is
a
great
example
of
the
advantages
of
a
federated
architecture.
Trino
is
not
actually
a
formalized
component
of
open
data
hub,
but
it's
being
deployed
in
quite
a
lot
of
our
use
cases,
including
the
data
commons
for
osclimate.
B
So
again,
open
data
hub
is,
you
know,
leveraging
everything,
that's
good
about
open
source
community
driven
upstreams
in
this
case,
in
the
artificial
intelligence
machine
learning
workspaces,
we
are
maintaining
the
actual
cluster
using
another
red
hat,
affiliated
project
called
operate
first
and
I'll.
Show
you
a
little
bit
more
about
what
that
means
later.
B
A
Both
notebook
can
be
stitched
together
into
end-to-end
data
pipeline.
You
know,
but
that
are
using
lyra
gonna,
think
I'm
on
the
wrong
screen.
Okay,
that
should
be
better
sorry
for
that
and,
however,
you
know
we
will
see
later
when,
when
I
cover
this
topic,
is
there
still
a
number
of
components
that
we
are
trying
to
to
add
to
the
platform
to
provide
better
manageability
across
the
community,
and
these
components
are
really
around
the
management
of
experiment
and
data
experiment,
as
well
as
the
metadata
associated
with
it.
A
A
So
that's
one
kind
of
first
problem
here
that
we
have
to
deal
with
now.
The
challenge,
though,
in
this
is
a
number
of
these,
like
data
sets
that
we
consume
may
be
public,
but
others
may
be
competitive.
Data
sets
so
provided
by
a
commercial
data
provider
which
may
be
okay,
with
the
data
being
used
to
improve
and
tune
certain
data
or
risk
model,
but
they're,
absolutely
not
okay,
with
just
distributing
their
data
totally
in
the
open,
so
they
want
security
and
compliance
to
be
managed.
A
So
I've
explained
it
before.
What
we're
trying
to
to
to
do
here
is
to
create
a
single
layer
of
compliance
and
security
across
the
pipeline.
So
what
this
diagram
shows
is
basically
a
kind
of
very
high
level.
You
know
skeleton
of
what
the
data
pipeline
typically
could
look
like.
You
have
some
primary
source
of
data,
which
is
nice,
but
not
necessarily
usable
as
it
is
by
the
community,
and
then
you
try
to
merge
it
with
some
kind
of
reference
data
set
that
will
help
people
make
sense
of
the
data.
A
So
one
example
of
this
is:
let's
say
you
have
carbon
emission
of.
You
know
factories
in
united
states.
Now
you
don't
necessarily
know
which
company
actually
owns
this
factory
and
how
to
actually
aggregate
all
this
data
into
a
way
that
is
meaningful
for,
say,
an
investor
which
would
be
looking
at
those
emissions.
For
specific.
A
A
So
the
first
thing
that
we're
trying
to
do
here
is
to
build
a
catalog
service
that
will
keep
track
of
data
set
version
data
set
metadata
and
also
what
we
call
pipeline
metadata,
which
is
every
time
you
generate
a
new
data
set
with
a
bit
of
code.
You
want
to
keep
the
code
version
and
you
want
to
associate
the
data
that
is
produced
with
the
latest
version
of
your
code,
so
people
understand
how
the
data
set
is
actually
being
built,
as
you
do
so,
as
you
can
imagine,
a
lot
of
data
sets
get
produced
now.
A
So
here
this
is
where
a
single
layer
of
data
federation
and
security
based
on
trino
actually
allows
us
to
build
a
single,
a
logic
of
data
access
across
multiple
data
sources.
So,
right
now
we
are
using
trino
to
do
exactly
this
to
make
sure
that
add
the
data
element
level.
We
have
consistent
security
at
any
point
of
the
pipeline.
So
let's
say
that
some
of
this
data
of
this
primary
data
set
is
actually
commercial
and
is
respected
in
access.
A
We
can
manage
and
make
sure
that
the
access
is
consistent
on
the
primary
data
set
layer,
so
the
source
data,
but
also
when
the
data
is
distributed.
So
that
is
extremely
important.
So
this
is
what
we
have
today,
but
something
that
we
are
still
working
on
is
kind
of
going
forward.
We
want
to
be
able
to
manage
access
at
the
data
set,
but
also
at
the
metadata,
so
you
know,
taking
certain
like
specific
datasets
itself
could
be
accessible
to
public.
A
You
know
one
example
of
this
is:
maybe
you
have
a
data
set
of
data
that
is
not
up
to
date.
Maybe
it's
data
from
six
months
ago.
You
want
to
make
it
available
to
the
public,
but
for
the
latest
data
you
need
to
have
like
a
commercial
licensing
to
do
so.
So
you
will
use
metadata
tagging
to
basically
tag
your
latest
metadata
set
and
make
it
subject
to
licensing.
A
How
do
we
make
this
happen
is
again
leveraging
open
source
project
to
build
and
integrate
security
and
governance
across
three
layer,
which
is
the
physical
data
layer.
So
how
is
we
store
data
and
serve
data?
The
virtual
layer,
which
is
how
do
we
consume
data?
How
do
we
query
data
in
the
pipeline
and
then
the
access
layer,
which
is
once
data,
is
ready
to
be
distributed?
A
A
You
know
update
and
read
of
the
data
across
big
data
set,
and
the
underlying
format
is
actually
a
parquet
format
which
makes
it
very
scalable
to
to
just
add
you
know,
gigabytes
or
petabytes
of
data
potentially
for
specific
data
set,
and
on
top
of
this
we
are
using
trino
as
a
distributed.
Sql
engine
so
trino
helps
us
to
federate
a
query
of
data
across
a
very
high
number
of
data
source.
A
The
work
that
we
have
not
done
yet
is
what
you
see
above
building
a
metadata
platform
and
some
data
security
layer
we've.
We
are
already
experimenting
with
linkedin
data,
our
project,
as
well
as
apache
ranger,
to
build
those
layer
and
a
lot
of
the
work
that
we
are
going
to
do
in
the
next
12
month
or
so
is
going
to
be
on
those
two
layers.
A
For
data
access
management,
as
you
can
see
today,
we
have
actually
a
very
granular
access
capability
already
with
the
trino
layer
and
rick
will
demonstrate
this
in
the
demo.
But
some
of
the
things
that
we
are
missing
is
is
kind
of
how
to
tie
the
security
to
the
metadata
management
layer
and
obviously
how
to
generate
catalog
automatically.
A
B
Excellent,
so
I'm
just
going
to
briefly
give
you
a
high-level
view
of
what
I'm
about
to
try
and
demonstrate
here,
as
I
mentioned
before,
we're
intending
to
make
heavy
use
of
eliara's
capability
of
supporting
jupiter
environments
for
both
actual
data
exploration,
data,
science
and
prototyping,
but
also
to
then
take
the
same
notebooks
and
be
able
to
run
them
in
repeatable
pipelines
and
so
sort
of
bridge.
The
gap
between
actual
data,
science
and
devops
and
along
the
way
I'll
show
how
we're
using
github
and
some
of
red
hat's
continuous
integration
bots.
B
And
then,
similarly,
you
know
from
the
slide
before
you
know:
lyra
we're
hoping
to
represent
actual
repeatable
etl
pipelines
ingesting
from
disparate
data
sources,
including
things
like
object,
store,
then
exposing
these
as
actual
sql
databases
and
from
sql.
You
know
it's
easy
to
consume
through
jupyter
or
tools
like
superset
to
do
actual
analysis.
B
Helping
to
onboard
different
community
members
who,
as
vincent
mentioned,
are
not
really
conversing
in
open
source
and
so
being
able
to
give
them
a
pathway
to
private
repos
on
github
and
then
eventually
to
public
repos
has
been
extremely
successful
and
actually
very
happy
with
how
much
work
now
is
being
done
completely
out
in
the
open
with
the
different
organization
members
and
similarly,
I've
had
a
lot
of
success.
Unifying
identity
management
through
github.
B
B
B
Here's
where
we've
actually
begun
to
already
interact
upstream,
so
we
want
to
unify
trino's
group
definitions
with
our
github
team
identities,
and
so
I've
been
prototyping,
an
actual
trino
plugin
that
does
group
provision.
So
it's
actually
extremely
helpful
and
we're
going
to
hopefully
test
this
out
and
actually
get
it
upstream
directly
onto
the
main
freeno
repository.
B
I
mentioned
the
operate
first
management
and
so
we're
actually
managing
the
entire
cluster
configuration
through
github,
and
so
here's
just
one
example
where
I
was
actually
resyncing,
our
trino
user
groups
with
the
github
teams
and
it
just
took
the
form
of
a
pull
request
and
in
standard
open
source
fashion.
We
discussed
it
and
talked
about
a
few
alternatives,
and
then
it
was
eventually
merged.
B
So
I
wanted
to
talk
about.
You
know
how
we're
actually
leveraging
the
platform,
especially
sort
of
like
one
of
the
central
obviously
features
is
the
jupiter
environment.
So
obviously
you
can
do
pip
installs,
but
I
already
have
these
installed
on
the
image
we're
managing
actual
credential
storage
with
the
python.m.
B
B
So
now,
let's
look
at
some
data
here
we
have
some
implied
temperature
rise
or
itr
data
that
I'm
loading
straight
off.
Our
trinomial
database
and
its
job
is
to
map
different
investments.
So
these
are
different
companies
in
the
energy
space
or
non-energy
space
as
well,
that
you
know
are
associated
with
implied
temperature
rise,
success
and
so
over
here
in
the
far
right,
there
is
an
achieved
reduction
targets,
and
so
there
are
various
points
of
progress
achieving
their
targets
and
each
of
these
is
associated
with
a
standard
identification
number,
an
isin
or
isin
number.
B
So
we
can
make
use
of
these
keys
to
do
some
interesting
things
now.
A
completely
separate
data
set
contributed
by
the
glyph
organization,
provides
some
key
data
services
like
mapping
these
ison
investment
identifications
to
the
legal
entities
that
actually
issued
them,
and
so
the
illegal
entity,
identification
or
lei
number
we're
planning
to
leverage
heavily
as
a
community
resource
for
getting
people
to
push
their
non-standard
ways
of
tracking
investments
into
standard
ones,
and
I
see
that
time
is
passing,
so
we
also
have
a
table
that
can
map
direct
issuers,
as
you
saw
above
to
ultimate
current.
B
So
this
is
to
track
corporations
and
corporation
subsidiary
ownerships,
so
that
you
can
actually
do
a
better
job
tracking,
true
climate
impact
up
through
company
ownership
chains,
and
so
what?
If
we
wanted
to
take
these
three
data
sets
which
come
from
two
completely
different
organization
members
and
get
some
kind
of
value
out
of
it.
So
here.
C
B
The
parents
are
all
empty,
which
I
say
these
are
all
top-level
corporations,
and
so
the
main,
the
main
value
here
is
that
I've
gone
from
standard
investment,
eisen
numbers
to
a
standardized
l,
legal
identification
number
to
get
the
achieved
reduction
and,
of
course,
there's
nothing
special
about
the
sql.
What's
really
special
here
is
that
we've
already
been
able
to
federate
data
sets
from
two
different
community
members
to
support.
You
know,
value,
add
using
standard
data,
science
techniques.
B
Yes,
so
I
got
all
seven
records
so
now
I've
done
some
data
science
in
our
notebook.
Again.
What
I
told
you
is
the
lyra
jupiter
environment
can
do
a
lot
of
fancy
things.
The
fanciest
thing
you
can
do
is
you
can
take
these
notebooks,
so
I
took
here's
that
notebook.
I
just
showed
you
copied
into
a
pipeline
editor.
B
B
I
can
look
at
its
log
output,
see
what
it's
doing
so
I
want
to
stress
that
it's
actually
running
the
notebook.
I
just
showed
you
with
exploratory
data
science
and
all,
and
of
course
you
could
remove
some
of
that
stuff
to
make
things
go
faster,
but
the
main
you
know
the
main
messages.
It
literally
gives
you
seamless
transition
from
data
science
notebooks
into
repeatable
pipelines,
and
it
should
shortly
give
me
a
message
that
it
succeeded.
Yes,
the
pipeline
ran
so
now.
B
C
B
A
B
B
Shame
on
you!
I
shouldn't
say
that
this
is
I'm
sure,
they're
doing
fine.
This
data
is
just
demo
old
data,
but
anyway
there
you
can
see
sort
of
like
the
entire
pipeline
from
exploratory
data
science
to
repeatably,
executable
pipelines
through
to
actual
end
user
dashboarding
and
charting
all
on
the
same
open
source
based
platform.
B
B
Oh,
I
want
to
show
the
actual
image,
so
the
container
image
I
used
to
run
that
pipeline
was
built,
I'm
also
off
of
the
os
climate
repositories
using
red
hat's.
Continuous
integration
build
robots.
So
here
you
can
see.
I
basically
filed
a
github
issue.
Say:
hey.
Could
you
please
build
me?
The
latest
image,
it's
version
0.1.1
and
the
bots
went
off
and
built
that
and
put
it
up
here
on
the
clay
repository.
So
you
can
see
here
it
was
built
yesterday.
I
think,
and.