►
Description
In this talk, Tristan Nixon, a Solutions Architect at Databricks and Ricardo Portilla, Lead Solutions Architect at Databricks, will demonstrate how data teams can leverage an open-source package tempo (available in Python and Scala) to advance time series use cases with Delta Lake and Spark.
In particular, we will show you how resampling to AS OF joins, and descriptive analytics of up to millions of time series can be done in parallel using a simple interface.
Speakers:
Tristan Nixon is a Solution Architect at Databricks. Tristan has been working in Data-science and ML engineering for over 15 years, in industries from Education to Telecoms and Chemical Manufacturing. He joined Databricks about a year ago where he acts as an SME for time series and Natural Language Processing (NLP).
Ricardo Portilla is a Solutions Architect at Databricks. Ricardo works with data teams to put data engineering, data analytics, and data science use cases into production. He as been at Databricks for ~3 years helping customers with use cases in all verticals and previously worked in the financial industry for 7 years. Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
B
All
good
cool
so
welcome
everyone
thanks
for
joining
us
if
you're
on
coming
from
youtube
and
I'm
just
gonna
check
linkedin
really
quickly
to
make
sure
everything
looks
good
there,
but
I'd
love
for
those
joining
us
from
youtube
and
linkedin.
If,
please,
let
us
know
where
you're
joining
us
from
we'd
love
to
love
to.
B
B
B
B
B
B
B
All
right,
so,
thank
you,
everyone
for
joining
us.
My
name
is
karen
baisa
here,
I'm
the
community
developer
community
program
manager
at
databricks,
and
today's
session
is
about
tempo,
distributed
time
series
analysis
with
apache
spark
and
delta
lake,
and
we
have
ricardo
and
tristan
here
from
databricks.
B
So
if
you're
joining
us
on
youtube
or
linkedin
here
is
the
link
to
join
our
data
plus
ai
online
meetup
group.
So
this
is
where
we
announce
all
of
our
events
and
all
of
our
our
upcoming
sessions.
So
I
would
love
for
you
to
check
us
out
there
and
join
the
group,
so
you'll
be
updated
and
notified
of
upcoming
sessions
and
you
can
join
join
the
zoom
zoom
session
if
you'd
like
and
answer
or
ask
questions
of
the
presenters
as
we
go
along
and
then
for
those
joining
us
on
zoom.
B
B
B
And
then
so
one
thing
I
wanted
to
mention
is
that
you
know
we
last
year
at
data
and
ai
summit
in
europe,
we
started
some
lightning
talks.
Some
community
lightning
talks,
and
I
wanted
to
carry
that
over
to
the
online
meetup
group.
So
this
is
kind
of
a
call
for
for
lightning
talks.
If
anybody
is
interested
in
submitting
your
project
so
we're
we're
asking
for
submissions
to
present
a
10
to
15
minute
lightning
talk
so
far.
B
I
kind
of
have
two
proposed
session
dates
so
march,
30th
and
april
27th
and
just
kind
of
depending
on
the
lightning
talk
submissions
that
we
get
we'll
kind
of
figure
out
what
what
dates
work
the
best
but
we'd
love
to
hear
from
the
community
and
hear
what
you
all
are
working
on.
So
topic
ideas
are,
are
not
limited
to,
but
you
know:
apache
spark,
delta
lake
koalas
ml
flow
and
we're
really
open
to
a
variety
of
things.
B
So
we'd
love
to
see
what
you're
working
on
and
I'd
love
to
to
get
those
lightning
talks
set
up.
So
I'll
drop
a
link
in
in
all
of
the
chat
spaces
for
you
to
kind
of
check
out
some
more
information
on
that
and
submit
your
project
and
then
so
just
before
I
kick
it
off
to
our
presenters,
ricardo
and
tristan.
I'm
going
to
stop
sharing
my
screen
because
I
just
have
a
quick
poll,
so
everyone
joining
us
in
zoom.
I
have
a
quick
poll
that
I'd
love
for
you
to
answer.
B
So,
given
that
we'd
like
to
get
some
lightning
talk,
submissions,
I'm
interested
in
just
hearing
from
you
who
would
be
interested
in
giving
a
lightning
talk
and
the
survey
is
not
anonymous.
So
I'll.
Have
your
your
contact
details
so
I'll,
send
that
that
that
google
form
out
to
you
directly
so
cool.
So
I
have
some
some
submissions.
B
B
So
we
have
a
lot
of
folks
saying:
they'd
love
to
attend
lightning,
the
community
lighting
talk
session,
and
we
have
a
few
that
said:
they'd
love
to
submit
a
project
cool
looks
like
almost
everyone
voted
in
zoom.
So
thank
you
for
that.
So
I'm
just
gonna
go
click
here
and
end
the
poll
and
I
guess
just
for
fun.
We
can
share
results
for
everyone
on
zoom.
Well,
it
looks
like
everybody's
really
excited
about
joining
the
session.
So
that's
awesome.
A
Thanks
tristan
awesome,
so
thanks
for
joining
everybody,
I'm
ricardo
and
tristan
will
introduce
himself
as
well,
so
we're
part
of
a
team
that
created
a
distributed
time
series
package
here
at
databricks,
and
so
we're
going
to
be
talking
about
that
today.
Some
motivations
on
you
know
why
we
created
this
as
well
as
some
of
the
functionality
in
a
demo
as
well,
so
excited
to
go
through
this.
A
He
helped
out
a
lot
with
the
initial
implementations
of
things
like
as
I've
joined,
tristan
who
joins
us
today
as
a
solutions
architect
as
well
scala
guru,
standards,
enforcer,
you
know,
has
done
an
amazing
job
of
making
sure
that
things
are
up
to
standards
on
unit
test
coverage.
Things
like
that
which
are
important
if
you
want
to
use
this
work
and
then
myself
as
well
solutions
architect
at
databricks.
A
So,
for
example,
if
you're
in
finance
oftentimes
you're
looking
at
not
one
ticker
or
symbol
worth
of
the
time
series
for
each
day,
so
something
like
a
price
per
day,
but
you're,
often
looking
at
potentially
hundreds
of
thousands
of
symbols
or
tickers,
and
each
one
of
them
isn't
really
a
daily
data
point.
The
index
is
more
a
time
stamp
which
can
be
completely
irregular.
It
doesn't
have
to
be
every
second
or
every
minute,
so
this
can
get
quite
big
and
all
of
those
different
time
series
are
in
the
same
data
set.
A
The
same
thing
applies
if
you're,
looking
at
things
like
energy
consumption,
where
you
want
to
look
at
forecasting
say
for
like
a
specific
node
in
a
grid.
Again,
you
you
likely
don't
have
one
specific
time
series
you're
looking
at
it's
all
of
them
at
the
same
time,
so
you
have
to
have
a
really
good,
robust
engine
for
processing
those
and
prepping
those
and
lastly,
the
the
other
common
one
that
we
see
very
often
is
a
retail
cpg
use
case,
so
basically
forecasting
sales
per
skewer
per
store.
A
A
And
so
just
to
reiterate
that
point,
you
know
time
series
is
so
difficult
because,
as
you're
looking
at
all
these
individual
time
series
within
a
much
larger
data
set,
which
can
be
billions
of
data
points
or
can
be,
you
know,
even
in
the
hundreds
of
gigabytes
of
data,
this
becomes
a
problem
traditional
sql
joins.
Sometimes
don't
perform
very
well
when
you're
overlapping
time
series.
A
So
if
I
want
to
tell
you
know
when
an
event
in
in
one
data
set,
if
I'm
doing
a
self
join
or
to
another
time
series
data
set,
the
sequel
way
to
express,
it
is
usually
something
something
like
an
interval
join
and
those
tend
to
kind
of
produce
cartesians
and
give
a
lot
of
problems
once
you
start
to
do
this
at
scale
slide
great.
So
what
we
did
was
we.
A
So
things
like
time
series
joins
as
moving
averages:
simple
moving
average
exponential
resampling,
as
well
as
some
delta
lake.
Optimization
these
all
these
things
are
baked
in,
so
that
there's
a
simpler
api
to
do
this
manipulation,
and
that's
really
kind
of
I
would
say
the
number
one
goal
that
we
were
looking
for
is:
how
can
we
make
you
know
data
engineers
or
data
scientists?
A
Analysts
lives
easier
by
kind
of
being,
making
a
simpler
api,
so,
instead
of
maybe
200
lines
of
sql
or
500
lines
of
sql
or
even
200
lines
of
python
just
provide
a
few
different
lines
to
do.
Common
operations
like
these
and
the
data
that
we're
seeing
that
this
can
be
applied
to
is
massive,
so
it
can
be
anything
from
iot
specific
data
sets
if
you've
got
device.
Data
sets
accelerometer,
which
we
have
an
example
of
alternative
data.
All
of
these
different
data
sets
are
typically
indexed
by
time
and
sometimes
it's
fairly
granular.
A
So
a
library
like
tempo
will
help
a
lot
for
your
processing,
which
eventually
converts
all
of
your
data
to
nice,
curated
reports
and
predictions
that
you
can
use
to
surface
to
either
internal
business
stakeholders,
or
just
externally
too
next
slide
awesome.
So
this
is
a
reference
architecture
again
to
give
context
into
where
this
package
would
typically
be
applied.
It's
you
know,
there's
a
whole
lot
of
data
system
left.
A
lot
of
them
are
a
little
bit
more
financial
financial
services
focus,
but
really
it
applies
to
a
lot
of
other
different
data
sets.
A
Anything
structured,
unstructured,
there's
a
whole
lot
of
methods
to
kind
of
get
these
into
what
we
call
a
database
kind
of
brand
of
the
lake
house
so
best
of
both
worlds
of
you
know
a
data
warehouse
and
the
data
lake,
so
you're
seeing
delta
lake
in
the
data
enrichment
portion
lend
a
lot
of
performance
optimization
and
where
tempo
helps
is
it
sits.
On
top
of
your
your
cloud
storage,
you
can
basically
run
these
manipulations
on
that
storage.
A
It
doesn't
have
to
be
restricted
to
delta,
but
we
recommend
it
for
many
reasons,
including
asset
semantics,
or
transactions
and
performance,
and
then,
once
that
data
is
prepped,
you
can
basically
apply
your
machine
learning,
algorithms
or
statistical
methods.
So
just
have
a
few
different
libraries
here:
xg
boost
pytorch
tensorflow,
which
eventually
feed
into
really
nice
fully
featured
data,
sets
that
you
can
analyze
in
any
bi
tool.
So
there's
a
few
listed
here
or
our
new
sql
analytics
preview
feature
as
well.
C
C
You
can
start
by
thinking
about
how
you
might
represent
such
data
in
a
conventional
spark
data
frame
right,
so
tabular
data.
Think
of
you
know
every
row
being
like
a
single
observation
against
a
single
time
series.
But
of
course
we
want
to
represent
multiple
time
series,
potentially
thousands
of
them
millions,
maybe
and
all
organizing
the
the
the
observations
and
working
with
them
in
time.
So
the
first
thing,
of
course,
is
that
you're
gonna
want
you
know
some
kind
of
a
temporal
index,
a
time
series
column
within
tempo.
That
could
be.
C
But
then,
of
course,
we
need
to
also
separate
out
each
logical
time
series
from
the
others
right,
so
you
can
have
some
set
of
partitioning
columns
which
are
used
to
identify.
You
know
which
which
observations
correspond
to
which
logical
time
series
right.
So
in
this
example,
you
know
this
is
kind
of
like
an
iot
example.
Perhaps
different
device
identifiers
represent
these
different
time
series
in
a
financial
use
case
that
might
these
might
be
the
symbols
for
different
assets.
They
sort
of
separate.
C
You
know
which
rows
correspond
to
which
logical
time
series
and
then,
of
course,
the
time
temporal
index.
The
time
series
column
will
order
those
in
time
within
each
one.
So
those
sort
of
form
the
structural
columns
like
it,
gives
the
structure
to
the
temporal
data
frame
and
then
the
rest
are
the
observations
you
know
so
different
measures
or
qualities
of
the
of
that
observation,
and
specifically,
of
course,
what
we
really
care
about.
C
You
know
starting
off,
you
know
we
have
a
tsdf
wrapper
object,
which
is
really
you
know
it's
it's
a
wrapper
on
top
of
that
underlying
spark
data
frame.
So
you
can,
you
know,
pass
that
into
us
and
then
give
us
pass
that
into
a
constructor
to
create
a
tsdf
object,
and
then
you
know
indicating
which
is
the
time
series
column.
What
are
the
partitioning
columns
from
there?
C
We
can
start
to
get
build
up
that
view
of
the
structure
and
the
the
different
measures
and
observations
that
you
have
within
that
within
your
time
series
data
from
there
we
can
go
into
what's
called
an
as
of
join,
which
is
really
a
way
of
joining
two
data
frames
together.
Temporal
data
frames
together
where
they
may
not
have
exact
matches
on
certain
keys,
but
you
what
you
want
to
do
is
you
want
to
represent.
C
You
know
the
the
records
from
one
set.
You
know,
as
of
the
latest,
the
latest
observation
corresponding
to
records
in
the
other
one,
so
thinking
thinking
of
it.
Logically
right,
you
can
imagine
you
know
two
different
sets
of
events
which
you
know
are
related
to
each
other,
but
coming
in
and
on
different
time
scales,
you
could
think
of
you.
C
Could
one
potential
way
of
doing
that
is
just
put
them
all
together
into
a
single
data
frame,
organized
by
event,
type
and
and
the
time
stamp,
but
you'll
see
that
you
know
you've
got.
You
know
different
different
observations,
different
values
for
different
rows,
because
they're
fundamentally
coming
from
different
sources,
then
you
could
then
collapse
that
down
so
that
you
have
one
set
of
common
observations
in
this
case.
C
It's
driven
off
these
trades
here
and
then
we're
bringing
in
the
quotes
values
as
of
the
time
so
the
the
quote
that
applies
the
most
recent
quote
that
applied
to
that
observation
based
on
the
course
overlapping
timestamps,
but
doing
this
in
this
way
like
actually
executing
it.
This
way
can
be
very
inefficient
right,
so
you're
talking
about
you,
know,
shuffling
and
sorting
of
large
amount
of
records
to
do
it
this
way.
C
Instead,
what
you
can
do
is
use
the
temporal
logic
and
and
windowing
and
spark
to
do
to
execute
this
in
a
much
more
efficient
way
under
the
hood,
and
so
that's
effectively
how
you
you
know
we
provide
that
as
of
joint
functionality
and
you're
going
to
get
massive
speedups
executing
it
in
this
way
versus
executing
it
in
a
different
way.
The
so
again,
you
know
looking
at
matching
up
timestamps
joining
different
data
frames.
C
Data
sets
together
when
the
timestamps
are
not
exactly
the
same,
but
trying
to
create
one
consolidated
view
of
the
data.
So
really
what
you're
doing
is
you're
saying
for
each
time
stamp
in
the
left.
Data
frame
find
the
last
observed
one
in
the
right
data
frame
and
here's
an
example.
You
know
we've
got
these
two
different
tsdf
objects
that
we've
created
from
two
different
data
frames,
and
then
it's
as
simple
as
just
saying
you
know
the
one
as
I've
joined
with
the
other.
C
This
data
there's
a
describe
function
that
sort
of
expands
on
the
existing
described
functionality
in
spark
data
frames,
there's
some
functionality
to
do
resampling,
so
you
can
resample
at
different
time
scales
and
then
various
rolling
averages,
including
simple
moving
average
exponential
moving
average
volume
weighted
as
well
as
you
know,
preparing
data
for
machine
learning,
in
particular,
creating
look-back
vectors
and
I'll.
Give
you
some
examples
of
what
that
looks
like
so.
C
Weighted
average
price
is
a
common
function
in
finance,
where
you
want
to
get
a
measure
of,
what's
the
sort
of
average
price
weighted
by
the
volume
of
trades
again,
you
know
it's,
you
know
just
you
know,
creating
a
simple,
a
function
that
allows
you
to
execute
that
really
easily,
once
you've
defined
the
data
frame.
The
time
series
data
frame
look
back.
Vectors
are
very
useful
when
setting
up
feature
vectors
for
machine
learning
models,
particularly
if
you're
going
to
be
building.
A
Yeah
thanks
tristan
so
yeah,
the
the
what
I
wanted
to
highlight
here
is
within
this
package.
We
have
a
number
of
these
different
transformations
and
joins
that
we've
discussed
so
far.
But
one
thing
we
wanted
to
make
sure
we
incorporate
as
well
is
some
of
the
the
delta
lake
features
to
make
access
of
time
series
data
sets
a
lot
quicker
and
a
lot
easier,
so
we'll
go
through
the
just
a
little
bit
of
intro.
First
next
slide:
okay,
perfect!
A
So
there's
a
you
know,
one
of
the
the
main
benefits
of
delta
lake
is
that
they're
it's
easy
to
compact
files.
So
sometimes,
if
you're
using
spark
and
parquet,
you
may
end
up
writing
lots
of
different
tiny
lots
of
tiny
files
to
your
cloud
storage,
and
so
what
we
tried
to
do
within
delta
lake
was
just
make
this
a
little
bit
simpler.
So
just
give
a
one
line,
you
know
sql
command
to
optimize
your
table
and
what
that
will
do
is
it'll.
A
Do
some
bin
packing
compact,
your
files,
so
that
when
you
query
your
your
data,
your
queries
come
back
a
lot
faster.
You
don't
have
to
manage
or
hard
code.
Any
of
the
the
number
of
partitions
within
your
your
spark
job
we've
actually
introduced
something
called
auto
optimize,
which
does
a
couple
of
things
it.
A
It
basically
coalesces
together
a
lot
of
the
data
in
memory
before
it
writes
data
out
to
your
delta
lake
table,
but
it
will
also
run
an
optimized
job
after
that
to
make
sure
that
the
data
is
properly
sized
on
disk
as
well
from
the
perspective
of
tempo.
We
wanted
to
basically
take
advantage
of
the
data
that
tristan
talked
about,
so
we
know
that
you
know
you,
as
a
user,
have
specified
the
partition
columns
as
well
as
the
the
event
timestamp,
for
example.
A
A
Okay,
great
so,
as
I
mentioned,
you
know,
we
have
advanced
knowledge
since,
if
you're
using
tempo,
you've
already
specified
the
partition,
columns
and
your
event
timestamp,
since
we
have
that
information,
we
can
take
advantage
of
this
order.
Feature
so
z.
Order
at
a
high
level
is
basically
excuse
me.
It
just
gives
you
a
way
to
specify
something
of
a
pseudo
index.
A
A
really
common
example
would
be,
you
know,
maybe
an
event
type,
a
more
granular
key
in
your
data
set,
something
like
a
user
id
or
a
store
id
where
it's
a
really
common.
Predicate
and
it'll
basically
allow
us
to
perform
a
lot
of
data
skipping
as
you're
querying
your
data,
and
the
whole
idea
is
to
call
like
locate
related
information
in
the
same
set
of
files
so
that,
when
you're
looking
for
a
particular
store
id,
a
particular
ticker
to
a
particular
device
id
we
can
skip
up
to.
A
You
know
90
percent
of
the
data
by
just
checking
the
the
delta
statistics
and
avoiding
a
lot
of
files.
So
it's
very
good.
For
that
reason,
we
actually
added
some
functionality
with
respect
to
bloom
filters.
That's
for,
like
that's,
going
to
be
better
for
if
you're,
just
looking
for
one
record
or
like
a
couple
specific
records
that
will
work
very
effectively
here,
we're
more
just
looking
to
optimize
query
patterns
such
as
my
time
is
between
you
know:
10,
30
and
10
35
and
skip
files
that
way.
A
I
think
we
talked
about
some
good
choices
for
z-order
columns
again
high
cardinality
used
very
often
in
query
predicates,
and
traditionally
you
know
we
wouldn't
suggest
using
any
anything
more
than
three
columns.
We
introduced
a
different
optimization
based
on
hilbert
facility
curves
and
that
actually
does
a
little
bit
better.
The
more
columns
you
add.
So,
if
you
have
you
know,
maybe
like
three
or
five
columns,
then
this
optimization,
we've
introduced
in
tempo
will
actually
work
pretty
well
next
slide.
A
Yeah,
absolutely
thanks
tristan
so
for
we
bloom,
filter
indexes
actually
are
supported
by
by
delta
lake.
It's
a
separate
setting
that
you
can
use
and
then
specify
some
of
those
parameters
to
to
have
bloom
filtering
enabled
for
a
specific
table.
So
that
would
be
a
different
pattern
like
I
said,
for
we're.
We're
just
trying
to
support
time
range
style
queries
for
this,
so
we
just
use
standards
ordering
not
the
filtering.
A
Okay,
great
so
we
talked
about
the
background
of
like
the
this
is
the
functionality
available
in
delta
lake.
What
we've
done
in
tempo
again
is
just
given
a
write
method,
I'll
review
it
when
we
go
over
the
python
api,
but
essentially
all
it's
going
to
do
is
you
say
timeseriesdataframe.right
give
it
a
table
name,
it
will
take
care
of.
You
know
the
spark
syntax
it'll
take
care
of
actually
creating
a
surrogate
or
look
at
the
event,
time
specific,
double
format
and
then
include
these
columns
in
the
optimization
columns.
A
Like
I
said
it
does
use
the
hilbert
optimization
for
z
ordering
and
the
reason
that
we
chose
that
is
it's
a
little
bit
more
distributed
in
terms
of
the
performance
optimization.
If
you
use
a
specific
optimization
for
for
the
space
filling
curves,
it's
very
frequent
that
you're
going
to
have
not
only
the
partition
key,
but
maybe
even
a
couple
of
extra
common
predicates.
So
that's
why
we,
we
chose
a
specific
design.
C
Cool
I'm
going
to
talk
a
little
bit
about
future
roadmap,
but
I
did
also
see
that
there
was
a
question
came
in
from
a
user
on
linkedin
it's
about
univariate
versus
multivariate
forecasting.
So,
with
the
would
certainly
with
the
the
look
back
building
the
look
back
tensors
you
can
support
both
univariate
or
multivariate
use
cases
we're
going
to
be
providing
some.
You
know
examples
in
the
near
term
about
you
know,
looking
showing
how
to
use
this
framework
to
build
certain
ml
models
and
forecasting
models,
so
we'll
definitely
include
both
univariate
and
multivariate.
C
So
as
we're
looking
forward
to
some
of
the
few
things
we're
working
on
in
the
future
for
tempo,
we've
got
a
number
of
of
key
things
that
are
coming
up
soon:
a
lot
more
support
for
machine
learning,
in
particular
we're
going
to
be
implementing
a
lot
using
the
spark
ml
transformers
framework,
and
so
that
you
can
chain
together.
A
number
of
you
know,
featured
transformations
and
feature
generators
into
a
spark
and
all
pipeline
for
those
of
you
not
familiar
with
it.
C
The
nice
thing
here
is
that,
of
course,
the
underlying
data
is
all
distributed,
spark
data
frames,
and
so
these
operator
can
operate
at
great
scale,
so
we're
going
to
be
providing
a
lot
of
different
feature
generation
and
transform
transformation
functions.
This
way
to
allow
you
to
build
features
for
your
machine
learning
models,
as
well
as
just
create
summary
statistics
using
those
we're
also
going
to
be
adding
support
for
natural
language
interval
definitions.
C
Another
another
common
use
case
for
machine
learning
that
we're
going
to
looking
to
support
is
training
models
per
time
series
for
certain
use
cases.
People
want
to
be
able
to
have
a
model
fit
to
each
of
the
different
time
series
in
in
a
large
data
set
and
being
able
to
train
those
in
parallel
can
get
you
a
lot
of
performance
advantages
so
leveraging
the
functionality
of
you
spark
user
defined
functions
udfs
in
order
to
be
able
to
distribute
that
model
training
at
scale.
C
C
So
it's
a
kind
of
an
example
of
what
that
looks
like.
If
you
think
about
this,
you
know
and
thinking
about
this.
This
is
a
different
layout.
There's
not
don't
think
of
this
as
a
data
frame,
but
think
of
this,
as
you
know,
every
row
is
a
logical
time
series
and
then
the
the
columns
here
are
sort
of
moving
through
time
from
from
zero
up
to
some
future
time.
C
T
if
you're
going
to
start
dividing
carving
up
your
your
data
set
into
training
and
validation
data,
you
have
to
be
very
careful
about
how
you
do
that
in
order
to
not
to
have
these
issues
of
leakage.
So
if
you're,
if
you've
got
something
like
a
device,
you
know
like
iot
type
example,
where
you're
wanting
to
train
a
single
model
that
then
has
to
generalize
to
new
devices
that
it
hasn't
seen
before
you
need
to
use
out
of
sample
validation
right.
C
So
you
need
to
train
on
complete
time
series
for
existing
known
devices
and
then
test
or
validate
on
the
on
completely
unseen
ones.
You
can't
just
randomly
assign
data
points
across
those
and
then
expect
to
have
a
reasonable
validation.
Then
there's
also.
So
this
is
what
you'd
end
up
with
you
know
doing
this
properly.
You'd
wanted
to
validate
this
way,
doing
it
with
if
you're
doing
forecasting.
C
You
need
to
do
both
out
of
sample
and
out
of
time
validation.
So
you
need
to
kind
of
follow
a
checkerboard
type
pattern
when
you're
doing
that.
So
this
is
just
some
of
the
things
that
make
you
know:
training
models
and
cross
validation,
tricky
with
time
series
data
we
plan
on
building
some
additional
tools
around
tempo
that'll.
Allow
you
to
do
this
more
easily.
You
know
just
define
the
strategy
that
you
want
to
use
and
then
have
the
algorithms
will
build.
The
the
cross
validation
sets
for
you.
A
Yeah,
absolutely
thanks
a
lot
tristan
super
interesting
stuff
on
cross
validation,
so
I
I
will
switch
to.
Let
me
share
my
screen
just
to
to
let
everybody
know
too
there
right
now.
There's
a
python
api!
That's
what's
primarily
in
the
repository
right
now,
but
we're
adding
a
scala
implementation.
It
should
be
actually
live
in
a
couple
days.
So,
if
you're
more
familiar
with
scholar,
you
can
use
that
right
now.
The
python
version
does
not
depend
on
the
scala
version.
A
So
there's
a
little
bit
of
flexibility
there
if
you're
interested
in
any
other
apis.
Definitely
let
us
know
specifically
for
r.
I
think
that
might
be
the
next
candidate
okay,
so
you
should
be
seeing
the
the
tempo
quick
start
notebook
right
now,
and
this
will
be
available
actually
again
on
the
github
a
little
bit
later
today.
So
we'll
post
this
it'll
just
be
in
the
main
docs
section
that
you
can
kind
of
check
out.
A
This
uses
a
public
data
set
from
uci,
which
is
a
human
activity,
recognition,
data
set
of
smartphones
and
smart
watches,
or
some
of
that
accelerate
acceleration
measurement.
So
you
can
kind
of
see
here.
I
just
posted
a
quick
picture:
it
just
measures
the
x
y
and
then
z
coordinates
for
for
movement
here
and
the
the
thing
that's
kind
of
paralleling.
What
we
already
showed
is
that
you
have
multiple
users
in
the
same
data
set,
so
you'll
have
all
these
measurements
and
the
actual
index.
A
The
time
index
is
absolutely
not
regularly
spaced,
so
you
won't
see
a
measurement
every
second
or
every
millisecond.
It's
kind
of
all
over
the
place,
which
is
what
this
mock-up
is
showing
so
you'll
be
able
to
kind
of
there's
a
databricks
notebook
within
the
within
the
repository
will
be
later
today.
There's
an
ipython
version
as
well.
A
So
some
of
this
code
is
actually
just
running
the
the
data
load
for
this
open
data
set
it's
a
zip
file,
so
there's
a
little
bit
of
prep
work
there,
but
other
than
that.
You
can
essentially
just
pip
install
this
directly
from
the
repository.
So
this
actually
points
to
you
know
a
fixed
revision.
So
this
is
reproducible.
A
That's
all
you
need
for
the
python
version,
so
this
will
just
install
it
for
you
for
subsequent
commands.
You
can
see
here,
I'm
actually
just
displaying
this
data
set.
This
data
set
didn't
come
with,
I
think,
a
specific
timestamp
column,
so
we
actually
just
converted
and
created
one
here
on
the
fly
so
that
we
could
we
could
use
it
within
the
tsdf
framework.
A
So
that's
this
event
timestamp,
but
other
than
that,
it's
fairly
straightforward,
just
a
set
of
12-13
columns
with
xyz
measurements
model
devices
for
these
specific
phones
and
watches,
and
then
the
user
as
well.
So
I
think
this
is
nine
users,
you
don't
have
to
know
the
cardinality.
So
we'll
show
you
some
interesting
display
commands
here.
A
So
this
is
how
you
get
it
into
the
the
tsdf
format.
Really,
if
you
just
have
any
spark
data
frame,
all
you
have
to
do
is
wrap
it
here
with
this
constructor.
Just
to
you
know,
say
what
the
event
time
is
say
what
the
partition
columns
are
and
again
these
are
logical
partition
columns.
It's
not
you
know
like
a
date
column
or
something
like
that,
it's
more
of
how
you
would
partition
the
data
by
a
specific
user
device.
So
it's
a
logical
partition
column.
A
A
So
the
first
thing
you
might
want
to
do
is
actually
visualize
your
data
again.
If
you've
got
a
billion
data
points,
it's
a
it's
impossible
to
collect
that
and
actually
draw
a
meaningful
picture.
So
one
of
the
first
things
that
you
might
want
to
do-
and
this
is
true
for
other
use
cases
as
well-
is
just
resample
the
data,
so
there's
a
quick
or
kind
of
a
simple
resample
function
here
you
can
specify
whatever
your
unit
is.
You
know
second
minute
hour
and
then
your
your
preferred
function.
A
A
So
we'll
get
those
data
points
and
it'll
do
it
for
the
metric
columns,
as
tristan
pointed
out
earlier,
so
there's
a
whole
whole
lot
of
structure
that
we're
kind
of
imposing
on
the
data.
Just
from
knowing
a
few
few
of
these
different
attributes,
I'm
actually
just
looking
at
one
day
for
visualization
purposes,
so
here's
my
user
base,
you
can
see
users
g
c,
a
and
h
actually
fall
into
this
day
with
active
measurements
and
it's
kind
of
perfectly
disjointed
data
over
different
time
periods
just
by
the
nature
of
the
data
set.
A
The
next
thing
which
we
wanted
to
talk
about
was
again.
I
think
that
this
package
is
largely
about
just
making
some
of
the
time
series
analysis
a
little
bit
simpler.
So
if
you've
got
this
data
set,
how
do
you
describe
it
and
what
we've
done
is
extended?
The
typical
spark
describe
functionality
so
with
when
you
describe
a
spark
data
frame,
you'd
get
things
like
per
column.
A
You
know
my
account
mean
standard
deviation
min
max,
so
these
are
pretty
useful
but
there's
other
features
of
the
data
set
which,
given
that
you
have
multiple
users
up
to
thousands
of
users,
you
know
what
else
might
you
want
to
know
about
that
data
so
unique
time
series
count
is
extremely
beneficial.
This
is
basically
how
many
logical
partitions
you
have
so
that's
encoded
in
here.
So
there's
nine
users
in
this
data
set
min
max
time
stamp
again
are
provided
in
addition
to
granularity.
A
So
what
we
actually
do
is
again,
since
we
know
the
time
stamp,
we'll
just
take
a
look
at
the
time
stamp
figure
out
what
is
the
most
granular
time
you're
dealing
with?
Maybe
it's
milliseconds,
maybe
it's
you
know
seconds
or
days,
something
like
that,
so
we
can
just
go
into
the
time
stamp,
analyze
that
column
and
figure
that
out.
A
Lastly,
we
actually
added
this
missing
value
percentage
so
for
this
data
set,
it's
actually
fairly
clean.
You
can
basically
just
on
each
column
level,
we'll
again
go
through
and
figure
out
if
there's
any
nulls
or
missing
values
and
then
roll
up
a
percentage.
Based
on
that.
Just
to
give
an
idea
of
you
know
what
you
need
to
clean
up.
If
there's
some
joints,
you
need
to
actually
fill
in
or
look
up
data.
You
can
figure
that
out
pretty
quickly
based
on
this
described
statement.
A
Okay,
so
that
was
a
little
bit
about
the
the
describe
function.
Resample
function,
all
I'm
doing
here
is
this
is
nothing
new.
It's
actually
just
displaying
some
of
the
watch,
accelerometer
data
and
aggregating.
I
think
this
is
just
you
know,
some
of
the
device
information
and
again
just
running
the
describe
again
quickly
on
this
you'll,
see
pretty
much
the
same
granularity
same
unique
time
stamp
account.
A
Sometimes,
if
you're
joining
different
data
sets
it's
important
to
know,
something
else
is
kind
of
more
slowly
moving,
so
maybe
it
could
be
days
which
is
really
really
often
the
case
for
financial
data
sets,
for
example,
okay.
So
the
first
thing
that
I
just
wanted
to
to
show
was
the
as
of
join.
So
we
talked
about
this.
I
think
tristan
cover
in
depth.
This
is,
I
would
say,
kind
of
the
heart
of
how
we
built
tempo.
A
We
came
up
with
an
as
of
joint,
and
then
we
came
up
with
an
optimized
implementation
for
sku
in
particular.
What
this
does
is
it
doesn't
care
about
irregularly,
spaced
time
it
doesn't
care
how
big
the
data
set
is.
It
avoids
a
joint,
a
standard
joint
at
all
and
instead,
under
the
hood,
does
some
of
what
the
visualization
tristan
showed
earlier,
but
anyway,
at
the
end
of
the
day,
what
you
should
expect
is
all
values
from
the
watch.
A
Accelerometer
data
set
we'll
just
produce
another
copy
of
it
and
append
the
columns
from
the
phone
accelerator
accelerometer
data
set
as
of
whatever
the
times
in
the
watch
data
set
would
be
so
again
they
could
completely
not
line
up
at
all.
These
are
fairly
granular
down
to
millisecond
and
we'll
figure
out
whatever
the
last
observation
from
the
phone
data
set
and
carry
that
forward
here.
So
just
a
overlay
of
the
two,
but
essentially
you're
just
making
sure
that
you
have
an
extra
column
for
all
the
watch
data.
A
So
this
is
the
the
as
of
join
again,
this
the
scales
up
to
terabytes
of
data
or
billions
of
data
points.
We've
tried
this
out
in
in
production
use
cases,
but
you
know
often
it
what
you're
going
to
see
is:
there's
massive
skew
for
a
lot
of
production
use
cases
so
in
energy,
energy,
specific
use
cases
or
financial
data
sets,
especially
if
you
have
a
really
big
ticker,
something
like
spy
spider
or
something
some
of
these
larger
traded
symbols.
A
Then
you're
going
to
have
too
much
data
to
fit
in
memory,
so
the
window
won't
really
fit
in
memory
for
that
partition
column.
So
one
of
the
first
things
that
we
added
was
an
optimization
to
essentially
take
among
your
partition
columns.
So
things
like
your
user
in
this
case
device
id
or
a
ticker
chunk
that
into
these
smaller
slices
these
time
slices
the
stamp.
A
The
default
is
actually
20
minutes,
but
you
can
control
that
with
this
timestamp
partition
value
and
then
we'll
chunk
it
up
into
smaller
pieces,
run
a
computation
to
get
the
last
observation
by
including
previous
partitions
worth
of
data,
and
this
will
perform
very
well
in
the
skewed
case.
So,
as
I
said
in
some
cases,
maybe
your
skewed
joins
whether
they're
sql
or
even
this
as
a
drain.
A
They
may
be
taking
hours
or
running
out
of
memory
if
you
use
this
with
the
default
values,
we've
seen
cases
where
it
may
decrease
the
job
time
from
something
like
you
know,
a
couple
hours
to
three
minutes
or
something.
So
this
does
a
really
small
scale
version
of
that.
I
think
this
is
maybe
something
like
a
3
3x
performance
increase,
but
your
mileage
may
vary.
It
kind
of
depends
on
your
data
set,
but
this
has
been
incredibly
useful,
so
kind
of
an
interesting
concept
here.
To
do
that.
A
The
last
thing
that
I
will
mention
is
just
the
the
right
feature
that
we
talked
about
earlier.
So
you
know
from
that
spark
data.
For
the
the
time
series
data
from
the
tsdf
object,
all
you're
doing
is
saying
right,
give
it
a
table
name.
As
you
can
see.
This
is
writing
it
to
delta
lake.
You
can
see
the
the
evidence
of
not
only
the
seat
tabs
but,
of
course,
the
optimize
command
that
we
wrote.
A
A
So
here's
just
a
quick
statistics
here
just
to
point
out
that
you
know
delta
is
using
statistics,
part
k,
stats
and
delta
stats
under
the
hood,
but
here's
an
example
of
some
of
that
time:
slice.
So
I've
just
taken
my
my
table
that
I
defined.
So
it's
really
just
a
table
accessible
via
sql.
I
can
restrict
to
the
user,
restrict
to
this
particular
time,
slice,
which
is
one
minute
and
I'll,
be
able
to
run.
These
queries
a
lot
faster
and
it's
not
a
lot
of
extra
setup
or
work.
A
You
know
from
you
as
the
user
of
something
like
this.
So
when
you're
trying
to
surface
your
your
time,
series
results,
maybe
your
silver
tables
or
your
data
science
copies
to
other
users.
This
is
incredibly
helpful.
It
just
helps
with
some
of
the
performance
and
really
common
query
patterns.
So
you
know
personally,
when
I
was
running
a
lot
of
time
series
queries
just
to
identify.
A
A
So
I
hope
this
has
given
a
good
introduction
just
to
to
the
usage
just
samples,
I
would
say
most
of
the
methods,
there's
a
few
more
like
simple,
moving
average
exponential
moving
average,
which
we
didn't
cover
in
the
quick
start,
but
those
are
those
apis
are
documented
on
the
the
github
page.
So
I
think
with
that.
That's
the
end.
Thanks
everybody
for
listening
to
this,
the
technical
portion.
I
think
we
have
some
questions
that
are
teed
up
here,
so
I
can
address
these.
A
A
Looks
like
daryl
had
some
questions
about
zeroing
performance
affected
by
order
of
columns.
This
is
a
really
great
question
and
that's
actually
the
reason
why
we
use
the
hilbert
version
of
the
space
filling
curve
optimization.
So
we
found
just
from
a
lot
of
examples
that
we've
tested
that
the
hilbert
optimization
does
distribute
more
evenly
on
the
more
columns
you
add,
so
the
leading
columns
used
to
matter
a
little
bit
more
now
they're
pretty
evenly
spaced.
So
you
don't
need
to
worry
about
which
column
you
select
first
as
much
with
that
specific
version.
A
A
Why
would
we
put
the
partition
column
in
the
z
order?
Okay,
it
looks
like
that
was
answered,
but
yeah
that
that
is
the
partition
column.
There
is
not
something
like
a
date
partition.
It's
more
of
the
logical
partition
that
you'd
set
up
for
your
time
series
data
so
hopefully
that
that's
been
answered.
B
C
Sure
yeah
I
mean
we've
so
yeah,
there's
a
question
on
linkedin
about
streaming
data,
and
I
think
it's
it's
one
of
those
things
where
you
know
streaming
is
immensely
complex
and
there's.
We
still
have
plenty
to
work
on.
Just
for
you
know,
batch
type
data.
Certainly
I
think
it's
on
our
our
longer
term
roadmap
to
to
start
adding
some
support
for
streaming,
particularly
when
it
comes
to
some
of
the
iot
use
cases
and
and
other
other
use
cases
which
may
be
more
real-time.
A
Yeah,
I
think,
for
streaming,
the
some
of
the
the
same
functionality
would
have
like,
as
of
joins,
would
apply
on
the
micro
batches,
but
we'd
have
to
re
kind
of
redo.
How
we
do
the
windows,
because
it's
going
to
be
different
for
those
two
apis
but
yeah.
That's
that's
great!
Thanks
tristan,
I
got
another
question
in
the
q
a
about.
Can
we
use
tempo
without
databricks?
A
Absolutely
yeah?
It's
just
a
it's
a
it's,
a
library,
that's
based
on
spark.
So
as
long
as
you're
running
some
sort
of
spark
implementation,
you
can
run
these
or
use
these
apis.
So
in
our
code
base
you
know
we
just
run
it
locally
when
we
run
our
unit
tests
to
do
all
these
different
joints,
and
things
like
that.
So
I
think
that
the
main
benefit
for
databricks
is
the
performance
optimizations
that
were
called
out
in
the
the
reference
architecture.
A
Things
like
adaptive,
query
execution
or,
if
we're
using
like
you're
using
pandas,
udfs
or
just
windowing,
all
those
different
things
that
are
included
in
the
runtime
you'll
just
get
a
performance
boost
from
those
and
obviously
when
you're
writing
to
delta
and
querying
the
tables.
That
way,
you'll
get
that
performance
boost
and
just
easier
to
manage
the
notebook
infrastructure
for
these
different
data.
Science
workflows.
A
A
We
wouldn't
advise
necessarily
partitioning
by
hour
because
your
files
might
be
too
small,
but
if
you
do
have
to
do
some
operations
again,
like
the
resampling
rolling
averages
and
start
appending
those
different
values
to
your
data
set,
that's
where
tempo
can
help
you
do
that.
Just
so,
you
have
those
in-memory
data
frames
or
you
want
to
write
it
back
out
to
your
data
lake
for
further
processing.
B
A
Yeah,
I
can
take
that
one.
So
tempo
itself
again
is
more
of
like
a
processing,
it's
apis
to
process
the
data
efficiently,
but
it's
definitely
restricted
to
more
batch
type
implementations.
Now
things
like
again
enriching
data
for
like
back
tests
or
prepping
the
data
for
really
granular
machine
learning,
algorithms
to
run
across
all
of
your
different
data
sets,
I
think,
we'll
we
do
want
to
add
functionality
to
aggregate
data
into
gold
level
tables.
It's
a
lot
smaller,
so
things
like
aggregating,
maybe
by
hour
and
saving
that
on
disk.
C
I
see
there's
a
question
in
the
q,
a
multiple
groups
of
time
series,
this
person's
got
data
from
five
parking
lot
buildings
and
then
different
wants
to
be
able
to
compare
availabilities
within
a
time
window
for
multiple
buildings
and
perform
time
series
predictions
on
multiple
time
series
yeah.
I
think
you
know
that
that's
a
great
use
case,
I
think
for
tempo.
I
would
you
know
you'd
want
to
consider
you
know,
organizing
all
of
your.
You
know
those
different
time
series
into
a
single
data
frame
where
you've
got
those.
C
You
know
the
factors
like
the
the
different
buildings.
You're
looking
at
here
would
be
the
partitioning
columns,
and
then
you
know
the
the
individual
observations
for
those
you
know
organized
by
their
by
time.
You
could,
then
you
know,
depending
on
the
types
of
analytics
you
want
to
perform,
you
can
perform.
You
know:
aggregations
rolling,
aggregations
across
buildings
or
across
groups
of
buildings,
we're
we're
still
building
out
some
of
the
the
functionality
for
doing
a
lot
of
those
types
of
rolling
aggregations,
but
those
should
be
coming
in
the
near
future.
C
To
do
you
know
more
different
variations
of
that
so
definitely
and
we'd
definitely
be
interested
in
hearing
more
about
your.
If
you
want
to
try
it
out
with
this
data.
Your
data
on
tempo
I'd
love
to
hear
about
your
experiences,
and
you
know
any
feedback
from
users.
We're
really
keen
on
that.
So
definitely
engage
with
us
on
the
on
the
github
channel
on
the
github
site.
A
Our
capital
markets
reference
architecture,
so
that's
actually
what
we
covered
today
and
I
think
the
question
was:
is
there
an
implementation
so
there's
actually
a
blog
that
we
wrote
recently
in
january
to
cover
a
lot
of
implementation.
It
uses
tempo
and
then
provides
an
implementation
kind
of
for
some
specific
use
cases
like
market
surveillance
and
spoofing
and
a
little
bit
of
volatility
calculations
as
well
so
prepping
for
that
using
tempo.
A
C
There's
another
question
about
running
arima
models
on
tempo,
we're
not
aiming
to
replace
other.
You
know
ml
frameworks,
or
you
know
statistical
frameworks.
Our
goal
is
really
to
provide
a
lot
of
you
know,
data
preparation,
functions
and-
and
you
know,
utility
functions
for
working
with
this,
but
you
know
our
goal
is
definitely
to
integrate
with.
So
you
could.
You
know,
for
example,
fit
arima
models
to
your
different
time
series
using
another
framework
like
whether
that
be
learn
or
another.