►
From YouTube: Data Collab Lab: Use Delta Engine to create a Hyperleaup from Delta Lake to Tableau Hyper
Description
Are you struggling with performance on your Tableau experience in the cloud? Do you use Tableau Server to Extract data from Delta Lake? Want to learn how you can create a hyperleaup from your Delta Lake to Tableau that will decrease latency to insights? Join us for another fun AMA session as we collaborate with Will Girten and Shoam Bhatt, and figure out what exactly a hyperleaup is!
Link to notebook: github.com/databricks/tech-talks
-----------------
Guests
-----------------
Soham Bhatt is a Senior Solutions Architect at Databricks based out of Seattle, WA. He is passionate about helping his customers in their journeys to build modern Data Lakes for Advanced Analytics and ML/AI. Before Databricks he worked at Toyota Motors on building their next generation Big Data Platform. Prior to that his background was in building Enterprise Data Warehouses for Fortune 100 companies with Kimball methodologies and now he loves guiding his customers with best practices as they convert those EDWs into modern Data Engineering Platforms in AWS and Azure.
Will Girten is a Sr. RSA at Databricks. He's helped some of the largest federal customers at Databricks build modern, enterprise Delta Lakes in the cloud. He specializes in building efficient and reliable ETL pipelines for fast data engineering and BI workloads.
-----------------
Hosts
-----------------
Franco Patano is a Solutions Architect at Databricks, where he brings over 10 years of industry experience in data engineering and analytics. He has architected, managed, and analyzed data applications both big and small, with open source and proprietary software, utilizing SQL, Python, Scala, Java, and Apache Spark, as well as experimenting with data science. Prior to Databricks, Franco worked as a Data Architect and Analyst in the Commercial Real Estate, Banking, and Education industries for organizations large and small.
Denny Lee is a developer advocate at Databricks, where he works on Delta Lake, Apache Spark, Data Sciences, and Healthcare Life Sciences. He has previously built enterprise DW/BI and big data systems at Microsoft including Azure Cosmos DB, Project Isotope (HDInsight), and SQL Server as well as the Senior Director of Data Sciences Engineering at SAP Concur. Denny holds a Masters in Biomedical Informatics from Oregon Health Sciences University.
To join the live chat, check out the meetup page: https://www.meetup.com/data-ai-online/events/273893266/ Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
B
Hey
everyone.
Thank
you
for
joining
us
today
on
another
episode
of
data
collab
lab
with
franco
and
dunny.
Thank
you
karen
for
introing
us
and
kind
of
sharing
all
that
awesome
new
news,
malcolm
gladwell.
I
am
super
excited
that
he's
coming
today.
Ai
summit
outliers
has
to
have
been
one
of
my
favorite
books
out
there.
B
So
I
am
super
excited
that
he's
coming
and
I'm
joined
here
by
a
couple
of
my
favorite
outliers
will
gertin
and
sohambot
and
danny
lee,
and
we
like
to
to
kind
of
show
you
what's
going
on
in
in
data
bricks,
land
and
something
that
happened
recently.
You
know
will
created
this
new
project
and
soham
and
I
kind
of
saw
what
it
was
and
we
got
super
excited
so
denny
and
I
were
talking
about
and
they're
like.
Maybe
we
should
bring
them
on
the
lab.
That's
what
what
do
you
think
denny
is.
D
B
E
Yeah
yeah
sure,
hey,
hey
everyone.
My
name
is
soham,
but
solutions
architect
with
databricks
based
out
of
seattle,
washington,
it's
it's
getting
cloudy
and
rainy
here
again,
but
the
summer
and
fall
were
beautiful.
My
my
background
before
joining
databricks
was
a
big
data
platform
architect
at
toyota
motors.
So
I
come
from
a
lot
of
data
lake
data
platform
background
and
before
that
I
did
a
lot
of
a
lot
of
etl
and
built
a
lot
of
data.
Warehouses
for
various
companies
so
excited
to
be
here.
B
That's
awesome
thanks,
so
will
you
work
with
federal
customers?
That's
that's
pretty
important
work.
You
probably
can't
say
a
lot
about
what
you
do,
which
we
got
to
respect
those
those
boundaries.
But
a
little
known
fact
I
know
about
you-
is
that
you
have
some
type
of
secret
clearance.
Is
that
right.
B
D
B
That's
right
awesome,
so,
let's
get
into
it
first,
I'm
gonna
just
share
my
screen
and
kind
of
show
you
what
what
the
a
notebook
that
we've
been
working
on
and
then
we're
gonna
talk
to,
will
and
see
kind
of
what
this
is,
what
what
is
hyperloop
so
will
you
want
to
take
us
there?
What
is
hyperloop?
How
did
how
did
you
even
think
about
this
and
kind
of
what
were
what
were
your
initial
findings
and
and
kind
of
just
tell
us
like
the
origin
story,.
D
Yeah
sure
so
hyperloop
is
the
library
I
created
so
basically
the
problem
that
it
solves
is:
yes,
it's
just
another
library
that
you
can
download
and
install
but
say
that
I'm
a
tableau
user
and
basically
I
come
in,
and
I
want
to
do
a
new
visualization.
So
all
my
data
lives
out
in
the
delta
lake
and
I
want
to
build
my
report
basically
off
that
data.
In
order
to
do
that,
I
fire
up
my
tableau
desktop
client.
D
I
use
the
brand
new
tableau
connector
to
database
cluster
and
then
I
begin
my
extract
and
then
I
wait
and
I
wait
you
know,
depending
on
the
size
of
the
extract,
I
might
be
waiting
around
for
a
very
long
time
on
the
order
of
maybe
several
hours
even
or
you
know
say
that
I
know
that
my
data
set's
pretty
large,
there's
millions
millions
of
rows.
I
could
what
I
could
do
is
basically
schedule
that
extract
as
a
job
within
tableau
server
and
have
that
job
run
overnight
and
then
you
know.
D
Basically
I
want
to
capture
the
latest
data
as
it
changes
up
on
that
delta
lake.
I
could
set
that
that
job
to
refresh,
but
over
time
as
my
reporting
team
grows
in
size,
there
might
be
more
and
more
people
running
these
extracts
that
pull
the
data
out
of
delta
lake
and
create
a
tableau
extract
and
over
time
what
happens
is
basically
the
the
tableau
back.
Our
rounder
can't
keep
up
long-running
extracts
jobs
take
a
long
time
and
they
basically
queue
up
over
time,
and
you
know
pretty
much.
D
The
table
backgrounder
can
become
a
single
point
of
failure
as
resources
get
exhausted.
So
what
hyperloop
basically
does?
Is
it
solves
that
waiting
around
problem
so
rather
than
pulling
data
out
of
the
delta
lake
across
odbc,
which
can
be
a
pretty
chatty
protocol?
What
it
does
is,
instead
of
pulling
it
it
kind
of,
does
the
inverse.
So
we
create
a
data
extract
from
the
database
cluster
as
the
data
changes
within
the
delta
lake,
and
then
we
push
it
rather
than
pull
it.
B
D
B
E
Yeah
they
were
super
excited.
I
mean,
first
of
all,
when
I
found
out
about
this
pattern
of
data
bricks,
actually
creating
the
tableau
extracts
and
pushing
it
into
tableau
like
I'm
like
wow.
This
is
great
because
the
typical
pattern
is
people
use
the
tableau
desktops
to
to
like
probably
download
the
extract
or
put
it
on
some
cloud
file,
and
then
they
use
a
tableau
desktop
to
do
this
process
of
creating
the
extracts
and
then
pushing
it
to
the
tableau
server.
E
So,
instead
of
that
here,
all
the
processing
of
the
creation
of
the
hyper
file
happens
on
databricks
cluster
and
as
part
of
the
same
etl
pipeline,
we
are
pushing
it
to
tableau,
which
is
amazing.
So
the
reason
some
of
my
customers
got
super
excited
is
because
they
are.
They
are
creating
a
curated
data
lake
right
because
they
have
structured
data,
unstructured
data.
They
want
to
convert
some
of
their
batch
etl
jobs
into
streaming,
etl
jobs
and,
as
you
as
you
all
know,
spark
and
databricks
does
an
amazing
job
with
streaming
as
well
right.
E
E
B
B
Is
that
you
essentially
in
the
cloud
you
have
the
network
to
contend
with
and
in
the
cloud
the
network
is
the
internet
right
and
routes
are
kind
of
you
know
up
in
the
air,
depending
on
what
you're
going
to
get
and
data
locality.
It's
important
now
right.
So
data
has
gravity
and
you
want
to
process
it
where
it
is
and
kind
of
the
different
types
of
ways
that
you
can
access.
B
The
data
is
either
using
data
bricks
to
perform
etl,
which
you
know
spark
is
a
great
etl
engine
delta
lake
brings
the
file
formats
and
the
extensibility
to
the
apis
to
make
that
type
of
work
very
simple,
and
then
there's
also
connecting
directly
with
the
delta
engine
or
with
data
bricks
itself.
Then
this
is
what
we're
talking
about
with
the
odbc
connection,
where
you're
connecting
tableau
to
databricks
itself
and
then
that's
connecting
to
your
cloud
object,
store
and
there's
all
these.
B
These
efficiencies
that's
built
into
that
connector
and
even
our
bi
team
has
spent
a
lot
of
time,
making
that
connector
better
there's
even
a
databricks
connector
with
inside
tableau
now
and
with
stuff
like
delta
caching.
The
experiences
have
gotten
better
but,
as
you
guys
have
seen
in
the
field,
requirements
are
getting
more
stringent.
People
want
their
the
latency
of
the
data,
so
the
data
freshness
right.
B
D
Yeah,
I
could,
I
guess
I
could
take
this
one,
so
you
know
basically,
basically
by
leveraging
this
hyperloop
library
and
to
create
these
data
extracts
a
lot
faster.
You
know
what
you
could
do
is
in
your
your
etl
pipeline
that
might
be
continuously
running.
You
could
tack
this
as
an
extra
step
onto
the
end
of
your
pipeline.
So
after
you
you've
created
your
cdc
and
you've
merged
it
using
delta
merge.
You
can
now
take
that
and
basically
create
an
extract
and
push
it
to
tableau
server.
B
Wow,
that's
awesome
so
essentially,
if
you
have
like
schedules
on
tableau
server
for
extract
refreshes
and
you
have
your
etl
schedule,
I
know
it
was
a
big
pain
for
me.
Back
in
the
day
when
I
was
trying
to
manage
like
warehousing
refreshes
and
tableau
refreshes
that
you
kind
of
have
to
manage
overlaps
right
and
maybe
put
some
buffer
and
then
all
of
a
sudden,
if
one
day
the
servers,
the
server
gods
were
shining
down
on
you
poorly
and
they
the
server.
B
You
know
everything
wasn't
moving
along
your
buffer
went
away,
and
maybe
your
schedule
missed
right
and
your
etl
didn't
finish
for
two
hours
later
than
it
normally
did.
But
your
extract
refresh
schedule
now
overlapped
and
now
it
causes
pain
and
latency.
That's
an
extra
thing
to
track
down.
Tacking
on
this
process.
To
the
end
of
your
etl
pattern
seems
like
a
pretty
pretty
good
architecture
pattern.
If
you're
trying
to
decrease
the
total
amount
of
latency
in
your
end-to-end
pipeline.
C
So,
actually,
oh
sorry,
I
wanted
to
chime
in
real
quick
just
because
we
had
a
couple
comments
on
linkedin
live,
so
one
of
the
call
outs
do
my
small
favor
franco.
If
you
can
just
scroll
up
back
to
the
original
the
per
to
the
other
slide
perfect
there,
you
go
the
use
cases.
There
was
a
call
out
on
linkedin
live
about
the
fact
that
you.
Why
aren't
we
talking
about
live
queries,
and
so
we
did
want
to
emphasize
the
point
that
no,
you
absolutely
can
do.
C
Tableau
live
queries
against
databricks,
exactly
what
franco
is
currently
circling?
Okay,
this
is
for
the
case
where
we're
going
to
be
extracting
out
millions
upon
millions
of
rows
right
that
you
want
to
bring
down
to
tableau
for
scheduled
extracts
for
whatever
process
that
you
want
to
do
downstream.
As
opposed
to
do
live
queries.
There
are
some
situations
where
live.
Queries
are
extremely
appropriate
and
exactly
you
can
connect
it
to
delta
engine
directly.
C
B
D
Yeah,
so
if
you
want
to
create
a
notebook,
basically
or
a
workbook
within
tableau-
and
you
want
to
basically,
you
know,
create
a
data
extract
that
might
not
fit
the
bill
or,
if
you
have,
you
know
a
really
large
data
set.
Where
you
know
your
query
might
take
a
very
long
time
and
it'd
be
better
just
to
create
a
data,
extract
kind
of
break
off
a
piece
and
do
your
ad
hoc
visualization
on
your
own
end.
B
Or
if
you
have
like
queries
that
return,
a
lot
of
rows,
for
example,
and
returning
a
lot
of
roads
over
the
wire,
can
be
a
bit
slow
right,
even
with
the
modifications
that
you
know,
we've
been
working
with,
our
bi
team
has
been
working
with
simba
and
tableau
to
make
it
faster.
So
like
a
arrow,
for
example,
apache
arrow
has
made
the
overall
transport
much
faster.
B
So
you
get
all
the
data
together
and
then
you
transport
it
over
the
wire
to
tableau
server
so
that
the
data
locality
is
more
local
to
where
you're
going
to
be
using
it.
So
the
data
in
the
data
lake
sits
on
delta
lake.
In
you
know,
the
cloud
and
the
data
center
on
its
side
tableau
might
sit
in
the
cloud
it
actually
might
sit
on.
Prem
right.
Have
you
guys
seen
instances
out
there
where
tableau
server
it
could
be
in
cloud
or
could
be
on-prem,
and
so
you
want
better
connectivity
to
your
on-prem
tableau
server.
D
B
Awesome
well,
let's
get
into
it.
Let's
start
playing
around
with
it
and
see
how
to
make
this
work
right.
We've
seen
enough,
we've
talked
about
a
little
bit.
What
I
I
prepared
today,
I
have
a
pretty
extensive
demo
that
goes
through
how
to
create
delta
tables.
Using
this
specific
thing
uses
tpch,
they
normally
use
it
for
benchmarking.
It's
just
sample
data.
B
What
I've
done
here
is
I've
created
a
metadata
database
on
databricks
and
I've
put
all
of
the
data,
the
tables
in
there,
and
now
we're
going
we're
going
to
use
hyperloop
to
extract
that
data
and
send
it
over
to
tableau
server,
where
we
have
a
dashboard
waiting
for
us
that
we're
going
to
kind
of
show
how
this
stuff
works.
So
we're
going
to
be
using
this
database,
so
I'm
using
primarily
sql
commands
here.
B
So
I'm
going
to
set
the
context
right
and
any
sql
editor.
You
have
to
use
the
database
that
you're
currently
in
so
I'm
going
to
say,
use
this
demo
database
that
I've
created
here
and
then
I
want
to
see
how
many
rows
we
have
so
here.
I've
got
about
29
million
rows
that
I
need
to
get
over
just
for
the
audience
sake.
If
I
were
your
experience,
wise
will
and
soham.
If
we
were
to
do
something
like
this
on
tableau
server
and
have
to
extract
29
million
rows
over
the
wire.
D
B
E
Oh
yeah,
typically
this
this
like,
if
you
are
going
to
do
an
extract
like
which
are
10
million
20
million
record
plus
it
could
take
hours
like
literally
two
to
three
hours
like
we
have
customers
where
there
are.
They
have
scheduled
all
these
extracts
they
run
for
hours
and
the
tableau
backgrounder
is
like
just
super
slow
beyond
a
certain
point.
So.
B
Wow,
that
seems
like
a
long
time,
especially
when
you
think
about,
like
the
total
amount
of
time,
that
it
takes
to
kind
of,
derive
an
insight
right
and
to
be
able
to
work
with
it.
Two
three
hours
is:
it
can
be
a
long
time
in
today's
world.
So
that's
interesting.
So
what
I'm
doing
here,
I
got
29
million
rows
and
I'm
gonna
be
just
taking
a
sample
of
the
data.
So
this
is
what
it
looks
like
I'm
gonna
be
adding
a
process
date
on
it.
B
C
And
one
thing
I
would
like
to
love
to
add
franco
as
if
you
scroll
back
up
just
to
add
that
cell
you'll
notice
that,
basically,
in
all
of
our
examples
here,
we're
using
good
old-fashioned
sql
and
we're
treating
everything
in
the
traditional
almost
like
data
warehousing
style,
queries
right,
in
other
words,
what
you're
used
to
within
that
realm
of
running
sql
and
and
database
relational
database
style.
It
all
comes
into
play
this
exact
same
example.
Everything
about
it
so
just
want
to
call
that
out
for
some
folks
are
wondering
hey
we're
using
databricks.
C
E
D
E
Here's
an
example
right
where
you
can
like
develop
as
a
team.
So
in
this
case,
franco
is
developing
some
queries
and
maybe
we
are
doing
some
code
review
or
we
are
developing
together
in
different
cells,
but
you
can
probably
I
can
go
ahead
and
change
the
code.
I
I
can
put
some
comments
and
then
we
can
work
in
a
collaborative
fashion.
B
B
B
So
what
I'm
going
to
be
doing
next
is
and
feel
free,
if
you
guys,
if
either
of
you
want
to
you,
know,
take
over
and
start
explaining
things
better
because
I
put
this
stuff
together
late
last
night
anyway.
So
we
need
to
configure
hyperloop
to
work
right
and
to
do
that.
We're
going
to
be
issuing
a
couple
commands
on
the
the
driver,
node-
and
I
do
that
in
data
bricks
by
issuing
the
magic
command
sh
percent
sh.
B
D
Yeah,
so
this
is
just
basically,
you
know
we
jumped
into
a
shell
into
the
driver,
node
and
basically,
what
we're
doing
is
installing
some
of
the
dependencies,
like
I
said
under
the
covers,
hyperloop,
really
leverages
the
the
tableau
hyper
api
and
allows
us
to
create
a
hyper
file
locally
on
this
driver.
Node.
B
E
So
a
lot
of
folks
may
not
be
aware
of
this,
but
if
you
remember
in
the
old
days
tableau
had
like
this
tds
files
or
tdsx
or
tableau
extracts,
they
are
essentially
upgrading
that
tdx
file
to
a
hyper
file
format,
which
is
like
a
columnar
compressed
file,
format
native
to
tableau,
and
what
bill
is
doing
here
is
creating
a
library
and
we
are
all
using
the
tableau
standard
hyper
apis
right.
So
just
want
you
to
know
that
this
is
the
standard
best
practice
suggested
by
tableau,
which
has
a
hyper
apis.
B
Thanks
so
hon
absolutely
actually
funny
story.
I
was
at
the
tableau
conference
the
year
that
they
announced
hyper
and
my
favorite
song
from
scooter
hyper
hyper
was
playing
on
the
speakers
almost
the
entire
conference.
It
was
epic
but
funnily
enough,
the
hyper
team,
which
is
from
germany,
which,
if
you
know
anything
about
edm
scooter,
is
from
germany
and
I'd,
ask
the
guys
I'm
like
so
you
like
scooter
and
they
were
like.
No,
I
don't
even
know
what
the
song
is
and
I
was
like
still
right
now
anyway.
B
So
what
I'm
going
to
be
doing
here
is
creating
the
the
hyper
file.
So
I'm
going
to
go
to
the
hyper
loop
package,
I'm
going
to
say
import
import,
the
hyperfile
command,
so
I
can
create
this,
and
what
I'm
going
to
do
is
I'm
just
going
to
put
sql
into
that
query.
I'm
going
to
kind
of
put
it
into
the
variable
query
and
it's
the
same
sql
command
that
I
had
run
up
here
just
to
kind
of
see
my
my
data.
I
can
see
my
data
with
inside
data
bricks.
B
That's
the
data
I
want
inside
tableau,
so
I'm
just
going
to
copy
that
sql
statement
here,
put
it
into
a
variable
and
then
tell
the
hyperfile
command.
I
want
you
to
call
it
tpch
extracts,
that's
what
we're
dealing
with
here.
That's
my
query.
Well,
do
you
want
to
tell
us
about
these
two
commands
here.
D
Yeah
sure
so,
at
the
core
of
everything,
just
like
you
were,
you
know,
run
this
on
a
tableau,
desktop
client,
yeah,
you're
gonna
create
an
extract
at
the
end
of
it
will
be
a
hyper
file.
So
this
is
the
main
kind
of
object
here
and
basically,
what
dbfs
enabled
flag
does
is
it
tells
the
hyper
hyper
engine
that
basically,
you
know
the
file
system
is
a
databricks
file
system.
D
This
can
you
know
you
can
easily
download
this
and
run
this
on.
You
know
any
any
ec2
instance,
or
you
know
locally
on
your
own
computer
as
well,
and
you
basically
would
say
you
flip
that
flag
and
say
it's
just
a
local
file
system
and
then
creation
mode.
Basically,
there
are
two
creation
modes
within
the
hyperloop
library.
One
is
a
little
bit
more
friendlier
to
to
things
like
null
values
and
that's
insert
so,
basically
what
it
does.
D
B
Interesting.
Thank
you
very
much.
That's
that's!
That's
quite
some
extensibility
there!
So
when
we
were
talking
earlier
about
how
long
this
stuff
would
take
on
tableau
server
to
extract
itself
we're
talking
hours
for
like
29
million
rows
right
all
right.
So
in
this
example
here
I
did
some
configurations.
Some
special.
You
know
performance
tuning.
B
Are
they
hacks
or
are
they
tricks?
I
don't
know
what
you
call
them,
but
I
kind
of
took
my
knowledge
of
what
was
going
on
with
this
project
here
and
I
created
a
special
cluster
for
it
and
I
actually
got
perfect
to
create
the
extract
file
in
like
eight
minutes.
You
can
see
here
this
command
for
29
million
rows
took
about
nine
eight
nine
minutes
now
we
want.
B
D
Yeah
correct
so
the
hyper
hyperloop
api
has
a
couple
handy
functions
within
it,
so
you
know
right
now,
like
you
just
said
franco,
the
hyper
file
is
currently
written
to
local
disk
on
on
this
cluster.
But
if
it's,
if
this
cluster
is
a
thermal,
gets
shut
down
and
basically
your
hyperfile's
gone,
so
I
have
a
function.
D
There
called
save
where
you
could
basically
save
this
to
dbfs
or
s3
in
this
case,
and
you
know,
come
back
the
next
day,
load
it
up
and
then
append
new
data,
as
it
arrives
near
delta
lake
to
this
this
hyper
file.
So
you
don't
have
to
go
back
and
recreate
it.
Every
time
you
can
save,
it
persist
to
the
cloud
storage
reload
it
and
then
append
new
data
as
it
arrives.
D
E
Tableau
every
every
every
night,
let's
say
once
they
discover
this
customers
love
it
because
now
the
extract,
let's
say
your
day:
zero
day:
one
load
is
on
dbfs
you're,
just
doing
the
daily
increment
appends
to
it,
and
then
you
then
you're
publishing
it
so
massive
saving.
In
terms
of
your
etl
pipeline,
it's
almost
like
a
cdc
into
tableau,
which
is
super
cool.
B
D
B
That's
awesome
so
that
I
know
that
back
in
some,
you
know
back
in
the
previous
life
that
the
backgrounder
is
on
the
same
server
as
it's
used
to
actually
serve
the
dashboards,
so
people
trying
to
get
it
to
a
dashboard
could
actually
that's
on
the
server
could
actually
be
throttled
by
the
back
the
background
the
extract
processor
running
so
this
could
actually
help
alleviate
stress
on
your
tableau
server.
So
you
could
get
more
people
concurrently
running.
Is
that
right.
D
B
Awesome
so
I
just
want
to
take
a
look
at
this
x,
so
we
said
on
tableau
server,
29
million
rows.
It
could
take
hours
order
manager
about
three
hours
to
do
a
refresh
over
the
wire
and
just
to
show
you
what
this
took.
It
took
eight
minutes,
eight
nine
minutes
to
prepare
the
hyper
file
and
then
to
transmit
it
to
tableau
over
the
internet.
It
took
three
about
four
minutes,
so
in
the
matter
at
night,
13
minutes
compared
to
three
hours
is
significantly
better.
E
Yeah
and
and
to
your
point
all
this
was
happening
initially
on
tableau,
server,
right
and
and
now
now
you
have
essentially
taken
a
big
part
of
the
processing
to
data
breaks.
You
can
run
it
in
parallel
on
on
ephemeral
clusters,
so
you
you
get
the
job
done.
You
shut
down
the
clusters
right
so
so
much
more
efficient
compared
to
doing
it
on
the
tableau
servers.
B
Awesome
and
then
now,
let's
just
say,
I
want
to
do
an
incremental
refresh
right,
so
I've
got
data
constantly
coming
in,
for
here,
I'm
just
going
to
emulate
it,
but
I'm
going
to
create
a
another
data
frame
of
my
data
that
I
previously
had.
But
this
time
I'm
going
to
change
the
process
date
to
10
20..
B
So
this
is
today,
I
processed
yesterday's
data,
but
now
I'm
going
to
put
today
data
today's
data
in
there
and
I'm
just
going
to
sample
this
down
a
very
common
function
that
I
like
to
use
in
the
field
is
sample
when
just
working
with
scales
of
data
by
the
way
funny
story,
you
can
actually
do
this
to
beef
up
a
data
frame.
So
if
you're
trying
to
test
scale
on
data
bricks,
you
can
arbitrarily
increase
the
size
of
a
data
frame
as
long
as
it's
a
decimal.
So
it
has
to
be
ten
thousand
point
one.
B
B
If
you
want
to
do
it
anyway,
so
here
I
just
basically
grabbed.
I
don't
even
know
what
that
is
point.
Zero
zero!
Is
that
a
a
millionth,
a
hundredth
a
thousand
anyway
I'm
off
on
my
math,
and
I
want
to
just
sample
it
down
just
a
couple
hundred
rows
and
then
I
want
to
append
it.
So
I
appended
the
data
frame.
I
appended
the
data
frame
to
that
and
it
only
took
25
seconds
to
basically
take
do
a
daily
increment
to
my
file
and
then
what
I
could
do
is.
B
B
Here
is
my
the
site
I
created
for
this,
and
this
is
a
sample
dashboard
of
tpch
data.
I
know
it
looks
very
homogeneous
and
there
aren't
very
many
outliers.
This
is
sample
data
right.
It's
emulated
data,
so
it's
not
really
going
to
show
anything
if
you
have
any
ideas
on
how
to
make
this
look
more
interesting,
I'm
really
open
to
them.
B
But
anyway,
we'll
take
a
look
here
and
essentially
when
I
loaded
my
extract
file,
it
got
put
here
to
my
extracts,
and
this
is
where
I
can.
I
can
see
my
extract
file,
so
my
extract
loaded
from
data
bricks
into
tableau
and-
and
this
is
where
it
is-
and
now
I
can
start
working
with
my
extract
file
on
the
fly
and
I
basically
sent
that
over
with
hyperloop.
C
No,
I
think
it's
just
really
nice.
So,
let's,
let's,
let's
just
dive
into
some
questions,
I'll
start
with
zoom
and
then
I'll
switch
over
to
to
linkedin
live
the
first
question,
actually
that
they
end
up
asking
on
first
call
out
on
tpc
was
that
tpch
is
really
boring.
So
that's
true
that
that's
actually
completely
fair
and
we're
gonna.
We
got
no
problem
with
that.
C
So
that's
that's
fair,
but
we
actually
now
have
an
actual
technical
question
and
I'm
going
to
direct
that
to
you
will,
since
you
had
helped
you.
This
is
your
pr
your
project.
What
is
the
size
of
the
hyper
file
out
of
care?
Actually
like?
Does
it
do
any
compression
as
you're,
basically,
for
example,
that
that
29
million
million
rows
that
was
initially
extracted?
D
Yeah,
so
you
know
with
regards
to
the
actual
core
technology:
it's
it's
proprietary
format,
tableau's
hyper
format.
I
you
could
easily
create
another
collab
lab
on
it,
but
basically
you
know
it's
it's
very
similar
to
parquet,
I
would
say
it's
calmer
store.
It
has
better
compression
and
for
a
size,
a
file
this
size
that
was
created
by
franco,
you're.
Looking
at
the
order
of
a
few
megabytes.
D
Yeah,
it's
an
excellent
question,
so
currently
it
only
supports
a
pen.
However,
the
future
road
map
will
support
upserts
in
the
future,
so
you'll
be
able
to
do
things
like
delete,
sub
updates
and
insert
now.
This
is
a
library
that
I've
created
in
my
spare
time.
So
it's
really
just
you
know
development
life
cycle,
for
this
is
really
just
my
own.
The
free
cycles
that
I
have
yeah
so
but
upsert
is
coming.
C
Perfect
actually,
franco,
why
don't
you
show,
since
you
happen
to
have
the
link
there
on
the
a
goodwill
punting?
That's
awesome!
I
love.
I
love
the
name
of
your
github
repo
dude
goodwill.
Funny,
that's
sweet,
yes,
and
so,
basically,
as
will
called
out
here,
is
the
code
base.
It
explains
why
hyper
extracts
are
so
slow
and
the
fir
sort
of
data
extracts
are
so
slow.
So
all
the
explanations
is
all
here:
take
a
look
at
it.
C
So
we
also
have
some
quick
logistical
questions
which
is:
can
we
get
a
replay
of
this
webinar
so
for
anybody
that
actually
wants
to
watch
this
webinar?
It's
actually?
Okay,
because
we
have
this
already
posted
to
youtube
live
as
we
speak,
so
you
can
watch
it
there
and
at
the
exact
same
time,
we're
posting
this
on
linkedin
live
as
well.
So
in
fact
we're
monitoring
all
the
questions.
C
So
if
you
have
any
more
questions
by
all
means,
please
chime
in
here
we're
looking
at,
like
I
said,
we're
looking
at
zoom,
we're
looking
at
youtube,
live
and
we're
looking
at
linkedin,
live
and
say
that
three
times
fast,
all
right,
so
I'm
gonna
continue
right.
Now,
I'm
just
trying
to
see
if
there's
any
other
questions.
E
Well,
I
guess
I'll
say
you
know,
I'm
thinking
more
about
like
with
malcolm
gladwell
and
all
the
reading,
like
I'm
thinking
more
about
like
happiness,
questions
right.
So
what
are
the
personas
that
make
that
are
like
happy
with
hyperloop
right?
So
so
the
bi
developers
are
happy
because
they
don't
have
to
use
tableau
desktop
to
do
all
this
right
that
the
data
engineers
are
happy
because
directly
from
databricks
they
can
write
to
tableau
right.
E
E
B
No,
that's
absolutely
right.
I
love
it.
That's
why
I
I
really
do
enjoy
working
here
at
databricks,
and
let's
segue
into
that.
If
you
are
currently
looking
for
a
new
adventure
here
at
databricks,
we
are,
we
are
hiring
so
if
you're
interested
feel
free
to
reach
out
we're
always
looking
for
for
good
people
to
to
join
the
cause
and
to
help
us
solve
the
world's
toughest
problems.
Just
like
will
here
created
this
awesome
project
and
soham,
and
I
are
out
there
trying
to
help
the
good
people.
C
Good
call
out,
no,
I
was
about
to
say
I
wonder,
and
before
we
transfer
to
karen,
I
did
want
to
call
out
exactly
to
your
point
if,
since
you're
all
into
vidcast
and
podcasts,
don't
forget
because
malcolm
gladwell
is
speaking,
you
know
back
to
that
revisionist
history,
excellent
podcast,
it
is
very,
very
enjoyable.
So
I
highly
highly
advise
you
guys
to
go.
Listen
to
that.
So
with
that,
I
think
we've
answered
all
the
questions
will
soham
thanks
very
much
for
this
awesome
session.
Franco!
Amazing
demo.
Thank
you
very
much.
A
All
right
thanks
so
much
franco
and
denny,
and
and
thanks
will
and
soham
for
joining
us
and
thanks
everyone
for
attending.
So
I
I'd
like
to
direct
you
to
the
youtube
live.
So
if
there's
any
kind
of
outline
questions,
please
drop
them
there
in
the
comments
once
this
broadcast
is
over
and
thanks
everyone
for
joining
us
hope.
You
all
attend
our
session
next
tuesday
and
use
that
code
for
for
summit.
We'd
love
to
see
you
there,
some
great
speakers.