►
From YouTube: Tech Talk | Simon and Denny Build a Thing! Part One
Description
Join us for the brand new tech talk series "Simon and Denny - Build A Thing!" where across a series of tech talks, we will build a data platform from ingestion to ETL to analytics and data science using the Azure Databricks platform. With our background in SQL Server and BI to Apache Spark and Delta Lake - we want to show you how to build your own lakehouse.
We would like this session to be interactive so come prepared to ask questions all throughout the session - the very first one will focus on the ingestion of data. Our conversations together will also guide what we will discuss in the next session! Be prepared for another geeky, trans-Atlantic event from two data nerds.
Github link: https://github.com/databricks/tech-talks/tree/master/courses/build-a-thing
Speakers:
Simon Whiteley - Microsoft Data Platform MVP, Databricks Nerd, Cloud Herder, one of those “London People”. Simon runs Advancing Analytics, a UK data consultancy as their Director of Engineering, hosts the Advancing Spark youtube channel and organises the Microsoft Data London meetup. He spends most of his time working with companies to apply "big data" thinking to traditional analytics problems, whether through cloud architecture, spark development or Azure wrangling. When not tinkering with tech, Simon is a death-dodging London cyclist, a sampler of craft beers, an avid foodie and a lover of all things nerdy.
Denny Lee is a developer advocate at Databricks, where he works on Delta Lake, Apache Spark, Data Sciences, and Healthcare Life Sciences. He has previously built enterprise DW/BI and big data systems at Microsoft including Azure Cosmos DB, Project Isotope (HDInsight), and SQL Server as well as the Senior Director of Data Sciences Engineering at SAP Concur. Denny holds a Masters in Biomedical Informatics from Oregon Health Sciences University. Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
A
A
B
A
C
C
No
we're
good
yeah
we're
good
danny.
I
just
dropped
you,
the
linkedin
live
and
the
youtube
live
links
so
welcome
everyone.
Thanks
for
joining
us,
the
spaces
were
just
kind
of.
I
was
just
getting
set
up
here.
I'd
love
to
hear
where
you
all
are
joining
from.
I
know
a
few
folks
are
already
chiming
in
and
linkedin
since
I
have
that
that
window
open.
C
We
have
some
few
folks
from
india
from
paris
welcome
and
then
I
also
I
just
had
a
pole
open
in
in
the
zoom
to
see
if
this
is
your
first
time
joining
us.
So
please
please
chime
in
and
let
me
know
we
always
like
to
see
if
you're,
a
newcomer
or
if
you've
joined
us
before
either
way
we're
happy
to
have
you.
So
I
am
oh
doug
from
sunny
discovery
bay,
where
it's
five
o'clock,
where
it's
always
five
o'clock-
five
p.m.
In
london,
some
folks
from
dallas
italy
cool
all
right.
C
Well,
I
just
have
a
few
intro
slides,
I'm
going
to
go
ahead
and
get
started
here
and
I
have
a
poll
so
I'm
going
to
stop
this
poll
for
now
looks
like
a
lot
of
folks.
Are
this
is
your
first
time
joining
us,
so
welcome
hope?
Hopefully
you
enjoy
the
session
and
you'll
you'll
come
back
next
time.
We
have
actually
a
lot
of
upcoming
sessions
for
the
rest
of
this
month,
which
I'm
really
excited
about
so
cool.
Okay.
C
A
No
problem
at
all,
as
you
hunt
down
for
your
slides,
welcome
everybody
for
attending
simon
and
denny
build
a
thing.
Now:
it's
not
a
random
thing.
We
are
not
going
to
be
doing
legos
today,
though
honestly
that
would
have
been
a
really
cool
thing
to
go.
Do
so,
but
instead
no
hey,
hey,
you've
got
your
scent.
I've
got
my
rastanante
here.
I
don't
know
if
you
can
tell,
but
I've
got
the
rasta
dante
here.
So
yeah
no.
A
Good
yeah,
so
instead
we're
actually
going
to
be
building
a
data
thing:
okay,
so
yeah
and
hopefully
that's
exciting
for
everybody
here.
So
my
name
is
denny
lee.
I'm
a
developer
advocate
at
databricks
and
long
time,
spark
guy
and
delta
late
guy
and
lake
house
guy,
even
though
the
term's
relatively
new
simon.
Why
don't
you
introduce
yourself,
hello,.
B
More
than
that,
okay
yeah,
I'm
sam
white
consultant
in
the
uk
run
a
company
called
advancing
analytics,
run
a
youtube
channel,
which
is
apparently
how
I
spend
most
of
my
time
these
days
talking
about
spark
and
data
right,
san
azury,
datay
things
and
yeah.
I
do
a
lot
of
data
lake
housing
type
stuff
and
it's
quite
fun,
but
it
can
be
a
little
bit.
B
A
A
A
Okay,
perfect,
all
right,
we
can
definitely
chat.
Also,
there
is
for
those
of
you
that
are
on
youtube
or
linkedin
go
ahead
and
chime
in
through
the
messaging.
The
chats
that's
there
as
well,
we'll
do
our
best
I'll,
be
the
one
monitoring
them
because
simon's,
the
actual
one,
with
the
knowledge
I'm
just
talking
ahead.
That's
all
this
is
so
at
least
that's
the
one
I
play
on
tv,
so
that
should
just
give
you
some
quick
context,
not
none
whatsoever.
It's
all
on
youtube.
Everything's
on
you,
okay,
understand
that
all
right.
C
C
C
Right
all
right,
so
this
is
part
one
in
a
new
series
with
simon
and
denny
part.
One
is
data
ingestion
and
I
wanted
to.
I
like,
I
always
like
sharing
the
links
to
our
meetup
group
and
our
youtube
channel
and
our
linkedin,
because
I
know
we
have
folks
joining
from
all
three
different
spaces
and
really
I'll
drop
these
all
in
the
chat.
But
I'd
love
for
everyone,
who's
joining
us
on
youtube
and
linkedin
to
join
our
online
meetup
group.
We
have
a
lot
of
really
great
upcoming
sessions
and
you've
joined
the
group.
C
You'll
get
notified
exactly
when
we
announce
events.
I
just
announced
two
this
week
alone
so
and
it's
also
fun
too,
to
join
the
zoom
stream
as
well.
We
try
to
do
a
few
different
polls
and
interact
with
the
chat
and
so
forth.
So
join
our
group
and
subscribe
to
our
youtube
channel.
We
have
all
of
our
recordings
there,
so
I
invite
you
to
check
out
that
all
the
sessions
that
we
run
live
are
recorded
and
hosted
there.
C
So
another
thing,
I
think
everybody
probably
notices
my
background.
Hopefully
it's
matching
the
data
in
ai
summit.
2021
registration
is
now
open,
so
that's
coming
up
in
the
end
of
may
and
we'd
love
for
you
to
join
us
there,
so
you
can
register
now
I'll
drop
the
link
to
register.
We
have
five
days
with
over
200
sessions
of
content,
we're
going
to
be
hosting
a
couple
meetups
and
I'm
pretty
sure
the
two
two
folks,
simon
and
denny,
who
are
joining
us
here,
might
make
an
appearance
on
those
beat
ups
as
well.
C
So
we'd
love
to
have
you
and
as
we
kind
of
go
through
the
weeks
of
our
these
online
meetups
and
as
I
get
more
details
from
the
team,
I'll
share
them
with
you.
So
if
we
have
cool
speakers
announced-
or
you
know
different
things
that
I
I
think,
you'd
like
you'd-
be
interested
in
I'll
show
them
as
I
get.
The
information
so
register
now
hope
you'll
join
us
there
and
then
quick
call
out.
C
So
I
I've
plugged
this
a
few
times,
but
so
we're
running
a
few
community
lighting
talk
sessions
and
we
also
have
community
lightning
talks
at
our
data
in
ai
summit.
So
I'd
really
love
to
run
two
sessions
leading
up
to
summit,
and
so
I
invite
you
to
submit
a
topic
really
it's
it's
open
to.
You
know
what
what
you're
working
on
we
want
to
see.
What
you're
working
on
and
a
lightning
talk
is
about
10
to
15
minute
presentation.
So
it's
really.
C
You
know
a
really
great
intro,
if
you're
interested
in
speaking
and
giving
kind
of
a
short,
quick
talk,
and
then
we
have
a
few
minutes
after
for
q
a
so
it's
a
really
great
way
to
kind
of
get
your
feet
wet
to
giving
a
talk
and
and
sharing
what
you're
working
on.
So
we
want
to
hear
from
you
and
also
if
you
do
submit
a
talk
through
that
link
and
it's
accepted
you'll
receive
some
swag.
So
look
out
for
that.
C
I
hope
I
hope
you'll
submit
your
project
and
with
that
I'm
going
to
stop
my
my
screen
share
and
just
do
a
quick
poll,
because
I'd
love
to
reach
out
to
you
with
more
details
on
giving
a
lightning
talk.
So
I'm
going
to
quickly
launch
that
poll
for
everyone
on
youtube
and
linkedin
if
you're
interested
in
giving
a
lightning
talk,
I'd
love
love
to
give
you
the
information
too
I'll
drop
the
link
there,
but
you
can
always
reach
out
to
me
directly
on
linkedin
and
I'm
happy
to
to
share
the
details
with
you.
C
C
Okay,
we
have
a
few
folks
on
zoom
that
are
interested
in
signing
up
to
give
a
lightning
talk,
but
it
looks
like
majority
are
excited
about
attending
the
session
and
cheering
on
the
other
presenters.
So
that's
awesome
too
cool.
Well,
I
think
I've
given
that
long
enough.
So
with
that,
I'm
going
to
end
this
poll
and
pass
it
over
to
simon
and
denny.
C
A
So
I'll
start
first
just
to
provide
context
before
you
do
all
the
actual
work.
Okay,
I
just
want
to
call
that
out.
Okay,
since
you
know
I
don't
actually
work
here
so
for
starters,
hi
everybody,
my
name
is
denny.
We
want
to
remind
you
that
we
are
doing
build
a
thing
now,
I'm
going
to
share
a
screen
and
I'm
actually
going
to
send
a
link,
the
link
to
all
three
channels
to
youtube
to
linkedin
as
well
to
zoom.
A
This
is
our
github
page,
where
we're
going
to
everything
we're
going
to
be
talking
about
during
this
series
is
actually
going
to
be
propped
right
into
this
github
page.
So
that
way
you
have
the
latest
information.
Okay,
so
we're
going
to
talk
about
data
ingestion
today,
actually
that's
our
primary
goal,
but
obviously
that's
not
the
end
goal.
We're
going
to
actually
ask
you
folks
to
we're
going
to
put
a
poll
at
the
near
the
end
of
this
particular
session.
A
Where
we're
going
to
ask
you
what
areas
you
want
us
to
dive
into,
whether
it
is
data
ingestion,
specific
or
etl
or
autoloader
or
anything
else,
and
as
we
progress
the
whole
end-to-end
pipeline
from
data
ingestion,
all
the
way
down
to
machine
learning
and
bi
we're
going
to
cover
all
the
bases.
So
for
those
of
you
that
are
going
like
hey
I'd,
love
to
go
ahead
and
ask
questions
like,
for
example,
there's
some
folks
who
are
asking
about
spark
python
versus
koalas
versus
spark
sql
right.
A
We're
not
going
to
answer
that,
but
not
because
we
don't
want
answers
right
now,
because
we
just
want
to
focus
on
data
digestion.
But
yes,
as
we
progress
down
we're
going
to
talk
about
things
like
koalas
and
sparks
equal
and
python,
and
things
of
that
nature.
Okay,
so
we're
just
we're
not
going
to
do
it
right
this
second,
okay,
so,
like
I
said
we're
going
to
send
you
that
link
and
the
end
goal
will
be
something
like
this
now.
A
It
won't
be
this
because
we're
not
going
to
be
using
this
data
set
we're
actually
going
to
be
using
a
very
different
data
set.
But,
for
example,
here
is
a
view
of
a
sql
analytics
dashboard.
This
is
based
on
the
2014,
yellow
cab,
new
york
city
taxi
data
set,
and
so
basically
we
have
like
26
million
26.9
million
pickups
with
drop-offs.
We
can
create
cool
little
graphs.
A
That
show
us,
for
example,
that
the
vast
majority
of
the
rides,
the
up
from
the
new
york
city
taxi,
were
done
by
a
credit
card
and
the
other
part
would
be
cash
and
then
the
rest.
It
is
what
it
is.
Average
trip
amount
payment,
but
then
also
funky
things
like
maps.
Okay,
so
we
can
dive
into.
What's
the
the
ride?
Sets,
that's
actually
happening
within
pickup
location
groups
by
rise
from
the
from
the
airport.
For
example,
sorry,
that'll
be
allergies
all
right.
A
Nevertheless,
excuse
me
it
tells
you
that
we're
live
at
least
nevertheless,
that
that
we're
going
to
be
going
ahead
and
actually
get
building.
Something
like
this
live
as
well.
Okay,
so
that's
not,
but
that's
not
what
we're
gonna
start.
We're
gonna
start
on
something
a
technical,
a
little
bit
more
boring,
which
is
more
on
the
data
ingestion,
but
extremely
important.
A
B
B
So
that
is
the
plan
in
we've
got
essentially
we're
scraping
some
data
and
we're
landing
it
into
a
blob
as
our
little
starting
point,
that's
kind
of
like
our
first
step
now,
we've
cheated,
we
have
I've,
gone
and
built
this
thing,
there's
a
little
little
bit
of
data
factory
working
away
individual
and
that's
like
the
littles.
Basically
it's
a
little
web
scraper.
So
I'm
going
to
give
you
a
little
bit
of
a
tour
of
the
thing.
B
A
I
think
so
I
think
it's
extremely
accurate
and
then
by
by
the
same
token,
the
the
movies
that
we
started
off
with.
I
don't
know
if
you
were
to
cover
that
yet,
but
the
movie
and
tv
shows
also
will
probably
give
you
a
little
a
view
of
what
how
exactly
simon
and
or
my
brain
works,
which
is
not
probably
not
the
best
thing
right.
It's
not
healthy
or
not
good
yeah.
I
don't
know
that's.
B
A
B
So
I
think
we're
not
really
going
to
go
into
over
on
our
side
sitting
in,
like
my
little
scrappy.
Sandbox
area
is
a
little
thing
that
essentially
calls
an
api,
calls
the
manipulator
and
says:
well,
that's
just
a
little
look
up
to
a
json
file,
so
I've
got
a
little
lake
of
my
own
looking
at
a
jason
fire,
getting
just
a
list
of
series
and
going
what
tv
shows
are
we
tracking?
B
So
we've
got
this
list
and
it
says
like
game
of
thrones
season,
one
given
throughout
season
eight
just
for
some
controversy,
it'll
just
say
how
many
episodes
of
each
of
those
things
and
then
for
each
of
those
episodes
the
little
copy
activity
that
goes
off
and
says
calls
that
api
says
this.
Is
the
id
bring
me
back
the
ratings
for
that
particular
episode
and
then
it
lands
it
down
as
a
json
file.
B
So
it's
actually
long
story
short,
there's
a
load
of
stuff
going
on
and
we
end
up
in
a
blob
store
at
basically.
The
start
of
our
story
is
a
blob
store,
that's
just
receiving
data,
and
that
I
don't
know,
but
you
guys
that
happens
to
be
a
lot
in
terms
of
I'm
kind
of
like
land
of
the
clients
and
like
so
we
get
sent
data.
We
have
no
control
about
it.
We
just
get
data
landing
in
this
bit.
We
need
to
go
from
there
and
that's
kind
of
what
the
story
we're
starting
with.
B
A
B
B
Just
written
some
code
to
call
it
and
land
it,
but
lazy.
I
could
have
written
a
logic
app
if
you
don't
do
that,
which
is
the
same
kind
of
thing
and
it's
much
more
lightweight
logic.
Apps
are
kind
of
better
at
understanding
things
like
json
doing
that
stuff,
but
I
guess
kind
of
tend
to
use
data
factors
it's
better
when
data
factory
is
better
for
doing
long,
running
data
processing
tasks.
B
So
for
this
thing,
when
it's
just
doing
a
little
web
scrape
thing,
it's
it's
like
a
sledge
hammer
for
a
note
right,
but
because
I
tend
to
do
lots
and
lots
of
data
bricks,
things
and
kicking
off
a
job
and
managing
that
job
while
it
runs,
it
makes
sense
to
use
the
same
tool.
So
you
can
have
like
an
end
to
end
pipeline.
That
comes
everything,
so
I
tend
to
use
data
factory
by
default,
because
that's
what
I
do
most
of
my
work
in
I.
A
It's
fine,
no,
no,
that
by
the
way,
in
all
seriousness,
even
though
simon
is
saying
ingest
that
it's
lazy,
it's
actually
we're
developers
so
yeah
we're
all
lazy.
Let's
be
honest,
right
we're
going
to
code
our
way
out
of
the
problem
and
we're
going
to
reuse
what
we
can
and
adf
is
an
excellent
lazy
way
to
do
it
in
the
azure
ecosystem.
Right
now,
no
there's
a
call
up
from
james.
The
best
developers
are
lazy,
absolutely
agree,
one
thousand
percent.
A
Oh,
I
just
noticed
hi
from
audi
by
the
way,
hey
adi,
how
you
doing
all
right,
but
nevertheless
the
context
here
is
that
right
now
we're
obviously
using
azure
database.
So
if
you're
using
google
or
if
you're,
using
aws,
we're,
probably
not
going
to
use
adf,
that
would
be
not
not
as
optimal
but
nevertheless
we
are
using
azure
databricks.
So
that's
why
we're
going
to
be
talking
about
that
using
that
as
our
starting
point?
B
Ecosystem
to
get
this
working,
oh
good
point!
Yes,
yes,
yes,
yes,
yes,
so
we've
got
an!
Is
your
subscription!
This
thing
we
we
have
our
builder
thing
resource
group
that
we're
working
from.
So
this
is
where
we've
got
all
our
different
pieces
and
again
no
data
factory
in
here,
so
the
data
factory
is
living
on
in
a
different
tenant
and
it's
just
delivering
data
as
if
it
was
some
third
party
thing
thing.
We
need
to
think
a
lot
in
this.
A
Yes,
so
so,
by
the
way
this
we
are
freely
emitting
our
lack
of
creativity
when
it
comes
to
naming
things.
So
so,
basically,
why
don't
we
explain
what
some
of
these
things
are
so
to
provide
a
little
context
so
as
as
simon
noted
right
now
we're
starting
off
with
our
build
a
thing
resource
group
within
azure,
our
azure
subscription,
but
it's
not
just
about
azure
data
bricks,
which
is
the
build
a
thing
dash.
Ws
ws
stands
for
workspace.
B
We've
got
distributed
cheap
storage,
so
we
can
just
dump
some
data
and
work
with
it
right
and
we've
got
two
different
sites,
so
they're
both
storage
accounts,
but
if
you're
building
a
lake,
if
you
want
to
have
security
in
the
lake,
you
need
a
thing
in
azure
called
hierarchical
main
space,
which
is
like
a
super
fancy
way
of
saying.
I
want
to
be
able
to
have
folder
objects
and
then
they
can
be
secure.
So
if
you've
got
like
normal
storage
in
azure,
then
you
end
up
with
a
blob
like
this.
B
C
B
See
season
8,
blah
blah
and
there's
a
lot
of
data
going
and
landing
there,
but
actually
that
whole
structure
all
of
that
stuff.
That's
just
it's
a
virtual
folder.
So
that's
part
of
the
file
name
that
then
the
browser
knows
to
kind
of
pull
that
apart
and
make
it
look
like
folders,
but
it
means
I
want
to
say
you
know
that
department's
over
there
you
have
access
to
this
folder,
whereas
you
can
read
that
folder.
You
can't
write
to
that
folder.
I
can't
do
that
in
here.
B
C
B
There
you
go
so
you
can
see
all
right,
album
data,
lake
storage,
gen
2.
It
has
hierarchical,
namespace
enabled
which
is
really
confusing,
because
loads
of
people
go
into
azure
and
like
I
need
to
build
a
data
lake
and
then
there's
no
option
for
data
lake
data.
Lake's,
not
a
thing
it
used
to
be.
You
have
deadly
gen
1
was
a
an
object
that
you
create
and
that's
a
data
lake.
Now
you
don't
have
a
data
lake
object.
You
just
have
a
storage
account
that
you
flick
a
switch
when
you
create
a
gang.
B
A
Yes
and
so
to
add
exactly
the
simon's
point
right.
So
the
background
is
that
if
you
recall
basically
we're
originally
started
off
with
blob
storage
right
and
then
then
it
went
to
adls,
gen
1
right
and
which
was
exactly
what
simon
was
talking
in
terms
of.
Let's
build
a
data
lake
versus
just
storage,
and
then
we
went
and
said:
okay
now
we
want
to
actually
do
adl's
gen
2,
but
it's
a
little
confusing,
because
even
though
it's
called
adl's
gen
2,
it
basically
is
blob
storage
with
a
hierarchical
namespace.
A
So
you
actually
have.
When
you
go,
create
the
storage
account,
you
actually
have
to
click
on
it
and
then
you
all
actually-
and
simon
was
the
one
that
reminded
me
by
the
way,
because
I'm
older,
so
I
actually
forgot
some
of
this
stuff.
So
simon
reminded
me
that
I
actually
had
to
click
on
enable
the
hierarchy
when
I
did
80
gen
2,
because
that
was
no
longer
by
default
yeah.
I
want
you
to
show
it
because.
A
B
B
A
B
B
But
it's
much
better
in
that
defense
because
back
when
they
had
just
their
leg,
store
gen
one
and
it
was
a
separate
thing:
it
had
a
different
api
to
blob
storage
and
it
had
different
hardware
to
blob
storage,
and
so
you
couldn't
get
it
in
all
of
your
regions
and
you
had
to
use
different
protocols,
maybe
used
if
you
had
something
built
to
put
a
file
in
the
blob
store.
You
couldn't
use
that
with
the
data
lake
storage.
B
A
Which
makes
sense
perfect
so
by
the
way,
there's
some
comments
on
linkedin.
So
let
me
go
ahead
and
actually
chime
in
for
a
second
just
to
answer
some
of
those
questions.
Some
there's!
Actually
it's
pretty
active
here
in
linkedin.
So
thank
you
for
the
folks
that
are
actually
going
ahead
and
calling,
but
I
did
think
I
did
want
to
call
up
camp
audi
actually
asked
a
great
question,
I
think,
is
a
great
for
everybody
to
know.
As
the
can
you
process
data
from
s3
with
azure
databricks.
Yes,
you
absolutely
can
azure.
A
Databricks
can
definitely
connect
to
not
just
blob
storage
or
adl's
gen
two.
He
can
absolutely
talk
to
s3
and
absolutely
talk
for
that
matter.
Google,
cloud
storage
and
for
that
matter,
if
you're
on
aws,
you
can
definitely
talk
to
the
database
on
aws
or
databricks
on
google.
You
can
absolutely
talk
to
s3
or
adl
gen,
2
or
blob
storage
or
google
cloud
storage
right.
They
you
can
all
talk
to
each
other.
A
So,
for
example,
if
you're
running
an
azure
data,
brick
system,
that's
talking
to
like
that's,
let's
just
in
east
u.s
too,
and
you're
talking
to
an
s3
account
right,
even
if
it's,
if
it's
s3
in
the
e
in
s3
in
east
for
aws,
there's
still
the
egress
cost
of
actually
grabbing
the
data,
so
in
other
words,
azure
database
has
to
go
across,
go
through
the
ada,
the
azure
network,
to
the
aws
network,
to
actually
request
that
for
that
data.
So
then,
usually
there's
like
a
higher
price
to
to
egress
that
data
out
okay.
A
Okay,
just
as
a
quick
call
out
all
right,
so
just
so
that
just
a
friday
context-
and
I
believe
sauron
had
already
answered
that
question
on
linkedin,
so
I
want
to
give
a
special
shout
out
to
him
because
he
was
the
one
already
answering
a
bunch
of
questions
linkedin.
So
thank
you
very
much
laura
for
doing
that.
The
other
question
I
did
call
out
that
I
did
want
to
answer
live
before
we
go
ahead
and
shift
back
is
that
for
data
engineering
scala
performs
ten
times
better
than
python.
A
Does
that
happen?
It
depends,
I
think,
maybe
you
may
be
referring
to
spark
in
which,
when
you're
running
spark
with
scala
versus
python,
if
you're
talking
about
rdds
or
and
it's
or
in
the
case
of
running
udfs-
that's
probably
true,
okay,
but
in
the
case
of
when
this
day
and
age,
when
we're
talking
about
spark
2.x
series
and
3.x
series,
that's
probably
not
true.
It's
going
to
be
roughly
running
almost
at
exactly
the
same.
A
The
reason
why
we're
saying
that
is
because,
basically
but
from
a
pi
spark
perspective,
it's
just
a
abstraction
layer
to
the
scala
anyways
and
so
because
of
data
frames
and
as
opposed
to
rdds,
it's
basically
the
same
speed.
Now
there
is
an
issue
when
it
comes
to
udfs,
but
if
you're
building,
pandas
udfs
or
some
people
know
those
as
vectorized
udfs,
then
bam
you're
good
to
go
right,
it'll
actually
go
ahead
and
be
able
to
be
roughly
the
same
performance
as
well.
A
So
this
we
actually
have
a
lot
of
good
blogs
that
talk
about
that,
so
highly
check
it
out,
just
go
to
databricks
blog
and
look
for
vectorized,
udfs
and
or
excuse
me.
Pandas
uds
and
you'll
see
a
lot
of
good
info
about
that.
Okay,
so
want
to
answer
those
questions
live,
so
we're
done
with
that
boom.
Simon
back
to
us,
let's
go
ahead
and
continue
showing
what's
going
on
here:
cool,
okay,.
B
C
B
A
B
B
So
if
you're
not
using
azure
too
much
keyboards
are
credentials
still,
it's
essentially
where
we
can
go
and
store
any
like
account,
keys
and
certificates
and
connection
keys
and
all
that
kind
of
stuff
in
a
secure
encrypted
way,
so
that
you
know
we're
not
hard
coding,
any
secrets
in
code
and
we're
not
putting
into
github
all
our
secrets
and
stuff,
which
is
bad.
So
there's
a
load
of
stuff
we've
pre-put
in
here
to
do
with
the
service
principle.
So
how
many
connectors
out
with
the
ideas?
What
subscription
ideas?
B
All
of
that
good
stuff,
a
blob
key
of
how
we
can
connect
into
that
so
we'll
step
through
when
we
do
some
auto
loadery
stuff,
we'll
kind
of
just
talk
about
when
we're
using
these
things
and
where
you
get
them
from.
But
just
we've
set
that
up
in
advance
because
watching
us
populate
a
key
vault
and
then
getting
things
wrong
and
then
having
to
redo
it.
Probably
not
that
interesting
to
watch.
A
B
All
right
so
yeah
mention
service
principles,
so
we
have
to
get
a
service
principle
set
up
right.
So
that's
a
an
active
directory
application
so
essentially
an
unattended
account
we
can
use
to
log
in
rather
than
like
log
in
as
me
or
danny.
We
can
just
go
log
in
as
the
service
principle
and
then
they'll
go
and
connect
to
stuff.
That's
most
things.
A
In
obviously,
we
want
to
log
in
as
ourselves
when
we're
doing
it
ourselves,
but
when
you're
running
a
process
that
you
want
to
automate,
this
goes
into
that
idea
of
productionized
systems,
then
yeah
in
reality.
What
we
really
want
is
something
like
the
service
principle
as
what
simon's
showing
right
now
to
provide
that
context,
which.
B
A
A
Okay,
all
right
there,
you
go
okay,
so
now
to
everybody.
That's
currently
watching
you're
going
to
wonder.
Okay,
the
the
the
between
these
two
they're
watching
burn
notice,
counterpart
game
of
thrones
gangs
of
london,
killing
eve,
the
expanse
of
one
division,
so
yeah!
So
is
there
a
theme
here?
Actually
I
have
no
idea
if
there
is
one
fairly
nerdy
yeah,
I
guess
that's
true,
fairly
nerdy
yeah,
that's
true
yeah,
apparently
they're
already
fairly
geeky.
A
Yeah,
that's
fair
enough:
okay,
so
anyways!
Yes,
so
nevertheless,
this
is
the
data
set
we're
working
with
later
on.
We're
actually
gonna
ask
a
poll
in
terms
of
what
data
you
want
us
to
join
this
data
with
by
the
way
as
well,
so,
like
obviously,
weather
data
would
be
bad,
but
for
safe
argument.
You
know.
Other
data
sets
whether
it's
like
related
time
or
whatever
else,
but
go
ahead.
I
want
to
show
this
the
data
that
we
have
here,
yeah.
B
So
yeah,
that's
that's
a
far
away
look
of
the
kind
of
data
we're
talking
about,
and
it's
that
kind
of
thing.
It's
the
name
of
the
episode.
It's
some
basic
information
about
it.
It's
got
some
writing
information
about
it.
It's
got
ids
to
other
imdb
stuff.
We
can't
get
up
to.
We've
got
a
link
to
a
poster,
so
we
can
see
about
doing
some
funky
stuff
about
making
some
visualizations
where
you
pick
a
thing
and
it
pulls
a
poster
in
and
something
that's
right.
A
Things
yes-
and
I
I
thank
simon
for
pulling
up,
of
course,
the
expense
which
is
easily
my
favorite
show.
So
I'm
feeling,
if,
if
it
wasn't
for
the
fact
that
we
had
to
talk
about
data
right
now,
I
would
actually
just
go
on
on
a
diatribe
about
how
the
expense
is
easily
the
best
book
series
and
best
show
ever.
But
I
shall
stop
now
before
simon,
just
shuts
off
my
mic,
which.
B
B
B
A
B
B
A
B
So
all
right
go
ahead.
Let's
lock
your
workspace
perfect
and
I
guess
we
need
to
talk
about
secrets
secrets.
Yes,
yes,
because
that's
the
one
final
bit
of
setup
that
we've
done
is
inside
databricks
super
zoomed
out.
We
want
a
databrick
to
be
able
to
see
everything.
That's
in
that
key
vault
and
so
you've
got
that
kind
of
really
awkward.
I
want
to
create
a
databricks,
well
a
key
vault-based
databrick
secret,
so
that
we're
inside
our
python
code
we
can
just
write
about
things.
B
A
I
feel
I
feel
personally
offended
that
I
wanted
to
write
the
entire
thing
with
curl.
What
are
you
doing?
Man,
I'm
completely
joking
as
much
as
I
love
as
much
as
I
love
love,
actually
curl.
So
not
that
not
girl!
No
simon
is
exactly
right.
Now.
Here
is
a
secret
pun
intended
on
how
to
work
with
secrets
now
being
simon,
simon.
A
A
Okay,
got
it
all
right,
so,
just
in
case
you
can't
see
it
okay
for
everybody.
Basically,
you
type
in
after
the
url,
you
type
in
secrets,
slash,
create
scope.
Okay,
we've
highlighted
it
it
it's
not
doesn't
show
up
on
the
administration
page
at
all
yeah.
Exactly
thank
you
very
much,
simon
yeah.
It
doesn't
show
up
on
the
page
at
all,
there's
no
place
to
click
on.
You
actually
have
to
add
it
in.
I
didn't
know
this
at
all,
that's
and
dude,
but
seriously.
Thank
you
so.
B
B
But
it's
like,
if
we're
using
data
factory,
and
we
want
a
data
factory
to
use
a
user
password.
I
want
a
database
to
use
the
same
password
and
we
want
to
share
passwords
across
a
load
of
different
things
without
having
to
if
we
cycle
that
password
every
30
days,
which
we're
obviously
going
to
do
for
this
right.
B
A
B
B
B
A
And
by
the
way
for
yeah,
for
you
folks,
by
the
way
against
simon
taught
me
this
again
see
that's
why
we
keep
on
saying
he's
actually
doing
the
heavy
stuff.
I'm
I'm
not
even
sure
why
I'm
here,
it's
it's
a
it's
a
extension,
a
chrome
extension
called
az
dash
mask,
like
literally
a
z,
dash,
mask
you'll,
find
if
you
just
bring
it
or
google,
it
you'll
find
it
and
you
can
install
it
on
your
chrome
or
onto
your
edge
browser
by
the
way.
So
just
another
quick
call
out.
A
B
A
B
Right
so
then,
so
the
command
line
in
face
I've
got
it
installed.
You
can
see.
There
is
a
one
called
secrets,
so
we
can
just
go
and
check
and
say
what
secrets
are
there
list?
Scopes
tells
me
what
secret
scopes
have
already
been
installed
on
the
given
workspace,
I'm
connecting
to
and
I've
set
up
a
profile,
so
I've
set
up
a
builder
thing
profile.
B
So
if
I
give
it
that
command,
that's
going
to
connect
to
that
workspace
that
we're
looking
at
and
say
what
scopes
are
in
there
and
there
we
go
so
build
a
thing:
scope.
Here's
the
one!
I've
already
got,
we
can
see
it's
key
vault
backed.
We
can
see
it,
it's
not
obscured,
but
you
can
see
it's
multi-thing
of
vault.net.
So
we
know
that
that
databricks
workspace,
we're
working
on
is
actually
properly
associated
to
a
key
vault
meaning
when
we're
writing
some
code,
we
can
go
to
the
keyboard
and
get
some
stuff
cool.
B
A
A
That
means
you're
as
your
key
vault,
contains
the
secret
okay,
that
not
only
azure
databricks
can
use,
but
other
systems
can
go
ahead
and
use
as
well,
which
is
really
important
right,
that
if
you
build
a
production
and
then
system,
it's
great
to
use
the
cli
secrets
with
data
bricks
by
itself
and
that's
fine,
and
if
you
are
typically
only
doing
that.
That
again
completely
makes
a
whole
lot
of
sense.
A
But
if
you're
actually
going
ahead
and
wanting
to
work
with
multiple
like
systems
right,
then
absolutely.
This
is
where
it's
going
to
be
very
handy,
where
everything's
in
key
vault
you're
good
to
go
from
there
and
then
not
a
big
deal.
They
all
will
use
the
same.
Login
excuse
me
secret
to
be
able
to
access,
and
so
now
your
azure
databricks,
your
key
vault,
whatever.
A
Oh
sorry,
your
azure
data
factor
or
whatever
else
that
one
is
using
boom,
you're
good
to
go,
and
in
addition
to
that,
I
just
want
to
call
that
some
folks
are
asking
questions
like
will
this
be
recorded?
It
is
currently
being
recorded
right
now,
as
we
speak,
so,
whether
you're
using
the
linkedin,
your
linkedin
or
youtube
it'll,
be
recorded
in
both
places
for
those
who
are
using
zoom,
we'll
send
out
the
youtube
link,
so
you'll
be
able
to
see
it
boom
all
right.
So
I
think
I
answered
that
question
anything
else.
B
A
B
A
Absolutely
so
autoloader
is
a
great
mechanism
for
okay.
Let
me
go
backwards.
A
little
bit.
Autoloader
basically
makes
it
easy
for
you
to
load
data
automatically
from
storage
directly
into
databricks.
Okay.
So,
basically,
as
data
is
being
added
into
the
system,
we
will
get
it
automatically
load
it.
Hence
the
name
now
when
you
write
it
within
within
spark.
A
Basically,
it's
typically
like
your
spark
dot
read
and
we're
going
to
show
that
code
in
a
second
and
by
the
way,
there
was
a
suggestion
that
later
on
that
we
actually
have
you
follow
along
or
write
the
code,
so
this
particular
session.
We
definitely
didn't
design
it
for
that,
but
I
think
our
I
believe
is
arno.
That
said,
it
has
suggested
yeah,
following
from
there
onwards,
actually
yeah
we'll
probably
try
to
do
exactly
that
which
is
go
ahead
and
actually
provide
you.
A
The
code
examples
right
from
the
get
go,
so
you
can
code
along
with
us,
okay,
but
the
context
is
basically
spark.read
and
then
you
basically
specify
cloud
files
that
little
specification
allows
databricks
to
automatically
know.
Okay,
I'm
supposed
to
basically
automatically
load
the
data
so
as
the
data
is
being
pushed
in
from
adf
okay
in
our
example
here,
which
is
basically
being
written
into
cloud
storage,
okay
into
our
hierarchical
store
that
we
created
all
right.
That's
for
the
sake
of
argument
here,
we're
pretending
that
that's
actually
somebody
else.
A
That's
running
maintaining
that
system,
which
is
very
typical
of
any
production
environment,
right
that
basically,
somebody
else
is
running
the
running
that
particular
system.
We
have
no
control.
All
we
do
is
have
access
to
the
data
from
the
from
the
from
the
storage
point
from
the
hierarchical
store
that
we
just
created
all
right,
perfect.
Well,
then,
how
do
we
automatically
load
data
when
there,
as
opposed
to
needing
to
like
you,
know,
schedule
it
or
time
it?
A
A
That
is
a
very
long
explanation,
and
so
I
don't
think
this
probably
makes
sense
in
this
particular
venue,
though
saying
that,
in
about
three
or
four
weeks
from
now,
we
will
actually
have
a
technical
dive
details
blog
actually
that
dives
into
all
the
details
around
how
the
autoloader
works.
You
know
basically
how
we
use
rocksdb
to
help
us
maintain
state
what's
going
on
in
the
background
with
the
checkpoints
things
of
that
nature,
okay,
how
we
actually
make
use
of
the
various
queuing
mechanisms.
A
You
know
that
are
involved,
so
there's
a
lot
that
goes
into
autoloader
that
you
won't
see.
So
we're
not
going
to
talk
too
much
about
that
today,
since
this
is
more
of
a
introduction
to
building
your
system
of
building
a
thing,
you
don't
necessarily
need
to
understand
all
those
details
to
build
it,
but
same
token.
We
will
go
ahead
and
actually
talk
about.
I
have
a
blog
that
said
coming
out
in
next
few
weeks
that
dives
into
those
details,
so
hopefully
that'll
answer
your
question
better.
Okay,
all
right
simon!
B
C
A
A
B
A
A
A
B
A
Okay,
so
perfect,
I
want
you
to
go
ahead
and
show
it
because
I
think
our
connection
is
a
lot
better
now
and
perfect.
I
will
present
my
screen
how's
that
all
right,
so
this
is
how
we
do
it.
Ladies
and
gentlemen,
because
simon's
internet
is
a
little
hinky,
I
will
be
presenting
my
screen,
while
simon
will
be
speaking
to
it,
how's
that
kids,
that
works
yeah
and
I
think
to
everybody-
that's
online-
is
that,
are
you?
Are
you
able
to
hear
simon
now?
A
A
In
case
you
want
dark
mode,
you
can
click
on
view
theme
notebook
like,
so
you
can
go
like
this
view
and
go
dark.
So
there
you
go.
So
if
you
folks
like
dark
dark
mode.
This
is
what
dark
mode
looks
like.
Okay
as
an
fy,
I'm
gonna
go
back
to
light
mode
because
at
some
point,
you're
gonna
need
to
put
some
branding
back
in
here
right.
A
B
B
Stuff
yet
so
that's
the
I
want
to
send
something
to
go
to
a
secret
secret
store,
one
of
the
the
scopes
that's
been
registered
and
pull
back
a
certain
thing,
so
in
our
key
vault
we
had
subscription
id
sas
key
and
we'll
go
through
what
each
these
things
are.
Essentially,
I'm
just
going
pillaging
the
key
vault,
storing
those
things
as
variables
so
that
we
can
then
use
it
in
a
second.
B
A
B
You
need
to
know
the
service
principle
id
you
need
to
know
the
active
directory
it
needs
to
know
which
subscription
and
which
resource
group
it's
using.
So
a
lot
of
this
stuff
is
just
kind
of
essentially
just
config,
so
that
it
can
go
and
create
a
thing,
so
I'll
need
a
little
load
of
that
information
and
what
it
does
is
that
that
trigger
kind
of
pulls
off
loads
loads
of
stuff
and
actually
dumps
it
into
a
dumps
it
into
a
blob,
storage
key
and
so
in
order
for
autoloader
to
then
read
that
queue.
B
It
also
needs
a
a
shared
access,
signature
key.
So
we've
got
that
q
sas
key,
so
it's
got
two
things:
it's
got
a
service
principle
that
can
create
these
file
watcher
objects,
and
it's
got
a
cue
sas
key
to
be
able
to
read
the
results
of
them,
and
it's
done
that
way.
So
you
can
split
the
two
up
you
could,
if
you
want,
go
and
create
all
the
event
grids
and
the
queues
and
do
all
that
yourself
and
then
just
have
autoloader.
Look
at
that
cue
and
just
read
out
the
list
of
stuff.
B
In
this
case
it
means
we
can
just
do
everything
at
once
and
we
just
go.
I
want
you
to
go
and
create
everything
for
me.
Go
nuts,
which
is
all
good,
okay
right,
so
next
thing
that
we
need
if
we
just
write
data
frame,
so
there
is
something
a
little
bit
hinky
with
autoloader
and
that
autoloader
needs
a
subscription
not
subscription.
What
am
I
talking
about?
It
needs
a
schema,
so
it
needs
it.
It
needs
a
structure.
A
B
B
So
I
wrote
a
quick
hack
when
I
was
having
to
play
around
so
I
barely
just
did
a
quick,
I'm
gonna
take
just
one
of
those
json
files.
I'm
gonna
run
a
quick
data
frame
over
it
and
that's
going
to
bring
back
just
a
data
frame
for
one
particular
json
file.
So
let's
just
try
and
run
these
bits
and
pieces.
B
B
Yeah,
just
rather
than
having
to
connect
and
give
all
the
passwords
and
give
it
all
the
code
each
time
just
said,
you
know
what
take
that
entire
blob
store.
Take
that
ratings
container
and
just
refer
to
that
as
mount
slash
ratings,
and
then
we
can
just
explore
the
rest
of
it.
So
that
is
real,
simple
little
data
frame
going
and
saying,
connect
to
that
one
file
bring
that
one
file
back
and
you
can
see
it's
understood.
It's
read
out
that
json
passed
it
for
us.
It's
got
the
kind
of
the
array
of
the
ratings.
B
It's
got
all
those
different
bits
and
pieces
and
then
we're
going
to
cheat
so
we're
going
to
use
that
as
the
as
the
schema
that
we're
expecting
and
it's
a
little
annoying.
That
autoloader
currently
expects
that
it
wants
us
to
give
it
that
each
time,
but
it
is
what
it
is.
It's
it's
the
penalty
for
using
autoloader
currently,
so
you
get
a
load
of
simplicity
in
terms
of
file
watching
and
automatically
knowing
what
it
needs
to
read
and
all
that
kind
of
stuff.
B
The
plan
is
to
be
doing
a
lot
of
the
coding
live
in
the
future
once,
but
I
guess
we
run
out
of
time.
So
I'm
just
pasting
in
some
pre-made
stuff
be
fine,
be
fine
right
and
then
the
final
bit.
So
this
is,
I
want
to
make
a
new
data
frame
again
slightly
bad
coding,
saying
df,
but
we
figure
that
out
so
spark
dot.
Read
stream-
and
that's
I
mean
that's-
that's
that's
kind
of
a
whole
bag
of
worms
in
itself.
Right,
so
autoloader
is
streaming.
B
B
A
A
Run
aquarium
so,
as
simon
is
doing
that
just
to
over
emphasize
this
little
sample,
df
is
simon's
fun
way
to
do
a
quick
cheat.
So
that
way
we
have
the
schema
which
is
defined
right
here.
Okay
and
all
the
the
options
for
the
cloud
file,
those
format
for
cloud
file
to
basically
use
the
auto
loader.
That's
actually
how
we
use
it.
This
cloud
file
here
is
defined
up
here.
Okay,
so
that's
actually
how
we
connect
to
it.
A
B
A
C
All
right,
thanks
cindy
and
simon
great
session
and
thanks
everyone
for
joining
us,
quick
reminder
that
the
session
was
reported
and
it's
available
on
our
youtube
channel,
so
check
us
out
there.
We
have
an
online
meetup
playlist,
all
the
content's
there,
and
I
just
launched
a
quick
poll
to
to
finish
us
out
here
on
zoom
about
what
everyone
would
like
to
see
next
session.
So
I'm
going
to
give
it
a
few
more
seconds
for
people
to
vote,
looks
like
we're
getting
a
lot
of
looks
about
50
or
so
have
voted.