►
Description
Change data capture is a popular method for unobtrusively ingesting data from SQL sources. In this talk, we will show how to easily incorporate your SQL data sources in near-real-time into Databricks and Delta Lake on Google Cloud. We will provide a short introduction to change-data-capture, Google Datastream (serverless CDC on Google Cloud), Databricks, and Delta Lake. In addition, we will also give a walk-through of our new open source Spark Structured Streaming connector which provides an easy-to-use / configure method of linking Datastream to Delta Lake.
Quick links:
https://delta.io/
https://github.com/badal-io/datastream-deltalake-connector
https://databricks.com/blog/2022/02/03/google-datastream-integration-with-delta-lake-for-change-data-capture.html
A
Doom
webinar,
like
features
newsletter,
so
as
a
host
of
events
like
I
know
about
all
these
cool
features
like
we
have
some
like
all
hands.
You
know
talks
that
we
zoom
for
and
they
always
play
music
at
the
beginning,
and
I
think
it's
coming
through
zoom
and
I
always
always
mean
to
try
to
check
that
out
and
play
some
music.
But
you
know.
B
A
C
A
Right,
I
think
I
got
the
thumbs
up,
that
we're
live
on
youtube
and
linkedin,
so
welcome
everyone
if
you're
joining
us
from
from
youtube
or
linkedin
we'd
love
to
hear
where
you're
joining
us
from.
I
am
in
oakland,
california,
it's
very
nice
and
sunny,
I
feel
like
I
always
my
instinct-
is
to
check
out
peek
my
head
out
the
window
and
see
what
the
weather
looks
like,
because
denny
always
talks
about
how
it's
gloomy
in
seattle
right.
C
B
A
C
A
Nice
we
have
some
folks.
We
have
a
couple
more
people
in
zoom
from
toronto,
adam
and
sohai,
I
think
sao
paulo
calgary
and
I'm
checking
linkedin
to
argentina,
bangalore,
austin,
texas,
st
louis
st
louis.
I
think
that's
san
antonio
texas,
dallas
michigan
somewhere
folks
from
seattle,
awesome,
belgium,
some
more
folks
from
india,
mumbai,
sweden,
cool
czech
republic,
lima,
mexico,.
A
C
A
A
A
So
quick
few
housekeeping
notes,
the
session
is
recorded.
We
are
running
it
through
zoom,
so
hello,
if
you're
joining
us
from
zoom
and
we
are
live
streaming
it
to
the
delta
lake
youtube
and
the
linkedin
too.
So
if
you're
joining
us
in
any
of
those
spaces,
there's
a
chat,
we'd
love
for
you
to
chime
in
ask
your
questions
and
we'll
we'll
collect
them
as
we
go
and
we'll
have
plenty
of
time
for
questions
at
the
end
and
throughout
the
presentation.
A
A
So
you
can
have
it
if
you'd
like
to
reference
it
at
a
later
date
and
we'll
also
update
the
copy
there
with
any
irrelevant
links
that
that
pop
up
throughout
the
presentation
so
check
that
out
there,
and
then
I'd
also
like
to
just
point
you
to
the
delta
lake
chapter
page.
So
it's
hosted
on
the
linux
foundation.
So
I'd
love
for
you
to
join
that
page
there.
A
That's
where
you'll
be
up
notified
of
upcoming
delta
lake
sessions
and
get
all
over
our
awesome
content,
and
we
also
do
a
week
sorry,
every
twice
a
month,
a
community
office
hour
session
with
vinnie
and
and
team,
and
so
we'd
love
for
you
to
join
the
sessions
as
well
and
then,
of
course,
subscribe
to
the
delta
lake
youtube
channel,
and
I
just
wanted
to
point
out
quickly
that
databricks
has
the
upcoming
summit.
The
data
and
ai
summit
really
exciting
this
year.
A
It's
going
to
be
hybrid,
so
in
person
and
virtual,
if
you'd
like
to
attend
virtually.
So
that's
going
to
be
the
end
of
june,
so
june,
27th
through
30th
and
there's
lots
of
awesome
sessions,
there's
some
training
sessions
at
the
beginning,
lots
of
amazing
you
know
technical,
deep,
dives
and
different
conference
sessions
and
we're
going
to
have
a
developer,
lounge
and
some
cool
spaces
that
you
can
hang
out
and
meet
different
people
in
the
community
and
delta
lake
contributors
and
and
lots
of
awesome
stuff.
A
So
I'd
encourage
you
to
register
and
and
join
us
there.
Our
our
whole
demo
team
will
be
there
and
also,
of
course,
most
importantly
tons
of
awesome
sessions
on
dudley
and
I
think
also
dude
denny
is
doing
a
live
podcast
series
with
brooke
182..
That
should
be
really
awesome,
so
lots
of
cool
stuff
all
right
so
with
that
I'll
pass
it
over
to
my
co-host
denny
to
kick
things
off.
C
Perfect,
hey
thanks
very
much
karen
by
the
way,
as
a
quick
call
out,
do
try
to
register
for
date
and
ai
summit
soon.
There's
both
the
virtual
component
as
well
as
the
in
person,
the
in
person
we're
actually
starting
to
run
out
of
tickets,
so
do
register
sooner
than
later.
Okay,
so
now
that
I've
done
that
little
tidbit,
I
without
further
ado,
let's
go
right
to
stephen.
I
think
he's
the
person
that
you're
actually
here
to
go.
Listen
to
so
stephen.
Take
it
away
as
people
know.
C
B
B
Just
give
a
brief
introduction
to
myself
and
the
people
involved
with
this
project
and
presentation
so,
like
I
said,
I'm
stephen,
it's
really
nice
to
meet
you
all,
I'm
a
data
engineer
at
a
company
called
bottle.io,
and
I
also
want
to
give
a
shout
out
to
eugene
nitzke
who's
also
with
bottle.
He
helped
build
the
spark
connector
which
I'll
talk
a
bit
more
about
later,
be
new
matthew,
who's
with
databricks,
and
he
helped
us
with
testing
the
connector
and
I'll
see
also
itai
margolin
who's.
A
product
manager
of
data
stream
at
google.
B
B
B
B
This
is
not
really
the
case
for
traditional
sql
databases,
which
may
provide
some
solutions
for
this
like
database,
replication
and
sharding,
but
are
much
more
difficult
to
implement
and
maintain
and
you'll
have
to
be
a
bit
imaginative
with
this
spark
analogy.
But
you
can
imagine
the
sailboat,
as
a
fleet
of
high-speed
motor
boats
capable
of
supporting
and
constructing
the
houses
on
the
lake.
B
B
In
addition,
we
likely
have
other
sources
of
data
which
we
need
to
which
require
other
periodic,
or
maybe
streaming
ingestion
strategies
like
the
plant
life
of
our
lake
or
possibly
one-time
day
loads
like
the
wood
holding
up
the
house.
Lastly,
what
is
a
good
lake
house,
if
not
to
support
life
and
social
activity?
B
So
I've
been
so
this
is
how
the
data
pipeline
actually
works.
So,
starting
from
the
left
side,
we
have
data
stream.
It's
a
gcp
service
which
provides
serverless
data
ingestion
from
sources
like
my
sql
oracle.
Data
stream
supports
two
processing
modes.
One
is
an
initial
backfill
and
second
is
a
streaming
cdc
mode.
B
Data
stream
exports
data
as
json
or
avro
files
into
gcs,
which
is
google's
distributed.
Blob
storage
in
the
future.
Google
pub
sub
will
also
be
a
supported.
Sync,
the
next
part
of
the
pipeline
is
a
spark
connector
which
our
team
has
developed.
It
uses
structured
streaming
to
move
data
streams,
gcs
output
into
delta
lake.
The
connector
can
be
run
using
a
databricks
cluster,
which
is
backed
by
google
kubernetes
engine
as
the
execution
backend
or
dataproc,
which
is
google's,
hosted
sparkrunner.
B
B
Then
a
containerized
app
could
be
deployed
onto
gke,
integrated
with
bigtable
and
make
transaction
time
fraud
decisions
and
an
exam.
Specific
example
of
this
could
be
building
sign
up
accounts
for
email,
ip
device
or
postal
code,
which
is
canada's
zip
code
for
american
audience
and
building
thresholds
on
top
of
these
counts
during
a
sign
up
flow.
This
is
a
simple
but
effective
technique
for
stopping
promotion,
abuse.
B
Okay,
moving
on
so
I've
mentioned
the
term
change
cdc
quite
a
bit,
and
it
stands
for
change
data
capture,
so
I'll
provide
a
brief
introduction
to
this.
Also
cdc
is
a
design
pattern
which
records
every
incremental
change
made
to
a
database
system
for
some
relational
databases.
These
cdc
events
are
usually
captured
on
the
server
in
an
ordered
log,
which
is
often
referred
to
as
the
bin
log.
B
B
B
B
B
You
do
some
pretty
straightforward
configuration
for
targeting
the
tables
that
you
want
to
ingest.
This
can
be
done
at
either
a
schema
or
table
granularity,
and
you
can
see
also
you
can
future
proof
your
pipeline
with
the
future
tables
option.
You
need
to
provide
a
backfill
strategy
so
currently
there's
either
an
automatic
or
manual
backfill.
The
difference
between
the
two
is,
as
backfill
is
basically
the
process
of
doing
an
idl
for
your
tables
and
it's
required
because
generally
the
logs
that
data
stream
rely
on
aren't
around
forever.
They
have
some
kind
of
retention
policy.
B
Something
like
a
week
is
pretty
normal.
So
normally
you
need
a
way
to
get
the
data
which
isn't
available
in
the
log
and
that's
what
backfill
is
so.
The
automatic
backfill
will
just
start
as
soon
as
you
create
the
stream
and
the
manual
mode.
You'll
have
a
option
for
triggering
the
idl
manually
in
the
ui
so
and
then
the
last
step
is
simply
just
picking
an
output
format
for
the
files
that
data
stream
writes.
B
Once
you've
created
your
stream,
they
provide
a
user
interface
for
keeping
track
of
some
statistics
at
a
table
granularity,
so
you
can
see
how
much
data
you've
processed
and
also
trigger
the
backfill.
Like
I
mentioned,
you
can
also
see
a
schema
using
the
view.
Details
they're
also
providing
some
kind,
some
straightforward
graphs
to
do.
Sandy
checks
on
your
data
stream
stream
once
you've
started,
you'll
start
seeing
your
data
landing
in
gcs
in
a
kind
of
per
table
and
date
time
partitioned
format.
B
Okay,
moving
on
to
delta
lake,
like
I
mentioned,
I
won't
spend
too
much
time
explaining
delta
lake
because
I'm
probably
not
the
best
person
to
do
it
and
also
there's
just
an
abundance
of
available
content
on
delta
lake.
But
here's
a
quick
executive
summary
also.
I
just
want
to
say
that
I'd
really
recommend
to
check
out
other
talks
on
delta
lake,
because
they're
really
good
for
not
just
understanding
delta
lake,
but
also
common
problems
in
data
engineering.
B
Okay,
so
moving
on
to
our
spark
connector,
so
we've
recently
collaborated
with
databricks
to
open
source,
a
spark
based
implementation
for
integrating
data
stream
and
delta
lake.
The
connector
utilizes
spark
structured
streaming
to
bring
your
cdc
data
into
delta
lake
using
structured
streaming.
You
get
benefits
such
as
checkpointing,
which
is
the
persistent
tracking
of
what's
been
unprocessed
and
sorry
processed
and
unprocessed
in
gcs,
you
get
the
regular
horizontally
scalable
data
processing
of
spark,
and
you
also
get
additional
debugging
support
with
spark
ui.
B
B
B
B
Running
via
spark
submit
is
somewhat
similar.
You
need
to
compile
that
jar
again.
You'll
need
to
make
that
jar
available
in
your
spark.
Submit
command
and
you'll
also
need
to
kind
of
create
a
configuration
file.
We
we
use
a
library
called
typesafe
and
it's
requires
a
kind
of
hokan
format
configuration
file,
so
you
can
see
an
example
of
that
in
the
not
just
the
project,
but
also
in
this
slide
here.
B
B
B
It'll
have
access
to
every
change
made
since
data
stream
has
started
running
because
any
of
the
data
that
came
from
the
kind
of
idl
portion
of
data
stream.
We
have
no
clue
how
the
state
of
a
given
row
got
to
where
it
is
because
the
log
is
gone
for
it.
The
second
table
that
it
generates.
We
call
the
merge
table,
but
sometimes
it's
also
referred
to
as
a
target
table
or
ods,
and
this
contains
the
actual
current
state
of
the
table.
B
So
this
query
is
something
that
is
run
on
every
micro
batch
of
data
in
the
structure
streaming
app
and
it's
run
on
a
per
table
basis,
so
per
table
per
micro
batch.
We
have
a
merge
query
that
gets
run.
That
merge
query
is
relying
heavily
on
the
metadata
that's
appended
by
data
stream,
such
as
what
are
the
primary
keys
for
this
row.
When
was
this
change
applied
and
what
sort
of
operation
it
is?
Was
it
so?
The
merge
query
itself?
B
B
So
again,
it's
done
on
a
per
primary
key
basis
and
we
have
basically
three
scenarios
that
we're
capturing
one
is
for
deletes.
One
is
for
updates,
and
one
is
for
inserts
for
the
delete
use
case.
We
want
to
see
if
the
change
that
we're
looking
at
is
was
applied
after
the
current
timestamp
of
the
row
that
we're
affecting.
B
If
it
was.
If
this
change
is
after
it,
then
we
just
delete
it
from
the
ods.
The
next
case
is
the
update
scenario,
which
again
we
need
to
check
if
the
change
is
applied
after
the
current
timestamp
of
the
ods's
version
of
that
row.
If
so,
then
we
replace
the
column
values
of
that
row
with
the
updated
values
and
then
the
last
scenario
is
for
insertions,
where,
basically
we
don't
need
to
even
look
at
the
timestamp.
If
there's
no
row
for
that
in
the
ods,
then
we
just
insert
it
with
the
current
recent
change.
C
C
B
C
No
not
at
all,
not
at
all,
okay,
so
well,
oh
hds,
the
I.e
the
log
table
is
your
historical
data
store,
so
basically
in
data
warehousing
terms,
it's
often
known
as
the
history
table
as
well.
If
you're
talking
about
like
a
hit
from
that
context,
but
basically
because
we're
talking
tables
it's
it's,
it's
the
history.
C
History
table
data
source
and
while
the
merge
table
is
the
operational
one,
and
so
that's
the
quick
context
behind
those,
and
so
whenever
you're
thinking
about
change
data
capture,
it's
often
in
that
form
of
what's
the
current
snapshot,
no
or
the
operations,
and
also
what's
the
log
or
the
history
that
goes
with
it.
So
that's
why
the
terminology
there
anything
else
that
I
I
might
have
missed
even
or
should
be
added.
C
B
B
We
have
a
step
to
kind
of
figure
out
what
is
the
most
recent
change
per
primary
key
within
that
micro
batch
of
changes,
so
that's
kind
of
like
the
spirit
of
what
this
query
is
like
that,
because
we're
ordering
by
timestamp
and
some
other
parameters
there,
but
mostly
it's
going
to
be
by
time
sum.
So,
whichever
change
has
the
large
the
most
recent
timestamp
within
that
micro
batch
will
be
the
entry
which
has
that
row
number
equals
one
there
in
the
query.
So
that's
how
we
pick
the
most
recent
change
between
that
yeah.
C
B
A
B
So
if
you're
choosing
the
merged
schema,
then
that's
the
strategy
used
for
every
single
table
in
a
given
instance
of
the
connector
and
yeah,
like
I
mentioned,
merge
schema
is
one
of
the
options.
The
one
other
one
is
override
schema.
I
highly
like
to
recommend,
merge,
schema
and
basically,
what
merge
schema
is.
Is
it's
a
schema
evolution
strategy
which
allows
for
some
categories
of
changes,
and
these
check
changes
are
the
ones
that
are
generally
considered
to
be
acceptable
and
won't
break
things
upstream
or
downstream.
B
So
the
type
of
changes
it
allows
are,
like
things
like
add
column
and
also
some
types
of
up
casting
so
yeah,
for
example,
bite
into
short
generally,
following
like
a
practice
like
this
across
your
whole
data
pipeline
is
going
to
lead
to
just
a
better
overall
data
practice.
I
would
say
the
other
option
is
overwrite
schema
and
basically,
whenever
a
change
is
detected,
it
will
just
replace
the
old
schema
with
the
new
one.
B
So
in
this
way,
your
your
this
application
will
never
break,
which
might
be
good,
but
probably
all
the
apps
that
rely
on
that
data
could
break
if
they're,
relying
on
any
column
which
had
a
breaking
change.
So,
in
my
opinion,
I
would
not
use
this
option
most
of
the
time,
because
personally,
I
find
the
mess
that
it
creates.
I
mean
no
matter
what
there's
a
mess
to
clean
up
on
a
breaking
change,
but
the
mess
is
harder
to
clean
up
if
you're
kind
of
propagating
it
up
to
the
applications
consuming
the
data
great.
B
So
the
next
topic
that
I
want
to
cover
is
tuning
and
compaction
right,
so
data
stream
and
delta
lake
are
both
using
files
to
store
their
data
in
the
case
of
data
stream
you're.
Just
writing
files
for
delta,
like
you're
doing
everything
possible
that
you
can
do
to
a
file,
so
achieving
appropriate
file
size
for
the
data
is
very.
B
So,
oh,
actually
one
more
thing.
Sorry,
also
just
like
a
yeah.
Why
is
even
like
the
file
size
like
an
important
thing
like
like?
If
your
delta
table
is
not
backed
by
enough
files,
then
every
single
change
that
you're
gonna
apply
is
going
to
be
updating
or
copying,
even
if
you're
only
affecting
a
few
rows,
you're
going
to
be
having
to
copy
over
a
ton
of
other
rows
which
were
in
the
same
file
where
the
change
is
applied.
B
So
the
kind
of
relative
rate
between
what
you're
updating
and
what
was
copied
is
not
going
to
be
a
good
balance
and
yeah
on
the
other
side
of
it.
If
you
have
too
many
files,
the
performance
of
the
merge
query
and
basically
all
operations
are
going
to
have
too
great
of
an
overhead
due
to
just
bringing
in
so
many
files.
C
C
Perfect
yeah,
it
goes
back
to
that
too
many
files
problem.
So
don't
forget
when
you're
talking
about
cloud
object
stores.
The
other
big
thing
is
that
the
listing
operations
actually
really
really
really
slow.
Okay,
I
cannot
overemphasize
that
statement
enough
and
so
because,
when
you're
dealing
with
too
many
files-
and
it's
really
slow-
so
that's
the
whole
point
like
you're
trying
to
then
have
spark
or
for
that
matter,
any
other
distributed
system
that
you're
this
is
I'm
documenting
just
in
general
purposes.
C
Here
that's
trying
to
go
ahead
and
read
the
file,
so
you've
got
so
many
files
you're
trying
to
list
all
those
files
that
ends
up
being
slowing
things
down
massively.
Okay,
so
that's
a
real
quick
call
out,
and
so
then
july
again,
I'm
pretty
sure
it's
an
aggregation
of
a
first
name
last
name
aston
zoom.
Basically,
how
do
you
handle
the
delta
history
log
size
growing
too
big
when
you're,
using
that
upsert
pattern,
and
so
just
to
provide
some
context
here?
C
Is
that
you,
you
actually
have
two
problems,
actually
not
just
the
log
files,
but
also
the
data
files?
Okay,
so
the
log
files
itself
actually
isn't
that
horrible,
because
what
happens
is
that
once
the
transaction
is
done
once
all
the
data
files
have
been
written
and
we
after
the
upsert
is
done,
it
basically
goes
ahead
and
just
simply
includes
in
the
json
file,
all
of
the
actual
file
names,
and
so
that's
it.
C
So
it's
not
like
it's
not
a
deduplication
for
the
snapshot
of
the
table
at
that
time
it
contains
all
of
those
parquet
files.
Now
the
problem
with
too
many
files
is
that
that
list
might
be
too
long.
So
that's
why
you
want
to
make
sure
it's
not
that
you
exactly
to
what
steven's
talk
about.
You
want
to
make
sure
there's
tuning
compaction.
C
In
other
words,
if
you
do
it
too
often,
and
so
over
times,
you're
going
to
need
to
run
vacuum
to
basically
what
to
clear
out
old
logs
as
well
and
so,
but
that's
a
whole
other
statement,
and
we
can
probably
discuss
that
in
the
delta
use
of
slack.
So
hopefully
that
answers
most
of
your
question
and
then
I
thank
you
for
letting
me
take
that
pause
stephen.
So
we
just
answer
these
questions.
B
No
worries
yeah
great
questions,
so
a
little
bit
more
on
compaction.
Obviously
it's
like
a
really
important
topic
when
it
comes
to
this,
so
we've
also
tried
to
provide
some
options
for
dealing
with
compaction
of
the
data
for
non-data
breaks,
use
cases
so
because
you
won't
have
auto
optimize
or
optimize
available
to
you
in
a
non-databricks
context.
B
So
we've
provided
three
configurations
that
together,
you
can
hopefully
tune
to
limit
the
problem.
The
first
and
you
have
to
kind
of
you
use
them
in
conjunction
with
each
other.
So
the
first
one
is
a
merge
frequency,
which
is.
How
often
do
we
read
data?
How
often
are
spark
structured
streams
triggered
the
second
or,
and
also
that
means?
How
often
do
we
check
for
new
files
outputted
by
data
stream?
B
The
second
one
is
the
file
read
concurrency,
which
is
how
many
files
are
we
reading
per
trigger
per
table
and
the
last
one
is
micro
batch
partitions,
which
will
coalesce
the
data
in
a
micro
batch
down
to
this
number
of
files.
So
using
these
together,
you
can
slow
the
growth
of
your
log
table
and
ods
quite
a
bit,
but
you'll
still
probably
need
to
schedule
something
to
run
outside
of
the
connector
so
which
will
be
something
like
a
repartition
of
your
delta
tables
from
time
to
time.
B
B
And
open
this,
so
the
notebook
for
actually
running
the
connectors
looks
something
like
this:
it's
pretty
straightforward
this
notebook's
a
little
bit
longer
because
I
did
a
little
bit
of
analysis
afterwards,
but
so,
basically,
like
we
have
our
imports
getting
the
just
bringing
in
like
the
classes
associated
with
the
connector,
and
then
we
also
have
some
types
related
to
the
configuration
of
the
connector,
hopefully
for
most
use
cases.
This
is
the
main
part
that
only
part
that
you'll
need
to
change,
but
the
configuration,
hopefully
is
pretty
straightforward.
B
B
B
C
B
No
worries
also
depending
on
if
your
configuration
can
be
achieved
from
just
modifying
these
variables.
You
may
have
to
change
this
part
of
the
notebook
a
little
bit.
These
ones
are
actually
the
kind
of
scala
configuration
classes
used
for
configuring
it.
So
obviously
it's
not
quite
as
user
friendly.
Since
now
everyone
knows
scala,
but
we
have
a
lot
of
documentation
on
this
on
our
github,
so
hopefully
it
can
get
through
it
and
then
yes
to
actually
start
the
connector.
B
B
And
then,
once
their
tables
are
being
ingested,
you
can
start
playing
with
them
immediately.
I
was
I'm
a
huge
basketball
fan,
so
I
was
watching
one
of
the
talks.
I
think
it
was
a
few
weeks
ago
where
charlotte
hornix,
charlotte
hornets
analytics
guy
was
in
to
talk
with
every
denny,
and
I
was
inspired
to
bring
in
some
nba
data.
So
for
this
I
mean
he
was
talking
about
this
spec
second
spectrum
data,
and
I
definitely
didn't
have
access
to
that
quality
of
data.
B
But
I
ingested
some
free
data
on
kegel
which
basically
just
has
like
box
score,
like
data
on
a
per
player
basis,
so
one
so
I've
ingested
that
data
in
our
pipeline
already
using
data
stream
and
everything
that
we
talked
about.
So
what
I'm
trying
to
do
is
there's
actually
well.
Another
kind
of
aside
is
I'm
a
huge
toronto
raptors
fan
and
we
currently
have
a
little
bit
of
a
war
going
on
with
a
certain
player,
joel
embiid.
B
So
what
I
wanted
to
do
is
there's
a
little
bit
of
feeling
amongst
raptors
fans
that
one
of
our
former
players,
marcus
soul,
is
the
player
that
turns
joel
embiid
into
a
hall
of.
He
turns
him
from
a
hall
of
fame
caliber
center
into
a
basic
everyday
player.
So
I'm
hoping
to
be
able
to
validate
that
assumption
that
us
raptors
fans
like
to
make
about
joel
embiid.
B
B
So
here
I'm
just
I
just
did
a
quick
way
to
kind
of
generate
statistics
for
every
single
kind
of
numeric
field
that
I'm
interested
in.
So
that's
what
I'm
doing
here,
just
basically
anything
that
we
can
get
like
an
average
sum
or
max
from
I'm
just
building
those
aggregations
here
or
defining
those
aggregations,
they're
not
actually
being
applied
yet
so
here,
just
building
a
kind
of
per
player
per
opponent,
type
data
frame,
and
here
just
applying
the
aggregations
that
I
define
ahead
of
time.
B
And
then,
oh
sorry,
let
me
change
this.
I
was
also
going
to
look
at
the
charlotte
hornets
players,
since
the
guy
who
was
on
earlier
was
the
charlotte
hornets
at
a
linux
guy,
but
I'll
skip
that
for
now
so
yeah,
like
I
said,
joel
embiid
is
the
player
that
we're
interested
in.
I
have
a
very
simplistic
and
very
bad
algorithm
for
what
a
bad
performance
is.
Basically,
if
they
don't
score
a
lot,
if
they
shoot
the
ball
at
a
low
percentage
and
if
they
lose
the
ball
a
lot,
that's
a
bad
performance.
B
So
we
could
see
here
based
on
my
very
simplistic
algorithm,
that
it
is
true
that
mark
gasol
has
a
very
odd
effect
on
this
player.
Joel
embiid,
you
can
see
just
for
reference.
This
guy
averages,
maybe
25
points
per
game
over
his
career,
so
he's
doing.
Half
of
that.
30
percent
is
a
beyond
horrible
field
goal
percentage,
especially
for
a
player
who
is
seven
feet
and
also
four
turnovers
is
quite
awful,
so
that's
all
for
the
demo.
B
C
Cool
stephen:
this
is
great.
We
did
have
some
a
couple
more
questions
about.
Let's
see
where
was
it?
Oh,
we
did
provide
access
to
both
the
blog
as
well
as
the
github
repo
already
so
everybody
you
should
have
access
to
the
blog
and
get
a
repost,
but
wondering
is
there
any
reason
why
you
couldn't
write
this
in
python
since
it
is
written
in
scala,
and
you
know
everybody's
like
going
like,
I
want
to
write
the
sucker
in
python
these
days.
B
C
Don't
worry,
we're
not
gonna
get
the
debate
on
this
one.
I'm
a
former
scholar
guy
myself
by
the
way
and
to
adam
who
just
chimed
in
from
just
chimed
in
from
zoom,
is
that
skull
is
fine
man
and
we
agree.
Actually
you
are,
you
are
among
friends,
but
for
the
python
sequel
folks,
the
context
is
you
just
have
to
include.
A
C
And
then
you
should
be
able
to
rewrite
most
of
the
stuff
in
python,
so
I
don't
think,
there's
anything
against
python,
just
it's
more
like
yeah
and
don't
by
the
way
note.
The
fact
that
what's
really
funny
for
me
is
that
I
am
a
scala
fo,
a
guy
who
ended
up
writing
learning
pie,
spark
because
of
data
science
and
then,
of
course,
holden
corral.
She
was
instrumental
in
creating
pi
spark.
B
C
I
think
so
I
think
so
so
and
then
ravi
just
call
out
that
basically
he's
all
for
alcoa
leonard.
So
that's
fair.
I
don't
think
we're
gonna
debate
that
particular
statement
these
days.
So
I
think
it's
actually
just
about
it.
I
think
we're
done
actually
with
most
of
the
questions.
So
my
only
response
before
we
switch
it
back
to
karen
is,
if
you
do
have
more
questions
for
us,
please
don't
hesitate
to
ping
us
also
in
the
delta
user,
slack
just
go
to
delta,
io
and
near
the
bottom.
C
There's
a
click
to
slack.
It's
actually
just
go
dot
delta,
dot,
io,
slash
delta
users,
if
in
case
you
want
to
remember
that,
but
otherwise
just
go
to
the
website
and
just
go
there
to
go,
do
it,
but
then
yeah.
If
you
have
more
questions,
absolutely
ask
us
there
and
or
basically
you
can
go
ahead
and
go
to
the
github
repo
that
we've
already
chimed
in
I'm
pretty
sure
you
can
chime
in
all
your
issues
there.
So
I
think
that
covers
for
those
logistics
karen.
A
Thanks
denny
thanks
so
much
stephen,
that
was
a
great
talk
and
awesome
job
so
and
thanks
everyone
for
joining,
and
so
I
just
was
grabbing
the
youtube
link
to
drop
that
the
recording
link
in
zoom
and
then
I'll
drop
it
in
the
linkedin.
So
with
that,
thank
you
again
thanks
everyone
for
attending
and
hope
to
see
you
for
our
next
tech
talk
thanks,
take
care.