►
Description
Deep learning has come a long way over the past few years, with advances in cloud computing, frameworks, and open source tooling, working with images has gotten simpler over time. Delta Lake has been amazing at creating a tabular structured transactional layer on object storage, but what about images? Would you like to know how to gain a 45x improvement in your image processing pipeline? Join Jason and Rohit on data collab lab as we find out! Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
Thank
you
karen.
My
name
is
lee
blackwell
and
I'm
a
solutions
architect
here
at
databricks.
I
recently
joined
after
a
pretty
lengthy
career,
all
in
the
big
data
space,
and
I'm
just
so
excited
to
bring
my
my
data
engineering,
knowledge
and
background
to
data
bricks
and
really
do
some
awesome
things
here.
I
haven't
candidly:
haven't
done
a
lot
of
image
processing
in
my
data
engineering
life,
so
it's
so
exciting
to
see
what
dataworks
can
enable
for
for
us
to
do
with
these
images
so
excited
to
bring
this
episode
to
you
guys.
B
D
B
C
C
What
this
use
case
did
is
we
wanted
to
look
at
images,
so
we
were
pulling
the
the
the
customer
we
were
working
with
was
pulling
lots
of
information
from
a
manufacturing
process
and
ultimately,
they
wanted
to
take
bring
in
the
power
of
deep
learning
using
some
gpus
to
detect
certain
conditions.
So
we
we
basically
came
in
and
helped
them
with
generating
of
images
in
this
case,
what
we'll
go
through
today?
C
We
I'll
show
this,
but
we
we
kind
of
shortcut
that
and
just
pull
some
images
for
classification
and
then
inference
and
and
send
out
notifications,
and
then
we
also
had
a
data
science
model.
What
was
kind
of
interesting
about
this
use
case
is
aside
from
the
fact
that
we're
dealing
with
image
data
instead
of
kind
of
what
we
normally
think
of
as
database
data,
is
the
the
cost
of
running
and
maintaining
gpu
instances.
C
And
so
what
we
wanted
to
do
is
come
in
with
spark
and
delta
lake
and
databricks
and
show
how
one
we
could
take
kind
of
what
they
were
doing
from
they're
using
pandas
and
some
traditional
single
core
type
stuff
and
really
not
have
them
changed
too
much
yet
come
in
and
let's
do
some
data
processing.
Now
and
let's,
let's
give
you
a
scalable
solution
out
to
the
rest
of
the
system.
A
C
A
Yeah,
I
think
that's
a
good
point.
I
mean
we're
kind
of
only
in
our
infancy
of
being
able
to
do
such
image.
Processing
right
I
mean
we're.
We're
the
big
data
world
is
really
tabular
and
you
know
more
traditional
business
reporting,
so
I
mean
it
just
it
might
be
some
sort
of
a
niche.
You
know
point
for
for
image.
Processing,
I'm
curious.
If
anyone
out
there
live
is
actually
working
with
image
data
or
has
a
use
case,
chime
in
we're
interested
to
see
what's
what's
out
there,
but.
B
Yeah,
what's
also
interesting
to
me
about
this
specific
instance.
Here
is
kind
of
some
of
the
the
decision
points
that
were
being
made,
so
this
was
when
we're
talking
about
like
they
had
to
have
all
these
gpu
based
machines.
These
were
capital
investments
for
on-premises
architecture,
correct
so
like.
When
can
you
talk
a
little
bit
about
how
painful
that
is
for,
for
this
instance,
just
doing
what
you're
trying
to
do
and
then,
when
you
went
to
the
cloud
right,
we're
taking
this
and
we're
helping
with
migrating
these
workloads
to
the
cloud?
B
C
It
was
impractical,
let's
say,
to
distribute
that
across
the
existing
infrastructure
based
on
licensing
and
all
this
other
stuff,
so
they
literally
had
just
these
large
boxes,
which
they
self-managed
self-clustered
and
then
developed
out
the
model,
and
so
one
of
one
of
the
first
things
that
was
a
concern
is
like
yeah
sure
you
guys
can
come
and
help
us
and
you're
doing
this
other
stuff
over
here,
working
with
text
data,
but
you
know
I'm
doing
images.
I've
had
to
customize
this,
it's
going
to
be
too
expensive.
Well,
no.
C
What
what
we
were
able
to
show
is
now
I
can
pull
in
and
really
use
gpus
on
demand
and
then
get
rid
of
them.
You
don't
have
a
capital
investment
if
you
come
back
to
it
in
six
months,
you
get
new
gpus,
which
might
have
the
latest
model
and
really
focus
on
using
that
here,
keeping
a
nice
streaming
just
regular
class
machines
as
it
turned
out
running
your
day-to-day
production.
C
So
it
really
gave
the
the
customer
the
ability
to
use
the
power
where
they
need
it
and
pay
for
it
only
when
they
need
it
and
then
really
give
them
a
economical
stream
stream
through
and
then
plus
since
we're
in
multiple
regions,
and
it's
easy
to
it's
easy
to
deploy
and
manage
data
bricks
across
multiple
regions
with
multiple
workspaces
and
such
now.
We
can
just
take
this
solution
and
effectively
copy
and
paste
it.
C
So
you
know
reality
probably
cicd
and
code
check
in
and
all
that
stuff,
but
I
really
just
move
it
out
so
that
we
are
effectively
bringing
compute
to
the
data
where
the
data
is
but
yeah
that
that
that
showed
a
lot
of
value.
And
what
was
neat
for
this
is
with
that
custom
hardware
solution.
What
what
we
got
in
this
was
a
a
big
improvement
and
just
being
able
to
use
the
binary
column
types
in
delta
lake,
which
I'll
walk
through
here
shortly
and
run
this
on
streaming
system.
C
C
I've
got
little
disks
here
from
to
represent
data
sources,
but
starting
with
that
source
and
then
getting
all
the
way
into
inference,
we
did
45
x
the
amount
and
when
I
did
a
as
as
much
as
I
could
kind
of
a
core
and
gpu
comparison.
Actually,
we
didn't
use
gps
on
this,
but
kind
of
a
core
and
ram
comparison
about
equivalent
hardware
with
what
they
were
doing
in
bare
metal
and
having
to
do
extra
work
to
kind
of
manage
the
distribution
and
such.
B
So
we're
talking
about
not
only
is
the
technology
itself
more
accessible
and
more
cost
efficient.
But
now
I
want
to
touch
on
something
you
mentioned
before
sampling
with
the
with
the
data
science
versus
now
we
can
do
it
on
the
entire
set
yeah.
What
does
that
really
mean
like
explain
like
the
business
impact
of
checking
every
single
widget
in
the
line.
C
This.
This
was
very
appealing
from
the
business
side
to
be
able
to
get
get
back
into
kind
of
a
full
check,
with
even
a
percentage
based
model
that
wasn't
human.
Obviously
we're
not
replicating
humans,
but
we're
getting
at
their
their
smarts
here.
So
two
aspects
really
helped
here:
the
yield
from
the
end
of
the.
From
the
end
of
the
day,
there
was
a
kind
of
a
long
process,
let's
say
the
manufacturing
process,
and
this
is
early
on
so
from
this.
The
idea
was
well
help
help
identify
these
things
up
early.
C
C
So
from
kind
of
a
time,
efficiency
perspective,
we
are
helping
them
with
that.
The
other
side
is
as
a
developer
as
a
data
scientist,
it's
sometimes
difficult
to
iterate
quickly
and
because
of
the
way
I
was
working
in
notebooks
and
rohit,
and
I
were
working
in
notebooks
because
of
the
way
that
databricks
is
already
set
up
to
kind
of
subset
things
for
you.
You
can
really
move
along
and
iterate
in
seconds
and
then
give
this
what
was
taken
30
minutes,
but
now
it's
taking
two
minutes
to
run
through
all
the
data
set.
C
You
can
get
that
full
data
set
while
still
being
able
to
iterate
on
a
small
sample.
So
then
you
get
time
efficiency
to
really
explore
ideas
within
even
a
production
cycle,
giving
you
the
ability
to
adjust
something
now,
instead
of
in
two
weeks
and
giving
you
the
ability
to
get
yield
in
two
months
and
know
that
you
can
guarantee
that
versus
waiting
and
having
to
throw
away
a
whole
batch
of
product.
C
C
Yeah,
let's
do
it
sorry,
so
yeah?
We
literally
used
notebooks
for
this,
and
let
me
let
me
go
through
some
of
this,
so
we'll
I'm
gonna
go
through
the
some
of
the
actual
code,
some
tricks
that
we
ran
and
ran
along
the
way
to
kind
of
make
this
good.
So,
as
I
tried
to
mention
earlier,
customer
already
had
some
python
code.
They
were
already
doing
some
image,
processing
and
and
didn't
really
want
to
go
and
redo
all
their
models
and
all
their
processing
work.
C
Just
to
try
a
new
platform
and
great
thing
with
databricks
is:
I
can
pull
in
what
we
need,
so
I'm
pulling
in
some
standard
libraries
here
and
then
at
this
I
say
the
actual
code.
This
is
obviously
not
what
we're
doing,
but
the
I
wrote
a
we
wrote
a
generate
image
function
so
for
this
use
case,
we're
not
starting
with
source
images
again,
so
I'm
just
pulling
them
together
and
in
this
case
I'm
doing
something
simple.
So
I'm
just
writing
a
dot
when
you
pass
in
a
set
of
input.
C
Input
rows
around
this
object
that
we
took
a
picture
of
really
the
deal
there
is.
I
want
to
show
how
to
kind
of
integrate
the
single
core
and
take
advantage
single
core
kind
of
mode
of
development
and
take
advantage
of
the
parallelism
offered
by
spark
so
then
getting
into
the
actual
here
this,
what
we
had
were
basically
two
tables
to
pull
together.
C
One
had
kind
of
summary
view
and
one
had
some
real
detail,
so
basically
in
spark
now
I'm
just
pulling
in
from
a
couple
tables
putting
them
together
and
then
getting
it
getting
it
set
up
so
that
I
can
do.
I
do
a
little
math
here,
but
getting
this
set
up
so
that
I
can
do
do
the
actual
image
generation.
C
C
After
a
few
edits,
it
becomes
clean,
so
I
I
do
try
to
make
this.
We
we
try
to
make
this
into
something
that
is
understandable
once
we
hand
off
our
plc,
so
the
okay,
so
yeah
back
to
so
right
now.
This
is
where
I'm
gonna
start
using
using
the
functions.
So
in
this
case
I'm
gonna
use
this
somewhat
new,
applying
pandas
function.
That
requires
a
grouped
data
set.
So
I'm
going
to
do
this
by
the
interest
id
and
then
it
also
requires
a
schema.
C
So
one
of
the
first
things
I
had
to
wrestle
with
is:
do
I
hard
code,
a
schema,
all
this
kind
of
stuff
that
you
sometimes
run
into,
and
you
know
just
a
quick,
aha,
hey
spark
is
lazy.
Why
don't?
I
just
pretend,
like
I'm
doing
a
select
just
to
get
the
columns
and
types
right
and
drop
that
down.
So
that's
what
we
did
here.
I
just
set
this.
This
doesn't
actually
create
any
data
operations.
C
It
just
does
a
one-time
schema
lookup
from
the
job
perspective,
provides
that
return
and
then,
as
long
as
I
have
my
distribution,
okay,
according
to
my
cluster,
I
can
just
call
that
generate
image
and
this
by
itself.
This
is
that
the
super
fast
image
generation,
because
I'm
able
to
distribute
out
chunks
of
chunks
of
my
incoming
image
information
and
then
distribute
that
across
the
cores,
so
that
turned
out
to
be
pretty
easy.
C
B
B
What
jason
just
did
was
he
took
that
regular
python
code,
not
pi
spark.
He
took
regular
python
function
and
he
put
it
into
a
function
called
applying
pandas
which,
under
the
covers,
will
distribute
that,
according
to
the
clause
in
group
by
so
in
this
instance,
is
taking
interest
id
right
and
it's
using
that
as
the
key
to
distribute
and
using
regular
python
code.
This
isn't
pi
spark.
B
C
Yeah,
so
I
I
I've
used
this.
I've
used
this
a
couple
times.
It's
really
every
time
I
start
with
somebody.
You've
already
basically
started
solving
the
problem
using
pandas
using
standard
image,
libraries,
this
kind
of
thing
kind
of
similar,
not
not
as
often
on
scala,
because
if
you've
gone
to
scholar,
oftentimes
you're
dealing
with
spark
directly,
but
this
proves
to
be-
I
mean
this
is
one
way
to
get
panda's
functionality.
You
can
also
directly
use
our
koalas
open
source
project,
which
is
pretty
neat
given
functional
compatibilities.
C
And
so
with
that
I
mean
there's
a
cup
there's
one
more
thing:
I
should
call
out
that
was
a
gotcha
here,
maybe
a
tangent
from
the
question,
but
the
other
thing
I
needed
to
do
here
was
really
filter
out
the
data,
because,
when
you
use
the
supply
and
pandas
to
do
your
regular
stuff
spark
is
literally
going
to
take
all
the
data
that
you
specify
and
send
it
over.
So
what
I
wanted
to
do
is
subset
that,
and
I
have
a
here-
we
go
okay.
C
Interest
with
image
should
be
what
this
is
there
we
go.
What
I,
what
I
needed
to
do
is
subset
that,
off
into
in
just
pulling
off
the
columns
that
we
need,
otherwise
all
of
that
data,
every
single
row
would
be
sent
here
and
so
and
then
the
other
thing
that's
kind
of
nuanced
about
using
this
is,
is
you
want
your
return
data
frame
to
match
the
schema
that
you
set?
C
So
I
I
had
to
do
the
same
thing
here
that
I
did
up
here
to
take
this
now
as
a
pandas
data
frame
and
spark
because
of
the
magic
of
this
function.
Spark
is
going
to
automatically
convert
that
when
the
data
comes
back,
it'll
immediately
go
from
the
pandas
chunks,
which
are
suitable
for
a
single
core
into
the
distributed
data
set.
That
comes
here,
and
this
is
probably
wrong,
but
I'll
fix
it
for
when
we
publish
this,
make
this
available.
B
Oh,
it's
totally
cool
in
the
interest
of
time.
I
know
we've
got
a
lot
to
cover
with
the
ml
stuff.
So
let's
let's
go
through
this.
This
is
also
another
big
point
here:
you're
how
you're
using
delta
lake-
and
this
is
this-
is,
I
think,
the
reason
why
I'm
super
interested
in
this.
Let's
talk
about
this
next
step.
C
Cool
so
yeah,
we'll
I'll
cover
this
kind
of
briefly
here
it's
the
the
magics
and
let's
take
advantage
of
the
fact
that
we
have
a
binary
and
column
type
and
then
just
put
your
images
there.
So
what
we
had
with
this
customer
case
is
we
had
a
bunch
of
potentially
small
files,
so
these
were
generating
small
images.
You
can
do
this
with
larger
images,
and
we
have.
C
I
haven't
personally
proven
out
the
the
ad
yet,
but
what's
nice
about
it,
is
you
let
delta
lake
manage
putting
even
several
images
in
the
same
file,
avoiding
ripping
through
your
block
size
and
such
so?
The
other
thing
that's
that
was
important
here
is
since
I'm
dealing
with
png
data
in
my
image
generator.
I've
already
got
it
compressed
and
the
rest
of
the
data.
C
Even
if
I
have
I
in
this
case
I
had
about
35
different
columns
to
put
in
next
to
the
image
which,
by
the
way,
makes
it
super
convenient
to
work
with
the
data
with
the
image
right
there
without
having
to
do
some
weird,
join
or
file
lookup
kind
of
a
thing
but
anyway.
So
when
I
had
all
that
all
that
data
together,
it's
you're
not
compared
to
the
image
data,
that's
in
here
I'm
not
going
to
get
a
lot
of
benefit
of
compressing
the
other
rows.
C
So
I
just
turned
compression
off
and
in
this
case
I'm
grabbing
what
it
was
set
to
then
I
run
through
and
just
say:
well,
let's,
let's
just
tell
delta,
to
do
its
normal
thing,
which
by
default
is
snappy
and
just
go
to
uncompressed,
write
it
out
and
then
then
restore
where
we
were
and
as
a
final
follow-up
on
that
I
actually
put
the
I.
This
is
how
I
like
to
do:
data
jobs.
C
C
Yeah
because
I've,
because
I've
put
brought
that
image
data
back
with
my
applying
pandas,
the
first
thing
I'm
going
to
do
is
I'm
going
to
join
that
back
with
all
because
I
subsetted
that
column
wise
so
that
I
wouldn't
have
a
lot
of
serialization
work
going
between
the
pandas
and
the
and
the
spark,
but
then
brought
that
back
together,
put
everything
together
and
then
with
that
image.
Now
it's
just
a
binary
column,
I'm
not
using
the
spark
image,
that's
useful
for
other
things,
I'm
using
a
binary
column,
and
now
it's
just
columnar
data.
C
A
C
Sweet
and
then
a
final
bit
that
is
cool
about
this
so
downstream.
Now
I'm
going
to
be
able
to,
in
this
case
I've
I'm
actually
showing
you
that
image
type.
So
this
isn't
what
comes
out
of
the
binary.
This
is
some
of
the
loading
that
rohit
is
going
to
go
through,
but
at
the
end
of
the
day,
when
I
want
to
use
this
guy
here,
it
is
what
we
were
able
to
do.
Is
we
in
this
case
we're
using
keras
and
tensorflow
and
we're
going
to
talk
about
that
shortly?
C
But
what
I
wanted
to
do
is
give
give
the
data
scientists
the
flexibility
to
bring
in
whatever
platform
and
bring
in
whatever
insights
they
wanted
to
ultimately
be
able
to
take
and
and
put
do
the
classification
that
they
needed
in
production.
So
you
want
to
be
able
to
do
that
without
this
huge,
coordinated
effort
across
lots
of
organizations
within
your
company.
So
in
this
case
extended
rohit
can
talk
about
this
a
bit,
but
we
extended
the
ammo
flow
python
flavor
and
grab
the
pipeline
code.
C
C
So
I
that
final
step,
like
we
took,
talked
about
image
generation,
and
now
this
is
the
inference
side.
Now
I
can
run
that
in
production
all
day
long
and
then
have
that
data
science
truly
as
a
as
an
out-of-bound
process
as
soon
as
it
publishes.
The
next
run
of
this
guy
will
get
this
pretty
the
production
version,
and
I
don't
even
have
to
care
if
it's
pytorch
or
classic
spark
or
whatever.
B
D
It
is
super
critical,
I
mean
yeah,
you
know
otherwise
I'd
be
spending
about
eighty
percent
of
my
time,
one
either
dealing
with
setting
up.
You
know
gpus
and
such
complicated
stuff
and
too
much.
More
importantly,
having
somebody
like
jason,
you
know,
figure
out,
hey
you
know.
Delta
is
pretty
cool
to
store
images.
So,
if
I'm
coming
in
and
building
a
you
know,
data
science
pipeline
that
makes
my
life
a
whole
lot
easier,
so
kind
of
taking
that
one
step
further.
D
You
know
we've
seen
some
examples
from
some
of
the
other
work.
We've
done
taking
a
sample
here
now
we'll
go
through
a
few
notebooks
kind
of
the
the
bottom
of
the
box.
That
jason
was
walking
us
through.
This
is
the
training
pipeline
right,
like
you
know
you
can,
you
can
have
the
training
done
in
data
breaks
with
with
distributed
computing,
and
then
you
can
have
a
stream.
That's
running.
D
You
know
24
7,
where
it's
actually
inferring
and
you
know
applying
the
model
that
you've
trained
onto
live
data
and
technically
you
know,
if
you
use
case
demands
for
something.
That's
you
know
ultra
low
latency.
You
can
even
deploy
it
on
a
edge
device
right.
You
can
take
a
model
that
we've
trained
here
and
then
deploy
it
elsewhere
too.
So
there
are
a
lot
of
options
depending
on
what
you
want
to
do
here
in
this
ng
step.
D
What
we're
going
to
look
at
is
kind
of
similar
to
what
jason
showed
us
in
zero
ingest,
but
we're
going
to
use
a
slightly
different
data
set
and
see
how
delta
you
know
handles
image
data
so
kind
of
jumping
right
into
it.
You
see
here
I'm
trying
to
list
and
see
hey
what
is
the
folder
structure
I
have
of
this
flower
photos.
You
can
see
that
there
are
a
few
folders
underneath.
D
I'm
just
gonna
see
hey
what
what's
underneath
that,
so
I'm
just
picking
daisy,
for
example,
there
are
a
lot
of
jpeg
images
down
here
so
now
the
tough
question
is
okay.
This
is
our
grade,
but
it's
unstructured
data.
How
do
I
put
a
structure
to
it?
Well,
it's
pretty
simple.
You
know
in
in
databricks,
you
can
just
say:
hey
spark.read.format
image
and
just
point
to
that
particular
folder.
D
D
Right
it
helps
you
with
qc.
It
helps
you
understand
what
you're
looking
at
you
know,
rather
than
looking
at
something
that's
a
little
too
complicated
and
for
those
of
you
you
know,
we
know
there
are
a
lot
of
data
scientists
and
data
engineers.
Here
you
can
definitely
look
at
the
more
interesting
stuff
under
the
hood
when
you
unselect
this
preview
and
you
can
then
see
it
has.
It
has
a
pointed
to
where
this
image
is,
what
its
you
know.
D
Metadata
are
and
the
number
of
channels,
in
this
case
its
color,
so
three
and
so
on.
So
it's
it's
very
useful
to
kind
of
to
be
able
to
do
that.
Let's
say
you
have
now.
This
is
a
case
of
you
know
kind
of
inferring
the
target
variable
label
automatically
through
the
folder
structure.
Let's
say
you
just
have
a
bunch
of
jpeg
images
in
a
folder.
You
know
you
can
simply
again
use
the
same
format
of
you
know,
read.format
and
pick
it
all
up.
D
So
that's
that's
pretty
nice,
as
you
can
see
here,
I've
you
know
a
whole
bunch
of
sunflowers.
Now
that
you
know
we
are
able
to
see
how
databricks
and
spark
lets
you
you
know,
pull
in
data
as
a
as
a
structured
table,
let's
kind
of
make
that
a
little
more
concrete.
What
we're
going
to
do
is
we'll
read
it
in
and
we
will
store
that
data,
so
I'm
just
going
to
reuse
the
same
stuff,
sparked.read.format
and
then
point
to
that
particular
location.
D
We've
seen
this
just
before,
but
this
time
around
I'm
going
to
save
it,
I'm
going
to
save
it
as
a
delta
table.
What
that
lets
you
do
is,
then
you
can
have
just
like
a
structured
table
but
actually
having
unstructured.
You
know
image
data
under
the
hood.
In
this
case
you
can
see
here.
I
have
this
particular
database.
I'm
just
gonna
take
a
look
at
this
table.
D
I've
created
right
here
as
a
delta
table
and
you'd
see
that
it
actually
captures
all
the
schema
and
the
sample
data
that
is
just
popping
up
so
that
that
makes
it
really
easy
for
users
to
come
in
and
check
all
the
you
know,
data
about
how
big
it
is,
and
things
like
that.
Another
cool
thing
that
delta
lets.
D
You
do
you
can
actually
have
time
travel,
so
you
know,
even
though
it's
image
data
as
you
keep
modifying
as
you
keep
adding
to
it,
you
can
actually
go
back
in
time
and
say:
hey
give
me
this
table
as
of
a
particular
time
stamp,
which
is
super
useful
for
a
data
scientist
when
you're
trying
to
reproduce
an
experiment.
Otherwise
you
know
you
have
to
tell
your
boss.
Well,
I
had
one
data
set.
I
tried
something.
I
got
some
results,
but
I
can't
reproduce
it
anymore.
D
D
So
here
I'm
just
doing
some
basic
transformations,
and
I
have
all
these
metadata
stored
in
and
I'm
going
to
do,
I'm
going
to
transform
the
label
to
an
index
and
just
finally
again
materialize
it
in
delta
format.
So
I
have
a
delta
and
I'm
going
to
write
it
again
as
a
table.
Like
jason
mentioned,
you
want
to
use
uncompressed
rather
than
snappy.
It
makes
it
extremely
fast
to
use-
and
that's
this
table
that
we
have
right
here.
D
Let
me
jump
to
that.
If
zoom
lets
me,
do
it?
Okay,
you
assume
it's
not
helping,
but
anyway
with
there
we
go
so
that
that's
the
that's
the
table
we're
talking
about.
You
can
see
the
binary
image
stored
here
now
that
we
are
able
to
get
to
that.
Let's
take
a
look
at
how
we
can
then
take
this
and
use
it
in
a
data
science
pipeline,
because
that's
the
that's
a
real
use.
What
we'll
show
you
is
how
you
can
load
the
data
in
spark.
D
D
So
note
that
you
know
here
the
particular
cluster
that
we're
using
actually
has
the
gpu
instances
that
under
the
hood,
which
makes
it
extremely
fast
and
powerful,
especially
for
training
libraries,
you
can
see
here-
it's
got
the
two
nvidia
gpus.
So
that's
the
kind
of
stuff
you're
using
for
training
aspect.
B
Yeah
I
got
a
question
from
linkedin
from
miguel:
can
we
reconfigure
parameters
and
deep
learning,
algorithms
approach
use
for
trained
model?
So
I
know
that
we're
covering
this
in
a
couple
minutes
here,
but
if
you
want
to
touch
on
that
for
a
second
before
we
get
there,
if
you
want
to
talk
about
that,
right
now
feel
free.
D
Yeah,
I
think
I'll
just
start
since
it
comes
through
in
the
flow
here,
but
yeah
you
can
you
know
you
can
do
things
like
distributed
hyper
parameter
tuning
and
you
can
pass
it
on
in
terms
of
distributing
the
parameters
that
you
use
in
a
particular
model.
Now
again
the
exact
parameters
that
really
depends
on
your
use
case.
What
we're
going
to
do
here
is
a
transfer
learning,
so
you
can
take
a
pre-trained
model.
You
rip
out
the
last
layer
of
it
and
then
you
just
retrain
it.
D
In
fact,
this
example
works
for
100
images,
so
you
know
many
people
think
about
deep
learning
as
requiring
terabytes
of
data.
That's
not
true!
You
can
do
it
with
very
small
data
as
well,
but
kind
of
jumping
right
in
what
we
do
here
is
we
set
an
mlflow
experiment
that
that'll
allow
you
to
track
the
the
progress
of
each
of
the
experiments
that
you
do
so
you
know
you're
building
many
things
you
lose
track
of
it
that
doesn't
help.
D
So
what
this
lets
you
do
is
it
will
keep
a
track
of
what
notebook
you're,
using
which
version
of
it,
what
data
source
you're
using
and
what
are
the
parameters
and
metrics
of
the
models
that
you
build,
makes
it
so
much
more
easy
for
you
to,
as
a
data
scientist,
to
understand
what
is
going
on.
We
have
some
simple
parameters
like
image
shape.
D
D
D
and
the
other
cool
thing
here
this.
This
is
something
we
learned
as
we
did
the
project,
so
petrostom
is
great
to
help
you
do
that
sort
of
a
transformation
from
a
delta
table
into
a
tensorflow
dataset,
but
we
run
it
to
some
issue
where
we
were
not
able
to
understand
why
you
know
the
amount
of
data
that
was
being
written
was
too
high,
and
then
we
realized,
as
you
convert
a
regular
delta
table
into
a
better
stamp
format.
D
The
parque
row
group
size,
which
is
the
amount
of
data
that
petastorm
caches
it.
Does
this
cache?
That's
the
library
that
we
set
here.
You
want
to
keep
that
to
a
relatively
lower
value
for
massive
data
sets,
because
otherwise
it
can
be
like
the
default
of
32
megabytes
can
be
too
high
and
you,
you
might
run
into
issues.
D
So
that's
a
pro
tip
that
we
learned
during
our
project,
but
once
you
convert
this
using
petastorm
for
converter,
you
can
then
directly
feed
it
into
your
libraries,
which
means
you
can
actually
just
switch.
It
out,
switch
out,
tensorflow
that
we
have
here
with
potash
and
everything
just
works
yep.
So
so
there
are
a
couple
of
things
that
we
really
focused
on
in
this
pipeline,
which
is
anything
a
data
scientist
does,
is
you
know
restricted
to
the
notebooks
that
he
or
she
is
working
on
anything
that
corresponds?
D
We
we
used
from
tensorflow's
samples,
you
see
all
of
this
by
the
way
is
python,
I'm
just
creating
a
simple
model
using
my
transfer
learning
mobile
v-net
model,
that's
trained
on
imagenet
weights,
so
it's
kind
of
pre-trained.
You
don't
really
need
a
whole
lot
of
data
to
retrain
this.
I
compile
it.
I
can
see
the
summary
of
the
model,
and
all
of
that
is
scarce
and
tensorflow.
D
Then,
once
you
have
a
model,
that's
ready
to
be
trained.
You
want
to
pre-process
the
data,
so
you
can
see
here.
This
is
again
a
nice
piece
of
info
because
our
information
is
stored
as
binary.
You
want
to
actually
use
a
byte
io
object
to
then
convert
then
and
then
feed
that
into
image
dot
open.
What
they'll,
let
you
do
is
then
do
any
transformations
you
want
to
do
like
resizing,
or
you
know,
rescaling.
So,
for
example,
pre-process
input
is
something
that
tensorflow's
mobile
v-net
to
you
know.
D
Library
provides
where
it
will
actually
rescale
things
down.
To
you
know,
an
image
that
is
ranging
from
zero
to
255
is
brought
down
to
minus
one
to
one
so
again,
very
nice
transformations
and
petastorm
also
allows
you
to
do
some
transformations
on
the
fly.
So
you
don't
need
to
do
everything
beforehand
and
that's
why
you
see
I'm
using
again
simple
python,
pure
python
functions,
to
do
my
transformations
once
I
have
that
I'm
gonna
jump
right
into
using
pedestal
to
kind
of
feed
batches
of
data.
D
So
you
don't
have
to
feed
10
terabytes
and
then
you
know
spin
up
a
supercomputer.
You
can
do
it
on
a
relatively
small
cluster
and
all
these
are
again
tensorflow
tensorflow
code
you're
just
fitting
the
model-
and
this
is
another
super
cool
thing
that
we
were
able
to
do,
which
is
how
you
ensure
that
the
inference
pipeline
doesn't
change,
whether
you
use
you
know,
tensorflow
or
python.
D
All
you
need
to
do
is
create
a
class
that
extends
a
pi
function,
model
implement
a
predict
which
does
all
the
transformations
of
data
that
you
feed
in
through
an
inference
pipeline.
So
in
the
inference,
instead
of
a
binary,
I'm
going
to
feed
in
a
regular
image-
and
this
takes
care
of
transferring
you
know,
transforming
that
into
something
tensorflow
understands
and
predicts
on
it.
So
you
can
see
here
it's
actually
applying
the
model,
so
the
data
scientists
can,
then
you
know,
deploy
a
pi
touch
model.
D
D
D
So
as
data
centers
develop
new
models,
the
inference
pipeline
doesn't
change
because
it's
just
saying
hey
pick
the
production
version
of
the
model
and
give
me
that
and
it'll
just
apply
it.
So
somebody
can
come
in
and
say
I
have
a
v4.
I
want
to
promote
that
to
production
again,
the
inference
pipeline
doesn't
change.
So
that's
another
critical
piece
you
can
see
here
it
actually
tracks
the
run
that
corresponds
to
the
one
that
created
that
model
and
when
I
come
down
here,
this
is
the
generic
python.
D
D
You
can
and
that's
a
good
point
you
can
see
here
all
the
metrics.
You
can
see
here
all
the
parameters
you
can
see
exactly
which
version
of
the
notebook
that
created
it
and
you
can
actually
go
ahead
and
click
reproduce
run
which
will
use
the
same
cluster
and
the
same
notebook
and
you
know
reproduce
the
exact
run.
So
if
your
boss
asks
you
hey
what
what
was
that,
you
can
actually
prove
that
to
him.
So.
D
D
D
You
just
need
to
make
sure
you
invoke
callbacks
that
allow
a
distributed
training
of
the
model
and
the
rest
of
it
is
practically
the
same
thing
as
what
I
had
earlier.
Just
make
sure
we
we
point
the
callbacks
and
that's
it.
You
know
again
do
know
that
all
of
this
is
python
code.
You
can
see
here
that
I
start
the
run,
I
just
say
hardware
runner
initiated.
D
D
All
of
those
are
are
being
used
to
do
a
distributed
training
of
this
particular
worker.
So
that's
that's
an
awesome
example
of
how
you
can
use
most
of
the
python
functions
that
you're
familiar
with
and
ensure
that
you
know
your
entrance
pipeline
is
really
solid
and
you
can
try
out
anything
you
want
and
ensure
that
it
goes
through
a
governance
process
of
model
deployment
with
that
I'm
gonna
pause,
not
sure
if
we
have
much
time
for
questions
but
happy
to
take
any.
B
B
I
think
that
if
we
have
time
to
answer
we
might
want
to,
but
there
was
a
question
from
amman
saying
that
what
about,
if
you
have
larger
images
like
satellite
images,
for
example,
where
they're
gigabytes
or
terabytes
or
petabytes
in
size,
my
initial
response
is:
I
thought
that
chunking
was
a
very
common
practice
with
that,
like
satellite
data
in
particular,
because
you
can't
process,
you
know
the
curvature
of
the
earth
and
all
that,
because
you
can't
really
put
it
like
that
jason
or
rohit.
Do
you
have
any
experience
with
that?
C
Yeah,
you
know
I
I've
worked
with
geospatial
data
a
bit
and
the
yeah.
I
I
usually
break
that
down
into
tiles
or
it's
already
broken
down
into
tiles,
and
then
each
of
those
titles
conveniently
fit
in
I've
got
some
other
use
cases
that
we're
developing.
We
need
to
push
the
limits
of
of
where
how
large
of
a
file
you
can
get
by
default
outside
of
images,
so
delta,
breakpoint
for
or
the
delta
right
size
is
around
two
gig
and
so
anything
up
to
that.
C
C
I
I
have
noticed
that
a
lot
of
them
are
actually
just
a
a
join
together
of
several
files
that
can
break
up
and
file
and-
and
I
can
turn
it
into
a
table
so
there's
some
future
work
to
do,
but
I
I
do
want
to
see.
I
haven't
taken
this
to
the
breaking
point.
I
know
we
can
get
up
to
two
gig
and
still
get
pretty
much
all
of
the
benefit
that
we've
covered
today.
D
And
just
add
to
that,
I
I
haven't
actually
implemented
using
better
storm
with
with
you
know,
geospatial
data,
but
petastom
exactly
helps
you
do
that
in
that
you
can
load
batches
of
parquet
data.
So
as
long
as
you
have
it
stood
in
a
you
know,
parque
file,
literally
terabytes
worth
of
it
pedestal
can
load
it
in
in
batches.
So
that
your
you
know,
your
server
is
able
to
handle
that
without
you
know,
running
out
of
memory
or
anything
like
that.
Yeah.
C
That
mapping
into
the
gpu
is
the
critical
bit
and
that
it's
overhead
showing
that
with
pedestrian
we'll
we'll
make
sure
I
mean
the
future
work
on
this.
We're
still
in
the
lab
here,
so
future
work
is,
is
to
really
push
the
extremes
and
and
see
where
we
can
go
and
keep
the
code.
This
simple,
basically.
B
C
C
It
is
a
common
practice
to
generate
extra
images
if
you
have
a
smaller
image
set
and
you
want
to
make
sure
class
is
represented
appropriately
through
rotations.
People
will
do
things
like
that.
I
don't
remember
that
specific
transform,
but
also
built
into
several
of
these
ml
toolkits
you'll
get
that
kind
of
rotation,
because
it's
a
it's
an
ideal
scenario
not
to
have
orientation
of
the
image
actually
affect
the
object,
detection
or
what
the
classification.
C
E
E
We
have
another
session
coming
up
in
a
few
weeks,
so
I'll
make
sure
I
pass
along
all
those
details
if
you're
joining
us
in
zoom
you'll
get
a
message
through
the
meetup
group
when,
when
that's
announced
and
if
you're
joining
us
on
linkedin
and
youtube
again,
I
encourage
you
to
join
the
meetup
group,
so
you
can
check
out
what's
coming
up
next,
so
this
video
it
was
live
and
it'll
be
recorded.
It
was
recorded
and
it'll
be
available
on
youtube
shortly.