►
From YouTube: S3select: Computational Storage in S3
Description
Presented by: Gal Salomon
S3select: Computational Storage in S3
S3 Select is an S3 operation (introduced by Amazon in 2018) that implements a pushdown paradigm that pulls out only the data you need from an object, which can dramatically improve the performance and reduce the cost of applications that need to access data in S3. The talk will introduce s3select operation and architecture. It will describe what the pushdown technique is, why, and where it is beneficial for the user. It will cover s3select supported features and their integration with analytic applications. It will discuss the main differences between columnar and non-columnar formats (CSV vs Parquet). We’ll also discuss recent developments for ceph/s3select. The presentation will show how easy it is to use ceph/s3select.
A
My
name
is
gar
Salomon
I
am,
and
I
am
a
member
of
dorado's
Gateway
team
for
the
last
three
years.
We
are
developing
this
to
select
this
talk
is
many.
An
introduction
to
S3
select
I'm
going
to
talk
about
the
concept.
I
will
explain
what
it
is.
Why
do
we
do
that?
How
does
it?
How
does
it
work
and
please
feel
free
to
ask
any
question
at
any
point
of
the
talk?
It's
allow
trailing
knowledge.
Okay.
A
Request
it
enables
the
client
to
push
down
an
SQL
statement
into
the
theft
storage,
the
SQL
that
we
are
using
there
is,
according
to
AWS,
specs
the
exercise.
It's
a
it's
a
new
S3
capability
designed
to
pull
out
only
the
data
that
you
need
from
the
object.
It
can
dramatically
improve
the
performance
and
reduce
the
cost
of
application
that
needs
to
access
data
in
S3.
A
Okay.
Why
do
we
need
that?
What
is
good
for
so
we
need
to
to
recall
that
sflect
is
actually
Implement
a
push
time.
A
push
down
paradigm
and
push
down
Paradigm
is
about
moving
or
pushing
the
operation
close
to
the
data
and
contrary
to
what
is
commonly
done,
which
means
moving
the
data
close
to
the
place
of
operation.
A
Push
down
does
not
consume
memory,
while
caching
does
consume
MOA
in
a
big
data
ecosystem.
It
makes
a
big
difference
upon
using
a
cache
paradigm.
The
flow
is
about
what
data
to
keep
in
Cache
and
what
data
should
invalidate
upon
using
push
down.
The
flow
is
about
what
the
operation
should
push
down
and
how
to
use
the
return
result.
A
So
for
one
example:
let's,
in
order
to
in
order
to
execute
the
left
sum
of
x
from
some
object
where
A
and
B
bigger
than
C,
it's
meant
to
fetch
the
entire
object
to
the
client
side
and
only
then
execute
the
operation
with
some
analytic
application.
We
push
down,
rf3
select,
the
entire
operation
is
executed
on
the
server
side
and
only
the
the
result
itself
returns
to
the
to
the
client
side.
A
So
an
SQL
query
that
needs
to
operate
on
CSV
or
Json
object
that
contains
many
columns
and
rows,
but
needs
only
a
very
small
portion
of
it.
A
few
columns
or
small
percentage
of
the
rows
will
return
only
a
small
amount
of
data,
which
means
less
operations
of
serialization
or
deserialization,
and
also
it
should
not
say
that
pushing
down
operations
close
to
data.
It's
not
an
easy
thing
to
do.
The
storage
is
usually
fragmented
and
walking
into
random
pieces.
A
A
What
is
asset
selected
capable
of
as
the
select
does
it
does?
Not
turn
our
S3
storage
into
a
database,
but
it
does
improve
greatly
the
efficiency
of
SQL
processing.
It
should
be
noted
that
SQL
dialect
provided
by
the
S2
select
is
not
is
only
for
read.
It's
only
the
select
statement.
There
is
no
notion
of
schema
or
table,
and
that
and
the
object
have
been
read
should
enable
tabular
operation.
A
Select
system
is
embedded
into
the
get
object
model.
What
makes
it
highly
efficient
for
push
down
operation
being
pushed
down?
The
object
is
fetched
and
each
fragment
of
the
object
is
processed
by
the
after
select
model.
This
lesson
select
is
embedded
into
the
S3
system.
There
are
no
redundant
copies
of
bytes
and
the
objective
processed
immediately
and
the
results
sent
back
to
the
client.
Moreover,
Ester
select
system
is
capable
of
possessing
CSV
Json
and
also
perki
the
same
engine.
By
the
way,
the
same
engine
is
processing.
All
of
this
type.
A
One
second:
okay:
why
using
why
using
SQL?
So
maybe
the
question
should
be
that
should
be
asked.
Is
why
why
what
is
needed
for
the
machine
learning
SQL
is
needed,
for
the
machine
language
is
the
in
practice.
This
is
the
language
that
is
over
covering
data.
It
should
be
noted
that
SQL
is
a
domain
specific
language.
More
than
40
years
it
was
designed
initially
for
manipulating
data
in
machine
learning.
A
Workflow
the
data
is
the
main
thing,
is
the
source
so
the
more
relevant
and
accurate
your
data
is
you
have
the
better
your
modeling
and
pattern
detection
will
be.
You
should
keep
in
mind
also
that
the
data
must
be
formatted
so
that
it
can
be
used
by
Machine
learning
algorithm
for
improved
pattern.
Detection.
A
Okay,
so,
as
you
know,
probably
know,
there's
the
storage
is
reliable,
efficient
and
relative
chip
storage
and
already
contains
trillions
of
objects.
It
contains
many
CSV
Json
and
poke
objects,
and
these
objects
contain
huge
amounts
of
data
to
analyze
and
HL
may
may
convert
these
objects
into
parquet
and
then
run
queries
on
this
converted
object.
The
parquet
is
by
the
way,
is
very
efficient
format,
but
converting
each
of
these
object
to
parquet.
It
comes
with
a
very
expensive
price.
Getting
all
of
these
objects.
A
A
A
A
A
Readers
of
the
different
data
types
are
decoupled
from
SQL
engine,
so
future
readers
for
other
data
types,
such
as
orc
for
one
example,
can
reuse
the
engine
code.
It
also
a
standalone
application.
The
same
as
the
select
engine
can
be
used
from
the
command
line
as
a
shell
command,
and
it
can
process
the
different
data
types,
the
CSV,
Json
or
pocket.
A
Coming
flights,
I
will
explain
about
the
different
flow
of
each
of
each
data
type
and
the
differences
between
the
data
source
types.
We
have
the
CSV,
the
Json
and
AppleCare.
We
start
with
the
CSV.
The
CSV
is
the
most
simple
format.
It
is
not
supporting
it's
not
supporting
data
types
and
as
a
trivial
meaning.
When
I
say
it's
not
supporting
data
types,
it
means
it
only
supports
string.
Of
course
you
need
to
cast
it
into
into
your
native
tab
like
integer
or
flow
to
something
like
that
and
as
a
trivial
schema
support.
A
A
A
Okay,
now
we
talked
about
going
to
talk
about
the
letter
K.
So,
as
mentioned
before
for
case
binary
file,
the
boundary
binary
object,
it
is
column,
column
now
object
by
saying
a
columnar.
It
means
that
we
can
access
directly
a
specific
column
without
scanning
awaiting
any
other
any
other
column,
which
is
a
usage,
a
user
advantage.
A
This
type
of
object
contains
a
lot
of
metadata.
It
enables
the
results,
as
I
said,
to
skip
non-necessary
information
at
first,
the
engine
verify
magic,
bytes
it
to
make
sure
it's
a
valid
pocket
object.
The,
Next,
Step
the
engine
purchase
the
predicate
the
predicate
columns
in
the
case,
the
predicate
returns.
True,
it
fetches
the
projection,
columns
and
calculate
the
projection
statement
for
anyone
to
know
SQL
statement
when
I
say
predicate
I'm
talking
about
the
workload
the
when
I
say,
projection,
I'm
talking
about
the
site
of
the
select
select
itself.
A
A
A
Okay?
Okay:
this
is
a
bit
weird
slide,
but
here
I'm
trying
to
explain
why
the
Json
reader
is
unique
compared
to
other
videos
upon
querying
a
Json
document.
The
SD
effect
engine
is
focusing
only
on
specific
parts
of
the
nested
and
compressed
Junction
data,
and
not
the
old
Json
data
is
actually
some
parts
it
ignore
and
some
parts
are
loaded
into
the
engine
itself,
its
meaning
that.
A
A
So
you
can
see
that
in
this
slide,
that
the
SQL
statement
is
focusing
it's
processing
on
the
specific
parts
of
the
document.
According
to
the
foreign
close,
the
phone
Clause
Define
the
boundaries
where
to
get
the
projection
and
the
predicate
predicate
a
column.
So
you
can
see
that
the
phone
call
says
the
and
it
says
the
form,
root,
glossary,
gloss,
deep
gloss
list,
gloss
entry,
gloss
Dev.
A
This
is
actually
a
defining
the
the
path
where
to
look
for
a
specific
column
for
the
select
statement
and
and
the
next
slide
you
can
see
here
and
the
new
statement.
It's
actually
the
same
document,
but
in
this
statement
the
SQL
statement,
the
phone
Clause
actually
Define
a
different
to
all
boundaries
compared
to
the
previous
one.
So
in
this
statement
the
readers
will
load
different
parts
of
the
documents
compared
to
the
previous
example,
as
I
mentioned
before.
So,
let's
talk
about
the
Json
video
and
the
Json
processing
flow.
A
Okay.
Actually,
now
we
cover
actually
all
the
the
current
supported
different
types
of
the
the
Json
the
cfp,
and
if
okay
now
we
talked
about
how
to
scale
it,
and
we
need
to
to
recall
that
running.
A
SQL
statement
on
a
single
execution
context
is
very,
is
very
inefficient.
When
talking
about
Big
Data,
we
strive
to
make
it,
of
course,
Much
More,
Much,
More
scale
at
least
scale
up,
so
the
ability
of
processing
data
stream
without
preloading
or
redirect
is
essential
for
the
processing,
a
huge
amount
of
data.
A
This
is
achieved
by
possessing
Raw
After,
all
that's
what
we're
doing
without
different
riddles
until
the
end
of
string
quite
similar
to
the
to
the
Linux
pipe
concept.
Having
that
as
the
select
function
consume,
little
memory
as
effective
is
able
to
split
and
the
query
a
processing
merge
the
results
upon
completion
into
a
single
result.
This
powerful
feature
is
a
is
very
powerful
for
the
Big
Data
ecosystem
in
the
next
slide.
A
I
will
explain
a
bit
more
okay
in
this
slide
and
try
to
see
to
explain
how
the
way
to
improve
dramatically
the
processing
of
this
huge
amount
of
data-
and
this
is
done
by
splitting
the
data
into
non-overlapped
overlap-
ranges
that
is,
the
system
should
run
the
same
query
on
non-overlapped
data
and
ranges
in
parallel,
of
course,
merge
the
result
and
upon
completion
and
return,
all
of
them
to
the
client
side.
As
a
single
one.
A
I
should
know
that
that
per
data
type,
the
the
data
splitting
flow
is
different
in
the
case
of
CSV,
the
object
is
split
according
to
the
object
size
and
the
row
delimiter
in
the
case
of
parquet,
the
this
unique
column
now
structure.
It
also,
it
is
already
divided
into
logical
groups,
so
it's
quite
easy,
but
with
the
Json
use
case,
it's
much
more
complex,
so
one
way
to
handle
that
is
to
get
getting
the
data
resized
on
many
many
objects
and
split
the
work
according
to
the
amount
of
objects.
A
Okay.
In
this
slide,
it
demonstrates
how
FC
select
engine
is
processing
in
aggregation
query
in
parallel.
In
this
case,
this
is
a
three
parallel
runs,
so
you
can
see
there
is
some
kind
of
select
segment
which
do
does
a
substring
while
using
a
mean
and
average
and
count
aggregation
functionality
with
some
predicate
a
were
closed.
So
this,
this
SQL
statements
are
run
in
parallel
on
different
range
of
the
of
the
data
upon
completion.
The
mid-stage
wait
for
all
these.
A
A
A
A
So
the
advantage
of
possessing
this
huge
size
object
using
as
to
select,
is
quite
obvious,
so
I
think
a
lot
of
network
and
also
CPU,
but
what
if
it
needs
to
run
a
single
query
on
many
small
and
medium-sized
objects?
This
may
happen
because
the
data
is
coming
in
a
small
packages
of
from
several
sources.
A
Okay,
until
now
we
saw
quite
a
complex
product
and
then
the
question
is
maybe
is
how
to
verify
that
the
result
of
copies
products
with
endless
combination
such
as
S3
select.
So
we
have
several
utilities
that
we
developed
in
our
team,
which
many
we're
talking
about
generator
of
expression
and
statement.
I
will
I
will
speak
about
several
of
them.
A
Expression
and
this
expression
can
actually
execute
it
on
C,
compiler
and
also
on
Azure
select
engine,
because
it's
the
same
syntax
and
the
result
should
be
the
same.
So
as
you
can
see
here,
it's
a
very
long
and
complex
expression
and
upon-
and
each
of
them
is
actually
actually
executed
by
the
C
compiler
and
also
by
the
asset.
Select
engine
and
and
results
are
of
course,
should
be
identical.
A
A
Also
so
for
one
example
in
this
you
can
see
here
are
two
SQL
statement
and
they
are
very
different
now,
in
this
case,
they
are
very
different
in
the
workload
One
is
using
a
time
using
a
a
Time
timestamp
functionality
and
the
other
one
is
using
a
string
functionality
and
still
they
must
return
the
same
result
for
any
random
input.
In
that
way,
we
actually
can
verify
that
these
two
statement
operate
correctly.
A
Another
example
for
that
you
can
see
here
and
the
same
thing
the
center,
the
syntax
is
very
different,
but
the
Thematic
is
the
same
so
here
I
can
see
also
this
step
in
our
completely
different
in
the
project
inside
and
also
in
the
in
the
predicate
side.
The
workloads
and
the
the
abstract
syntax
tree
and
the
execution
flow
that
evolved
from
this
statement
is
completely
different,
but
still
the
result
should
be
the
same
for
this,
for
any
random
input.
A
Okay,
another
type
of
generator
is
a
a
query:
generator
an
SQL
generator.
We
develop
a
something
like
that:
a
generator
which
can
actually
can
generate
a
endless
number
of
SQL
statement
complex
for
our
SQL
layer,
parser
and
to
run
them
both
on
and
run
it.
This
is
certain
on
both
on
self
engine
on
in
and
also
on
AWS
engine
and,
of
course
the
result
should
be
identical,
and
this
is
also
another
system
that
help
us
to
find
issues
in
the
system.
A
A
But,
as
said,
it
is
very
inefficient
use
of
storage,
and
that
is
because
it
needs
to
retrieve
all
objects
participate
in
the
query.
But
what?
If?
What?
If
upon
using
Esther,
select
a
request
instead
of
the
get
object
request-
and
this
will
improve
and
the
query
performs
dramatically
and
F6
effect-
is-
is
actually
an
opt.
You
can
figure
if
assets
as
an
Optimizer
for
this
eye
level
application.
A
It's
me
to
understand
that
there
is
a
great
advantage
using
SQL,
analytic
application.
Since
so
many
end,
user
applications
are
written
for
spark
and
Trina
or
presto.
What
is
needed
for
for
from
this
analytical
application
is
to
identify
what
parts
of
the
query
can
be
pushed
down
to
S3
storage
and
send
it.
Then
this
this
part
is
a
subgrade
to
as
as
to
select
a
request
and
should
bear
in
mind
also
that
this
type
of
Optimizer
already
exists
in
AWS
VMR.
A
Okay,
as
mentioned
before,
the
SQL,
the
asterisk
select
can
be
used
as
a
command
line
utility
as
any
other
shell
command.
So
we
can
walk
on
on
standard
input
or
our
files
or
all
files.
If
I
don't
know
on
your
local
file
system,
you
can
review
the
GitHub
repo
of
SF
elect
and
send
that
for
yourself.
I
will
give
some
some
examples
for
that.
A
The
first
to
SQL
statement
is
actually
running
on
the
on
the
standard
output
of
the
PS
command.
Okay
and,
as
you
can
see,
you
can
actually
Define
what
is
the
column
delimiter
in
this?
In
this
case
it
was
it's
blank
and
also
Define
that
there
is
this,
the
other
info.
So
what
you
can
see
here
6ps
is
a
printing
out
the
name
of
the
each
columns
when
you're
doing
PS
the
first.
The
first
row
is
actually
the
name
of
the
columns.
A
We
can
use
that
in
the
the
select
statement
you
can
see
that
we
do
select
a
PID
command
from
standard
input.
Well,
parent
PID,
equal
one
is
so
easy.
You
can
do
it
from
the
command
line.
That's
it
and
the
second
example,
you
can
see
an
SQL
statement
processed
by
the
by
container
and
I
will
explain
about
this
container
later.
The
input
data
is
packed
into
the
Container
itself.
A
So
again,
this
container
is
actually
a
container
that
lets
you
not
just
running
after
select
from
the
command
line,
but
actually
you
can
you
can
review
the
code.
You
can
change
the
code,
you
can
actually
build
it
rebuild
it,
and
this
content
also
contains
some
build
utilities.
So
it's
quite
easy
to
to
to
get
some
impression.
A
So
you
actually
in
in
this
case
you
can
see
that
I'm
running
here
a
statement,
SQL
statement
on
some
file
on
the
host
and
the
OS
machine
after
mapping,
of
course,
mapping
the
the
directory
of
the
awesome
machine
to
to
The
Container
itself
and
yeah.
That's
it
I
will
be
glad
to
to
hear
for
any
type
of
question.
If
you
have.