►
From YouTube: Kubernetes SIG Big Data Meeting (August 16 2017)
Description
No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).
B
C
Another
some
people
I
don't
recognize,
grant
Nicholas
Jean
from
school,
underneath
hi.
D
Yeah,
so
it's
great
Frances
from
the
air
for
community
he's
a
grant
in
Lincoln
Mitchell
months
and
our
core
it
with
the
airflow
and
proving
that
he's
initiatives.
E
E
F
Greens,
aggregation
not
segregation;
okay,
okay,
actually,
I
can
write
it
down
Columbus,
it's
very
easy.
You
extreme
like!
Essentially,
if
you
look
at
a
regular
coastal,
you
you're
using
dry,
cl'
attached
storage,
or
we
provide
like
creo
student
government
for
direct
attached
storage,
but
you
can
dynamic
on
the
line
comment.
How
many
drives
you
walk
on
a
given
server?
Oh
it's.
F
C
So
knock
you
get
enough.
Announcements
is
most
you
already
know.
Of
course,
we
finally
dropped
the
sparked
2.3
rebase
release
and
yes
tip
for
getting
it
adopted
upstream.
Hopefully,
this
release
cycles,
so
that
was
a
big
deal
and
thanks
to
every
to
rule
for
all
the
work,
you
guys
put
it
on
doing
that,
and
people
well
spent
a
lot
of
time
on
mouths
and
the
lock
will
be
able
to
see
the
fruits
of
that
library
sometime
between
now
and
the
February
believe
yeah.
I
I
spoke
briefly
with
Renault,
and
he
said
he
would
talk
to
the
other
PMC
members
about
getting
us
support
on
that
side,
but
to
all
the
people
that
came
the
post
if
you
can
go,
and
you
have
your
plus-one
and
the
reason
why
this
should
be
up
streamed.
That
would
be
super
helpful,
I
think
those
people
here
have
already
done
that.
But
just
you
know
there
and
again
has
except
thank
you
for
all
the
efforts
around
dogs.
The
you
know,
diagrams.
B
I
C
I
notice
that
there
have
been
at
least
a
few
totally
unfamiliar
people
Kinney
adding
+1.
So
it's
really
good
sign
and
I,
don't
really
like
discussing
without
legal
and
stuff.
What
you
can
talk
about
in
terms
of
the
deployment
sizes
but
I
think
anybody
anybody
wants
to
like
discuss
to
whatever
degree
of
detail
that
allowed
like
the
use
cases
that
they've
exercises
on
there
definitely
will
help
and
as
I
recall,
honoree
of
all
Reynolds
Pacific,
Way,
Africa,
I,
think
yeah.
D
C
C
C
C
C
K
Sure
so
the
use
case
is
basically
that
sale
40
go
for
an
application
from
yarn
you,
but
you
end
the
migration.
You
make
all
of
your
jars
into
your
docker
images,
but
yours
still
want
to
use
smart
dot
files,
basically,
like
the
add
files
feature
to
ship
over
this
trip
over
like
some
files-
and
in
this
case
these
files
are
very,
very
small
and
it
seems
a
little
heavyweight
to
have
to
install
a
resource
staging
server
and
also
have
to
use
a
and
youth
in
an
entertainer
in
this
case.
So
what
this
PR
does
is.
K
It
adds
an
alternative
mode
for
mounting
only
file.
It's
gone
up.
If
this
jars
and
like
this
thing
will
work,
but
if
you,
if
you
only
have
files
to
add
and
there
under
I,
believe
it's
ten
thousand
bytes
as
open
case,
and
it
will
instead
of
using
a
I,
absolutely
ten
dozen
cubed
listed
so
like
all
files
total
so
like
the
under
that
under
that
amount,
the
text
said
the
files
are
small
enough.
Then
it
will
use
a
secret
amount
added
files
include
the
driver
and
include
the
executor
and
yeah
the
other.
K
Does
it's
more
of
a
use
case
specifically
for
specifically
for
those
who
are
before
over
applications
from
yarn,
and
this
is
like
I,
just
more
distinct?
This
is
up.
This
is
a
distinct
use
case
from
adding
our
preferred
volumes.
L
J
K
If
we
create
a
single
secret
for
the
entire
file,
that's
our
profile.
Okay,
these
are
you,
create
a
single
secret
for
every
file.
I've
already
had
higher
for
all
of
the
files
that
you're
writing
is
put
into
one
picker
object
and
we're
going
to.
We
just
laid
out
the
keys,
like
specifics,
just
a
file
names,
and
then
we
just
not
the
same
directly.
Okay,
this.
J
K
G
K
We
are
we
we
limit
for
total
size
of
all
files
to
be
under
ten
thousand.
That
means
that
it
that
all
of
the
files
should
be
able
to
fit
in
one
secret
should
we
find
their
clothes
in
there.
Probably
that's
an
Ontario
correct
that
he
could
probably
I
should
probably
change
it.
So
it's
totalize
after
after
basically
for
encoding
right
now,
I
think
it's
total
size.
The
four
basic
super
encoding
I
guess
it's
more
correct
to
be
after
bases
for
encoding,
though
you.
C
I
These
files
are
likely
going
to
get
encrypted
in
its
CD
I
mean
granted
it's
not
going
to
insulate
that
much
I
still
found.
It
would
be
better
to
keep
it
on
the
lower
side
in
case
there
are
lots
of
these
because
storing
like
a
thousand
one
megabyte
junction
at
CD
completely
kills
it
seen
that
already.
I
K
Powers
of
the
phone
would
do
it
after
that.
It's
not
immediately
clear
to
the
user.
How
like
what
do
I
like
how
much
they
have
to
reduce
your
size
if
it
goes
over,
I,
don't
know,
whereas
if
we
calculate
strictly
for
before
then
like
we
just
say
like
your
trouble
size
was
this
and
yeah
okay
to
do
this.
C
K
K
I'm
running
into
a
problem
right
now
we're
so
with
the
way
smart
dog
files
works.
Is
that
and
it
is
actually
exactly
a
problem
regardless
of
use
register
or
state
server
or
not
Prez
basic
speaking,
implementation
also
has
problem
smart
contract
if
that
files,
either
through
sports
out
files,
are
added
to
the
working
directory
they
do
have
to
be
in
the
working
directory.
Do
like
the
driver
needs
that
your
processor,
so
that
you
basically
know
where
to
find
them
afterwards.
That's
because
this
is
a
way
to
do
that,
but
the
problem
is
right.
K
Now
we
set
the
working
directory
to
be
just
market
installation
directory,
so
I
guess,
theoretically,
at
a
file
called,
say,
item
file
called
spark
batch
class
and
then
or
that
it
have
only
better.
A
better
example
to
get
is
how
you
add
a
file,
that's
like
equal
to
the
names
we
want
to
indirectly
prepare
and
then,
like
you
copy,
and
then
you
possibly
add
you
put
the
file
not
in
the
working
directory
button
to
do
CPE
and
then
like
source
destination.
K
But
if
this
H
is
a
directory,
then
I
mean
you
can
end
up
like
putting
files
in
like
the
wrong
place
or
end
up
over
writing
files
entirely.
So
I
need
to
change
the
working
directory
to
be
an
empty
directory.
I
tried
making
the
files
not
over
rideable,
but
soccer
fire
doctor
like
file.
Permissions,
is
kind
of
weird
plus
cost.
K
K
Orga
directory
is
Kurt.
The
working
directory
is
per
the
only
positive
docker
image.
So
what
we
don't
allow
configuring
the
working
directory
right
now
from
smartphones
parks,
wet
I,
know
you
can
put
the
working
directory
inside
the
pods
back.
I
decided
to
take
this
like
what
we
that
we
don't
expose
it
as
an
option
right
now.
I
think
it's!
Okay!
If
you
don't
yet.
J
There's
one
minor
follow
path
and
I
have
so
if
you
actually
don't
use
empty
turn
and
then
just
use
a
you
know,
whatever
directory
that's
available
inside
darker
image,
you
ends
up
touching
files
in
the
archive
storage,
back-end
like
overlay,
FS,
o
FS
or
whatever
right
and
depending
on
which
storage
back
and
you
choose.
You.
J
J
K
B
K
I
think
it
does
all
the
court
contemporary
directors
which
we
have
so
we
have
to
work
out
a
solution
for
making
the
temporary
directories
not
that
Bobby's
right
now.
I,
don't
think
we'll
do
that.
So
we
should
probably
probably
do
that
and
but
a
button
footage
this
kind.
You
wear
this
particular
use
case
I,
don't
think
I
just
study
the
working
directory
so
mannequins
we
find
okay.
J
J
K
K
I
I
But
it
sounds
like
we
should
be
writing
into
an
integer
volume
like
a
moon
set.
So
maybe
this
deserves
more
investigation.
Can
someone
write
up
an
issue
to
track
this?
Oh.
J
J
I
Not
really
and
I
mean
it
I,
don't
think
we
would
have
done
it.
The
sightless
would
be
the
same
constraints
that
you
have
on
the
existing
set
up,
and
it
is
its
lifetime,
is
managed
by
by
the
system
itself,
so
as
I
think
it
would
be
upgrade
from
where
we're
at.
C
I
I
I
guess
he
isn't
here,
but
in
any
case,
I
wanted
to
quickly
get
some
feedback
on
what
you
guys,
think
of
enabling
client
mode,
and
so
I
tried
this
out.
Actually
the
computer
thing
seems
to
well
to
point
to.
It
seems
to
have
some
issues,
but
mostly
just
works
where
we
can
use
dynamic
allocation
and
all
of
that,
so
it's
really
nice
to
have.
But
yeah
I
was
asking
Saahil
if
he
had
looked
at
like
Jupiter
hub
and
trying
to
get
that
to
work
rather
than
single
Jupiter
instances.
K
Well,
I
commented
I
commented
on
the
pull
request.
I
think
the
current
way
that
client
mode
is
implemented
is
not
exactly
is
required.
Some
work.
Currently
it's
still
using
a
submission,
quiet
and
that's
incorrect
supply
motion
should
be
running
the
scheduler
backend
directly
inside
this
works
with
my
process
and
that's
not
currently
being
done.
So
we
need
to
make
sure
that
that
tweak,
that's
actually
what
is
done.
K
That's
tricky,
because
the
scheduling
back
end
is
pretty
opinion
is
currently
opinionated
towards
having
certain
like
a
big
property
set
by
the
submission
client
which
deploys
it
in
cluster
mode.
So
we
need
to
have
it's
a
bit
like
right
now
we
have
like
kubernetes
cluster
scheduler
back-end,
you
might
need
you
have
a
pretty
high
scheduler
back-end.
This
is
design
a
logistic
yarn.
Has
this
exact
exact
division?
K
K
This
analogous
er
what
we
need
yeah.
So
that's
that's
like
that's
like
my
thoughts
there.
If
that
works
and
you're
within
a
cluster
or
like
grooves
you're
within
a
pod
in,
like
my
I
imagine
a
it
could
be
something
like
you're
just
within
a
pond,
and
you
started
like
your
own
travel
process.
However,
you
wanted
and
then
you
just
create
a
single
spark
context
could
burn
any
quick.
You
forbidding
me
timer
and
that
should
just
create,
like
this
controller
back-end
like
immediately
and
yet
is
it
that
should
just
work.
K
N
This
is
Tristan
from
clutter
and
unfortunately,
only
have
five
minutes.
I
have
a
meeting
coming
up
but
and
I
don't
think
the
heel
is
on
the
call.
Who
is
really
the
one
who's
look
at
this
closely,
but
just
so
I
understand
and
I
have
not
looked
at
the
code
in
depth
so,
but
just
so
I
understand
the
issue
exactly
Oh.
What
is
the
problem
if
this
is
creating?
So
what
what's
the
problem
I
mean
when
I
saw
the
demo
of
this?
At
least
it
worked
as
I
would
have
expect
from
an
end
user
perspective.
N
K
Problem
is
that
you're
we
creating
a
you're
creating
a
driver
pod
when,
in
fact,
that
pod
that
you're
running
it
I
should
be
the
driver
pod
itself,
so
like
I,
say
I
logic,
I
want
to
have
a
process
and
I
say
run
my
spark
application
and
like
performed
within
that
pot,
I,
say
I
want
to
start
my
spark
application
right
now.
I
understand,
I,
understand
it,
based
on
based
on
some
of
the
based
on
some
of
the
comments
that
were
written
and
based
on
understanding
the
code
it
will.
K
It
will
create
a
well
for
another
process
that
creates
another
pod.
That
then
runs
that
then,
once
the
driver-
and
that
seems
like
a
little
bit
see
it
seems
like
actually
needs
to
have
like
both
both
pods
in
that
case
right.
So.
N
N
You
know
the
he
was
running
Jupiter,
you
know
using
importing
PI
spark
and
I
would
assume
that
the
driver
would
have
to
run
in
that
process
where
Jupiter
is
running
or
where
the
Python
kernel
is
running
so
you're
suggesting
that
there
might
be
an
extraneous
pod
being
created,
that's
kind
of
like
a
pseudo
wrecker
pod
or
something
somehow.
The
code
is
kicking
off
a
driver
as
well
I.
Think.
K
J
K
B
N
That
was
my
certainly
impressive,
live
again
about.
This
was
me
as
the
end
user.
It
did
exactly
what
I
expected
as
an
end
user.
You
know
what
I
could
serve
now.
Things
could
be,
you
know,
created
really
fast
and
stop,
and
you
know
there
could
be
something
weird
going
on,
but
as
an
end
user,
it
looked
like
it
was
doing
exactly
expected.
N
K
A
C
N
K
Basically,
like
I
should
never
be
able
chat
all
in
this
process,
so
we
should
be
hated.
We
should
do
some
like
code
trace
micro
tracing
in
the
IDE
and
make
sure
we're
not
doing
that.
It's
pretty
tricky,
because
sports
limit
invoke
instantiates
like
the
classic
the
trial
classes,
whether
legal
mission,
client
class
or
the
main
class
of
the
user
and
like
eveni
our
client
question,
was
it
saturated
reflectively.
So
it's
kind
of
hard
to
trace
down
like
where,
like
whether
or
not
we
actually
make
it
I.
J
I
C
So
I
just
wanted
to
drop
a
comment,
and
maybe
I
should
actually
just
file
an
issue
this,
what's
going
through
the
release
process
there,
the
2.2,
Scott,
then
I
realized,
and
we
can
feel
like
an
all-clear.
What
command
flag.
We
should
use
to
build
early
polls
to
make
them
pass
a
little
sting
and
stuff
like
that.
So
I
got
something
that
basically
eventually
passed
or
that
reported
the
command
labels
on
the
ticket.
K
Andrew,
can
you
share
what
we've
been
used
because
I
we've
we've
also
been
building
I
thought
week,
I
think
we've
also
been
releasing
these
I
think
we
handle
some
ones
released
in
the
past.
If
you
get
you,
can
you
post
a
command
that
you
use,
maybe
to
maybe
do
it
maybe
to
the
get
up
issue
or
jogging
stock
set
like
the
Eric?
Is
it
yep.
L
Yeah
I
have
a
script
that
runs
through
a
bunch
of
things.
It'll
need
some
slight
tweaks,
because
we've
used
in
addition
to
forking
spark.
We
also
afford
to
do
so.
It
runs
the
pouncer
version
of
it
as
well
make
a
couple
small
changes,
but
I
can
collect
script
if
you're
very
easily
convertible.
Sorry.
K
L
K
H
I
K
O
C
P
Yes,
so
I
guess
four
four
and
four
I'm
running
out
the
universe
that
otherwise,
when
we
should
be
looking
when
migrating
to
two
points
to
accretion
cool
the
only
goal.
One
thing
that
I
found
kind
of
odd
was
I
was
trying
to
run
two
different
tests
on
top
of
each
other,
one
which
runs
on
a
K
in
it.
P
While
reading
key
tab
sleep
ass
into
summation
five
and
one
that
requires
commissioning
the
keeps
has
insufficient
client
arm
and
the
first
time,
I
run
it
with
the
key
tabs
everything
passes
and
then,
when
the
precision
volumes
are
taken
down
with
the
second
run
within
the
same
namespace,
sorry
within
a
different
name
space,
even
the
KDC
note
is
saying
that
that
the
key
tab
that
was
created
was
ordered.
That
already
exists.
P
So
it's
almost
like
the
person's
volume
didn't
actually
approach
to
all
the
combines
and
it's
being
destroyed,
which
I
thought
was
quite
weird,
I,
don't
know
if
there
was
ever
a
case
where,
after
you
do
dot
deleted
volumes
that
persists
anyway,
what
kind
of
persistent
volume
it
is
it
I
did
it
on
host
path?
Call
me
quickly
check
the
gamma
five,
its.
I
Host
pad
there
is
no
cleanup
per
se,
so
if
you
were
to
attach
back
to
that
same
path,
it's
going
to
stick
around
on
the
node.
So
till
we
have
precision
local
storage,
yeah
host
Pat
volumes
are
not
going
to
purge
it
for
you,
if
you
want
to
use
something,
that's
managed
like
send
an
empty
door,
is
the
better
choice.
J
P
J
P
J
I
So
if
you
add
an
owner
reference
from
one
kubernetes
resource
to
another,
when
the
owner
dies
or
is
you
know,
deleted
from
the
API
there's
the
pod
or
actually
there's
a
garbage
collector
and
inside
kinetic,
that
would
take
care
of
cleaning
up
all
the
dependence
on
that
resource.
So
right
now
we
use
that
within
the
driver,
part
like
all
the
executor
pods
have
owner
references,
pointing
back
to
the
driver
and
the
secrets
that
Matt
was
talking
about
for
mounting
files.
J
J
You
actually
have
a
20
front
code
paths
if
I'm
correct.
Oh,
you
know,
one
code
path
is
where
the
driver
actually
creates
the
secret
file
and
own
it
or
the
lifetime
of
it.
There's
a
the
other
foot
path
is,
of
course,
a
pre-existing
secret.
The
user
is
applied
to
the
driver
right.
So
for
the
second
case,
you
didn't.
B
J
I
I
C
I
L
I
P
D
Yeah,
so
there's
been
a
lot
going
on
in
the
air,
take
one
start
being
a
lot
of
bloopers
on
my
I,
my
microphone,
but
yeah.
So
I
never
see
Cooper
News
team
last
week
with
Anirudh,
as
well
as
a
few
members
of
his
team,
and
got
some
really
good
feedback
on
that.
We
have
grant
Nicholas
and
Victor
Montero
who
are
joining
us
and
what
I
would
really
love
to
do
is
I'd
love
to
just
kind
of
go
over
the
basic
road
map.
D
I
don't
set
up
on
you
reading
my
discuss
to
just
have
get
a
sense
of
like
what
you
guys
think
would
be
the
best
routes
forward,
how
to
make
sure
we're
staying
inside
of
like
today's
best
practice
it
and
yeah.
Let
me
let
me
quickly:
grab
both
items
and
I'll,
just
gonna
dump
them
in
and
whatever
we
can
get
through
in
20
minutes.
So.
D
The
first
thing
that
we
wanted
to
discuss
was
using
custom
resources
for
deployment.
Can
someone
turn
off
their
might
their
microphone
I
think
that
I'm
just
getting
a
lot
of
feedback.
D
So
I
think
that,
if
there's
almost
as
though
they're
like
any
strong
objections,
I
think
that
being
able
to
work
towards
like
the
future,
as
opposed
to
kind
of
keeping
things
in
the
air
plus
source
code,
where
it
would
become
problematic
for
maintenance
for
three
months
for
open
ships,
I
think
would
be
a
fair
trade-off.
I
D
Yeah
I
mean
we
I
would
definitely
be
find.
I
talked
about
that.
Maybe
we
could
chat
up
up
the
meeting,
but
I
know
that
for
a
lot
of
the
work,
we're
doing
because
we're
kind
of
trying
to
abstract
things
away
from
our
users
so
for
one
example
is
like
if
we
want
to
launch
a
spark
cluster
rather
than
forcing
people
who
might
not
necessarily
know
those
details
to
like
say:
oh,
we
want
four
gigs
for
executors
and
sixteen
executors.
D
D
One
other
thing
that
another
thing
that
seemed
to
be
discussed
was
basically
that
trying
to
merge
in
the
operator,
and
the
executor
makes
for
a
very
large
clunky
PR,
and
so
it
might
be
better
to
kind
of
merge
in
the
just
the
kubernetes
operator
code
as
the
first
release
and
then
kind
of
build
up
some
goodwill
with
the
airflo
community
as
well
as
get
in
some
of
these.
The
changes
we
want
to
make
to
the
airflow
internal
code.
Before
we
try
to
merge
in
the
executor,
grant
and
Victor.
D
Airflow
air
flow
has
was
what
are
called
it,
so
air
flow
operators
are
kind
of
what's
on
the
user
side,
so
the
user
actually
defines
like
I
want
this
to
be
a
Python
operator,
and
then
you,
the
Python
function
or
I,
want
to
be
a
spark
submit
operator.
You
put
a
spark
to
make
commits.
The
executor
is
actually
what
works
internally.
So
what
happens
in
air
flow
is
that
air
flow
has
an
internal
scheduler
and
every
time
that
scheduler
is
a
heartbeat.
D
It
checks
to
see
if
there's
any
tasks
in
a
queue,
and
if
there
are
any
tasks
that
all
their
dependencies
have
been
met,
they
then
use
the
whatever
executor
has
been
defined
to
say:
hey
executor,
we
have
this
task.
You
figure
out
a
way
to
run
this
task
so
right
now,
air
flow
has
like
a
celery
executor.
That'll
use
pickles
to
send
it
to
a
celery
slave.
It
has
a
Mesa
executor
that
also
uses
typical,
listen
to
a
Mesa
slave,
and
so
with
the
kubernetes
executors.
D
I
B
A
See
a
lot
of
oozy
and
we
see
a
little
bit
of
airflow
I,
don't
think
that
anything
to
take
away
from
the
airflow.
It's
just
you
know.
We
have
a
lot
of
customers
who
have
things
that
are
in
production
for
a
long
time.
So
they've
got
a
lot
of
history.
Okay,.
D
Yeah,
so
actually
one
thing
I
did
want
to
really
bring
up
with
with
the
general
kind
of
larger
consortium,
because
when
when
Anirudh
and
I
were
talking
on
Thursday,
we,
we
weren't
really
able
to
figure
out
like
a
really
good
solution.
For
that
this
and
like
I
guess
the
the.
D
What
we're
kind
of
trying
to
figure
out
is
a
best
solution
for
data
for
the
actual
transportation
of
the
dyed
files,
from
the
the
kind
of
main
airflow
instance
to
the
slave
instances,
and
the
problem
we
ran
into
is,
according
to
the
members
of
the
kubernetes
team,
the
git
repo
volume
actually
was
not
made
with
maintainability
in
mind.
So
there's
a
lot
of
kind
of
code
concerns
with
it.
D
D
But
since
now,
I
know
that
the
git
repo
volume
is
kind
of
questionable,
like
one
thought
would
be
actually
placing
that
placing
those
kind
of
placing
a
special
case
for
github.
In
the
custom
controller
that
we're
building
but
I'd
also
be
open
to
hearing,
if
you
guys
have
good
experiences,
any
form
of
distributed
file
system
that
you
would
recommend
it's
kind
of
like
a
baseline
for
transporting
these
tags
between
the
master
allayed.
When.
D
C
Yeah,
like
I'm,
not
sure
if
it's
like
actually
github
specific
I'd,
have
to
look
into
that,
but
I
found
it
I
on
it
through
some
github
blog,
where
you
can
basically
install
a
plugin,
and
it's
specifically
designed
to
support
like
managing
very
large
files
through
your
git
repository
interesting.
B
D
J
A
D
Don't
think
that's
a
huge
concern,
because
unless
you
know,
unless
you're
I'll
say
that
if
you're
pulling
so
many
Pollock
parallelism
is
a
setting
in
airflow
that
you
can
say
this
is
the
max
number
I
actually
want
to
run
at
one
time,
and
if
you're
running
that
many
tasks,
then
that's
kind
of
like
a
that's.
But
that's
something
that
you
need
to
manage
incentive
like
if
you're
running
ten
thousand
tasks,
all
of
which
pulling
20
megabytes
each,
then
maybe
you
should
just
lower
the
parallelism
level
or
the
other
thing
that
we
did.
D
Basically,
dagon
quarter
like
a
dag,
important
and
basing
we're
going
to
offer
github
and
maybe
two
or
three
others,
and
if
you
really,
if
you,
if
you
have
like
a
large
scale
solution
and
you
have
some
preferred
distributed
file
system,
we
give
you
the
ability
to
just
basically
create
your
own
file
amounts
and
then
that
will
immediately
attach
you
to
all
the
slaves.
So.
A
If
I
understand
that
right,
you're,
basically
saying
in
a
normal
case,
you
don't
mind
that
there's
all
these
multiple
requests
to
an
external
resource
that
are
going
to
get
the
same
answer
because
in
most
cases
you're
assuming
it's
not
going
to
be
so
big
that
you
need
to
make
the
network
local
the
cluster.
Exactly.
D
D
So
I
mean
I
realize
that
the
distributive
there's
no
there's
no
good
answers
in
this.
The
distributed
file
system
world
right
now,
I
know
that,
like
NFS
has
a
lot
of
issues
with
like
locking
and
stuff
like
that.
But
there's
there's
been
some
stuff.
What
anirudh?
What's
called
Mineo
or
yeah
yeah,
so
I
mean
like
that
might
be
saying
like
something
that
might
be
worth
looking
into
in
terms
of
like
I.
Don't
know
if
anyone's
working
on
this
Anirudh
but
I
I
might
be
if
it
was
working.
D
C
Been
some
problems
with
Niall
like
we
were
exploring
using
it,
Burano
I
guess,
look
kind
of
similar
use
cases
like
you
know:
containerized
s3,
inside
of
a
coop,
slash,
openshift,
cluster,
okay
and
I'm,
not
sure
like
to
what
degree.
This
was
like
some
weird
intersection
of
men
I
own
spark,
but
there
is
some
bug:
that's
preventing
us
from
using
it,
and
it's
been
outstanding
for
a
depressingly
long
time
now
and
I.
C
Yeah
I
mean
they
were
responsive
in
a
sense
that
we
got
like
you
know
some
diagnosis
and
some
roadmap,
but
it
had
to
do
with
like
upgrading
to
Hadoop
like
2.8,
which
of
course
you
know
itself
took
a
while,
and
then
they
claimed
that
he
had
to
run
some
kind
of
cross-platform
unique
test
found
some
reason
they
didn't
like.
For
some
reason
it
took
forever
to
like
run
this
testing
I.
Can
it's
not
everything
I
mean
currently,
people
use
Mineo,
so
I'm
like
it
may
yet,
where
you're
useless,
okay
I
mean.
D
The
the
the
two
thoughts
I'm
having
there
is
it.
On
the
one
hand,
it
concerns
me
that
it
takes
a
community
that
long
to
handle
something
with
such
a
valuable
use
case,
but
it's
also
that
I
imagine
that,
from
what
we're
doing
is
we're
a
much
simpler
use
case.
We
would
literally
just
be
like
running
a
simple
like
hey
give
me
this
folder
or
whatever,
but
sorry
I
cut
them
out
there
I'm.
J
E
C
J
C
C
D
D
I
mean
we
looked,
we
did
take
some
time
to
look
into
WebDAV
solutions,
although
I
think
that
I'm
not
I'm,
not
sure
that
that
would
solve
the
problem,
because
people
would
still
be
like
importing
the
entire
directory,
which
would
basically
all
the
same
things.
What
the
good
thing
is,
I
guess.
C
D
Ok,
I
see
what
you're
saying
like
in
a
sense
of
like
have
have
some
sort
of
a
web
dab.
That's
just
continually
synced
with
github
and
in
doing
so
like
the
slave
instances,
would
query
the
web
dab
instead
of
the
external
github,
which
could
kind
of
reduce
the
possibility
of
like
overloading
the
github
instances,
and
also
we
could
just
have
like
multiple
instances
of
this
web
dev
running
to
kind
of
keep
up
with
the
scalability
of
the
system.
Yeah.
C
F
D
K
B
K
Probably
someone
from
use
you
can
make
to
it
with
a
to
make
it
scale
out
better,
because
right
now,
just
sorts
on
it
just
sort
of
darker
disk.
It
doesn't
use
any
and
although
I
guess
you
could
configure
the
volume
so
pretty
pretty
easily
yeah
good
one
optimist
and
and
and
I
suppose
it
uploaded
a
download
end
point
we're
just
like
just
takes
an
input
just
like
you
just
provide
it
with
the
cursor
diamond
data
input
and
it
looks
like
a
treatment
in
on
output.
What's
what's
particularly
unique
about
the
resource
ad
server?
K
Is
that
once
you
push
data,
you
receive
a
secure
token,
you
have
to
provide
a
secure
torque
in
order
to
receive
the
file,
so
that
basically
gives
you
like.
It,
gives
you
a
very
simple
and
yet
for
our
use
cases
it's
mark
at
least
effective
mode
of
security
and
authorization
where
you
have
to
like
the
only
thing
that
uploaded
this
thing
can
be
the
thing
that
downloads
the
same
so
I
don't
matter,
but
for
that
I'm
not
too
sure.
K
D
K
O
I
think
Jude's
kind
of
like-
and
this
is
grant
by
the
way,
there's
like
two
separate
problems
here,
which
is
one
getting
airflow
to
recognize
the
bag
and
then
two
getting
all
the
workers
to
actually
like
pick
up
the
code,
so
they
can
execute
it
and
I
think
those
should
be
separate
because
publishing
like
a
dag,
so
it
can
be
run
versus
you
know
actually
running.
O
It
are
two
separate
things,
and
so,
whatever
like,
what
I
wouldn't
want
to
see
would
be
like
when
having
a
solution,
we're
uploading
code
to
some
like
you
know
some
some
web
dab
type
thing
where
requires,
if
you
have
like
a
dag,
that's
already
running,
would
like
over
write
the
code,
that's
already
existing,
so
just
kind
of
like
separating
up
those
two
things.
You
need
to
be
careful.
D
D
D
What
grant?
What
do
you
think
about
that,
like
just
like
kinda
having
like
that
when
you
do
the
query,
you
have
to
query
it
with
the
reverse
revision
version
and
when
the
executor
creates
the
creates
the
slave
instance,
it
creates
it
with
like
an
environment
variable
or
a
config
map,
just
pointing
to
the
like
with
that
revision,
so
that
when
it
starts
up
and
knows
what
to
pull
yeah.