►
From YouTube: How Spotify Leverages Flyte To Coordinate Financial A... Haytham Abuelfutuh & Dylan Wilder Patterson
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How Spotify Leverages Flyte To Coordinate Financial Analytics Company-Wide - Haytham Abuelfutuh, Union.ai & Dylan Wilder Patterson, Spotify
Kubernetes’ popularity stems from its declarative yaml based API and focus on versioning and other best practices of developing micro-services. Therein also lies its weakness, where users that are not conversant with infrastructure are inadvertently discouraged from leveraging the power of Kubernetes. Using an example application at Spotify that powers their financial platform, the talk examines:
* The specific operational needs of ML and Data Engineering (MLOps/DataOps) in contrast to DevOps.
* Best practices for developing maintainable ML and Data applications.
* How Flyte can be used to bridge these gaps for users with varying technical proficiency.
A
We
are
going
to
talk
today
about
how
spotify
has
used
flight
to
revamp
their
financial
analytics
platform,
and
my
name
is
haitham
abruptor.
I
am
one
of
the
co-founders
and
maintainers
of
the
flight.
Open
source
projects
and
joining
me
today
is
dylan
wilder
from
spotify,
but
unfortunately,
due
to
personal
circumstances,
he
wasn't
able
to
be
here
in
person,
but
he
was
gracious
enough
to
share
his
thoughts
in
a
recorded
video.
So
without
further
ado,
I
let
him
introduce
himself
and
talk
about
what
spotify
has
been
trying
to
do.
B
My
name
is
dylan
oops,
I'm
an
engineering
manager
on
a
team
called
vivaldi
within
spotify.
We
work
in
the
finance
department
and
I'm
also
the
tech
lead
on
a
project
called
one
model,
and
so
that's
what
this
is
about
and
one
model
is
really
powered
by
flight.
I
just
want
to
give
some
background
on
what
we're
talking
about.
So
this
is
really
one
model
is
a
financial
forecasting
problem
and
quickly.
What
financial
forecasting
is
is
every
quarter
as
part
of
compliance
objectives.
B
B
What
revenues
do
we
think
will
take
in
what
cost
do
we
think
we'll
have
to
spend
in
order
to
get
there,
as
in
addition
to
just
being
part
of
you
know,
requirement?
It
also
forms
a
big
part
of
spotify's
business
planning.
You
can
imagine
this
is
useful
for
investment
investors,
it's
just
as
useful
internally
for
the
people
making
decisions,
the
c-suite
etc.
B
B
We
deal
with
licensors
music
law
stuff
like
that
and
so
the
expertise
in
order
to
kind
of
run.
This
system
is
scattered
across
a
lot
of
different
domain
experts,
and
so
as
it
stands,
this
process
takes
about
three
to
four
weeks.
Every
quarter
span
as
many
as
eight
different
teams.
These
teams
work
in
silos.
Some
of
these
pieces
are
excel
models
and
then,
at
the
end
of
the
day
you
might
imagine
team
one
is
handing
off
a
google
sheet
to
team
two
to
start
running.
B
B
We
want
to
fix
these
manual
handoffs.
We
want
to
speed
up
the
the
feedback
time
to
do
this
process
and
we
want,
to
you,
know,
reduce
for
errors
and
a
secondary
piece
of
this
is
that
because
it
takes
so
long
to
run,
we
can
only
really
do
this
once
per
quarter.
It's
too
much
of
an
effort,
investment,
and
so
we
think
by
automating
this
we
can
really
unlock
the
second
piece
of
the
problem,
which
is
business,
casing
and
scenario
analysis
if
we
can
run
this
problem
end
to
end.
B
You
know
within
a
couple
hours,
then
these
people
making
the
decisions
can
come
in
and
say
well
what
happens
if
we
increase?
You
know
our
subs
by
20
million
people
in
india
versus
what
happens
if
we
open
a
new
market
in
eastern
europe,
or
something
like
that,
so
you
can
start
to
ask
these
questions
more
frequently
without
having
you
humans
in
the
loop
in
order
to
answer
them.
B
There
we
go
so
this
is
high
level
overview.
What
it
looks
like
there's
again
it's
a
complicated
process.
Each
of
these
nodes
in
this
graph
is
actually
you
know.
This,
isn't
one
flight
task
per
se?
It's
a
bunch
of
different
things.
These
are
just
high
level
logical
pieces
to
spotify's
business
model
and,
as
you
can
see
here,
each
individual
one
of
these
might
be
owned
by
a
different
team
and
there's
lots
of
you
know
complex
dependencies
between
teams
and
there's.
B
No,
I
mean
this
is
our
best
shot
at
what
this
thing
looks
like
there's,
no,
like
person
who's
in
charge
of
like
the
overall
end-to-end
thing.
Who
knows
exactly
what
this
looks
like
it's
it's
very
much
a
distributed
ownership
model,
so
yeah,
one
model,
automating
all
the
components,
and
once
we
do
that
we
can
kind
of
unlock
this
scenario.
This
business
case
analysis
scenario,
and
you
know
we
think
that'll
be
really
cool.
A
All
right
and
thank
you
dylan,
so
I
thought
this
you
can
think
of
this
part
as
well.
This
is
our
sort
of
requirement
gathering
phase.
Now
we
have
a
problem
statement
and
this
is
spotify
right.
You
can
imagine
that
problem
replicates
to
a
lot
of
other,
even
smaller
companies
that
don't
have
the
capabilities
they
do
on
the
infrastructure.
They
do,
and
I
thought
before
I
talk
about
how
flight
has
helped
them
and
has
been
used
in
lyft
as
well.
A
A
B
A
Systems
and
monitoring-
and
you
know
all
of
these
things-
incident
management
and
responses,
and
you
can
see
a
lot
of
them
in
the
showcase
floor
downstairs
if
you've
been
to
and
a
couple
of
years
after
that,
this
term
came
to
be
again.
Another
term
got
coined
rpa,
robotic
process,
automation
and
it
was
more
of
the
the
business
side
of
you
know.
A
A
And
since
then
again
we
have
been
developing
tools
and
and
processes
and
trying
to
make
you
know
the
life
of
male
engineers
and
data
scientists
better.
I
you
know
maybe
before
then
I
would
say
there
are.
There
were
a
lot
of
attempts
to
sort
of
force,
some
of
the
dev
tools,
devops
tools
onto
ml
engineers
and
they
conceptually
sort
of
work.
A
If
you
want
to,
you
know,
deploy
your
model
yeah,
you
can
write
a
ci
system
for
the
ci
use,
one
of
the
existing
ci
tools
for
it.
If
you
have
you
know
multiple
steps
and
training
and
pulling
data,
you
can,
you
know
over,
impose
it
into
onto
an
existing
ci
tool,
but
you
know
it
didn't
really
quite
work.
A
I
wanted
to
summarize
a
little
bit
some
of
the
the
key
differences
I
see
between
the
requirements,
for
you
know
the
devops
and
then
ml,
ops,
data
and
email,
ops,
and
you
know
again
from
a
10
000
foot
view
they
look
pretty
much
the
same
set
of
problems,
but
the
the
when
you
look
at
the
details.
The
inputs,
for
example,
for
a
data
rml
pipeline
or
training
model
might
be
it's
not
just
the
code.
You
wrote
to
train
a
model,
it's
also
the
terabytes
of
data.
A
You
want
to
train
the
model
and
when
those
changes,
your
outputs
change
in
the
dev
world,
you
just
have
code
and
it's
you
know
usually
in
megabytes
and
it's
okay.
If
you
have
you
know,
10
or
100
different
ci
pipelines
that
just
check
out
the
code
every
time
right,
because
it's
it's
it's
cheap.
You
can't
really
do
the
same.
A
Similarly,
for
the
actual
job
that
runs
in
in
the
dev
world,
yeah
the
ci
system
maybe
takes
minutes
to
finish.
I
know
20
minutes
if
it's
a
bit
slow
and
you
start
getting
grumpy
and
start
optimizing
it
in
the
ml
world.
If
you
have
a
training
job,
you
know
it's
very
normal
for
these
to
take
hours
even
days.
In
some
cases
we
have
cases
where
they
they
run
at
lyft,
training,
building
the
map
of
the
world
essentially-
and
that
gets
refreshed.
A
Last
time
there
are
a
lot
of
other
requirements
that
stem
from
just
this
one
change
in
the
requirements,
whereas
in
for
the
iteration,
for
example,
they
people
still
want
pretty
much
the
same
flexibility
right.
They
want
to
make
a
change
want
to
test
it
locally
write
unit
tests.
Somebody
else
makes
a
change.
I
want
to
make
sure
their
change
is
validated
before
it
runs.
A
You
want
to
be
able
to.
You
know,
promote
things
that
succeed
to
production.
So
you
want
the
same
semantics.
You
want
the
monitoring
part.
You
want
the
incident
management
and
alerting
all
of
these
things.
You
want
the
the
as
goals
the
trick
is
to
make
sure
they
we
offer
them
or
make
them
available
to
you
in
where
you
are,
where
you
are,
writing
your
code
or
where
you
are
developing
your
workflows
and
obviously
the
stakeholders
are
different
between
the
two
and
I
want
in
the
next
few
slides.
A
I
will
talk
about
how
like
flight
addresses.
Some
of
these
concerns.
It's
really
our
take
on
on
the
problem.
Flight
has
been
developed
at
lyft,
we
open
source
flight
two
years
ago,
and
since
then
there
are
tens
of
companies
there
that
adopted
the
open
source
flight
in
for
for
various
different
platforms
inside
companies
from
building
ml
platforms
on
top
data
and
data
ops
on
top
to
just
purely
using
flight
flights.
A
There's
companies
that
sort
of
resell
flight
or
repackage
flight
with
a
you
know,
vertical
integration
in
certain
fields
and
and
flight
would
power.
You
know
the
underlying
orchestration
parts
flight
is
a
workflow
automation
platform
for
business,
critical
data
and
ml
processes.
It
might
sound
mouthful,
but
hopefully
it
will.
It
will
be
clearer
as
we
go
on.
A
We
have
a
ton
of
integrations-
and
I
think
at
this
point-
it's
probably
fair
to
say
most
of
these
are
built
by
the
community
more
than
you
know
the
original
authors
of
the
platform,
so
they
were
all
built
to
be
used.
They
were
not
just
built
to
you
know,
somebody's
keeping
count
of
how
many
integrations
are
on
the
platform.
A
A
This
is
our
and
you
know
any
any
product
I
guess
starts
with
some
assumptions,
and
this
is
our
assumption
for
how
we
think
an
ml
engineer.
You
know,
go
through
the
process
of
developing
a
workflow
or
a
pipeline
or
training
model
or
data
scientist
does
the
same.
They
start
on
the
left
here
with
some
idea,
so
they
have
some.
A
Business
logic
to
try
out
something,
maybe
query:
some
data
process
data
a
certain
way
three
in
the
model
things
like
that,
and
they
want
to
write
it
locally.
They
want
to
start
up
their
laptop
open
up
their
favorite,
ide
or
jupyter
notebook,
or
what
have
you
and
just
write
it
there?
If
you
start
asking
them
to
well,
you
can't
do
that.
You
have
to
go
somewhere
else
to
start
writing.
Then
things
fell
apart.
A
They
want
to
be
in
their
favorite
tools,
they're
comfortable
environments,
where
they
can
pull
in
requirements
that
they
have,
or
they
want
the
library
requirements
and
just
write
their
code.
You
want
to
take
you
know,
restrictions
on
which
versions
of
libraries
they
can
and
cannot
use
because
some
other
you
know
person
in
the
team
is
using
an
older
version.
Now
they
cannot
upgrade.
You
know
things
like
that.
They
just
want
to
sit
down,
write
the
code.
A
They
want
to
write
and
test
it
right
there,
but
at
the
same
time,
once
they
exceed
the
capacity
or
maybe
of
their
laptop,
maybe
they're
just
running
on
like
sample
data
on
their
laptop
and
now
they
want
to
run
on
the
full
data
set.
They
should
also
be
able
to
very
confidently
run
the
codes
that
they
wrote
in
a
remote
environment
where
they
can.
A
You
know,
leverage,
bigger
and
more
beefier
machines,
and
you
know
larger
compute
environments
to
process
bigger
and
bigger
data
sets
and
that
process
should
give
them
should
be,
should
guarantee,
to
a
certain
degree
that
the
code
will
behave
exactly
the
same.
If
it
worked
on
my
machine,
it
should
work
remotely.
You
know
all
the
problems
of
well,
you
have
a
mismatching
requirements
or
mismatching
versions
of
some
dependency,
like
all
of
that
should
just
not
be
a
thing
that
they
think
about
once
they
are
ready.
A
A
You
should
also
be
able
to
promote
the
successful
workflows
right
or
things
that
produce
the
artifacts
that
you
can
you
deem
you
know
good,
maybe
through
running
other
tests
and
so
on.
You
should
be
able
to
promote
them
to
production.
Just
like
we
do
with
services.
You
know
you're
deployed
to
staging
environment,
and
you
like
multiple
versions
and
the
version
that
passes
all
the
end-to-end
tests,
integration
tests
and
so
on.
You
promote
that
to
production.
It
should
give
you
the
same
semantics
but
again
within
an
environment
that
looks
familiar
to
them.
A
Once
again.
Those
things
happen.
You
should
be
able
to
run
on
schedules
right,
especially
for
data
processing
pipelines.
We
see
that
very
often
right
data
keeps
coming
in
from
external
sensors
from
you
know,
external
sources,
and
you
want
to
keep
processing
the
data
as
they
come
in,
so
you
want
to
trigger
those
pipelines
on
schedules
or
based
on
events,
and
you
want
to
monitor
them.
A
You
want
to
get
notified
when
things
fail
alerts
and
right,
like
all
of
the
conv,
all
the
things
we
take
for
granted
as
software
engineers
like
they
need
to
be
offered
for
these
pipelines
as
well
and
again.
The
key
point
here
is:
we
should
meet
them
where
they
are
shouldn't.
Ask
them
to
like
adopt
all
of
these.
You
know
different
tools
for
to
make
this
work.
A
A
A
We
do
require
you
to
clear
those,
and
we
think
I
know
it's
a
contentious
point,
but
my
take
is
any
production
code
should
really
be
strongly
typed.
I
know
types
are
options
in
python
and
in
many
cases
when
you
prototype,
you
don't
necessarily
care
about
them.
But
if
this
is
code
that
you
are
writing
to
be,
you
know
productionized,
I
think
they
should
be.
A
You
should
declare
the
types
and
flight
uses
the
types
in
various
other
ways,
we'll
we'll
talk
about
in
as
we
progress
to
ensure
you
know,
tasks
fit
together
and
to
ensure
you
know,
caching,
works
and
so
on.
So
once
you
have
your
function
right,
you
can
run
it
locally.
You
can
write
unit
tests,
it's
just
a
python
function.
All
you
need
to
do
is
add
this
top
annotation
the
add
task,
and
now
it
becomes
a
task,
a
flight
tasks.
Now
flight
knows
that
this
is
a
sort
of
an
atomic
execution
unit.
A
It
knows
that
it
can
take
this
plus.
You
know,
of
course,
any
other
function.
It
call
and
run
that
in
separate
hardware,
sandbox
environments,
it
can
run
in
parallel
if
it
needs
to
it's
like
a
you
know,
it's
a
stand-alone
sort
of
function
or
task
in
our
language,
and
we
wrote
a
couple
of
those
and
then
the
third
one
here
again
a
function,
but
we
marked
this
as
a
workflow.
A
What
does
what
that?
Does?
It
tells
flight
that
for
when
you
execute
this
workflow,
you
see
the
body
here.
Don't
really
call
the
tasks
right,
so
we
you
see
me
calling
pay
multiplier
and
total
spend.
It
will
not
really
call
these
functions.
All
it
will
do.
Is
it
will
record
that
you
want
to
call
these
tasks?
This
is
the
data
flow.
You
want
to
pass
the
inputs
that
come
from.
You
know.
The
workflow
are
sent
as
inputs
to
this
task,
and
the
output
of
this
task
is
sent
to
this
as
right.
A
So
it's
it
builds
up
the
three
of
the
the
graph
or
the
dag
based
on
executing
the
code.
You
write
here
and
yes
in
this
example-
is
just
a
one
line,
but
this
can
be
arbitrary,
complex
and
we
will
just
execute
that
once
statically,
with
no
inputs
to
understand
the
graph
of
execution
and
from
that
point
on
it
gets
transferred
into
our
own
representation
of
graphs.
We
use
protobuf
to
represent
the
typing
and
the
graphs
and
all
of
these
information,
and
it
enables
flight
to
be
targeted
through
different
sdks.
A
We
started
with
python,
but
spotify
contributed
a
java,
sdk
and
a
scada
sdk
and
you
they
can
all
interrupt
right.
So
some
people
write
like
a
spark
java
task
to
process
things,
but
then
their
teammates
write
python
tasks
and
python
workflows
and
they
call
this
java
task
right.
They
are
all
completely
addressable
and
uniquely
addressable
and
run
in
complete
sandbox
environments.
A
A
In
this
example,
everything
here
is
going
to
be
in
one
container,
so
you
will
care,
but
but
yeah
in
in
a
real
environment.
These
things
could
be
spread
over
multiple
github
repos
like
just
completely
independent
on
each
other
and
then.
Lastly,
even
after
you
know
marking
things
as
this,
you
can
still
obviously
call
them
as
python
functions,
because
they
are
python
functions.
So
you
can
write.
A
You
know
you
can
use
pi
test
frameworks
like
all
the
testing
you
can
do
locally
in
python,
so
you
can
add
as
many
checks
as
you
want,
for
you
know,
for
anyone
else
who
comes
in
and
modifies
this
code
to
make
sure
you
know
that
it
behaves
as
you
expect
it
to
given
the
inputs,
and
there
are
a
couple
of
nifty
features
here.
You
can
add
on
top
right
there's
this
on
top.
I
added
cache
equals
true
to
this
one
task,
and
this
is
me
telling
flight
that
I
know
this
task
is
deterministic.
A
A
So
if
somebody
else
in
my
team
or
in
the
company
calls
the
exactly
the
same
task
with
the
same
version
with
the
same
inputs,
I
will
not
try
it
again.
Flight
will
not
run
it
again.
It
will
just
say
well,
here
is
the
output.
Somebody
else
you
know
saved.
You
the
time
and
and
ran
this
and
again
all
of
this
is
tracked,
so
you
can
always
go
and
see
well,
which
execution
exactly
produced
like
ran
the
first
time
for
this
input,
and
you
can
go
there
and
look
at
the
logs
and
all
of
that.
A
So
it's
all
tracked
in
the
system
once
you
know
things
run
locally
and
you
are
happy
with
what
you
see
and
you
want
to
maybe
run
in
a
remote
environment
with
more
resources.
We
give
you,
you
know,
capabilities
to
define
or
ask
for
more
resources
for
given
tasks,
and
maybe
one
task
needs
gpus.
The
other
tasks
don't
need
that,
so
you
don't
really
need
to
be
paying
the
cost
of
a
gpu
machine.
A
A
All
you
had
to
do
is
say
well,
the
task
config
is
spark
and
flight
will
take
care
of,
or
the
spark
plug-in
in
flight
will
take
care
of
initializing
or
setting
up
a
spark
cluster
for
you,
controllerized
spark
and
initialize
the
spark
context
and
the
connectivity
and
all
of
that,
and
then
your
function
will
start
running
and
you
will
have
that
spark
context.
If
you,
you
know,
need
it
in
this
case,
I
it
will
automatically
convert.
We
also
have
type
transformers
that
would
automatically
convert
from.
A
You
know
append
this
to
the
frame
to
a
pi
spar
data
frame.
You
don't
have
to
do
anything.
You
just
take
this
and
start
running
your
code.
Your
spark
code
right
here
and
you
can
see,
I
still
use
it
exactly
the
same
way
right.
I
still
am
calling
totally
spend
in
the
workflow
exactly
the
same
way.
Nothing
changes
here,
but
instead
now
it's
a
spark
task
and
you
can
do
all
the
mapreduce
magic
that
spark
lets
you
do
and
finally,
at
the
bottom
we
run.
We
have
this
concept
called
launch
plans.
A
Most
plans
are
that
can
overlay
on
top
of
a
workflow
where
you
can
define
certain
behavior
for
certain
inputs.
You
can
also
define
schedules.
You
can
say
you
will
run
this
every
hour
on
the
hour
or
you
know
you
can
have
chrome
schedules
or
fixed
rates.
You
can
have
notifications
when
things
fail,
so
you
can
integrate
this
into
your
existing.
Maybe
incident
management
systems
and
maybe
slack
notifications,
email
applications.
All
of
that
it
will
integrate
well
into
your
existing
infrastructure.
A
And
finally,
flight
also
integrates
well
into
your
existing
githubs
tools.
Right
if
you
have
a
ci
system-
and
you
want
to
maybe
at
the
end
after
merging
a
new
workflow
into
you-
know,
github
repo-
you
want
to
kick
off
the
run
of
that
workflow.
You
can.
You
know,
use
this
flight
ctl
to
cube
cattle
inspired
cli
that
can
do
pretty
much
everything
like
the
api
layer
in
flight
exposes,
as
you
can
create
executions
register.
You
know,
all
of
that.
You
can
wait
for
executions
more
like
consume
or
download
the
outputs
of
executions.
A
A
That
shows
you,
you
know
the
past
few
runs
how
long
they
took
and
and
whether
they
succeeded
or
failed
there's
a
list
of
versions.
So
every
every
change
you
make
to
a
task
or
overflow
is
tracked
as
a
separate
version.
So
everything
is
immutable
and
that's
that
enables
us
to
always
be
able
to
go
back
to
the
exact
version
of
the
code
that
produced
you
know
certain
artifacts
and,
and
that
obviously
applies
to
all
of
your
new
dependency
tasks
and
everything
else
you
call
on
the
right
here
is
a
launch
form.
A
A
If
it's
a
string,
you
get
this
there's
one
for
arrays
and
so
on,
and
this
is
also
pluggable.
You
can
enrich
this.
If
you
have,
you
know
new
input
types
once
you
run
or
you
start
an
execution,
you
get
a
big
graph
of
you
know
what's
happening
and
you
can
see
how
much
is
being
what
the
progress
is.
What
tasks
succeeded?
What
failed
you
can
see.
There
are
like
boxes
in
boxes
here,
because
this
is
one
of
the
more
complex
graphs
we
have
that.
That
uses
composition
right.
A
So
we
have
a
workflow,
that's
calling
other
workflows,
it's
calling
other
workflows
and
with
like
branching
and
if
conditions
and
so
on,
and
the
ui
is
able
to
like
render
the
nesting
level
multiple
nesting
levels
here.
We're
experimenting
with
a
few
different
ways
of
rendering
the
nested
graphs
that,
because
this
looks
a
bit
more
complex
than
you
can,
you
know
easily
understand.
But
but
you
know,
we'll
feedback
is
always
welcome.
A
So
let
us
know
when
things
go
bad
and
as
they
you
know,
will
at
some
point
or
another
flight
also
tries
to
pull
in
the
logs
stress,
looked
traces,
stack,
traces
or
as
much
information
as
we
can
up
bubble
that
up
in
the
ui.
You
can
get
exactly
the
same
information,
obviously
from
the
cli,
or
you
know
your
the
the
library
I
showed
you,
the
remote
library,
so
we
try
to
make
this
as
actionable
as
possible.
You
know
if
it's
an
image,
pool
failure
from
kubernetes.
B
Really
quickly,
I
thought
this
slide
was
fun.
Why
did
we
pick
flight?
This
is
actually
written
by
our
data
science
team.
So
this
is
not
from
us,
but
we
think
it's
really
cool,
because
as
engineers,
a
lot
of
this
might
be
kind
of
table
sticks
for
us,
but
for
the
data
scientists
being
able
to
get
it
up
and
running
on
flight
and
getting
all
of
this
stuff
for
free
has
been
a
really
big
win
for
them
and
they've
been
really
excited
about
it.
B
You
know
the
ability
to
share
models
with
each
other
and
compose
things
easily
out
of
the
box.
Parallelism
I
mean
again
may
not
seem
like
the
biggest
deal,
but
for
them
when
you're
writing,
python
scripts
and
everything
runs
and
takes
a
certain
amount
of
time.
Whereas
now
for
free,
we
get
parallelism
across
tasks.
I
think
they
think
that's
really
cool
the
caching.
A
Yep,
so
we
talked
about
composition,
yeah,
you
can
call,
you
know
workflows
from
one
another
you
can
share.
We
haven't
really
talked
much
about
this,
but
everything
in
the
system
is
tracked
and
addressable
globally
addressable
within
a
deployment,
and
you
can
other
people
can
start
using
your
tasks
and
workflows.
A
One
of
the
teams
we
worked
with
they
their
their
infrastructure,
team
or
platform
team
built.
You
know
one
flight
project
with
just
common
tasks
that
everybody
else
uses
things
like
you
know,
query
table
blah
with
you
know
dates
or
you
know
things
like
that
and
partisan
as
dylan
talked
about
it.
Caching,
we
talked
about
in
storage
and
yeah
compiled
client
validation
that
we
are
also
very
big.
We
feel
strongly
about
that
that
most
of
the
errors
you
get
should
really
surface
as
early
as
possible.
A
A
A
A
A
Yes,
please
say
it
again:
streaming
data.
Do
we
handle
streaming
data,
not
natively,
but
we
one
of
the
one
of
our
customers
wrote
a
flink
plug-in
for
flight
so
and
it
lets
you
do
that,
but
it
doesn't
so
so
that
I
guess
the
question
really
is:
can
can
you
execute
these
workflows
in
a
stream,
or
can
it
consume
a
stream
and
just
run
the
workflows
right?
The
execution
that
requires
workflows
to
run
in
you
know
seconds
right
very
quickly.
A
In
overhead
kind
of
situation
and
most
of
the
workloads
we
saw
run
in
containers
and
there's
always
the
container,
startup,
runtime
and
so
on.
So
at
the
moment
the
answer
is
no,
but
but
it
is
something
we
are
thinking
about,
how
to
how?
How
can
we
integrate
well
with
streaming
solutions,
and
how
do
we
offer
more
importantly,
how
do
you
offer
you
sort
of
a
unified,
dev
experience?
A
A
A
A
That's
another
feature
I
did
not
talk
about.
You
can
declare
retries
on
individual
task
level
and
flight
makes
sure
you
start
from
a
sort
of
a
sandbox
environment.
We
have
the
data
separated
right.
So,
like
the
next
three,
try
will
not
sort
of
read
like
corrupted
data
because
of
the
previous
retry.
You
know
things
like
that,
so
we
we
do
a
lot
to
make
sure
these
start
up
individually,
but
but
yeah
they
can
be
arbitrary,
large
number
of
containers.