►
From YouTube: Running Massively Parallel Deep-learning Inference Pipelines on Kub... Suneeta Mall & Martin Abeleda
Description
Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Running Massively Parallel Deep-learning Inference Pipelines on Kubernetes - Suneeta Mall & Martin Abeleda, Nearmap
Nearmap captures terabytes of aerial imagery daily. With the introduction of artificial intelligence (AI) capabilities, Nearmap has leveraged Kubernetes to generate AI content based on tens of petabytes of images effectively and efficiently.
This talk covers how using Kubernetes as the backbone of our AI infrastructure, allowed us to build a fully automated deep-learning inferential pipeline that despite not being embarrassingly parallel is actually massively parallel. This talk explains the architecture of this auto-scalable solution that has exhausted all K80 spot GPUs across all US data centres of AWS for weeks. This system has already produced semantic content on over a million km2 area at resolution as high as 5cm/pixel in just 2 weeks. In this talk, you will learn about the joys of building and running this system at scale, challenges encountered, their resolution, & future work.
A
Near
map
is
the
leading
provider
of
high-definition
aerial
imagery.
We
specialize
in
providing
current
and
crisp
content,
and
we
do
so
in
various
formats,
such
as
2d,
3d,
intelligently,
derived
layer
and
more.
We
frequently
capture
urban
areas
of
Australia
and
New,
Zealand,
United,
States
and
Canada.
To
do
so,
we
fly
our
plane
at
the
height
up
to
18,000
feet
and
that
allows
us
to
capture
thousands
of
kilometers
Square
area
in
a
single
flight
at
a
very
high
resolution,
such
as
five
centimeter
per
pixel.
In
just
year,
2019,
we
captured
2.5
million
kilometer
Square
area.
A
If
we
were
to
print
out
our
image
and
stack
them
up
in
just
one
year,
we
would
have
created
for
Mount
Everest,
equivalent
content,
and
it's
got
to
be
this
way
because
we
are
data
as
a
service
organization,
and
we
truly
are
a
big
data
company
today
I'm
going
to
be
talking
about
what
we
do
with
AI
at
near
map.
What
are
the
expectation
when
we
have
when
we
talk
about
offline
inferences
and
then
we
talk
a
bit
more
about
our
solution
that
we
call
Sayer
martin
will
pitch
in
and
talk
about.
A
Some
of
the
adventure
we
have
had
been
running
this
solution
at
scale
and
he
would
also
talk
about
some
of
the
annoyances
that
we
have
encountered
with
respect
to
it.
Cloud
infrastructures
and
I'll
come
back
into
talk
about
some
of
the
misadventures
and
then
we'll
open
the
floor
for
questions
at
new
map.
The
REI
team
has
been
focused
on
associating
semantics
to
the
pixel
and
we
do
so
to
find
information
about
location.
A
What
you're
seeing
here
is
Sydney's
most
livable
place.
This
is
Milsons
point
and
lavender
Bay
from
the
eyes
of
our
machine.
This
is
Roof
swimming
pool,
water
body
and
building
outline
overlaid
on
top
of
it.
Of
course,
we
do
a
lot
more
than
these
handful
of
attributes
I'm.
Talking
about
what
we
do
is
we
generate
better
bytes
of
derived
content
from
terabytes
of
aerial
imagery,
but
we
already
got
petabytes
of
imagery.
So
you
see,
we
are
exponentially.
Growing
are
already
big
data
set.
The
solution
we
are
talking
about
today.
A
We
call
seer
can
produce
semantic
content
at
a
rate
of
70,000
kilometers
quite
a
day.
We
could
probably
do
a
lot
more,
but
we
are
limited
by
the
availability
of
spot
GPUs
in
the
cloud
now.
The
data
that
we
generate
are
we
have
already
generated,
is
already
being
put
to
good
use
in
the
industry,
it's
available
for
consumption
under
AI
offline
beta
program,
and
it's
already
delivering
solution
in
the
space
of
5g
insurance,
energy,
utility,
urban
planning
and
lot
mode.
A
But
we
don't
just
stop
it
associating
semantics.
We
want
to
provide
insights
from
micro
level
such
as
pixels
all
the
way
up
to
macro
level
such
as
world,
and
we
are
already
doing
that.
Some
of
the
examples
are
all
on
the
screen
now
to
do
all
that
we
observe
a
workflow
end-to-end,
AI
workflow,
somewhat
somewhat
similar
to
what's
shown
on
the
screen.
Now
we
own
everything,
starting
from
the
data
collection
and
to
model
operations
such
as
training
tuning
improvising
and
also
to
inference
and
at
the
end,
to
insights
as
a
service.
A
Now
today,
we
are
only
going
to
be
talking
about
offline
inference,
which
is
the
section
of
this
complex
workflow
shown
on
the
screen
in
red,
highlight
when
we
talk
about
offline
inference.
The
modus
operandi
really
is
for
that
operation
is
what
we
call
dag,
which
is
a
series
of
operation
applied
on
an
input
data
set.
Our
expectation
with
the
dag
is
that
each
step
in
the
in
the
dark
process
needs
to
massively
parallel
eyes.
A
We
need
to
be
efficient
in
terms
of
course,
turnaround
time
and
utilization
of
the
resource
that's
available
to
us,
and
we
need
to
be
resilient
with
our
workload,
because
that
helps
with
chaos,
and
it
also
helps
a
lot
with
the
cost
and
we
don't
need
any
manual
bottleneck
in
our
system
because
we
can't
be
running
at
that
scale
if
we
have
a
manual
bottleneck.
So
that's
really
a
key
component
of
our
system.
A
At
the
time
of
building
the
solution,
we
looked
at
off-the-shelf
available
solution.
Now
these
are
great
tools,
but
they
they
weren't
quite
production,
ready
and
weren't
quite
doing
what
we
wanted
of
our
go:
pachyderm
and
cube
flow
path,
pipelines
which
is
a
wrapper
over
our
go.
They
are
great
tools
and
we
already
are
using
them
in
some
capacity,
but
particularly
for
offline
inference.
They
weren't
quite
doing
what
we
wanted.
Most
couldn't
cope
up
with
the
load
and
in
some
cases
there
were
over
or
under
provisioning
or
execution
model
was
too
expensive.
So
what
did
we?
A
Do?
We
build
our
own
solution?
That's
sale.
We
are
talking
about
how
we
did
that.
Is
we
exploited
kubernetes
extension
in
point,
that's
known
as
custom
restores
the
source
definition.
Excuse
me:
if
you
aren't
familiar
CR
DS,
then
you
might
have
heard
term
operator
which
is
which
works
on
a
similar
paradigm
it
just.
There
is
only
slight
pragmatic
difference
between
the
two
so
now
I
show
you
the
architecture
of
Seir,
which
is
which
looks
somewhat
like
this,
and
you
might
think
there's
a
lot
going
on.
Let
me
take
you
through
in
a
step-by-step
fashion,.
A
A
A
Now
this
is
a
synonym
of
what
our
workflow
would
look
like
and,
as
you
can
see,
some
steps
are
independent
and
singleton,
but
other
steps
run
in
at
high
parallelism
in
Gang
shedule
semantics,
as
I
said
before
now
they
consume
their
own
respective
datums
and
perform
the
operation
at
the
of
the
operation.
We
collect
the
data
feed
into
the
next
step.
We
keep
doing
that
until
we
have
reached
to
the
final
stage.
A
That's
when
we
clean
up
after
ourselves
and
remove
all
the
resources
now
this
solution,
with
an
extension
to
cube
annuities,
creates
ephemeral,
cubes
for
us
now.
We
we
don't
run
them
forever.
We
run
them
as
on
need
basis.
These
are
self
managed
multi,
as
a
cube
in
it
is
cluster
provision
with
chaox.
They
we
primarily
run
on
AWS,
but
we
often
see
the
cases
when
spot
GPUs
run
out
on
AWS
in
their
data
centers,
because
we
are
running
at
such
a
capacity
in
endow.
In
those
cases
we
go
out
and
run
on
GC
P.
A
So
that
means
this
solution
is
cloud
agnostic.
This,
like
I,
said
this.
This
component,
this
solution
is
multi
a-z.
We
run
with
three
master
up
to
three
on-demand
nodes,
and
then
we
run
a
fleet
of
four
GPUs,
hundreds
of
CPU
compute
and
thousands
of
GPU
nodes,
and
we
we
don't
just
have
one
communities
like
that.
We
would
have
multiple
communities,
depending
on
the
capacity
of
the
system
not
to
deal
with
the
scale
and
the
volume
of
data
that
we
are
talking
about.
A
We
are
running
to
many
nodes
and
to
many
parts
we
are
talking
about
of
the
order
of
tens
of
thousands
of
odd
and
thousands
of
node.
We
need
to
be
careful
with
the
placement
of
our
resources
and
utilizing
the
resources
that's
available
to
us
in
best
of
its
capacity
to
do
that.
We
we
utilize
placement
predicates
quite
heavily,
and
some
of
these
are
selector
predicates
until
affinities
and
into
affinity
Stainton
tolerances,
in
terms
of
quality
of
services.
A
Unfortunately,
due
to
nature
of
our
Lord,
not
many
are
guaranteed,
but
we
do
our
best
to
ensure
that
they
are
bustable
wherever
not
possible.
We
fall
back
to
best
effort,
but
together,
all
these
components
with
the
help
of
autoscaler
allows
for
the
scale
that
we
are
running
at
and
our
helps
us
realize
our
goal.
Now
Martin
will
be
taking
over
to
talk
about
some
of
the
adventure
we
have
seen
in
running
this
solution
at
scale.
Thank.
B
You
Sunita
also
no
project
is
complete
without
its
fair
share
of
adventures
and
running
thousands
of
nodes,
and
tens
of
thousands
of
jobs
in
the
cluster
has
been
quite
a
challenge
with
this
kind
of
load.
We
can't
guarantee
that
there's
enough
resource
in
the
spot
market
to
run
everything
in
parallel,
and
so
we've
got
contention
between
resources
with
lots
of
pending
pods,
except
not
all
of
these
pods
are
created
equal.
We
really
need
our
controller
and
our
master
pods
to
have
priority
when
we
allocate
them.
B
B
The
second
component
is
a
scheduler
which
monitors
the
state
of
jobs
in
the
in
the
cluster
and
looks
at
the
the
capacity
of
the
cluster
and
decides
whether
or
not
we're
able
to
shade
your
new
jobs.
And
this
allows
us
to
trigger
and
cue
our
workflows
for
processing.
And
we
can
ensure
that
everything
runs
in
the
right
order.
That
we
don't
overload
the
kubernetes
api
and
that
we're
feeding
our
resources
efficiently
as
possible.
B
So
at
this
point,
we're
running
multiple
workflows
simultaneously,
each
of
them
with
thousands
of
jobs
scheduled
at
a
time
and
we're
running
a
stateful
workload
here.
So
we
require
across
a
Z
persistent
store
and
we're
accessing
files
repeatedly
so
with
thousands
of
nodes,
each
doing
a
mix
of
read
and
write
operations
we're
starting
to
see
ourselves
hitting
the
limits
of
the
shared
file
stores.
B
So
our
solution
here
is
to
isolate
the
persistent
volume
for
each
of
these
workflows
and
again
we
provision
across
a
Z
and
FS,
but
share
this
only
between
the
jobs
in
that
workflow,
and
this
provides
a
good
isolation
between
each
of
those
and
then
we
can
throw
this
away
once
our
processing
is
finished.
So
it
allows
us
really
to
keep
our
processing
times
in
check
by
keeping
our
GPUs
fed
and
running
high
concurrency.
B
So
introducing
these
Peavey's
has
also
added
some
stability
issues
and
we
see
particular
nodes
I'm
failing
to
map
the
volume,
and
this
manifests
itself
in
pods,
which
is
stuck
in
a
container,
creating
state.
And
so
at
the
moment,
it's
a
still
an
open
issue
that
we're
seeing
in
the
but
for
now
to
hit
full
animation.
B
We've
gone
with
a
custom
solution
and
what
we've
done
is
we've
deployed
an
agent
onto
the
cluster,
and
this
resolves
this
issue
when
we
see
it
by
draining
the
the
unhealthy
nodes
and
again
remembering
that
we're
running
a
self-managed
kubernetes
cluster
with
burstable
or
best-effort
jobs,
we
see
occasionally
that
we
still
node
resources
from
system
components
like
the
cubelet
and
again.
This
is
an
issue
that
many
people
in
the
community
have
seen.
B
But
for
this
one
there's
a
well
documented
best
practice
solution,
and
so
what
we've
done
is
we've
manually
reserved
the
resources
for
the
cubelet
in
the
cluster
spec?
And
although
it's
not
necessarily
difficult
solution,
we
think
that
it
should
really
be
pre-configured,
since
kubernetes
should
be
is
intended
to
run
at
the
scale
out
of
the
box.
B
So
now
we're
at
a
point
where
each
of
our
pods
can
run
without
harming
system
components
or
running
out
of
I/o
or
hitting
I/o
limits,
and
given
that
we're
scheduling
in
the
order
of
tens
of
thousands
of
these
at
a
time,
we
need
thousands
of
nodes
to
process
these
in
parallel
and
so
with
the
D.
With
a
default
container
network
interface,
we're
limited
to
a
route
table
limit
of
50
entries
and
we
see
degradation
if
we
cross
this,
and
we
could
either
raise
this
limit
with
AWS
at
the
expense
of
a
slight
performance
degradation.
B
B
At
this
point,
we're
running
a
healthy
cluster
with
a
configuration
that
allows
us
to
orchestrate
tens
of
thousands
of
pods,
but
it's
not
completely
pain-free
and
we
start
to
see
some
issues
with
the
cluster
autoscaler
when
going
beyond
650
nodes.
In
this
case,
the
cluster
continues
to
be
to
look
healthy,
but
the
autoscaler
can't
reach
our
masters,
and
we
suspect
that
this
is
due
to
a
time
at
issue.
Again.
B
This
is
an
open
issue
which
we
don't
have
a
resolution
to
at
the
moment,
but
we
have
a
bandit
like
solution
here,
good
or
bad,
and
when
we
observe
autoscaler
failures,
we
fall
back
to
our
pre
cooks,
it
script,
which
acts
as
our
autoscaler
and
as
the
secondary
measure.
This
does
the
job
so
now
we're
running
at
the
scale
we
want,
but
we
need
to
finish
as
fast
as
possible
and
you
know
workflow.
B
The
GPU
step
is
by
far
the
most
expensive
operation,
both
in
coil
in
terms
of
cost,
but
also
in
compute
time,
and
we
typically
see
processing
times
of
up
to
three
hours
for
each
of
these
datums.
So
really
two
processes
and
in
any
acceptable
time
frame.
We
need
to
parallelize
this
as
much
as
possible,
using
as
many
GPUs
as
possible,
but
who
would
have
thought
that
there
wouldn't
be
enough
spot
GPU
in
the
cloud
so
going.
B
Multi
AZ
wasn't
enough
and
we've
embraced
a
multi
region
approach
and
what
we've
done
is:
we've
provisioned
a
cluster
in
every
AWS
region
that
has
enough
spot
GPU
availability.
To
achieve
this,
we've
adopted
a
git,
ops
workflow,
and
we
orchestrate
the
clusters
with
cops
and
terraform,
and
every
code
change
triggers
a
CI
workflow
which
propagates
throughout
through
to
our
clusters,
and
this
gives
us
all
the
benefits
of
version
control,
but
also
folds
into
our
existing
code
review
practices.
B
So
we've
gone
out
to
multiple
regions
and
we
have
a
mushroom
effect
with
our
clusters.
Queuing
and
scheduling
mechanisms
have
been
decentralized,
and
this
requires
a
bit
more
work
to
operate,
considering
that
we
want
to
really
be
in
a
fully
automated
manner.
So
we
want
to
centralize
our
queueing
in
chechenya
mechanisms
and
we've
done
this
with
Sto.
So
each
of
our
clusters
has
its
own
service
mesh,
which
together
form
one
logical,
hybrid
mesh
and
with
multi
cluster
sto.
B
So
now
we're
in
a
place
where
we're
running
thousands
of
nodes
generating
terabytes
of
data
per
job,
and
the
next
challenge
is
pulling
this
data
into
our
object
stores
so
that
we
can
serve
it.
It's
not
quite
as
simple
as
running
a
single
s3
sync,
as
others
have
found,
and
we
observe
missing
files,
especially
when
we're
working
with
terabytes
of
data.
B
We
also
tend
to
see
interesting
things
like
this.
We
get
corruptions
when
GPUs
crash
and,
as
you
can
tell
these
on
rubes,
even
though
that's
what
they're
supposed
to
be
so
we've
added
a
final,
very
vacation
step
was
a
which
is
a
comprehensive
verification
and
is
our
final
guarantee
that
everything
is
there
and
valid.
B
We've
also
managed
to
hit
quite
a
few
bottlenecks
within
the
infrastructure,
because
it
turns
out
that
cloud
providers
aren't
used
to
seeing
people
hitting
this
scale.
The
first
one
of
these
is
hitting
node
provisioning
limits,
because
we
want
to
be
able
to
scale
up
to
thousands
of
nodes
in
just
a
matter
of
minutes,
but
by
default
AWS
provisions
nodes
in
blocks
of
10,
which
takes
about
40
minutes
to
hit
that
on
the
scale,
and
it
turns
out.
That's
not
all
we're
hitting
infrastructure
limits
everywhere.
B
A
You
Martin
no
adventure
is
ever
possible
without
a
few
handful
of
misadventures
and
that's
what
makes
our
experience
interesting.
So,
let's
look
at
some
of
these
sharing
GPU.
What
do
we
do
when
our
GPU
workload
isn't
quite
GPU
efficient?
We
share
the
GPU
right,
so
we
thought
and
when
we
ventured
out
to
this
exercise,
first
attempt
at
this.
A
We
exposed
all
the
GPUs
of
a
node
to
all
the
pods
deployed
on
the
node
and
only
share
a
fraction
of
the
memory
through
tensorflow
configuration
turns
out
that
even
tensorflow
doesn't
guarantee
the
amount
of
flexion
that
you
are
intending
to
allocate
with
the
traction
configuration
and
that's
not
it.
There
are
various
Furious
error
that
you
would
see
with
your
stack
crashing
and
GPU
compute
crashing.
A
Some
of
them
are
so
nasty
that
they
would
fail
with
without
they
would
not
outright
fail,
they
would
fail
silently
and
you
would
have
absolutely
no
idea
of
crash
and
what
that
leads
to
is
perfectly
readable
output
but
complete
garbage
one
such
example
is
shown
on
the
screen.
Now,
when
we
run
with
this
particular
attempt,
we
generate
thousands
of
these.
A
These
are
of
no
use
to
us
and,
of
course,
that's
not
quite
right
what
we
are
trying
to
do,
because
there
is
no
isolation
guarantee
on
GPU
the
the
GPU
providers,
such
as
Nvidia
they
for
security
and
isolation
purposes.
They
don't
recommend
this
approach,
but
fun
item
10
one
exercise
wasn't
enough:
we
still
hell-bent
on
it.
We
tried
one
more
exercise.
We
we
looked
at
exposing
GPU
memory
as
an
extended
resource,
similar
to
what
Alibaba
cloud
exposes
in
another
semantics.
A
It
works
for
a
lot
of
people,
but
it
didn't
quite
work
for
us
because
again,
like
previous
discussion,
we
would
have
spurious
of
our
mirrors
and
and
complete
rubbish
files
getting
generated
all
over
the
place
turns
out.
All
we
needed
to
do
was
optimize
our
code,
so
it's
efficient
on
GPU
and
it
wasn't
as
much
effort
as
it
did
for
us
to
run
through
one
and
two
now
GPUs
are
expensive
right
and,
as
we
said,
we
run
thousands
of
them.
A
So
when
we
run
with
spot,
we
can
get
them
at
about
40
percent
price,
so
we
thought
and
then
we
then
we
bid
for
them.
They
bid
for
Macs
on
demand
price
and
then
we
get,
for
whatever
is
the
relevant
price.
That
sounds
perfectly
reasonable
right.
Just
when
you
run
start
running
thousands
of
those
and
drain
the
cloud
or
the
region
with
spa
GPUs,
then
the
market
starts
to
go
up.
A
You
know
how
spot
markets
work
right,
though,
as
the
demand
goes
up,
the
price
also
start
going
up,
but
when
your
system
is
automated
and
you
are
bidding
for
the
max
price
and
happily
sipping
your
drinks,
then
things
like
these
happen-
the
price
spike
from
6%
all
the
way
two
hundred
and
twenty
percent-
and
yes,
we
ended
up
realizing
that
when
we
got
our
bill
and
yes
I
still
have
my
job.
Thank
you
for
your
concern.
A
So
the
solution
for
this
was
quite
simple.
All
we
had
to
do
was
bid
for
the
price
that
we
are
comfortable
with
now.
Monitoring,
GPU
monitoring,
GPU
isn't
as
simple
as
monitoring,
CPU
and
memory
when
you
lend
out
VMs
from
cloud
infrastructure
because
that's
something
that
they
own
and
it's
easy
to
isolate
from
the
virtualization
point
of
view,
not
quite
so
far
GPU,
because
you
still
need
to
install
the
drivers
and
toolkits
to
to
be
able
to
prove
the
GPUs.
A
But
we
needed
to
ex
to
explore
that
because
we
wanted
to
build
the
silk
bill,
improve
our
code.
So
it's
hundred
percent
efficient
from
GPU
view
point.
So
what
we
did,
what
a
common
solution
would
be,
we've
deployed
at
them
and
said
that
that
frequently
poles
into
pulls
into
the
GPU
to
find
the
utilization
but
turns
out
we're
now
running
so
many
demons
head
that
we
can't
that
scheduler
can't
cope
up
with.
We
already
have
thousands
of
POD.
Now
we
have
flannel
for
networking
component.
A
We
have
monitoring
them
in
sets,
we
have
device
plug-in
and
we
also
have
GP
utilization
and
system
is
starting
to
freak
out.
We
could
probably
do
better
with
these
demonstrates.
So
what
we
did
was
we
co-located
these
with
the
device
plug-in
and
that
seemed
to
work
for
us,
but
of
course,
now
NVIDIA
has
the
GP
operator,
so
that
takes
away
the
pain
of
deploying
the
Devon
sets.
So
thank
you
and
media
there.
Now.
A
We
would
like
to
thank
you
all
for
coming
to
this
talk
and
we
hope
that
our
experience
was
interesting
to
you
and
you
find
you
have
learned
something
from
our
experience
now.
I
would
like
to
take
questions,
but
before
I
open
the
floor
for
that
I
would
like
to
let
you
know
that
we
are
hiring
for
AI
team
and
we
have
two
open
positions.
One
is
data
engineer
and
other
one
is
machine,
learning,
engineer
Martin
and
myself
would
be
hanging
around
for
rest
of
the
day,
so
please
come
and
find
us.
A
C
A
A
You
know
how
you
expose
memory
or
CPU,
and
you
sort
of
say
that
I
only
want
one
CPU
for
this
part
and
only
give
me
that
in
the
similar
semantics
you
would
say
that
I
only
want
say
400
and
they
are
all
or
one
third
of
a
memory
from
the
GPU
and
semantics
like
this,
and
you
would
expose
like
that
under
a
certain
key
and
you
deploy
your
parts.
But
fundamentally
the
GPU,
like
Nvidia
itself,
doesn't
guarantee
isolation
on
your
GPU.
A
So
you
can
take
one
GPU
and
slice
the
memory
and
and
and
use
your
workload,
but
it's
not
guaranteed
that
you
will
not
step
over
and
consume
any
more
memory
if
your
workload
start
to
spike
up
and
use
more
memory
and
similar
to
the
attempt
that
we
had
one
the
same
problem
exists
in
attempt
to.
There
is
no
isolation
guarantee.
A
So,
even
if
you
are
asking
for
half
of
memory,
you
are
not
quite
guaranteed
to
get
it
and
that's
the
problem
that
we
had
running
in
the
temptu
because,
like
I
said
before,
my
solution
isn't
quite
guaranteed
from
from
the
GPU
providers
itself.
There
are
security
and
isolation
concerns
to
start
with,
and
if
you
read
up
their
design
document,
alibaba
clouds.
They
are
very
clear
in
saying
that
it's
not
guaranteed.
It
works
for
some
cases
like
if
you
are
running
very
small
load
with
GPU,
and
you
kind
of
guarantee
the
same
throughput
with
your
workload.
A
D
A
Great
questions
so
I
have
to
start
with
the
notion
that
by
the
time
we,
like
you
know
when
we
started
thinking
about
KF
serving
and
Q
flow
serving
was
still
evolving.
We
already
had
this
solution
running
and
producing
at
scale.
We
are
talking
about
running
over
a
million
kilometres
choir
back
in
February
March
timeframe.
So
that
time
we
already
had
solution
running
at
scale
and
doing
the
things
that
we
are
talking
about
and
at
that
time
these
tools
were
not
evolve,
not
evolved.
Yet,
like
you
know,
Q
flow
is,
is
not
quite
production
ready.
A
Yet
we
are
talking
about
January
for
that
to
be
production,
ready
and
like
you
looking
at
the
scale
of
things
we
are
doing,
we
are
doing
you,
you
would
want
it
to
be
production,
ready
and
really
have
past.
The
stretch
stress
test
that
wasn't
quite
done
for
us,
and
these
are
great
tool
like
I
I
am
all
for
Q
flow
k
of
serving
sounds
really
great
great
and
the
the
fundamental
idea
of
Q
flow,
taking
away
the
like
bringing
in
the
abstraction.