►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
eBay Search On K8s - Mohnish Kodnani & Yashwanth Vempati, eBay
eBay currently has billions of items available for search. The search engine at any given time can get around 100K’s of queries per second for search within this inventory. In order to support this scale of traffic and the size of the inventory we need thousands of servers. The inventory is sharded and then replicated across these servers to handle the traffic. In this talk we will go through how we migrated the application to kubernetes and its deployment architecture while meeting some of the business requirements for resiliency and availability. We will also go through our index distribution architecture that leverages kubernetes principles. At the end we will also share our challenges and learnings while deploying the application on kubernetes.
https://sched.co/UaXv
A
All
right,
while
the
speakers
make
their
way
to
the
front
hi
I'm
Doug,
Davis
chocos,
for
this
session
I
see
we
talked
about
you
based
search
on
kubernetes,
a
couple
of
things.
First,
there
is
a
survey
with
associate
with
each
session,
so
please,
after
the
session,
is
over
please
rate
the
presentation
and,
of
course,
the
presenters,
much
appreciated
and
with
that
now
that
they're
here
or
knurl,
miked
you're
all
set
all
right
right
here.
Not
you
guys.
B
All
right,
thank
you,
hi
everyone.
My
name
is
Manish
I'm,
a
developer
with
eBay
search
team,
and
this
is
my
colleague
Ashwin
and
he's
a
developer
with
eBay's
platform
team,
which
is
responsible
for
all
things
kubernetes
within
eBay
and
will
walk
you
through
a
journey
of
how
we
took
eBay,
store
Hugh's
case
and
ran
it
on
top
of
kubernetes.
B
So
if
there's
nothing
else
that
you
remember
from
this
talk
at
least
remember
that
it's
possible
to
run
a
large-scale
latency-sensitive
application
like
a
search
application
and
run
it
on
the
kubernetes
platform
natively
and
hopefully,
with
that
said,
I'll
sort
of
walk
you
through
what
search
looks
like
in
eBay?
What
searches
scale
is
and
then
my
partner
will
go
into
detail
on
what
our
current
design
looks
like
when
we
model
this
use
case
on
top
of
kubernetes.
B
So
let's
look
at
searches,
background
eBay
has
its
own
custom
search
engine
built
in-house,
it's
called
Cassini,
it's
a
distributed,
search
engine.
It
says
various
use
cases
within
eBay.
One
of
the
most
prominent
use
cases
is
what
we
call
as
active
listings
and
to
give
you
an
idea
at
any
point
in
time.
This
search
engine
is
serving
around
1.4
billion,
active
listening,
so
listening
that
what
we
call
the
things
that
you
can
transact
on
site
so
buy,
sell
them,
etc
and
active
is
what
is
currently
available
as
opposed
to.
B
Hopefully,
these
two
numbers
together
at
least
give
you
a
rough
idea
that
it's
not
possible
to
load
all
of
this
data
on
a
couple
of
machines
and
be
able
to
serve
the
query
volume
that
we
are
talking
about
within
the
stipulated
SLS
for
our
queries,
and
so,
in
fact,
within
eBay
search
use
case
typically
takes
around
30
to
40%
of
our
data
certain
footprints.
So
that's
how
many
servers
thousands
of
servers
we
are
talking
about.
B
We
have
five
nines
of
availability
of
SLS
that
we
have
to
maintain.
Every
second
search
is
down
is
actually
costing
eBay
money
because
you
can't
use
our
sight
anymore,
and
so
lot
of
our
design
is
governed.
Keeping
this
in
mind
that
we
have
to
be
available
all
the
time
with
that
said,
let's
dive
a
little
bit
deeper
into
how
a
based
search
architecture
looks
like
so
broadly,
you
can
think
of
eBay
search
architecture
divided
into
five
layers.
B
So
when
a
user
comes
into
ebay.com
and
types
in
a
query,
USD
sweatshirt,
for
example,
it
goes
to
our
first
layer,
which
is
the
software
load
balance
earlier.
These
are
custom,
inbuilt
software
load,
balancers
and,
as
the
name
suggests,
their
job
is
very
simple.
They
find
the
least
loaded
node
in
the
downstream
layer
and
forward
the
query
to
that
particular
node,
which
brings
the
query
to
the
second
layer,
which
is
the
query
transformation
layer.
So
the
co,
a
transformation
layers
main
responsibility,
is
to
take
the
users
query
and
enhance
it
and
annotate
it.
B
What
I
mean
by
this
eBay
analyzes
a
lot
of
user
data
click
data?
That's
going
on
and
be
mind.
What
is
the
demand
and
supply
for
various
things?
So,
for
example,
we
have
might
have
noticed
that
people
who
type
in
USD
sweatshirt
are
also
interested
and
click
on
hoodies
and
tend
to
buy
OD.
So
we
will
add
that
keyword
to
the
original
query
and
add
hoodies
for
something
we
can
also
do
spell
correction
at
this
layer,
add
brands
etc
to
this
coil
and
transform
it.
And
then
we
send
this
query,
transform
query.
B
The
next
layer
is
what
we
called
us:
the
top-level
aggregators,
the
top-level
aggregators
responsibility,
is
to
take
this
user
query
and
fan
it
out
to
multiple
use
cases.
If
the
intent
is
such
so,
it
can
send
the
same
query
to
our
active
listings
and
also
to
our
sold
listings
and
then
on
the
way
back.
It
merges
the
results
for
the
upstream
layers
to
look
to
get
the
whole
relevance
and
recall
for
that
particular
query.
The
top-level
aggregators
once
they
fan
out,
send
it
to
low-level
aggregators
or
ll
s.
B
They
are
essentially
the
same
as
the
top-level
aggregators,
but
they
work
within
the
use
case.
So
they
don't
go
out
to
multiple
use
cases.
They
work
within
the
use
case
and
fan
the
query
out
for
the
next
layer,
which
brings
me
to
the
most
important
layer,
which
is
the
fifth
layer
which
we
call
as
the
quarry
serving
stat.
B
This
is
the
layer
which
is
essentially
responsible
for
taking
the
query
running
that
query
across
the
billion
items
that
we
have
figuring
out
the
relevance
of
the
intended
relevance
of
the
incoming
query
to
the
item
and
then
score
them
rank
them
and
send
it
back
upstream
to
the
user.
So
this
essentially
is
the
most
complex
layer
for
us,
and
our
idea
was
that
if
we
tackle
this
layer
correctly,
other
layers
will
take
care
of
itself
or
themselves.
So
this
is
the
layer
that
we'll
be
focusing
this
talk
on.
B
As
we
saw
earlier,
we
have
around
1.4
billion
listings,
so
we
cannot
all
fit
this
in
the
same
machine.
So
we
shard
this
data,
so
you
can
think
of
shards
as
columns
which
hold
a
percentage
of
our
inventory
so
every
or
buckets
rather,
which
hold
a
percentage
of
our
inventory,
and
then
we
have
replicas
within
the
shard
or
what
we
call
rows
for
handling
the
query
volume.
B
Every
second
items
are
getting
bought,
items
are
getting
sold,
prices
are
changing,
and
so
what
we
do
in
our
design
today
is
we
take
a
snapshot
of
the
whole
inventory
and
we
ship
it
to
this
query
serving
grid
grid
multiple
times
a
day.
What
this
means
is,
whenever
the
query
serving
grid
has
to
load
this
new
snapshot,
it
has
to
be
restart
the
grid,
while
serving
traffic,
so
that
some
servers
have
loaded
this
new
snapshot
and
the
rest
of
working
on
working
its
way
through
also
from
the
point
the
snapshot
was
taken.
B
We
continuously
send
all
of
these
servers
data
messages
that
tell
you
the
current
state,
because
not
everything
is
captured
in
the
snapshot.
Things
are
happening,
and
this
is
only
happening.
Few
times
a
day,
so
prices
could
be
changing
every
second,
so
then
we
continuously
keep
sending
it
data
to
keep
the
query
serving
grid
up
to
this
factors
in
this
complexity
of
shipping
data
and
restarting
the
grid,
while
serving
traffic
factors
in
and
is
an
important
part
of
how
we
model
this
use
case
when
we
ran
it
on
top
of
kubernetes.
B
We
looked
at
how
search
works
and
so
I
searched
101
for
eBay,
and
now,
let's
look
at
the
kubernetes
fit
went
within
eBay,
so
the
team
that
handles
all
the
kubernetes
clusters
within
eBay
is
called
tests.
At
I/o
we
have
around
60
plus
production
clusters
of
various
sizes,
some
ranging
from
hundreds
of
nodes
to
some
that
are
in
upwards
of
2000
nodes
in
totality.
We
are
handling
hundred
and
sixty
thousand
pods
application
pods
across
30,000
bare-metal
machines
today,
and
these
clusters
serve
multiple
internal
and
external
use
cases.
B
B
The
these
clusters
are
also
spread
across
the
world
into
around
15
regions.
What
this
gives
applications
is
an
opportunity
to
be
closer
to
the
user
closer
to
the
edge,
so
some
of
our
kubernetes
clusters
are
deployed
on
the
edge
and
we
are
using
on
web
proxies
and
software
load
balancers.
To
give
them
that
capability,
so
which
brings
me
to
the
why?
Why
did
search
decide
to
run
on
kubernetes?
Why
can't
it
stay
for
what?
B
B
So
we
saw
earlier
that
our
kubernetes
clusters
are
deployed
to
the
edge,
which
means
that
if
we
are
able
to
offload
or
if
you
are
able
to
design
our
workload
on
top
of
kubernetes,
it
also
gives
eBay's
search
to
be
closer
to
the
edge
by
just
being
on
run
by
being
able
to
run
on
kubernetes.
So
there
are
lots
of
use
cases
if
you
think
you
are
typing
iPhone,
and
you
see
all
these
completions
that
happen
right.
Those
things
make
sense
to
be
closer
to
the
user,
because
every
millisecond
matters
for
search.
B
Automation
so,
as
I
said,
one
of
the
main
factors
for
search
in
designing
is
availability.
We
have
to
be
available
all
the
time,
which
means
we
lay
out
our
grid,
our
topology,
the
matrix
that
we
talked
about
in
a
very
specific
manner.
For
example,
when
we
create
a
column
or
a
chart,
we
make
sure
that
every
single
replica
is
not
on
the
same
rack
just
because
if
a
rat
goes
down
and
it
takes
the
whole
shard
with
it,
that
part
of
the
inventory
essentially
is
invisible
to
the
user.
B
So
it
took
out
the
whole
that
chunk
of
the
the
bucket
basically
is
out,
and
so
we
cannot
show
the
items
that
were
in
that
bucket.
So
things
like
that,
so
what
we
did
was
we
use
Ford
affinity,
anti
affinity,
rules
that
kubernetes
provides
and
bake
this
into
the
platform,
as
opposed
to
carefully
manually
hand-holding.
How
we
layout
the
topology.
B
B
C
My
niche,
the
first
part
of
this
journey,
was
to
convert
this
search
application
into
micro
services,
so
the
first
important
micro
service,
which
we
call
as
a
query
serving
pot.
This
has
three
major
containers.
The
first
container
is
a
query
serving
container.
This
contains
all
the
application
logic
that
is
required
to
serve
the
end
user
queries.
The
second
container
is
the
log
exporter
container.
This
container
pushes
all
the
application
logs
into
our
logging
system.
The
third
is
a
matrix
exporter.
This
container
converts
all
the
application
matrix
into
Prometheus
format
and
ships
to
our
metric
system.
C
The
second
important
micro
service
is
the
data
distribution
agent,
for
the
primary
responsibility
of
this
part
is
to
download
the
data
on
to
the
node.
This
data
is
used
by
the
query
serving
fault
to
serve
the
user
queries
we
deploy.
This
part
is
a
daemon
set
on
all
our
caged
nodes.
We
also
have
few
metric
collection
paths
that
collect
both
the
container
matrix
and
the
node
level
metrics
for
the
KH
norm.
We
store
this
data
on
our
local
persistent
volumes.
C
This
persistent
volume
is
shared
both
by
the
data
distribution,
agent,
pod
and
the
query
serving
for
now.
The
next
part
of
the
journey
is
to
layout
the
search
grid.
Further
for
the
search
use
case,
when
we
looked
into
the
case
ecosystem,
we
could
not,
we
could
not
find
any
operator
or
a
pattern
that
would
help
us
to
deploy
the
application
in
the
form
of
a
matrix.
The
existing
patterns,
such
as
deployments
run
stateful
cells
are
uni-dimensional,
so
what
we
did
was
we
set
out
to
create
an
operator
to
achieve
this
use
case.
C
We
call
this
as
a
matrix
deployment
operator,
so
when
an
operator
wants
to
deploy
a
search
grid
for
a
specific
use
case,
she
goes
and
creates
a
CRD
for
the
matrix
deployed.
As
you
can
see
in
this
particular
example,
spec,
the
operator
wants
to
create
a
matrix
with
four
columns
and
3
rows.
He
had
columns
referred
to
the
stateful
sets
in
the
search
grid
and
the
rows
referred
to
the
replicas
for
each
column.
C
This
also,
this
bike
also
has
additional
information
like
the
container
versions
which
need
to
be
deployed,
and
also
the
data
packages
that
need
to
go
as
part
of
the
pod.
We
also
track
the
status
of
this
deployment
grid
so
once
the
CRD
instance
is
created,
the
controller
kicks
in
and
starts
creating
the
stateful
sets,
as
Mohnish
mentioned,
the
charts
of
each
column
needs
to
be
distributed
across
the
racks.
C
So
we
use
a
combination
of
pod
and
DF
3d
and
affinity
rules
to
achieve
this
kind
of
layout,
so
the
statehood
sets
are
created
on
and
the
grid
has
formed,
as
we
just
illustrated
since
the
operator
has
asked
for
four
columns
and
3
rows.
We
create
a
search
grid
with
this
configuration
now,
while
these
parts
are
getting
created,
the
service
controller
kicks
in
and
creates
the
headless
services
for
each
of
these
pots.
C
This
we
create
headless
services
because
we
want
the
upper
level
stack
to
be
oblivious
of
the
infrastructure
changes
underneath
them
now,
as
you
can
see,
this
design
helps
us
to
be
very
flexible
with
our
search
grids.
It
provides
us
the
speed,
for
example,
if
we
have
want
to
increase
the
grid
size.
All
the
operator
needs
to
do
is
change
the
spike.
C
It
also
provides
us
the
flexibility
of
moving
our
infrastructure,
for
example,
if
you
know
that
a
specific
use
case
is
going
to
get
more
traffic,
while
the
other
doesn't
get
that
much
traffic,
so
we
can
decrease
the
search
grid
for
the
other
use
case
and
move
the
infrastructure
here.
So
it
actually
provides
the
flexibility
in
terms
of
infrastructure
movement.
C
We
also
monitor
our
search
grids
using
the
metric
server.
Whenever
there
we
look
at
the
CPU
and
the
memory
utilization
and
whenever
it
goes
beyond
a
certain
percentage,
we
use
the
pod
auto
scaling
capabilities
to
increase,
have
a
good
sighs
now
that
we
have
laid
out
the
grid
and
the
next
part
of
the
journey
is
to
distribute
the
data.
C
Since
this
is
a
stateful
application,
it
is
very
important
for
us
to
maintain
the
right
state
on
these
parts,
so
we
set
out
and
wrote
an
other
operator
which
we
called
as
a
file
distribution
operator
to
distribute
the
data
across
the
grid.
So,
whenever
there
is
new
data
for
the
use
case
that
in
a
deployment,
Orchestrator
gets
the
grid
metadata
for
that
particular
use
case.
This
includes
information
like
number
of
rows,
number
of
columns
and
so
on.
The
orchestrator
then
creates
the
spec
on
the
ATA
server,
for
example,
in
this
spec.
C
We
are
mentioning
that
for
shard
1
what
data
needs
to
be
downloaded
and
where
it
needs
to
be
downloaded.
Once
this,
we
also
track
the
status
of
the
data
distribution
using
this
spec
things
like
the
download
speed
and
the
progress
of
the
download
is
tracked
here,
which
is
in
turn
used
by
the
orchestrator
for
the
distribution
of
the
data.
C
Once
this
year
is
created,
the
data
distribution
controller
starts
to
push
data
in
suddenly
to
all
the
columns
in
the
search
grid.
Once
the
data
lands
on
the
top
of
the
column
pod,
it
gets
replicated
to
its
charts,
which
is
robeast.
This
is
done
by
the
data
distribution
agent
pod
now
mohnish
mentioned
that
we
push
the
data
a
few
times
in
a
day,
as
well
as
the
incremental
data
to
the
search
grid.
Both
these
mechanisms
happen
to
look
through
the
same
process
once
the
data
is
downloaded
to
all
the
pods.
C
C
The
solutions
that
Cuban
at
his
office
today
doesn't
help
us
to
solve
the
problem.
The
way
we
Internet
do
in
order
to
solve
this
problem,
we
created
a
mutating
weapon
to
make
sure
the
same
PVC
it
was
distributed
is
shared
by
both
the
pots.
Let
me
walk
through
how
it
how
this
goes
so.
The
data
distribution
agent
Paul,
as
I
mentioned
earlier,
is
running
as
a
daemon
set
on
all
our
cases.
Notes
when
a
query
serving
part
lands
on
that
particular
node.
We
create
a
PVC
by
binding
a
little
pv
to
that
particular
PVC.
C
Once
this
query
serving
part
is
up
and
running,
we
go
ahead
and
we
start
the
data
read
my
side
during
the
restart
process
the
mutating
webbook
kicks
in
and
what
it
does
is.
It
gets
all
the
list
of
all
the
PVCs
that
are
created
on
that
particular
node
by
the
query,
civic
pod.
We
get
the
PVCs
and
attached
to
the
speck
in
the
mutating
maple.
So
when
the
data
distribution
demented
part
gets
recreated
on
the
node,
it
has
a
list
of
all
the
PVCs
that
it
needs
to
share
with
the
query
serving
pot
this.
C
This
is
one
of
the
solutions
that
we
have
built
in
order
to
achieve
this
particular
grid
pattern.
Once
we
have
laid
out
the
grid
pattern
that
the
next
thing
we
set
out
to
do
is
to
look
at
the
performance
of
this
particular
application
right
out
of
the
box,
we
saw
a
10
to
12
percent
degradation
in
the
performance
at
around
20
percent
CP
utilization.
C
We
wanted
to
be
as
close
to
our
bare
metal
performance,
because
this
is
a
latency
sensitive
application
and
we
cannot
compromise
on
the
performance
of
the
application.
So
we
set
out
to
look
at
the
performance
of
the
of
our
case,
node
that
we
on
which
the
search
is
hosted
on.
So
what
helped
us
move
the
curve?
There
are
three
important
things
that
helped
us
move
the
curve:
the
kernel,
CPU
and
networking
we
made
sure
all
our
Colonels
are
upgraded
to
the
latest
version
on
order
KH
nodes.
C
The
second
factor
is
the
CPU.
We
started
with
the
CPU
manager,
which
is
a
feature
which
is
available
in
Cuba
natives.
As
many
of
you
are
aware,
this
actually
helps
in
better
distribution
of
the
workloads
by
exclusively
pinning
the
CPUs
to
that
workload.
However,
this
particular
change
did
not
help
us
much.
We
then
disable
the
irq
balancer
on
the
node.
This
is
to
make
sure
we
disable
the
hardware
interrupts.
This
did
not
know
the
calibrator.
C
Then
we
set
out
to
tune
our
p
state
and
c
state
parameters
to
enable
the
turbo
boost
technology
on
the
bare
meter.
This
actually
had
the
smoother
curve
significantly
high.
However,
we
were
not
close
to
the
bare
metal
performance
here,
so
we
wanted
to
get
as
much
close
as
possible.
We
started
looking
into
our
networking
on
the
gates,
not
for
normal
workloads.
We
use
OBS
bridge.
C
However,
when
the
network
packet
transfer
happens
from
the
bare
metal
host
to
the
pod
interface,
we
see
that
there
are
a
lot
of
soft
irq
in
droves
happening
during
this
process,
thereby
reducing
the
network
throughput
and
also
increasing
the
CPU
usage.
So
we
switch
to
IP
wheel
and
networking
this
networking
change
reduced.
The
number
of
soft
irq
interrupts,
thereby
increasing
the
network
throughput
on
the
node.
So
this
brought
us
closer
to
the
performance
that
we
wanted
to.
C
After
making
these
changes,
we
were
nearly
able
to
match
the
bare
metal
performance
at
around
80%
CPU
utilization
at
20%
and
50%.
We
are
exactly
the
80%.
We
are
close
to
the
pavement
of
performance,
so
in
short,
we
were
able
to
achieve
the
speed
flexibility
of
the
search
application
without
compromising
more
much
on
the
performance
of
that
particular
application.
B
So
Cassini
is
all
now
it's
it's
new
for
us,
but
it's
still
older.
So
it's
like
it's
been
around
for
seven,
eight
years
that
we
know
of
so
over
the
years
we've
accumulated
a
lot
of
TechNet.
It
was
really
difficult
for
us
to
take
this
monolithic
application
and
break
this
down
into
micro
services
or
containers
and
trying
to
figure
out
how
the
containers
will
communicate
to
each
other.
It
was
painful,
it
was
really
painful,
so
that
is
one
of
the
things
that
I
would
suggest.
You
start
early
in
their
design
about
that.
B
This
is
not
a
short
journey
for
us.
It's
not
something
that
will
happen
in
a
matter
of
few
months
that
all
of
the
search
will
suddenly
move
on
top
of
kubernetes.
It's
a
long
journey.
That's
going
to
take
a
few
years,
so
we
wanted
to
make
sure
that
we
minimize
the
disruption
to
our
clients,
our
users,
and
so
we
have
to
do
things
in
a
way
that
sort
of
doesn't
impact
a
lot
of
our
clients.
B
What
I
mean
is
we
are
writing
matrix
in
the
kubernetes
world,
infra
meatiest
format,
but
we're
also
writing
them
in
the
old
format.
We
are
setting
out
alerts
on
top
of
the
Prometheus
metrics,
but
we
are
also
setting
alerts
on
top
of
the
older
format,
just
to
make
sure
that
our
tools
and
capabilities
that
we
have
already
worked.
But
this
also,
what
this
also
did
was
sort
of
maintain,
gave
us
a
chance
to
get
the
old
system
and
the
new
system.
B
One
more
example
is
we
had
to
develop
a
proxy
layer
which
abstracts
out
for
our
clients,
all
of
the
underlying
topology
or
the
layout
of
the
grid,
our
clients
code
to
that
proxy
layer
using
restful
services,
and
this
proxy
layer
then
hides
out
whether
search
is
running
on
bare
metals
or
is
it
running
on
kubernetes?
So
essentially,
if
we
ever
now
decide
to
change
technologies
again,
hopefully
it's
causes
less
disruption
for
all
of
our
clients,
and
we
can
just
change
the
proxy
layer
and
it's
ignores
and
the
clients
above
are
agnostic.
B
For
example,
as
you
saw,
we
had
to
create
a
mutating
web
book
so
that
we
can
mount
the
PVCs
with
on
these
two
separate
parts
now,
but
it
was
important
for
our
teams
to
do
it
this
way,
because
the
life
cycles
of
those
two
things
is
completely
separate
and
they
wanted
to
keep
them
separate
and
not
intermingle
than
with
one
another.
If
they
have
to
do
upgrades
and
such
or
such
things.
So
we
had
to
jump
through
hoops
to
basically
make
this
work
and
sort
of
mutating
where
books
saved
us
their.
B
Notes
eventually
died.
Things
go
wrong.
What
we
had
naively
assumed
is
that
we
are
running
on
top
of
kubernetes.
If
a
pod
dies,
the
stateful
set
will
spawn
it
again,
not
quite
true,
so
we
didn't
sort
of
get
this
design
thoroughly.
When
we
started
this
journey,
we
are
not
using
networked
PVCs.
We
are
using
local
PVCs
for
performance
reasons
and
when
a
node
dies
exactly
when
is
it
okay
to
now
delete
the
PVC
so
that
the
stateful
set
can
launch
another
pod
on
another
node?
That
is
available?
B
It's
still
something
that
we
are
working
on
an
ironing
out.
So
that
is
not
something
that
we
fully
vetted.
When
we
started
this
journey
and,
of
course,
performance,
you
can
never
start
early
with
performance.
It
took
us
a
lot
of
trial
and
error.
A
lot
of
runs
of
our
search
grid,
hats
off
to
our
performance,
guys
over
here
sort
of
figuring
out
what
works
and
what
doesn't
work,
and
especially
for
a
latency
sensitive
application
where
every
millisecond
counts.
B
Even
though
we
had
started
early
in
the
design
that
we're
gonna
measure
everything
from
the
get-go,
you
can
never
start
early.
You
know
so.
Yeah,
that's
one
of
the
lessons
learned
future
work.
So
this
brings
me
what
we
are
excited
to
work
on
next
now
that
we've
proved
that
search
can
run
on
top
of
kubernetes
and
we've
modeled
our
workload
you
saw
earlier
in
the
slides.
We
have
this
data
Orchestrator
layer
that
we
use
to
to
sort
of
download
this
new
snapshot
of
the
inventory
to
our
search
grid.
B
We
also
have
to
restart
all
of
these
nodes
when
the
new
snapshot
is
available.
This
takes
time
in
the
default
strategies
that
are
available
with
stateful
sets
when
we
started,
because
when
you
update
the
stateful
set
it
go,
it
went
pod
by
four,
which
meant
it
could
take
tens
of
hours
for
our
size
of
search
grid.
To
finish
that,
just
one
data
upgrade
so
are
we
use
the
data
or
test
return
to
parallel
East?
We
start
by
rows
now
in
the
newer
versions
of
kubernetes.
B
We
have
this
max
unavailable
strategy,
and
so
we
are
excited
to
sort
of
try
and
use
that
and
offload
some
of
more
work
to
the
platform
itself
recently,
if
also
hard
talks
about
volume
cloning,
so
you
saw
earlier
that
whenever
there
is
load
on
the
search
grid,
we
scale
and
bring
up
new
pods,
but
the
data
still
has
to
come
to
the
new
pods
and
that
takes
time
sometimes
can
take
hours.
So
we
are
going
to
experiment
and
see
if
we
can
implement
something
around
volume.
B
Cloning
with
when
local
PVCs
are
installed,
such
that
whenever
a
new
pod
comes
in,
you
can
download
this
data
from
a
pure
pod,
rather
than
going
across
to
our
Duplessis,
where
the
data
is
yeah.
No
remediation
with
local
PVC
really
needs
to
be
ironed
out
for
us
at
which
piece
in
the
node
remediation
workflow.
Do
we
listen
for
an
event
and
say:
ok,
the
return
of
investment
on
now
deleting
this
data
is
good
enough
that
we
should
delete
it
now
so
that
the
pod
can
come
up
on
another
node
versus.
B
Maybe
a
few
restarts
we'll
fix
it
and
it'll
come
up
in
a
matter
of
seconds,
so
we
are
still
going
to
work
on
ironing
out
that
thing
as
search
on
boards.
More
and
more
on
top
of
kubernetes
or
cluster
sizes
are
going
to
grow.
We
have
use
cases
that
require
sometimes
7000
servers,
and
we
want
this
to
be
more
manageable.
B
We
might
have
to
break
it
into
multiple
clusters,
which
means
that
the
controllers
and
operators
that
we
write
should
have
the
ability
to
have
this
uber
view
across
the
clusters,
and
so
the
like
say,
make
matrix
deployment
controller
or
the
data
distribution
controllers
that
we
have.
We
want
to
add
multi
cluster
support
to
it
not
also
use
cases
are
very
latency,
specific,
typically
for
latency
sensitive
applications
with
our
use
cases
for
search.
We
don't
share
our
cluster
with
anybody
else,
but
not
all
the
CLA's.
B
The
other
workers
that
are
running
in
the
cluster
can
be
preempted,
so
we
haven't
played
with
this
much,
but
we
intend
to
play
with
this
in
the
next
few
months,
all
right
so,
which
brings
me
to
the
end
of
the
talk
so
reiterating
back,
it's
absolutely
possible
to
run
a
latency-sensitive,
large-scale
application
state
full
application
on
top
of
kubernetes,
while
keeping
all
of
the
performance
or
while
keeping
the
performance
as
close
to
bare
metal
as
possible
and
leveraging
all
the
agility
flexibility
that
kubernetes
platforms
brings.
Thank
you.
B
B
E
B
The
immediate
roadmap,
it's
something
that
we
have
just
started
talking
about,
but
it's
not
immediately
something
that
we
are
working
going
to
work
on
in
the
next
year
or
so,
because
there's
lots
of
migration
effort
from
our
side.
That
needs
to
happen
before
we
can
think
start
thinking
about
using
service
mass
for
architecture.
D
C
B
B
So
search
is
an
eventual
consistent
system.
We
never
guarantee
that
you
will
have
full
consistency,
so
at
any
point
in
time
one.
So,
let's
say
there
are
two
snapshots
of
an
inventory
right.
We
don't
be
comment
load,
the
that
snapshot
inventory
inventory
in
a
single
short
across
all
the
servers.
So
at
any
given
time,
some
servers
will
have
the
newer
snapshot.
Some
servers
will
have
the
older
snapshot,
but
we
are
also
incrementally
sending
it
data
all
the
time
right.