►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Running Envoy as an Edge Proxy - Bala Madhaven, eBay + Qiu Yu, eBay
At eBay we run Envoy as our Edge proxy in PoPs (Point of Presence) across the globe. In this talk, while we walk through eBay's journey of running Envoy at all eBay POP's, we will discuss the following key aspects:
Architecture overview of eBay Edge Proxy
Managing Envoy in Kubernetes.
Consistently developing the entire stack across all edges of Helm.
Building a metrics and access log based pipelines for better visibility using Prometheus and ELK stacks.
Migrating Hardware LB configurations to Envoy and validating them using automation tools.
Integrating Envoy at ATS (Apache Traffic Server) to enable dynamic caching at eBay.
In this talk, we will touch on the challenges and the solutions that were applied while we completely moved all eBay mobile traffic from Hardware LB to Envoy.
A
A
So
here
is
our
agenda.
In
the
beginning,
we're
going
to
briefly
talk
about
the
edge.
Then
the
software
stack
of
the
edge,
the
elf,
oh
and
I,
was
7
controller
and
data
plane.
What
we
did,
how
we
did
it
and
after
well
cover
the
monitoring
and
the
deployment
aspect
of
the
edge
proxy
and
towards
the
end,
we
will
talk
about
some,
some
learnings
and
the
current
session.
A
So,
first
of
all,
why
edge
so
in
the
past,
eBay
only
has
data
center
in
the
US
as
traffic
grow
from
global
user.
We
will
realize
having
the
infrastructure
that
can
deliver
content
faster
to
our
end.
User
is
really
critical
for
us.
It's
not
just
about
ensure
better
user
experience
for
people
visiting
our
website,
but
they're
also
transmitted
into
more
revenue,
more
GMB
for
e-commerce
companies
like
eBay
and
all
that
translate
into
getting
closer
to
the
end
user.
B
A
So
in
eBay
we
have
cognitive,
Castille,
deploying
each
of
the
pub
and
by
using
communities
as
a
general-purpose
running
platform,
we're
on
our
edge
proxy
stack
there
and
by
having
at
proxy
in
the
pub
we
are
able
to
terminate
SSL
connection
closer
to
the
end
user
and
save
expensive
SSL
round-trip
latency
across
the
continents
that
actually
increased
the
page,
page
low
speed,
and
once
we
started
that
journey
by
converting
into
a
software
based
at
proxy
on
edge.
We
realize
that
there
are
much
more
things
we
can
do
than
just
increase
the
speed.
A
A
So
all
of
our,
for
instance,
will
be
in
active
mode,
be
actively
part
of
the
classroom
and
once
the
packet
get
there
they
are
for
will
determine
well
to
send
a
packet
to
for
the
l7
in
a
back
and
each
l7.
Sorry,
each
l4
uses
the
render
was
hashing
to
determine
the
schedule
result
so
for
the
packet
belong
to
the
same,
GDP
flow
is
guaranteed
to
reach
the
same
l7.
So
with
the
random
who's
hashing.
It
helped
us
to
operate
our
seven
culture
in
a
controlled
manner.
A
For
example,
in
case
we
want
to
roll
our
7
software
upgrade,
we
can
simply
believe
a
single
l7,
so
that
will
trigger
the
connection
training
from
l4
2000.
Once
the
connection
is
chain,
4
we'll
be
removing
the
house
7
from
the
hash
table
and
that
will
not
have
an
impact
for
the
connections
between
the
l4
and
the
remaining
of
the
earth
sevens,
meaning
no
SSL
disruption,
adding
a
l7
proxy
to
the
cluster.
A
On
the
other
hand,
well,
now
have
any
impact
for
the
establish
nation'
between
l4
and
our
7th
hour
only
triggered
at
least
amount
of
has
hash
repast.
So
that's
l4,
so
our
architecture
actually
follows
the
same
design
pattern
for
both
alpha
and
o7.
So
post
has
a
control,
plane
and
a
data
plane.
So
for
l4
we
use
IP
vsfs
our
data
plane.
We
have
a
homegrown
project,
called
yuppie
return
for
the
control
plane
for
the
l7.
We
use
envoi
as
our
data
plan
and
we
use
ingress
control
as
our
control
plane.
A
So
everything
is
a
Spectre
even
give
happy
model
driven
so
to
operate
a
cancer
we
create
sparks
through
kubernetes
api
server.
Then
our
alpha
and
our
seven
going
to
translate
aspects
into
the
alpha
and
a
seven
configs
accordingly,
and
since
it's
a
cube
native
department,
everything
runs
in
a
path,
so
we
can
easily
horizontal
scale
either
l4
l7.
A
A
little
bit
more
regarding
the
elf
for
software
load
balancer.
As
mentioned
earlier,
it's
a
BBS
based
data
plane.
We
have
an
in-house
build
kernel
model
called
IP,
vs
TBL.
The
TBL
stands
for
the
table
so
that
kernel
module
will
schedule
the
packet
based
on
the
the
hashing
table
and
that
hashing
table
is
generate
from
the
user
space
by
the
control,
plane.
Y
batata
and
ya
be
two
other
programs.
A
The
kernel
by
pushing
down
that
data
into
kernel,
zoonatic,
each
elf
or
pal,
has
a
unique
container
that
is
recent
responsible
for
load
the
kernel
module
into
the
into
the
community.
No
first
time
is
being
scheduled.
It
also
has
a
health
check.
Sidecar
check
the
house
between
the
L
4
&
7.
Obviously
only
after
passed
the
house
chapter
7
can
be
added
to
the
L
form.
A
Once
the
l
7
has
the
house
check,
l
4
will
actually
start
a
BGP
advertising
all
the
traffic
we
also
used
DSR
or
the
direct
server
returned
for
the
egress
path,
so
meaning
when
the
l7
returns.
The
response
back
to
the
end
user
well
bypass
the
L
fault
here
and
direct
a
return
path
to
the
end
user.
That
allows
us
to
run
much
less
l
4
to
match
more
else
events.
C
I'm
going
to
walk
you
through
in
much
more
detail
as
to
how
we
run
on
why
and
like
how
we
we
manage
it
and
also
the
cases
like
what
are
the
control
plane,
how
we
have
integrated
with
kubernetes
and
how
is
the
data
plane
and
what
are
the
features
and
optimizations
that
we
have
done,
and
one
can
also
discuss
about
one
specific
use
case
that
we
built
on
top
of
on
why
to
call
the
dynamic
caching.
So
let
me
start
so.
C
So
basically
things
like
when
you're
running
on
Y
or
any
else
around
proxy,
you
would
want
to
roll
out
new
features,
pretty
often
as
well
as
like,
or
you
want
to
do
some
software
upgrades
or
anything
that
you
want
to
do
pretty
often,
so
you
will
always
want
to
have
live
traffic.
Being
received,
and
then
you
still
want
to
have
a
much
control
when
you
do
this
so
coconut
is
provides
the
rolling
update
strategy.
C
C
Deployment
controllers
helps
you
to
scale
up
and
scale
down
if
you
want
as
well
as
it
gives
you
more
much
more
flexibility
cases
like
if
you
want
to,
if
you
find
out
you're,
all
out,
didn't
go
as
expected,
and
you
want
to
pass
them
or
probably
like
even
rollback
them.
So
so
that's
the
reason.
Even
for
the
deployment
controller
option,
as.
A
C
Colleague
said
before,
so
we
are
everything.
Income
rate
is
a
spec
driven.
So
this
is
what
this
part
is
going
to
say
like
how
we
are
going
to
create
the
Envoy
configuration
dynamically
and
again
we
have
a
bunch
of
specs.
So
basically,
the
ingress
spec
is
what
we
use
for
specifying.
What
is
the
listener?
Ip,
you
want.
What
are
all
the
features
that
you
want
to
turn
on?
Do
you
want
TLS
or
termination,
or
it's
just
going
to
be
HTTP,
or
do
you
want
G's
or
any
of
these
features?
C
You
could
also
say
like
what
port
you
want
to
listen
on,
and
things
like
that.
So
this
all
these
details
that
basically
typically
goes
under
the
listener
configuration
goes
on,
there
are
introspect
and
for
the
routes,
basically
the
ones
that
envoy
l7
routes
goes
as
a
part
of
the
CRD
socking
in
c
rd
is
as
a
type
of
an
api
resource
object
that
gives
you
the
option
to
create
your
own
schema
and
kubernetes
will
take
care
of
managing
it
throughout
the
lifetime
of
that.
C
Similarly,
for
the
endpoints,
basically
the
the
clusters
that
the
backend
pools
you
could
use
the
endpoint
spec
and
so
how
this
works
is
once
you
once
you
end
the
service
pack.
So
once
you
create
all
these
spec
on
the
AP,
a
QB
API
server,
so
the
cube
API
server
realizes
these
using
a
controller
called
the
ingress
controller.
So
the
English
controller
again
is
this
kind
of
pretty
dynamic
you
could?
It
could
basically
get
the
task
from
the
API
server
and
then
it
could
convert
it
into
these
specs
as
as
config
maps,
again
kubernetes
resources.
C
So
these
conflict
maps
will
again
be
picked
up
by
the
discovery
service
that
will
be
running
as
a
sidecar
container
with
envoy,
so
I'll
be
discussing
that
in
the
next
slide.
So
what
the
ingress
controller
basically
does,
is
it
kinds
of
whatever
Roxy
you
might
be
using
it
kinds
of
tries
to
convert
your
your
legacy
style
specs
into
configs,
that
you
could
easily
dynamically
load
and,
as
you
know,
let's
say
if
just
one
endpoint
Changez
only
the
endpoint
spec
would
get
modified
and
only
your
endpoints
config
map
would
be
changed.
C
That
way,
all
the
other
configuration
would
still
be
the
same.
So
this
is
this
slide.
Tells
you
more
about
like
how
every
part
l7
pod
runs.
So
basically,
as
you
see,
you
have
the
the
on
Y
container
and
a
discovery
service
and
a
bunch
of
other
stats.
D-File
beat
containers
so
I'll
be
mostly
talking
on
the
discovery
and
the
Envoy
and
a
little
about
the
unit
container.
So
the
unit
container
is
hugging.
C
One
good
option
where,
when
you
want
to
when
you
want
to
like
when
have
some
tasks
that
runs
to
completion
things
like
you
want
to
fetch
a
token
or
if
you
want
to
do
some
unit
settings
like
changer
congestion,
control,
algorithm
or
your
TCP
socket
settings,
you
could
have
the
unit
container.
Do
that
for
you
before
the
Envoy
comes
up
that
way,
things
are
up
and
running
for
you
and
and
then
the
discovery
container,
so
the
discovery
container
mainly
watches
on
the
config
Maps.
C
So
what
it
does
is
it
tries
to
see
if
there's
any
change
in
the
config
map
and
uses
the
envoys
LDS,
RDS
EDS
and
the
serious
features
to
try
to
update
the
configuration
and
there's
an
envoy
agent
that
runs
along
with
the
Envoy
container.
It's
a
sidecar
process
that
kind
of
is
is
the
is
the
pup
is
the
one
who
tries
to
boot
up
on
voice.
C
So
it
sees
that
there
is
a
change
in
the
config
and
then
it
loads,
and
then
it
uses
the
on
voice
hot
restart
feature
to
try
to
dynamically
restart
the
Envoy
without
any
loss
in
traffic,
and
we
also,
let's
say
you're
you're
doing
all
these,
and
then
you
are
also
using
the
readiness
and
the
liveliness
Pro.
So
the
readiness
for
again
this
is
something
that's
that's
part
of
the
kubernetes.
C
So
the
the
readiness
probe
is
something
like
like
you
want
the
container
to
be
a,
but
you
don't
want
it
to
be
getting
the
traffic
as
soon
as
it
comes
up,
you
want
to
have
your
own
dynamic
discoveries
or
whatever
that
you
are
trying
to
load
our
applications
to
get
ready.
So
you
you
basically
have
your
own
healthy
hand,
point
as
a
part
of
on
why,
and
you
could
say
they
know
what.
Only
after
all
these
are
probably
like.
C
The
listeners
talk
head
is
opened
right,
you'll
have
things
like
these,
and
then
you
could
say
you
know
what
one
day
after
this
deemed
this
container
is
ready.
So
what
this
does
is
tell
that
it,
the
Elfa
load
balancer,
would
not
load
balance
any
traffic
to
the
L
sevens
and
only
after
the
readiness
protein
and
then
is
successful.
You
basically
start
giving
traffic
to
the
Envoy.
The
library
nurse
power
bond
is
basically
to
continuously
monitor
unwise.
If
it's
running,
if
the
containers
for
some
reason
crashes,
it
is
immediately
moved
out
of
the
concision
hashing
table.
C
C
So
so,
as
you
know
like
when
we
are
creating
this
edge,
we
are
basically
trying
to
replace
all
the
hardware
L
bees
with
envoys,
and
the
main
task
that
we
had
to
do
first
was
to
identify
what
are
all
the
features
that
that's
that
were
being
used
in
the
current
Hardware
elby's
and
then
trying
to
find
equivalence
of
on.
Why
on?
C
So
again,
we
came
up
with
a
few
set
of
the
way
to
reverse
engineer
based
on
the
path
or
the
prefix
or
the
query
match.
We
could
dynamically
generate
request,
that's
going
to
hit
every
specific
routes
and
then
try
to
use
this
tool
to
send
traffic
to
both
the
hardware
and
the
software
and
then
see
if
the
response
you
get
back
is
exactly
the
same.
C
We
also
had
rum
that's
basically
to
to
monitor
more
on
the
performance
side
whether
your
edge
network
is
working
as
expected,
or
maybe,
if
you
roll
out
some
new
features,
are
you
seeing
any
degradation
in
performance,
and
things
like
that?
So
rum
basically
stands
for
real
user
metrics,
so
this
is
part
of
on
every
ebay
pay
once
the
actual
content
is
delivered.
C
You
have
a
small
JavaScript
that
tries
to
download
some
from
the
pop
and
then
gives
you
a
much
better
information
about
like
how
the
what
was
the
speed
I
mean
how
how
close
it
was.
It
was
any
transmissions
or
things
like
that,
so
that
was
mostly
the
migration
part.
Along
with
this
further
we
did
a
brain
to
get
up
and
running.
We
had
to
integrate
with
a
bunch
of
proprietary
assert
management
systems
that
you
want
to
manage
like
SSL
service
or
anything.
So
we
once
we
got
this.
C
We
are
now
looking
into
the
data
plane
features
that
we
want
enable
I'm
not
going
to
list
the
awesome
features
that
onwhy
provides,
but
because
it's
going
to
be
unique
for
every
specific
use
case,
I'm
just
going
to
touch
a
little
bit
on
the
certain
optimizations
that
we
did
on
top
of
envoy
and
that
so
one
major
requirement
we
had
was
because
it's
an
edge
use
case.
We
always
want
a
much
better
connectivity
from
the
pop
to
the
data
center
right.
C
So
what
we
did
was
on
why
we
were
able
to
like
have
connection
prefetching
based
on
the
dynamic,
dynamically
traffic
rate.
So
you
would
basically
go
and
see
that
if
the
traffic
rate
is
higher,
you
would
want
to
pre
create
or
prefetch
connections
with
SSL
talk
with
SSL's,
so
that
you
don't
want
any
of
your
requests
to
be
queued
at
any
time.
And
this
this
was
this.
C
After
this
we
based
on
the
stat,
we
were
able
to
see
that
the
cube
connection
count
went
down
to
zero,
so
delivering
much
better
experience
to
UV
users,
and
also
we
had
a
couple
of
use
cases
where
you
wanted
to
like
drop
or
reset
a
particular
connection.
Maybe
during
for
security
reasons
or
whatever
we
don't
want
them
to
go
all
the
way
up
to
the
DC.
So
we
wanted
them
to
do.
We
handle
that
pop.
C
So
what
we
did
was
we
created
an
l4
filter
that
kind
of
sits
along
in
on
Roy
that
kind
of
watches
for
whatever
you
don't
want
and
then
drops
or
resets
back
the
connection
so
and
also
we
we
wanted
a
way.
Let's
say
if
the
connectivity
from
the
edge
to
the
data
center
is
is
bad
or
went
down.
You
don't
want
to
your
customers
to
see
503
upstream
timeouts,
so
we
came
up
with
a
small.
C
The
feature
again
says
basically
called
the
default
cluster,
where
you
want
to
gracefully
handle
Chris
will
handle
the
way
that
you
want
to
show
your
user
better,
a
much
better
experience,
rather
than
showing
a
blank
page
so
yeah.
These
are
the
customizations
that
we
did
and
like
it
was
very
easy
to
do
it
on
Envoy
because
of
the
existing
architecture.
C
Along
with
this
on
the
on
the
operating
system,
we
were
also
able
to
tweak
a
few
settings
to
again
to
have
a
much
better
experience.
So
one
was
we
were
only
running
cubic
and
as
the
TCP
congestion
control
algorithm
and
we
wanted
to
adopt
beabea.
So
we
went
with
enabling
PBR
for
the
RAM
traffic
and
we
saw
like
tremendous
improvement
in
the
rate
traffic
rate
at
which
you
deliver
it
back
to
the
client,
and
we
didn't
want
to
miss
that
out.
C
So
we
went
ahead
and
enable
bbr
and
as
I
told
before
it's
using
the
unit
containers,
you
could
turn
on
PBR
and
then
turn
on
the
on
voice.
So
similarly,
we
did
a
bunch
of
TCP
optimizations
as
as
simple
as
turning
off
slow
start
or
having
a
better
socket
receive
or
send
buffers.
So
this
is
a
quick
glance
on
how
it
went
through.
So
if
you
see
the
first,
the
line
on
top
is
basically
what
is
the
response
times?
C
If
you
have
it
directly
terminated
at
the
data
center
from
all
over
the
world,
but
the
one
on
the
in
in
between
is
what
let's
say:
it's
pretty
much
close
to
the
hardware.
Lb
that's
running
on
the
power,
let's
let's
say,
but
we
always
don't
want
to
settle
with
something
that
is
equal,
but
we
want
to
beat
them
right.
So
what
we
did
was
along
with
on
wise.
C
We
also
did
the
TCP
optimizations
and
ran
the
bbr
optimizations,
and
if
you
see
to
the
to
the
blue
part,
we
are
now
clearly
beating
the
numbers
that
and
any
current
or
edge
Hardware
in
B's
could
provide
with
with
own
voice
and
with
our
customizations.
So
that's
one
of
the
the
best
things
that
you
would
want
to
and
yeah,
so
that
that
concludes
on
the
data
plane
path.
C
So
this
is
something
that
I
want
to
say
like
what
you
can
do
on
top
of
Envoy
when
you
run
it
on
the
edge,
so
we
created
again
a
new
filter.
That's
that's,
basically
does
caching
and
when
I
mean
caching,
it's
dynamic.
Caching,
so
dynamic
has
in
like
this
is
something
that,
in
a
page
where
you
have
content
that
is
specific
to
geography
like
if
it's
a
if
a
German
customer,
you
want
to
show
him
in
his
specific
dollars
or
whatever.
So
so
we
were
able
to
we
using
the
filter
that
we
created.
C
We
were
able
to
do
it
based
on,
let's
say,
a
path
or
a
specific
query
or
a
specific
user.
So
we
we
were
able
to
achieve
this.
As
a
part
of
this
we
did
when
we
created
the
filter.
We
did
not
have
the
cash
stores
as
on
wire
themselves,
but
we
just
used
ATS
or
any
other
store
cash
stores
that
he
could
use.
C
But
again
we
use
the
ring
hash
she'll,
be
in
envoy
to
basically
to
get
your
sharding
turn
on,
because
you
won't
want,
let's
say
multiple
ATS
running
and
you
want
the
cash
to
be
stored
and
retrieved
on
one
of
them,
and
you
also
always
want
to
access
them,
and
that
again
has
to
be
a
something
that
is
part
of
a
path
prefix
or
query.
Or
you
want
to
crunch
that
data,
so
we
use
the
ring.
C
That's
like,
as
I
said
it's
ten
times
much
better
and
not
only
the
T
TFB,
but
because
you
are
delivering
the
content
really
fast
to
the
user.
You
are
trying
to
like,
engage
them
much
like
for
them
to
skim
through
to
see
what
is
the
pricing
or
what
is
is
this?
The
item
I
am
expecting
or
things
like
that
you're
kind
of
engaging
them
and
don't
want
them
to
go
to
other
websites.
So
so
that's
one
major
advantage
that
we
were
seeing.
C
We
also
have
a
few
in
the
roadmap,
something
like
we
want
to
say
if
someone
is
searching
for
for
an
item-
and
you
want
start
caching
proactively
the
top
five
things
that
basically
is
being
listed
as
a
part
of
eBay
site,
so
that
that's
something
that
we
are
thinking
about
again.
This
is
a
very
overview
of
what
we
got
out
of.
Let's
say
the
response
times.
If
you
directly
hit
the
DC
and
the
yellow
line
is
with
having
it
as
an
edge
proxy
and
what,
if
you?
What
is
the
benefit
that
you
see?
A
Coming
to
the
monitoring
or
the
observability
part
of
the
department,
we
use
promises
for
the
monetary
purpose
and
we
replicate
the
same
thing
to
all
our
pops.
So
it's
kind
of
a
self-contained
stuff.
So
in
each
hop
up
we
run
premise
as
a
pair,
we
use
at
a
nose
for
the
promise
hae
from
there,
the
data,
the
mattress
being
merged
and
duplicated
and
returned
to
the
core
interface
we're
on
Mineo
in
each
of
the
pops
to
further
to
keep
the
matrix
for
a
longer
time
period.
A
We
haven't
got
to
the
part
to
use
promise
or
channels
for
aggregation
across
the
globe,
but
we
have
a
centralized
growth
fauna
that
can
read
the
promise
data
from
each
of
the
pops
for
both
L
for
our
seven
metrics,
as
well
as
a
metric
from
other
supporting
stacks
and
also
measures
from
other
data
sources
as
well,
such
as
a
unity
search.
So
a
little
bit
more
recording
and
more
specific
things.
A
We
use
teste
Explorer
as
a
sidecar
running
together
with
my
to
receive
the
histogram
Tara
and
that's
test
exported
intern
exposed
to
the
premise
interface
for
screen
server
to
scrubbing
from
we
use
FA.
Okay
for
the
access
log,
so
F
stands
for
the
FRP,
so
inside
each
of
the
l7.
How
the
therapy
is
responsible
for
shipping,
the
axonal
out
to
a
centralized
locks,
their
farm
from
there,
the
lock
being
passed
process
and
then
sent
to
a
calf,
cart
and
I
think
and
then,
in
turn
being
sent
back
to
the
data
center.
A
We
have
a
managed
electives
are
classes
there
from
there
we
run
analytics
jobs
for
for
analytics
and
alerting
purpose
as
well,
alerting
we
have
a
manager
deploy
in
each
of
the
parts.
We
also
have
external
charts
that
check
our
check
on
our
public
web
so
ensure
everything
works
from
end
to
end.
We
have
post
etiquette
based
alerts,
that's
the
manager.
We
also
have
dynamic
base
dollars.
That
is
the
anomaly
detection
based
on
the
research
data.
A
We
use
home
for
our
deployment
management
so
that
it
can
give
us
a
consistent
stack
across
all
the
pops.
Actually,
it's
not
just
about
part
but
also
consistent
department.
You
know
Democrats
and
our
production
cancer
as
well
as
you
can
see
from
the
screenshot
here.
So
everything
is
packaged
into
a
hump
chart.
A
B
A
C
Yeah,
so
now
that,
like
most
more
than
50%
of
the
worldwide
ebay,
traffic
is
being
deduct
redirected
through
the
edge
and
it's
on
why
that
is
basically
taking
the
traffic.
We
wanted
to
share
a
few
learnings
that
we
we
we
we
experienced,
and
we
also
wanted
to
highlight
what
are
the
challenges
going
forward?
I'm,
pretty
sure
the
part
of
the
community
is
going
to
do
the
same
so
in
the
learnings.
C
Basically,
what
we
see
is
because
this
is
like
everything
is
new
here
edges
new
software
stacks
are
new,
so
you
had
to
relay
a
lot
on
testing
and
testing
is
definitely
the
the
key
part
that
you
want
to
make
sure
that
you,
you
spend
an
effort
and
time
on
to
get
your
thing
initiative
successfully.
So
the
ways
that
we
basically
did
was
the
hardware
to
software
cross
validation.
That
I
explained
rum
for
let's
say
for
continuously
looking
for
performance
metrics
that
you
don't
want
to.
C
Let's
say
after
a
specific
rollout
of
a
node,
you
would
probably
see
that
some
difference,
so
you
will
want
to
make
sure
that
your
performance
is
always
being
met
as
well
as
like
new
cases.
Like
things
like
what,
if
a
DC
goes
down
right,
what
if
one
other
pop,
goes
down
right?
So
how
are
you
going
to
handle
these
cases
right?
So
we
have
to
do
a
thorough,
rigorous
testing
of
each
of
these
cases
before
we
could
start
taking
production
traffic.
So
we
so
that's
that's
one
key
step.
C
Testing
is
key
and
the
second
part
is
okay.
Now
that
you
have
tested
and
got
everything
built
and
everything
ready,
how
are
you
going
to
transition
it
like?
Are
you
going
to
turn
on
a
switch
and
then
make
sure
all
the
hundred-person
of
the
traffic
is
going
to
go
to?
The
software
will
be
no
right,
so
you
would
want
to
do
that
in
a
much
controlled
manner.
C
Where,
like
you
know
that
you
know
what
let
me
put
1%
5%,
you
want
to
go
in
slices
and
for
this
you
would
want
to
like
spend
some
effort
in
integrating
with
existing
systems
in
your
company,
where
you
would
want
to
like
finally
send
traffic
to
multiple
systems
and
then
monitor
them.
So
these
are
the
things
that
we
thought
is
is
going
to
be
useful
for
the
community
as
well,
and
the
challenges
in
the
challenges
part.
As
you
know,
I
mean
we
are
always
growing,
so
the
number
of
pops
keeps
increasing
every
month.
Right.
C
So
now
comes
the
pain
part
of
it
like.
How
are
you
going
to
manage
the
ops?
How
are
you
going
to
make
sure
there's
no
drift
in
the
configuration
that's
being
running
in
each
of
these
pops
right,
so
we
are
still
thinking
about
solutions
for
these
and
that's
the
that's
one
of
the
challenges
and
also
again
with
much
more
power
you
get
more
I
got.
You
have
to
be
careful
because
now,
with
sitting
the
edge,
you
know
that
exactly
what
is
the
amount
of
data
that
you'll
be
getting?
C
You
know
where
the
users
are
you
get,
and
then
you
know
like
how
far
the
users
are
from
the
pop
and
where
the
users
are
growing
and
they're
coming
from
mobile
or
they're
coming
from
desktop,
and
things
like
that,
you
would
want
to
do
a
lot
more
with
such
amount
of
data.
So
that's
one
of
the
pleasant
challenges
that
we
have
that
we
would
want
to
handle
and
make
it
of
more
use
for
eBay,
and
the
second
thing
is
because
we
moved
from
legacy
hardware
to
coupon.
C
Ids
like
now
comes
the
problem
of
debugging
and
troubleshooting
right.
How
are
we
going
to
do
that?
Let's
say
if
there's
a
p1
case
like
how
are
you
going
to
quickly
go
ahead
and
turn
on
certain
things
and
start
debugging
it
right
from
because,
as
you
know,
coverage
is
a
really
complex
system.
So
so
these
are
the
challenges
we
are
still
addressing
and
we
just
thought
it
would
be
good
to
share
it
so
yeah
and
yeah.
If
you
can
go
with
any
Q&A
and
we
are
yeah
one
or
two
questions.
Please.
C
Yeah
definitely,
as
I
said,
we
did
contribute
some,
but
these
things
are
more
of
like,
basically
that
the
dynamic
prefetching
was
based
on
a
small
algorithm
that
we
wrote.
We
wanted
to
test
it
out
well
and
then
probably
push
it
upstream.
That
way,
it
would
be
perfect
when
it
everyone
runs
it
so
yeah.
We
will
definitely
and
then
also
be
very
in
a
rush
to
get
all
these
up
and
running
in
a
year
or
a
year
and
a
half.
A
Chaos
testing
with
this
pre
menu,
I
would
say
so
basically
just
to
ensure
when
things
go
south
our
Intel
Intel
stack
can
still
function,
as
is
so
basically,
which
is
random,
kill
this
certain
part
of
the
stack,
the
l4
and
a7,
to
ensure
the
house
checking
the
other.
Everything
can
still
work
expected.
Meanwhile,
we
have
the
monitoring
going
on
to
make
sure
we
capture
all
the
events.
C
C
Yes,
so
every
pop
is
a
kubernetes
cluster
and
you
have
a
helm
to
know
what
is
the
config
that's
actually
running.
What
is
the?
What
is
the
latest
software
version?
What
are
the
IPS
that
you
are
using
and
things
like
coloring
and
all
those
everything
is
specific
yeah
we
use
help
to
roll
out
all
the
changes.