►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Grow with Less Pains - Meshing From Monolith to Microservices - Leo LIang, Cruise Automation
This talk will walk you through the adventure, learnings and culture shift on how we evolve A high growth sartup architecture into microservice world; We used L5D, Consul, Nginx, Prometheus, and customized L5D plugins to build up the service mesh that is taking care of billions of request every day. The service mesh manages all traffic from edge to Inter-services and is working with a heterogeneous type of service nodes from EC2 box, ECS, K8S to Lambda. The flexibility, observability and stability enable the business and teams to move fast with confidence. Contentof the presentation will be from 1st https://goo.gl/cE5e6Y 2rd https://goo.gl/czHkex 3rd https://goo.gl/c2ucMu
https://sched.co/MPfS
A
Hello,
I'm,
Rita
and
I'm
a
contributor
on
the
linker
new
project
and
I'm
here
to
introduce
Leo,
who
will
be
speaking
to
us
about
how
his
company
brought
a
service
mesh
into
their
infrastructure
and
use
it
to
serve
billions
of
requests
per
day
and
what
they
learned
from
that.
This
is
a
reminder
also
to
rate
each
talk
on
SCAD
comm.
After
each
session,
Leo
has
a
ton
of
industry
experience
with
distributed
systems
having
worked
at
Microsoft,
Amazon,
AWS
and
Twitter.
So
let's
hear
what
he
has
to
say.
Please
welcome
Leo.
B
Thanks
for
everyone
here,
so
my
my
name
is
Leo
and
I
actually
work
at
currently
I'm,
actually
working
in
cruise
automation,
a
self-driving
car
company.
These
toys
actually
made
me
my
my
work
in
my
previous
company
as
well
as
actually
my
as
a
role
contributor
of
legatee.
So
here's
actually
my
little
profile,
let's
actually
get
started,
I
try
to
actually
describe
a
start-up
in
Python
code.
This
is
usually
how
it
works
in
the
early
stage
startup.
Somehow
you
actually
pick
a
domain
right
and
then
that
domain,
you
don't
know
the
difficulty
of
the
domain.
B
Maybe
and
then
you
actually
pick
pick
a
goal
which
measure
in
progress
and
then
every
day
your
company
or
your
team
make
some
progress,
bring
some
money
and
tomorrow,
if
you
still
have
some
money
left
and
you
still
have
something
to
do
and
do
again
until
some
point,
you
either
run
out
of
money
or
either
you
finish
your
progress
right,
that's
actually
fairly
simple.
This
is
a
hypothetical,
fairly
simple
case
of
a
startup.
B
So
this
is
something
that
you,
by
the
way
she
shouldn't
be
random,
but
it
should
be
a
functions
somewhere
that
this
is
something
that
you
should
actually
optimize
for.
You
want
to
actually
maximize
your
iteration
progress.
Okay
in
every
single
iterations
you
want
actually
maximize.
Your
show
puts
how
to
do
that.
B
Usually
you
actually
hire
a
bigger
team
right
and
then
you
get
every
team
member
I
have
better
productivity.
We
got
him
and
when
you
have
a
bigger
team,
what
can
you
do
and
usually
according
to
comments
law
and
any
organization
that
designed
a
system
in
effort,
ibly
produce
a
design
whose
structure
is
a
copy
of
the
organizational
communication
structure?
So
that
implies
two
things
a
you
neither
actually
design
your
organization
first
and
then
you
have
your
architectures,
your
architecture
mimic
your
organization,
whatever
you
look
like
or
usually
have
a
architecture
first.
B
B
We
actually
start
actually
build
up
a
new
stack
in
2016,
very
early
of
2016.
At
that
time,
linka
D
just
released
the
first
version.
I
can
remember,
excited
emotion,
number,
it's
open
a
something,
and
at
that
point
of
the
time
the
company
actually
has
10
engineer,
okay
and
then,
by
the
end
of
life,
by
the
middle
of
that
2018,
we
have
a
hundred
and
then
2016,
there's
a
one
month,
one
big
fat
monolith
and
then
2018.
B
We
have
40
services
and
300
million
requests
a
day
by
that
time
and
by
2018
that's
actually
two
to
billions.
So
if
we
actually
measure
oh
by
the
way,
the
companies
that
actually
purely
on
AWS,
they
actually
host
their
own.
The
easy
tools
at
that
point
that
we
have
like
tens
of
boxes
and
then
by
18
we're
for
a
couple
hundreds
of
boxes.
B
First
of
all,
let's
talk
about
what
sitting
caddy
right.
This
thought
is
specific.
Our
experience
on
linker
db1.
What
is
an
Caddy
is
a
is
a
very
feature-rich
proxy,
so
it
provide
manageability
and
observability
of
your
traffic's
within
any
two
points,
and
it
is
actually
based
on
a
fairly
sophisticated
stack.
Mainly
the
link.
Id
theme
is
actually
a
group
of
gang
they're
actually
from
Twitter,
so
they
actually
do
on
a
very
mature
to
a
technology.
At
a
time
it's
an
epi
pen,
angle,
Scala
and
frif
and
Lindy
use
a
domain-specific
language
called
DTaP
for
routing.
B
B
What's
special
about
linka
DB,
one
DS,
it
provides
a
very,
very
powerful
architectures.
Allow
you
to
actually
write
any
write
code
that
actually
intercept
in
its
pipeline.
This
enables
us
to
actually
make
the
linkage
between
any
two
nodes
programmable.
What
does
it
mean?
That
means
we
can
actually
implement
filter
transformer.
We
can
manipulate
the
result
and
manipulate
the
payload
of
the
traffic
between
any
two
points.
So.
B
What
this
is
what
what
is
cost
about.
This
talk
is
about
architecture,
evolution
for
them
from
monolith
to
micro
service,
driven
bytes
of
his
mash
in
2016,
at
that
one
Koopa
naughty
DCOs
are
still
in
its
very
early
stage
and
serval
is
actually
on
the
rise
and
as
a
start-up,
we
can
actually
bet
on
any
kind
of
this
technology.
At
that
point
of
the
time
for
now,
I
think
you
should
be
feel
free
to
just
bet
on
Cuba
naughty.
That's
why
we
are
here
right
so
and
also
as
a
start-up.
B
What
we
talked
about,
essentially,
we
try
to
actually
present
a
very
pragmatic,
a
systematic
journey
about
how
we
actually
from
mana
life
from
one
service
through
like
40
services,
and
it's
imperfect.
So
we
are
not
coming
here
to
actually
present
a
perfect
solution,
but
instead
is
we
try
to
be
actually
very
pragmatically
and
realistically
explain
what
we
did
and
walk
through
the
actual
tree
of
that
we
have.
B
So
this
is
a
gender
first,
we
want
you
to
talk
about
edge,
observe
and
then
we
will
actually
talk
about.
The
edge
is
actually
a
critical
component.
Here
we
actually
deep
dive
into
that,
and
then
we
were
to
talk
about
how
actually
be
of
the
mesh
internally
for
the
service,
and
then
we
will
talk
about
observability
and
at
the
end
we
will
actually
talk
about
some
of
the
learning
and
conclusion
that
we
have
here.
B
So
what
is
the
edge
energies
and
is
a
service
component
that
exposed
to
public
internet
and
usually
it
is
a
gateway
and
need
the
wide
internal
and
external
services
from
the
high
level?
How
we
actually
implement
the
edge
service
is
actually
combination
of
nginx
and
link
ID
together.
You
may
actually
ask
why
you
come
by
to
proxy
together
right.
It's
weird
our
each
other
explained
there
here
nginx,
because
a
lot
of
startup
right
when
you
actually
running
really
fast
and
you
have
a
monolith.
What
would
you
do
you?
You
want
to
implement
API
and
someone?
B
When
you
actually
move
from
a
monolith
to
a
micro
service,
you
really
actually
can't
just
saying
at
this
corner
all
the
old,
cryin
and
I
need
to
achieve
all
the
Oakland
to
a
new
crime
format.
It's
impossible
to
do
that
and
we
actually
use
the
nginx
to
actually
do
the
long,
URL
normalization
as
well
as
actually
performing
policy
in
post
enforcement.
B
So
let's
just
get
to
this
is
a
high
level
of
what
link
ID.
Actually
look
like
the
pipeline
at
you.
Look
like
you
can
see
the
pink
color.
This
is
actually
a
pipeline
and
then
the
blue,
the
blue
thing
is
actually
where
you
can
actually
write
a
plug-in
to
inject
into
that
in
every
single
stage,
particularly,
you
actually
more
focus
on
the
in
the
identifier
stage
here.
So
we
would
have
a
lot
of
example.
Later.
B
So,
okay,
let's
actually
get
into
how
let's
getting
into
the
nginx
lair
yell,
be
a
request.
Gesture
from
a
mobile
to
a
to
yell
be
and
yell,
be
actually
for.
The
request
into
engine
makes
a
lair
and
what
ngx
actually
do
for
at
that
point
is
to
perform
some
policy
and
security
check,
for
example,
dealing
with
CSRF
header
validations
and
also
actually
doing
Heather
normalizations.
There
try
to
remove
unnecessary
header
inject
some
of
the
header
there,
and
then
also
this
is
a
keeper
for
that
is
it
does
URL
normalization.
B
B
The
first
part
for
that
is
slash,
h1,
initiated
protocol,
so
we
can
actually
extend
that
to
h2
in
future
and
then
the
data
center
us
is
1/8,
that's
actually
obviously
digested
we're
not
like
is
a
Tobias
domain
and
prot
is
a
stage
so
in
on
the
edge
we
actually
encode
these
things
actually
inside
the
nginx
deployment,
so
the
nginx,
you
will
actually
form
the
canonical
URL
in
that
form.
This
allow
us
to
have
multiple
data
centers.
B
Let's
take
a
look
at
in
the
engine
in
the
link
ID
layer
when
we
actually
have
when
we
actually
have
the
that
the
nginx
actually
passed
down
the
canonical
URL
to
to
lenka
d.
What
it
does
is
the
link
ID
actually
throwing
that
canonical
URL
into
name
ID,
name,
ID,
again,
name
ID,
just
like
DNS
right.
It
will
actually
translate
the
particular
path
like
each
one.
Us
is
one
a
pro
slash,
four
slash
bar
a
equal
to
one.
Two
three
become
some
host
port,
slash
fubar,
this
translation
process.
It
is
actually
called
resolutions.
B
From
here
you
may
actually
notice
that
I
have
actually
to
this
one
there's
actually
two
boxes
here,
one
it
is
actually
the
Authenticator
another
one.
It
is
actually
experiment
decider,
they
are
actually
the
packing
in
the
link
ID.
So
I
was
really
try
to
explain
what
they
do
and
how
we
actually
implement
that
in
the
following
slide.
B
This
is
the
first
thing
that
you
usually
actually
worry
about,
because
you
want
people,
you
want
your
engineers
to
actually
be
able
to
implement
service
in
anything
work
that
that
that
they
like
or
in
different
language,
that
they
want,
and
you
need
to
actually
get
a
consistent
way
or
easy
way
to
actually
authenticate
the
request.
In
our
specific
case,
we
were
actually
using
django.
Django
is
actually
based
on
a
cookie
oven
occasions,
and
we
then
actually
enable
jadibooti
JSON
web
token
as
an
secondary
way
to
actually
authenticate
request.
B
B
B
What
the
logic
actually
do
is
explain
in
this
slide.
On
the
left
hand,
side
it
is
actually
config.
It
actually
has
a
proxy
config
and
a
GWT
key
config,
ok
and
then,
on
the
right
hand,
side.
This
is
actually
the
logic,
the
high-level
logic.
As
you
hear
the
original
implementation.
Obviously
it
is
actually
in
the
JBLM
jvm
language,
that's
actually
in
Scala
and
I.
Don't
want
to
present
the
Scala
here,
because,
given
the
fact
that
life
is
very
hard
to
read,
so
I
try
to
reform
the
logic
using
a
Python
for
pseudocode.
B
So,
as
you
can
see
here,
the
logic
is
on.
The
high
level
is
fairly
simple.
Right.
First,
try
the
GWT
of
an
occasion.
If
we
actually
fell
it
tried
to
proxy
the
code
into
your
authentication
service,
which
is
a
micro
service.
Actually
is
our
your
stack
in
that
way?
We
get
the
best
of
both
work.
So
if
something
is
actually
if
this
authentication
is
required,
this
particular
class
is
authenticated
by
one
of
the
authentication
methods
is
success,
and
then
we
actually
pass
the
identification
stage
to
the
next
step.
B
B
Because
fundamentally,
when
you
actually
design
on
eyes,
have
a
lot
of
service?
Actually
in
your
step,
what
you?
The
the
first
thing
that
you
want,
is
you
want
the
POI
area
time
deploy
anytime
right?
You
want
multiple
service
being
able
to
deploy
at
the
same
time,
and
then
they
you
want
them
to
be
able
to
do
the
experiment
to
do
progressive
rollout.
They
don't
need
to
actually
stay
in
that
queue
right,
so
this
will
enable
that
particular
scenario.
B
Fundamentally
we
want
to
actually
do
is
for
service
X.
We
need
Y
percent
of
the
traffic
go
into
the
X
cannery,
respecting
the
request
context.
Why
we
need
to
what
to
see
mean
by
request
context
here.
So
just
give
you
an
example:
if
I
am
actually,
as
a
user
I
send
a
request
into
your
stack,
say:
I
want
to
get
all
the
items
right
and
then
item
service.
It
is
actually
doing
some
sort
of
an
experiment
for
now
is
rowing
out
a
new
amp
new
item
service
version.
You
really
what
you
really
want.
B
Is
you
want
for
me
for
a
particular
user?
You
neither
actually
stick
with
a
production
or
canary.
You
don't
want
him
to
actually
jump
across
every
single
time
they
refresh
they
would
get
different
m1.
That's
the
turbo
experience,
that's
what
I
mean
by
context-sensitive
experiment,
so
we
actually
implement
this
another
plug-in
actually
in
link
ID.
What
it
does
is
using
murmur
literature
using
murmur
hash
to
actually
shut
a
user
ID
or
some
sort
of
context
that
inside
the
request
and
ensure
a
particular
user,
always
stick
with
a
particular
service
step.
So
that
means
it's
like.
B
This
is
a
lay
layer
to
actually
control
our
based
on
some
sort
of
context.
So,
as
you
can
see
here,
this
is
actually
the
high
level
overview
for
that,
so
a
user
request
comes
in
it
get
into
your
edge
and
then
your
edge
actually
somehow
actually
synchronize,
with
some
sort
of
conflict
in
cs3
that
particle
convex
specific
light
for
particular
service.
X
percent
go
through
this.
Each
person
go
to
canary
wipers
angle,
to
canary
we
ban,
etc.
B
B
So
you
can
you,
actually.
We
actually
allows
you
to
actually
update
the
s3
file
on
the
fly
and
then
the
routing
edge
actually
periodically
refreshed
the
professor
value
from
the
s3.
Usually,
it
actually
pick
it
up
like
in
in
30
seconds
and
that
will
allow
you
to
distribute
the
traffic
so
Lincoln
di
enabled.
This
particular
scenario
gives
you
a
very
nice
programming
interface
to
control
traffic
up
to
tak,
pernah
realities.
B
Okay,
this
is
actually
what
are
what
you
look
like
inside
the
inside
the
edge
so
on
the
high-level
yell
become
yell,
be
the
requests
coming
from
miyabi
and
then
the
requests
going
into
nginx
and
engine
makes
normalize
the
yo
IO
and
forward
that
to
link
ID
a
link.
Id
actually
has
couples
of
plucking
the
first
one,
acting
as
a
filter
and
apparently
the
second
one,
acting
as
a
transformer
and
and
then
after
Lucas
is
done.
They
deliver
to
some
sort
of
an
endpoint.
B
The
endpoint
can
be
service,
another
another
set
of
service
or
it
can
be
actually
a
API
carry
which
is
mounting
a
service.
A
service
attack,
so
here
are
pros
and
cons.
The
pros
for
that
is.
We
now
actually
have
a
horizontal
scale
ready
step,
so
you
just
add
more
boxes
when
the
traffic
comes,
it
gives
you
effects
ability
it's
very
flexible,
because
engine
mix
is
super-powerful.
B
Linga
T
is
super
programmable
and
then
you
get
the
best
of
both
world
and
you
have
to
have
food
observability
from
Inca
tea
as
well.
What's
the
con
was
a
Tang
sign
for
that
Papa.
You
may
actually
ask
we
are
doing
double
passing,
which
is
a
request
going
into
nginx
and
then
going
into
Ranga
D.
So
what's
a
trade-off
for
that,
we
actually
waste
50%
of
the
CPU
so
well.
B
This
is
actual
trade-off
that
we
have
for
this
kind
of
accessibility.
It's
because
the
number
of
the
H
boxes
that
we
have
is
actual
range
from
3
to
30
through
the
day,
because
our
traffic
is
such
a
very
time-sensitive
so
and
we
actually
use
like
50,
paisa,
wondrous
and
more
of
the
of
the
CPU
there,
but
giving
this
flexibility
and
giving
the
capacity
that
we
have.
We
actually
decide
that
this
is
actually
a
good
trade-off
for
us.
B
A
B
To
be
able
to
enable
engineer
to
use
the
best
tool
for
the
best
purpose,
what
does
it
mean?
A
usually
means
is
like
me
to
achieve
support
a
lot
of
languages.
Okay,
not
all
the
language.
We
we
end
up
actually
design.
We
support
Java,
Python
and
go.
We
want
to
actually
support
different
kind
of
framework
that
each
of
language
actually
really
good
there.
We
want
to
support
different
kind
of
messaging
protocol,
specific
rest,
HTTP,
v,
1
and
v,
and
then
now
the
challenge
actually
for
that.
Is
it
in
this
very
hybrid
or
heterogeneous
environment?
B
The
challenge
actually
for
us
as
a
as
a
platform
engineering
is
it's
super
challenge
to
to
really
actually
manage
all
these
or
all
these
different
kinds
of
traffic,
because
if
any
of
you
have
been
actually
paying
with
java
or
playing
with
python,
then
you
actually
know
how
difficult
really
getting
the
connection
pool
actually
currently
set
up
right
in
the
specific
framework.
That's
usually
is
a
mission
impossible
and
if
you
x
like
like
the
number
of
the
framework,
the
number
of
the
language
itself,
that's
actually
pretty
much.
It's
not
a
very
enjoyable
work.
B
Lindy
is
actually
deep
oak,
as
we
actually
mentioned.
Our
service
tag
is
actually
peel
is
actually
on
ec2
boxes
at
that
point,
so
we
actually
deploy
link
that
the
SSI
car,
as
you
can
actually
see
that
from
from
this
graph
here
from
as
from
an
application
standpoint,
the
application
has
no
external
touch
point.
B
That
means
doesn't
the
all
the
traffic
actually
getting
into
the
application
is
from
local
host,
which
is
from
the
alinker
desai
car,
because
every
single
traffic
actually
getting
into
the
box
always
catch
through
the
link,
ID
and
link
ID
actually
forward
into
the
applications,
and
when
the
application
need
to
actually
reach
out,
and
then
it
we
don't,
we
have
the
firewall.
We
don't
allow
the
application
to
actually
reach
out
I,
really
it
has
to
actually
go
through
the
link
ad
itself.
B
Why?
Because
this
way
we
have
a
fairly
consistent
connor
shampoo
management
policy,
and
we
have
a
really
good
soccer
braking
and
we
have
four
hours
of
ability
well
in
capacity
voluntary
HIV
pro
by
and
also
when
we
have
back
pasture,
which
means
by
pressure,
really
means
it's
like
if
some
of
the
service
actually
like
at.
B
B
B
As
you
can
actually
see,
I
sense,
you
guys
see
here.
Linka
D
is
actually
running
as
a
side
car
inside
the
box,
and
we
also
have
an
answer.
You
see.
This
is
up.
Another
box
theses
are
not
the
plucking
that
we
actually
write
will
actually
go
into
detail
later.
Oh
by
the
way
you
see
that
on
the
egress
point
here,
we
have
also
have
an
experiment
designed
that
what
does
it
mean?
B
So
that
means,
if
you
say,
for
four
percent
of
the
5
percent
of
the
traffic
go
to
the
new
item
service
and
then
because
your
item
service
has
to
theta
2
main
source
of
the
traffic
coming
from
right.
First
part
et
cetera
from
external.
The
second
part
is
from
internal
and
by
having
a
Yi
quest
in
the
US.
We
actually
install
our
plug-in
in
that
way
that
were
actually
enable
us
to
actually
doing
the
robot
even
for
internal
traffic's,.
B
So
we
just
finished
the
overview
about
of
what
the
service
machine
look
like
and
in
specific
I
want
to
actually
talk
to
one,
not
a
very
fun
topic:
how
service
and
service
talk
to
each
other
I
named
this
part.
As
for
this
curry,
we
actually
have
two
stage:
we
we
evolve
our
study
into
phrase.
One
is
discovery
v1
and
the
other
part
is
scared.
Me
too,
let
me
get
into
the
detail
here.
B
So
discovery
v1,
it's
fairly
simple,
this
one
the
diagram
actually
looks
much
complicated
than
the
actual
solutions.
It
is
actually
this
part
we
actually
use
console.
So
we
step
statically
at
your
register,
the
service
in
the
console
by
the
EOB
name.
So
on
the
high
level,
what
is
Carrie
do
what
discovery
actually
does
is
to
mapping
from
a
service
to
either
host
or
either
DNS
or
either
a
bunch
of
IP
right.
So
this
is
why
it
does
we
actually
statically
actually
register
the
service
in
the
console.
B
So
we
have
a
name
mapping
to
a
particular
ERP
and
with
the
help
of
with
the
help
of
name
ID,
every
single
requesters
are
coming
in
here
and
nemeth
e
will
actually
tell
the
link
ID
the
IP
address
of
of
a
yuppie
okay,
and
then
the
request
has
to
go
from
there,
as
you
can
actually
see.
That
is
now
for
every
single
service
talk
to
the
other
servers
you're,
always
going
through
a
particular
yarby.
B
B
Because
the
kind
always
actually
catch
the
DNS
and
whenever
you
have
caching
and
then
your
link
meteors
talk
about
your
vacation,
talk
about
all
the
other
stuff
that
is
fairly
complicated
and
also,
more
importantly,
a
lot
of
the
very,
very
fancy
or
matured
feature.
It's
actually
a
whole
space.
What
does
it
mean?
Retry
projects
right
manner?
We
try
to
my
never
hold
such
study
fail
and
then
we
retry
it's
based
on
a
host
and
sucker
braking
is
based
on
a
host.
B
So
if
you
want
to
the
richer
budget
or
to
the
circling,
Soaker
breaking
actually
on
a
yuppie
in
that
sense,
anything
goes
wrong
and
then
you
shut
down
the
particular
hobby.
You
lost
the
content
of
the
old
holes
behind
behind
the
particle,
a
yuppie
which
is
not
a
good
idea,
and
that
makes
your
belief
control.
You
become
a
single
point
of
failure.
B
How
are
we
gonna
fix
it?
Discovery
to
this
current
video
is
SSR
the
base.
We
use
the
SRA
based
routing,
it's
a
jumper.
No,
it's
a
to
appear
to
appear.
What
does
it
mean?
We
go
around
the
Yuppie,
we
don't
you
say,
yeah
be
anymore.
We
actually
enable
host
to
talk
to
the
other
host
tyranny
on
the
fly,
and
fortunately,
lingered
II
can
actually
do.
The
fell
is
in
fairly
easily
because
Venango
actually
support
given
kind
of
low
balancing
algorithm
is
very
mature.
B
Let's
get
to
what
s
our
bill
is
SRV,
it
is
a
kind
of
DNS
record.
You
can
consider
that,
as
actually
as
a
seeming
for
a
group
of
hosts
I
think
every
every
DNS
provider
now
where
they
actually
support
that,
particularly
we
actually
use
in
a
raffle
history
in
each
of
the
SRV
record.
I,
remember
that
will
allow
us
to
support
up
to
200
host.
So,
basically,
if
you
have
a
service,
you
give
a
DNS
name
of
a
particular
service.
B
B
So
this
is
how
it
works,
because
every
single
issue
box,
we
call
that
a
note
know
automatically
actually
announce
to
a
fleet
EP
fleet.
Eb
is
a
database
that
we
implement,
that
in
implement
that
using
a
dynamo
to
actually
implement
the
fleet
EP
with
some
sort
of
decay
and
fleecing
service.
I
children,
listen
to
the
fleet,
D
be
using
dynamo
stream
and
then
register
the
DNS.
B
The
new
coming
DNS
interrelate
history
and
peat
register
things
that
has
been
the
oil
already
expire
and
we
actually
implement
a
SRV
record
resolver
in
the
name
of
the
Sony
MIDI,
now
understand
as
a
via
record
and
whenever
a
request
coming
in
to
a
link,
ID
link
ID
dedicate
the
resolution
to
name
of
the
name
ID,
give
it
link
ID
back
a
list
of
the
host
and
link
ID,
actually
do
the
low
balancing
locally
and
the
fleecing
service.
The
thing
that
in
the
in
the
yellow
box,
he
actually
what
it
does.
B
It
is
actually
free
form
service
that
a
children
running
in
lambda
and
we
periodically
actually
wake
up
that
particular
lambda.
He
actually
used
that
to
actually
monitor
the
flea
TB
and
listen
to
ec2
event
when
I
used
to
box
actually
died,
and
then
he
tried
to
remove
that
entry
and
he
also
tried
to
periodically
rip
rip
off
the
outdated
informations
in
this
design.
B
Obviously,
one
of
the
key
thing
that
we
actually
trade
off
a
teacher,
the
consistency
so
basically,
whenever
at
any
point
of
the
time
when
you
actually
get
back
a
list
of
the
IP
that
you
want
to
balance
to,
we
think
it's
possible
that
some
of
the
IP
may
be
already
outdated
for
the
particular
service,
which
is
fine
in
our
case,
because
Lindy
actually
is
a
low
balancer
itself.
He
knows
how
to
when,
whenever
they
initially
thought
were
not
actually
dead,
they
were
actually
had
to
rotate
to
another
one
and
the
actually
maintain
the
balance.
B
Other
the
low
balances
said
actually
in
memory
and
availability
is
actually
strong
and
felt
horrid
is
strong.
Thanks
to
we
actually
use
the
cloud-based
solution
like
route
53
dynamodb.
So
we
don't
need
to
maintain
those
infant
structures,
and
particularly
in
this
way,
we
no
longer
need
to
actually
maintain
the
console
so
because
console
is
a
very
good
product,
but
just
until
that
it
failed
a
master
and
then
get
into
the
master
election.
Then
that's
another
nightmare.
We
no
longer
have
those
kind
of
problem.
B
The
graph,
the
graph
it
is,
such
as
a
new
connection
can
so
before.
After
so
before,
we
deploy
the
service,
and
then
you
can
see
that
we
have
a
lot
of
new
connections
after
we
actually
deployed
the
service.
We
can
actually
see
that
the
new
whole
nation
actually
dropped
significantly
because
linka
d2
lincolni
collisions
is
actually
much
more
well
behaved
than
Lincoln
t2y
lb
and
year
beetling
kadhi,
so
yeah
be
as
you
can
remove
it,
and
then
we
have
a
much
more
stable
condition
pool
in
some
sense.