►
From YouTube: Service Meshes: The Production Readiness Checklist for the Rest... - Zachary Arnold & Austin Adams
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Service Meshes: The Production Readiness Checklist for the Rest of Us - Zachary Arnold & Austin Adams, Ygrene Energy Fund
At KubeCon Austin in 2017 during the Keynote sessions it was declared: "2018 is the Year of the Service Mesh." At Ygrene, this has been a true statement. SM's have been a great evolution in service-to-service communication. However, the long and short of it is, these tools are not straightforward to simply drop into our clusters and watch them magically work for us. Because of this, we evaluated several service mesh tools using a rubric of our company's needs, and developed a production readiness checklist that would be relevant for any service mesh. We'll share with you our evaluation rubric and production readiness checklist which will hopefully aid you on your journey to better service-to-service communication.
To learn more: https://sched.co/Grag
A
So
it's
1:45,
so
I
guess
we'll
get
started.
Hi
welcome
to
service
meshes
the
production,
readiness
checklist
for
the
rest
of
us.
We
called
it
the
checklist
for
the
rest
of
us
because
we're
targeting
small
but
innovative
teams
who
have
the
same
problems
as
the
big
teams
with
armies
of
people
to
solve
all
of
these
infrastructure
and
networking
problems.
A
A
We
we're
targeting
you
because
we
know
that
you
don't
have
armies
of
people
to
dedicate
to
this
problem,
and
we
know
that
your
time
matters
a
lot
to
you
and
we
know
you
don't
have
a
million
test
environments
and
a
billion
Google
credits
to
do
all
of
your
stuff
with.
So
that's
where
this
is
for
you're,
probably
already
managing
some
services
and
production
you're,
probably
already
experiencing
the
pain
that
comes
with
having
a
few
people
managing
lots
of
services.
A
Maybe
you
even
bit
into
micro
services
a
little
too
fast
guilty,
but
now
you're,
seeing
the
wonderful
world
that
is
micro
services
and
all
the
benefits
that
come
with
them,
but
also
some
of
the
pain
and
thank
you
for
your
questions.
They're
just
rolling
in
hopefully
I'll
finish
quick
and
we
can
get
to
Zacks
part
and
get
to
some
questions
who
this
is
not
for
is
you
know,
people
holding
PhDs
and
networking,
and
probably
some
of
you
aren't
already
in
production
with
your
service
mesh.
But
if
you
are
here,
you're
obviously
welcome.
A
But
you
might,
you
might
take
an
opportunity
to
go.
Take
a
nap
in
that
dark
corner
over
there.
So
without
further
ado,
let's
let
the
load-balancing
begin
so
I'm
Austin
Adams.
This
is
Zach
Arnold.
We
work
at
a
company
called
Y
Greene.
We
are
a
financing
company.
We
do
specialized
financing
using
the
equity
in
a
building
to
provide
clean
energy
upgrades.
It's
really
cool,
really
legal.
A
No,
it
is
legal.
Sorry,
there's
a
lot
of
legal
we've
done
over
a
billion
dollars
of
financing
in
about
six
years.
Every
second
of
downtime
is
about
a
hundred
and
forty-four
dollars
of
lost
business.
So
anytime,
we're
down
our
cio
is
just
like.
When
are
we
going
to
be
back
up,
want
to
be
back
back
up,
so
many
of
you
will
probably
have
the
same
problem.
We
have
eight
time
full-time
engineers
and
loads
of
contractors,
just
loads
of
them,
and
so
the
eight
to
fight
full-time
engineers
take
all
the
ownership
of
what's
happening.
A
A
And
we're
hiring
more
of
them.
So
please
come
see
us.
We
have
an
office
in
Seattle
and
you
get
to
work
with
this
man
over
here.
Okay,
so
moving
right
along.
So
let
me
quickly
define
service
missions
for
you.
If
you
don't
already
know
they
are
a
dedicated
piece
of
infrastructure
surrounding
your
network,
abstracting
it
for
you,
so
that
you
don't
have
to
think
about
it.
Except
for
that
you
do
with
every
service
mesh
I
want
to
read
this
slide.
Get
bigger
comes
a
proxy!
That's
pretty
much
the
common
denominator
between
any
service
mesh.
A
It's
a
TCP
proxy
that
wraps
your
services
and
you
know,
captures
the
traffic
there,
so
it
can
make
it
more
reliable.
It
can
add
other
features
like
flow
control,
security,
stuff
authorization
policy
can
do
some
cool
routing
and
you
get
it's
a
great
place
to
chuck
in
some
observable
and
metrics
and
tracing.
So
that's
you
know
just
at
a
very
in
a
nutshell
what
it
is
so
there's
lots
of
service
messages,
there's
actually
there's
probably
a
new
one,
born
right
now
and
now
another
one
another
wrapper
around
envoy.
A
A
lot
of
people
are
doing
that
are
making
their
own
envoy
management
layers
just
going
through
the
landscape.
Really
quick,
we've
got
linker
D
one
and
two,
both
great
products
and
the
linker
d2
team
is
really
rad
I
like
them.
A
lot
you've
got
sto,
which
I
know
none
of
you
have
heard
of,
but
it's
actually
a
pretty
good
product.
A
It's
coming
out
of
Google
e
and
IBM,
and
you
can
wrap
your
own
proxy
there
in
Gen,
X
and
envoy
are
the
most
popular
ones,
there's
aspen
mesh,
which
is
a
managed,
sto,
a
sure,
Oh
aw
sap
mesh,
which
managed
envoy
as
your
service
fabric
mash.
Please
don't
hurt
me
if
I
get
this
wrong,
but
it's
basically
just
an
sdk.
You
put
into
Java
and.net
projects
that
manages
a
connection
in
between
the
fabric
of
services
fabric.
A
Then
you
got
gke
manage
dis,
do
which
I
think
that's
the
thing
right
now,
I,
don't
know.
If
it's
a
publicly
available
thing,
we
definitely
didn't
get
access
to
it,
but
probably
some
of
you,
cool
companies
did
and
then
there's
index
plus,
which
don't
forget
about
engine
X,
they're,
really
cool.
You
can
pay.
Somebody
they
can
help
you
out
that
are
really
cool.
A
So
for
us
a
case
study
side
of
things.
We
took
a
four
step
process
to
getting
a
service
mesh
into
production.
The
four
steps
that
we
followed
were
assess,
select,
commit,
implement
and
release.
So
I'll
cover
the
assess,
selecting
commit
the
manager,
II
portions
and
then
the
actual
work
will
be
covered
by
Zack,
because
he's
the
only
person
that
actually
does
work
in
our
company.
A
A
So
you
need
to
POC
these
things
out.
One
key
two
key
questions:
I
think
that
everyone
should
ask
himself
after
they've
asked
that
they
need
a
service
mashes.
Can
our
team
handle
the
complexity
of
the
service
mesh?
Maybe
you
really
want
one,
and
it's
just
like
the
microservices
question:
can
we
really
handle
this?
Do
we
actually
need
this?
Ask
yourself
those
questions.
I
think
the
CNC
F
is
cool
enough,
where
no
one's
going
to
judge
you.
If
you
don't
have
a
service
mesh
I
mean
they
might
look
at
their
shoes
and
decide.
A
They
don't
want
to
talk
to
you
anymore,
but
maybe
they
won't
judge
you
and
can
your
application
handle
the
service
mesh?
You
wouldn't
believe
that
the
weird
stuff
that
happens
in
applications
and
then
you
wrap
it
in
a
proxy
and
all
of
a
sudden.
It
just
doesn't
work
anymore.
That's
actually
happened
to
us
and
we
had
the
instrument
and
change
a
few
things
in
in
one
of
our
applications
to
actually
make
it
work.
A
So
service
meshes
have
a
lot
of
different
use
cases.
The
most
important
one
for
us
was
the
security
stuff,
so
we're
a
financial
company.
We
have
a
lot
of
financial
data
on
people
and
we
that's
our
top.
Priority
is
keeping
that
stuff
secure
Zack
likes
to
call
it
our
don't
be
Equifax
policy,
that's
not
a
shot
again.
X
against
Equifax
I
know
they
work
really
hard.
That's
just
a
joke.
I
think
I
had
to
say
that
for
legal
reasons.
A
So
as
you
can
see,
this
is
a
very
complicated
architecture,
diagram
that
took
us
months
to
create,
and
it
accurately
represents
some
of
the
services.
You
know
the
red
ones,
the
super
secure
one.
We
want
to
make
sure
that
the
gold
one
never
talked
to
the
red
one,
and
we
want
to
enforce
that
in
a
place.
That's
easy
to
enforce.
You
know
you
can
get
really
crazy
with
AWS.
I
am
permissions
and
blah
blah
blah.
A
But
you
know
service
meshes,
make
that
a
great
place
to
actually
implement
that
policy,
because
it's
right
there
at
the
outgoing
connection,
the
actual
event
level.
So
for
us
we
wanted
to
make
sure
that
everything
was
encrypted
everywhere.
We
wanted
to
uphold
that
standard
through
our
adoption
of
kubernetes,
and
you
know
we
were
manually
managing
all
of
that
stuff.
A
One
of
the
numbers
on
our
feature
map
and
gets
put
into
place
along
with
all
the
others,
because
we
want
to
make
sure
any
decision
that
we
make
is
going
to
be
stable
in
a
long
term
as
much
as
we
can
help.
So
we
look
at
the
open
source
project
from
the
from
the
areas
of
how
long
it
has
it
has.
It
existed,
how
popular
is
it
in
terms
of
contribution
and
usage?
How
well
sponsored
is
it
like?
Maybe
IBM
or
Google
is
behind
one
of
them.
A
You
know
just
for
example,
and
you
know
how
recently
has
it
changed
you
know,
have
they
has
a
contribution
died
off,
you
know
so
at
Y
green,
you
know
when
we
have
to
use
open
source
technologies.
That
pains
me
to
say
it,
because
we
have
such
a
small
team
week.
We
favor
manage
services,
but
we
always
end
up
a
lot
of
times.
A
We
end
up
having
to
roll
our
own
because
of
encryption
requirements
and
security
requirements
and
Ben
always
tells
us
we're
wrong,
because
you
know
he's
like
mega
hacker
and
he's
like
this
is
the
five
ways
it'll
get
hacked
so
yeah.
So
our
advice,
our
quick
advice
and
then
we'll
tell
you
what
service
mesh
we
chose.
A
So
if
you
are
a
small
team
that
just
needs
a
service
mesh
for
reliability
and
tracing,
and
things
like
that,
try
out
link
your
d2,
our
recommendation,
but
if
you
need
a
service
mesh
for
almost
anything
else,
you
should
probably
check
out
this
to
you
and
don't
ignore
linker
d2
our
link,
Rd
one,
that's
been
out
there
for
quite
a
while.
It's
pretty
pretty
solid
and
you
can.
You
can
do
em
TLS
with
that.
So
without
further
ado,
I'll
show
you
what
our
choice
was.
A
B
B
The
other
thing
is
in
order
to
get
to
your
questions
which
I
want
to
do.
I
will
have
to
speed
through
this
next
part,
so
you
may
have
to
watch
this
on
youtube
and
slow
it
down
to
like
0.25
speed.
Actually
now
you
guys
will
get
it
you're
awesome,
okay,
so
we
did
this
do
awesome.
After
talking
to
some
of
you
this
week
we
changed
this
part
of
the
top
you're
like.
B
Oh,
you
know
what
we're
not
addressing,
where
we
are
they're,
not
being
forthright
enough
or
we're
not
actually
addressing
everyone's
real
concerns
so
scrapped
it
and
start
it
over,
and
so
now
what
we're
talking?
What
we're
going
to
talk
about
is
shoehorning
is
do,
or
at
least
the
envoy
sidecar
proxy
into
your
app
and
getting
engineer
buy-in
and
what
that
looked
like
for
us
at.
B
Why
green,
the
other
bits
and
pieces
of
sto
that
don't
work
in
eks,
yet
we
are
on
Amazon
we're
not
in
Amazon
in
production,
but
you
can
okay
and
and
then,
of
course,
how
we
got
into
prod
just
so
we're
absolutely
clear.
We
picked
this
do
for
security,
so
everything
I'm
talking
to
you
about
we
picked
for
security.
Canary
recent
releases
are
really
cool,
but
if
you
roll
out
5%
of
your
financing
that
gets
underwritten
with
one
criteria
and
95%
that
gets
underwritten
with
another
you
get
talked
to
by
the
SEC.
B
No
one
congratulate
you
for
having
a
really
successful
rollout
of
two
different
type
of
criteria,
so
blue
green
releases
are
something
that
we
have
to
do.
It
has
to
be
a
hundred
percent
one
way
or
the
other.
All
this
exciting,
advanced
traffic
routing
wasn't
really
important
to
us.
Currently
in
the
mortgage
industry.
The
average
underwriting
time
is
two
to
three
days
for
us.
We
go
through
a
very
similar
process
for
them
as
the
mortgage
industry,
and
it
takes
us
two
minutes,
so
it
goes
from
geologic
time
to
understandable
in
human
people.
B
When
we
get
to
sub
millisecond
latency,
then
we
will
definitely
tell
you
way
more
about
all
the
other
awesome
features
about
on
voice.
So
just
a
level
set
like
if
you
were
like
I'd
like
to
know
about
how
any
way
that
we
probably
won't
be
able
to
answer
with
a
lot
of
clarity
right
now.
Okay,
are
we
cool
everyone?
Good
could
move
on
awesome,
implementing,
not
just
the
demo
app
the
HTTP
bin
app
that
we've
all
seen
on
the
ISTE
o
page?
Sometimes
it
doesn't
even
work
yes,
this
is.
B
We
did
the
literal
opposite
of
what
we
heard
from
the
stripe
talk
and
so-
and
that's
actually
been
our
experience
with
all
CN
CF
projects.
We
have
this
one
huge
thing
in
our
environment
called
manager
and
manager
is
this:
is
this
Ruby
on
Rails,
amazing,
API,
Orchestrator
I,
say
amazing,
a
little
bit
tongue-in-cheek?
It
pays
the
bills,
but
obviously
we
are
really
actively
trying
to
get
stuff
out
of
it
fast,
but
it
takes
about
two
minutes
to
startup
it's
a
five
gigabyte
docker
file,
our
docker
image.
B
B
We
terminated
SSL
at
the
load
balancer
level
and
for
the
brief
bit
where
stuff
was
traveling
through
the
Amazon
V
PC
to
the
node
before
it
was
reinterred
in
talking
between
the
rest
of
the
services.
That
was
technically
plain
text.
We
wanted
to
do
better
for
our
customers,
because
we
really
love
encryption
and
we
think
that
your
data
is
worth
protecting,
so
the
app
also
has
all
the
other
peripherals.
There
are
party
services,
RDS
Redis,
the
gambit.
B
You
know
we
manage
a
huge
AWS
bill,
so
you
can
guess
that
there's
probably
a
lot
of
AWS
services
of
use.
It's
also
the
app
that
we
lovingly
call
the
edge
case
factory,
because
you
would
not
believe
the
number
of
things
that
this
has
literally
caused
us
in
production
alone,
so
the
rest
of
our
service.
So
so
sorry,
we
did
that
first,
once
that
thing
started
working,
we're
like
holy
God,
this
could
work
for
us.
Sto
might
be
a
thing.
B
We
might
be
like
a
cool
company
here
soon
finance
my
thing
so
the
rest
of
our
services
inherit
from
a
common
base.
So
we
don't
run
a
polyglot
anymore,
or
at
least
we're
almost
all
the
way
out
of
a
polyglot
in
production
right
now,
originally,
we
were
kind
of
like
bring
your
own
language,
bring
your
own
framework,
and
then
we
were
like.
Oh
my
god,
this
is
a
nightmare.
B
So
everything
comes
from
a
common
base
which
is
really
awesome
because
pushing
a
config
update
to
the
upstream
everything
inherits
downstream,
and
that
was
how
fast
it
was
to
get
dev
bian.
They
didn't
even
notice
it.
They
can
keep
working
on
their
apps
and
I.
Don't
have
to
lose
sleep
at
night
when
I
get
paged
so
or
when
I
don't
get
paged.
So
that
dev
bian
was
simple.
B
This
may
not
be
the
case
for
a
lot
of
you
guys,
but
for
us
it
actually
was
really
great
to
have
a
common
base
to
push
everything
to
and
then
inherit
from
all
right,
implementing
a
suggested
strategy.
Again,
if
you
are,
if
you're
doing
for
security,
this
would
be
useful.
I
don't
have
memes
I
have
architecture
diagrams,
although
I
like
to
be
the
funny
one
between
Austin
and
I.
So
this
is
ironic
for
me.
If
you
can
use
helm,
helm
has
a
really.
It
was
made
by
the
ISTE
of
people
they're
pretty
smart.
B
They
know
how
their
product
works.
If
you
really
want
to
spit
out
all
of
the
ammo
files
and
understand
all
that
components,
that's
terrific
for
us
not
necessary
the
more
hell
meanest,
the
better.
Using
your
selected
features
for
us.
We
knew
we
were
going
to
have
to
focus
on
Citadel
Citadel
Citadel,
which
is
ISTE
o--'s
CA,
slash
all
the
identity
and
M
TLS
eNOS
that
gets
distributed
to
the
proxies.
So
we
highly
recommend
working
with
instrumenting
metrics
early.
B
It
will
save
you
time,
diagnosing
problems
in
the
long
run
and
oh
for
those
of
you
that
are
in
eks,
we
apologize,
but
there
was
a
lot
we
had
to
turn
off
at
the
time,
I
told
that
this
is
different.
Now
it
still
isn't
for
us
at
the
time
admission
controller,
validating
webhooks,
which
is
how
sto
validates
config
did
not
exist.
B
We
don't
use
it
in
prod,
but
we'll
see
more,
we'll
see
more
fun
stuff
with
eks
in
just
a
moment
implementing
moving
on
to
the
metrics,
we
used
our
own
Prometheus
plug
unit
into
mixer,
our
own
fluid
D,
our
own
Gravano
Jaeger.
Turning
this
on
was
just
one
switch
in
the
helm.
Chart
incredibly
useful
for
debugging
the
service
ingress.
I
cannot
say
enough
great
things.
B
Now
we
don't
have
the
header
respecting,
so
we
can't
see
all
the
traces
throughout
the
rest
of
our
application
yet,
but
for
getting
traffic
into
the
very
first
place,
which
was
that
huge
manager
app
we
were
just
talking
about.
This
was
like
paramount
key
at
2:00
a.m.
when
I'm
crying
and
holding
my
teddy
bear
I
read
I
was
like
oh,
my
got
a
Mis
name
to
the
port
and
it
was
attempting
to
do
something
it
shouldn't
have.
B
Oh,
that
was
in
the
documentation
moving
on
to
then
now
so
we've
we've
we
bootstrapped
sto
we've
got
our
metrics
in
now,
it's
time
to
start
shoving,
proxies
in
and
praying
so.
You
inject
the
sidecar
into
your
service
or
services,
your
call
and
then
start
creating
virtual
services
for
the
ones
that
need
to
be
routable
externally.
B
So
virtual
services
allow
sto
to
take
host
headers
that
are
presented
to
the
ingress
gateway
and
turn
them
into
places
where
traffic
and
flow
to
inside
of
the
side,
cars
a
relatively
simple
concept,
but
is
easy
to
kind
of
mess
up
and
I
would
also
say.
This
is
where
we
saw
the
most
problems
week.
We
very
recently
solved
a
production
error
where
there
was
random
503
s
happening,
and
it
was
because
kubernetes
was
rescheduling.
B
Pods
and
stuff
would
just
trip,
and
this
guy
would
get
paged
and
he
would
tell
me
lots
of
really
French
words
and
and
and
as
it
turns
out,
there
was
actually
retry
logic
that
we
could
let
sto
do.
That
would
eventually
allow
this
traffic
to
flow
through
which
it
also
does
exponential
back-off.
So
you
don't
get
thundering
herd.
It's
amazing,
ok,
next
thing,
so
you've
got
traffic
flowing
in
and
all
of
a
sudden,
your
app
doesn't
start.
Why?
B
Because
it
can't
talk
to
anything
why
not
sto
has
a
secure
by
default
setting,
because
obviously
this
isn't.
This
is
the
kubernetes
of
networking.
Of
course
it
would
give
you
a
billion
switches
just
enough
rope,
hang
yourself.
So
the
the
next
thing
that
we
had
to
do
is
whitelist
all
outbound,
HTTP
and
TCP
traffic
to
third-party
vendors.
So
your
RDS
instances,
your
ElastiCache
or
Redis
or
memcache
D.
B
That's
going
to
be
important
for
us
credit
report,
vendors,
anything
that
has
an
addressable
thing
that
you'd
like
to
see,
leave
your
proxy
and
eventually
return
as
a
round
trip
request
that
will
have
to
be
configured,
and
so
you
can
see
the
red
arrows
that
point
to
the
things
that
you
will
definitely
have
to
configure.
You
can
configure
an
egress
gateway.
If
you
would
like
to
get
more
tracing
information
wasn't
necessary
for
us
V
PC
endpoints
will
screw
you.
If
you
are
not
paying
attention.
B
Eks
is
the
Amazon
uses
the
Amazon
CNI,
which
is
based
on
calico,
but
the
Amazon
CNI
at
like
layer,
2.5
ish
to
between
2
and
3
uses.
The
V
PC
is
the
transit
mechanism.
Ok,
what
does
that
mean?
That
means
that
there
is
an
elastic
network
interface
bound
to
every
single
node
for
every
single
pod.
So
it's
addressable
inside
of
your
V
PC.
Well,
guess
what
else
has
a
private
IP
address
everything
else?
B
That's
private
inside
of
your
V
PC,
and
if
your
cluster
happens
to
share
the
same
V
PC,
because
you're
concerned
about
security,
then
sto
is
going
to
try
to
broker
the
connection
between
the
two
entities
and
one
is
in
the
mesh.
One
is
not,
and
so
a
certificate
presented
by
kms,
that's
signed
by
Amazon
CA
is
not
one.
That's
signed
by
the
Citadel
CA,
which
is
inside
of
your
cluster
and
immediately
you'll,
see
that
explode,
which
was
kind
of
crazy
for
us
and
was
one
of
the
more
aggravating
things
to
debug.
B
We
still
cool
everyone
follow
me
all
right
awesome.
We
are
almost
releasing
ok,
so
you
have
sworn
underneath
your
breath
in
amazon
for
some
interesting
things
in
the
cni
and
then
you
are
now
getting
ready
for
deployments.
You
can
use
sto
destination
rules
or
kubernetes
services.
If
you
want
to
do
fun
things
for
routing
again,
finance
company
security,
not
cool.
If
people
get
loans
that
are
underwritten
with
two
different
criteria,
so
traffic
splitting
here's,
a
diagram
of
how
it
works.
Documentation
is
great.
B
Obviously,
security,
provisioning,
ssl
certificates
for
public
domains
that
you
want
to
be
routable.
Inside
of
your
mesh
early,
we
switched
from
using
Amazon
ACM,
which
was
terminating
connections
at
the
load
balancer
and
instead
used
an
elf
for
proxy.
All
the
way
into
our
ingress
gateway
and
use
jet
stack.
Jeff
stack
is
awesome.
It's
like
if
kubernetes
and
certificates
got
together
and
made
sure
that
you
never
got
an
accidental
certificate
in
validation,
error.
Again,
that's
beautiful,
so
unreleased
night.
B
We
have
the
benefit
of
being
again
a
financial
institution,
which
means
we're
allowed
to
have
downtime
I
tried
really
hard
to
make
sure
that
we
didn't
and
we
did
not,
which
was
awesome.
It
says,
because,
obviously
then
this
would
not
be
a
worthwhile
talk
for
you
guys,
but
we're
allowed
to
basically
have
eight
hours
of
downtime
after
9:30
p.m.
Pacific,
which
is
like
probably
insane
to
some
of
you,
people
which
sorry
money
does
sleep
for
us
at
night,
so
switching
DNS
over
it's
an
alias
record
so
that
usually
switches
over
in
seconds
internally
in
Amazon.
B
We
gave
it
about
15
minutes
before
we
went
ahead
and
terminated
the
old
load
balancer
and
switched
the
service
object
to
a
cluster
IP
so
that
it
didn't
try
to
rebound
sir,
and
so
now
we
see
traffic
flowing
in
and
the
ingress
gateway.
We
went
over
to
go
to
the
next
slide
and
switched
the
NT,
the
destination
rule
to
doing
sto
mutual
and
the
policy
in
the
mesh
to
be
permissive.
So
all
of
the
services
will
send
TLS
traffic,
but
if
there
are
lingering
requests
or
DNS
cache
somewhere
else,
it
can
still
receive
plain
text.
B
One
last
little
bit
after
we
were
reasonably
satisfied
that
that
was
no
longer
the
case.
We
switched
the
policy
to
strict.
We
run
all
of
our
checks
and
we
prove
that
we
are
now
fully
encrypted
all
the
way
through
every
single
possible
place
in
our
cluster,
with
no
configuration
or
respect
for
that
in
any
of
our
applications,
which
is
beautiful.
B
The
final
step
for
us,
obviously,
as
Austin
was
saying,
and
his
beautifully
done,
architecture
diagram.
We
can
all
thank
Austin
for
that.
Well
done,
you
obviously
don't
want
two
services
talking
to
each
other.
If
one
secure
one
is
less
secure,
like
you
know,
you
don't
want,
like
things
that
are
I'm
gonna,
make
something
up
things
that
are
hold
all
the
keys
to
the
kingdom
talking
to
like
a
blog.
B
That
would
be
a
super
big
bummer,
so
using
Sto,
our
back
rules
allows
you
to
enforce
and
we
did
allows
us
to
enforce
that
two
things
won't
talk
to
each
other.
If
you
are
using
the
Calico
CNI
like
we
are
cops
in
production,
you
can
also
use
Network
policies
and
sto
will
respect
those
which
is
terrific.
It
will
distribute
all
that
configuration
to
the
Envoy
proxy
okay,
that
was
it
and
I
have
eight
minutes
for
questions.
So
thank
you.
B
Let
me
see
if
I
can
show
these
all
right
present,
that
is
so
cool
okay,
which
service
mission
is
the
best
multi
cluster
Cates
decay.
It's
multi
in
for
a
case
to
be
M
support,
I,
would
say
fairly
unequivocally
sto.
One
of
the
things
that
we
noticed
with
the
helm
chart
is
it
provisions
and
opens
up
ports
on
the
load,
balancer
ingress
that
we
figured
out
actually
we're
addressable
to
the
pilot
and
mixer
control
panel.
So
you
can
actually
have
some
external
cluster
to
internal
cluster
support
that
didn't
really
matter
to
us.
B
B
B
We
do
not
do
that.
We
do
not
again
so
that
requires
a
admission
controller
webhook,
which
we
did
not
have
the
leisure
of
having.
So
we
used
st
OCT
l
cube
inject
so,
and
it
also
gives
us
a
really
strong
guarantee
that
we
know
exactly
which
pods
are
going
to
always
get
this
ike
are
so
anonymous.
Thanks
anonymous.
I
guess
hackers
are
here:
IP
tables
IP,
set
kernel
space
operations
limitation
in
throughput
latency
to
wanted
to
gigabyte
gigabits
per
second.
B
B
If
you're
talking
about
updating,
IP
tables
for
routing
information,
that's
actually
all
done
in
memory
and
it's
distributed
by
sto
pilot
to
the
Envoy
sidecars.
The
most
recent
releases
of
Sto
have
drastically
improved
the
speed
at
which
configuration
is
distributed
to
all
of
the
Envoy
side
cars,
but
they
pretty
much
guarantee
less
than
10
seconds
and
the
entirety
of
your
envoy
deployment
will
have
any
updated
configuration
changes.
You.
A
B
But
what
experience
do
you
have
with
publicly
accessible
endpoints
the
sto
ingress
gateway?
Perfect
literally
perfect?
For
us,
we
have
two
public
with
three
publicly
accessible
endpoints
in
the
cluster.
They
all
have
the
exact
same
DNS
address
and
based
on
host
header.
It's
a
configuration
file.
You
can
upload
to
sto,
it's
called
a
gateway
and
then
a
virtual,
corresponding
virtual
service
that
will
allow
you
to
decide
specifically
which
host
headers
are
routed.
B
I
would
also
recommend,
when
you
define
a
gateway,
that
you
automatically
redirect
any
HTTP
requests
to
HTTPS
so
that
your
application
doesn't
have
to
worry
about
that.
Does
our
service
mash
have
the
n-squared
problem?
Yes,
it
does
yeah
sto
does
also
I
would
say:
I
spoke
with
all
I
forgot
his
name.
It
was
named
Zack
Zack,
who
is
working
for
tetra
and
is
an
awesome
guy.
Sto
1.2
will
allow
a
configurable
amount
of
configuration
to
be
distributed,
but
right
now,
every
single
other
side,
car,
every
envoy
proxy,
trying
to
get
this
right.
B
Every
envoy
proxy
will
get
the
location
of
all
the
other
proxies.
So,
yes,
it
is
over
distributing
information,
especially
if
you
have
on
the
order
of
millions
of
side
cars,
so
that
is
going
to
be
a
problem.
If
you
try
to
do
this,
do
right
now
is
do
1.2,
I'm,
guessing
mid,
2019
or
late.
2019
will
totally
be
the
the
tool
of
choice
for
you.
There
do.
We
know
any
funny,
jokes,
I'll.
Let
you
did
this
one
keyword
funny.
B
A
B
Service
of
messages
allow
for
Bluegreen
deployments.
If
so,
what
would
be
a
good
way
to
do
so?
Okay,
so
there's
two
ways
to
do
it.
Obviously,
kubernetes
has
great
label.
Selection.
Sdo
recognizes
kerbin
IT
services
as
as
a
first-class
object,
so
all
criminate
services
are
routable
by
default,
so
you
can
do
subsets
in
ISTE
o,
which
allow
you
to
route
traffic
based
on
label
selection.
You
define
the
subset
or
you
can
use
kubernetes
services
and
switch
a
label.
B
What
will
happen
invariably
is
that
update
has
to
be
observed
by
sto
the
control,
plane,
consumed
and
then
distributed
to
envoy.
So
if
the
amount
of
time
that
it
takes
to
consume
the
update
of
the
label
switch
in
kubernetes
and
then
distribute
that
information
to
envoy
is
a
problem
for
you,
I
think
it's
on
the
order
of
a
split
second.
But
if
that's
a
problem
as
far
as
Bluegreen
like
it
must
be,
a
very
clean
split,
then
use
Sto,
I'm,
sorry
use
subsets,
which
are
a
a
portion
of
a
destination
rule
in
this
deal
No.
B
Yes,
this
we're
all
HTTP
for
those
of
you
that
have
custom
TCP
protocols.
There
are
a
couple
of
ways
to
solve
this:
one
is
in
roll,
your
own
envoy
and
and
up
and
use
the
same
interfaces
that
they
have
for
defining
G
RPC
HTTP.
The
I
think
Redis
is
now
supported
and
MongoDB
as
a
TCP
protocol
to
do
that
or
create
a
mechanism
by
which
you
sort
of
Marshall
and
unmarshal
requests
of
your
custom
protocol
to
something
that
does
speak.
G,
RPC
or
HTTP.
A
B
We're
just
gonna
keep
rolling
through
questions
out
loud
until
people
just
kick
us
out.
So
do
we
need
cluster-level
ingress
lb
proxies
when
you
have
mesh
sidecar
proxies?
No,
you
do
not
sto
does
its
own
I'm.
Sorry
envoy
does
its
own
low,
balancing
based
on
its
own
internal
algorithms.
You
can
control
that
algorithm,
so
you
do
not
need
cluster
level
ingress
load,
balancer,
proxies,
sto,
ingress
gateway,
also
provisions,
whatever
cloud
provider
has
so.
A
B
Yeah,
well,
we
don't
even
know
how
do
you
store
your
persistent
data
for
micro
services,
RDS
Redis
in
elastic
cache,
pretty
much
anywhere
when
when
it
comes
to
data,
we're
just
not
confident
yet
in
our
own
ability
to
run
stateful
sets
in
kubernetes
we
delegate.
All
of
that
I
mean
again
I'm
a
team
of
one
I
hold
the
pager
for
infrastructure
problems
so
as
much
as
possible.
I'd
love
to
have
the
Amazon
people
chewing
Rolaids
all
night,
not
me
so.
B
Yes
and
how
do
we
share
data
between
services
when
there's
a
logical
relationship,
we
have
architecture
diagrams
where
it's
legal
to
write,
have
two
services
to
write
to
the
same
database
that
feels
illegal
in
my
heart.
We
don't
obviously
do
that.
Oh,
we
don't
do
it
anything
okay.
So
then,
if
there's
a
logical
relationship
between
those
two,
we
either
communicate
over
a
message,
bus
or
HTTP.
B
You
are
done
migrating
upgraded,
something
new
and
there's
something
new
in
the
market.
How
often
you
keep
catching
up?
Oh
yeah,
this
is
like
a
real
problem,
so
obviously,
there's
always
going
to
be
something
new.
Just
like
Austin
said,
I'm
sure
there
was
a
service
mesh
conceived
and
built
at
this
conference.
B
We
have.
We
are
engineers
like
Kelsey
Hightower
put
it
so
perfectly.
Our
purpose,
as
eight
engineers
is
to
it
against
the
goals
of
why
green
our
mission
is
to
make
sure
that
earth
can
be
here
in
the
future.
So
we
finance
energy-efficient
upgrades
insofar
as
we
can.
We
use
the
technology
that
works.
That's
why
I
like
people
shudder
when
we
say
we
used
Ruby,
but
Ruby
works,
and
it's
understandable
for
literally
everybody
on
our
team.
So.
A
B
We
did-
and
this
is
why
we
picked
Sto-
we
did
not
Minj
our
own
CA.
We
did
not
have
a
certificate
revocation
list
if
we
had
to
rotate
a
certificate.
It
was
a
super
bad
process
been
up.
Here
was
the
one
who
helped
us
do.
Self-Signed
cert
I
knew
none
of
this
ahead
of
time,
but
he
would
be
able
to
expertly
answer
this
question.
Thankfully
we
got
to
rip
all
of
that
out
and
put
SDO
in
and
now
we
can
manage
it
with
that.
How.
A
B
Authentication
is
you
are
who
you
said:
you
are
authorization,
as
you're
allowed
to
do
what
you
said.
You
could
do
so-and-so,
authentication,
yes,
authorization
only
if
it's
service
to
service,
but
we
do
not
do
like
API
object
level
decisions
in
ISTE.
Oh,
that's!
Actually
something
we're
trying
to
consider
plugging
into
mixer,
but
basically
like
user
a
is
allowed
to
do
something
because
they
presented
a
valid
Jaa
token,
which
means
it's
them,
but
that
doesn't
necessarily
mean
that
they're
allowed
to
perform
this
function
on
this
other
object
and
service
be
like
you
can
imagine.
B
An
underwriter
would
like
to
click
something
that
changes
a
decision
in
a
dashboard,
but
if
they're
not
an
underwriter,
they
shouldn't
be
allowed
to
change
that
decision.
So
that's
like
four
different
things
that
just
happen
there
in
terms
of
our
services
and
all
of
those
services
are
authorized
and
authenticated
to
talk
to
each
other,
but
that
person
might
not
be
so.
That's
still
done
by
permissions
inside
of
our
main
manager,
app.
A
B
Beta
features
in
kubernetes,
we
both
participated
in
contribution
to
upstream
open
source.
Kubernetes
beta
features
are
generally
and
it's
the
same
team
essentially
doing
this
stuff
in
sto.
Alpha
features
are
the
ones
that
are
super.
Experimental
beta
is
usually
supposed
to
be
stable
enough
to
rely
on,
but
productions
like
yeah,
maybe,
but
that
means
that
there
aren't
going
to
be
huge,
braking
API
changes
if
it
does
go
to
G
a
mine.
A
B
B
Experience
with
this
deal
horrible,
but
that
was
because
we
had
to
turn
off
config
validation,
not
because
sto
isn't
a
good
product.
So
if
you
turn
off
config
validation,
you
only
know
that
everything
is
broken
or
nothing
is
broken
and
did
it
actually
work
so
and
in
fact,
the
the
issue
of
ended
version
of
authentication,
TLS
checking
that
we
had
did
not
actually
work,
I
had
to
compile
it
from
source,
so
that
was
kind
of
a
red
flag
at
first,
but
it
was
pretty
bad
I'll.
Take
this
one:
okay,.
A
B
You
mentioned
the
ingress
controller,
implements
AMPA
config.
Does
it
mean
that
you
don't
use
gateway,
API
resources,
but
regular
ingress
with
the
is-2
ingress
class?
Oh
SVO
has
two
different
ways:
you
can
use
the
kubernetes
ingress
or
you
can
use
an
st
of
ingress
gateway.
An
in-studio
ingress
gateway
is
configured
as
a
kubernetes
service
object
that
has
a
type
of
load
balancer,
so
it
is
separate
and
it
is
managed
by
sto
and
the
Envoy
control
plane.
I
think
we're
almost.
We
are
super
over
time.
What
did
the
zero
say?
The
eight
nice
belt
awesome.
B
Ok,
that
was
a
joke.
Node
side
car
versus
pod,
sidecar,
pod
sidecar
cubelets,
are
not
supported.
Yet
the
plan
is
to
do
that
1.2.
So
you
can
do
health
checking
with
TLS
services
shoutouts
for
team
of
one.
Thank
you
with
eks.
It
has
a
public
API
endpoint.
Is
that
security
concern?
If
you
trust
AWS
H
Mac
off?
No,
it's
not.
How
do
you
have
history
related
reload
with
cert
managers?