►
Description
Nobody likes the idea of unscheduled downtime, downgraded performance, or high costs due to unforeseen traffic demands. However, the solutions to these challenges aren’t always straightforward. Typically, the first response would be to spread your application for high availability (HA). However, you have to consider how traffic will be balanced in such topologies. How will you manage cross-zone or cross-regional traffic costs? How can you optimize your load balancing with unequal demands in different zones and regions? If that’s not enough, you also have to think about the underlying computing resources. How do you scale your cluster automatically whilst catering to scheduling constraints, pod requirements, and compute costs all at once? In this talk, Lukonde Mwila will walk through these challenges and share how teams can overcome them using pod anti-affinity, Istio, and Karpenter.
A
B
So
thanks
for
coming
to
the
talk,
as
you
can
see,
I'm
going
to
be
speaking
about
spreading
applications
controlling
traffic
and
optimizing
costs
in
kubernetes.
As
William
already
pointed
out,
my
name
is
Le
Conde
muella
or
you
can
also
call
me
Luke
and
I'm.
A
developer
advocate
for
kubernetes
at
AWS.
B
So
if
any
of
you
want
to
get
in
touch
with
me,
you
can
feel
free
to
do
so.
You
can
reach
out
on
LinkedIn
I
use
my
full
name,
so
you
can
search
for
me,
hopefully
you'll,
be
able
to
recognize
me
also
I
produce
a
lot
of
kubernetes
and
Cloud
native
related
content,
and
you
can
search
for
that
on
YouTube
subscribe
to
the
channel
if
you
find
it
interesting,
but
also
the
developer
advocacy
team
that
I'm
a
part
of
has
a
dedicated,
YouTube
channel
called
containers
from
the
couch.
B
Great
we're
going
to
jump
right
in
so
this
talk
was
inspired
by
a
combination
of
things
from
the
past
and
the
present
the
first
one
being
an
event
several
years
ago
on
a
specific
Black
Friday
and
at
the
time
I
was
working
as
a
developer
for
a
software
company,
not
the
same
one
as
William,
and
all
the
developers
worked
together
in
a
shared
open
space.
So,
from
my
vantage
point,
I
could
see
the
screens
of
some
of
my
work
colleagues
now
because
it
was
Black
Friday
as
you'd
expect.
B
A
number
of
my
work,
colleagues
were
especially
excited
to
take
advantage
of
the
discounts
on
a
particular
e-commerce
website
and
that
site
was
really
popular
amongst
most
of
the
folks
at
the
company.
Unfortunately,
on
that
day,
some
of
you
might
know
where
this
is
going.
The
site,
crashed
and
I
could
see
this
from
where
I
was
seated.
B
They
didn't
have
that
bit.
There
about
trying
to
restore
balance
in
the
force
I'm
a
Star
Wars
fan
so
I
threw
that
in
there,
so
just
for
the
sake
of
historic
accuracy
that
wasn't
there
now,
when
you
experience
something
like
this
as
a
developer
and
you're
the
end
user,
or
even
if
you're
just
involved
in
software
architecture,
and
you
understand
it
to
some
degree,
you
can't
help
but
ask
yourself
the
question:
what
would
I
do
if
I
was
in
that
position
or
how
would
I
have
prevented
that
kind
of
situation?
You?
B
You
can
kind
of
empathize
with
the
people
involved
in
that
situation,
because
they're
also
involved
in
software.
So
naturally
it
spawned
a
lot
of
discussions
between
myself
and
some
of
my
work
colleagues
at
the
time,
but
even
over
the
years,
because
it
was
a
relatively
significant
incident
and
the
general
consensus
from
a
lot
of
those
discussions
was
that
if
you
want
to
prevent
something
like
this,
then
you
should
probably
have
some
form
of
high
availability
and
auto
scaling
mechanisms
involved
in
your
architecture.
B
Now
I
just
want
to
be
clear.
We
didn't
have
context
as
to
what
the
root
causes
were,
so
we
were
basically
guessing,
but
that
was
a
safe
guess
in
terms
of
the
kind
of
measures
that
you
want
to
have
in
place
to
mitigate
the
risks
of
such
an
outage,
especially
for
a
business
of
its
stature
and
on
such
an
important
day.
B
You
want
to
eliminate
having
a
single
point
of
failure,
in
addition
to
that,
it
improves
performance
by
essentially
taking
care
of
the
fact
that
you
have
you
don't
want
to
have
a
minimal
number
of
resources
that
gets
strained,
but
bearing
that
in
mind,
you
need
to
be
aware
of
what
your
particular
constraints
are
for
a
particular
project
in
order
to
implement
the
solution
in
the
best
possible
way.
So
there
are
a
lot
of
costs
associated
with
the
load
balancing
strategy.
B
In
addition
to
that,
we
also
have
to
consider
Auto
scaling,
so
this
traffic,
that
is
being
proxied
from
the
load
balancer,
is
headed
up
to
Upstream
destinations
and
one
of
the
other
common
pitfalls
is
wasted.
Compute
capacity,
even
while
I've
been
here
speaking
to
a
couple
of
people
who
shared
that
they've
also
been
seeing
this
a
whole
lot.
Do
your
nodes
actually
align
with
the
workload
requirements,
and
so
Auto
scaling
is
important,
but
another
thing
that
we
see
that
happens.
B
A
whole
lot
is
over
provisioning
of
compute
capacity,
and
so
it's
great
that
now
you're
your
infrastructure
is
essentially
scaled
out
to
accommodate
those
events
where
you
need
to
scale.
However,
there
are
situations
where
you've
provided
a
whole
lot
more
than
what
you
actually
need.
The
reason
for
that
is
probably
using
a
cluster
Auto
scaler
that
has
a
more
static
approach,
working
with
a
scaling
group
in
a
particular
cloud
provider
that
you
may
be
working
with.
B
So
these
two
areas
incur
you
a
whole
lot
of
costs
and
now
it's
a
whole
lot
more
critical,
because
we're
in
a
difficult
economic
time
and
a
lot
of
teams
do
want
to
take
advantage
of
high
availability
of
their
workloads
and
their
infrastructure
with
kubernetes.
And
they
want
to
achieve
that
in
a
cloud
environment
because
of
what
it
has
to
offer,
including
the
elasticity
they
can.
They
can
get,
but
they're
also
trying
to
find
the
best
ways
to
address
the
operational
costs
associated
with
such
a
model
and
that's
what
I'm
going
to
be
focusing
on.
B
So
the
first
thing
that
I
want
to
look
at
is
spreading
of
your
application
and
they're.
Two
different
features
that
you
can
make
use
of
in
kubernetes
in
order
to
spread
your
application.
The
first
one
that
I
want
to
focus
on
is
POD
Affinity
rules
and
so
pod
Affinity
rules.
How
you
can
essentially
apply
scheduling
constraints
through
the
form
of
certain
rules
that
Define
a
relationship
between
different
workloads.
It
could
be.
It
could
be
the
same
workload
as
well,
and
that's
what
you'd
essentially
want
in
the
case
of
achieving
High
availability.
B
Now,
what
we're
interested
in,
as
you
can
see
in
this
diagram
over
here,
is
an
anti-affinity
rule,
so
an
affinity
rule
would
essentially
be
if
we
want
our
pods
in
close
proximity,
whether
that's
based
on
a
host
topology
or
if
we
want
it
to
be,
if
we
want
them
to
be
in
the
same
AZ
with
anti-affinity,
that's
essentially
wanting
to
have
somewhat
of
a
repelling
effect.
We
want
the
pods
to
repel
each
other,
so
that
they're
spread
across
the
different
topology
domains,
and
so
that's
how
we
would
achieve
High
availability
with
our
application.
B
Next
I
want
us
to
consider
pod
topology
spread
constraints.
So
this
is
the
other
approach.
Now
pod,
Affinity
rules
or
rather
pot
anti-affinity
rules
can
help
you
achieve
High
availability,
but
the
issue
with
them
is
that
it
doesn't
necessarily
address,
or
rather
it
introduces
a
problem
of
fault
tolerance,
pods
apology,
spread
constraints,
not
only
provide
you
with
high
availability,
but
also
deal
with
the
issue
of
fault
tolerance.
It
gives
you
more
control
over
how
you
actually
want
your
pods
to
be
distributed
across
the
different
topology
domains
in
your
cluster.
B
So
if
you
take
a
look
at
this
diagram
over
here,
you
see
that
we
have
10
pod,
replicas
and
they're
close
to,
even
when
it
comes
to
doing
this
in
three
different
availability
zones.
We've
got
a
spread
of
three
four
three,
so
this
is
something
that
you
can't
really
achieve
with
pod
anti-affinity
rules
and
the
reason
for
that
is
because
they
have
an
anti-affinity
toward
each
other.
B
So
you
could
easily
end
up
with
a
situation
where
you
have
a
single
replica
running
on
a
node
which
is
not
good
for
fault
tolerance
and
it's
not
good
for
resource
utilization
either.
So
this
works
out
a
whole
lot
better
if
you're
trying
to
not
only
address
availability
but
also
false
tolerance
of
your
workloads,
and
because
this
is
the
one
I'm
going
to
be
showing
you
or
demonstrating
a
little
bit
later,
I'm
going
to
focus
on
the
kind
of
properties
that
you
would
be
defining.
B
So
these
are
the
properties
that
you
would
primarily
be
concerned
with.
If
you're
going
to
use
pod,
topology
spread
constraints,
I'm
going
to
start
off
by
focusing
on
Max,
skew
and
Max
Q
is
how
you
would
Define
the
maximum
point
to
which
you
want
imbalance
or
inequality
for
the
distribution
of
your
pods
across
the
different
topology
domains.
So
if
we
take
the
example
from
the
previous
diagram
just
going
to
go
back
quickly,
so
you
can
see
that
so
that's
10
replicas
for
three
different
availability
zones,
there's
no
way
to
equally
spread
the
number
of
PODS.
B
But
what
we
can
do
is
say
we
want
to
maximum
imbalance
or
inequality
of
one,
which
is
why
we've
got
three
four
three,
so
that
could
easily
end
up
also
being
four
three
three
now
the
max
Q
can
be
anything
between
the
value
of
one
and
the
number
of
replicas
that
you
have.
So
in
that
case
it
would
be
between
1
and
10..
B
It's
going
to
be
one
of
the
labels
and
it's
how
you
essentially
Define
the
kind
of
topology
that
you
want
to
work
with,
whether
you
want
it
to
be
a
zonal
approach
or
if
you
want
it
to
be
a
host
and
when
unsatisfiable
is
somewhat
self-explanatory.
This
is
how
you
want
the
scheduler
to
respond
in
the
case
that
it
can't
meet
these
scheduling
constraints.
B
That
would
be
how
you
would
achieve
availability
of
your
application.
But
the
next
thing
that
we
want
to
consider
now
is
to
start
addressing
those
two
main
areas
of
optimizing:
our
costs,
the
load,
balancing
area
and
then
we'll
get
to
the
nodes
a
little
later.
But
let's
start
off
with
controlling
traffic
and
I'm
going
to
demonstrate,
or
rather
speak
about,
two
different
approaches
that
you
can
take.
B
The
first
approach,
I'm
going
to
discuss,
will
be
through
the
use
of
istio
for
those
who
aren't
familiar
with
istio,
it's
an
implementation
of
a
service
mesh
and
the
job
of
a
service.
Mesh
in
a
nutshell,
is
essentially
to
unburden
applications
from
having
to
deal
with
networking
concerns,
and
they
generally
do
that
in
these
four
main
domains
connecting
of
the
applications
securing
those
connections
controlling
traffic
or
traffic
Management.
B
In
addition,
with
that
also
adding
resilience
mechanisms
and
as
well
as
having
observability
features
now,
what
we're
concerned
about
in
this
particular
case
is
controlling
traffic,
so
changing
the
way
load
balancing
is
actually
going
to
work.
So
if
we
look
at
the
diagram
over
here,
you'll
notice
that
we
have
a
request
that
starts
from
a
particular
end
user
and
their
request
goes
through
to
the
load
balancer
that
is
exposed
by
the
istio
Ingress
Gateway,
that
traffic.
That
traffic
then
gets
proxied
to
that
component.
There.
That
says
virtual
service.
B
The
job
of
a
virtual
service
in
istio
works
a
lot
like
Ingress.
So
if
you're
familiar
with
the
Ingress
resource,
this
is
essentially
how
routing
takes
place.
You
define
your
routing
rules
using
virtual
Services.
After
rooting
takes
place,
you
can
apply
additional
policies,
such
as
what
we're
actually
about
to
carry
out
which
is
controlling
the
traffic,
and
you
would
do
that
with
destination
rules
and
with
the
destination
rules.
What
we
can
actually
do
is
essentially
say
for
traffic
coming
from
a
certain
point
of
origin.
B
We
want
it
to
go
to
a
certain
destination,
and
so
some
of
you
might
already
be
familiar
with
these
Concepts
when
you
think
of
different
deployment
strategies
directing
a
certain
amount
of
traffic
to
a
certain
version
of
an
application
for
either
a
blue
green
deployment
or
a
canary
deployment.
But
in
this
case
what
we
actually
want
to
be
doing
is
taking
advantage
of
istio's
feature
known
as
locality,
weighted
load,
balancing
and,
if
I'm,
correct.
B
Other
service
measures
also
have
this
so
istio
essentially
takes
the
information
of
the
topology
domains
in
your
cluster
and
uses
that
information
in
order
for
you
to
actually
carry
this
out
so
for
a
load
balancer,
which
is
highly
available
and
I'll
demonstrate,
demonstrate
that
a
little
bit
later.
So
let's
say
it:
we
have
our
cluster
running
in
the
region.
Eu
West
one
and
we've
also
got
a
load
balancer,
that's
highly
available,
so
it's
across
three
different
availability
zones,
EU
West,
one
A,
B
and
C
traffic
that
hits
that
particular
load
balancer.
B
We
want
to
configure
it
in
such
a
way
that
we
say
for
traffic
coming
from
EU
West
1A.
It
should
go
to
an
application
that
is
running
in
EU,
West
1A
and
we
can
determine
how
much
of
that
traffic
goes
where
so.
In
this
case,
you'll
see
with
this
particular
diagram,
I'm
saying:
60
percent
of
traffic
to
EU
West
1A
40
to
EU
West
1B-
and
this
is
a
powerful
mechanism,
because
this
is
one
of
the
ways
that
you
can
drastically
reduce
the
egress
traffic
costs.
B
And
so,
if
you
have
a
highly
available
cluster
across
EU
West,
one
A,
B
and
C,
the
endpoint
slice
controller
is
going
to
be
responsible
for
allocating
the
different
endpoints
into
these
different
topology
domains.
Now,
when
you,
when
you
do
that,
the
next
thing
is
the
Q
proxy
is
a
Daemon
set.
That's
running
on
each
of
the
different
nodes
and
the
Q
proxy
is
also
serving
a
form
of
internal
routing
and
what
it
does
is
it
consumes
from
the
endpoint
slices
now
outside
of
topology.
Aware
hints.
B
Each
of
these
endpoints
just
has
information
about
the
specific
pod.
Its
IP
address
the
node
that
it's
running
on
and
any
additional
topology
information.
But
when
you
have
topology
aware
hints
enabled
this
is
what
happens
so
you
can
see
without
hints
the
endpoint
slices
have
endpoints.
That
essentially
say
we
serve
traffic
for
just
about
any
Zone,
but
when
hints
are
enabled
it
essentially
has
there's
an
endpoint
with
a
specific
availability
Zone
based
on
the
one
where
that
pod
is
actually
running.
B
So
if
we
have
a
pod
running
an
EU
West
1A,
that
information
is
stored
in
a
particular
endpoint
and
then
the
endpoint
slice
controller
is
going
to
add
a
hint
saying
that
you
should
serve
traffic
coming
from
EU
West
1A.
So
that's
the
mechanism
to
essentially
control
traffic
to
be
within
a
specific
Zone
to
minimize
the
costs
associated
with
egress
cross
Zone
traffic.
B
Hopefully
that
helps
you
with
when
you're
trying
to
consider
which
approach
to
take
again
having
a
service
mesh
does
come
with
operational
complexity,
additional
domain
knowledge,
and
it
might
be
a
case
where
your
team
is
not
in
a
position
to
take
on
that.
Alternatively,
if
you
decide
to
go
with
topology
aware
hands,
you
just
need
to
be
aware
of
the
specific
version
of
kubernetes
that
you
would
have
to
be
running
in
order
to
take
advantage
of
that
approach.
B
But
the
process
of
actually
going
getting
running
with
topology
aware
hints
is
really
simple:
it's
just
ensuring
that
your
nodes
have
the
relevant
topology
domains
and
if
you're
running
a
kubernetes
cluster
in
a
cloud
environment
that
automatically
gets
generated
for
you,
those
topology
domains
are
attached
as
labels
to
the
respective
nodes,
and
then
you
would
simply
need
to
add
an
annotation
to
the
service
that
you
want
to
have
actually
managing
the
the
service.
That's
going
to
be
proxying
traffic
to
the
different
pods
in
order
for
hints
to
work
out
all
right.
B
Okay,
most
people,
which
is
great,
and
it's
a
fantastic
project,
and
several
years
ago
before
I
was
in
developer
advocacy
and
was
still
Consulting.
That's
the
project
that
we
primarily
used,
but
a
challenge
that
we
run
into
then
was
the
fact
that
the
cluster
Auto
scaler
takes
a
static
approach
to
scaling.
B
And
what
Carpenter
does
differently
is
the
fact
that
it's
more
Dynamic
and
instead
of
working
with
a
scaling
group.
Instead,
it
looks
at
pods
that
are
in
a
pending
State
and
takes
into
consideration
into
consideration
their
specific
pod
requirements,
as
well
as
their
scheduling
constraints
and
then
reaches
out
directly
to
the
ec2
API,
in
order
for
it
to
provision
the
nodes
that
are
actually
needed
for
those
particular
workloads,
and
so
this
drastically
improves
your
research
resource
utilization
as
well.
In
addition
to
that,
Carpenter
also
has
a
feature
called
cons.
B
Now,
obviously,
the
big
challenge
with
Carpenter
is,
even
though
it's
an
open
source
project
at
this
particular
point
in
time,
it
only
has
there's
only
support
for
the
AWS
cloud
provider,
so
there's
still
an
open
issue.
This
is
also
somewhat
of
an
invitation
for
more
people
to
get
involved
with
the
project
in
order
for
it
to
be
extended
to
additional
Cloud
providers
as
well.
B
Now,
one
of
the
other
things
that
I
love
about
Carpenter
is
the
fact
that
it
also
respects
scheduling
constraints
so
to
Circle
back
to
where
I
started
from
in
terms
of
high
availability.
When
you
put
particular
topology,
spread
constraints
in
place
for
your
workloads,
Carpenter
will
be
able
to
respect
those
things
when
it's
adding
nodes
to
your
cluster.
B
B
B
Now
both
of
them
have
topology
spread
constraints
applied.
Now,
if
you
take
a
look
at
this,
you'll
see
that
this
is
the
exact
same
code
block
that
I
walked
through
earlier,
when
I
was
talking
about
topology
spread
constraints.
In
addition
to
that,
I've
got
a
node
selector
to
ensure
that
these
pods
are
only
placed
on
nodes
that
are
added
by
Carpenter.
B
So
that's
our
application
and
they're
going
to
be
fronted
by
a
cluster
IP
service
that
proxies
to
those
two
different
applications,
treating
them
as
if
they're
the
same
one
or
rather
two
different
versions
of
the
same
application.
Then
next
over
here
is
a
custom
resource
definition
file
known
as
a
provisioner.
So
this
is
the
file
that
would
essentially
control
the
life
cycle
of
your
nodes
in
Carpenter,
and
you
can
have
multiple
provisioners
where
you
can
have
a
single
one.
So
in
this
case,
I've
got
a
provisioner
dedicated
to
my
Express
test
workload.
B
As
you
can
see
here,
it's
called
Express
test
and
you
can
apply
different
parameters
or
constraints
for
how
you
want
Carpenter,
to
pre
to
add
nodes
to
your
cluster.
So
in
this
case
it's
already
defined
to
add
nodes
to
every
one
of
the
availability
zones.
In
the
case
that
you
wanted
to
add
constraints
to
that,
you
can
do
that
if
you
want
to
restrict
it
to
only
adding
spot
instances,
that's
something
that
you
can
also
apply
in
my
case.
B
I
want
both
spot
instances
and
on-demand
instances,
and
you
can
also
add
further
configurations
like
defining
the
instance.
Families
you
want,
which
is
a
really
powerful
feature,
because
obviously
there
are
certain
instance,
families
that
are
more
costly
than
others,
and
so
you
can
be
able
to
manage
that
and
then
right
at
the
top.
Here
is
consolidation,
and
you
can
see
it's
enabled
so
remember
this.
B
All
right
next
up
over
here
is
our
destination
rule.
This
is
that
custom
resource
definition
file
for
istio
and,
as
you
can
see
over
here,
I'm
just
going
to
focus
on
the
distribution
section
and
you
can
see
what
I'm
saying
is
for
traffic
coming
from
EU
West
1A
I
want
80
of
it
to
go
to
EU,
West,
1A
and
10
to
be
in
10
to
C,
and
the
other
sections
are
very
similar.
B
B
Before
we
get
to
that
before,
I
run,
it
just
want
to
quickly
show
you
over.
Here
is
Express
test.
Two,
it's
already
running
in
my
eks
cluster,
and
here
we
have
Express
test
one
Express
test.
Two
is
running
on
this
node
over
here
you'll
see
two
one:
six
five
and
express
test.
One
is
running
on
zero,
zero.
Eighty
seven
so
I
want
to
quickly
come
here,
so
you
can
see
that
you
see
we
have
zero
zero.
Eighty
seven
over
there
I'm
going
to
describe
that
and
I'm
going
to
scroll
down.
B
B
There
we
go
so
we've
got
responses
of
version
1.1.2
and
some
from
version
1.1.4.
Now,
if
we're
honest,
it's
really
hard
to
deduce
that
the
destination
rules
are
actually
working
for
all.
We
know
it's
just
the
random
approach,
so
the
best
way
to
verify
this
is
to
actually
go
back
to
our
code,
editor
and
modify
the
destination
rules,
and
instead,
what
we're
going
to
say
is
we
want,
regardless
of
where
the
traffic
is
coming
from?
We
want
98
percent
of
it
to
be
sent
to
EU
West
1A.
C
B
D
D
B
B
So
it's
totally
up
to
you,
depending
on
the
type
of
application
you're
running
and
how
critical
it
is
to
continue
running.
Obviously,
the
the
desired
approach
is
to
have
high
availability
across
your
different
topology
domains,
but
then
this
is
an
attempt
by
the
scheduler
to
take
those
rules
into
consideration
when
it's
applying
them,
so
it
just
comes
down
to.
How
do
you
want
it
to
respond
in
the
case
that
those
constraints
can't
be
satisfied?
B
C
All
I
would
like
to
ask
you
something
related
to
Carpenter,
because
I
I've
been
using
it
in
production
for
over
a
client
of
mine,
yeah,
and
one
of
the
issue
I
find
out
is
that
while
Carpenter
worked
really
well
with
scaling
and
regarding
like
memory
or
CPU
or
this
kind
of
statistic,
disk
space
is
actually
a
problem
like
I
had
no
the
crushing
like
application,
crushing
just
simply
because
the
node
was
that
this
space
was
not
enough
and
stuff
like
that.
Right.
B
E
B
E
Talk
yeah
I
have
a
question
regarding
when
you're
talking
about
the
topology
spread
like
in
the
beginning
and
the
fact
that
we
have
quite
a
lot
of
information
or
on
where
our
workloads
are
actually
deployed
in
which
zone
in
the
given
cloud
provider.
We
have
this
information
on
the
pods,
because
polls
are
in
the
nodes,
yeah.
B
E
Also
have
this
resource
called
the
endpoint
so
where
we
know
how,
where
to
root
our
traffic,
so
that
it
ends
up
in
the
particular
availability
Zone.
However,
I
also
noticed
that
in
one
of
the
configuration
files
when
you're
showing
how
to
set
up
istio
yeah,
there,
you
had
the
configuration
on.
Basically,
you
had
the
paths
configured
in
such
a
way
that
there
were
names
of
these
zones
in
the
paths
yeah.