►
From YouTube: TGI Kubernetes 097: Cluster Autoscaler
Description
Notes: https://github.com/vmware-tanzu/tgik/blob/master/episodes/097/README.md
- 00:00:00 - Welcome to TGIK!
- 00:04:04 - Week in Review
- 00:18:40 - Autoscaling Poll
- 00:23:13 - Autoscaling Overview
- 00:42:00 - Deployment Resource and Limits
- 01:00:00 - Horizontal Pod Autoscaler Overview and Configuration
- 01:12:49 - Cluster Autoscaler Overview and Deployment
- 01:38:00 - Autoscaler Implications for Cluster API
- 01:46:00 - Quick Overview of Vertical Pod Autoscaler
Come hang out with Josh Rosso as he does a bit of hands on hacking of Kubernetes and related topics. Some of this will be Josh talking about the things he knows. Some of this will be Josh exploring something new with the audience. Come join the fun, ask questions, comment, and participate in the live chat!
A
A
So
it's
great
great
to
be
here
with
you
all,
as
you're
joining
in
if
you're
comfortable,
feel
free
to
say,
hey
and
Chad
and
where
you're
signing
in
from
I'm
out
of
Boulder
Colorado
by
the
way,
so
we're
kind
of
in
that
stage
of
kind
of
wintery,
but
also
kind
of
summery,
it's
its
high
elevation.
So
it
tends
to
vary
quite
a
bit.
So,
let's
see
who
do
we
have
joining
us
so
far,
Olaf
from
Denmark
awesome
welcome.
Olap
joy
from
Richmond
thanks
for
joining
us
Peter
from
Spain,
very
cool
Maddy.
Welcome!
A
Welcome
glad
to
have
you
back,
Kristoff
Germany
welcome
Christophe.
Thanks
for
joining
us,
hey
Peter,
San
Diego.
Welcome!
Welcome
jacquees
from
New
York
City.
Yeah
awesome.
Welcome
thanks
for
joining
us
from
the
East
Coast
there.
No
it's
a
little
bit
later
into
your
day
than
ours.
Jose
from
Madrid.
Welcome.
Welcome
Christian
from
Germany
Steve
nice
to
see
you
Steve
Steve
used
to
be
on
our
direct
team
with
Duffy
and
I,
but
he's
moved
on
to
some
more
engineering
focused
tasks.
A
It's
great
great
to
see
your
face
again:
Steven
if
it's
just
a
little
thumbnail
myth
of
San
Francisco
welcome.
We've
got
Prasad
from
Switzerland
very
cool,
welcome,
Elco
Eelco,
hopefully
I'm
saying
that
somewhat
close
welcome
from
Holland
Sandeep
from
Poland
welcome,
Sandeep
great
to
have
you
with
us
Martin
from
the
net.
So
does
that
mean
the
Internet
welcome
Martin
glad
to
have
you
Gustavo
from
Chicago.
Welcome!
A
Welcome
Shahar
from
Atlanta
awesome
welcome,
al
ski
from
Russia
Alec
sea
from
Russia,
hopefully,
I'm
saying
that
one
somewhat
close,
but
welcome
from
Russia
awesome
how's
that
how's
the
weather
in
Russia
right
now.
Is
it
a
is
it
cold?
Is
it
warm
Martin
Netherlands
welcome,
David
from
Minnesota
Cecilia
from
Germany,
all
right
cool,
so
welcome
everyone
I'm
sure
we'll
have
more
people
join
as
we
go.
I'm
already
excited
to
see
the
representation
from
all
over
the
world.
Toma
Tama
from
Romania
welcome.
A
Super
super
excited
to
have
you,
so
you
know
in
in
normal
tgia
format,
we'll
start
off
here
by
just
talking
a
little
bit
about
some
news
stuff
in
the
in
the
community
and
by
the
way
I've
got
George
joining
me.
You
all
probably
saw
he
linked
the
the
notes
for
the
hack
of
MIDI
as
well.
He's
gonna
be
helping
out
and
chat
here.
So
thanks
so
much
to
George
for
the
support
and
let's
go
ahead
and
talk
a
little
bit
about
some
news
so
into
the
notes
in
Allah.
A
I'll,
pull
it
up
on
my
screen
here
and
switch
over
for
you
all.
So
there
we
go
cool
all
right,
alright,
so
again
episode
97
our
focus
this
time
make
my
camera
a
bit
higher.
Maybe
there
we
go
our
focus.
This
time
is
around
cluster.
Auto
scaling
and
auto
scaling,
as
you
probably
know,
has,
has
multiple
facets
to
it.
We're
we're
mostly
focused
on
cluster,
auto
scaling,
I
guess,
but
really
the
other
facets
of
auto
scaling
played
together,
so
we'll
be
talking
about
a
lot
of
the
different
pieces.
A
Overall,
I'm
actually
really
excited
about
this
topic,
because
this
might
this
might
surprise
you
I
my
role
along
with
Duffy's
role.
As
we
help
we
help
a
lot
of
companies
and
large
orgs
implement
kubernetes,
and
for
me
personally,
you
know
I've
consulted
on
and
talked
about
like
scaling
methodologies,
an
HPA
and
cluster
autoscaler,
but
it's
actually
been
a
really
long
time
since
I've
touched
it
like
actually
tried
to
deploy.
It
tried
to
use
it
back
in,
like
the
the
cube
160
or
something
like
that.
We
we
deployed
it
to
Amazon
with
a
customer.
A
I
was
working
with.
In
fact,
we
were
we're
doing
some
auto
scaling
stuff
that
also
like
integrated
with
spa
tents,
to
make
like
bids
on
different
instance,
types
and
stuff,
so
I'm
really
excited
to
kind
of
revisit
this
and
and
mess
around
with
it.
I
expect
a
lot
to
potentially
go
wrong
as
we're
playing
around,
but
that's
kind
of
the
joy
of
it
right,
very
cool.
So
before
I
get
in
the
news,
let's
see
who
else
we've
had
join
cool
I
think
I
may
have
said
everyone
so
far.
A
So
let's
talk
a
bit
about
some
news.
Now
news
this
week:
it
guess
it's
like
a
fairly
lighter
week.
Maybe
we've
got
the
beta
release
for
kube.
1.17
is
live,
so
you
early
adopters
and
testers
check
that
out
download
the
binaries
mess
around
with
it
give
feedback,
and
then
code
freeze
for
those
of
you
who
are
maybe
in
like
cigs
and
working
groups,
is
coming
up
on
November
14th.
So
just
be
cognizant
of
that.
We
have
a
couple
of
interesting
things
coming
from
around
the
web
this
week.
A
So
inside
of
our
show
notes,
we've
got
a
link
to
an
interesting
project,
called
chaos
cube
and
I've.
Seen
certain
projects
like
this
propagate
I've,
seen
a
lot
of
my
my
customers.
I
work
with
you
know
frankly
build
this
from
scratch
over
and
over
again,
but
chaos.
You
might
be
an
interesting
project
to
look
at
if
you're,
looking
for
something
that
can
go
in
and
provide
some
intentional
instability
to
your
platform
right,
go
in
and
kill,
pods
and
and
make
certain
events
occur.
A
So
you
can
ensure
that
your
your
workloads
on
kubernetes
are
as
resilient
as
they
should
be,
so
so
would
be
one
to
check
out.
I
haven't
personally
tested,
chaos
cube
yet,
but
definitely
definitely
a
growing
area,
which
is
you
know,
doing
chaos,
engineering,
the
top
of
kubernetes
GoDaddy
open
sourced
their
kubernetes
gated
deployments
tool
I
like
the
name,
because
it
is
extremely
specific
right,
but
basically
from
what
I've
read
and
I
haven't
I
haven't
tested
it.
Yet
it
looks
like
a
way
to
do.
You
know
kind
of
automated
and
more
elegant
kind
of
a
be.
A
You
know,
canary
style,
testing,
where
you
can
do
some
amount
of
introduction
of
an
behind
a
service
that
might
be
a
new
version
of
the
app
and,
as
you
many
of
you
know,
with
kubernetes,
we
can
validate
that
that
new
version
works
and
if
it
does,
we
can
make
sure
that
all
new
traffic
propagates
into
that
new
version
of
the
app
and
I
again
have
a
dog
super
deep
into
it.
Of
course,
but
I
would
imagine
that
there
are.
You
know,
roll
back
and
in
those
types
of
those
types
of
things
built
into
this.
A
The
system
as
well
so
kind
of
an
interesting
one,
especially
if
you're
not
already
like
completely
bought
into
service
smash,
land
and
you're.
Not
already
trying
to
do
this
with
you
know,
ISTE,
o
or
linker
D,
or
something
this
could
be
a
could
be
a
pretty
interesting,
interesting
thing
to
check
out
cool
another
one
that
was
really
cool
that
is
reading
through
George
posted
inside
of
the
slack
of
D.
That
I
thought
was
really
slack
him
D
hackin
D
that
I
thought
was
really
cool.
Was
the
Java
in
a
container
resource
allocation
guidelines?
A
So
you
know
some
of
us
are
really
lucky
and
we
live
in
a
place
where
we
just
create
a
statically
linked
binary,
and
we
put
that
in
a
really
minimal
container,
and
then
we
ship
it
and
that's
it.
But
reality
is
you
know.
Most
of
the
world
is
not
doing
that.
Most
of
the
world's
using
I
don't
yeah.
Most
is
probably
fair,
like
most
people
are
using
some
type
of
runtime,
along
with
their
application,
and
the
JVM
is
an
extremely
common
one.
A
B
A
Right
inside
of
a
container
I
should
say
you
know
the
common
things
that
I've
seen
customers
run
into
so
much
like
you
don't
want
to
go
in
and
set
your
heap
space
explicitly
on
on
a
JVM
level.
You
want
your
your
java
application
to
be
able
to
respect
the
limits
that
you
set
up
inside
of
the
container
right,
because
you
don't
want
that.
The
multiple
levels
there
to
conflict
with
one
another,
there's
a
really
interesting
one
to
about
some
parameters
that
you
can
set
on
it
around
min
and
Max
RAM
usage,
particularly
you
know.
A
An
interesting
point
is
that
by
default,
the
JVM
heap
gets
25%
of
the
containers
memory
and
if
you
want
to
not
explicitly
go
in
and
set
the
the
JVM
parameters
to
explicitly
set
the
memories,
you
can
go
in
and
set
the
percentages
relative
to
what
would
propagate
from
the
limits,
if
that
makes
sense
right
so
kind
of
an
interesting
one.
If
you,
if
you
play
in
the
Java
world-
or
maybe
you
do
kubernetes
and
there's,
there's
Java
SH,
you
know
teams
Java
shops
in
your
organization.
This
might
be
a
really
cool
one.
A
To
kind
of
look
at
it's
always
interesting
to
to
get
into
the
weeds
for
me
again
to
get
out
of
my
my
world
of
statically,
linked,
binaries
and
and
so
on.
So
that's
really
cool.
This
is
a
one
that
we're
really
excited
about.
Of
course,
selfishly
Steve's
on
the
Steve's
on
the
the
line
right
now
so
congrats,
Steve
I,
know,
you've
been
you've
been
putting
in
work
both
historically
and
currently
to
contour.
A
So
contour
is
an
ingress
controller
that
is
based
well,
I
should
say
it
integrates
with
envoy
it
uses
envoy
as
the
technology
to
actually
send
traffic
into
pods
that
has
finally
reached
one
dot.
Oh
one
of
the
kind
of
unique
things
about
contour
for
those
of
you
who
haven't
checked
it
out.
Is
it
leverages
a
CR
D
that
allows
you
to
express
really
common
traffic
patterns
that
you
might
need
to?
A
A
Sorry
to
say
this
to
you
all,
but
some
of
the
values
of
those
annotations
were
JSON.
It
was
totally
crazy,
so
imagine
like
you've
got
a
yeah
mole
key
and
then
the
value
of
that
is
some
JSON
structure
right.
So
it's
it's
super
interesting
how
it
all
works
out,
but
one
of
the
cool
things
about
contour
again
is
they
kind
of
got
ahead
of
this
curve
and
did
a
really
good
job
of
defining
a
CR
D,
which
is
now
called
HTTP
proxy?
A
Those
of
you
who
use
contour
back
and
day
you
may
be
familiar
with
ingress
route
in
with
HTTP
proxy.
You
can
do
a
lot
of
fancy
things
in
a
structured
way,
a
lot
of
it
more
advanced
kind
of
forwarding,
type
things
and
rules
around
TCP
proxying
and
where
to
terminate
your
TLS,
you
wonder
on
the
pod
level,
do
you
wanna
do
it
on
the
load
balancer
level
right?
It
just
supports
those
more
intricate
configurations,
and
then
you
know
last
thing:
I'll
mention
about
contour.
A
If
you
do
check
this
out,
is
it
does
have
a
bunch
of
really
interesting
multi-tenant
capabilities
built
in?
That's.
That's
quite
neat,
because
you
know
a
lot
of
times
when
you
all
run
your
kubernetes
cluster
is
at
an
organization.
You
might
have
like
a
fleet
of
ingress
controllers
that
serve
most,
if
not
all
the
pods
and
workloads.
So
it's
like
a
multi
tenant,
ingress
layer
right
so
a
lot
of
times.
A
You
want
the
ability
to
make
sure
that
when
people
apply
ingress
resources,
they're
asking
for
their
specific
domain
name
or
their
specific
path,
right
or
maybe
it's
TLS
stuff-
and
you
want
to
make
sure
people
are
using
using
only
their
certificates
right
when
they
set
up
their
ingress
resources
and
so
on.
So
there's
a
lot
of
kind
of
multi
tenant
features
built
into
contour
that
helped
you
solve
those
types
of
problems,
so
super
psyched
to
say
that
it's
one
dot,
oh
great
job,
contour
team,
Steve
on
the
on
the
line
too.
A
If
you've
used
contour,
give
them
a
congrats
because
I
know
they
put
some
blood,
sweat
and
tears
into
that
thing,
and
it's
it's
really
really
cool
and
then
the
last
I
guess
I
have
two
more
two
more
small
pieces
of
news.
Scaffold
is
GA.
I'm,
not
gonna
lie,
haven't
gotten
the
chance
to
use
scaffold
yet
but
I
think
conceptually
it's
a
it's
a
great
idea.
Anything
to
help
make
the
developer
experience
a
top
of
kubernetes
more
straightforward.
A
I
am
a
massive
massive
fan
for
our
fan
of
so
check
out
scaffold
if
you
haven't
yet
it's
it's
GA
now.
So,
if
you're
kind
of
thinking
about
like
the
dev
axe
experience
around
kubernetes,
this
could
be
a
cool
project
to
evaluate
and
see
what
it
can
offer
for
you
and
then
the
last
chunk
of
news
I've
got
for
y'all
on
this
lovely
Friday,
one
that
I'm
kind
of
excited
about
it's.
A
It's
kind
of
a
more
of
a
directional
thing,
I
guess,
but
there
was
a
pull
request
that
got
put
in
and
I
and
I
haven't
looked
into
it
and
super
great
depth.
But
it
looks
like
we're
starting
to
do
some
work
to
make
namespace
deletion,
or
at
least
the
things
that
are
happening
behind
the
scenes
when
a
namespace
gets
deleted
more
elegant,
now
again
I'm
not
going
to
commit
to
whether
this
exact
pool
request
helped
significantly
with
that,
but
a
lot
of
times.
What
I've
seen
is.
A
You
know
when
you
go
to
delete
a
namespace
there's.
A
lot
of
these
controllers
that
are
operating
on
things
and
and
the
operator
and
the
controllers
that
are
operating
on
things
will
see
that
the
namespace
is
deleting,
or
at
least
they'll,
get
an
error,
because
the
namespace
is
being
deleted
and
then
get
into
some
kind
of
cycle
and
there's
some
weirdness
and
it
can
either
like
slow
down
the
name
space
creation
altogether.
A
A
You
know
I
I
had
a
an
error
and
that
error
was
source
from
the
fact
that
I,
my
name
space
was
terment
terminating
from
what
I've
read
very
briefly
in
the
code
before
this.
Before
this
presentation
I,
it
seems
like
it
kind
of
exits,
quicker
I'll
put
it
that
way
right.
So
that's
true
that
reduce
a
little
bit
more
stability
for
us
long
term,
I'm,
looking
forward
to
testing
that
out
and
kind
of
seeing
what
it
has
to
offer
so
yeah.
A
That's
again,
that's
it
for
news
check
out
chaos
cube
for
chaos,
engineering,
stuff
check
out,
Go,
Daddy's,
gated
deployment
tools,
if
you're
interested
in
doing
kind
of
like
a
be
like
deployments
in
an
automated
fashion,
people
in
the
Java
world
or
have
consumers
of
their
key
platform
there
in
the
Java
world
check
out
that
rent
run
time,
resource
contours,
1-0
Congrats
on
that
scaffold
is
now
GA
and
then
a
really
cool
pool
requests
on
namespace
deletion.
So,
let's
see
if
we
had
anyone
else
joins
since
I.
Looked
last
Ramesh.
B
A
Don't
know
if
I
saw
your
name
last
time
from
SF
welcome
super
glad
to
have
you
Herman
yeah
thanks
yeah
we're
happy
to
be
using
your
idea,
so
so
shout
out
to
you.
This
came
from
a
issue
in
our
TGI
K
github
right
so
for
those
of
you
who
maybe
haven't
visited
it
in
a
while.
Let's
see
if
I
can
pull
it
up
quickly
here
we
do
genuinely
try
to
to
to
look
through
and
groom
these
things
right.
A
So,
if
you're
ever
interested
in
an
idea,
you
know
if
you're
interested
in
Joe
and
Duffy
you
just
like
diving,
deep
into
some
really
cool
topic
that
you're
interested
in
add
an
issue
to
the
to
the
github
page.
Here
this
is
where
I
got.
The
idea
for
cluster
autoscaler
I
was
literally
just
woke
up
one
morning:
grab
some
coffee
scrolled
through
it
and
saw
somebody
requested.
Oh
hey,
I
want
to
do
cluster
autoscaler
I
want
to
hear
about
it
right
and
I
thought
it's
a
brilliant
idea.
A
I
talked
to
customers
about
that
all
the
time,
so
you
know
add
tickets
in
here.
If
you
have
some
ideas,
we
love
the
feedback,
so
cool
cool
other
people
we've
got
in
we've
got
philip
from
germany.
Welcome
philip
Hameed
hamid
from
ireland,
jason
from
my
team
as
well
Jason's.
Also
a
Colorado
person
down
in
Denver,
so
he's
a
little
bit
south
of
me.
Welcome
Jason,
but
Jordan
from
Belgium
welcome
and
yeah.
So
and
abc123.
That's
a
great
point.
We
need
to
I.
Guess,
I
wasn't
being
super
honest
in
the
grooming
aspect.
A
A
I
did
like
PSPs
and
network
policies
which,
for
some
reason,
like
I've
done
with
almost
all
my
customers,
like
it's
an
exhaustively
written
in
my
brain
right,
and
this
one
like
I
had
mentioned
to
you
early
on,
is
like
something
I've
talked
to
customers
about,
but
I
just
haven't
gotten
a
chance
to
like
do
it
myself.
So
I
think
you
know.
A
I
set
up
some
AWS
infrastructure
for
us
to
test
this
all
out
with
and
we're
just
gonna
play
around
and
see
if
we
can
bring
these
concepts
together,
so
chat,
I
would
love
your
thoughts
and
feedback.
Some
of
you
have
kind
of
done
this
stuff
before
too.
So
you
know
don't
be
shy,
speak
up
and
chat,
I'd
love
as
much
involvement
as
I
can
and
speaking
of
involvement.
Let's,
let's
start
off
with
a
quick
question.
So
so
my
chat
people
out
there.
A
Okay,
who
has
used
some
type
of
auto
scaling
so
HP
a
VP,
a
or
cluster
autoscaler.
So
if
you
want
to
just
put
in
here
for
an
answer,
say:
I've
used
X
for
one
of
them
or
you
could
say
new
to
all
or
some
variation
of
that.
So
so
we'll
take
a
moment
here.
Chat
kind
of
give
me
some
background
who
here
is
totally
new
to
the
world
of
auto
scaling
and
cube,
and
then
who
has
used
one
of
these
three
horizontal
pod
autoscaler,
which
is
HP
a
vertical
pod,
autoscaler
and
then
cluster
autoscaler,
which.
A
Okay,
some
good
backgrounds,
that's
that's
cool
ABC
as
well
on
the
custom,
auto
scalar
based
on
CPU
usage,
to
add
more
notes
to
the
cluster
I'll
be
super
interested
when
we
get
into
scaling
our
nodes
out
kind
of
how
that
compares
and
contrasts
with
kind
of
like
the
upstream
approach,
if
you
will,
we
have
a,
we
have
a
really
interesting
direction.
This
is
all
headed
and
I'll
wrap
up
this
talk
with
it
around.
Like
I,
don't
know
if
all
of
you
have
kind
of
seen.
B
A
Cluster
epi
is
taking
lifecycle
management
with
kubernetes,
but
for
those
of
you
who
know
Matt
about
cluster
API,
now
I'll
elaborate
on
this,
but
if
we
do
cluster
API
right
and
it
becomes
like
the
ubiquitous
way,
we
manage
kubernetes.
It
actually
has
a
really
interesting
impact
on
how
auto
scalars
can
be
built
and
I'm
super
excited
to
kind
of
brainstorm
that,
with
you
all
cool,
so
yeah
got
a
hugely
mixed
audience,
which
is
great.
A
B
A
That'll,
be
that'll
be
great,
so,
as
we
dive
in
here,
you
know,
feel
free
to
hop
in
chat
and
add
some
context
and
if
I,
if
I
say
something
silly,
please
please
jump
in,
don't
hesitate,
alright
cool,
so
we've
got.
We've
got
a
really
good
audience
here.
I'm
selfishly
just
excited
to
play
with
this,
so
let's,
let's
do
it
and
we'll
try
to
wrap
up
our
Friday
on
some
amount
of
success.
Okay,
so
couple
topics
I
want
to
get
over
in
full
transparency.
A
My
last
couple,
TGI
K's
kind
of
just
try
to
pack
way
too
much
into
them.
I
think
so.
My
thought
process,
especially
with
our
mixed
audience,
I'd
love
to
get
everyone's
expertise
here.
Right,
we'll
start
off
with
just
like
scaling
at
a
high
level,
we'll
talk
about
resource
scheduling
because
it
heavily
impacts
how
these
scaling
models
work,
we'll
get
into
HP
a
we'll
get
into
VP
a
we'll
get
into
a
cluster
autoscaler.
Also.
A
Metrics,
and
things
like
that.
So
I
was
thinking
for
this.
First
one
will
lay
the
foundation
with
like
scaling
in
general
and
then,
if
it's
a
hit
and
if
you
all
like
it
to
some
degree,
we
can
have
a
follow-up
session
where
we
dive
into
like
integration
with
metrics.
You
can
scrape
like
Prometheus
like
metrics,
right
and
and
how
you
can
get
more
fancy
and
kind
of
tuning.
A
All
of
all
of
these
bits
cool,
so,
let's,
let's
dive
in
cool
all
right,
oh
wow,
did
you
hear
that
Alexa
in
my
apartment
is
talking
to
me
so
cool?
Alright,
let's,
let's
dive
in
on
then
we'll
look.
It
we'll
take
it
from
there.
So,
let's,
let's
kind
of
lay
out
some
context
here
now,
I
I
wanted
to
hook
my
iPad
up
again.
So
I
could
draw
things
with
you
all
you
might.
You
might
have
seen
some
of
my
more
childish
drawings
that
I've
done
in
the
past.
A
Forty
gik,
but
I
lost
my
Apple
pencil,
which
is
terrible
so
high
I
can't
do
that
so
we'll
we'll
use-
and
we
will
will
draw
some
also
child,
looking
drawings,
just
via
just
via
this
method
here,
okay,
so
just
kind
of
laying
the
land
for
us
all
for
just
a
moment
here,
alright.
So
if
I
kind
of
come
into
this,
the
screen
box
and.
B
A
Think
about
scaling
as
a
whole
right,
so
we
know,
kubernetes
is
composed
of
a
bunch
of
these
worker
nodes
right,
that's
kind
of
a
primary
principle
of
kubernetes.
Alright,
and
you
know
you
might
stand
up
a
cluster
and
I'd
be
willing
to
bet
that
your
cluster
has
a
lot
more
nodes
than
my
sample.
One
does
here,
but
let's
assume
that
we
started
off
with
with
two
worker
node
two
worker
node
cluster
right.
So
so,
basically,
what
we're
hoping
to
kind
of
think
about
here
is:
can
our
cluster
be
more
elastic?
If
you
will
right
so?
A
Can
we
take
our
cluster
and
based
on
scheduling
constraints,
determine
when
we
need
to
add
more
nodes,
add
more
nodes,
add
more
nodes.
Now
kind
of
a
piece
of
this
is,
of
course,
the
the
workload
component
that
will
drive
that
right.
So
if
we
think
about
the
the
workload
again,
I
told
you,
these
drawings
would
be
bad
today,
but
that's
okay.
These
workloads
are,
of
course,
the
pots
and
most
of
us
we
know
this
super
super
well
and
the
pods
can
have
kind
of
like
these.
A
A
Just
implies
I'll,
throw
you
on
a
host
where
I
perceive
there's
room,
use
some
sea.
You
use
some
memory
but
hey.
If
the
host
comes
under
contention
with
other
things
that
have
reserved
that
memory
and
potentially
CPU
CPUs
a
little
more
complex,
but
at
least
memory
you
might
get
booted
or
you
might
get
killed,
or
something
like
that
right.
So
these
pods
get
created
and
as
the
pods
get
scaled,
you
end
up
with
the
need
to
kind
of
create
these
new
pods
here
right,
which
I
will
draw
again
poorly
with
our
dotted
our
dotted
lines.
A
Okay,
so
this
is
our
our
other
pot.
Now
this
is
a
dotted
pod
because,
as
you
can
tell,
this
box
does
not
fit
in
our
other
box
right.
So
what
we
would
need
to
do
here
if
we
were,
if
we
were
just
in
a
one
worker
mode,
is
we
would
need
to
realize
hey
you
know,
I
am
I,
am
one
CPU
short
right,
so
one
CPU,
alright
and
or
you
know
some
amount
of
memory
short
or
whatever
right,
and
then
the
job
of
the
cluster
autoscaler,
of
course,
on
the
on
the
cluster
level.
A
A
That's
super
obvious,
but
you
know
that
kind
of
that
kind
of
sets
the
stage
for
for
what
we're
trying
to
achieve
here.
Right
now,
you
could
imagine,
while
conceptually
simple,
there's
a
whole
potential
set
of
challenges
right.
There's
things
like
how
do
I
make
sure
overnight
when
my
my
team's
Java
app
freaks
out
that
I
don't
wake
up
to
a
1,000,
kubernetes
cluster
right.
A
Additionally,
one
thing
we
hadn't
really
thought
about
is
like
what's
making
these
pods
kind
of
scale
out
right,
because
if
you
think
about
it
as
our
as
our
friendly
users
over
here
right
kind
of
send
requests
into
the
pots.
You
know,
if
we're
thinking
about
like
a
simple,
a
simple
application
where
you
know
if
traffic
comes
in
it
does
something
it
sends
a
response.
Well,
maybe
it's
starting
to
get
overloaded
right
and
if
it's
starting
to
get
overloaded,
then
maybe
we
can
alleviate
the
load
by
scaling
the
pots.
A
So
you
can
probably
see
where
this
is
going
right.
Your
pods
start
scaling.
They
eventually
get
to
the
point,
as
we
saw
down
here
in
the
bottom.
They
eventually
get
to
the
point
where
they
don't
have
the
available
infrastructure
to
land
on,
and
this
is
where
the
ca
can
kind
of
take
over
okay.
So
there's
there's
one
more
thing
that
I
just
want
to
throw
out
there
with
this
diagram
too,
because
it
sometimes
causes
a
point
of
confusion
right
in
AWS.
A
As
many
of
you
know,
you
have,
let's
see
if
I
have
my
ec2
console
still
open,
I'll
show
you
all.
You
have
Oh
perfect,
auto
scaling
groups
right,
okay,
so
the
idea
with
AWS
auto
scaling
groups.
This
is
one
of
my
node
pools
right.
I've
got
a
min
account
of
nodes
right
now,
which
is
one
I've
got
a
max
count
of
nodes,
which
is
also
one
all
right
now,
probably
not
a
actually.
No
it's
five
right
now.
Isn't
it
okay
cool?
So
my
min
is
one.
My
max
is
five.
A
Now,
if
I
were
to
go
in
here
and
set
some
constrained
stuff,
I
could
say
things
like
if
I
reach
certain
amount
of
CPU
right,
I
want
to
scale
my
host
out
I
want
to
add
more
hosts
and
then
maybe
in
Reverse
like
if
I
have
enough
idle,
maybe
I
want
to
like
collapse
them
to
some
degree.
Maybe
you
do
something
similar
with
other
metrics
as
well,
but
you
know
the
thing
to
make
sure
you
have
your
mind
wrapped
around
with
cube.
A
A
When
you
look
at
your
cluster
here,
your
pods
might
get
added
right
and,
like
we
may
be
aware
and
I'll
talk
about
pods
have
the
ability
for
you
to
say,
give
me
the
sea
poops
much
CPU.
Give
me
this
much
memory,
it's
especially
important
when
you're
running
you
know
multi
team
or
multi
tenant
clusters.
So
if
you
start
scheduling
a
bunch
of
pods,
the
scheduler
is
gonna
kind
of
look
into
kubernetes
and
be
like
alright.
I
have
room
for
you
here.
Okay,
this
pod
wants
you.
B
A
Two
CPUs
and
five
gigs
of
memory
I
carve
it
out
I
place
you
here,
so
the
scheduler
and
the
controller
components
they
know
about
this
right.
They
can.
They
can
kind
of
go
okay,
cool
I,
know
where
you
fit
in
here,
but
if
you
just
have
auto
scaling
mechanics
at
that
higher
level
like
in
the
case
of
just
an
auto
scaling
group,
you
might
actually
be
at
capacity
right,
but
you're
not
actually
seeing
the
utilization
to
trigger
the
scaling
event
right.
A
So
the
example
we're
at
where
I'm
headed
here
is
you
schedule,
100
pods
right
and
those
pods
are,
on
average,
only
using
five
percent
of
their
their
requested
CPU.
The
scheduler
doesn't
want
to
schedule
anything
new
because
it
doesn't
think
it
has
room.
But
if
you
look
at
your
nodes,
your
average
node
is
maybe
like
10
to
15%
CPU
usage
right.
So
the
key
thing
with
these
Auto
scalars
is
they
live
in
the
cluster
right
like
anything
else,
so
the
idea.
B
A
If
we
deploy
this
thing,
it's
it's
cognizant
of
cube
from
an
API
perspective
right,
so
it
understands
the
schedulers
hitting
limits
it
understands
and
it
knows
how
to
scale
based
on
those
things.
So
that's
kind
of
our
our
high-level
overview
here,
all
right
and
I'm,
just
gonna
just
gonna.
Take
a
quick
look
at
chat,
real
quick
here
and
ABC.
We
took
total
ratio
of
total
cluster
resource
usage
availability
divided
by
the
number
of
nodes
in
the
cluster
and
scale
based
on
that
interesting,
okay,
yeah
I
see
what
you're
doing
there
so
you'll
see.
A
Obviously
a
simplified
model
of
that
today,
I'll
be
interested
once
we
talk
more
about
kind
of
cluster
API
how
that
works,
and
if
you
can
disclose
it
to
a
VC
I'd,
be
curious.
What
what's
your
provider
under
the
fighter?
Are
you
scaling
on
something
like
Amazon,
or
is
this
like
internal
infrastructure
to
some
degree
like
vSphere,
or
something
like
that?
That
made
a
pretty
interesting
piece
of
data
AWS.
A
B
A
Let's,
let's
take
this
a
step
deeper,
we've
got,
we've
got
our
conceptual
overview
here.
Alright
are
very
crude
conceptual
overview,
so
the
world
of
scaling
will
call
that
we'll
call
that
good
now
resource
scheduling
alright.
So
this
is
where
we're
gonna
we're
going
to
kind
of
start
using
an
example.
The
first
thing
that
we
need
to
do
is
we
need
to
have
an
application
that
we
can
use
to
just
sort
of
simulate
this
right
now.
So
the
first
thing
I
want
to
show
you
all.
A
Let
me
get
a
move
this
guy
over
to
9
and
then
open
up
a
terminal
here,
all
right,
cool,
great
all
right.
So
the
first
thing
I
want
to
do
with
you
all
is
just
have
kind
of
a
sample
application
that
we
can.
We
can
start
playing
around
with
now.
I
I
went
ahead
and
built
some
of
this
stuff
out,
just
because
I
didn't
want
to
waste.
The
majority
of
this
having
you
watch
me
stumble
through.
A
Writing,
go
so
I
just
wrote
a
really
simple
application
that
will
do
what
we
need,
but
to
give
you
some
quick
context
on
the
cluster.
Let's,
let's
see
what
we
got
here
cool,
so
nothing
has
changed.
What
I
can
tell
I've
got
a
cluster
running
in
Amazon
right
now,
two
node
cluster,
as
you
can
tell
by
the
rules,
I,
have
got
a
master
node
here
and
I've
got
a
worker,
node
plain
and
simple.
Another
thing
that
we
can.
We
can
look
with
this
data.
You
just
get
that
name
again.
A
Right
is
if
you
were
in
kubernetes
and
you
wanted
to
know
like
hey.
What's
what's
available
to
my
to
my
node
right,
we
could
go
in
and
we
could
do
a
cute
cuddly
at
notes.
We
could
ask
for
this
specific
node
right,
actually,
an
even
easier
version,
I
think
to
look
at
it
is:
let's
just
do
describe
node
right
and
IDO
is
a
short
for
node.
So
if
we
do
describe
node
towards
the
bottom
here,
you
can
actually
see
some
of
the
allocated
resources
that
are
available
to
us.
A
Don't
judge
me
if
my
math
is
terrible
but
I
think
we're
somewhere
in
the
room
of
one-and-a-half
CPUs
right
and
then
obviously
I'm
using
some
stuff
up
with
like
system
resources,
you
can
see
actually
just
look.
Look
a
little
bit
higher.
There
we've
got
Canal
node,
we've
got
cube.
Dns
we've
got
cube
proxy.
A
lot
of
vendors
will
ship
their
containers,
their
workloads
with
with
the
allocated
CPU
and/or
memory
set
in
the
manifest
and
I
personally
kind
of
think.
That's
the
best
practice
like
because
you.
A
The
workload
in
especially
something
that's
as
critical
is
like,
in
this
case
Canal
node,
which
is
going
to
be
working
on
the
networking,
and
you
want
to
know
that
it
has
a
consistent
amount
of
resource
available
to
it.
No
matter
where
you
deploy
that
way,
someone
doesn't
deploy
it
and
it's
best
effort
and
being
thrown
off
and
all
that
all
that
kind
of
stuff.
So
cool
we've
got
yes,
we've
got
it.
We've
got
some
resource
on
this
node.
That's
that's
the
simple
way.
To
put
it
all
right.
So,
let's
look
at
our
our
workload
here.
A
Should
I,
what
should
I
put
together
real,
quick,
that
that'll
produce
some
load
inside
of
inside
kubernetes
and
and
as
one
would
do,
let's
just
make
a
loop
that
counts
up,
assigns
of
to
a
value
as
it
counts
up
we'll
have
it
print,
and
then
we
will
have
it
go
ahead
and
and
write
a
response
to
us
right
and
actually
that
that
right
line
is
totally
wrong.
I
didn't
update
it,
but
it
doesn't
matter
we
can.
We
can
get
jump
back
on
more,
so
I'm,
more
so
interested
in
the
status
code.
A
We're
gonna
get
back,
so
those
of
you
that
aren't
as
familiar
with
go.
You
know
just
think
of
this,
like
a
web
server
that
will
receive
traffic
and
every
request
it
receives,
it's
going
to
count
to
this
number
right
and
then
it's
gonna
respond
to
the
clients
synchronously,
so
count
all
the
way
up
respond
to
the
clients
and
what
we
should
see
if
all
goes
well
here
is,
as
we
start
sending
some
load
into
this
thing.
We
should
theoretically
see
some
some
usage
go
up
from
like
a
CPU
standpoint.
In
this
case.
A
Will
be
very
CPU
focused
in
this
in
this
use
case,
all
right,
okay,
so
the
only
other
thing
I
want
to
show
you
before
we
start
deploying
in
and
we
get
to
the
part
that
I'm
interested
in
which
is
trying
to
scale
this
thing.
I
want
to
show
you
the
the
the
metric
server.
So
as
we
as
we
kind
of
move
along
here,
we're
gonna
be
looking
at
different
ways.
You
can
determine
whether
you
need
to
scale
right
and
there's
actually
a
couple
different
ways.
You
can
kind
of
integrate
this
right.
A
So
if
I
do
cube,
cuddle
get
pods
for
the
namespace
cube
system,
make
it
a
little
bit
smaller.
Here,
it's
still
not
small
enough.
Let's
try
that
beautiful,
you'll
notice
that,
in
this
sample,
cluster
I
have
deployed
a
metrics
server,
okay,
so
for
for
lack
of
a
lack
of
a
better
description
for
right
now.
Think
of
this
as
a
required
component
to
propagate
metrics
about
things
that
are
running
in
my
cluster,
do
you
might
be
familiar
with
man
I'm
already
forgetting
a
name
keep
stir
it's.
A
Heap
stir
that
used
to
do
similar
things
as
metric
server
does
but
I'm
running
metric
service
server
in
the
cluster,
which
is
enabling
me
to
do
some
things
that
I
want
to
be
able
to
show
you
all.
You
know
an
example
of
that
being:
let's
do
cute
cuddle,
get
nodes
again
right
and
if
I
grab
this
this
worker
node
right
here
command,
you're
gonna
see
me
do
a
lot
through
today.
Yes,
assuming
it
works,
okay
for
us
is
you're.
Gonna
see
me,
do
cube
cuddle
top
node
leave
and
then
just
the
node
name.
A
Is
that
right?
Okay,
great?
So,
let's
make
a
cleaner
here.
So
if
I
do
this
request,
this
is
going
to
give
me
some
data
about
the
actual
nodes
so
kind
of
similar
to
to
what
we
had
seen
before.
So
it's
letting
me
know:
hey
you've
got
this
node.
Okay,
you've
got
about
three
percent
CPU
usage
right
now,
right.
A
To
be
to
be
clear
right
you
this
these
numbers,
don't
add
up
the
numbers.
We
were
looking
before
cuz.
Those
were
like
the
the
reserved-
and
this
is
my
my
usage
right
so
because
I
have
metrics
server,
running
I'm
gonna
be
able
to
run
these
commands
and
just
get
information
about
things
like
a
knows,
usage
I
should
also
be
able
to
go
in
and
do
let's
grab
a
random
pod
from
cube
system,
real
quick.
So
if
I
grab
Canal
master,
we
will
do
a
top
on
that
pod.
A
Okay,
and
we
will
put
the
pod
name
in
here
and
now
we
can
see
our
usage
from
a
pond
level
right.
We've
got
Canal
master,
12m,
CPU
usage,
sixty-eight
mi
memory
usage
as
well,
alright.
So
what
we'll
want
to
do
here
is
get
this
app
deployed.
That
I
showed
you
and
then
use
commands
like
this
use
the
data
out
of
the
metric
server
to
see
how
it
gets
impacted
from
all
this
good
stuff.
Alright.
So
so,
let's
see,
if
we
can,
we
can
deploy
this
real,
quick
in.
A
Interesting
data
from
the
metric
server
to
compute
that
Pro
Oh
program
stress:
what's
what's
that
project?
Is
that
a
way
to
give
metrics
or
is
that
a
way
to
like
stress
test
things?
I
know
if
I
don't
know
if
I've
come
across
that
one
to
be
honest
with
you
mystic
the
terminal
I'm
using
is
our
xvt
Unicode,
it's
one
that
runs
on
Linux.
So
it's
just
like
a
super
basic
terminal
that
takes
a
massive
text-based
configuration
file.
I
love.
It,
though,
because.
A
Can
kind
of
tune
it
the
way
that
you
know
stress
past
okay,
that
that
makes
total
sense,
so
you're,
gonna,
you're,
gonna,
get
get
to
see
me
use
the
lovely
tool
jmeter
today
for
those
of
you
who
remember
jmeter
from
you
know
doing
a
lot
of
java
based
work.
I
I
just
installed
it
today.
You
might
have
seen
and
it
like.
It
actually
gave
me
a
nostalgia
opening
up
jmeter.
It's
been
so
long
since
I've
used
jmeter.
That
I
was
just
like
you
know
it
actually
installed
and
try
using
it
for
this.
B
A
A
We're
not
doing
anything
fancy
here
right.
So
if
we
did
like
for
I
in
something
or
like
you
said,
while
and
bash
all
we
have
to
do
is
say
curl
this
thing
and
like
you
can
even
spawn
multiple
threads.
If
you
did
curl
blah
blah
blah
right
like
for
in-1
one
dot
dot,
it's
been
a
while
since
I've
done
these
types
of
commands.
So
let's
see
if
it
even
works,
do
curl
google.com,
we'll
DDoS
Google,
here
Google
comm
and
then
you
could
spawn
off
theoretically
40
threads
right.
A
If
you
did
ampersand
done
and
I
think
great
yeah,
so
you
can
have
like
40
concurrent
clients
to
test
out
HP
a
to
some
degree
right.
So
you
don't
have
to
do
anything
fancy
here,
but
that's
that's
a
great
call!
Maddie
you
could
you
could
keep
it
as
simple
as
bash.
You
could
be
as
fancy
as
using
some
tool
as
well,
so
great
great
call-outs,
everyone
all
right.
Let's
deploy
this
terrible
application
and
see
how
it
goes.
What
do
you
say?
A
Alright,
so
I
did
throw
it
appoinment
together
and
I
was
messing
around
with
it
a
bit
last
night.
So,
let's,
let's
talk
about
this
real,
quick,
okay,
great
yeah!
This
is
this
is
pretty
much
the
crux
of
it
all
so
again,
I
just
wanted
to
save
some
time
with
like
the
container
image
stuff,
but
but
we'll
start
here
as
our
as
our
starting
point
and
we'll
just
build
from
here.
A
So
just
you
know
just
to
make
sure
you
don't
feel
like
I'm,
like
you
know,
doing
something:
fancy
behind
the
curtain
here,
I've
uploaded
that
go
file
as
a
static,
binary
and
I
think
in
either
scratch
container
or
an
Alpine
Linux
container
I
put
it
up.
I
think
docker,
definitely
docker
hub
based
on
the
image
tag
and
just
put
it
there.
Okay,
so
the
things
that
I've
started
off
with
I've
called
this
app
counter
and
I'm
gonna
be
using
contour
during
this
demo.
So
don't
get
the
to
confuse.
A
This
is
the
counter
app,
not
the
contour,
ingress
controller.
Okay,
very
simple,
as
you
probably
saw
in
the
go
code
and
I've
done
the
same
thing
on
the
container
level,
it
will
expose
port,
8080,
okay
and
then
we'll
have
a
resource
limit
and
and
something
that
maybe
we
should.
We
should
pause
for
a
moment
to
make
sure
all
on
the
same
page
and
see
if
you
all
have
any
questions
with
this
right.
A
What
I'm
gonna
do
to
sample
CPU
is
I'm
gonna
set
a
limit
of
250
milli
CPU.
Is
that
how
you
say
it?
Someone
correct
me
if
I'm
wrong?
So
basically
you
can
go
to
the
communities,
dock
and
learn
about
these
different
metrics,
but,
like
that's
a
TGI
K
in
and
of
itself,
in
my
opinion,
think
of
a
thousand
as
a
CPU.
If
memory
serves
correctly
yeah,
a
thousand
should
be
a
CPU,
so
I'm
using
like
a
quarter,
I
guess
of
a
CPU
or
at
least
requesting
that
much.
A
With
you
all
to
see
kind
of
your
familiarity
here,
okay,
so
let's
it!
Let's
do
this
so
chat.
Tell
me
who
has
used
requests
in
kubernetes
and/or
limits?
Okay.
So
what
I
want
you
to
respond
with
here
is
tell
me:
I've
used
requests
before
I've
used
limits
before
or
I've
used,
both
or
I've
used.
None.
So
just
so,
I
have
a
kind
of
a
context
of
where
you're
all
out
so
again
answer
in
chat.
For
me,
who's
used,
requests
in
kubernetes,
Hughes
used
limits,
who's
used
both
and
then
who
is
used.
A
A
A
Of
these
things,
so
just
just
to
just
to
kind
of
lay
lay
the
land
out
here.
Right
cube
is
gonna.
Let
us
do
cube
is
gonna.
Let
us
do
CPU
memory
and
more
to
some
degree
and
specify
requests
in
limits
right.
So
it's
more
intricate
than
what
I'm
going
to
describe
here,
but
I
just
want
to
lay
the
foundation
right
for
requests.
You
kind
of
think
of
this
as
like
what
the
scheduler
should
carve
out.
A
A
With
requests
in
limits
is
generally,
requests
are
enforced
at
runtime
right
from
like
a
C,
groobie
type
level.
Now,
there's
actually
some
inaccuracies
here
for
those
of
you
who
are
like
really
deep
in
this
stuff,
but
for
simplicity,
sake
like
the
limit
on
a
CPU
would
say:
okay,
you
requested
a
quarter
CPU.
If
you
start
ramping
up
your
usage,
I'm
gonna
be
an
upper
bound
that
tries
to
make
sure
I
keep
you
on
some
like
check
interval
underneath
or
at
that
bound
right.
A
So
the
the
limits
are
gonna
have
kind
of
an
impact,
whereas
if
you
just
had
a
request
and
you
scheduled
theoretically,
it
could
use
90%
of
the
CPU
if
it's
available
to
it
right
now.
Technically,
if
the
host
came
under
contention,
there's
things
that
would
happen
around
like
CPU
shares
that
would
ensure
that,
like
the
CPU
for
what
requested
it
would
get
it
and
things
like
that.
So
it's
a
much
more
deep
topic
like
you
should
check
it
out.
A
It's
there's
zalando
did
a
really
great
talk
a
long
time
ago,
or
maybe
a
year
ago,
I
don't
know
about
like
resource
allocation.
It's
if
you
Google's
the
Lando
resource,
request
limits
or
something
it's
a
fantastic
talk.
They
talk
about
their
struggles
with
it,
why
they
only
decided
to
use
requests,
I
believe
for
CPU
and
not
limits,
and
yet
it's
really
interesting
and
you
can
learn
about
like
how
limits
are
enforced.
A
With
this
thing
in
the
kernel
called
the
completely
fair
scheduler,
which
is
an
apt
name
for
for
what
it's
trying
to
do
is
trying
to
make
sure
that
you
don't
use
everything
when
you
can
and
keep
it
fair.
Any
there's
reasons
why
you
might
use
one
or
the
other
and
I
just
want
to
make
sure
you're
all
on
the
same
page.
A
Now
limits
is
what
I
decided
to
focus
on
here
right
and
the
reason
I
decided
to
focus
on
limits
is
because,
in
the
absence
of
requests,
my
requests
will
become
my
limit
because
think
about
that
for
a
moment
right,
if
you
set
a
limit
in
place-
and
you
say,
give
me
a
quarter-
CPU
limit
me
there,
but
you
don't
specify
a
request.
Well,
obviously,
it
makes
the
most
sense
to
just
have
the
request
equal.
A
B
A
Okay,
so
enough
enough,
talking
and
you're
welcome
to
tell
me
to
shut
up
if
I
go
if
I
go
off
course
on
these
things,
all
right.
So,
let's,
let's
see
if
the
deployment
even
works.
So
the
first
thing
we'll
do
we'll
open
up
a
bottom
buffer.
Here
you
all
probably
know
I'm
a
huge
fan
of
watches,
so
watch
cube,
cuddle
get
pods
for
the
name,
space,
TGI
K.
That
does
remind
me,
let
me
see,
if
I'm,
making
a
TGI
kein
namespace
here,
how
beautiful,
okay,
sometimes
sometimes
I,
think
ahead.
A
It's
rare,
it's
rare,
but
it
happens
so
keep
cuddle,
apply
deployment
right,
cool,
okay,
counter!
Please,
please
run
all
right,
so
we
have
got
the
the
counter
running
on
the
node.
Quick
and
easy
all
looks
pretty
good
cool.
So
now
that
it's
running,
we
should
be
able
to
start
sending
some
load
into
this
thing
and
trying
to
observe
what
it
has
available
to
it
right,
but
let's
just
start
off
by
trying
to
figure
out
what
is
available
to
it.
A
So
if
we
were
to
go
into
our
counter
here
right
and
we
were
to
say,
cube-
cuddle
get,
let's
do.
Let's
do
a
top
actually,
so
we
learned
that
we
can
do
top
pod
hit
this
send
in
the
namespace
for
TGI
kay,
and
currently
our
usage
is
0m,
so
you
know
it's
running
an
HTTP
server
and
go
which
I'd
imagine
is
pretty
lightweight
by
default.
We're
not
using
any
memory.
Nothing
like
that
or.
A
Lot
of
memory-
okay,
so
let's
do
that.
So
let's
make
another
watch
I
guess
so
in
my
top
buffer
here,
I'm
watching
and
then
in
my
bottom
buffer
here,
I
guess
the
section
isn't
that
important!
So
we've
got
our
we've
got
our
pod,
we're
looking
good
cool.
So
maybe
we
can
start
off
what
do
you
all
say
with
putting
some
load
in
here?
A
Okay,
so
there's
some
really
cool
cloud
native
eTools
to
do
this,
but
we're
not
going
to
use
that
we're
gonna
use
jmeter
right,
let's,
let's
boot
it
up
here,
you
all
y'all
are
gonna,
laugh
at
my
fiery
burning
obsession
with
jmeter.
It's
it's
actually
like
borderline
concerning
I,
probably
should
see
somebody
but.
A
To
have
any
of
you
use
jmeter
before
it
like
I,
don't
know
how
to
use
it
in
five
years
and
I
just
opened
it
up
and
was
like
I.
Remember
like
trying
to
figure
out.
Why?
Oh
why
this
JVM
app
was
slow
and
it
was
because
this
sinc
function
was
getting
called
when
jmeter
called
this
particular
API
like
I,
don't
know,
I
just
spent
a
lot
of
time
in
this
thing
and
I
hope
I
still
remember
how
to
use
it.
A
So
here's
what
we're
gonna
do,
and
this
could
be
kind
of
an
introduction
to
load
testing
tools.
If
you
haven't
used
one
before
that's
kind
of
cool,
so
most
load
testing
tools
have
an
ocean
of
users
or
threads.
They
usually
mean
the
same
thing.
So
to
kind
of
start
us
off
here,
we're
gonna
start
off
with
two
users.
Okay,
so
these
are
people
that
are.
My
computer
will
translate
to
threads,
not
real
people.
A
I,
don't
have
a
Steve
and
Jason
behind
the
scenes
like
making
API
requests
right,
but
this
will
want
to
hit
enter
spawn
off.
Ideally,
I
think
two
threads.
You
know
maybe
in
the
JVM
and
then
shoot
off
the
request
to
them,
so
our
loop
count
will
just
say
forever.
So,
as
we
start
sending
requests
in
and
testing
out
these
auto
scalars
we'll
use
that
okay
and
then
for
the
requests.
If
we
go
down
to
the
let's
see,
I
think
it's
a
sampler
right,
we'll
do
an
HTTP
request.
A
B
A
B
A
Let's
go
ahead
and
see
if
I
still
have
that
file,
contour
cool,
so
I'll
spare
you
the
gory
details
of
deploying
contour.
It's
not
that
important
for
what
we're
trying
to
talk
about
so
service
contour
created
namespace,
created,
okay,
good
good!
Let
me
see
what
my
what
is
my
contour
version
I'm
using
here.
Actually,
let
me
just
check
that
real,
quick,
okay,
that's
good!
That's
what
I
was
expecting
good
good.
All
right,
so
keep
cuddled
get
pods
for
the
namespace
tik,
okay,
great.
A
You
can
see
that
the
similar
names
now
here's
the
problem,
I
was
worried
about
so
contours
running
and
counter
is
running
now,
just
to
make
sure
you
you're
understanding
here,
contour
I've
got
it
deployed
with
host
port,
so
it's
kind
of
like
host
networking,
but
it
actually
like
is
a
little
different.
There's
performance
reasons
why
you
might
use
hosts
networking.
Nonetheless,.
B
A
I'm
trying
to
get
at
here
is
contours
gonna
be
attached
to
the
edge
of
my
note.
It's
gonna
consume
port
80
on
my
node,
and
traffic
is
gonna
kind
of
shoot
through
and
disperse
to
theoretically
counter
here,
so
I
also
put
up
an
ingress
to
save
you
some
time
from
watching
me
write
it
still
called
ping
yeah
Mille
yeah.
It
is
okay,
sorry,
so,
basically,
the
idea
here
is,
if
you
send
a
request
into
this
domain
name
on
this
path,
we
will
send
that
request
into
any
pods
that
are
selected
by
this
service.
A
Okay,
so
check
contour
receives
traffic,
sends
it
to
the
pots
plain
and
simple.
So
we'll
do
a
cube.
Cuddle
apply
to
ping
yeah
Mille
again,
that
should
be
probably
called
ingress,
yeah
Mille,
and
now
we
should
be
able
to
send
traffic
in
okay.
So,
let's
see
if
we
can
do
that-
and
this
is
probably
if
anything
goes
wrong.
This
is
probably
where
the
the
biggest
issue
will
happen.
So
we'll
just
open
up
our
counter
one
more
time.
A
Let's
bring
our
friend
jmeter
back
I
told
you
we'd,
bring
them
back
and
let's
go
ahead
and
put
in
some
data
here.
So
the
first
thing
that
we
need
to
do
is
we
need
to
figure
out
our
node
right
because,
like
I
mentioned,
I'm
gonna
have
contour
running
on
the
edge
of
one
node.
Ok,
just
again
for
simplicity
sake,
so
theoretically,
I
just
need
to
call
this
IP
address
right
here
right,
so
we'll
go
back
to
jmeter,
we'll
put
in
this
IP
address.
A
I've
got
contour
exposed
on
port
80
for
the
non
HTTP
ask
traffic
so
we'll
still
go
through
port
80
I,
don't
think
I'll
need
to
set
a
port
number
here.
Path
is
fine,
as
is
we're
gonna
trick
the
test,
though,
if
we
go
here
and
do
a
HTTP
header
manager,
because
I
don't
have
DNS
set
up
in
the
request.
We're
gonna
set
the
host
header
to
be
the
value
of
that
yeah.
Mille
I
just
showed
you
right,
so
this
is
kind
of
like
setting
an
Etsy
host
on
your
system.
A
If
you've
ever
done
that
before
to
make
a
host
auto
resolve
to
an
IP
address
that
you
specify
we're
not
doing
that,
we're
basically
just
going
directly
to
the
IP
buts,
the
Hatter,
so
when
it
gets
the
request,
it's
like.
Oh,
you
asked
for
this
hostname
blah
blah
blah.
So
if
we
go
into
our
manifest
here,
let's
do
a
cat
on
what
was
it
called
ping
right
ping,
yeah
Mille,
we'll
grab
the
counter
here,
we'll
bring
our
friend
jmeter
back
and
we'll
put
the
value
in
all
right.
A
B
A
A
Maybe
if
I
want
it
to
be
used
by
multiple
thread
groups,
I
can't
remember
anyways,
so
we'll
go
into
the
HTTP
request
and
we're
gonna
add
in
a
post,
processor
or
sorry
a
is
it
a
listener,
yeah
listener
that
gives
us
a
summary
report,
which
I
think
should
tell
us
if
things
are
failing
or
succeeding
and
then
we'll
go
in
and
add
in
one
more
listener
which
will
give
us
graph
results.
Yeah.
That
seems
pretty
good
and
actually
there's
a
response
time.
B
A
A
A
B
A
B
A
A
A
It's
running,
summary
report,
all
right,
we
got
throughput
and
we
have
zero
percent
error.
So
far,
that's
good!
So,
let's
well!
This
is
going.
Let's
make
sure
we
have
our
top
window
up
here,
setup
with
top
again
cool,
so
we'll
keep
that
on
now
and
we
got
some
load
coming
in
too.
That's
good,
that's
good!
All
right!
So
we're
really
starting
to
set
the
stage
for
how
things
are
gonna
scale
pretty
soon
right.
So
it
looks
like
our
app
is
not
struggling
too
much
when
the
throughput
is
in
total,
at
least
so
far.
A
A
To
see
initial
kickoff
and
then
we're
kind
of
trending
down
okay,
so
you
know,
based
on
this
initial
initial
metrics,
it
seems
like
with
the
usage
of
61
milla
CPUs.
We
can
satisfy
this
amount
of
requests.
Okay,
so
there's
really
no
need
for
scaling
so
far,
all
right!
So
if
we
go
into
the
thread
group-
and
what
do
you
say
we?
What
do
you
say
we
beef
it
up?
What
do
you
think
chat?
What
should
our,
which
our
thread
count
be
here?
I'll,
let
you
I'll,
let
you
ponder
that.
A
So
let
me
let
me
know
in
chat.
What
do
you
think
are?
What
do
you
think
are
number
of
users
if
this
app
can
take
on
Oh?
Actually,
it
looks
like
if
you
look
at
the
top.
There
looks
like
our
CPU
is
increasing
a
little
bit,
so
we're
maybe
a
little
bit
too
quick
to
judge
it's
starting
to
start
and
take
a
little
bit
more
load
all
right.
A
So,
while
you're
thinking
about
the
the
number
of
threads
I'm
gonna
go
to
the
throughput,
because
that's
actually
what
what
matters
here
right
at
least
per
thread
and
then
we'll
bump
this
up
to
like,
let's
say
25,
maybe
and
then
we'll
bump
our
thread
counts
up
to
in
this
case
just
5.
So
while
you
should
do
this
scientifically
I'm
not
going
to
we'll
stop
and
we'll
hit
run
again,
let's
clean
the
results
out
too
and
we'll
keep
our
summary
report
here.
Ok,
cool
all
right!
A
If
you
will,
but
I
would
expect,
with
as
big
of
a
jump
as
I
just
did,
that
this
will
probably
hit
250,
which
again
remember
that
was
our
max
Milla
CPU
unless,
unless
I
underestimated
our
app
and
it
can
handle
more
than
that,
which
is
I
suppose
not
to
impossible.
But
you
will
notice
here
that
it
probably
isn't
handling
it,
because
the
air
percent
right
now
is
is
not.
A
Clearly,
something
is
failing
in
these
requests
here.
Ok,
so,
while
that's
while
that's
going
we'll
let
what
jmeter
and
and
the
pod
counter
kind
of
get
some
data,
there
were
250
now,
ok,
so
we're
we're
maxed
out
all
right.
So,
let's,
let's
talk
a
little
bit
about
horizontal
pod,
auto
scalars
that'll
be
our
first
place
to
start
right,
horizontal
pod,
autoscaler,
ok,
so
the
horizontal
pod
autoscaler,
as
the
name
implies,
is
going
to
scale
your
pods
by
adding
new
replicas
of
them
right,
and
it
has
a
bunch
of
different
ways
that
it
can
do
this.
A
A
new
pod
then,
like,
as
the
requests
came
in
right
that
other
pod,
if
it's
not
being
hit
through
the
ingress
controller
or
balanced,
it's
not
going
to
take
any
utilization
right,
so
HPA
is
gonna
realize
alright,
I
only
need
to
scale
it
by
one
replica
and
now
my
overall
count
percentage-wise
is
even
way
lower
than
80%.
It's
not
just
gonna
keep
scaling
right.
It's
gonna
need
to
calculate
that.
A
Based
on
you
know,
your
your
current
usage
of
the
pods
that
the
HPA
is
is
looking
over
so
somewhere
in
here,
there's
a
command
where
we
can
set
up
kind
of
a
shorthand
command
to
get
you
started
here.
It
is
I
like
this
one
cube,
cuddle,
Auto
scale.
Okay,
now
an
interesting
thing
about
this
command.
I'm,
pretty
sure
that
this
is
correct
and
someone
can
chat
can
correct
me
if
I'm
wrong
at
all.
A
But
if
we
introduced
a
horizontal
pod
autoscaler
okay
to
our
to
our
replica
set
here,
I
actually
think
I
think
how
this
works
in
cube
is.
If
you
have
a
deployment
managing
the
replica
set,
you
need
to
point
the
auto
scale
at
the
deployment,
or
else
the
replica
set.
The
deployment
will
just
like
keep
resetting.
A
Under
the
hood
I,
just
it's
an
observed
behavior
that
I've
had
before
so
we're
gonna
do
an
auto
scale
on
the
deployment,
okay
and
the
deployment
we
know,
because
you
can
always
look
at
your
pod
right.
This
is
your
deployment
name.
This
is
your
replica
set
ID,
and
this
is
your
pot
ID
right.
So
we
can
kind
of
look
at
that
and
figure
out.
What's
what
so,
if
we
go
to
deploy
and
we
do
counter
okay,
let's
see
what
do
you
think
so,
we'll
do
I
guess
our.
A
A
Let's
try
that
cool
all
right,
so
we'll
keep
top
going
up
here
now,
top
might
not
immediately
show
the
pods,
it
doesn't
mean
they're
not
creating
like
if
I
go
in
and
do
cube,
cuddle
get
pods
name
space,
TGI
K,
no
another
one
hasn't
created
just
yet.
Oh
there
we
go.
Okay.
I
spoke
too
soon,
but
you
can
see
how
it's
now
respecting
that
minimum,
but
top
isn't
showing
it
quite
yet.
Cuz
guess
what
it's
it's
not
taking
load
yet
right.
So
we
still
need
the
load
to
come
into
this.
It
needs
to.
A
It
has
a
readiness
check.
It
needs
to
pass
its
readiness
check
so
that
traffic
can
come
in
and
and
all
that
good
stuff.
So
that's
all
fine
and
dandy.
We'll
wait
for
top
to
kind
of
show
it
up
here.
Let's
look
at
what
that
manifest
kind
of
looks
like
right.
So
if
we
do
a
cube,
cuddle
get
pod
know,
get
HPA,
sorry
get
HPA.
In
fact,
I'll
even
show
you
a
simplified
view
of
this
real
quick
get
HPA
for
the
namespace
TGI
ko
wide,
give
it
just
a
sec
here.
A
Cool,
so
you
can
see
some
interesting
data
here
right.
First
off
we've
got
in
HPA.
Now
it's
it's
a
counter
and
I
guess.
One
thing
I
should
have
unveiled
is
HP
as
a
controller
like
a
lot
of
other
things.
Okay,
so
if
you
haven't
seen
it
Duffy
has
been
doing
awesome
sessions
on
grokking
kubernetes,
where
he
dives
deep
into
this
stuff
and
I.
A
Think
he's
already
done
controllers
someone
correct
me
if
I'm
wrong,
if
he
hasn't,
but
you
should
check
out
that
TGI
K,
it's
awesome,
a
controller
right
that
is
being
used
here
is
actually
HPA,
so
I
didn't
install
anything
to
run
HPA
per
se
right
I'm,
just
using
kubernetes.
The
counter
is
set
up.
You
can
see
that
it's
letting
me
know
that
currently
my
I'm
at
a
hundred
percent
out
of
eighty
percent
so
on
the
interval
that
it's
going
and
checking
and
trying
to
make
scaling
events
happen
right.
It's.
A
Is
needed
at
this
point?
There's
there's
probably
a
little
question
about
that.
We
can
see
our
min
pods,
our
max
pods
are
replicas
and
all
that
right
now
this
is
I.
Guess
more
so
just
to
feed
your
interest,
because
we
won't
have
time
today
to
get
into
it,
but
you
can
do
fancier
stuff
and
what
I'm
just
doing
with
CPU
right
one
of
the
things
that
maybe,
if
it's
interesting
to
you
all
we
could
dive
into
someday,
is
talking
about
kind
of
tuning.
This,
like
there's
the
whole
notion
of
like
cooldown
and
delay.
A
You
know
just
want
to
just
start:
blasting
resources
everywhere
and
ddossing.
Your
whole
system
there's
notions
of
using
met
more
than
one
metric
which
I'm,
obviously
just
using
CPU.
You
can
use
custom
metrics,
which
is
something
I,
be
selfishly
love
to
dive
into
you
like
triggering
off
of
other
things
and
all
that
good
stuff.
Additionally,
I
think
I
was
looking
at
it
last
night.
Maybe
it's
this
one,
there's
some
examples,
just
to
kind
of
show
you,
okay,.
A
A
There
we
go,
so
you
can
kind
of
see
where
this
can
hack
right,
like
potentially
packets
per
second
could
induce
scaling
events.
It's
really
like
I
guess
at
that
point.
A
business
and
technical
question
on
an
app
side
so
like
CPU,
is
an
easy
thing
for
us
to
model,
but
you
can
get
much
fancier
with
like
what
you're
looking
up
and
the
events
you're
scaling
on
right,
so
cool,
okay,
good
deal!
So
let's
go
back
and
let's
see
what
we're
we're
scaling
up
here.
A
So
if
I
do
cube,
kuddle
get
pods
name
space,
TGI,
K,
whoa,
we're
scaling.
Okay.
This
is
good,
so
we're
starting
to
bring
more
and
more
pods
online.
All
right
and
we
should
have.
I
wonder
if
this
pod
stuck
yes,
okay,
we
might.
We
might
have
a
lucky
slash
good
transition
here.
So
why
do
you
all
think
this
pod
is
stuck?
So
what
do
you
think
chat?
Why
do
we
have
a
pod
stuck
intending
here?
What
do
you
think's
going
on
so.
B
A
Have
gasps
so
what's
interesting,
a
BC
yet
you're
right
out
of
resources!
So
what's
interesting
about
this
in
going
into
kind
of
your
answer
and
rinse
or
a
mesh
which
is
probably
not
exactly
what
you're
thinking
so
I'm
just
gonna
like
I'm
gonna
misconstrue,
what
you're
saying
it
kind
of
has
been
scheduled
right
like
it
knows,
go
to
this
node,
but
you're
out
of
resources
right
so
you've
hit
the
do.
We
still
have
our
ugly
diagram.
Let's
see
yes,.
A
A
Now
I
guess
it
would
depend
on
how
it's
sliced
up
anyways.
We
know
for
sure
that,
but
from
a
scheduling
perspective,
we've
sliced
out
more
than
we
want
to
schedule
on
top
of
okay.
So
thinking
this
through,
how
can
we
kind
of
check
it
out?
You
know
one
thing:
I'm,
not
sure
I,
wonder
if
HPA
tells
you
in
the
event
API
that
this
is
hitting
a
scaling
limitation.
B
A
That's
I
actually
looked
into
the
events
of
the
HPA
yeah,
which
is
clearly
extremely
helpful
right,
so
it
doesn't
know
that
the
pod
couldn't
get
scheduled.
It
seems
which,
which
is
totally
I,
can
reason
with
that
because,
like
it's,
not
the
HP's
concern
really
like
whether
there's
infrastructure
for
the
pod
right,
but.
A
A
Name,
space,
TGI
K
and
we
will
will
describe
pod
here,
cool
alright.
So
this
is
probably
a
message.
Those
of
you
who
run
kubernetes
have
read
a
thousand
times
over
at
this
point
for
various
reasons.
Right,
it's
one
I
always
see,
and
it
sometimes
confuses
me.
So
what
it's
saying
here
is
I
looked
at
zero
out
of
two
notes:
okay,
one
of
them
had
insufficient
CPU
for
scheduling
common.
The
other
one
had
tanks
that
this
pod
could
not
tolerate
right.
A
So
what
this
is
saying
in
plain
English
is
I
tried
to
go
on
the
node
that
you're
putting
everything
on
and
I
don't
have
CPU,
and
then
it's
also
saying
I
also
looked
at
your
master
node,
but
it's
got
tanks
on
it.
That
say
it's
no
schedulable
based
on
this
roll
master
right,
so
I'm
not
gonna
land.
There
either
right.
B
A
A
B
A
A
A
moment
like
right
now
we
have
to
have
all
these
different
docks
and
all
this
different
stuff,
but
that's
mostly
because,
like
how
it
talks
to
EWS
is
obviously
gonna
be
different
than
the
API
that
Azir
exposes
okay.
So
let's
go
in
here.
Real
quick
cluster
autoscaler
AWS,
like
I,
said
that
last
time
I
touched.
This
thing
was
in
like
one
six,
so
we'll
see
how
it
goes.
A
I
looked
at
it
a
bit
last
night,
but
you
know
we'll
see
how
it
goes
in
a
real
cluster,
so
we
want
to
scale
our
cluster
out
horizontally
now,
fair,
so,
okay,
we
need
an
IM
policy
for
the
node
group.
That
makes
sense.
What
do
you
all
think
are
you?
Are
you
betting
people?
Do
you
think
I
already
have
my
I
am
set
up
correctly?
Should
I
just
skip
this?
Let's
just
skip
this:
let's
see
what
happens:
AWS
credentials.
Okay,
that's
probably,
if
you're
not
using
an
IM
role
attached
to
the
node
okay
cool.
A
Now,
here's
where
it
gets
interesting
and
and
I
I
saw
something
in
chat
that
might
have
looked
similar
to
this
I.
Don't
know
maybe
see
if
you
had
brought
it
up.
Maybe
you
were
talking
about
something
different,
but
you
might
need
to
run
multiple
aSG's.
If
memory
serves
correctly,
one
of
the
should
I
say
limitations
of
cluster
autoscaler
in
AWS.
If
I,
if
I
recall
correctly,
is
that,
like
you
know,
if
you
schedule
a
node
a
pod
somewhere
and
it
has
EBS
volume
attached
and
again
chat,
keep
me
honest,
I'm,
not
an
AWS
expert.
A
The
EBS
volume
is
going
to
be
in
that
availability
zone.
So,
if
you
have
auto
scaling
groups
that
span
available
I,
think
one
of
the
things
you
can
theoretically
run
into
is
an
auto
scaling.
Event
occurs
and
it
starts
a
node
in
a
place
where
things
might
not
be
available
that
you
expect
things
like
EBS
volumes
or
other
things.
Let's
see.
Actually,
let's
just
look
at
the
docs,
it
probably
says
it
here:
availability
zones,
cluster
autoscaler,
does
not
support
multiple
availability
zones.
A
B
A
A
A
B
A
Examples,
cluster
autoscaler
run
on
master.
That
seems
like
a
pretty
good
name,
we'll
grab
this.
We
will
bring
this
down
locally,
so
let's
call
it
C
Auto
Gamal
that
wait.
Let's
W
get
it
first
right
and
then
we
will
rename
it
C
Auto
Gamal,
okay,
cool
all
right.
So,
let's,
let's
take
a
look
at
this
everyone
and
see
what
we
can,
what
we
can
come
up
with
all
right,
so
cluster
rolled,
do
we
got
just
some?
Are
back
cluster
roll
binding
roll,
binding
deployment?
Okay,
here's
what
we
want!
A
What
we
want
to
look
so
we've
got!
Cluster
autoscaler
looks
like
it's
got
some
Prometheus
metrics
that
it
exposes
that's
good.
Our
image
is
right
here:
okay
and
okay.
So
it's
yeah.
What's
interesting
about
cluster
autoscaler
and
I
guess:
I
had
forgotten
this
detail,
those
of
you
who
have
played
with
like
kubernetes
CSI
or
cloud
providers.
You
probably
know
like
at
one
point
we
baked
a
lot
of
a
lot
of
cloud
provider
logic
into
core
kubernetes
and
with
the
introduction
of
CSI
and
for
some
other
good
architectural
reasons.
A
We've
sense,
extracted
cloud
provider
logic
into
cloud
provider
controllers.
If
you
will
I
think
is
a
couple,
is
a
CC
on
Cloud,
Controller
manager
and
and
what's
interesting
about
this-
is
this
is
kind
of
using
sort
of
the
old
model
to
some
degree
and
that
if
it,
if
I'm
reading
it
right,
it
looks
like
there's
one
single
image
and
you
set
a
cloud
provider
flag.
A
So,
like
we're
gonna
be
running
this
in
our
cluster
and
I'm
guessing
it
probably
has
the
logic
and
code
for
like
azure
in
it,
even
though
it's
not
live
right,
which
is
kind
of
an
interesting
thought,
so
cool
anyways,
that's
just
me
on
a
soapbox.
This
thing
also
has
limits
in
requests
which
we
talked
about
our
important
things.
So
that's
great
all
this
looks
good.
These
can't
be
right,
so
let's
do
nodes
1
and
then
node
max
20.
So
this
is
gonna,
be
probably
my.
A
My
min
and
Max
I
think
what'll
be
interesting
here
and
I'll
take
guesses
or
knowledge.
If
you
already
have
it,
but
I've
already
got
an
ASG
right,
which
is
down
here
and
with
this
ASG.
My
min
and
Max
is
one
in
five.
So
what
do
you
all
think?
What
do
you
think
chat?
Do
you
think
this
will
get
updated
based
on
this
deployment,
or
do
you
think,
there's
kind
of
two
levels
I
need
to
change
it
in
AWS
and
change
it
in
the
cluster
autoscaler
yeah,
so
kind
of
a
interesting
thought
to
noodle
on
there.
A
I
guess
we'll
learn
soon
enough
and
then
the
name
of
the
ASG
I,
guess
I,
think
I
I
think
I
do
remember
this
being
the
actual
name
and
not
like
an
AR
n
or
anything
so
we'll
grab
this
name
paste
it
in
okay.
That
all
looks
good
to
me.
What
do
you
think
chap?
Does
that
manifest
look
good?
You
think
we
think
we
don't
have
any
mistakes,
I'm
hoping
so.
Okay.
B
A
Gonna
deploy
this
real,
quick
and
then
I'll
read
through
chat,
and
it
sounds
like
you
guys
are
having
real.
You
all
are
having
really
interesting
conversations
around
just
general
stuff,
but
I'll
see.
If
there's
any
questions
in
there
too,
so
I'll
save
that
up
for
C
Auto.
Let's
go
ahead
and
again,
as
usual,
we'll
do
in
our
bottom
buffer
I
guess
this
time
we'll
do
a
watch
for
cube
cuddle,
get
pods
for
the
namespace
cube
system,
which
is
where
we're
deploying.
So
that
makes
sense
here
right.
Does
that
make
sense
to
you
all?
A
B
A
A
A
A
No
version
116,
if
that's
in
regard
to
me,
Herman
I,
don't
know
what
version
my
using
in
my
on
114
or
115
I
just
used
a
tool
to
boot,
wrap
this
cluster
real,
quick,
so
I'm
probably
sent
an
old
version.
There
is
definitely
a
version.
116
cool
and
it
looks
like
the
rest-
is
just
implementation,
details
very
cool,
so
I
guess
yeah
another
big
thing
about
kind
of
understanding.
This
is,
you
might
not
find
this.
A
Auto-Scaling
model
works
for
you
right,
like
this
autoscaler,
as
you're
going
to
learn,
is
going
to
be
based
on
and
ability
to
schedule
due
to
lack
of
resources.
There
are
other
reasons
why
you
might
scale
and
a
lot
of
my
customers
when
I
talked
to
them.
We
actually
talked
about
building
something
that
does
is.
It
sounds
kind
of
like
with
with
your
use
case,
ABC
where
it
was
like
yeah.
This
is
nice,
but
like
we
need
to
think
about
this
more
strategically
for
our
needs
and
scale
based
on
that
right.
B
A
A
This
one
I
guess
describe
let's,
let's
look
at
the
events
so
we'll
describe
this
pod,
real,
quick!
Oh,
it
doesn't
match
the
taint
okay.
That
would
be
an
important
thing
so
same
error
from
before,
but
for
it
for
a
different
reason,
I
suppose.
So,
if
we
look
at
see
auto
here
and
let's
find
the
the
taint
that
it
tolerates
it's
all
the
rate-
shins
yep.
Okay,
that's
all
good!
That
looks
fairly
normal.
Let's
see
here.
A
So
if
I
do
a
cube,
cut,
Oh,
get
nodes,
rolla's
master
and
that's
the
right
key
I
know
that
just
from
doing
cube
stuff,
let's
describe
this
master
node
real,
quick,
so
cube
cuddle
describe
node
here,
you
know
we're
gonna,
get
the
yellow
that'll
be
easiest.
So
let's
get
node,
oh
yeah,
mall
and
let's
this
is
this
is
silver
searcher.
A
What's
the
issue
No
Oh,
our
key
is
different
check
that
out.
So
maybe
this
is
actually
a
version.
Mismatch
thing
or
just
someone
chose
to
use
a
different
key,
so
the
taint
key
is
supposed
to
be
this.
It
looks
like
right,
so
let's
go
back
here
and
oops,
oh
geez,
okay,
cool
and
let's
do
that
and
bring
in
node
alpha
kubernetes.
That
looks
good.
So
that's
the
key.
The
value
is
master,
it
looks
like
and
equal
effect
is
no.
So
that
looks
all
right
yeah.
A
What
do
you
all
think
that
seems
that
seems
better
for
the
cluster
autoscaler?
So,
let's,
let's
see
if
that,
if
that
works,
okay,
all
right
cool,
so
we
do
a
cube,
cuddle
I'm,
usually
a
fan
just
to
make
sure
I'm
not
on
I'm,
not
on
a
back
off
for
these
types
of
scenarios.
I'll
start
off
by
deleting
that
all
right
and
then
we'll
do
a
cube.
Cuddle
apply,
C,
Auto.
A
Oh
interesting
rest
in
peace
CREP
replaces
AG.
Is
that
a
is
that
a
competitor
has
AG
to
saw
her
silver
serger
decided
to
go
to
our
P
grab.
I
haven't
I've,
never
heard
of
our
P
grabs,
so
that's
cool.
If
there's
something
even
better
I
will
check
that
out
afterwards
for
sure.
Oh,
it's
in
rust,
Wow
yeah,
that
is,
that
is
bleeding-edge
I,
like
it
cool
I'll
check
it
out
for
sure
awesome.
Thanks
for
the
heads
up,
okay,
so
it
deployed
okay,
we're
may
be
getting
farther.
This
time
can
run
container
air.
A
This
node
describe
node
right,
so,
let's
describe
node,
oh
geez,
cube
cuddle
get
nodes.
Will
copy
no
wait.
Actually,
sorry
everyone,
let's
do
cube
cuddle,
get
pods
that'll
be
easier
for
the
namespace
cubes
system.
We're
gonna
grab
the
autoscaler
and
we
will
once
again
do
a
describe
so
describe
this
pod.
That's
this
one.
The
namespace
is
cube
system
ooh.
B
A
Did
you
know
that
no
that's
weird
Oh
cuz
above
interesting
is
the
mount
path
that
put
in
the
container
is
CA
certificates,
okay,
anyways.
Let's
try
that
I
think
that
I
think
it
might
be
as
simple
as
that,
but
it's
it's
been
a
while
and
remember
we're
core
OS
necessarily
puts
that.
Ok!
So
again,
let's
do
a
delete.
Ok,
then,
let's
do
an
apply,
ok
and
then
let's
watch
Hey.
Look
at
that
we're
running
that's
exciting!
Okay,.
B
A
We're
almost
there
everyone
we're
almost
done
at
least
for
this
to
this
first
session.
So
let's,
let's
make
this
work.
So
all
we
got
to
do
is
see
some
node
scale,
let's
start
off
with
logs
here.
So
if
we
do
a
cube
cuddle
logs
make
this
a
little
smaller,
keep
cuddle
logs,
no
and
err.
Okay,
let's
see
what's
come
on
fine
cube
cuddle
logs
for
this
pod
in
the
name,
space
cube
system.
A
B
A
What
I
should
be
using
here
is
I
should
be
using
a
thing
that
delegates
in
a
more
secure
fashion,
my
I
am
access
right,
so
ABC,
exactly
cube,
I
am
Kai
and
I
know
I'm,
saying
it
wrong.
So
someone
cracked
me
if
I
am
I,
think
it's
I
think
it's
Kai
am
I'm,
probably
saying
it
wrong
wrong.
Anywho,
so
I
should
use
something
that
is
going
to
provision.
This
I
am
role
for
me.
A
I'm
stubborn,
like
when
I
try
to
put
together
furniture
and
stuff
I,
didn't
read
the
instructions
and
I
hose
myself
right.
So
let's
go
ahead
and
let's
add
this
in
so
we
have
a
statement
here.
I'm
not
gonna,
introduce
cube
to
I
am
just
to
warn
you
all
I'm,
sorry
to
give
your
hopes
up
or
cayenne
for
that
matter,
but
I
am
gonna,
go
into
my
node
role
and
do
something
that
I
probably
shouldn't.
So,
let's
go
to
instances.
A
I
am
role
there.
It
is
it's
been
a
while
sorry
AWS
you
I
edit
policy,
okay,
okay,
it's
still
somewhat
intuitive
here
so
ooh.
It
was
there,
though.
Oh
you
know
what
that
was
my
worker
node.
Wasn't:
let's
go
back.
That's
my
worker
node
yep
I
can
see
it
from
here.
Excuse
me
remember
we're
deploying
on
top
of
our
master
nodes.
So
let's
check
that
one
real,
quick
I
am
roll
policy
edit
policy.
Okay,
let's
introduce
this
here.
B
A
Theoretically,
if
we
come
in
I'm
guessing
it's
as
simple
as
that
maybe
seem
okay
to
you
all
I'm
gonna
go
with
it
review
policy,
Save
Changes
back
to
our
terminal,
all
right.
What
do
you
think
if
we
delete
this
pod?
Will
it
work?
Can
we
can,
when
you
wrap
this
thing
up
on
a
good
note,
let's
see
all
right
so
autoscaler
keep
kuddle
delete,
pod,
name,
space,
cube
system.
B
A
A
A
And
you
guys
just
explained
that
in
chat
anyway,
so
thank
you
for
backing
that
up,
I
should
have
looked
down
and
read
that
hey
we're
72
seconds
in
this
is
promising
right.
I
think
we
can
ship
this
thing
for
production.
Okay,
so,
let's
see
the
logs
now
keep
cuddle
logs
for
the
autoscaler
logs
for
this
namespaces
cube
system.
A
Okay,
this
looks
fairly
healthy
to
me.
Let's
go
ahead
and
tail
these
logs
out
and
we'll
just
keep
an
eye
on
it.
So
I
wonder
if
it'll
pick
up
the
need
to
do
a
scaling
event
automatically
scale
down
status
we
can
see
here
is
unneeded
starting
main
loop.
One
unregistered
nodes
present
I'm,
not
exactly
sure
what
that
means.
No
hon,
and
maybe
that
means
it's
ooh.
Look
at
that
upcoming.
A
One
notes:
I
wonder
if
that's
a
scaling
event
based
on
our
load,
so
let's
bring
our
our
buddy
jmeter
back
in,
because
I've
missed
in
it's
been
so
long
since
we've
taken
a
look,
we're
still
at
a
2%
error,
failure,
there's
lines
that
have
colors
that
apparently
depict
that
our
throughput
and
such
is
stabilizing
a
bit
okay.
So,
let's,
let's
see,
let's
see
what
we
got
here
that
so
if
I
go
down
to
Amazon,
let's
start
there
so
obviously.
A
Scaling
of
that
will
take
some
time
because
we
got
stand
up
the
know
there
might
be
user
data
that
needs
to
run
blah
blah
blah
to
keep
components
need
to
initialize.
Let's
see
what
we
got
so
we
go
to
ec2.
Oh
I
think
that
number
is
a
good
thing
check.
It
out.
Success,
exciting
stuff,
look
at
that,
so
I
hope
you're
all
as
excited
as
I
am
so
we
have
a
node.
That's
kicked
on
it's
initializing.
If
we
go
back
to
our
tests
here,
I
think
we
should
see
a
decrease
in
our
throughput
eventually.
A
So
there's
a
couple
I
mean
we
already
saw
that
that
log
line
right
up
here
I
think
it
said,
like
upcoming
one
notes
that
was
that
was
cool
to
see.
If
we
clear
this
out
and
then
we
do
cube
cuddle
get
nodes.
So,
let's
start
by
watching
that
Oh
check
it
out.
It's
registering
in
cube
seven
seconds
good
deal,
we'll
make
we'll
make
a
couple
things
here.
So
let's
do
that?
A
Let's,
let's
set
up
a
window,
that's
going
to
be
dedicated
to
watching
the
top
four
pods
inside
of
the
namespace
TGI
Kay
wait
top
o
j--
cube
cuddle
would
be
helpful,
I,
don't
think
I,
don't
think
just
Tov
is
gonna.
Do
the
trick.
Okay,
so
we've
got
these
pods
here
this
bottom
one.
You
just
bring
that
down
a
bit
okay,
so
again
we're
in
the
three
pods
and
then
down
here,
we'll
watch
cube
cuddle,
get
pods
for
the
namespace
TGI
K
get
pods;
alright,
oh
hey
that
one
initialized.
A
So
that
would
make
sense,
because
now
our
extra
node
is
ready.
Okay,
so
we
talked
about
kind
of
the
the
multiple
factors
here
that
are
in
play.
I
think
what
I
would
expect-
and
let
me
know
if
you
all
agree,
I
think
what
we
should
have
happened
now
is
HPA
should
continue
to
be
able
to
kick
off
right
because
there's
no
longer
that
inability
to
schedule
so
there's
no
longer
that
no
that's
basic
or
that
pod
that's
taking
zero
percent
cpu.
A
So
I
think
we
should
start
see
scaling
events
on
our
pods
and
if
that
new
node
fills
up,
we
should
then
again
see
another
scaling.
Events
on
our
cluster,
if
all
goes
well
right,
so
we'll
wait
for
that
for
a
second,
then
we'll
wrap
all
this
stuff
up
in
a
bow
and
call
it
complete
for
Friday,
so
that
all
looks
good.
Our
pods
Oh
check
it
out.
I,
don't
think
we're
hitting
a
card
enough
anymore.
Are
we
so
you
can
see?
B
A
Enough
of
these
enough
of
this
child
play
right.
We're
gonna,
we're
gonna
go
all
in
so
by
the
way
I'm
running,
jmeter
locally.
On
my
machine.
You
would
never
do
this
in
real
life,
so
I'm
gonna
introduce
twenty
threads.
I
am
gonna,
put
my
constant
throughput
timer
up
to
100
and
let's,
let's
get
crazy
okay,
so
this
is.
This
is
on
each
thread
too,
by
the
way
so
like
the
factor
to
which
we
are
going
up
here
is
kind
of
ridiculous.
So
we'll
stop
we'll
start
and
let's
see
what
happens.
A
B
A
Once
I
killed
J
meter
and
for
some
duration
of
time,
it's
starting
to
realize:
hey,
I,
don't
need
these
nodes
anymore
right.
It
actually
should
start
taking
the
nodes
and
scaling
them
down,
and
this
is
you
know
one
thing
that
operations
oftentimes
like
like
you
know:
if
you
have
a
dev
cluster
one
one
instance
I
set
up
was
we
had
a
dev
cluster
and
we
we
used
a
project
speaking
of
Zalando
similar
to
cute
janitor.
Is
that
the
name
of
it
you
basically
put
like
TTL
annotations
on
your
pods?
A
B
A
Event
would
occur
on
night
would
sweep
up
a
bunch
of
deployments.
They
would
go
away.
All
the
sudden
would
be
underutilized
knows
so
we
were
running
cluster
autoscaler
and
cluster
autoscaler
would
realize.
Oh
hey,
I
can
start
killing
off
nodes
because
I
no
longer
have
that
pod
resource
utilization
that
I
used
to
need
right.
So
that's
that's
kind
of
a
cool
thing
to
think
about
check
it
out.
We
introduced
a
new
node,
so
that
is
that's
good,
we're
clearly
hitting
it
harder
than
than
we
otherwise
expected
right.
So
that
should
be.
A
That
should
be
scaling
up
as
well.
So
I
will.
Let
me
let
me
wrap
this
this
topic
up.
While
we
see
if
another
node
comes
online
with
cluster
autoscaler.
Oh
you
know
what
one
thing
we
meant
to
check.
I
had
mentioned
to
you
all:
cuz
I,
wasn't
sure
of
the
answer.
If
we
go
to
auto
scaling
groups,
okay,
yeah,
so
it
didn't
change
the
men
in
the
max
based
on
what
I
said
in
the
manifest
right.
A
A
Let
me
see
if
I
can
find
it
here,
real
quick.
So
if
you
are
actually
gonna
play
around
with
us
and
deploy
it
I
think
this
is
the
way
to
go.
If
I
recall,
correctly,
Auto
discovery
mode,
I'm,
pretty
sure
works
off
of
an
AWS
tag.
So
what
you
can
do
is
you
can
set
that
min
and
Max
in
one
place
right
and
inside
of
the
AWS
console
and
then
Auto
discover
it
will
see
the
tag
that
ASG
and
will
just
automatically
respect
those
settings.
A
So
you
all
probably
saw
me
going
and
I
like
put
the
name
of
the
ASG
I
put
them
in
Max,
the
ASG
Auto
discover
if
your
tags
are
correct
and
you
have
the
right,
privileges
should
automatically
be
able
to
detect
those
and
then
just
start
controlling
those
aSG's
simple.
As
that.
So
that's
that's
kind
of
another
approach
that
you
could.
You
could
take
and
look
amazon's
trying
to
help
me
save
money
on
compute,
which
we
respectively
trying
to
do
to
some
degree
here
as
well,
although
I
guess
we're
actually
spending
more
at
this
point
right.
A
B
A
Up,
though,
check
that
out
more
and
more
pods
I
think
we're
gonna
hit
it
soon.
So,
let's
talk
about,
let's
talk
about
one
more
concept
before
we
before
we
before
we
end
the
day
here.
So
you
know.
Kind
of
the
last
thing
that
I
wanted
to
talk
about
is
that
this
auto
scaling
model
is
something
that
we
can
deploy.
It
solves
a
common
problem,
but
one
of
the
things
that
I
think
is
kind
of
interesting
about
this
is
that
the
autoscaler
needs
to
know
about
how
to
talk
to
the
cloud
provider.
A
A
Idear,
who
knows
cluster
api
really
well,
so
he
can
help
me
out
with
with
gory
details
or
thoughts.
But
what
I'm
excited
about
is
something
you've
probably
heard
about
on
this
channel
quite
a
bit,
which
is
how
things
are
headed
towards
a
declarative
model
in
regards
to
how
we
manage
the
lifecycle
of
clusters
right,
so
the
thought
process
pink,
those
of
you
not
that
close
to
cluster
API
is
in
there
it's
more
complex
in
this,
but
think
about
it
like
we
can
have
a
cluster
that
understands
how
to
talk
to
the
infrastructure
provider
and
as.
A
Like
you
do,
for
any
other
any
other
resource
and
cube,
you
can
actually
put
a
manifest
and
then
like
basically
kind
of
describes
a
cluster
right
and
then
that
thing
could
ingest
and
be
like.
Okay
cool.
So
like
a
couple
nodes,
you
need
this
that
the
other
thing
you
need
this
pod
cider
and
it
can
actually
go
out
and
talk
to
the
provider
and
instantiate
clusters.
So
it's
like
cube,
instantiating,
cube
and
then
cube
to
some
degree
in
managing
cube
right.
Actually,
let's
just
let's
go
back
to
our
terrible
diagramming
tool.
A
Since
that's
a
that
seems
to
be
our
jam
here
and
let's
say:
let's
introduce
a
new
one
here,
so
think
of
it
like
we
have
this
cluster
API
thing
right.
Just
think
of
it
like
some
nebulous
thing
right
now.
Okay-
and
this
is
you
know-
don't
shoot
me
I'm
not
trying
to
be
technically
accurate
here,
but
basically,
as
you
know,
we
go
in
and
we
submit
manifests
right
these
guys,
okay,
we'll
give
it
to
cluster
API
and
then
cluster
API
will
be
able
to
instantiate
and
manage
cube
clusters.
So
this
is
cube
cluster
0.
A
This
is
cube.
Cluster
1
right
now,
the
great
thing
about
this
model
is
that
you
think
about
where,
where
is
your
cloud
provider?
Logic
exists
right?
Well,
in
this
case,
if
you
implement
a
provider
like
let's
say
an
AWS
provider
and
or
a
vSphere
provider
right,
what
this
means
is,
as
you
put
in
these
declarative,
manifests
into
cluster
by
a
cluster
API
into
this
cluster
right.
It's
called
a
let's
call
it
a
management,
cluster
I
guess
for
lack
of
a
better
word.
A
B
A
And
it
can
actually
add
new
cube
clusters
and
manage
them
now
where
this
is
headed
right
is
think
about.
If
you
wanted
to
add
a
node,
you
don't
need
to
talk
to
Amazon
to
Adam
no
right.
How
did
you
add
nodes
in
the
first
place?
Well,
in
this
model,
you
used
this
declarative
API
right.
You
use
this
Mammal,
manifest
that
apparently
you
like
physically
handed
to
some
cloud
right
and
that.
A
A
Terminology
here
right,
give
me
five
nodes:
six
nodes
and
just
by
increasing
that
declarative
count
the
copy
the
the
cluster
API
provider
you
have
can
actually
make
that
happen
on
a
vSphere
AWS
level.
Now
why
this
is
cool
and
why
I'm
bringing
it
up
is
I
actually
think
it's
gonna
introduce
opportunities
to
simplify
the
concerns
of
cluster
Auto
scalars.
So
ABC,
remember,
remember
your
talk
about
like
I,
know
we're
doing
these
custom
things
and
so
on
and
so
forth
to
kind
of
scale.
B
A
Well,
like,
theoretically,
if
you
took
that
same
need,
but
you
kind
of
cut
off
your
need
to
use
like
AWS
SDK,
to
talk
to
aw
asking
you
literally
just
had
to
go
into
an
API
or
to
a
Hamelin
and
basically
increase
the
count.
You
take
your
replicas
from
30
to
32
and
all
the
sudden
that
would
happen
from
a
cluster
api
perspective
in
your
cluster
would
scale
so
again,
it's
kind
of
delegating
those
concerns
to
the
provider
and
not
having
to
embed
cloud
provider
logic
into
an
auto
scaler.
B
A
Brought
up
a
great
point-
and
that
is
like
if
we
can
move
to
this
model
in
the
future
right
where
we
centralized
through
a
provider
where
the
cloud
provider
logic
goes
in
kind
of
like
we
did
with
like
CSI
stuff,
integrating
with
cloud
controller
managers
right
then
building
cluster
Auto
scalars
will
be
so
easy.
It'll
just
be
a
minute.
What
the
implementation
or
provider-specific
you'll
be
able
to
go
in
and
be
like
yeah.
A
I
think
there's
been
a
lot
of
opportunity
to
do
fancier
stuff
with
auto
scaling
and
cube,
especially
building
your
own
extensions
and
I.
Think
if
we
can
remove
the
need
to
implement
the
cloud
provider,
logic
I
think
we're
gonna
see
a
lot
more
cool,
interesting
stuff
come
out
of
the
cluster
autoscaler
space.
I
really
do
so
I,
don't
know.
B
A
Something
to
noodle
on
I
don't
know
chat
if
that
resonates
with
you,
but
that's
kind
of
kind
of
an
interesting
thought
about
the
direction
that
this
all
could
head
with
with
our
lovely
cluster
autoscaler
diagram
here
and
capi
diagram,
so
check
it
out.
Okay,
I
think
I
think
we
can
wrap
this
thing
up.
So
we've
now
introduced
another
node,
so
clearly
we've
scaled
up,
I
will
I
will
kill
my
throughput
here
now,
I
don't
know
depending
on.
A
If
you
all
want
to
stick
around
I'm,
not
I,
know
it
scales
down,
but
I've
seen
this
happen
before,
but
I'm
not
sure
how
long
it
takes
about
the
defaults
are.
But
theoretically
the
utilization
should
go
down,
which
means
our
pods
can
go
down,
which
then
means
our
nodes
should
stop
being
is
utilized
and
they
should
go
down
as
well.
If
we
look
at
our
response
time
graph,
you
can
see
some
of
the
initial
requests
when
we
really
started
hitting
it
I'm
just
guessing
here,
but
maybe
this
is
where
it's
actually
hard
to
say:
I.
A
Is
probably
our
initial
hit
where
our
test
results
went
when
we
add
in
a
node
and
then
maybe
with
three
nodes,
possibly
but
I?
Don't
think
we've
had
three
nodes
online
that
long
so
any
who
clearly
we've
seen
this
kind
of
dip
and
dip
and
dip
and
dip
so
really
really
interesting,
stuff.
All
right,
so
I'm
gonna
look
at
chat
one
more
time.
If
you
have
any
outstanding
questions
or
thoughts,
please
let
me
know,
let's
see
what
else
we
got
here.
B
A
A
We,
if
we
ever
do
do
a
follow-up
on
this
topic.
We
can
dive
deeper
into
it.
I
never
talked
about
vertical
pot
autoscaler,
okay,
so
let
me
just
mention
it
for
a
moment,
there's
vpa,
which
is
vertical
pot
autoscaler,
it's
something
that
we
could
have
used
as
well
vertical
pot
out
of
scalars
name
implies,
is
basically
a
way
for
us
to
scale
the
resources
available
to
the
pod
vertically
right.
A
It
more
memory,
there's
there's
really
good
docs
on
and
check
it
out.
I
think
your
common
use
case
with
vpa
I
actually
haven't
personally
used
VPI
extensively.
It's
mostly
been
like
my
customers
using
HPA,
but
vp
is
probably
better
if,
like
stateful
workloads,
where
you
can't
just
like
you
know,
replicate
replicate.
B
A
To
know
that,
if
you're
scaling
in
the
opposite
direction
and
vertically
there
is
something
called
vertical
pot,
autoscaler
will
let
you
set
up
rules
not
too
dissimilar
to
what
we
saw
here.
Obviously,
some
of
the
big
differences
are
we're
not
going
to
be
replicating
pots
right
and
there's.
Also
some
intricacies
like
when
you
add
more
resource
to
the
pod
restart.
Should
it
just
expand
if
it
can
like
what?
What
should
that
look
like
right,
so
doo-doo-doo
be
sure
to
check
out
vertical
pot.
A
Okay,
so
I
think
everyone
that
is
maybe
all
we've
got
for
today.
So
we
we
succeeded
so
again
like
have
an
awesome
Friday.
Thank
you.
So
much
for
hanging
out
I
hope
you
found
this
kind
of
interesting
and
informative
to
play
around
with
I
would
say
you
know
if
this
topic
is
interesting
to
you
and
taking
it
a
step
further.
As
far
as
custom
metrics
go
like
you
know,
making
this
fancier
than
just
scaling
based
on
CP,
you
add
a
ticket
to
TGI.
K
have.
B
A
They
get
to
the
second
and
third
page
I.
Sometimes
don't
look
so
much
at
them,
but
yeah
I
can't
thank
you
enough
for
joining
us
this
week,
thanks
so
much
for
hanging
out
and
yeah
I'm
sure,
Duffy
and
Joe
will
be
back
soon,
but
I'm
sure
I'll
see
you
all
soon
in
the
future
as
well.
So
have
a
great
Friday
thanks
again.