►
From YouTube: Battle of Black Friday: Proactively Autoscaling StockX - Mario Loria, Carta & Kyle Schrade, StockX
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Battle of Black Friday: Proactively Autoscaling StockX - Mario Loria, Carta & Kyle Schrade, StockX
When we started our Kubernetes journey, manual responses to traffic spikes were the norm, causing significant brown-outs greatly impacting the customer experience. Graduating into an autoscaling implementation for both workloads and cluster nodes took time but yielded immense results. With this talk, we’ll explore how StockX has grown to solve these challenges operationally and developmentally, even enabling a proactive solution in addition to our reactive methods. Attendees will gain advanced clarity on autoscaling tools like the HorizontalPodAutoscaler and Cluster-Autoscaler, difficulties managing at scale, challenges in educating a rapidly growing team, and the unique intersection of e-commerce marketing events with infrastructure reliability efforts. Finally, we’ll announce an in-house tool we’re Open Sourcing to help aid others in their proactive autoscaling efforts!
A
A
Before
we
start,
we
kind
of
want
to
introduce
ourselves
and
introduce
stockx
a
little
bit,
so
some
of
this
stuff
makes
a
little
bit
more
sense
when
we're
walking
through
it.
Like
I
said,
my
name
is
kyle,
I'm
a
senior
software
engineer
at
stockx.
I
like
graphql,
probably
just
a
little
too
much.
If
you
guys
ever
want
to
talk
about
it.
Our
twitter
handles
are,
in
the
lower
right-hand
side,
feel
free
to
hit
me
about
anything.
Graphql.
I've
talked
at
a
few
other
conferences
about
it.
B
I
do
thank
you
kyle,
I'm
mario
lauria.
I
actually
just
love
stockx
a
few
months
ago
for
carda,
I'm
a
yama,
aficionado,
devops,
sre
engineer.
I
love
everything.
Infrastructure.
Of
course,
love
kubernetes
and
auto
scaling
is
one
of
the
funnest
challenges
and
kyle,
and
I
have
a
lot
to
discuss
today.
It's
going
to
be
a
fun
session,
so.
A
Cool
so
like
we
mentioned,
this
happened
at
stockx.
A
lot
of
the
stuff
is
based
off
of
work.
We
did
at
stockx.
Stockx
is
a
stock
market
of
things.
We
are
a
live
bid
and
ask
model
where
people
will
actively
bid
and
ask
things
just
like
a
stock
market,
but
instead
of
trading
stocks,
they
will
trade
normally
some
type
of
product.
So
in
the
normal
instance,
the
stuff
that
most
people
know
is
some
type
of
jordan
or
some
type
of
nike
card.
We
just
expanded
to
collectibles.
A
So
trading
cards
are
a
big
thing
that
we
do
now
as
well.
We
also
do
some
electronics,
but
again
it's
that
live
stock
model
where
people
will
be
actively
asking
for
something
and
bidding
on
something
and
when
they
match
that's.
When
you
get
your
item,
so
the
first
thing
we
want
to
do
is
we
want
to
really
frame
this
problem
of
what
was
our
problem
that
we
had
to
solve?
And
what
is
this
battle
that
we
had,
and
the
big
thing
is,
is
it's
the
surges
in
traffic?
Our
traffic
is
very
unnormal.
A
So,
like
a
lot
of
places,
you
would
expect
you
know
your
traffic
to
go
up
through
the
day
and
down
at
night
and
kind
of
follow
that
nice
little
easy
wave.
We
don't
have
that.
Unfortunately,
we
have
these
very,
very
spiky
trends,
where
all
of
a
sudden
kanye
will
feel
like
he
needs
to
drop
a
shoe
and
all
of
a
sudden
all
of
our
stuff
tanks,
and
we
go.
You
know
from
zero
request
to
a
million
request
almost
instantly
and
that's
kind
of
where
this
a
lot
of
these
talk
will
come
from.
A
Is
we
have
these
massive
massive
traffic
traffic
spikes
very,
very
fast
that
cause
us
to
do
some
pretty
cool
things
and
to
give
you
kind
of
a
real
graph
on
kind
of
what
happens.
This
is
actually
our
request
per
second
and
our
edge
layer
of
what
happens
when
a
push
notification
for
a
marketing
team
happens.
As
you
can
see,
we're
hanging
out
around
15
000
requests
per
minute
and
then.
A
80K
within
30
seconds,
and
how
do
we
handle
that
without
going
down
without
having
a
lot
of
problems
and
originally
what
we
we
did
is
we
would
do
some
good
old,
coop
ctl
commands
and
make
some
things
work
and
scale
some
stuff
up
and
that's
how
we
do
it
we
would
manually
go
through,
we
would
add
nodes
to
our
cluster
and
we
would
also
go
around
saying:
hey.
Did
you?
Did
you
scale
your
stuff
up?
We
have
something
coming
and
the
thing
is.
A
The
question
is:
does
this
work
and
I
mean
it
kind
of
worked
it
kind
of
made
stuff
work.
We
didn't
go
down
all
the
time,
but
we
sometimes
went
down
and
that's
kind
of
what
happened
right.
We
had
these
two
major
problems
with
that
approach.
One
is
we
had
tons
of
tedious
human
interaction
where
you
had
to
remember
you
had
to
do
it
before,
and
there
was
no
formal
plan
around
this.
A
There
was
a
lot
of
just
me
and
mario
and
a
few
other
people
on
our
teams
running
around
saying:
hey,
dude,
you
know
marketing
is
going
to
do
something
at
one
o'clock
today
we
think
or
kanye
might
drop
something
tomorrow.
Are
you
guys
ready?
What
time
do
we
need
to
do
all
this
stuff
at,
and
that
was
very
time
consuming
and
painful?
B
A
Would
have
up
to
100
micro
services,
and
then
we
would
have
to
run
around
make
sure
we
had
all
of
these
things
at
once,
which
is
really
really
painful.
The
other
thing,
too,
is,
as
our
push
notifications
evolved.
It
also
started
hitting
different
surfaces.
So
all
of
a
sudden,
what
we
would
work
yesterday
would
no
longer
work
today
because
there's
a
new
usage
pattern
in
this,
and
that
was
always
hard
and
trying
to
keep
up
with
all
of
the
stuff
as
it
was
going
and
as
more
things
are
happening.
B
I'm
first
going
to
talk
about
what
we
did
in
the
reactive
realm,
which
was
very
sre
focused
for
the
most
part,
and
I
really
believe
that
dialing
in
the
reactive
components
with
hpa
and
cluster
autoscaler,
which
I'll
get
into,
is
really
really
pivotal
for
you
to
understand
the
types
of
challenges
that
you're
gonna
have
in
in
your
reactive
scaling
journey
before
you
go
on
to
build
something
else,
so
let's
get
into
it,
let's
first,
basically,
the
the
key
thing
with
our
infrastructure
at
stockx.
You
know
all
eks.
B
We
use
the
cluster
autoscaler,
which
just
the
existing
helm
packages.
For
that
pretty
simple,
we
fully
manage
our
nodes,
which
means
we
use
aws
unmanaged
nodes.
This
gives
us
access
to
configure
cubelet
certain
ways
or
run
things
like
node,
local
dns
and
then
with
with
cluster
autoscaler.
As
you,
you
work
on
that
it
actually
automatically
taps
into
the
aws
apis
to
interact
with
the
auto
scaling
groups.
B
So
there's
a
vod
that
you
get
for
free,
but
there's
a
lot
of
setup
as
well,
and
a
lot
of
dialing
in
tolerances
figuring
out
the
optimal
intervals.
We
don't
want
flapping.
We
don't
want
nodes
that
were
just
added
two
minutes
ago
to
get
removed
a
few
minutes
later
and
really
ironing
out
that
core
decision
making
of
when
do
we
scale
up
versus
when
we
scale
in
and
those
are
both
calculated
in
different
ways
with
cluster
autoscaler.
B
There's
plenty
of
documentation
on
this
on
the
kubernetes
github,
as
well
as
the
kubernetes
document
sites
as
well,
and
so
you
really
pretty
much
can
get
really
really
far.
However,
there's
there
are
definitely
some
nuances.
This
is
not
not
perfect.
Things
like
running
cluster
autoscaler
on
multiple
across
multiple
zones
takes
a
little
bit
more
effort
and
a
little
bit
more
watching
and
maintaining.
What's
going
on
dialing
in
those
those
configuration
values
and
and
the
stuff
right,
it's
very
dependent
on
your
workloads
as
well.
B
Of
course,
as
I
was
saying,
you
know,
definitely
not
perfect
getting
scale
in
just
right
is
really
tricky
and
you
obviously
don't
want
to
waste.
You
want
scaling
to
work,
but
it
is
really
something
that
you
have
to
iterate
on.
There's
lots
of
testing
we
had
to
do
there.
We
saw
a
lot
of
unbalanced
decision
making,
which
was
hard
to
understand.
B
I
mean
we,
we
again
could
see
the
status
of
the
cluster
autoscaler
and
what
it
was
doing,
but
we
were
wondering
a
lot
of
times
why
it
was
making
certain
decisions.
So
definitely
you
know
you're
gonna
need
to
watch
this
and
again
dialing
in
those
tolerances
will
help
you
get
there
and
then
definitely
actions
after
burst.
So
let's
say
we
just
added
six
nodes
to
the
cluster,
any
actions
that
needed
to
happen.
B
Where
we
added
more
nodes,
a
few
minutes
later,
would
actually
be
somewhat
hindered
by
probably
aws
api
rate
limits,
which
is
never
fun,
and
that
is
not
very
clear
to
you
right.
So
that's
something
you
have
to
dig
into,
and
then
we
had
to
even
customize
our
asgs
beyond
the
the
standard
that
was
upgraded
issues,
the
auto
scaling
groups,
we
had
to
do
a
suspended
action
on
those,
or
else
they
would
kind
of
fight
against
the
cluster.
B
It
was
making
a
good
decision,
not
the
best
ever
decision,
not
the
most
cost-effective
decision,
but
it
was
better
than
what
we
had
before,
which
was
nothing
right,
and
so
we
had
the
sense
that
our
clusters
could
scale
in
and
out
with,
the
capacity
that
we
needed
and
scaling
out
is
actually
a
pretty
fast
operation
when
it
comes
to
cluster
and
auto
scaler.
So
definitely
good
enough.
I'm
sure
it's
a
lot
better,
and
nowadays
that
takes
us
to
the
horizontal
pod,
auto
scaler.
B
We
with
the
current
versus
the
desired
metric
value.
So
we
actually
did
all
of
our
scaling
with
cpu
and
so
really
for
optimizing.
The
hpa,
there's
kind
of
different
facets
that
you're
looking
at
you're,
focusing
on
obviously
one
application,
how
it
is
working
and
so
with
cpu
you're,
looking
at
your
resource
requests
and
limits,
mainly
the
requests,
and
so
how
we
did
this
is
you
know
at
stockx,
because
I
focused
on
one
particular
service,
and
this
was
our
front
end
and
you
know
front
end.
B
Written
in
node
actually
lived
in
its
own
cluster
environment,
and
so
it
was
an
easy
kind
of
boundary
where
we
felt
comfortable
working
on
just
this.
This
instance.
Of
course,
we
did
this
in
in
devastation
environments
before
that
solely
on
cpu,
and
so
a
lot
of
this
was
analyzing
the
trends.
Of
course.
This
is
all
manual
work
you're,
using
your
your
data
dog
tools,
you're
using
your
k9s
terminal
user
interface,
to
look
at
the
kind
of
live
values
for
our
front
ends.
B
The
the
thinking
we
generally
did
was
tuning
our
requests
first
to
ensure
we
were
requesting
something
that
was
sane
and
then
we
could
scale
based
on
what
that
was
and
again.
This
is
the
requests,
allow
you
for
cpu
and
memory,
but
the
horizontal
scaler
only
cared
about
cpu,
and
then
we
would
tune
the
cpu
utilization
for
our
horizontal,
auto
scaler
definition
and
those
were
co-located
with
the
applications
themselves
in
their
helm,
deploy
one
of
the
key
things
that
we
really
like
like
and
I'll
mention.
B
This
multiple
times
is
building
in
buffers
building
in
a
safe
overflow.
So
if
an
application
is
deployed-
and
it
has
six
instances-
we
would
have-
maybe
you
know
eight
instances
or
twenty
five
percent
or
more
capacity
inherently,
that
we
wouldn't
consider
as
waste,
because
if
there
was
some
sort
of
operation
or
marketing
push
as
kyle
mentioned,
we
would
be
able
to
at
least
handle
that
traffic
for
some
period
time
until
we
got
more
scale.
B
The
other
part
of
this
is
events
and
knowing
what's
going
on,
especially
during
the
testing
phase.
Kubernetes
makes
this
relatively
easy
to
find
it's
a
little
bit
tricky.
Looking
at
the
deployment
object
and
looking
at
the
hpa
object,
you
can
get
a
sense
of
this
and
a
lot
of
platforms.
I
know
we
were
using
datadog
will
let
you
feed
this
in
and
then
it's
an
event
that
you
can
act
on
as
well.
B
Those
are
really
important,
and
so
you
can
see
that
we
managed
to
get
to
40
000
requests
per
minute
under
300
millisecond
response
times,
which
is
a
huge,
huge
win
from
where
we
were
before
and
again.
This
is
just
kind
of
flattening
those
curves
out
when
it
comes
to
response
times.
Making
sure
error
rates
are
staying
low,
that's
huge
and
we
again
good
enough.
It
was
never
perfect
there's
this
is
unlimited.
You
can
dial
in.
B
You
can
kind
of
react
as
much
as
you
want
to
and
try
to
get
it
as
perfect
as
possible.
That's
unlimited
time
you
can
spend
on
that.
We
we
couldn't
do
that.
We
definitely
spent
plenty
of
time
testing
it.
We
had
many
different
scenarios,
as
kyle
mentioned
when
kanye
feels
like
doing
a
drop,
we
had
plenty
of
data
to
go
off
of
so
definitely
dialed
it
in
really
well-
and
this
is
me
listening
to
music,
as
I'm
coding
and
working
on
hba
so
hpa
was
fun.
A
Thank
you,
mario
and
cool.
We're
going
to
go
back
to
the
other
side
of
this
we're
going
to
start
talking
about
some
proactive,
auto
scaling.
An
hpa
is
great,
but
it's
not
always
enough.
It's
great
when
we
have
these
organic,
normal
ups
and
downs
of
traffic.
That
normally
happen
in
services,
but
when
we
have
traffic
drops
you're
fine
too
right,
like
it's
really
only
a
problem
when
you
have
a
dramatic
step
increase
in
traffic,
and
you
can
kind
of
see
that
mario's
graphs
before
when
that
traffic
would
sharply
increase.
A
That's
really
the
only
time
you'd
have
a
problem
when
we
had
these
drops-
or
you
know
it's
just
a
normal,
slow
flow
up
or
down
of
traffic
you're.
Normally,
okay
and
that's
what
the
problem
we're
trying
to
solve
is
that's
where
our
battle
was
is
when
we
have
these
very
big
spikes
in
traffic,
due
to
a
push
notification
and
or
a
drop
or
something
that
would
cause
a
ton
of
people
to
instantly
open
up
an
app
or
instantly
show
up
to
a
site.
A
We
also
kind
of
hold
ourselves
back
when
we
go
way
over
that
and
we're
constantly
going
up.
It
actually
waits
for
a
little
bit
before
it
actually
starts
scaling
yourself
up
to
make
sure
it's
something.
That's
not
going
to
cause
these
up
and
down
this
flapping,
and
that's
also
one
of
the
problems,
but
there's
also
a
few
other
things
layered.
On
top
of
that,
that's
kind
of
what
happens,
but
the
reason
that
happens
is
a
few
things
mainly
between
your
configure
delay,
which
you're
telling
it
to
wait
for
and
you're
also
your
metric
intake.
B
A
And
say:
hey:
we
have
these,
this
large
increase
in
traffic
before
you
can
even
really
start
to
say,
hey,
we
should
start
scaling
or
in
a
lot
of
cases
hey.
We
should
scale
in
five
minutes
if
we
still
have
this
hard
traffic,
maybe
not
five
minutes,
but
you
know
some
time
frame
in
the
future
and
again,
a
lot
of
this
stuff
is
really
time
sensitive.
So
if
you
wait
that
time,
you
wait
that
configured
delay
time.
Your
app's.
A
Point
most
of
the
times,
so,
if
you
think
of
a
marketing
push
notification,
if
you
open
up
your
phone
and
say
hey
there's,
you
know
this
new
shoe
or
there's
this
new
thing
you
hit
it
and
the
app
crashes
or
the
app
you
know
gives
you
a
light
screen.
You're
normally
dead
at
that
point
right,
because
people
normally
don't
stay
around
and
keep
trying
to
open
your
app
or
keep
trying
to
get
to
a
page
unless
they
really
want
something,
but
a
lot
of
times.
A
People
will
open
it
up
and
then
kind
of
you
know:
hey
it's
not
working
and
then
leave
and
that's
kind
of
where
our
problem
is.
We
had
these
massive
massive
massive
step
increases
in
traffic.
That
would
really
cause
us
these
huge
problems,
because
we
would
send
out
a
marketing
marketing
push.
Everybody
would
get
a
little
banner
you'd
hit
the
banner
and
then
all
of
a
sudden
boom
our
traffic
would
go
up
and
a
lot
of
people
would
ask:
how
much
does
your
traffic
go
up?
Well
through
our
math?
A
It
actually
goes
up
about
533
in
less
than
30
seconds.
So
that
is
a
lot
of
people
hammering
our
app
very,
very,
very
fast,
so
where
we
would
have
these
hp's
kick
in
in
the
future.
It's
very
hard
to
have
them
scale
for
533
increase
almost
instantly
and
then
not
have
them
do
that
in
other
times.
So
you
would
ask
you
know
hey
what
what
does
this
look
like?
Give
me
some
data
of
what
it
would
look
like
and
here's
our
data
right.
A
A
Is
you
can
see
these
lines
constantly
going
up
and
at
some
points
we
would
hit
over
90
errors
and
sometimes
we'd
even
hit
as
high
as
95
percent
errors,
which
means
you
know
one
out
of
20
people
is
not
having
a
problem,
but
19
people
are,
and
that's
really
really
hard
to
deal
with
and
going
back
to
the
scaling
parts
here.
This
is
our
pod
counts
and
our
pod
charts.
You
would
see
here
right.
A
We
have
these
very
nice
little
curve
up
curve
up
curve
up
where
we're
trying
to
catch
up
and
they're,
almost
perfectly
incremented
to
where
that's
where
our
delays
would
kick
off,
like
we've
delayed
for
so
long
start
scaling
up
to
the
point
where
you're
good
at
your
next
point
and
take
some
metrics,
try
it
again
and
we
would
level
out
sometime
after
the
push.
But
again
at
that
point
we've
lost
so
much
traffic,
so
many
users
that
it's
not
worth
it
anymore.
A
We
would
want
this
to
happen
all
before
and
that's
where
we
created
something
called
barricade.
Barricade
is
basically
our
version
of
a
warming
solution
where
we
would
spin
up
all
these
pods,
but
we're
going
to
do
that
before
we're
not
going
to
do
that.
We're
not
going
to
wait
for
the
hpa
to
wait
for
stuff
we're
actually
going
to
do
it
before
and
that's
what
we
called.
It
is
a
warming
system,
so
it
warms
our
kubernetes
servers
and
systems
to
basically
be
ready
for
that
push.
So
we
would
do
this
before.
A
After
the
fact-
and
this
is
what
it
looked
like,
isn't
it
you
know
a
practical
example
of
our
impact
on
pots,
we
put
it
on
one
service
to
test
it
out,
and
this
is
the
graph
we
saw.
There's
a
box
in
the
left
hand
side
here
where
you
see
that
real,
really
really
really
strong
jump
up,
that's
equivalent
to
all
of
the
smaller
jumps
you
saw
before,
but
it
happened
before
anything
actually
happened,
which
is
kind
of
the
point
of
this.
A
So
now,
when
that
service
got
hammered
with
stuff,
we
would
actually
be
ready
for
it
we're
no
longer
waiting
for
everything
to
reactively
scale
and
hopefully
that
it
would
scale
all
the
way
up
to
the
point
that
it
needed
to
we're
telling
it
before
hey.
This
is
where
we
need
to
set
it
so
we're
okay.
After
the
fact
or
when
this
push
does
hit,
we're
not
going
to
be
underwater,
you
know
get
to
the
end
point
of
all
of
our
pods.
A
To
start,
and
again
I
can
say
all
this
stuff,
it
doesn't
mean
anything
unless
we
have
data
to
back
it
up
and
data
of
why
this
is
a
good
idea.
These
are
side-by-side
charts
of
a
push
notification
before
and
a
push
notification
after
we
added
barricade,
as
you
can
see
the
graph
from
before
we
had
these
huge
air
spikes.
We
would
hop
up
to
95
errors
at
points,
but
we
were
consistently
over
85
errors.
A
But
if
you
look
after
when
we
just
do
basically
what
the
hpa
would
do
throughout
this
process
and
after
we
hit
zero
percent
air
rate,
most
of
the
time
we're
under
one
percent
air
rate,
which
normally
is
due
to
you,
know,
hey
a
bad
you,
you
are
not
a
bad
uuid,
a
bad
product
id
or
somebody
not
being
authenticated.
You
know
the
normal
stuff
of
people
just
using
the
website,
but
we're
no
longer
getting
these
problems
of.
A
We
don't
have
enough
resources
to
really
help
you
with
your
request
and
now,
let's,
let's
get
into
a
little
bit
of
what
is
what
happened
when
we
created
barricade,
we
really
had
a
few
driving
forces
in
it.
One
of
the
easiest
ones
that
we
had
was
really
keeping
it
simple.
We
had
a
process
that
worked
before
where
we
would
truly
just
go
and
do
this
stuff
manually
and
we
would
manually
go
around
the
cube,
ctl
commands.
We
would
scale
our
stuff
up
and
we
hope
we
got
everything.
A
All
we
really
did
is
take
that
same
idea.
Keep
it
really
simple
and
just
have
a
service.
Do
it
for
us,
so
that
service
now,
which,
instead
of
us
running
that
kubectl
command
the
service,
runs
it
and
the
service
is
normally
much
more
reliable
than
we
are
because
it
remembers,
and
it
has
a
system
inside
of
it
to
remember
when
and
where
to
do
all
this
stuff,
and
the
other
thing
that
we
really
kept
simple
is
it
has
three
jobs?
We
didn't
try
to
overload
this.
A
We
didn't
try
to
add
a
ton
of
things
to
it.
We
truly
just
said:
hey,
it's
gonna
intake
events,
it's
gonna
warm
stuff
and
it's
gonna
cool
stuff,
nothing
more,
nothing
less!
It
does
one
very
pointed
job
and
it
does
it
very
well.
But
let's
start
talking
about
some
of
the
processes
we
use
to
make
this
really
nice
because
it
really
illustrates
how
nice
it
is.
A
Now
versus
what
it
was
before
so
now,
what
ends
up
happening
is
we
have
a
lambda
that
pulls
a
third-party
system
to
see
if
there's
something
going
on
for
marketing,
if
it
does
see
something
it
sends
it
to
barricade,
barricade
then
ingest
that
that
event
knows
when
it's
going
to
happen
and
then
that
internal
crown
job
will
eventually
pick
it
up
and
just
run
that
event
warm
our
pods
and
cool
our
pods
off
as
well.
We've
also
seen
a
few
other
things
where
we
have
a
micro
service.
Do
this.
A
So
if
ours
team
is
putting
something
in
our
internal
services
that
schedule
this
stuff
to
go,
do
something
like
an
email
or
whatever
they
can
go
talk
to
barricade
and
the
beauty
of
keeping
barricade
as
simple
as
we
did
is
anything
can
truly
do
this
and
that's
the
beauty
of
it.
We
no
longer
have
to
have
everybody
know
this
mat.
You
know
this
kubernetes
knowledge
to
know
what
and
when
to
go,
do
stuff,
and
you
know
how
should
I
put
it
up?
How
should
I
put
it
down?
Where
do
I
put
my
pod
counts?
A
How
far
should
I
you
know,
go
with
the
cpu
scaling?
The
beauty
of
this?
Is
you
have
an
api
call.
You
make
that
api
call
just
like
you
would
if
you
were
trying
to
update
a
user
or
reset
your
password
or
sign
in
and
barricade
will
do
the
rest
for
you.
You
don't
have
to
worry
about
anything
else
and
that's
the
beauty
of
it
and
just
to
kind
of
recap,
quick.
What
makes
barricade
so
useful?
Is
it's
really
easy?
A
But
if
you
have
other
application
developers
on
your
team
at
your
company,
they
might
not
know
that
as
much
and
the
ease
of
use
of
basically
saying
hey,
here's,
your
endpoint,
here's
the
few
things
you
send
it
and
it
just
works,
and
you
don't
have
to
deal
with
it-
is
a
huge
huge
benefit
because
you
don't
have
to
share
this
knowledge
all
the
time
and
force
people
to
learn
stuff.
They
might
not
be
comfortable
with
the
other
huge
thing
about
it.
Is
it's
really
set
that
you
can
fire
and
forget
it?
A
You
put
it
in
barricade.
Barricade
will
handle
it
from
there
unless
everything
you
put
in
barricade
completely
wipes
itself
out
somehow
crashes,
your
databases
crash,
you
have
a
full
system
outage.
You
can
forget
about
it
and
it'll
just
work
at
the
end,
which
is
a
huge
huge
benefit
and
the
biggest
part
of
this.
A
I
don't
know
how
I'm
ever
going
to
start
to
feel
better
about
these
graphs
after
this,
because
now
I
have
a
90
that
I
can
drop
on
a
slide
and
say
it's
nice,
but
again
it's
a
huge
win.
It's
a
huge
thing.
If
you
just
proactively
scale
for
things
that
are
gonna
crash,
your
system,
you
can
see
these
awesome
takeaways
and
now
mario
is
coming
back
with
just
a
few
nice
things
to
take
away
from
this
talk.
B
All
right,
so,
thank
you.
Kyle
here
are
the
takeaways.
Of
course.
We
didn't
do
everything
perfect,
I
guarantee
it
so.
First
I
wanted
to
cover
hpa
and
if
we
were
to
attack
it
today
and
implement
it
for
our
services,
some
of
the
things
we'd
be
thinking
about
the
space
has
gotten
a
lot
better
with
many
more
people
focusing
on
this
problem
of
resource
management.
B
Stormforge
is
a
fantastic
company,
formerly
carbon
relay,
and
they
have
just
made
massive
strides
on
providing
that
loopback
of
testing
continuously
to
get
a
sense
of
what
your
resources
should
be
and
what
makes
the
most
sense
for
the
long-term
scalability
of
your
application.
Keto,
which
is
a
cloud
native
computing
foundation.
Cncf
project
is
really
really
cool,
letting
you
scale
on
things
like
your
q
size
from
sqs
and
other.
B
B
There's
a
slew
of
new
hpa
features
that
are
more
than
just
what
is
our
target
percentage,
but
you
can
actually
leverage
custom
and
external
metrics,
which
we
would
have
done
a
lot
more.
I
know
datadog
now
offers
you
the
ability
to
use
any
metric
in
datadog
and
scale
on
that
there's
also
more
container
focus
metrics.
So
it's
not
just
the
pod
metrics
of
which
the
hpe
makes
a
decision.
B
You
can
make
decisions
on
solely
containers
within
that
pod
and
then
behavior
policies
with
rate
and
window
support
is
huge
for
hpa
and
really
ensuring,
like
I
was
saying,
with
closer
autoscaler
that
we
have
safe
intervals,
that
we
are
we're,
not
thrashing
thresholds
are
saying,
etc,
and
this
is
unique
to
everyone
so
making
those
something
that's
configurable
is
really
awesome.
So
I
think
that's
good
for
that
side,
yeah.
So
if
we
look
at
a
few
other
things
I
wanted
to
mention,
you
know
always
keeping
sufficient
buffer.
B
I
mentioned
that
before
that
will
always
be
helpful.
I
think
hopefully
you're
doing
that
already,
preferring
more
instances.
We
always
told
our
developers
if
you
were
ever
questioning
how
many
instances
lead
on
the
higher
side.
We
are
willing
to
spend
the
money
up
front
to
ensure
that
your
application
has
the
capacity
it
needs
when
a
traffic
event
does
a
hit.
B
Slis
slos
slas,
the
sre
methodologies
and
principles
are
really
important.
Imparting
these
on
developers
is
definitely
another
story,
but
there
are
lots
of
organizational
things.
You
can
do
12-factor
methodologies
and
other
resources
out
there
that
you
can
share
and
help
implement
these
work
with
developers,
not
against
them.
You
know
there's
a
lot
of
misconceptions
around
the
kind
of
offside
of
the
table
and
the
the
development
side
of
the
table
where
developers
are
throwing
things
over,
they
don't
understand
they
don't
care.
A
B
They
should
care
as
a
service
owner
and
with
the
long-term
life
cycle,
and
you
know
they
don't
want
to
get
pinged
by
pager
duty.
So
you
know
there's
many
many
ways
you
can
impart
knowledge
in
a
more
flexible
way.
What
happens
when
what
happens
when
you
do
a
deployment
and
you're
currently
in
a
traffic
event,
and
you
are
not
sure
that
that
is
something
you
wanted
to
happen,
because
you
just
want
things
to
be
stable,
so
this
this
dials
into
from
code
going
through
a
pipeline
getting
deployed.
B
What
are
the
controls
that
you
have
in
place
there?
This
really
relies
a
lot
on
your
pipeline
and
how
you
structure
your
deployments.
Of
course
everyone
should
be
considering
open
source
and
third-party
tools.
I
mentioned
goldilocks
there's,
you
know,
stormforged,
there's,
there's
a
lot
of
tooling
out
there
spot
to
io
also
helps
with
waste
and
nodes
as
well.
So
there's
a
lot
to
consider,
I
think,
just
soaking
for
a
little
while
testing
is
really
important
and
I
think
just
a
couple
more
things.
B
B
You
don't
have
to
worry
about
the
nodes
being
spawned
the
size
of
those
nodes.
The
types
allocated
versus
actual
capacity
you
know
so
moving
on,
I
know
hps
are
great.
They
have
no
set
of
challenges.
I
hope
you
you've
seen
that
again.
It's
it's
trial
and
error
kiss
keep
it
simple
stupid
one
of
my
favorite
favorite,
mottos
and
manual
first
at
automate.
B
I
think
it's
one
of
those
things
where
you
have
to
do
it
the
hard
way
before
you
can
do
it
the
easy
way
to
understand
the
nuances
of
of
how
something
actually
works
and
what's
going
on
behind
the
scenes,
so
I
think
the
excellent.
Let
me
talk
about
myself,
some
more
so
I'm
mario
lauria.
Thank
you
so
much
to
everybody
for
tuning
in
today.
I
look
forward
to
some
amazing
questions.
I
work
at
a
company
called
carta.
B
If
you
have
stock
options,
you
might
have
heard
of
carta,
we're
also
doing
a
lot
more
around
409
evaluations
and
releasing
card
x,
which
is
letting
people
now
invest
in
private
companies,
which
is
really
really
cool.
It
helps
people
holding
stock
options,
get
some
liquidity
as
well.
So
you
can
visit
me
on
the
web.
Mario
laureate
dev
links
to
all
of
my
other
resources,
and
I
think,
with
that
I'm
gonna
hand
it
off
to
kyle
to
talk
a
little
bit
more
about
himself.
A
As
mario
and
I
mentioned
earlier,
we
both
work
at
stockx.
I
am
still
at
stockx.
If
you
guys
are
interested,
we
are
hiring
at
all
positions.
If
you
go
to
stockx.com
scroll
down
to
the
bottom,
there's
a
jobs,
link
click
that
everything
is
listed
out
there.
We
have
a
tech
blog,
it's
recent
there's
a
lot
of
graphql
stuff.
Like
I
said
I
like
graphql
a
little
too
much
so
you'll
probably
see
my
face
on
a
lot
of
the
stuff
feel
free
to
go.
A
Read
it
go
talk
to
me,
go
tell
me
it's
either
great
or
bad
whatever
my
twitter
is
on
there
as
well.
So
if,
whatever
you're
feeling,
let
me
know
it's
a
good
time
and
with
that
I
would
like
to
say
thank
you
guys.
Thank
you,
everybody
for
the
talk
for
listening.
Thank
you,
everybody
at
the
cncf
for
putting
this
on.
It's
an
awesome
time.