►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Predictive Application Scaling with Prometheus and ML - Chris Dutra, Schireson
Schireson, a New York City based data science and engineering firm, overcame a challenge in how to properly scale kubernetes services for their application stack. Consuming data from several tools in the CNCF portfolio (such as Envoy and Prometheus), Schireson developed home-grown machine learning to actively predict the resource requirements for its services at any given time. The models act in a semi-supervised state to ensure the overall stability of their data science platform. This talk will illustrate the steps taken to construct the models, and offer suggestions to the larger DevOps community on how to implement Predictive Application Scaling in their organization.
To learn more: https://sched.co/GrVz
A
Well,
thanks
everyone
for
attending
this
is
a
huge
crowd,
so
really
appreciate
you
guys
shown
up.
My
name
is
Chris
nutria
I
run
site
reliability.
Engineering
at
Shearson,
Shearson
is
a
data
science
company,
headquartered
in
New,
York
City,
and
we
do
qualitative
quantitative
market
research
as
well
as
data
science
and
media.
Optimization
long
story
short.
We
help
our
clients
with
a
variety
of
different
things
in
marketing
and
advertising.
You
can
find
me
on
github.
You
can
find
me
on
Twitter
links
here
and
also
after
this
presentation.
You
have
any
questions
or
anything
like
that.
A
What
that
really
means
is
that
we
have
data
scientists
that
typically
would
put
this
work
together
and
send
it
off
to
a
client
and
our
job,
as
engineers
has
been
to
build
the
data
science
processes
and
actually
make
them
a
platform.
That
is
a
service
for
the
clients
to
be
able
to
run
the
data
science
workloads
themselves.
So
hard
reason
here,
because
we
are
running
kubernetes,
we're
using
a
lot
of
neat
technology
and
due
to
the
advertising
industry
and
kind
of
not
really
having
any
regulations.
A
A
So
that
was
a
initial
hurdle
of
thinking
of
data
scientists
writing
things
to
their
hard
disk
on
the
laptop
and
then
using
that
putting
that
into
volumes
that
that
we
had
to
do
a
lot
of
that
kind
of
retrofitting.
For
for
those
processes,
also
in
order
to
get
our
clients
to
actually
use
this.
Instead
of
calling
the
data
scientist,
we
had
to
achieve
at
least
similar
performance
to
what
a
data
Tess
was
running
on
a
laptop,
so
they
had
big
beefy
desktops
that
we'd
be
basically
competing
with
with
the
kubernetes
clusters.
A
That
was
another
interesting
challenge
and
also,
as
I
mentioned,
before,
providing
data
science
as
a
service
to
clients.
So
we
had
to
make
it
intuitive.
We
had
to
make
it
easy.
They
had
to
understand
what
they
were
doing.
You
know,
as
they
were
adjusting
different
parameters
or
choosing
different
data
sources,
so
these
are
all
things
that
kind
of
came
in
to
how
we
started
moving
these
data
science
workloads
into
a
production
grade
cluster.
So
our
first
attempt
that
this
was
well.
Let's
try
to
put
this
into
a
deployment
that
did
not
work
out
too
well.
A
As
you
can
see,
there's
a
lot
of
spike
enos
with
data
science
processes.
We
had
a
lot
of
evicted
pods,
we
had
a
lot
of
killed
nodes.
It
took
us
a
long
long
time
to
realize
that
the
traditional
way
of
deploying
an
application
was
not
gonna
work
for
data
science.
Also,
you
can
see
from
here.
This
is
actually
memory
in
our
GU
fauna.
It
takes
up
a
lot
of
memory,
so
we
also
realized
that
isolation
was
pretty
important.
A
A
What
happens
when
our
processes
start
to
crash,
or
you
know
we
needed
to
account
for
those
types
of
situations,
so,
as
you
might
have
been
able
to
guess,
we
actually
ended
up
going
with
daemon
sets.
Part
of
this
was
is
that
we
actually
retain
a
lot
of
the
orchestration
capabilities
that
we
have
the
rest
of
our
software,
so
applications.
A
We
were
essentially
just
adding
new
nodes,
to
instance,
groups
and
kubernetes,
and
the
rest
of
the
process
was
taken
care
of
pretty
seamlessly,
and
this
is
kind
of
just
a
short
example.
I
spent
up
in
a
femoral
cluster
just
to
kind
of
explain
this.
So
the
you
know,
here's
a
sample,
one
master
lots
of
worker
nodes
and
we
would
use
a
node
select
there.
Just
isolate
resources
and
the
daemon
set.
You
can
use
a
node
selector
to
say.
A
I
only
want
these
nodes
to
have
this
run,
and
then
our
back-end
API
nodes
would
just
have
the
same
notes
like
there,
but
for
the
API.
So,
as
you
can
see
here
with
the
pods
underneath
these
data
science
processes
just
essentially
had
remove
running
on
those
five
nodes
that
are
the
ones
that
are
labeled
here.
So
this
ended
up
working
out
great
and
kind
of
that
led
us
to
where
we
are.
Today
we
have
the
data
science
processes
running
as
daemon
sets.
A
We
have
Python
and
go
micro
services
that
are
backed
by
a
Ambassador,
API
gateway,
which
is
using
envoy,
underneath
we
have
Prometheus
core
fauna
elk
and
view
jeaious,
nginx
front-end,
also
some
middleware,
as
well
as
helm
and
fluency
as
part
of
our
logging.
The
we
also
have
some
managed
databases
such
as
redshift,
RDS,
s3
and
we've
been
pretty
successful
with
airflow
as
far
as
data
processing
and
pipelines.
A
But
this
is
great,
this
solves
I
would
say
half
the
problem.
The
other
half
of
the
problem
is,
is
how
do
we
scale
and
there's
three
major
considerations
here?
We
don't
want
the
person
doing
the
scaling,
a
human,
because
we're
gonna
burn
out
feel
like
a
better
term.
Also
I
can't
just
spin
up
200
nodes
to
run
data
science
because
my
boss
is
gonna,
call
and
say:
why
is
the
bill
so
high
I'm
sure
public
cloud
vendors
don't
mind
that,
but
I'm
sure
our
company
does,
but
also
to
is.
A
A
So
we
do
have
data.
We
are
running
Prometheus
and,
as
we
were
looking
at
some
of
our
data
science
nodes,
we
can
BC
know
places
where
we
can
actually
scale
down.
But
the
challenge
here
is:
is
that
the
time
slice
will
you
go
from
needing
a
node
to
having
one
is
very
impactful,
because
you're
still
waiting
to
be
able
to
take
on
that
next
data
science
workload?
So
we
actually
need
it
to
be
about
10
15
minutes
ahead
of
knowing,
when
we
needed
to
have
another
node
spun
up
to
handle
another
data
science
workload.
A
So
you
know,
as
you
can
see
here,
this
is
a
dace
add
a
time
slice
of
the
CPU
utilization
and
we
kind
of
came
to
the
conclusion
that
we
could
probably
scale
down
some
of
our
resources
here
and
save
some
money,
especially
when
you
think
of
hourly
prices
and
also
some
of
the
licensing
that
we
have
first
different
algorithms
that
run
so
our
solution
essentially
was
to
use
timeseriesforecasting.
We
had
this
great
time
series
database
Prometheus
and
we're
actually
using
it
to
anticipate
changes
to
key
metrics.
A
So
our
SRE
team
can
write
size
infrastructure
provide
more
redundancy
during
peak
times.
So
if
you
know
a
clients
on
the
west
coast,
we
know
that
we
just
scale
up
resources
there,
instead
of
us
East
and
also
reduce
cost
during
non
busy
periods
so
employing
that
we
have
a
collection
of
data.
Scientists
I
took
this
problem
to
them
and
said:
what's
the
best
way
that
we
can
utilize
all
of
this
data
and
they
were
really
excited
about
it.
So
we
ended
up
going
back
and
forth
and
to
be
fair,
there's
a
lot
of
different
ways.
A
You
can
look
at
this
data
and
model.
It
there's
a
ton
of
really
great
machine
learning
resources
out
there,
but
we
ended
up
going
with
LS
TM
and
just
a
quick
show
of
hands
how
many
people
are
familiar
with
neural
networks:
okay,
Wow,
okay,
and
how
about
long
short-term
memory
and
one
more
caris?
How
many
people
are
working
with
carrots
today?
Okay,
all
right!
A
A
In
short,
I
mentioned
before
LST
M
stands
for
long
short-term
memory.
It's
a
type
of
neural
network
called
an
RNN
or
recurrent
neural
network,
and
it's
initially
designed
to
recognize
patterns
and
sequences
of
data.
So
a
good
example
of
this.
You
might
have
seen
some
papers
I.
Think
Google
is
working
on
something
with
image,
recognition
and
like
working
on
different
sequences
of
numbers
and
matching
patterns.
That's
one
of
the
common
use
cases
for
it,
but
when
you
think
of
a
situation
where
I've
been
talking
for
about
you
know
10
minutes
or
so
now.
A
A
But
it's
probably
not
going
to
be
something
like
bananas
or
sleeping.
So
you
know
what
a
neural
network
is.
Gonna
do
is
predict
that
next
sequence,
what's
the
next
word
and
potentially
going
to
be
a
couple
of
other
neat
ways
that
the
community
for
machine
learning
and
LS
TM
have
been
doing
it?
Is
we
think
of
Mozart,
for
example?
A
And
let's
say
you
listen
to
10
hours
of
music
of
Mozart
and
you
have
a
new
song
and
you're
10
seconds
in
you
know
the
model
would
be
able
to
predict
the
next
musical
note,
perhaps
or
be
pretty
close
to
what
that
would
sound.
Like.
Let's
say
you
took
every
work
of
Shakespeare
and
you
loaded
that
in
it
essentially
works
of
art
or
you
know
their
sequences.
You
know
the
sequence
of
words
there.
So
what
would
be
you
know
if
you
got
a
paragraph
of
Shakespeare?
A
If
some
title
ii
didn't
know,
could
you
predict
the
next
word?
That's
another
thing
that
an
LS
TM
model
would
do
also
timeseriesforecasting.
So
this
is
something
that's
kind
of
more
common
in
business.
So
if
you
think
of
you
know
sales
revenue,
you're
always
going
to
want
to
forecast
out
I
see
investment
bankers
nodding
their
heads,
that's
good,
so
yours
gonna
want
to
forecast
out
you
know:
what's
the
next
quarter
or
the
next,
you
know
two
quarters
to
be
able
to
kind
of
gauge
your
business,
so
LS
TM
models
also
play
an
important
role
there.
A
So
we
decided
that
we're
going
to
go
down
this
path.
We
were
working
with
data
scientists
and
we
had
all
this
awesome
data
and
Prometheus
that
we're
going
to
use
to
actually
shape
how
our
kubernetes
cluster
operates
and
the
underlying
infrastructure.
So
we
would
collect
the
metrics
and
then
those
metrics
would
be
fed
into
build
and
deploy
and
Alice
TM
model.
A
That
model
would
then
be
able
to
take
in
metrics
that
are
streaming
real
time
for
Prometheus
and
make
predictions
and
we'll
talk
about
that
a
little
bit
later
and
then
what
we're
working
on
now
actually
is
taking
those
predictions
and
making
decisions
on
the
underpinning
infrastructure.
So
there's
certain
things
that
we
can
do
with
kubernetes.
A
So
you
think
of
you
know
if
I
predict
fifteen
minutes
out
that
I
need
to
scale
up
my
pods
I'm
going
to
want
to
be
able
to
connect
to
the
communities
API
and
scale
that
up
likewise,
if
I
need
more
infrastructure,
then
I'm
gonna,
you
know
talk
to
my
cloud
vendor
and
say:
I
need
more
infrastructure
through
another
API.
The
reason
why
we're
not
there
yet
is
because,
as
we're
still
training
these
models,
I
don't
want
to
bring
down
production.
It's
so
not.
A
This
is
not
something
that's
deployed
in
production
yet,
but
what
we're
working
on
is
the
observability
of
the
predictions
that
are
coming
out
and
the
arm
is.
Every
team
is
actually
adjusting
that
as
we
go,
so
we're
still
kind
of
fine-tuning
and
we're
still
going
through
the
discovery
phase,
but
I'm
going
to
walk
it
through
each
of
these
steps.
Next,
so
the
first
one
which
is
pretty
straightforward,
is
that
we
need
to
acquire
these
metrics
with
this.
You
want
to
play
with
the
data
range
because,
ultimately,
you're
gonna
do
a
range
query
from
Prometheus.
A
The
the
key
thing
here
is
is
that
you
don't
want
to
take
such
a
large
data
range,
because,
if
you
think
of
your
application-
let's
say
your
startup
and
what
your
app
look
like
six
months
ago
as
far
as
traffic
may
not
be
what
it
is
today.
So
you
how
that
model
is
trained,
is
gonna
be
on
that
full
data.
So
you
want
to
be
kind
of
cognizant
of
you
know.
A
Do
I
want
how
heavily
do
I
want
to
look
at
my
traffic
from
six
months
ago
and
have
that
play
a
role
in
how
predictions
are
generated.
So
what
we
do
is
actually
three
weeks
and,
as
you
know,
more
folks
are
coming
on
and
using
the
platform
we
have
a
nice
sliding
scale
of
how
the
predictions
are
being
generated
from
there.
A
We
format
the
data,
you
know,
Prometheus
we're
calling
the
API
getting
JSON
back
so
will
formatting
it
into
a
way
that
can
be
loaded
into
a
data
frame
in
Python,
and
you
know
some
example
metrics
that
we've
played
around
with
just
to
kind
of
get
an
understanding
of
how
modelling
and
so
forth
would
work.
Replica
counts
are
kind
of
straightforward,
yet
pod
counts.
Maybe
you
have
an
HPA,
that's
already
doing
some
stuff,
and
you
want
to
just
predict
where
you
are.
That's
one
great
way
to
do
it
RPS
or
requests
per.
A
Second,
that's
a
key
one
that
we've
seen
a
lot
where
you
know
every
graph
on
a
dashboard
I
feel
like
has
one
that
is
on
display
in
the
showroom
also
upstream
latency,
so
ambassador
has
been
great
and
we've
been
using
the
underlying
Envoy
stats
as
a
way
to
look
at.
You
know
which
api's
of
our
of
our
kubernetes
service
are
being
hit
the
hardest,
and
we
can
actually
use
that
to
figure
out
like
where
do
we
need
to
scale,
and
we
can
train
a
model
off
of
that
as
well.
A
A
So
that's
first
step,
so
you
have
your
data,
but
one
quick
thing
is
that
there
is
a
limitation
in
Prometheus,
but
I
wouldn't
call
it
limitation
because
you
don't
want
to
bring
down
your
Prometheus
server.
So
when
you're,
making
these
large
queries,
you're,
probably
gonna,
want
to
batch
them,
and
the
reason
for
that
is
is
because
there's
a
maximum
resolution
of
eleven
thousand
points
for
time
series.
A
So,
if
you
try
to
query
your
Prometheus
server
and
say,
give
me
everything
from
six
months
ago
to
today,
I
wanted
a
rate
of
five
minutes,
not
gonna
work,
they're
gonna
come
back
and
say,
reduce
your
time
series.
So
a
couple
of
things
here.
This
is
just
kind
of
some
commented
out
go
code,
so
we
are
basically
taking
the
time
and
putting
it
into
unix
time.
So,
if
you
aren't
familiar
with
UNIX
time,
it's
a
64
bit
integer
counting
the
number
of
seconds
since
January
1st
1970.
A
That's
basically
what
we
theist
does
when
you
set
up
the
range
of
your
range
query
so
from
the
from
the
so
that
way
you
can
batch
that
out
really
nicely
and
then
basically
aggregate
it
together
and
put
it
into
the
file
for
the
data
frame.
I've
got
a
brief
demo
of
it
right
here,
so
this
is
just
one
day
and
it's
gonna
basically
grab
60
minutes
at
a
time.
You
see
the
UNIX
timestamps
here,
pulling
all
that
together
and
then
we
basically
formatted
it
into
a
CSV.
A
So
it's
just
easy
to
load
into
another
other
file
system
or
another,
the
next
part
of
the
application.
So
it's
gonna
take
a
look
at
the
CSV.
This
is
our
dev
server,
so
there's
not
a
ton
of
activity,
but
as
we
scroll
through,
none
of
our
developers
are
working
late,
just
disk,
but
then,
as
we
see,
we
go
through
here
and
there's
some
RPS
that's
coming
through.
So
this
is
kind
of
what
the
file
looks
like
as
you
hand
it
off
to
the
next
step.
A
This
has
been
super
helpful
to
go
from
nothing
to
something
very
quickly,
and
so
the
big
thing
that
you
want
to
do
when
you're
starting
out
is
take
your
entire
data,
set
split
it
into
what's
test
and
training
sets
I
recommend
with
a
time
series
doing
we
do
60%
it's
the
first
60%
of
the
data
set
is
training
and
the
remainder
is
test
done.
There's
some
transformation
steps
you
need
to
do
basically
transform
the
data
into
what's
called
supervised.
Data.
A
I
could
talk
about
that
a
little
bit
later
and
then
also
scaling
so
scaling
basically
takes
it.
The
most
common
one
to
do
is
the
hyperbolic
tangent
takes
all
of
your
data
points
and
goes
from
negative
one
to
one
and
then
from
there
you
fit
the
model
and
there's
a
couple
things
here
that
our
data
scientists
do
to
prevent.
A
Overfitting,
you
may
have
heard
of
that,
which
basically
means
that
you
don't
want
your
model
to
match
your
training
set
exactly
because
predictions
then
are
going
to
be
are
not
going
to
be
based
off
of
anything
other
than
the
training
set.
So
you
know
we
do
early
stoppage
and
kind
of
more
advanced
topics,
but
I'm
I'm
not
gonna,
not
going
to
cover
that
for
this
talk,
but
just
know
that
there
are
ways
of
you
know
configuring,
your
epics
and
neurons
on
how
you
arch
fitting
your
model
and
then
after
you
do,
the
training
set.
A
You
want
to
evaluate
the
model
against
the
test
set
and
that's
the
importance
behind
splitting
them
out
is
that
you
can
actually
take
different
data
that
was
not
trained
and
evaluate
how
predictions
look.
We're
gonna
look
at
a
quick
model
of
that
right
now,
so
I
have
for
you
a
ipython
notebook
good
showing
up,
and
this
is
actually
taking
a
very
small
data
set
because,
like
you're
gonna
run
Karis
really
quick.
So
this
is
just
showing
some
server
latency
that
we
grabbed
from
an
envoy
metric
from
Prometheus.
So
you
see
here's
your
unix
timestamp!
A
So
we
see
here
this
is
doing
the
training.
The
loss
here
is
saying
that
okay,
it
got
a
little
bit
better
over
time,
expand
this
out.
So
now
our
model
is
complete.
That
was
relatively
quick
because
the
data
set
was
small
and
otherwise
we'd
be
staring
at
this
for
a
little
bit,
and
here
we're
gonna,
take
a
look
at
our
test,
set
that
we
have
and
we're
gonna
run
that
so
we
see
pretty
close,
it
seems
like
there
was
some.
A
So
we're
gonna
rerun
this
really
quick,
and
so
we
might
need
to
do
a
little
bit
more
fitting.
Typically,
what
you're
looking
for
is
linear
kind
of
like
x,
equals
y
on
the
top
right
here.
Expand
this
out,
but
this
shows
the
predicted
distribution
on
the
left
and
the
actual
on
the
right,
and
these
are
the
correlations
between
the
two,
so
you're
shooting
for
y
equals
x,
but
see
here
we
have
some.
We
have
to
probably
do
some
different
tuning
and
so
forth.
A
Oh
is
gonna
there
we
go
okay,
so
now
we
have
our
model,
so
we're
gonna
now
pass
our
model
in
to
start
generating
some
predictions
now.
This
is
where
it
gets
a
little
bit
interesting,
because
you're
gonna
choose
a
smaller
data,
set
that
you're
gonna
feed
in
in
real
time,
and
but
you
still
want
to
have
some
context
of
what's
happened,
for
example
in
the
last
hour,
because
if
your
models
saying
like
it's
dead,
there's
nothing
going
on,
there's
no
request
activity,
but
the
real-time
predictions
are
saying.
No.
A
A
So
we're
gonna
basically
use
about
an
hour's
worth
of
data
to
predict
the
next
10
minutes,
for
example,
but
you
can
kind
of
adjust
this
as
you
go
and
then
we're
gonna
split.
The
data
set
again
at
the
training
and
test
data
sets
and
the
training
set,
which
could
be
say,
40,
minutes
or
so
is
used
to
forecast
the
data
in
step
and
then
we're
gonna
use
the
test
set
to
basically
do
the
next
prediction.
A
A
If
you
wanna,
if
your
step
is
a
minute,
if
each
of
one
of
these
is
a
minute
and
that's
a
new
prediction
in
actual
then
what
you're
going
to
want
to
do
is
keep
in
mind
the
hot
you're
gonna
have
a
higher
rate
of
error.
The
further
you
go
out
so
if
I
use
this
model
and
it's
a
minute,
the
predictions
that
are
15
minutes
out
are
going
to
be
more
accurate
than
the
ones
that
are
60
minutes
out.
A
So
that's
how
that
in
in
a
nutshell,
we
generate
the
predictions.
So
now
we
have
predictions
to
play
with,
and
this
is
some
we
have
some
actionable
data.
So
what
we
have
here,
we
have
no
traffic,
so
our
predictions
are
saying
from
T
0
to
T
15
that
there's
nothing.
So
what
does
this
mean?
For
me,
as
an
SRE
means,
I
may
have
the
opportunity
to
scale
down
replicas
and
scale
down
some
infrastructure.
A
A
This
is
just
kind
of
a
very
basic
example.
So
if
I
had
stateless
resources
deployments
pods,
you
know
there's
some
sample
go
code
here.
The
basically
says:
I'm
gonna
set
thresholds
that
basically
say
the
you
know
if
we're
above
a
certain
RPS
or
a
below
a
certain
RPS
and
take
action.
Otherwise,
let's
sleep
and
wait
for
the
next
streaming
predictions
to
take
take
hold
also,
but
I'm
gonna
want
put
gates
in
here,
and
this
is
why
I
actually
mentioned
it.
A
It's
like
semi-supervised
I,
don't
know
if
that's
the
exact
term,
but
there
is
human
interaction
here
we
are
preventing
this
from
going
way
too
high
and
way
too
low
again
we're
trying
to
not
have
a
crazy
cloud
bill,
but
we
also
want
to
have
our
service
up.
So
these
are
things
to
just
keep
in
mind
as
you're
going
through
it
and
I
can
make
these
codes
than
if
it's
available
on
a
guest
or
something-
and
that's
that's
where
we
are
today
from
a
data
science
side
we're
currently
exploring.
A
Can
we
do
better
we're
looking
at
kind
of
tuning
those
parameters,
but
also
looking
at
something
called
multivariate
LX
l
STM?
This
correlation
between
some
of
your
data-
you
know
you
may
look
at
CPU
percentage
and
RPS
and
you
may
want
to
put
both
in
a
model
and
generate
a
different
model
off
that
that
may
have
more
context
to
the
rest
of
the
cluster
or
maybe
you
have
two
different
services
that
have
different
end
points
of
latency
and
maybe
those
models
shape
something
else
so
we're
looking
at
multivariate
lsdm
as
well.
A
Another
thing
is:
reinforcement,
learning
or
cue
learning
if
you're
not
familiar.
This
is
what
alpha,
0
or
alphago
is
based
off
of
the
we're
looking
at
this
as
trying
to
reinforce
good
behavior.
So
if
we,
in
a
very
simplistic
term,
if
we
say
keep
costs
low
but
make
sure
the
client
experience
is
optimal
and
we
define
parameters
around
how
that
works,
then
we
can
use
cue
learning
and
use
that
to
find
a
balance
between
maybe
our
threshold
numbers,
as
well
as
our
min/max
numbers.
A
We
were
talking
about
in
the
predictions
and
we're
looking
at
other
things
that
we
can
forecast.
We
started
this
out
looking
at
how
we
can
increase
and
decrease
our
infrastructure
for
our
data
science
workloads.
Now
we're
looking
at
it
for
replicas
and
we're
looking
at
it
for
different
types
of
pods,
even
looking
at
it.
For
you
know
some
of
the
elastic
resize
stuff
that
came
out
for
the
databases,
that's
something
that
we're
seeing.
If
you
know,
if
down
times
low,
maybe
we
can
do
some
scaling
there
and
also
cluster
management.
A
Maybe
there's
situations
where
we
can
predict
and
move
our
clusters
around.
You
know
work
in
a
hybrid
environment
and
so
forth,
but
I
think
the
goal
behind
the
data
science.
Part
of
this
is
how
low
touch
can
we
get.
You
know
how
what's
the
minimal
amount
of
time
and
sre
needs
to
manage
a
kubernetes
cluster
and
how
much
machine
learning
and
potentially
AI
do
we
need
to
get
there.
A
Another
thing
that
we're
looking
at
is:
how
do
we
make
this
intuitive
and
deployable?
So
we've
been
looking
at
HP,
a
horizontal
pod
autoscaler
and
the
model
around
that
and
seeing
if
there's
a
way
that
we
can
take
everything
we
discussed
and
actually
put
it
into
a
CR,
D
or
custom
resource
diagram,
and
then
it's
just
as
simple
as
doing
a
coop
cuddle
create
paths.
A
I
come
from
the
PA's
world,
so
that's
kind
of
weird
but
coop
co-create
paths,
that's
something
that
attaches
to
your
deployment
or
daemon
set,
and
it's
just
into
it.
If
you
pass
some
parameters
and
the
machine
learning
starts
to
kick
off,
so
this
is
something
that
we're
actively
working
on
and
that's
it
so.
A
Yeah,
so
the
question
was
is
to
take
into
account
seasonality
and
that's
something
we
looked
at
because
I
think
the
seasonality
would
like
Saturday
Sunday,
especially
with
what
we
were
seeing
with
the
Gryphon
Oh
charts.
That
is,
was
kind
of
a
primary
driver
for
us.
So
when
you're
picking
your
slice
of
data
that
your
initial
you're
putting
in
to
fit
the
model,
that's
important
because
then
you're
looking
at
you
know,
fall
holidays,
other
types
of
things
that
actually
gonna
get
incorporated
in.
A
Sure
so
it's
a
good
question,
so
the
question
was:
if
the
predictions
were
wrong,
how
do
we
adjust
for
something
like
that?
So
typically,
what
we've
been
doing
is
that
we
also
have
a
kind
of
a
reactive.
So
if
the
model
is
bad
or
maybe
there's
you
know
it
made
it
an
incorrect
prediction
and
we
need
additional
infrastructure.
A
A
It's
it's
so
the
question
is
around
scale
and
how
would
this
operate
at
scale?
That's
something
we're
looking
into
now.
I
think
we
are
I,
think
what
we've
initially
done
is
Unleashed
it
onto
our
development
environment.
So
there's
you
know
it's
smaller
in
scale,
but
as
we
start
to
look
at
it
a
little
bit
further
out,
I
actually
think
that's
where
you
see
if
you
decentralize
and
have
like
the
model
and
have
like
a
CR
D
for
each
different
service,
and
you
know
you
datasets
or
you
make
your
datasets
a
little
bit
smaller.
A
Right,
so
what
we,
what
we
are
doing
right
now,
the
question
was:
are
you
automating
the
Refresh
of
the
model?
Yes,
we
are
so
basically
at
midnight
we
take
in
the
next
three
weeks.
The
next
rolling
three
weeks
generate
the
model
of
that.
If
the
model
fits
the
evaluation
set,
then
we'll
we'll
deploy
it
and
that'll
be
like
the
model
that
will
be
used
for
the
next
day
and
then
from
there.
That
process
continues.
A
A
A
That's
actually
what
we're
doing
now,
so
we're
not
we're
not
actually
taking
the
actions,
we're
kind
of
observing
how
the
model
is
predicting
and
if
it's
like,
if
it's
a
ridiculous
number,
that's
we're
actually
sending
like
a
slack
message
to
our
team
to
be
like
hey
the
predictions
way
off.
Can
we
look
at
it?
How
can
we,
you
know,
is
something
wrong
with
the
model
and
we
kind
of
do
an
investigation
from
there.
A
Sure
so
the
some
of
the
things
we're
doing
is
like
rates
arrayed
over
five
minutes
so
and
that
actually
plays
into
the
resolution
a
bit.
So
you
may
have
to
do
smaller
batches.
When
you
do
the
rate
resolution
over
five
minutes,
we've
noticed
that,
as
we
started
smoothing
a
bit
the
resolution
for
the
time
series
got
crazy,
so
I'll
take
one
more.
A
Sure
it's
a
good
question,
so
the
why
we
use
daemon
sets
instead
of
jobs
we
initially
went
down.
Daemon
sets
were
gave
us
a
bunch
of
benefits
in
the
sense
of
we
knew
that
we
could
run
exactly
one
by
using
daemon
sets
on
that
particular
node.
Whereas
jobs,
we
didn't
really
get
into
heavy,
like
affinity
or
node
selection
with
jobs,
yet,
but
the
the
it's
essentially
triggered
and
the
daemon
set
is
just
listening
for
a
job
to
appear
in
our
messaging
queue,
so
we're
actually
just
waiting
for
it
to
pick
it
up.