►
From YouTube: Observability is the Key Tenet of Running a Multi-Tenant K8s Environment - Thom McCann
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Observability is the Key Tenet of Running a Multi-Tenant K8s Environment - Thom McCann & Amreth Chandrasehar, T-Mobile
How do you achieve 200 Million requests per day and no down time for 2 years? T-Mobile has been driving containerized workloads for many internal application teams. Running an internal multi-tenant environment can be challenging but has significant benefits In this session we'll dive deep into observability components such as a large scale telemetry system built on Prometheus serving 4000+ requests per second and millions of metrics over 6 clusters. This open source system is built across 3 AZs in each region (US West and US East) and is federated across multiple Prometheus clusters enabling distributed queries and limitless scale. We'll dive deep into how our operational teams can view integrated monitoring for infrastructure, hosts, VMs, containers and application level and integrate alerts with slack, pager duty and other real-time systems.
To Learn More: https://sched.co/GrXF
A
A
This
is
called
a
observability
at
scale
and
the
play
on
words:
observability
is
a
key
tenant
of
multi-tenancy
for
us
at
t-mobile,
and
we're
going
to
talk
to
you
about
a
specific
solution
that
we
have
using
containers
at
scale.
To
give
you
some
idea
about.
T-Mobile
here
comes
to
commercial
we've
grown
quite
a
bit
over
the
past
four
years.
T-Mobile
five
years
ago
was
the
fourth
largest
telecom
provider
wireless
provider
in
the
u.s.
with
about
35
million
customers.
A
It's
now
the
third
largest
with
77
billion
customers
over
the
last
five
years,
and
the
reasons
for
that
is
really
we'll
focus
on
customers
providing
things
like
Netflix
on
us.
We
have
actually
really
humans
doing
customer
service
taxes
included
on
the
bill,
unlimited
voice,
dated
text
and
would
give
you
something
every
Tuesday
and
case.
You
didn't
know
that
people
really
like
free
pizza,
so
then
give
you
so
many
frame
pizza
usually
puts
a
pizza
shop
out
of
business
for
a
day
or
two
we
didn't
charge.
Do
a.
B
Handoff
here
so
tom
said,
I
grew
from
25
million
customers
to
avoid
like
7
billion
customers
today,
so
most
of
the
pretty
large
systems
that
we
run
are
running
on
cloud,
and
for
most
of
that
we
do
the
observability
based
on
this
particular
system.
What
we
will
so
t-mobile
operates
in
two
brands.
What
as
t-mobile
handled
recently
metal
PC
has
got
to
be
branded
to
Metro
by
t-mobile,
and
there
were
50,000
employees
which
I
came
to
know
just
few
days
back.
A
In
our
public
cloud
team,
we've
started
with
docker
based
applications
about
three
years
ago
and
have
built
up
slowly
to
a
large
share
of
multi-tenant
environment
with
multiple
cluster
technologies.
So
it's
been
about
six
years.
We
have
about
70
plus
applications
and
applications
is
mostly
defined
across
AWS
and
Azure,
primarily
AWS,
and
we
have
are
all
of
our
region's.
A
Our
commerce
there's
just
a
lot
of
things
that
we
do
in
terms
of
serving
customers
in
different
markets
that
add
up
to
large
number
of
tea
and
serving
those
large
number
of
teams
for
77
million
customers
that
ends
up
being
challenging.
So
we
thought
we
pass
on
some
lessons
that
we've
learned
and
going
about
that.
A
So,
as
I
said,
we
started
three
years
ago
adopting
containers
and
we
built
a
shared
cluster
and
there's
some
things
on
here.
That
are
actually
true
and
that
can
sound
unbelievable.
But
we
really
think
there's
a
big
win
and
take
an
approach
if
you're
an
organization
like
ours,
it
has
multiple
what
we
call
customers.
A
We
have
not
had
a
single
minute
of
downtime
in
the
production
clusters,
we
have
had
0p1
customer
incidents
as
a
result
of
things
running
on
these
clusters
and
the
biggest
reason
for
that
is
observed,
observability
and
walk
through
what
we've
done
to
make
sure
that
that's
a
default
choice
for
every
application
team.
The
second
biggest
reason
for
that
is
luck.
A
And
you
all
know
that
to
be
true,
we
have
done
everything
three
weeks
ago
and
it
CD
cluster
went
down,
got
corrupted
that
happened
in
non
prod.
We
had
we've
been
this
close
a
number
of
times
of
having
this
streak
broke
and
even
in
the
last
week
where
we
had
a
misconfigure
dl
being
a
misconfigured
console
service,
and
but
we
caught
it
in
testing.
So
we've
been
very
close
a
number
of
times
ruining
this
record.
A
But
by
luck,
and
by
doing
some
of
these
principles
that
we'll
talk
about
we've
avoided
actually
having
these
systems
go
down,
we
have
built
a
set
of
tools
that
we
call
internally
conductor.
You
can
see
the
logo
up
there.
We
could
spend
a
whole
session
talking
about
that
logo
because
it
has
abduct
in
it,
but
we're
not
gonna
do
that
right
now.
Actually,.
B
You
can
see
if
whatever
semblance
of
the
kubernetes
logo
there's
a
70
70
shell,
sided
figure
and
I
can
see
our
conductor
as
well
as
78
figure.
So
if
we
just
took
that
as
an
example-
and
you
can
see
a
que
with
the
dark
or
that's
gay
for
Cuban
riders,
so
that's
how
we
decided
to
play
around
and
then
we've
we
actually
started.
Creating
an
internal.
Do
there's
a
whole
lot
of
thing.
B
We
started
doing
with
conductor
so
like
two
three
years
ago
and
before
we
started
creating
these
platforms,
they
used
to
be
like
two
Altium
swim.
You
to
go
to
just
you
know,
go
live
with
a
single
container
in
production,
say
your
DNA.
Is
you
have
to
create
a
login
system?
You
have
to
get
certificates
and
all
those
things
you'll,
basically
automated
a
lot
of
these
things.
So
all
these
are
the
integrated
services.
B
What
we
automated,
including
security
scanning,
self-discovery,
secret
management's
and
with
Cuban
letters,
and
with
today,
like
we
have
like
about
like
14
cumulus
clusters,
which
is
like
pretty
not
scale
which
can
handle
like
600,000
requests
per
second,
so
we
did
both
tests.
We
did
a
lot
of
things
which
is
a
base
of
what
conductor
is.
So
that's
what
you
see
like
the
different
Lobos,
like
whatever
we
use
series
to
Banaras,
obviously
having
four
deployments
and
Hettie
Oh
has
been
a
partner
for
us
too.
You
know,
scan
and
also
check
the
reliability
of
our
cameras.
B
Custer's,
we
don't
talker.
Are
we
just
testing
with
Content
ID
as
well?
Use
we've
Co
DNS,
HCD,
nginx,
International
English?
Basically,
so
that's
where
you
know
we
post
us
at
like
600,000
requests
per
second,
basically,
because
it's
large-scale
themes
because
he's
our
platform
team,
generally
teams,
don't
come
to
us
and
tell
us
that
hey,
we
are
going
live
unscaled
of
the
systems,
they
just
go
live
and
they
go
live
from
like
two
characters
per
second
to
like
a
14-carat
cross
once
again
within
seconds.
B
So
nobody
tells
us
that
okay,
we
are
going
live
and
I
mean
there,
so
some
metrics
on
you
know
how
it
happened.
So
we
prepared
is
one
of
the
things
and
observability
plays
a
key
role
in
that
because
we
know
how
much
the
systems
can
handle
and
when
the
system
can
break.
So
you
know
that's
where
all
these
different
tools
and
services,
what
we're
going
to
talk
about
is
is
very
important.
We
use
ansible
for
automating
the
provisioning
of
kunis
clusters.
We
used
to
see
emotion
of
keyboardist
cluster,
which
means
whatever
is
open-source.
B
We
take
that
we
automated
using
ansible
puppet
as
well
and
terraform.
Therefore
these
are
our
factory.
That's
our
centralized
docker
registry,
but
like
I,
think
there's
possibly
are
like
15,000
or
20,000.
Docker
images
used
to
scan
all
those
docker
images,
and
then
the
teams
are
here.
You
know
these
are
vulnerable
and
you
need
to
go
and
fix
those.
B
We
are
playing
with
harbor
as
well
for
signing
those
doctor
images
and
making
sure
only
scienter
images
are
being
run
so
because
they're
like
have
two
different
teams
like
on
us
on
engineering,
which,
on
datacenter
we
as
public,
you
are
going
to
make
sure
every
cluster
is
going
through
the
same
process
of
signing
those
images.
Of
course,
we
run
on
AWS
on
a
pretty
large
scale,
yeah.
A
When
we
talk
about
some
of
the
data
were
collecting,
but
first,
as
we
talked
about,
is
building
the
culture
when
you
start
you're,
basically
starting
with
data
collection,
you
know
even
last
week,
we've
run
into
some
an
application
where
they
didn't
have
logging
the
beta
it's
just
not
being
collected.
So
it's
a
larger
organization,
who's
lost
a
legacy.
A
You
know
applications
out
there,
so
we're
trying
to
continue
to
make
sure
we
improve
that,
and
what
we're
gonna
do
today
is
show
a
demo
after
this,
where
the
de
you'll
see
as
you
build
an
application,
you
just
basically
can't
avoid
logging.
Unless
you
really
screw
things
up,
you
can't
avoid
having
a
graph
on
a
dashboard.
It
just
is
built-in,
so
we
want
to
encourage
application
teams.
We
have
a
model
where
we
have
proactive
engagement,
I'd,
say
another
factor
in
us
being
able
to
keep
our
production
clusters
up.
A
Is
we
treat
our
non
production
like
production
and
we
proactively
engage
development
teams.
When
we
see
an
error,
we,
you
know,
we
have
all
of
our
alerts
flowing
into
slack
channels
and
many
other
places
to
be
where
the
developers
are
and
our
team
proactively
engages
them.
When
we
see
those
there
in
order
to
get
to
what
we
think
is
the
promised
land
of
observability,
which
is
you
got
to
walk
through
each
step
of
this?
You
got
a
date,
your
data
collection.
A
You
got
to
turn
that
data
into
information
through
telemetry
tools,
dashboards
reports
and
service
health
monitoring.
Then
you
have
to
create
the
culture
that
uses
those
things
doesn't
matter
if
you're
creating
dashboards,
so
I'm
just
got
to
use
them.
You
got
to
create
alerts
on
those
things.
Those
alerts
need
to
have
more
noise
than
signal
and
you
have
to
integrate
telemetry
and
all
the
tools
you
build.
We
do
build
internal
tools
that
help
people
on
to
this.
So
we
want
to
integrate
telemetry
into
those
tools
and
integrate
all
the
data
into
those
tools.
A
A
big
factor
for
us
being
proactive
is
chat,
ops
and
slack
and
making
sure
we
have
dedicated
teams
and
dedicated
channels
to
support
teams
and
applications
in
in
slack.
So
then,
we've
walked
three
steps
of
the
path,
but
then
there's
another
place.
Another
direction
we
can
go
is
now
we
have
all
this
data.
We
have
data
coming
from
Prometheus.
We
also
have
another
piece
of
data
on
here
that
I've
added
you
probably
haven't
seen
in
observability
talk,
which
is
cost.
A
We,
since
we're
on
the
pub
cloud,
you
have
to
look
at
cost
pretty
rigorously,
and
so
we
integrate
the
cost
data
into
this.
If
we
do
trend
analysis,
we
do
some
optimization
and
we're
starting
to
get
to
the
point
where
we're
adding
machine
learning
to
that.
So,
ultimately,
where
we
want
to
be
is
today
we're
in
a
situation
where
every
team
who
on
board
something
is
making
a
request
for
more
memory
and
CPU.
A
B
We
have
seen
some
applications
actually
for
one
request
per
second:
they
have
like
four
CPUs
and
16gb
of
memory
seriously,
so
we
do
talk
to
these
application
teams
and
tell
them
that
you
know
you
need
to
do
in
an
actual
performance
review
and
do
a
performance
optimization
of
these
applications.
It's
pretty
easy
to
scale
containers
right
I
mean
you
can
even
have
like
a
10
CPU
on
a
50,
GB
memory
and
then
scale
like
100
pods,
and
you
can
run
it,
but
do
we
actually
need
it?
B
So,
since
we
do
a
loss
of
cost
analysis
and
everything
we
do
get
back
to
the
team
and
tell
them
that
you
know
whatever
the
costing
is
like
$3,000
an
hour.
That's
a
lot
of
money,
so
go
ahead,
go
fix
your
particular
applications
and
since
we
are
a
platform
team,
we
know
what
all
services
are
not
runs
on
a
platform
and
we
can
actually
go
and
tell
them.
That
is
a
very
inefficient
way
of
running
it.
B
So
in
the
process
of
that
we
actually
have
teams
who
have
reduced
their
consumption
by
1/10,
which
is
actually
quite
significant
because
some,
like
40
containers
to
like
four
containers
and
it
basically
on
a
bigger
picture.
You
can't
actually
see
millions
and
millions
of
dollars
to
order
right.
So
it's
it's
actually
a
quite
a
lot
of
money.
B
This
is
a
two
box,
as
you
can
see,
we
don't
make
28
Prometheus
clusters.
So,
as
I
said,
we
have
like
14
cupidus
clusters.
Each
of
the
stupidest
clusters
has
a
Prometheus
cluster
and
each
Prometheus
cluster
is
highly
available,
which
mean
to
say
that
at
least
to
run
on
each
of
those
clusters,
and
we
have
integrated
that
with
Thanos,
which
is
an
open
source
solution
for
it
to
have
an
infinite
retention
period.
B
So,
even
if
you
go
two
years
behind,
you
can
actually
see
the
data
for
each
of
the
system,
including
your
own
production
and
production,
instead
of
all
the
process.
A
lot
of
money
in
this
case,
because
we
are
storing
this
in
s3,
which
is
like
very
cost
efficient,
probably
like
a
300
terabyte,
might
cost
you
like
few
thousand
dollars,
maybe
so
we're
just
not
that
much
amount
of
money,
given
the
fact
that
you
have
to
have
like
if
you
want
to
go
on
a
phone
lines
or
whatever
it
is
every
year.
B
You
want
a
good
look
back
over
a
year
and
see
that.
How
much
is
your
resource
consumption
for
it
and
how
much
time
you
will
be
prepared
for
something,
because
we
are
a
telecom
provider,
we
need
to
have
all
those
data
or
metrics,
and
it's
it's
quite
useful.
We
store
like
about
1.5
million
metrics
in
total
and
more
than
customer
interest
that
comes
at
times
or
telemetry.
Requests
are
more
like
we
get
like
around.
4,000
requests
a
second.
There
are
metrics
what
we
ingest
into
our
Prometheus
clusters.
B
We
do
get
a
significant
amount
of
data,
what
get
dumped
into
Prometheus
and
we
do
a
federation
on
top
of
that
and
collect
valuable,
metrics
and
store
a
different
Prometheus
cluster,
and
that
gets
collected
to
a
data
lake
where
we
do
all
those
machine,
learning
and
data
processing
for
cost
analysis,
build
dashboards
that
you
know,
botox
application
teams
are
basically
this
is
how
much
you
have
grown
over
a
period
of
time.
This
is
your
consumption
in
total.
This
is
how
much
you
can
save.
B
This
is
where
you
have
seen
issues
on
your
application
because
of
CPU
and
memory
utilization.
We
have
seen
applications
with
like
one
percent
utilization
and
we
go
on
I
mean
I,
think
pretty
much
everybody
here.
I
would
have
worked
with
this
application
of
built
applications
like
this
and
if
you
actually
do
back
into
those
telemetry
systems,
accuracy
like
this
is
not
the
most
efficient
way
of
running
your
services.
B
In
the
phone
our
perspective,
we
run
a
hundred
fifty
odds
in
total,
which
mean
to
say,
like
there
are
at
least
50
60
applications
with
non
production
and
production
put
together,
it'll
be
like
hundred
plus
they're,
like
thousands
and
thousands
of
dashboards,
that's
built
by
our
team,
that's
built
by
the
application
teams
and
they're
like
more
than
800
and
20
alerts.
Just
creating
alerts
is
not
most
useful.
You
have
to
go
and
revisit
the
thresholds
for
those
alerts
as
well
like
to
nose
down.
B
Ten
requests
per
second,
but
now
we
have
grown
to
like
2,000
acres
per
second
and
it
will
keep
alerting
once
you
keep
on
loading,
it's
going
to
be
a
spam
on
your
mailbox
or
on
your
slack
channels
and
nobody's
gonna
go
and
look
at
verse.
So
it's
very
important
to
go
and
revisit
those
alerts
that
actually
make
sense.
B
That
includes
Network
thresholds
as
well
the
bandwidth
for
each
of
those
instances
a
Tobias
keeps
optimizing
their
instances,
so
we
go
and
move
to
those
instance
types.
That
means
you
say
you
have
to
go
back
and
fix
those
thresholds.
Whatever
you
have,
you
have
done,
and
every
single
of
the
clusters
are
highly
available
so
graph,
one
itself.
B
B
Region
so
in
even
or
single,
we
tell
our
teams
like
our
customers
or
our
application
teams,
that
of
its
99.95%
available.
We
do
not
want
a
region
if
the
entire
region
goes
down,
grifone
are
still
available.
It
can
still
alert
us
on
top
of
the
idea.
Build
the
social
dashboard
even
of
the
entire
cluster
goes
down.
That
system
would
actually
a
lot
us,
even
if
everything
goes
down
that
is
sitting
just
outside
the
cupidus
cluster
outside
the
graph
on
our.
It
is
entirely
independent
of
quality
systems,
so
that
is
in-house.
B
Build
system
which
we
plan
to
open
source
actually
want
open
source
today,
but
will
probably
open
source,
probably
few
weeks
down
the
lane.
So
we
saw
need
for
a
service
l
dashboard
that
can
support
both
measures
and
cuban
adders,
so
this
can
actually
go
and
monitor
every
single
part
that
is
running
every
single
service
that
runs
on
kubernetes
at
every
single
service,
a
transfer
and
even
maiasaurs.
You
can
have
an
how
a
generic
alert
for
tcp
or
even
HTTP
HTTP
endpoint,
and
you
can
subscribe
for
email,
slack
notifications.
B
I
can
pretty
much
do
everything
on
that.
Even
prometheus
is
highly
available.
We
have
built
Prometheus
on
a
multi
region,
so
both
the
regions
go
and
collect
information
from
all
these
pods.
That
is
learned,
even
if
both
Prometheus
clusters
on
the
west
region
goes
down.
We
have
on
the
East
region
that
is
already
highly
available.
We
are
working
with
uber
m3,
which
actually
does
the
data
application
between
Prometheus
clusters.
We
are
doing
a
POC
with
that
as
well,
because
firm,
it
is
a
standard
on
our
time
series
data
base.
B
It
doesn't
do
it
application
between
databases,
so
we
do
work
on
that.
So
you
know
are
the
some
of
the
tools
like
Prometheus
define
a
fluent
II
collect.
D,
actually
is
what
we
use
for
log
applications
coming
data
collection,
sorry
and
we
use
elasticsearch
Gabbana's
plunk
Thanos.
They
just
talked
about
in
flux,
TV
and
C.
At
myself'
accumulators
we
collect
metrics
from
all
the
pods
or
node
exporter
of
Prometheus
to
collect
data.
We
did
collect
as
much
data
as
possible
and
make
meaningful
use
of
those
data.
Just
collecting
the
data
and
storing
is
not
useful.
B
A
Another
concept
is
Bill,
as
I
said
earlier,
build
telemetry
everywhere.
So
one
of
the
things
we
do,
as
you
see
in
the
right-hand
side,
as
we
integrated
into
slack,
we
drive
we
public
internally
public
channels
on
our
every
alert,
the
fires
in
our
clusters,
so
that
anybody
can
see
it
and
importantly,
our
team
responds.
A
We
built
the
COI
internally
such
that
we
can
combine
all
telemetry
aspects
into
our
CLI,
so
you
can
actually
get
not
just
container
level
data
cluster
level
data,
but
ec2
level
data
all
in
one
place
in
this,
and
you
can
see
this
particular
example.
We
filtered
by
ec2
instances
so
just
making
sure
the
concept
is
be
where
developers
are
live
where
they
are
and
provide
the
tools
in
every
place.
They
might
be.
B
A
So
we
try
to
get
to
a
demo,
so
we
rip
through
as
long
as
it
goes.
Well,
no
Democrats,
you
feel
for
us
rip
through
some
of
the
telemetry
gas
board,
just
because
we
like
to
show
cool
dashboards
and
you
know
first
level.
Obviously
us
we
have
to
do
good.
You
have
ourselves
of
making
sure
we
have
dashboards
available
to
our
own
team
and
we
have
probably
100
or
more
dashboards
just
specifically
to
our
platform
measuring
things
like
health
status.
A
Sorry,
this
is
a
one
of
the
important
concepts
is
when
you
put
a
tool
out
there
for
our
application
teams.
We
would
encourage
them
to
use
that
tool
and
the
self
service
and
self-sufficient,
and
this
is
one
area
where
a
team's
we're
not
generating
these
dashboards.
Our
teams
are
to
us
that's
a
great
sign
that
we've
reached
a
certain
point.
A
When
you
get
this
level
of
detail
about
one
specific
application
and
then
see
you
know
something
like
an
API
dashboard
for
the
same
application,
where
they're
measuring
each
component
of
the
application,
the
response
time
request
rate
cache,
hits
all
different
sorts
of
attributes
of
the
dashboard.
So
I
just
think.
A
Then
what
you
get
after,
that
is,
people
learn
that
they
can
do
different
things
in
grow
fauna,
and
this
is
an
example
of
a
call
flow
dashboard
and
an
API
where
you
can
see
that
they're
measuring
the
call
flow
as
it
goes
through
in
some
places,
it's
green,
which
means
it's
operating
within
its
SLA
and
some
place.
It's
orange,
which
is
not
quite
in
its
SLA,
and
so
they
can
look
at
every
level
in
the
request.
A
Here's
another
one,
which
is
interesting,
that
we
have
a
retail
application,
that's
deployed
both
to
the
east
and
west
region,
and
they
can
see
the
traffic
flowing
east
or
west,
so
they
can
determine
the
health
based
on
region
availability.
So,
just
a
wide
variety
of
we've
seen
developed
dashboards
we've
seen
developed
by
our
customers,
which
are
internal
teams
and
yet
another
one
here,
just
different
ways.
You
wouldn't
think
of
using
poor
fauna
and
this
telemetry
platform
to
see
order,
numbers
and
just
a
table
format.
A
This
one
I
love
because
I
don't
know
if
you
know,
but
coroner
can
have
white
backgrounds
and
I
for
one
exactly
so,
I
don't
know
where
I'm
going
on
this
I'm
a
person
when
I
look
at
white
text
on
black
screen,
I
get
dizzy
and
I'm
when
I'm
curious,
how
many
people
in
this
audience
or
like
it
or
is
it
just
me?
So
just
so,
we
have
this
really
comb,
so
yeah,
so
mojave
a
dark,
no,
not
a
fan,
not
a
fan
at
all,
so
I
mean
the
reality
is
a
for
dashboarding.
A
A
A
We
look
at
you
can
see
in
the
top
right
here
in
corner
I
drill
down
from
the
bottom
left,
which
allows
you
to
look
at
a
specific
application
and
you
can
see
the
breakdown
of
the
cost
on
AWS
line
items
and
then
further
break
it
down
by
usage
type
in
800.
So
if
you
really
know
all
the
line
items
in
the
bill,
you
can
get
really
granular
about
looking
at
what
you
need
to
get
and
we
also
have
the
same
capability
on
Azure
as
well.
A
As
you
see,
on
the
right
hand,
side
in
addition
to
provide
providing
folks
with
a
daily
trend
as
well
when
you're
doing
things
like
launches
and
bringing
up
new
applications,
so
cost
is
a
really
important
part
for
us
in
helping
the
drive
system
performance
and
as
Amarth
mentioned
earlier,
we
pull
all
this
data,
all
the
Prometheus
a2,
all
the
cost
data
into,
but
data
Lee
and
we
analyze
it.
We
look
at
utilization,
we
did
a
project
where
we
spark.
Let's
look
at
the
utilization
and
we
found
that
some
interesting
things.
A
Some
some
are
obvious,
but
some
obvious
things
only
jump
out
when
you
put
them
in
the
in
in
data
and
look
at
it.
So
we
saw
that
our
average
utilization
for
memory.
The
first
thing
we
found
out
is
that
whenever
you're
talking
about
utilization,
oftentimes
people
think
about
CPU
well
in
the
stateless
environment
and
for
a
ninety
seven
point,
nine
percent
of
our
applications,
CPU
is
just
isn't
a
factor,
it's
memory
and
that
might
seem
obvious
to
us.
A
But
when
you
look
at
the
data,
it's
even
more
obvious
and
so
51
percent
utilization,
not
so
bad
across
a
wide
variety
of
applications,
but
there's
a
number
that
should
stand
out
to
you.
That
looks
just
really
strange
here,
and
that
is
the
standard
deviation
of
that
number
right
and
so
we've
kind
of
looked
at
a
standard
deviation
of
30
seems
you
know,
kind
of
high
for
any
data
set.
So
we
look
at
that
and
started
measuring
the
standard
deviation
of
memory
utilization
across
our
entire
set
of
clusters,
which
will
turn
out.
A
B
One
more
point
here
is
the
number
of
short
run
containers,
so
starting
of
the
earth
we
in
February
actually
gave
those
terror
cells
like
being
ran
like
about
like
60,000
or
90,000
Shortell
of
containers
and
just
february
month
alone.
Later
in
august,
it
was
four
hundred
fifty
thousand
containers
short-lived
containers
and
in
number
December
time
frames
like
600,000
short-lived
containers.
So
what
a
short-lived
containers
those
containers
that
one
less
than
a
day
or
short,
one
continuous.
B
It
can
run
two
seconds
because
of
some
restarts
or
it
can
run
like
our
Jenkins
slaves
run
on
our
as
a
docker
container.
It
runs
depending
on
the
pipeline
length.
It
could
be
for
few
minutes,
so
even
the
demo,
hopefully
I'm
gonna
show,
is
going
to
run
like
that.
It's
going
to
run
as
a
docker
container.
So
those
are
what
the
short
run
containers
are.
A
Yeah
yeah
I
mean
that's
actually
an
interesting
point
and
one
that
kind
of
really
gets
at.
You
know
everybody's
talking
about
DevOps
and
agility
and
continues
to
literally
you
see
a
very
high
percentage
of
your
containers
that
run
short-term.
You
know
you're
starting
to
get
there
that
you
see
you
know
really
quick
iterations.
We
have
very
few
over
a
month
you
might
have
less
than
5%
of
the
containers
run
the
entire
month
and
so
just
another
kind
of
thing.
Another
little
data
point
you
can
look
at
to
do
that
so
show
the
demo
here.
A
B
Hopefully
the
demo
gods
are
there
with
us
and
everything
works,
so
this
particular
demo
is
about
usually
developers
day
how
it
goes
so
$1
per
comes,
and
you
know
it's
like
there's
an
issue
for
this
particular
application
and
he
wants
to
have
more
replicas
for
high
availability
of
his
application.
So
say
he
just
wants
three
replicas
for
ass
application
or
even
make
some
code
changes.
B
B
So
this
is
an
open-source
tool
that
we
use
for
testing
and
we
use
this
tool
to
run
600,000
requests
per
second
or
nginx
clusters.
So
this
can
stay
and
how
much
ever
you
want
so
say
an
application
developer
checks
in
the
score
dot.
Now
he
sees
the
metrics,
you
wanna
do
a
performance
testing
on
top
of
a
supplication
what
he
has
so
we're
going
to
use
this
tool
to
load
tests
on
this
particular
service,
and
you
can
see
all
those
graph
on
our
graphs.
Those
dynamically
ask
for
whatever
request
we're
getting
here
so.
A
B
Yeah,
so
this
that's,
what
is
standard
for
everybody,
so
this
is
a
self-service
dashboard,
so
any
so,
even
if
you
have
like
hundreds
and
hundreds
of
applications
and
they
have
like
thousands
of
pods
and
thousands
of
containers
that
runs
all
the
developer
has
to
go
and
do
is
like
go
to
the
hosts
and
choose
whatever
host.
You
have
go
to
the
upstream
and
choose
your
particular
namespace,
and
this
was
one
of
the
previous
example
and
not
only
that
you
can
actually
go
to
the
squads
and
see
how
much
is
a
CP
utilization.
B
How
much
is
your
memory
utilization?
What
is
the
consumption
of
network
bandwidth
for
for
your
different
services?
So
basically
a
developer
did
not
do
anything.
All
he
did
was
deploy
his
service
into
kubernetes
and
he's
seeing
all
these
metrics
it's
already
available,
so
I'm
just
checking
if
the
Jenkins
yeah
so
actually
now
the
deployment
is
done.
So
it's
asking
for
a
staging
deployment
which
I'm
gonna
wait,
so
you
can
actually
see
what
average
and
then
I
IATI
on
a
time
stamp.
So
now
it
is
available.
B
The
DNS
is
now
available
and
I'm
going
to
refresh
this
system
and
it
probably
just
got
deployed
you
can
see
those
metrics
started
coming
immediately.
So
previously
you
saw
everything,
that's
not
applicable,
because
that
service
was
not
running
and
the
service
was
not
available
for
into
the
parameter
system.
And
now
the
data,
the
C
advisor,
the
colibri
agents
that
was
running
on
the
host
is
collecting
all
the
system
and
it
is
now
sending
angriff
Anna's
getting
those
data,
and
you
can
see
all
this
so
now.
A
B
Start
you
know
trying
to
see
like
okay,
how
much
requests
can
my
application
having
so
you
can
see
immediately,
there's
like
it
has
started
running
hundred
users.
This
can
scale
to
like
a
100,000
200,000
users
depending
on
how
many
slaves
it
has.
We
have
tested
it
to
its
limits
and
we
made
some
lot
of
customizations.
On
top
of
it,
you
can
see
designing
a
312
across
for
a
second.
B
This
is
what
the
load
testing
tool
is
actually
showing,
and
if
I
am
refreshing
here
it
is
it
started
to
get
that
data,
so
you
can
actually
see
those
data
that
is
coming
here
and
you
can
see
all
those
server
requests
and
you
can
actually
see
those
upstream
requests
and
everything
that
is
starting
to
come
in.
You
can
actually
see
like
the
recursive
second
has
gone
to
like
four
and
54.
B
It
goes
per
second,
you
can
see
these
upstream
across
as
well,
so
this
particular
system
is
actually
collecting
all
those
information
and
it
is
showing
all
those
errors
latency
bandwidth,
and
not
only
that
it
can
actually
see
the
show
the
CPU
utilization
and
utilization
of
these.
You
can
actually
see
this.
A
little
spike
in
CPU,
so
what
it?
What
you
can
see
here
now
is
like
there's
an
inefficient
way
of
using
CPU
like
basically
I'm
requesting
for
a
lot
of
CPU,
but
the
CPU
is
in
utilization.
B
Here,
it's
like
very,
very
nice,
so
I
have
a
lot
of
room
for
optimizing.
My
CPU
my
memory,
without
even
having
any
sort
of
application
monitoring
solution
in
place
with
your
application,
so
not
only
with
pods.
You
can
actually
go
to
a
container
dashboard
and
you
can
see
the
number
of
pods
that
is
running.
You
can
actually
three
three
pods
are
running
and
it
can
even
show
you
on
which
particular
host
it
is
running.
It
can
show
you
information
about
each
pod
and
what
is
the
utilization
for
each
pod
like
over
the
period
of
time?
B
I
mean
I
I
put
a
period
of
time
once
it
goes
to
production.
We
have
seen
applications
with
like
4,000
requests,
a
second
or
even
like
10,000
rows
per
second,
and
you
can
actually
see
those
utilization
metrics
for
each
of
those
services.
There
are
times
where
application
teams
will
run
like
450
containers.
You
need
450
containers
for
like
100
cores
per
second,
so
so
this,
actually,
you
know,
helps
application
teams
to
determine
what
is
their
consumption
and
how
much
they
want
to
use,
and
so
you
can
actually
see
this
load.
B
Testing
tool
is
actually
showing
all
those
information
as
well,
but
it's
probably
available
to
a
load
tester,
but
now
this
is
also
available
for
an
application
developer
to
go
and
check
it
so
along
the
Disney
I
want
to
show
or
service
and
dashboard
as
well.
So
this
is
our
line
production
system,
how
much
data
it's
actually
getting
consumption
you
can
actually
see
our
production
system
is
getting
slammed
with
10,000
requests
per.
Second,
you
can
actually
see
how
much
it
does
go
on.
B
There's
about
3,000
services
with
6,000
plus
container
that
runs,
and
you
can
actually
see
how
much
is
on
maiasaurs
and
how
much
is
on
Cuban
artists
and
you
can
actually
search
with
different
platforms
together
across
or
different
clusters.
This
is
what
we
actually
plan
to
open
source
in
some
time,
so
he'll,
please
watch
out
for
that,
and.
A
One
more
thing
is
show
on:
this
is
the
integration
here.
Anybody
can
subscribe
to
a
particular
application
and
send
these
health
alerts,
for
example,
to
a
slack
dashboard,
and
that's
just
something
that's
built
in
to
the
application,
so
just
again,
building
the
tools,
so
the
maple
developers
to
have
this
all
by
default
and
available.
So.
B
System
that
is
running
outside
prometheus
outside
dha'fana.
This
is
just
a
simple
chat
tool.
All
it
does
is
talk
to
cube
natives.
It
does
a
he'll
check,
talk
to
me
and
see
the
head
check
and
then
it
stores
that
information-
and
you
know,
tells
it
okay.
What
is
your
availability?
How
much
is
the
request
per
second
for
each
of
those
applications,
so
yeah.
A
A
A
The
answer
is
not
a
hundred
percent.
Yes,
yet
I
think
that
has
to
be
a
coordination
between
the
application
team
and
the
platform
team
to
make
sure
the
application
knows
how
to
go
up
and
down
we've
seen
with
scaling.
The
bigger
problem
is
not
scaling
up,
it's
scaling
down
and
if
you
see
where
an
application
might
have
ten
containers
where
it
gets
to
be
problem
as
applications
that
might
have
one
service
running
in
one
container
and
have
to
having
to
relocate
that.
A
B
A
The
tool
we're
using
for
cost
is
that
something
that
could
open
source
it's
a
little.
Basically,
what
we
do
there
is
we
would
we
could
open
source
like
the
data
model
that
we
have
right.
So
that
would
be
one
thing
we
could
do
on
that.
We
basically
I
mean
that
was
power
bi
that
you
saw
so
that
data
is
loaded
up
into
Azure,
actually
and
lunched
in
there.
So
yeah
we
could
the
open
source
part
of
that
would
just
be
kind
of
like
how
do
we
do
this.
A
Snow,
it's
our
question.
The
question
is:
is
a
perfume
team
different
than
the
money?
No,
it's
the
same
team.
We
run
our.
We
build
our
kubernetes
clusters,
we
build
our
monitoring
stack,
we
build
tools
on
top
of
that
and
we
build
the
a
bunch
of
tools
so
to
onboard
and
make
all
these.
You
know.
Logging
is
automatic.
We
can
show
a
demo
of
logging
which
isn't
really
exciting,
but
logging
becomes
automatic,
just
like
telemetry
is
automatic
and
some
of
these
other
services
are
automatic.
A
How
big
is
how
big
is
their
team,
so
our
team
is
also
responsible
operations
across
the
board
of
all
cloud
applications
around
the
clock.
We
have
an
FTE
part
and
a
in
a
partner
part
of
our
team
located
in
multiple
places.
So
we
have
the
world
covered
from
Seattle
we're
located
here:
t-mobile's,
headquartered
here,
Dallas
Atlanta
and
in
India
and
soon
Estonia,
so
that
team
overall
is
probably
similar
around
60
people
that
covers
all
70
applications
that
are
on
a
cloud
and
all
70
applications
that
are
inbound
on
the
cloud.