►
From YouTube: How T-Mobile Built and Scaled Kubernetes On-Prem for the 2018 Retail Season - James Webb, T-Mobile
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
How T-Mobile Built and Scaled Kubernetes On-Prem for the 2018 Retail Season - James Webb, T-Mobile
Have you ever supported a major retail craze like the immediate spike in sales during a new smartphone launch? We have! We've deployed Kubernetes on-premise to support upcoming new smartphone launches and the busiest days of the year for T-Mobile including Black Friday & Cyber Monday. Through our “fail fast, fail forward” approach, we learned many valuable lessons on the way to being successful. Tons of research and testing took place, because failure in production was not an option. Millions of customers purchase smartphones on several key dates, producing huge spikes in transactions across dozens of internal business applications. Come hear our story of how we planned a Kubernetes project for the real world, including the biggest surprises, and keys to success. We’ll help you understand all the things you may not know about deploying on-premise at scale.
To learn more: https://sched.co/GrSD
A
A
It's
more
a
container
story,
it's
about
what
we've
been
doing
at
t-mobile
for
the
past
couple
years
and
successes
we've
had
and
and
what
led
up
to
our
decision
to
use,
kubernetes
and
kind
of
how
we
made
the
choices
to
deploy,
what
we
deployed,
how
we
leveraged
our
prior
successes
using
cloud
foundry
which
we'll
get
to
and
where
we're
going.
So
it's
it's
less
technical,
more
of
a
journey
story
so
who
we
are
so
we
are
t-mobile
pop
we're
engineering,
Brendan
and
I,
and
another
product
manager
started
the
team
in
mid-2016.
A
A
Short
timelines
is
why
we
was
one
of
the
main
reasons
we
chose
a
product
provided
by
a
vendor.
It's
we
want
to
be
able,
hit
the
ground
running
and
deploy
immediately
and
use
expertise
that
already
existed,
that
we
did
not
have,
though,
later
that
the
platform
was
wildly
successful.
We're
now
a
team
of
25
and
we
supports
I
as
Kaz
and
Paz
platforms
are
our
team,
most
of
which
is
sitting
on
the
audience.
A
Have
a
broad
background:
instructs
engineer
system
as
traders
developers,
platform
admins
product
managers
and
we've
also
become
customer
success
engineers.
So
it's
not
just
enough
to
run
a
platform
for
your
customer.
You
also
have
to
engage
the
customer
and
make
sure
they're
successful.
We
had
trouble
finding
people
to
staff
the
team
with
our
initial.
Our
initial
thoughts
were
we'll
go,
find
people
who
know
Cloud
Foundry.
Well,
those
people
don't
really
exist,
so
you
you
have
to
go
in
and
you
know
we
found
people
who
were
familiar
with
the
technical
technical
technology
stack.
A
B
In
general,
we
found,
if
you
know,
cloud
foundry
or
you
know,
kubernetes
you're,
already
employed
and
pretty
much
every
company
hiring
for
those
spots.
They're
always
hiring
those
spots
because
they
can't
find
enough
people
so
you'll
see
a
lot
of
people
move
it
around.
You
come
to
the
company
as
they
try
to
reinvent
their
same
success
over
and
over
which
they
can
usually
do
depending
on
where
they
go.
A
We
ask
our
engineers
not
to
put
a
cloth
around
their
LinkedIn,
because
I
usually
get
job
requests
right
away.
Great
way
to
get
stands
yep,
so
we
are
part
of
a
larger
organization
supporting
all
on
Pyke,
all
on
premise:
I
team
structures
for
t-mobile,
so
there's
two
I
t's
within
t-mobile,
there's,
there's
the
IT
that
supports
the
cellular
network
and
then
there's
the
IT
that
supports
what's
more
like
a
classic
IT,
so
we're
on
the
side,
not
the
celery
Network
site,
what
we
manage.
So
we
manage
right
now,
three
main
offerings:
Cloud
Foundry
or
its.
A
We
have
grown
from
a
single
foundation
hosting
about
3
to
400
apps,
to
12
foundations,
hosting
3,400
application
instances.
Roughly
40%
is
that
its
production
60%
non
production.
We
take
in
our
production
foundation,
roughly
300
300
production,
300
million
production
transactions
a
day,
it's
kind
of
hard
to
catch.
It's
it's
an
interesting
metric.
We
actually
the
foundation
takes
the
foundations,
take
a
lot
more
traffic
than
that,
but
a
lot
of
that
traffic
has
health
checks
and
so
self
referential
stuff.
This
registry
calls
things
like
that
yeah.
A
So
it's
hard
for
us
to
as
a
plateau
platform
team
is
hard
for
us
to
isolate,
what's
a
production
transaction
versus
what
what's
an
overhead
transaction,
but
we
reach
down
to
some
customers
and
some
of
our
larger
customers
and
found
that
300
300
million
is
a
pretty
reasonable
number.
It's
probably
a
little
higher
than
that.
But
that
was
a
good
guess.
B
Yeah
in
terms
of
total
traffic,
it's
probably
a
little
more
than
double
that
we
see
about
600
700
million
transactions
a
day
coming
through
the
entire
routing
layer
of
the
platform,
but,
like
Jim
said,
there's
a
lot
of
route
of
hairpin
traffic
leaves
the
foundation,
hits
the
service
registry
back
in
the
foundation
or
hit
something
else
inside
health
check
traffic.
Things
like
that.
We
don't
consider
to
be.
You
know
a
real
business
transaction.
B
A
Also
have
some
data
services
associated
with
the
platform
that
are
accessible
by
service
broker,
so
some
of
our
customers
use
subscribe
to
our
my
sequel,
RabbitMQ,
gem
fire,
witness
services,
and
we
we
don't
manage
those
services
as
experts
on
Redis
experts
on
rabbitmq,
but
they
are
available.
So
it's
a
it's.
You
can
subscribe
to
it,
but
then
it's
self-managed
pass
that
we
have.
A
We
have
an
on-prem
Cabernets
offering
now
we
have
several
flavors
of
it
right
now:
24
clusters,
mix
of
various
various
technologies,
PKS
some
open
source
and
some
older
stuff-
that
from
other
vendors,
that
we
have
live
traffic
going
through
our
clusters.
Now
we
don't
have
it
so.
When
I
wrote
this
abstract,
we
thought
we'd
have
probably
about
five
to
six
more
times
traffic
going
through
the
cluster,
as
we
do
today.
Nope.
If
that's
a
lesson
learned
reality.
A
Bureaucracy
sometimes
slows
things
down,
but
we
do
have
live
traffic
going
through
we're,
taking
about
a
million
production
transactions
a
day
through
our
largest
clusters
now,
and
we
have
teams
that
are
actually
actively
ramping
that
up
and
within
the
next
two
weeks
this
should
double
or
triple.
We
also
are
huge
fans
of
Bosh.
We
use,
we
use
it
for
a
ton
of
our
own
internal
needs,
and
we
also
offer
it
to
some
of
our
some
of
our
some
of
the
customers.
A
We
work
more
directly
with
to
offer
basically
data
services
managed
to
Bosh
so
that
they
don't
have
to
and
the
primary
use
case
there
was
rabbit.
We
didn't
want
them
to
use
the
become
noisy
neighbors
on
the
cloud.
Foundry
broker
rabbit
instances,
so
we
set
up
Bosch
supplied
instances
for
them
to
work
with.
It's
been
working
great,
but.
B
B
Here
yeah,
so
we
were
kind
of
you
know
the
reason
for
Paz.
In
the
first
place,
we
were
in
a
situation
where
we,
as
a
business,
could
not
deliver.
Business
needs
fast
enough.
It
took
us
on
average,
seven
months
and
seventy-two
steps
to
get
a
single
line
of
new
code
to
production.
That
is
a
long
time
you
think
about
adapting
to
your
competitors
to
market
change.
To
you
know
the
whims
of
a
somewhat
eccentric
CEO
and
it's
hard
for
us
to
keep
up
and
deliver
things
fast
enough.
B
This
got
us
into
a
bad
situation
more
than
once,
where
we
had
to
outsource
services
to
other
companies
which
then
more
or
less
held
us
hostage,
and
we
could
not
then
and
move
away
from
them
when
we
needed
to
so
we
had
this
problem
where
we
could
not
move
fast
enough.
We
had
really
no
options.
You
know
we
could
give
somebody
a
server
in
you
know
less
than
a
day
or
so
hopefully
10
or
15
minutes,
but
once
you
had
that
you
so
to
deal
with,
you
know:
load
balancers,
DNS
records,
firewall
rules.
B
All
these
things
before
you
can
even
put
your
application
on
top
of
it
and
do
all
your
extensive
manual
testing
and
integration
and
deployments
config
manage
of
patching.
It
was
just
very
laborious,
very
painful,
and
it
really
is
zero
value
at
work,
so
we
decided
we
needed
to
go
towards
a
path
that
would
make
this
stuff
easier
for
all
of
our
developers.
So
what
used
to
take
us
seven
months
in
72
steps
can
now
be
done.
The
same
day
people
find
in
fix
bugs
and
patch
them
same
day
going
through
dev.
B
You
know
about
10
or
15
test
environments
and
going
on
the
production.
This
is
all
daytime
work,
zero,
downtime,
I,
think
90,
95
percent
of
deployments
in
paths
happened
a
time
with
zero
impact
to
our
end
customers.
So
that's
been
just
a
huge
benefit
for
us.
We
can
now
adapt
much
more
quickly
to
change.
Also,
the
general
transition
from
model
is
to
micro.
Services
has
been,
you
know,
enormous
benefit.
B
It
used
to
be
extensive
regression
testing
on
the
thousand
functions
in
some
of
these
middleware
services,
and
now
they
have
a
few
functions
instead
of
a
small
micro
service.
They're
able
to
move
much
more
quickly,
much
more
reliably
test
in
a
much
smaller
scope
and
they're
able
to
do
these
changes
with
zero,
downtime
and,
more
importantly,
zero
incidents
caused
by
change
I.
Think
in
the
two
and
a
half
years
we
doing
this,
we
haven't
seen
a
single
set
one
or
P.
B
You
started
out
with
once
you
finally
get
a
to
production
again,
the
day
time
changes
we
can
now
do
canary
testing
and
about
in
a
much
more
small
and
reliable
way.
We
don't
have
to
worry
about.
You
know
we
cut
these
50
stores
over
to
this
one
instance
of
an
app
that's
running
it
could
now
be
you
know.
5%
of
all
traffic
now
goes
to
you
know,
version
2.
Instead
of
version
1
and
we're
able
to
test
these
things
much
more
reliably,
we
used
to
have
deployment
weekends.
B
We
still
do
for
some
apps
that
aren't
running
in
paths
where
it
would
be
sixty
seventy
eighty
hundred
developers
in
a
room
for
a
whole
weekend
drinking
Red,
Bull
eating
sandwiches,
because
every
app
being
pushed
is
all
tightly
coupled.
They
must
succeed
and
when
an
app
that,
when
I
have
deployment,
must
succeed,
you're
shooting
from
the
hip
you're
making
changes
last-minute
you're
touching
production
in
ways
that
you
probably
shouldn't
be,
and
you
never
make
those
changes
back
down
from
prod
to
all
your
sub
products.
B
So
next
time
around
you're
in
an
even
worse
state
than
you
were
before.
Beyond
that,
just
the
inefficiency
of
having
developers
and
operations
team
worried
about
the
state
of
these
states
very
stateful
servers
this.
You
know
the
pets
they've
had
around
for
three
or
four
years
that
all
have
names
and
very
important
meaning
to
them.
We
had
teams
that
had
maintenance
windows
from
10:00
p.m.
to
4:00
a.m.
five
or
six
nights
a
week,
and
they
worked
those
maintenance
windows,
10
p.m.
to
4
a.m.
5
or
6
nights
a
week.
B
They
did
patchy
and
dated
configuration
management.
They
did
application
deployments.
They
had
whole
silos
of
hardware
where
they
would
have.
You
know
an
identical
stack.
You
know
a
few
racks
over
and
they
would
say
this.
This
stack
right
here
is
no
longer
active
tonight.
We're
gonna
do
all
our
work.
They're
eventually
cut
back
over
to
it.
When
we
have
our
next
maintance
window
tomorrow
night,
with
the
benefits
we
get
from
passing
and
tater
ization,
these
developers
and
devops
teams
can
now
sleep.
They
can
spend
time
with
their
family.
B
They
can
focus
on
operating
their
apps
in
a
way
that
makes
them
more
efficient.
Instead
of
just
you
know,
the-the-the
endless
monotonous
of
zero
value
out
of
work
they
were
doing
previously
and
one
of
the
biggest
benefits
we've
seen
from
a
very
hard
cost
perspective
is
just
consolidation
of
workloads,
some
of
our
most
important
middleware
applications.
We
had
about
2,000
physical
servers
writing
these
loads
before
they.
You
know
very
painful
to
operate.
B
We
had
a
lot
of
redundancy
built
in
there,
we're
now
running
those
same
workloads
on
less
than
a
hundred
50
physical
servers,
along
with
everything
else
running
in
paths.
So
incredible
consolidation.
Here
we
have
much
higher
utilization
on
this
physical
hardware
and
we're
able
to
decommission
all
those
servers
that
required
maintenance
and
support
contracts
and
all
the
patching
that
went
along
with
that
slide
right
into
practice.
B
Similar
similar
themes
to
previous
talks,
so
if
pads
worked
so
great,
why
are
we
why
we
adhere
at
kubernetes
con
coupon
talking
about
kubernetes
and
why
we're
going
that
route?
So
Paz
is
great.
If
you
have
a
workload
that
fits
in
Paz,
if
you
don't
you're,
gonna
have
a
pretty
bad
time.
We
had
some
applications
that
were
vendor,
delivered
containers
it's
much
more
popular
now
than
it
was.
B
You
know,
even
three
or
four
years
ago,
where
they're
not
giving
you
an
RPM
or
Deb
file,
they're,
saying
here's
our
container
just
run
it
in
or
work
just
fine.
That
works
fine
for
some
containers,
but
other
ones.
We
have
in
particular
one
that's
an
order
management
system.
It
has
a
local
cache
that
has
to
warm
up
every
time
it
starts
out.
Then
it
has
to
offer
out
a
armi
port
that
allows
it
to
receive
updates
to
that
cash
anytime.
There's
updates
made
so
in
the
cat
in
the
past
world.
This
is
a
bit
non-standard.
B
You
have
you
know
what
should
be
persistent
storage
underneath
that
we're
not
offering,
and
then
you
have
this
kind
of
non-standard
TCP
routing
instead
of
standard
HTTP
routing.
So
we
had
candidates
like
that
that
we
said
this
will
never
work
properly
in
Cloud
Foundry
and
somehow
they
got
it
working.
They
had
huge
agility
gains,
just
like
we
saw
with
other
apps
moving
in
there,
so
they
wanted
to
keep
on
running
in
this
environment,
but
we
found
those
as
we
did
upgrades
to
our
paths.
We
didn't.
B
We
didn't
have
problems
with
the
native
apps
that
ran
with
the
you
know
the
Cloud
Foundry
build
packs.
We
had
a
problem
with
these
docker
apps
things
would
change
in
the
docker
run
time
they're
using
other
security
standards
that
were
tightened
down,
and
these
apps
running
in
docker
would
always
be
the
ones
that
caused
us
problems.
In
addition,
because
of
the
way
they
had
TCP
routing
for
some
of
these
apps,
it
wasn't
really
a
great
use
case
to
have.
B
You
know
this
almost
a
pet
running
inside
a
container
from
the
management
perspective,
so
they
had
to
scale
out
in
a
very
non-standard
way
that
led
them
to.
You
know
that
they
couldn't
they
couldn't
be
as
agile
as
they
wanted
to
be.
So
this
kind
of
bore
the
need
for
a
Kaz
inside
t-mobile.
Instead
of
just
two
standard
paths
where,
instead
of
having
the
the
strictly
opinionated
approach,
they
can
run
a
much
more
generic
container
in
kubernetes.
B
You
know
get
get
best-in-class
orchestration
and
be
able
to
really
adapt
to
the
needs
they
had
be
able
to
deliver
services
from
vendors
or
from
you
know,
docker
hub,
whatever
they
want
to
do,
they
can
run
these
things
much
more
easily
and,
most
importantly,
with
persistent
storage.
There
were
some
patterns
for
persistent
storage
and
paths.
We
don't
want
to
adapt
those
because
it's
based
around
NFS
and
it
wasn't
really
a
scalable
pattern,
long
term
for
us
that
we
could
properly
secure
and
then
you
know
last
one
was
just
the
platform
frustration.
B
The
ability
to
have
you
know
multiple,
apps
or
deployments
as
part
of
a
stateful
staff
that
are
able
to
have
inter
dependencies
if
something
crashes
that
bring
up
something
else.
You
know
we
didn't
really
have
that
ability,
because
we
had
much
more
independent
instances
in
paths
that
you
know
didn't
really
fit.
The
needs
of
some
of
these
people
that
had
much
more
complex
tax
where
they
had
inter
dependencies
I
mean.
A
We
actually
had
a
lot
of
teams
that
were
flush
that
did
not
want
to
move
or
did
or
didn't
think
that
their
apps
could
move
to
class
andriy,
but
also
we
when
they
called
the
obstruction
team
and
said,
do
you
guys
have
a
standard
car
from
we
did
it,
so
they
would
do
is
go
provision
a
VM
and
basically
run
their
own
docker
right.
So
we
have
this
put
the
shadow
docker.
You
know
infrastructure,
that's
pull
defer
Aidid
through
the
through
the
enterprise,
and
we
know
the
risks.
There's
a
lot
of
risks
associated
with
that.
B
Many
cases
we
saw
teams
would
take
their
existing
application
servers,
wipe
them
clean
and
just
installed
base
OS
and
docker,
and
use
that
purely
as
a
delivery
mechanism,
because
they
get
enormous
benefit
from
doing
that.
But
you
know
in
terms
of
long-term
scalability
stability,
if
they're
still
tied
to
that
one
IP
address
still
tied
to
the
same
firewall
problems
they
had
previously.
So
it's
better,
but
it's
not
really
revolutionarily
better.
So
they
wanted
to
have
something
better
for
actually
orchestrating
the
containers
instead
of
just
running
them.
A
Why,
on
Prem
we
get
this
question.
A
lot
is:
there's
cloud
providers
that
offer
fantastic
manage
kubernetes
services.
Why
would
you
go
off
of
the
effort
to
build
it
on
prem?
Well,
there.
There
is
a
lot
of
truth
to
the
data
center
to
date
of
gravity
and
I
call
it
data
center
gravity
because
there's
a
lot
of
other
there's
a
lot
of
other
issues,
not
just
data
right.
So
data
is
the
first
right.
We
have
a
massive
amount.
You
know
petabytes
petabytes,
petabytes
of
storage
of
data
local
to
our
data
center.
A
If
we
try
and
move
that
in
and
out
of
the
cloud
right,
that's
that's
a
huge
cost
right
and
not
just
a
huge
cost
in
terms
of
money,
but
also
a
huge
cost
in
terms
of
latency
effort.
So
we
want
to
keep
an
a
player
very
close
to
what
our
systems
record
are
in
the
data
center.
You
know,
what's
the
future
of
that
who
knows,
but
this
is
where
we
are
right
now:
network
latency
matters
right,
wait
for
a
lot
of
our
mobile
wraps.
They
are
very
latency
sensitive.
A
We
have
certain
SOA
as
we
try
and
meet
with
retail
stores
call
centers
and
the
web,
so
we
want
to
make
sure
that
we
keep
that
as
low
as
possible.
There's
also
partners
that
terminate
in
data
centers.
So
there's
there's
a
lot
of
there's
a
lot
of
other
network
concerns
that
that
really
factor
into
running
on-premise
well
security.
A
We,
the
all
our
security
controls,
are
written
or
most
of
our
security
crowell
could
overridden
for
on-prem.
So
when
an
app
team
wants
to
do
something,
they
know
how
to
do
it
on
prem
when
they
go
to
the
cloud.
It's
it's
a
it's
a
new
discovery
process
and
it's
completely
doable.
But
it's
a
lot
of
extra
overhead
for
those
teams
to
move
out
their
organizational
right
again
going
to
public
cloud.
Not
not.
Every
team
has
some
public
out
cloud
expertise.
We
have
a
excellent
public
cloud
team.
A
They
do
a
great
job,
onboarding
customers,
but
they're
still
a
certain.
You
know
overhead
that
teams
have
to
accept
if
they
want
to
go
out
to
the
public
cloud
in
a
lot
of
cases.
There's
also
a
lot
there's
also
a
lot
of
our
compliance
documents.
In
terms
of
Sox
know,
things
are
written
around
on
Prem
processes,
so
those
those
you
know-
that's
that's
not
ad,
that's
not
even
add-on
pram
or
a
cloud
problem.
A
That's
actually
a
cloud
native
problem
because
it
introduces
a
whole
new
class
of
systems
that
our
compliance
documentation
doesn't
really
have
terms
or
documents,
or
you
know,
wait
ways
to
deal
with
so
we're
going
through
enough
of
that
cost
is
another
strong
factor.
A
lot
of
a
lot
of
cloud
is
about
velocity.
We
have
data
centers.
If
we
can
execute
we,
there
are
huge
savings
just
to
put
the
playing
on
print
right.
It's
not
not
for
every
service,
but
for
just
like
pure
raw
hardware
capacity
and
memory
storage.
A
There's
it's
actually
a
big
cost
savings
to
stay
on
prim
in
a
lot
of
ways
and
then
cost
modeling
right
on
Prem
tends
to
be
capex
off.
Prem
tends
to
be
op
X
right,
so
something
not
everyone's,
just
it
to
how
those
two
things
play
out.
Nother
things.
Do
we
want
o
in
our
own
destiny
right
when
we
go
to
the
public
cloud
or
kind
of
a
hold
in
to
certain
to
certain
things
being
on?
Prem
gives
us
a
lot
more
control.
A
We
do
have
a
public
cloud
team
I
see
some
of
them
out
in
the
crowd.
We
love
them,
so
it's
they
offer,
they
all
have
their
own
Cade
service
and
a
lot
of
other
things.
So
it's
been
a
great
partnership
and
we
we've
been
converging
with
them
to
figure
out.
What's
our
common
offering
gonna
be
and
how
do
we?
How
do
we
help
each
other
direct
customers
to
the
hypothesis
yeah.
B
It's
very
important,
Marc
standpoint,
that
when
teams
move
between
on-premise
and
cloud
providers,
they
don't
have
to
learn
a
whole
new
set
of
workflows
or
API
calls
so
using
kubernetes
as
that
abstraction.
You
know,
coop
controller.
The
kubernetes
api
is
what
you
have
to
learn.
You
don't
have
to
learn.
You
know
AWS
versus
szura
versus
tcp.
You
can
focus
on
that
common
abstraction
and
move
move
much
more
quickly
between
on-premise
and
cloud
providers.
C
A
Wouldn't
late
2017
or
early
2018,
we
start
coming
up
with
what
are
our
requirements
for
a
platform
right,
so
platform
team.
You
know
a
common
checklist
right.
We
need
to
be
highly
available
at
every
level.
We
need
automated
deployment.
That
was
that
was
a
huge
lesson
learned
on
the
on
the
cloud.
Foundry
side
is
everything
we
installed
to
start
with.
We
we
we
installed
by
hand
right
and
now
someone
else
takes
care
of
it
and
it's
it's
not
easy
and
there
they're
automating
the
amount.
C
A
That
we
left
behind.
Thank
you
guys,
but
what,
on
the
cast
side,
we
wanted
to
start
with
automate
everything
we
possibly
can
not.
Just
control
playing
for
cluster
builds
no
downtown,
no
downtime
lifecycle
management.
So
one
of
the
huge
advantages
again
of
the
cloud
foundry
side
is
that
we
can
patch
and
keep
the
environment
up-to-date
without
our
customers.
A
Knowing
so
we
want
the
same,
we
want
the
same
ability
on
the
Cate
side,
because
that's
actually
huge
value
to
customers
as
well
lets
them
focus
on
app
development
and
and
supporting
that
application,
not
on
keeping
OSS
patched
or
keeping
versions
of
various
tools.
Up-To-Date
LDAP
integration,
you
know,
API
give
a
really
and
automated
ops,
so
that's
kind
of
our
high-level
list
of
what
we
had
requirements.
We
talked
with
some
teams
and
gathered
their
requirements
as
well.
A
Some
application
teams
they
wanted
to
have
a
native
Khachaturian's
container
orchestration
kind
of
they
wanted
to
have
a
they
want
to
have.
We
set
a
very
high
bar
for
the
support,
except
for
how
we
supported
them
on
the
on
the
path
side.
So
we
wanted
to
at
least
for
the
bulk
of
the
workloads,
give
them
a
very
similar
experience
right
out
of
the
box
load,
balancing
out
of
the
box
a
wildcard
certificate,
so
they
don't
have
to
register
their
own
OS,
patching
in
structure,
maintenance,
persistent
storage
and
all
the
all
the
different
flavors.
A
With
that
one
of
the
things
that
it's
a
little
more
painful
in
the
cloth
farmer
side
is
TCP
ingress,
so
we
wanted
to
give
a
service
type
load
balancer
and,
most
importantly,
it's
centralized
logging
and
metrics.
That's
it's!
That's
a
big
big
deal
and
just
the
fact
that
if
you
can
spin
up
your
application
and
not
have
to
write
your
logs
to
standard
out
and
they
go
in
the
right
place,
that's
that's
incredibly
powerful
and
again
lets
them
focus
on
what
they
want
to
focus
on,
which
is
app
development.
Not
how
do
I
log.
A
A
A
Point
of
view
we
again
married
what
we
did
for
Cloud
Foundry
is.
We
basically
went
with
a
shared
nothing
architecture
where
each
each
column
there
is
a
single
rack
or
Hardware,
with
network
top
rack
network,
compute
and
storage.
So
there's
no
common,
no
common
elements
between
the
three
racks,
even
the
the
networks
they
go
to.
The
three
racks
are
different:
it's
isolated
behind
a
spine
leaf
so
that
our
our
traffic
doesn't
doesn't
see
that
never
sees
the
core.
Only
only
customer
traffic
in
and
out
its
T's
any
other
network
elements
right.
A
It's
very
high
bandwidth,
low
latency,
east-west
region.
The
max
capacity
is
about
55
terabytes
of
memory,
22
enter
cores
and
to
two
petabytes
of
storage.
We
could
probably
go
bigger
than
that
on
a
couple
of
these
things.
It's
just
a
matter
of
cost
efficiency
in
terms
of
dim
size
and
in
the
and
compute.
So
this
was
kind
of
a
first
guess
at
what
we
would
need
we'll
see.
You
know
time
will
tell.
A
In
a
couple
of
months,
we
deploy
multiple
of
these
regions
to
a
single
data
center,
with
the
idea
that
that
there's
there's
production
regions,
there's
non
production
regions
and
there's
multiple
production
regions
within
a
single
data
center,
so
that
teams
can
do
a
near
near
deployment
strategy.
So,
even
though
we
have
multiple
datacenters
and
would
like
to
be
active,
active
across
data,
centers
back
to
the
data
gravity
right
most
apps
tend
to
run
in
one
location.
A
A
So,
on
that
hardware
right,
what
we
wanted
to
get
kind
of
kind
of
established
was
what
what
we
call-
and
we
stole
this
term
from
something
else-
is
that,
like
the
kubernetes
downtown
right,
we
want
to
be
able
to
just
provision
clusters
and
get
kubernetes
out
of
these
regions
with
with
very
low
overhead,
and
very
you
know,
just
press
a
button
get
a
cluster
not
quite
there
yet,
but
we're
getting
there.
The
product
we
chose.
We
ended
up
selling
on
was
pivotal
container
services.
It's
there's
it's
a
framework
that
essentially
is
a
service
broker.
B
Yeah
it
really
built
is
built
around
the
concept
of
you
know:
you'd
have
your
GK,
he
or
IKS
eks
on-premise,
where
you
had
an
API
endpoint
as
an
authenticated
user
and
you
get
a
kubernetes
cluster
spit
back
out.
So
we've
worked
with
them
a
bit
to
determine
what
actually
the
makeup
that
could
be
a
t's
cluster,
what
it
would
look
like,
and
you
know
what
it
means
to
get
a
cluster
and
who
we
expect
to
be
operating
as
well.
A
Once
a
cluster
is
deployed,
we
have
some
more
pipelines
to
kick
in
and
and
and
basically
to
mobilize
the
cluster.
We
install
monitoring
it
puts
it's,
a
storage
ingress,
the
logging.
You
know
it's
still
some
manual
steps
to
get
the
external,
the
bounces
configured,
we're
working
on
that
and
that
once
that,
once
that
we
consider
the
cluster,
you
know
production-ready
do
some
some
boy
and
other
other
sort
of
conformance
tests
against
it.
And
then
say
this
is
a
live
cluster
then
day
two
opt
kicks
in
right:
organ
user
management.
A
We
have
a
get
ups
framework
for
our
path
side,
we're
currently
working
on
putting
that
over
the
community
side
and
then
just
a
time.
Support
and
support
right
now
is
onboarding
customers
and
it's
back
to
the
customer
success
engineering
world
right.
It's
it's
be
an
ad
we're
being
advocates
for
the
platform,
we're
not
it's
not
the
old
days
where
we
just
say:
here's
your
here's,
your
here's,
your
thing
use
it.
We
are
actively
helping
users
develop
good
patterns,
it
helps
us
understand
their
workloads.
It
helps
them
understand
our
concerns
as
well.
A
B
It's
also
not
the
old
days
of
they
can
throw
things
over
the
fence
at
us
and
we
just
fix
it
for
them.
We
make
sure
they
understand
that
you
know
we'll
give
back
the
same
level
of
effort
that
you
guys
give
us.
So
if
you
say
this
is
broken
fix
it
we're
gonna
say
why
is
it
broken
and
the
queues
troubleshoot
your
own
problems?
If
you
come
to
us
and
say,
I've
tried
XYZ,
it's
still
not
working,
we'll
help.
B
You
figure
out
why
it's
not
working,
but
it's
it's
not
a
situation
where
we
can
support
all
of
our
customers
on
the
path
side.
We've
got
over
a
thousand
developers
that
operate
on
our
platform.
We
can't
handhold
each
one
of
those
people
when
they
teach
them
how
to
fish
themselves,
they're
able
to
operate
their
own
applications
in
their
own
platform
without
us
having
to
do
it
for
them.
A
So
this
was
a
healthy
debate
we
had
early
on
is
who
owns
clusters
right,
and
it's
it's
we're
looking
to
provide
these
for
the
initial
for
the
initial
around
a
curated
environment.
We're
not
we're
not
handing
over
clusters
with
with
you,
know,
cluster
minister
access
and
they
go
to
town
right.
We
are,
we
are
providing
a
resource.
Customers
come
to
us
on
board.
A
We
give
them
a
name
space
right
that
that's
that's
the
model
that
that
we
are
starting
with,
and
it's
the
it's
the
model
that
our
customers
are
most
comfortable
with,
at
least
the
majority.
Our
customers
are
most
comfortable
with,
and
it's
working
out
so
far
right
at
some
point.
We'll
extend
that
to
be.
You
know,
we
know
we
have
some
more
advanced
customers
or
some
more
demanding
customers
that
are
looking
to
run
their
own
run,
their
own
cluster
into
end.
A
We're
thing
what
that
looks
like,
but
for
now
we
manage
the
clusters
right
and
then
provide
access
to
the
teams,
and
they
they
you
know,
do
their
work
and
if
they
need
something
that
they
ask
us
and
we
do
our
best
to
help
them
with
any
problems.
They
have
multi-tenant
clusters
are
interesting
things,
so
one
of
the
great
things
about
paz
is
that
within
a
single
foundation,
we
just
pack
containers
in
like
nobody's
business.
B
A
Early
successes,
so
we
we
have
some
live
apps
and
this
is
pretty
cool.
It's
been
a
long
time
coming
with
the
first
app
actually
went,
live
in
August
and
it's
maps
to
mobile
comm.
You
can
go
to
that
on
your
phone,
it's
just
a
it's
a
coverage
map
where
you
put
in
a
location
and
it
shows
you
t-mobile
coverage
in
that
area.
This
is
also
a
cool,
a
very
cool
example
because
they
are
running
in
three
or
four
different
places
and
load
balancing
across
them.
A
So
they're
running
in
kubernetes
in
our
on-prem
kubernetes
they're
running
in
our
on-prem
paths
and
they're
running
in
the
in
the
public
cloud,
I,
don't
know
if
that's
on
kubernetes
or
in
the
meso
stuff,
it's
so
they
are.
They
are.
They
have
been
one
of
our
leading
customers
in
terms
of
doing
what
we'd,
like
all
our
customers,
to
do,
which
is
distribute
your
application
to
multiple
places.
So
if
we
have
a
failure
in
one
place,
it
keeps
running
so
it's
a
simpler
app
to
do
that
with.
A
But
it's
it's
also
a
great
pattern
that
we
refer
to
other
other
customers
too.
We
also
have
some
order
management.
We
have
our
mobile
point-of-sale
systems
that
also
connect
to
the
canaries
clusters.
Again,
they
are
they
now
that
the
primary
you
know
part
of
the
holiday
season's
over
in
terms
of
sales.
A
They
are
now
actively
moving
their
work
load
onto
our
clusters,
so
it's
very,
very
exciting
to
see
what's
gonna
happen,
there
we're
already
seeing
a
benefit
from
the
seamless
upgrades
right,
especially
with
PKS,
where,
where
you
know
pivot,
will
provide
to
a
new
new
version
for
us.
We
ingest
it,
run
it
through
some
staging
clusters
and
then
push
it
to
our
production
clusters.
We
singles
the
upgraded
from
110
and
111
and
another
example
is
the
other.
Success
is
the
same
day
that
the
111
5
patch
was
released
for
the
for
the
API
CVE.
A
We
were
patched
within
36
hours
and
we've
seen
this
back
on
the
path
side
as
well,
where
usually
our
systems
are
patched
far
ahead
of
anyone
else's
systems,
because
it
is
a
single.
It
is
a
single
action
to
to
initiate
a
patch,
and
then
we've
been
we've.
One
of
the
big
things
we
look
at
for
the
kunai's
platform
is
persistent
storage.
We
had
fantastic
success.
There
we've
been
very
happy
with
our
solution.
It's
in
use
by
us.
It's
any
spy,
our
customers.
A
We
think
this
is
going
to
be
one
of
the
critical
capabilities
offered
by
the
platform.
So
it's
we're
looking
to
really
build
on
that
some
challenges.
Tcp
ingress,
so
one
of
the
problems
is
running
on
Prem-
is
that
we
don't
have
a
service
type
load
balancer
out
of
the
box
with
PKS
you
could
install
NSX
T.
We
didn't
lots
of
reasons.
They're
not
gonna,
go
into
them
too
much
here,
but
we're
still
we're
still
looking
at
that
and
we're
valuing
evaluating
some
of
those
solutions
there.
Right
now
we
have
some
workarounds.
B
It's
kind
of
an
example
of
how
we've
changed.
We
started
out
saying:
no
ports,
not
an
option.
We're
not
gonna
offer
that
to
our
customers
and
then
first
customer
says
I
need
this.
We
say
well
yeah.
You
definitely
need
that.
We
can't
really
just
say
no
and
say
you
can't
get
this
traffic
it.
So
it's
example
a
place
where
we
haven't
found
a
good
solution.
Yet
for
an
automated
TCP
ingress,
we
keep
on
hoping
all
the
various
ingress.
Flavors
will
give
us
an
option,
but
currently
it's
mostly
CR.
B
A
Adoptions,
although
slow
its
kubernetes
as
a
huge
there's,
a
very
steep
learning
curve
right,
so
we're
we're
ramping
up
internally
on
that
we
have
a
cloud
native
Celia,
that's
helping
up,
helping
out
to
get
that
get
our
developers
be
more
productive
in
kubernetes
and
to
help
them.
You
know
establish
what
our
best
patterns
for
the
enterprise
instead
of
everyone
having
to
do
go,
do
their
own
research
and
then,
finally,
you
know
peak
ass
is
great
for
some
stuff.
A
That
still
has
some
issues
that
we're
looking
for
them
to
help
us
with
one
of
the
one
of
which
is
it's
it's
not
as
configurable
as
we'd
like
it
to
be.
We
can't
use
PUD
security
policies.
We
can't
use
pod
presets,
so
we
are
working
around
those.
We
were
looking
very,
very
hopefully
to
the
day
when
that
will
be
available
out
of
the
box
quickly
time.
Lessons
work,
so
the
lessons
aren't
just
for
kubernetes.
A
It's
for
it's
been
for
this
whole
three
year
exercises,
and
the
first
thing
is
on
what
you
can
it's
it's
you
know
if
we
want
to
deliver
quickly,
just
like
we
provide
a
platform
for
our
customers
to
go
faster.
We
have
to
go
faster
as
well,
so
we've
actually
taken
on
extra
load
for
infrastructure
components
that
we
would
not
traditionally
own.
A
We've
had
great
management
support
to
enable
that,
but
in
some
cases
where
we
don't
own
it,
we
have
to
form
closer
closer
partnerships
with
teams
that
own
our
services
and
impart
to
them
the
importance
of
velocity
and
how
why
they
need
to
move
more
so
quickly
with
us
limit
blast
radius.
So
we
started,
we
had
one
production
foundation.
A
You
know
we
started
losing
sleep
because
if
it
went
down,
it
was
everything
so
now,
we've
learned
from
day
one
tell
your
customers
to
deploy,
to
provide
your
customers
multiple
places,
to
deploy
and
tell
them
to
the
point,
multiple
places
so
that
you
don't
in
you,
don't
end
up
with
80%
your
workload
in
one
place
and
five
percent
utilization
in
two
roles
upgrade
impact.
Often-
and
this
is
we
didn't
do
this
and
we
let
a
couple-
customers
hold
us
back
and
huge.
A
A
So
so
our
new
much
argue
mantra
is
like
we.
We
do
not
inform
our
customer
or
reinforce
our
customers
when
we
were
going
to
upgrade.
We
did
not,
we
don't
ask
them,
we
don't.
We
can
slowly
do
a
little
bit
of
consulting.
We
know
we
don't
want
to
do
it
on
the
busiest
day
of
the
year,
but
we
essentially
when
we're
ready
to
patch.
A
We
run
it
through
our
staging
environments
and
them,
and
then
we
patch
we're
working
on
having
repaved
happen
automatically,
just
even
if
it's
not
needed
just
to
refresh
the
environment
from
time
to
time.
That's
good
for
us,
because
that
keeps
our
environments
fresh.
It's
a
so
good
for
our
customers,
because
it
kind
of
helps
them
build
the
muscle
of
what
a
repave
looks
like
and
how
they
need
to
deal
with
it
and
how
they
need
to
look
at
it
and
again:
don't
let
it
don't
let
individual
apps.
You
know
we're
shared
platform.
A
A
The
primary
foundation
is
it's:
it's
become
a
pet
right,
it's
not
a
cattle
anymore,
so
we
mean
we're
trying
to
figure
out
how
to
avoid
that
from
the
community
side.
Automate
everything
yeah,
that's
of
course
another
bit
another
big
thing:
it's
back
to
the
it's
back
to
the
you
know,
you
know
non
non
technical
requirements,
but
it's
create
community
for
your
customers.
Right
art.
We
created
a
cloud
founder
channel.
We
have
a
kubernetes
channel
now
and
we
already
have
in
both
places,
customers
answering
other
questions,
other
customers,
questions
which,
for
us
is
absolutely
fantastic
right.
A
That's
what
we
were
hoping
for,
what
we
were
looking
for,
but
it
takes
the
burden
off
of
us.
It
makes
everyone
feel
empowered
makes
everyone
feel
valuable.
You
get
immediate
feedback,
we're
not
on
there
all
the
time
it's
been.
It's
been
just
a
tremendous
tremendous
success.
How
many
people
do
we
have
on
the
cover
under
channel
like
doing
good
over
a
thousand.
C
B
A
Aware
of
it
and
our
our
pipelines
that
do
repays
and
those
kind
of
work
actually
post
to
the
channel
to
notify
customers
when
it's
happening
and
when
it's
finished
and
then
on
the
horizon,
we're
really
looking
for
kubernetes
we're
really
keen
forward
to
being
able
to
provide
hosted
data
services
and
not
just
that
we
provide,
but
that
any
team
can
provide.
We
have
awesome,
on-prem,
Data,
Services
teams
that
are
looking
down
boards
on
the
platform,
so
they
can
figure
out
how
to
move
their
customers
from
the
existing
legacy
infrastructure
onto
on
to
more
modern
platforms.
A
Service
smash
is
going
to
be
a
big
thing
for
us.
Caves
is
gonna,
be
a
big
thing
for
us.
It's
this
has
been
interesting.
Recurrent
topic
in,
but
I've
seen
here
is:
should
developers
be
using
cute
cuddly?
Probably
not
so
what's
what's
the
next
thing,
that's
going
to
be
the
abstraction
layer
that
helps
that
helps,
application
developers
and
owners
have
a
more
pad
like
experience
on
top
of
on
top
of
the
Cavs
platform.
So
that's
me:
that's
that
amazing
space
that
there's
a
lot
of
like
there's
a
lot
of
work
going
on
it.