►
From YouTube: Bringing 1976 into 2016: Ticketmaster’s Public Cloud and Kubernetes Strategy by Justin Dean
Description
Bringing 1976 into 2016: Ticketmaster’s Public Cloud and Kubernetes Strategy - Justin Dean, Ticketmaster.com
"Ticketmaster redefined an entire market by being the first to put computers in box offices. We now have four decades worth of products and tools that connect fans with the live events they love. When you are the big fish in the pond you have to reinvent yourself; even if it might be painful at times. Containers and enabling technologies like Kubernetes allow us to keep pace like never before.
In this talk, we’ll discuss the challenges that we’ve faced with having tens of thousands of servers and decades of technology that power the world’s largest ticketing engines. Imagine the need to build out your infrastructure to support demand spikes that go from idle to 100% capacity in only a few seconds. Now retrofit all that infrastructure and the applications to be cloud native before a hard deadline of data center lease renewals. What technologies would you use to make that possible?"
About Justin Dean
SVP, Platform and Technical Operations, Ticketmaster
Justin is the SVP of TechOps for Ticketmaster.com
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hello
alivelu,
so
we're
just
letting
a
little
time
elapse.
We
know
that
the
lightning
rounds
just
ended,
but
hopefully
that
give
you
a
little
like
view
of
of
what
we
do
and
hopefully
we
can
make
kubernetes
power.
All
of
that
which
is
pretty
cool,
so
we
get
started.
So
thank
you
so
much
for
for
having
us
we're
excited
to
be
here.
A
So
I
am
Justin
Dean
I'm,
the
SVP
of
platform
and
technical
operations
at
Ticketmaster
I've
been
there
for
a
year
and
a
half
and
I'm.
You
know
extremely
passionate
about
building
high-performance
teams,
high-performance
organizations
and
giving
the
scale
of
complexity
challenges
in
the
Ticketmaster
world.
I
think
I'm
in
the
right
place,
I'm
also
kind
of
nerdy
about
automating
things
that
you
might
not
should
be
automating
like
beer
and
barbecue.
So,
if
you're
into
stuff
like
that,
definitely
hit
me
up,
pitmasters,
pyro
and
github
kind
of
cool,
clever
and.
A
Cool,
so
why
are
we
here?
Why
did
we
come
here
today?
Really
we're
excited
about
kubernetes
we're
kubernetes
is
going,
and
we
really
wanted
to
give
some
perspective
on
like
how
does
a
company
like
Ticketmaster,
think
about
kubernetes
right?
Where
does
it
fit?
Where
do
we
see
it
unlocking
value
for
large
enterprise
companies
and,
ultimately
we're
helping
to
are
hoping
that
we
can
help
foster?
A
You
know
more
enterprise,
adoption
of
kubernetes
the
whole
ecosystem
of
software
out
there
right,
because
you
know
if
we
can
share
sort
of
our
path
and
sort
of
how
it
leads
into
the
business
and
our
business
strategy.
Hopefully
it's
relevant
to
some
of
you
to
some
of
your
shops
other
shops
out
there.
A
Ultimately,
hopefully,
we
hope
that
it
will
inspire
some
confidence
in
the
ecosystem
of
the
software
that
its
enterprise
caliber,
and
that
you
can
use
it
to
grow
and
expand
your
businesses,
get
more
confidence,
get
more
folks
involved
and
to
sort
of
join
up,
share
nodes,
quick
little
history
about
Ticketmaster.
So
one
thing
when
you,
when
you
join
the
company,
you
immediately
start
to
understand
the
gravity
of
the
fact
that
we
are
a
40
year
old
technology
company
all
right.
So
we
you
know
digitalized
tickets
in
1976
built
a
built.
A
A
ticketing
system
that
is
still
in
use
today
still
powers
a
lot
of
this
right.
Just
like
any
industry
like
Airlines
or
whatever,
like
you,
you
build
these
things,
mainframe
style.
They
don't
go
away
right,
you
you
may
expand
and
add
to
them,
but
there's
still
part
of
your
fabric,
part
of
your
story.
In
2010
we
partnered
with
Live
Nation
to
become
the
world's
largest
live
entertainment
company
and
then
in
2013
we
started
to
you
know
as
as
all
industries
all
companies
started
getting
really
disrupted.
A
We
made
a
conscious
decision
as
a
company
that
we
were
going
to
start
disrupting
ourselves
and
part
of
that
is
what
we're
going
to
talk
about
and
then
we're
at
you
know,
like
I
mentioned
we're
part
of
the
Live
Nation
family
and
on
a
revenue
side
we
do
twenty
five
billion
dollars
worth
of
Commerce
on
the
platforms
that
we
build
right.
So
we
are
hoping
to
sort
of
inspire
that
confidence
of
running
twenty
five
billion
dollars
on
kubernetes,
for
instance,
and
we're
a
top
five
e-commerce
site.
A
This
is
becoming
one
of
my
favorite
images
because
it
really
captures
what
it
is
that
we
do
right
if,
somewhere
in
the
world,
every
20
minutes,
there's
a
Live
Nation
event
I
mean
the
scale
of
that
statement.
The
gravity
of
that
is
is
pretty
phenomenal
right
and
then
and
to
think
about
what
we
do
at
at
Ticketmaster
and
what
our
technologists
do
is
build
the
tools
and
platforms
to
actually
power
this
stuff
right.
So
it's
a
pretty
cool
job,
to
say
the
least.
A
Our
company
mission
is,
you
know
we
power
unforgettable
moments
of
joy
and
a
couple
months
ago,
I
took
my
family
to
that
same
Coldplay
concert,
which
is
why
I
had
to
like
had
these
on
my
phone
right
and
then
I
saw
you
know
that
image
was
like
I
have
to
use
these
right.
Is
this
the
same
concert
right
and
and
I
start
thinking
about
the
gravity
of
what
it
is
that
we
do
right
and
sort
of
the
tools,
the
platforms,
the
systems
that
we
build
right?
A
A
As
a
company,
we
do
that
for
530
million
fans
in
37
countries
all
right,
so
just
like
the
video
clip.
If
you
saw
that
right
I
mean
it's
concerts,
sports
arts
venues,
clubs,
anything
live
right.
We
we
build
the
tools
that
that
power
that
I'm
not
going
to
go
into
this.
This
is
the
speeds
and
feeds
slide,
but
you
know
one
thing:
I
noticed
when
you
talk
to
people
about
Ticketmaster
and
sort
of
our
scale
of
challenges,
immediately,
people
think
about
the
website.
Ticketmaster,
calm
right.
A
If
you've
ever
tried
to
buy
tickets
for
something,
let's
say
you
wanted
to
go
to
Adele.
You
know
what
it's
like
dealing
with
the
website
and
it's
not
the
necessarily
the
technical
portions
of
the
website.
It's
the
fact
that
there's
not
enough
inventory
to
go
around
right,
but
the
story
is
so
much
larger
than
that
right
for
us,
and
especially
as
we
look
at
our
platforms
as
we
look
at
how
we
roll
out
software,
it's
not
just.
How
do
we
make
the
website
faster
and
better
right?
It's?
A
How
do
we
power
all
of
the
things
required
to
run
a
fortune,
500
company
and
an
enterprise
that
has
to
deal
with
all
kinds
of
complexities
that
you
would
get
from
venues
to?
You
know
some
of
some
of
these.
You
know
we
do
festivals,
but
you
know
some
of
these
stats
are
pretty
telling
right.
We
do
a
half
a
billion
ticket
sold
so
think
about
the
Commerce
and
the
backend
channels
to
make
that
happen.
All
right,
21
million
app
downloads,
70
million
unique,
hits
to
our
website
every
month.
A
Right
and
these
are
these
are
not
quick,
like
you
know,
quick
social
hits
right.
These
are
I'm
here
to
spend
thousands
and
try
to
add,
say
season
tickets
for
your
favorite
football
team.
Right
I
mean
imagine
one
transaction
of
that
of
that
scale
right.
So
it's
it's
definitely
part
of
our
fabric,
part
of
our
story,
part
of
our
challenge,
so
we
also
have
a
large
competitive
landscape.
That
is,
is
a
driver
for
what
we're
doing
and
a
catalyst
to
to
force
us
to
evolve
right
it
just
like
every
other
business
out
there
right.
A
So
when
you're
the
the
market
leader
like
like
we
are
in
this
space,
we
are
in
a
position.
You
know
it's
great
to
be
the
market
leader.
It's
also,
you
know
a
terrifying
place
to
be.
You
know,
when
you've
got
every
company
out
there,
basically
trying
to
get
a
portion
of
your
business
right
so
Ticketmaster
our
ecosystem
is
we
have
a
huge
surface
area,
one
little
piece
of
that
surface
area
could
be
an
entire
business
for
a
start-up
or
a
smaller
company
right.
A
A
On
top
of
that,
we
have
a
our
fair
share
of
tech
complexities.
You
know
one
part
of
being
a
40
year
old
company
and
successful
in
a
market
leader.
Is
you
end
up
with
every
era
of
software
right,
and
you
may
not
believe
this,
but
some
of
its
not
ready
for
clouds
and
containers
right.
So
you
know
we're
that's
a
part
of
our
challenge.
You
know
the
the
70s
tie-in
right
is.
A
The
main
nucleus
of
our
ticketing
system
is
VAX
right,
so
we've
emulated
it
because
you
can't
buy
Hardware
on
eBay
anymore
for
facts,
but
you
know
how
do
you?
How
do
you
get
backs
stuff
to
run
inside
of
a
kubernetes
ecosystem
right?
So
we're
actually
working
on
that?
Craig
will
go
into
a
little
bit
more
details
on
that.
A
But
you
know
I
mean
we
have
every
air
era
of
you
know
whatever
the
the
the
architecture
the
drawer
was
at
that
time
and
then
whatever
API
layers
on
top
of
that
right
and
then
we
have
a
huge
footprint
of
you
know
what
used
to
be
fairly
modern
tech.
Five
years
ago
right
we
have
a
huge
private
cloud,
Zen
base,
you
know,
just
you
just
look
up
what
private
cloud
architectures
were
five
or
six
seven
years
ago.
We
have
that.
A
So
we
we
have
this
pre
sort
of
modern
technology
complexity,
but
it
runs
a
very
successful
business
right.
We
have
21
full
blown
ticketing
systems,
250
unique
products
that
run
on
top
of
this
stuff.
Right
this
is,
you
know
from
every
you
know:
20
40
year
old
company.
Part
of
that
is
you.
You
end
up
doing
acquisitions.
Everyone
who's
been
through
acquisitions.
Understand
like
how
challenging
that
is
right.
A
You
may
be
in
the
same
market,
but
you
never
really
quite
tie
everything
into
one
platform
right,
so
we
have
21
of
those
full-blown
businesses,
1,400
people
in
our
technology
organization
right.
So
that
means
that
these
1,400
people
really
have
to
be
up
to
speed
on
whatever
platform
stuff
we
put
out
there
right
so
how
we
deliver
them.
Kubernetes
right
is
a
factor
when
twenty
fourteen
hundred
people
have
to
understand
how
to
use
those
technologies
right.
A
We
have
a
huge
private
cloud
that
we're
working
on
migrating
22,000,
vm7
global
data,
centers,
15,000,
Network
endpoints,
it's
kind
of
a
unique
thing
for
us,
because
you
know,
if
you
think
about
the
ticket
Ticketmaster
ecosystem,
we
have
touch
points
everywhere.
You
walk
into
a
venue.
Alright,
you
go
to
the
Seahawks
game.
You
scan
your
ticket.
Boop
I
mean
that
eventually,
that
stuff
routes
in
in
sort
of
our
products
base
into
our
world,
we
validate
the
tickets
in
real
time
right.
A
So
we
have
a
million
machines,
also
hitting
us,
and
then
we
make
it
harder
on
ourselves
by
also
telling
everyone
what
time
they
go
on
sale,
right,
Friday,
turn
o'clock,
be
there
like
it's
so
set
your
clocks,
so
it
just
adds
to
the
complexity
of
the
the
you
know:
complex
business
and
web-scale
challenges,
so
recapping
real
quickly,
right,
public
company
pressures,
all
across
the
board.
With
that
market
pressure
are
highly
competitive
right.
We
make
money
by
selling
software,
we're
a
software
company
right.
A
A
It
takes
to
run
a
business
of
this
size
and
what
that
means
is
we
spend
a
lot
of
time
paying
the
interest
rate
on
that
debt
in
the
form
of
operations,
and
you
know
like
most
folks:
we
want
to
have
more
people,
you
know
future
stage
strategic
focused,
not
operating,
and
then
we
have
Black
Friday
every
day,
so
all
roads
for
us
as
a
company
right
to
get
better
as
a
company
all
roads
lead
to.
We
must
get
faster
right.
We
have
to
be
able
to
like
be
nimble
and
quick
right.
A
So
for
us,
what
that
means
is
we
have
to
simplify
our
platform
right,
so
how
we
get
here
our
sort
of
business
case
for
this
right
is:
we
need
more
revenue,
more
market
share
right,
that's
probably
everybody's
challenge:
we
need
better
products,
more
features,
better
features
all
right.
How
do
we
get
that?
We
have
to
deliver
products
faster,
all
right
and
our
bet,
and
what
we've
been
doing
over
the
last
couple
years
is
creating
teams
that
are
autonomous,
alright.
So
back
to
the
other
slide,
we
have
the
big
shark
and
a
bunch
of
fish.
A
We
are
trying
to
create
a
bunch
of
fish
inside
of
the
big
shark
right,
so
we
have
basically
micro
businesses
that
can
run
their
own
stacks
completely
right,
instead
of
relying
on
every
end
of
the
business
to
release
something
so
autonomous
product
teams
that
can
keep
up,
keep
pace
and
beat
pace
of
startups,
and
in
order
to
do
that
and
enable
of
that,
we
have
to
simplify
the
platform
that
we
offer
those
teams
to
make
it.
You
know
something
that
they
can
consume
all
right.
So
every
presentation
has
to
have
this
meme
here.
A
It
is
we're
working
on
simplifying
all
of
our
things.
So
how
do
we
do
it?
And
you
might
I
have
to
address
the
fact
that
there's
this
image
of
me
and
this
70
stuff,
we
had
a
recent
70s
party.
So
we've
got
a
lot
of
like
stop
footage
of
this.
We
thought
it
fit.
It
fit
tight
and
tight
and
well
so
our
goal
of
all
of
this
right
is
let
the
makers
make
right.
A
We
have
an
amazing
company
of
makers,
creators,
visionaries,
innovators,
you
know
people
who
can
focus
on
delivering
products
to
market
and
that's
where
they
should
like
figure
out
the
next
big
thing
to
power
our
business
and
to
make
it
better.
We
do
not
want
to
burden
them
with
also
having
to
figure
out
how
to
deploy
infrastructure
around
to
support
their
software
right.
It
doesn't
make
a
lot
of
sense
to
have
the
people
who
are
delivering
software
to
make
money,
also
deliver
software
to
deliver
software
right.
So
for
us,
it's
about
removing
that.
A
So
we
you
know
to
get
conviction
as
a
company
in
every
big
company
like
you
have
to
get
you
have
to
get
you
make
a
business
case
that
and
make
it
a
business
case,
not
a
technology
case
right,
so
for
us
it
was.
We
need
to
be
faster
as
a
company.
This
is
the
solution.
Right
that
way,
we
get.
You
know
clear.
Executive
level.
A
So
a
little
bit
about
our
journey,
how
we
got
here
so
big
picture
wise
we've
been
disrupting
ourselves
for
the
last
couple
years.
We
are
now
in
sort
of
this
space
towards
the
end
where
we're
in
the
public
cloud
adoption
and
we're
moving
into
our
Cooper
Nettie
phase.
On
top
of
that,
so
I'll
go
quickly
through
just
to
give
you
some
context,
because
you
can't
just
start
at
the
kubernetes
phase
and
especially
in
a
big
company,
so
self
disruption
right.
Everybody
understands
this
like
change
or
disappear
right,
so
we
got
clear
about
that.
A
A
few
years
ago
we've
been
disrupting
the
hell
out
of
ourselves.
We
want
to
be
Ebor,
not
yellow
cab
right.
End
of
story,
don't
be
yellow
cab,
no
offense
to
yellow
cab
folks
out
there.
So
part
of
that,
for
us
was
what
we
call
the
lean
transformation.
We
got
super
super
laser
focused
about
what
what
does
what
do?
We
really
need
to
get
done
as
a
company
instead
of
working
on.
A
You
know
a
hundred
different
things,
work
on
six
that
are
going
to
be
meaningful
in
the
market,
take
the
workforce
and
get
them
laser
focused
on
those
six
to
deliver
right.
So
what
that
looked
like
for
us
is,
we
ended
up
with
sixty
five-plus
cross-functional
teams,
those
micro
businesses
and
TAC
practically.
What
that
looked
like
for
us
was
merging
products,
dev
QA
UX,
into
these
teams
with
a
mission.
They
got
really
really
really
good
at
writing
software.
A
They
did
not
get
any
better
at
delivering
software
right.
There's
a
huge
gap
between
we
wrote
it
and
it's
making
money
in
prod
right.
So
all
roads
led
to
we're
blocked
by
ops,
and
we
didn't
scale
up.
You
know
so
entered
our
insert
our
DevOps
phase.
We
DevOps
hard
as
a
company
I,
won't
go
into
that.
Everybody
probably
is
wildly
aware
of
the
challenges
of
you
know
that
the
culture
changes
and
everything
it's
a
tectonic
shift
for
companies
to
deal
with,
especially
ones
with
40
year
old
muscle
memory.
A
So
we
spent
the
last
couple
years
really
really
doing
this
to
the
point
where
work
changes
and
the
big
point
was
really
taking
the
people
who
build
and
support
a
particular
product,
take
them
out
of
the
central
ops
areas
and
put
them
closer
to
like
embed
them
into
the
product
teams
right.
So
you
get
a
stick:
self-sufficient
business
teams
right,
product
teams,
you
build
it,
you
run
it
you
own
it
you
optimize
it.
You
monetize
it
right,
and
then
you
hit
them
with
the
chargeback
P&L
stuff
to
keep
them
honest
across
the
board.
A
So
we
did
that
and
then
that
took
us
or
public
cloud
phase,
and-
and
so
you
know
the
relevance
in
the
story
there
is.
Is
we
got
to
a
point
where
we
had
these
teams
who
were
able
to
they
had
the
destiny
was
in
their
hands?
Their
tools
were
not
ready
right,
so
the
for
us.
We
were
in
this
public
cloud
phase
right
now,
where
we
were
using
it
as
a
giant
carbon
filter
for
all
of
our
products.
A
Right
we're
thinking
about
it
a
little
bit
different
than
we
don't
care
who
we're
paying
for
resources,
compute
stores
it
doesn't
matter
right.
It
is
the
deep
introspection
of
every
product
that
we
have
every
piece
of
software,
making
it
cloud
native
doing
whatever
we
need
to
do
to
get
the
illah
DS
up
to
speed
right.
You
can't
run
in
a
public
cloud
or
in
a
kubernetes
ecosystem.
A
If
you
don't
have
you
know
the
basics
around
your
application
in
a
modern
fashion
or
I
need
to
be
repeatable,
scalable
reliable,
you
know
all
those
things
but
like
resiliency
is
a
huge
one,
so
we're
using
that
as
our
vehicle
right.
So
when
we
get
out
on
the
other
side,
our
products
are
up,
leveled
and
we're
kubernetes
fits
for
us
is
the
tooling
and
the
self-service
abilities.
A
As
our
simplification
layer
tell
us
where
you
want
it
to
run
how
many
replicas
and
we
will
abstract
all
of
the
complexities
on
the
back
end
right.
We
don't
want
hundreds
of
teams
out
there
learning
how
to
deliver
infrastructure
right,
keep
that
stuff
hidden
from
them
and
with
that
Craig
will
go
into
a
little
bit
of
more
detail
of
around
all.
B
Of
that
all
right,
so
we're
heading
into
the
public
lab
and
it's
just
I
was
saying
you
know
we
in
order
to
do
that,
we
got
a
simple
higher
platform.
So
in
order
to
simplify
it,
you
have
to
know
what
you
have
so
we
started
with
40
years
of
acquiring
businesses
that
had
their
own
products,
creating
our
own
businesses
that
had
their
own
products,
but
we
didn't
have
any
canonical
list
of
what
our
products
actually
were.
So
we
created
this
product
inventory
management
tool,
which
is
the
single
location.
B
We
can
go
to
get
all
the
information
about
what
we
have
and
the
state
of
it
who
owns
it,
where
it
runs
who's,
operating
it
and
any
other
interesting
pieces
of
information,
and
then
we're
you
know
now
that
we
know
we
have.
We
can
start
to
migrate
it
to
the
public
cloud
and
we're
using
this
as
an
opportunity
to
really
re-evaluate
all
of
these
products
and
figure
out
which
ones
you
want
to
take
with
us,
which
ones
we
want
to
improve,
which
ones
we
you
know
we
really
don't
feel
like.
B
We
need
to
invest
more
and
may
die
in
our
data.
Centers
and
then
we're
using,
we
created
this
tech
maturity
model
to
have
a
common
grading
system
for
all
of
our
products
and
where
they
are
with
their
technology.
So
you
know,
maybe
some
of
them
are
using
different
coding
practices
that
we'd
like
to
improve,
or
you
know
they
don't
have
h
a
mullet
or
something
like
that.
This
is
giving
us
a
common
score
for
all
of
these
different
things
we
open-source
this
earlier.
B
Today
it
was
on
the
Ticketmaster
tech
blog
and
then
it
was
also
a
guest
post
on
the
aprender
blog.
So
you
can
take
a
look
at
it
see.
What's
in
there,
there's
like
40
something
capabilities
that
are
measured.
We've
got
the
cloud
enablement
team
and
these
guys
are
all
about
helping
engineering
teams
migrate
themselves
into
the
public
cloud.
So
we've
got
250
products,
so
we've
got
200
ish
teams.
We
don't
want
200
different
ways
of
using
the
cloud
this.
B
This
team
is
full
of
people
figuring
out
all
of
the
public
cloud
solutions
to
our
private
cloud
problems
and
then
documenting
and
educating
the
rest
of
the
workforce
on
how
to
do
this
stuff.
So
they
don't
have
to
do
and
figure
it
out
on
their
own,
and
then
we've
got
kubernetes
right,
so
the
CET
team
is
all
about
getting
people
into
the
cloud
as
quickly
as
possible.
That's
definitely
not
the
most
optimal
use
case.
So
kubernetes
we
see
is
a
way
of
optimizing.
B
How
we're
using
the
public
cloud
and
getting
you
know
more
rapid
development
and
everything
else
we
can
get
by
using
leveraging
the
api's
and
primitives
in
it.
So
the
tech
maturity
model
is
saying:
it's
got,
40
capabilities
in
these
dimensions
of
code.
So
that's
like
how
you're
writing
your
software
is
the
code
reusable
we
build
and
test
so
like
CI
types
of
thing,
automated
testing
release,
so
how
you
deploy
it
to
your
different
environments,
you're
doing
CD.
Are
you
doing
you?
Basically,
all
those
practices
operate
so
how
you
run
it
when
it's
there
you
like?
B
Are
you?
Have
you
know,
monitoring,
alerting,
dashboards,
that
kind
of
stuff
and
then
optimizes
how
you
come
back
around
on
the
products?
And
you
know:
are
you
resolving
your
root
causes
to
business
disruptions
and
are
you
dealing
with
your
tech
debt
so
that
was
open
sourced
today
you
could
check
the
aprender
blog
or
the
I
think
we
probably
tweeted
it
too.
We've
created
this
tech
materiel
portal
that
we
have
internally.
It
rolls
up
all
the
information
and
all
the
scores
for
our
different
products
into
the
different
business
verticals.
B
So
you
can
go
and
see
how
they're
operating
we're
going
to
open
source
version
of
that
later.
This
year
is
currently
being
worked
on,
but
what
you
know
the
tech
maturity
model
allows
you
to
do
is
quantify
things
that
were
used
to
be
hard
to
quantify
like
tech
debt.
So
you
can
have
this
clear
score
in
a
roadmap
of
where
you
want
to
go
to
improve
your
product.
B
This
is
what
it
looks
like.
If
you
look
on
the
left
hand
side,
you
can
see
that
this
team's
score
they're
better
at
Cody,
building
and
testing
and
optimizing
than
they
were
at
say
operating.
So
that
highlights
you
know.
Maybe
they
have
some
improvements
they
can
make
to
how
they
monitor
and
run
their
application,
but
they're
really
good
at
dealing
with
the
problems
that
come
up
by
not
doing
that
and
then,
as
just
I'm
saying,
you
know,
we
DevOps
hard
and
it
shows
here.
B
The
Green
Line
shows
that
they're
their
highest
score
is
on
DevOps
practices.
So
if
you
look
at
this
capability
level,
one
is
basically
like
throwing
your
application
over
the
wall
and
letting
some
other
team
deal
with
it
and
then
level.
Four
is
the
full-on
DevOps
where
the
engineering
team
owns
and
operates
it
in
production.
B
Here's
one
more
screenshot
from
the
tool
on
the
right-hand
side.
You
can
see
this
min
cloud
readiness
column,
so
we
picked
a
subset
of
capabilities.
We
felt
like
we
were
critical
for
moving
into
the
public
cloud
and
like
what
those
levels
should
be
before
you
even
consider
the
migration.
You
can
see
that
on
this
one,
the
first
red
one
there
on
the
right
is
the
three
we're
saying
you
know
you
should
have
active,
actively,
build
and
maintain
unit
test,
functional
test,
critical
path
test
and
all
these
other
tests
right.
B
But
these
guys
are
a1,
they
don't
have
any
testing
at
all.
So
in
order
to
remove
it
into
a
cloud
you
should
know
if
it's
working
and
then
the
cloud
enablement
team.
So
this
is
a
group
of
architects
and
they've
been
fault.
You
know
figuring
out
all
the
ways
to
use
the
public
cloud
rather
than
the
the
ways
we've
been
using
the
private.
So,
for
example,
if
you're
using
Amazon
you've
got
a
bunch
of
static
content,
you'll
probably
want
to
put
that
in
s3.
B
But
if
you've
got
your
own
datacenter,
you
may
have
an
NFS
server
where
you,
through
all
that
kind
of
stuff
in
this
server,
so
they've
been
figuring
out
all
this
stuff
documenting
it
writing
tools
to
make
it
easier
and
enable
our
engineering
teams
to
do
that.
The
you
know
we
spent
the
first
quarter
working
with
the
team,
building
the
documentation
and
the
tools.
Second
quarter.
We
gave
it
to
another
team,
had
them
migrate
themselves
and
then
you
know
kind
of
watched
how
it
worked
and
filled
in
any
gaps
and
then
in
the
third
quarter.
B
We
only
sit
on
the
rest
of
the
company,
so
they
can
start
doing
by
themselves.
It's
gone
pretty
well,
so
you
know
the
CT
method
was
hey,
just
get
everybody
moving
right,
so
there
was
only
three
demands
we
made.
First
one
was
to
docker
eyes
the
app
with
we
figured
if
we
can
just
get
them
to
put
into
a
container.
We
could
always
come
back
and
improve
things
underneath
it.
We
said
running
on
core
OS,
we're
huge
fans
of
core
OS
and
the
one
thing
that
core
OS
Linux
gives.
B
You
is
a
like,
a
container
running
appliance
so
that
you
don't
have
to
go
and
get
into
like
the
worlds
of
chefs
and
puppets
and
anibal's
and
that
kind
of
stuff-
and
we
were
saying
terraform
the
cloud
infrastructure
because
we
didn't
want
people
going
and
building
a
bunch
of
custom-built
certain
infrastructure
for
the
application
through
the
UI
and
they're
not
being
able
to
reproduce
it.
So
this
definitely
has
some
challenges
that
come
along
with
it
right.
Development
with
terraform
is
really
slow.
It's
got
to
use.
B
The
Amazon
API
is
gotta,
wait
for
ec2
is
says
to
spin
up
that
kind
of
thing,
we're
launching
all
over
to
containers
with
cloud
in
it
and
that
you
know
that's
not
good
at
all
right.
Every
time
you
need
to
change
something
interchanges,
auto
scaling
group
wait
for
a
bunch
of
stuff
to
change
and
then
it's
totally
inefficient
use
of
the
cloud
resources,
because
we
were
firing
above
these
VMs
and
we're
spinning
up
the
docker
containers
on
top
of
it,
we're
getting
this
one-to-one
mapping
and
we're
not
being
able
to
do
any
kind
of
compaction.
B
So
that's
where
we
see
kubernetes
coming
in
and
helping
right.
So
there
was
people
all
over
the
company
starting
to
use
kubernetes
within
their
own
teams,
just
as
a
way
of
deploying
their
application
and
managing
it.
You
know
the
community
is
great.
We
used
it
in
a
POC
to
solve
a
problem
with
open,
TCB
and
I'll.
Talk
about
that
in
a
second
I
really
think.
It's
way
ahead
of
everything
else,
that's
out
there
right
now
right,
there's
some
other
stuff,
but
it's
really
not
quite
this.
What's
what
kubernetes
has
you
know?
B
The
API
is
the
primitives
are
all
awesome
and
it's
getting
us
better
compaction
of
our
ec2
instances.
So
this
POC
II
open
tastes
to
be
if
you're
not
familiar
with.
It
is
a
lot
like
graphite
or
in
flux.
Db,
it's
an
open
time.
It's
a
time
series
database,
so
we've
pumped
in
about
a
half
a
mo
metric
per
second
into
it.
We
use
it
for
all
of
our
monitoring
when
we're
doing
these
ticket
sales.
But
what
would
happen
is
everybody
would
go
to
it
during
a
ticket
sale
and
then
we
would
have
our
own
internal.
B
You
know
DDoS
right,
so
the
services
would
get
overwhelmed,
they'd,
lock
up
or
crash,
and
somebody
have
to
may
only
go
in
there
and
bring
it
back
up
by
putting
into
kubernetes
and
leveraging
the
pod
health.
Jackscrew
brands
can
catch
that
for
us
and
bring
it
back
up
for
us.
So
it
took
something
that
we
don't
have.
We
don't
have
to
go
in
there
and
manually
manage
it
anymore,
right
it
just
kind
of
does
its
own
thing,
it's
great,
but
it's
not
perfect.
We've
had
some
problems
with
it.
B
Where,
like
the
automated
core,
OS
operating
system
would
upgrade,
but
the
services
with
kubernetes
wouldn't
come
back
up
by
themselves.
We've
tracked
some
bugs
down.
We
haven't
looked
into
it
too
closely,
and
then
we
got
the
hosts.
So
the
host
is
the
original
ticketing
engine
that
powers
Ticketmaster.
B
B
So
the
project
was
to
deploy,
take
a
master
product
into
a
production
grade,
kubernetes
cluster
and
equipped
the
team,
with
all
the
knowledge
to
be
able
to
support
it
in
production.
Those
it's
a
fully
remote
team
of
six,
most
of
them
right
here
today,
even
in
this
room.
So
we
got
a
lot
of
work
to
do
right.
We
don't
really
know
how
many
clusters
we
need
to
have
for
our
environment
right.
We've
got
to
figure
that
out.
We
don't
know
which
architecture
is
quite
right
for
what
we
need
to
do.
B
We
don't
know
how
we're
well.
We
know
this
one
now,
but
how
we're
gonna
plan
to
deploy
and
test
the
cluster
once
we've
built
it
and
we're
really
not
sure
about
what
the
best
networking
option
is
to
use
at
Amazon
when
you
start
to
factor
in
the
multi
easy
deployments,
but
it's
a
totally
legit
team.
We've
got
our
own
logo.
We've
got
our
own
beer
koozie
up
here
somewhere
and
we've
got
stickers.
So
these
are
the
milestones.
B
Basically,
you
know
we
start
with
a
simple
cluster
and
then
we
start
to
add
some
complications
on
top
of
it
right.
You
know
we'll
figure
out
how
to
optimize
it,
how
to
make
it
Enterprise
ready,
how
to
make
it
a
che
and
make
it
easier
to
use
by
application
teams
figure
out
how
we're
gonna
do
the
on-call
production
support
and
all
these
other
stuff.
But
you
know
now:
we've
got
tectonic,
we
partnered
up
with
core
OS
yesterday.
B
We
really
see
this
as
a
way
of
weakness
to
making
this
whole
thing
happen
within
a
few
weeks,
rather
than
a
few
months,
they've
got
a
lot
of
the
enterprise
stuff
figured
out,
so
you
know
really
planning
to
take
advantage
of
that
and
we
brought
in
other
people
to
help
along
the
way.
So
JJ
was
here
right
now,
but
so
he
had
a
prin
to
come
in.
We
spent
three
days
locked
up
in
a
room.
Reviewing
everything
Ticketmaster
does
reviewing
everything.
B
Qu
brandies
can
do
and
kind
of
trying
to
map
the
two
together
to
figure
out
what
the
best
bottle
is
for
us
to
manage
it,
both
like
technically
and
as
an
organization.
So
these
are
all
the
kinds
of
things
we
cover.
I
mean
pretty
standard
stuff,
and
then
we
had
Kelsey
Hightower
come
in
and
work
with
the
team,
so
he
really
walked
us
through
a
lot
of
standard
production
deployments
that
he's
seen
working
with
other
companies.
Kelsey's
got
a
lot
of
real-world
knowledge
and
really
is
like
a
great
person
to
talk
to
you.
B
B
Don't
go
complicate
it
until
you
have
to
go,
complicate
it
right
and
then
he
gave
us
a
bunch
of
unbiased
recommendations
on
kubernetes
deployments
and
then
we
brought
in
core
Wes
and
they
gave
us
a
bunch
of
bias
recommendations
like
River
days
deployments,
but
it
was
great
because
their
their
architecture
was
a
lot
like
the
architecture
we
were
planning
to
build.
So
we
had
a
ton
of
stuff
in
common.
We
all
got
along
really
well
and
that's
really
what
led
to
the
partnership.
B
B
So
you
know
now
that
we
have
this
where
we
were
planning
to
go
with
it,
you
know
we're
really
into
linker
D
the
guys
over
there
have
been
great.
We,
the
client-side
low
balance.
You
could
do
with
it.
It's
awesome,
you
know
when
Twitter
came
out
with
finagle,
it
looked
great,
but
you
can
only
use
it
if
you're
using
services
that
were
like
Twitter's,
link
or
D
makes
this
you
can
use
it
on
everything
right
we've
been
around
for
so
long.
B
We've
got
every
version
of
software,
that's
come
around
in
the
last
40
years
we
can
almost
use
linker
D
and
between
all
of
those
things
right.
It
speaks
HTTP
also
super
impressed
with
Prometheus.
We've
already
started
to
convert
some
applications
that
are
using
open,
TCP
to
emit
metrics
in
Prometheus
and
then
converting
them
back
into
open
TCP.
Just
because
we
know
we
want
to
get
rid
of
open,
TCP
and
start
using
Prometheus
there's
also
some
really
cool,
open
tracing
stuff
happening.
Basically,
anything
CN
CF
is
doing
we're
trying
to
do
now.
A
Cool
so,
as
I
mentioned
earlier
right,
our
our
goal
is
really
to
share
about
what
we're
doing,
how
we
think
about
kubernetes,
how
we're
solving
our
business
problems
with
kubernetes
in
the
ecosystem
in
general,
with
container
orchestration
in
general,
right
and
and
how
to
how
we're
thinking
about
solving
enterprise,
caliber
problems
and
running.
You
know:
25
billion
dollars
on
of
Commerce.
A
The
kubernetes
community
is
the
community
that
we
want
to
be
part
of,
and
we
want
to
sort
of,
encourage
others
to
join
us
along
the
journey
right
and
and
and
be,
you
know,
add
more
anchors
of
big
companies
into
the
community
so
that
it
can
continue
to
thrive
grow
so
that
some
of
these
problems
get
solved
and
we
divide
punker
right,
like
somebody
eventually
is
gonna
figure
out
what
the
best
way
to
do
off.
Somebody's
gonna
figure
out
all
the
problems
that
you
have
to
do.