►
From YouTube: Case Study:The Road to Kube at Ticketmaster, Tim Nichols, OpenShift Commons Gathering Seattle 2018
Description
Case Study: The Road to Kubernetes @ Ticketmaster, Tim Nichols at OpenShift Commons Gathering Seattle 2018
https://commons.openshift.org/gatherings/Seattle_2018.html
A
A
Okay,
that's
a
lie:
I
am
a
nerd,
so
I
care
personally
about
tick
kubernetes.
But
let
me
explain
what
I
mean
when
I
say:
I
don't
care
about
kubernetes
about
two
years
ago
I
was
my
son
was
turning
13.
His
birthday
is
on
New
Year's
Eve
and
my
wife
and
I
decided.
We
wanted
to
give
him
his
first
real
rock
concert
experience.
A
So
he
was
this
huge
twenty
one
pilots
fan.
Do
we
have
any
twenty
one
pilots
fans
and
then
the
yeah
yeah.
You
can
applaud
that.
You
know
they're
they're,
their
favorite
or
their
one
of
their
most
popular
songs.
He
would
wander
around
my
house
singing
it's
blurryface.
My
name
is
blurryface
and
I
care
what
you
think
right
to
which
his
loving
siblings
would
look
at
him
and
say.
Well,
we
think
you
should
stop
singing,
there's
a
lot
of
love
in
my
house,
so
we
took
him.
A
We
took
him
to
Sasquatch
festival,
I,
don't
know
how
many
people
here
have
been
to
the
gorge.
This
I
took
this
picture
at
the
concert
that
I
took
him
to
it's
a
it's
an
amazing
venue.
It
is
so
much
fun
beautiful.
You
camp
kind
of
back
behind
where
these
folks
are
at
and
what
I
did
was.
You
know
I
drove
out
four
hours
from
here
with
my
son
and
we
spent
we
spent
some
time
in
the
car.
A
This
is
kind
of
what
it
takes
to
get
a
thirteen-year-old
to
talk
to
you
as
a
parent,
and
then
we
spent
two
days
at
a
Sasquatch
festival.
He
got
to
see
21
pilots,
but
he
also
got
to
open
his
eyes
to
a
whole
bunch
of
other
experiences
and
it's
something
that
him
and
I
will
go
home
and
we'll
never
forget
right.
So,
when
Ticketmaster
talks
about
how
we
power
unforgettable
moments
of
joy,
this
is
what
we
do
are
we're
in
the
business
of
giving
people
amazing
lifetime
experiences
that
they'll
never
forget.
A
So
when
I
say
I
don't
care
about
kubernetes,
I
don't
care
about
kubernetes
the
technology
I
care
about
how
kubernetes
will
help
me
do
this
for
more
people
right
and
live
nation,
our
parent
company.
We
do
one
of
these.
We
do
something
like
this
every
20
minutes.
On
average,
it's
pretty
cool,
it's
really
cool
to
work
for
a
company
that
our
job
is
to
make
people
happy
just
make.
People
have
fun,
and
it's
not
always
easy.
So
some
of
these
really
huge
events,
they're
really
high
demand
events.
A
We
might
have
a
you
know,
tens
of
thousands
of
tickets
and
hundreds
of
thousands
of
people
who
want
those
tickets.
So
when
the
tickets
go
on
sale,
we
see
something
that
looks
like
this
in
our
network
and
on
our
in
our
infrastructure.
It's
effectively
a
Black
Friday
and
Cyber
Monday
combined
event,
right,
Taylor,
Swift
or
Garth
Brooks,
or
somebody
like
that.
When
they're
going
on
sale,
everybody
wants
those
tickets
and
it's
not
like
a
typical
retail
retail
event.
A
A
So
that
leads
to
some
interesting
scale:
we're
not
a
Google
or
any
Amazon
for
sure,
but
we
have
27
ticketing
systems.
We
have
about
1400
people
across
our
tech
and
product
organization,
and
you
know
we're
here
to
talk
about
kubernetes
right.
So
to
date
we
have
about
a
thousand
nodes
running
on
kubernetes
16,000
pods
across
our
clusters,
and
we've
been
growing
like
mad,
so
this
is
spread
across
AWS
and
our
on-prem,
so
we're
hybrid,
with
kubernetes
we've
almost
doubled
this
year
next
year
is
gonna,
go
even
bigger,
I'm
sure.
A
So
I'll
introduce
myself.
My
name
is
Tim
Nichols
I
run
the
hybrid
cloud
kubernetes
and
developer
platform
organization
that
Ticketmaster
I'm
gonna
tell
you
basically
our
story.
How
we
got
here
this
it's
really
hard
to
have
this
story
without
talking
about
our
DevOps
transition
that
led
to
it,
because
kubernetes
is
really
ultimately
the
the
culmination
of
that
story.
So
when
I
started
it
take
a
massacre
about
six
years
ago
we
were
dead
and
we
were
ops
right,
most
come
I.
Think
most
companies
were
sort
of.
A
At
that
same
stage
we
had
developers
who
would
want
to
resource
they'd
want
a
server,
let's
say,
and
they
they'd
open
a
ticket.
They
didn't
talk
to
ops,
they
open
a
ticket
right
and
then
ops
would
build
the
server
to
the
spec
that
they
got
and
they'd
hand
it
back
and
dev
would
say:
that's
not
what
I
wanted
and
then
they'd
open
another
ticket
and
they'd
go
back
and
forth
like
this.
Sometimes
it'd
take
a
week
for
someone
for
a
developer
to
get
resources.
A
Do
you
any
of
you
guys
remember
what
that
was
like
still
have
that
experience
today,
sometimes
probably
yeah.
So
we
dug
in
really
hard.
We
knew
we
had
to
fix
this
because,
as
a
big
leader
in
the
industry,
we
were
going
to
get
eaten
alive.
If
we
didn't
change
our
pace,
so
we
dug
into
DevOps
and
we
started,
we
started
a
developer
cross-training
program.
A
We
started
teaching
developers
how
to
do
the
operational
work
because
they
they
hadn't
seen
any
of
that
before
we
started
giving
them
access
to
their
servers
that
they
had
never
had
before,
and
then
this
really
accelerated
for
us.
But
we
had
a.
We
still
had
another
problem
with
tech
debt
40
year
old
company
right
we
did
and
we
needed
to
create
something
of
a
carbon
filter.
We
needed
to
figure
out
a
way
for
developers
to
uplevel
their
products,
so
we
brought
in
some
containers
and
we
brought
in
some
AWS.
A
We
gave
them
all
the
operational
access
we
gave
them
taught
them
how
to
how
to
build
everything,
and
we
gave
them
toolkits
to
do
that,
but
across
an
organization
of
a
thousand
developers
that
turns
into
something
else.
It
starts
to
look
a
little
bit
more.
Like
managed.
Chaos
managed
right,
and
we
at
this
point
knew
we
had
to.
We
had
to
make
some
changes
right
as
with
any
agile,
scrum,
DevOps
transition.
A
It's
all
about
the
learning
right
fail,
learn,
iterate
fail,
learn
iterate.
So
what
did
we
learn?
I
heard
somebody
one
of
the
other
speakers
earlier
talked
about
guardrails.
This
was
something
that
we
we
felt
pretty
strongly
right
from
the
beginning
when
we
dug
into
our
DevOps
journey
guardrails,
not
gates
right,
but
what
we
learned
was
when
we
were
lifted
the
gates.
The
guardrails
were
pretty
hard
to
build.
A
They
weren't
as
simple
they
weren't
so
simple
to
just
it
wasn't
as
simple
to
just
say:
oh
well,
let's
build
a
tool
and
it's
gonna
define
what
everybody
wants
to
do.
These
are
nuanced
things
and
we
didn't
invest
enough
in
it
early
on,
and
so
that
was
one
of
our
learnings.
We
needed
to
invest
harder
on
making
sure
that
those
tools
were
in
place.
We
also
learned
that
autonomy
tends
to
be
the
enemy
of
alignment,
especially
at
scale.
A
Ultimately,
that
means
what
we
wanted
to
do
was
give
them
the
freedom
to
innovate,
but
not
the
freedom
to
reinvent.
So
when
you
think
about
I
think
about
what
the
mission
of
a
given
team
is
they're
they're.
Ultimately,
what
you
want
them
to
do
is
to
stay
in
their
Lane.
They
give
them
a
mission,
give
them
clear
responsibilities
within
what
they
want
to
do,
and
you
want
them
to
have
lots
of
freedom
to
operate
to
solve
that
mission.
A
But
you
don't
necessarily
want
them
to
be
distracted
by
something
by
something
else
right,
and
we
got
this
feedback
from
our
development
teams.
They
really
they'd
get
to
where
they'd
spin
their
wheels
on
how
to
how
to
stand
up
an
auto
scaling
group-
or
you
know,
load
balancers
in
AWS
or
whatever
it
was
because
it
really
wasn't
their
core
their
core
skill
set.
So
we
learned
that
we
needed
to
give
them
the
opportunity
to
innovate,
but
not
to
reinvent
and
summit,
and
some
of
that
comes
out
of
building
good
obstacles.
A
A
They
actually
don't
need
as
much
access
as
they
used
to
either
so
at
ticketmaster
we
went
down
the
path
of
using
core
OS
in
AWS
and
then
making
immutable
Lois's.
So
there's
no
SSH
access
in
our
AWS
space.
Actually,
because
we've
built
these
tools
out
to
limit
that
even
for
our
administrators,
they
it's
a
break
glass
model
to
get
into
the
base
of
us.
So
what
is
this?
How
does
this
tie
back
to
kubernetes?
A
Well,
kubernetes
is
basically
all
of
those
things
right.
First
off
we
knew
from.
We
knew
early
on.
We
needed
to
do
container
orchestration,
but
we
we
we
sort
of,
did
a
stepping
stone
model
right.
We
we
gave
people
the
tools
to
run
containers
inside
of
a
VM
early
on,
because
we
didn't
want
to
over
complicate
the
process,
but
we
we
had
an
absolute
evolution
plan
to
get
to
orchestration
and
we
knew
we
needed
it.
A
Kubernetes
was
there
as
a
competitor
with
swarm
and
various
other
tools,
and
we
wanted
to
give
it
some
time
to
see
what
was
going
to
grow.
This
abstraction
of
primitives
is
a
is
all
about.
Staying
in
your
lane,
teams
want
that
ability
to
ask
for
a
resource
and
infrastructure,
but
they
don't
want
to
have
to
be
specific
about
the
way
that
they
get
it.
They
just
want
a
resource,
and
that's
really.
What
kubernetes
is
all
about
is
defining
those
abstractions.
A
A
We
had
gotten
to
the
place
where
all
of
the
infrastructure
operations,
especially
in
AWS,
was
back
on
the
developer
team,
and
then
they
were
getting
totally
consumed
by
that
operations
task,
even
if
it
didn't
have
much
to
do
with
running
the
application
I
mean
the
kubernetes
is:
does
a
really
good
job
of
doing
that
separation
between
operators
and
bringing
the
infrastructure
back
to
a
to
a
core
administration
team?
Then
the
development
teams
get
to
focus
on
what
they're
actually
running
and
stay
in
their
Lane.
A
So
at
that
point
we
we
knew
it
was
kubernetes
for
us.
We,
we
made
a
strategic
partnership
with
core
OS,
because
tech
tonic
was
this
product
that
brought
enterprise
features
to
kubernetes
two
years
ago
that
we
didn't
that
you
just
didn't
see
out
in
the
open
source
right.
The
big
one
for
us
was
our
back.
We
need,
we
knew
we
needed
to
run
kubernetes
at
scale
and
and
we
needed
to
be
able
to
isolate
teams
from
each
other.
A
We
couldn't
do
that
without
our
back
and
tectonic
had
it
built
in
so
that
was
that
was
fantastic.
We
started
talking
to
these
guys
and
we
just
we
sort
of
fell
in
love
with
them
and
and
and
did
a
fantastic
partnership
with
them
over
the
last
couple
of
years
and
then
hired
and
and
recruited
the
internally,
a
core
team
that
has
really
done
some
amazing
work.
A
This
isn't
everybody,
it's
everybody,
I
could
fit
and
sort
of
the
core
of
who's
been
doing
some
major
contributions
back
into
the
into
the
community.
As
you
can
see,
you
know
we
did
a
lot
of
work
on
the
ale
being
us
ingress
controller.
We
did
some
quite
a
bit
of
work
in
the
in
helm,
charts
prometheus
operator,
external
DNS.
You
can
read,
but
this
this
has
been
something
that
ultimately,
we
hired
a
team.
A
Ultimately,
what
I
care
about
is
the
Ticket
Master
mission
right
and
how
how
take
a
master
is
contributing
to
that.
So
what
are
we
doing?
I
wanted
to
make
this
sort
of
consumable
by
you
guys.
We've
done.
You
know
we're
running
kubernetes
in
a
lot
of
different
places,
but
if
you're,
how
many
people
have
bought
a
ticket
from
Ticketmaster
in
the
last
year,
yeah
a
lot
of
hands
raised
right,
so
you've
probably
seen
this
page.
A
This
is
how
you
select
your
seat
and
your
and
your
your
price
level,
where
you
want
to
be
all
of
that
stuff.
This
this,
of
course,
isn't
run
by
a
single
service
is
run
by
many
services
in
the
background,
but
ultimately,
every
time
you
buy
a
ticket
you're
doing
it
you're
using
kubernetes
to
do
it.
So,
even
in
these
high
volume
sales
modes,
like
a
tailor
or
a
Garth,
then
you're
you're
consuming
kubernetes
to
buy
those
tickets
and
it's
similar
in
the
in
the
model
and
how
you
get
in
the
door.
A
A
20:19
we're
just
we're
all
about
reap
lat.
Forming
kubernetes
is
going
to
be
a
major
portion
of
where
we're
going
so
we're
we're
driving
much
of
what
we
do
onto
onto
kubernetes
platforms
and
onto
new
cloud
platforms.
A
lot
of
work
ahead
of
us
and
I
see
the
adoption
growing
exponentially
so
that
really
that's
really
yeah.
A
If
you
have,
if
you
have
a
great
interest
in
figuring
out
how
to
get
your
own
son
or
daughter
to
their
first
concert
or
your
spouse
to
the
Paul
Simon
concert
or
this
new
clean
tour,
that's
going
around
I'm
totally
excited
about
that
for
next
year
and
you're
in
figuring
out
ways
to
make
that
happen
faster
and
better
with
kubernetes
come
and
talk
to
me
or
the
team,
we're
absolutely
hiring
and
we
need
help.
So
thanks
a
lot.