►
Description
Reworking Your Team for Kubernetes [B] - Hans Kieserman, Yahoo
Launching a product on Kubernetes involves a lot more than just “containerization”. It’s an interesting challenge to train team members for expertise in networking, monitoring, debugging, and reworking legacy apps to perform better under orchestration. Beyond the common issues of optimizing Docker containers, you’ll need a whole new understanding in order to create the right team, schedule, and ambitious but realistic adoption path.
We can help your efforts towards adoption by explaining common problems with getting up to speed. With several Kubernetes projects in the works across multiple teams, Yahoo has recent experience that can benefit any company.
About Hans Kieserman
Hans Kieserman is an infrastructure architect at Yahoo, currently working to build out as much Kubernetes as he can get away with. He has previously worked on payments infrastructure, web services, chaos engineering, and CI/CD systems. The power of open source initiatives are almost as fascinating as the culture that enables them to be successful in the enterprise.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Welcome
everyone.
Thank
you
for
joining
lucky
for
you
that
the
color
of
cube
con
is
blue
I'm
wearing
my
blue
shirt.
Instead
of
the
bright
purple
that
we
encouraged
at
Yahoo.
I
want
to
talk
a
little
bit
in
general
about
a
an
effort
to
migrate
to
kubernetes
that
we've
undergone.
We
have
been
working
on
a
lot
of
different
solutions
for
a
long
time,
and
hopefully
some
of
the
lessons
here
can
be
useful
to
all
of
you.
A
Yahoo
is
a
huge
driver
of
traffic
on
the
web,
with
billions
of
pages
service
and
millions
of
users.
Every
day
we've
had
a
long
history
in
internet
years,
which
means
lots
of
different
engineering
teams,
diverse
workloads,
lots
of
acquisitions
and
lots
and
lots
of
different
legacy
systems
that
are
all
being
updated
over
time.
This
can
get
really
really
expensive.
We're
looking
for
optimizations
that
may
be
large
or
small
to
increase
utilization
and
decrease
application.
Maintenance
and
kubernetes
is
great
candidate
for
the
next
generation
of
our
infrastructure.
A
We
have
several
teams
using
public
clouds.
We
also
have
a
large
internal
cloud
which
is
running
hundreds
of
thousands
of
requests
per
second
across
many
products
and
over
the
years
we've
experimented
with
various
orchestration
frameworks.
We
were
advised
to
not
make
jokes
about
the
different
options,
so
I
won't
do
any
of
that
by
the
including
several
homegrown
ones,
with
varying
levels
of
success,
but
most
of
our
traffic
still
goes
to
directly
allocated
hardware
in
data
centers
and
I.
A
Don't
know
if
anyone
here
is
using
bare
metal,
but
it's
a
big
deal
and
a
lot
different
than
many
of
the
automated
setup
that
you
saw
this
morning
for
for
Google
Cloud,
for
example,
there's
no
one
click
deploy
a
few
percentage
points.
Difference
in
utilization
can
lead
to
millions
of
dollars
of
savings.
So
this
is
something
we're
really
excited
about,
and
a
few
percent
improvement
and
usability
can
lead
to
millions
of
whatever
happiness
is
measured
in
units.
A
My
name
is
Hans
kottmann
orky
thurman
in
Germany
I'm,
an
architect
for
the
engineering
services
team.
At
Yahoo.
We
build
tools
that
are
used
across
the
company.
We
spent
many
years
building
really
powerful
systems
to
get
the
job
done.
There
were
many
many
custom
internal
products,
and
this
makes
the
migration
even
more
challenging
they're
even
used
to
be
an
enterprise
guide.
That
said
that
just
don't
pull
the
latest
open-source
thing
off
the
shelf.
Linux
is
an
exception.
Look
at
all
these
startups
that
had
to
be
bought
out
to
actually
scale
and
I.
A
Think
that's
really
really
changing
now:
two
things
that
have
actually
been
around
for
a
long
time
as
I'm
sure
you're,
aware
elastic
cloud
infrastructure
and
also
large
organizations
contributing
to
open
source
Yahoo
has
done
a
lot
of
open
source
work
in
the
great
computing
area.
Hadoop
Apache
serving
stacks
nodejs
community
software
build
infrastructure
and
more,
but
we
have
not
done
so
much
in
the
cloud
world,
and
this
is
one
story
about
trying
to
make
that
transition.
A
So,
whenever
you're,
looking
at
a
new
transition
like
this,
there
are
a
few
steps
that
you
should
really
follow,
and
this
was
successful
for
us
really
questioned
everything
open
it
up,
figure
out
what
the
inflection
point
will
be
for
making
the
change
where
you
need
to
invest.
First,
second,
build
out
your
team
to
respond
to
those
challenges
and
third
tune
for
the
real-world
use
cases.
A
So
you'll
hear
a
lot
of
talks
at
kubernetes
over
the
next
few
days
today
and
tomorrow
to
understand
all
the
specific
features
that
are
available
in
kubernetes
you'll
hear
a
little
bit
less
about
the
pain
point,
so
hopefully
I
don't
sound
like
I'm
complaining
for
a
half
an
hour
here,
but
there
are
a
lot
of
other
issues
that
we
need
to
think
through
in
addition
to
how
to
deploy
after
you
deploy.
How
do
you
route
after
you
route?
How
do
you
make
sure
that
things
are
secure.
A
A
So,
as
I
mentioned,
my
goal
here
is
to
give
you
a
sense
for
what
the
real
world
problems
are
beyond
the
how-to
documentation.
You
can
make
more
informed
decisions
about
your
own
choices
and
will
have
to
answer
these
separately
for
every
organization.
I,
think
all
of
the
answers
here
are
positive
for
almost
everyone
these
days
and
it's
getting
better
every
day.
A
Kubernetes
is
very
easy
to
run
scripts
on
it's
very
easy
to
get
running
in
five
minutes,
but
that
doesn't
mean
it's
ready
for
production.
If
you
have
a
small
deployment,
you
can
often
get
away
with
running
on
a
public
cloud
and
they're
great
vendors
here,
who've
been
contributing
a
lot
to
the
community.
There's
consulting
from
organizations
at
the
booths
outside.
Don't
go
the
bare
metal
route
with
being
fully
without
being
fully
aware
of
what
you're
getting
into
it
can
be
very
successful,
but
it
requires
care
and
time.
A
Also.
The
implementation
path
is
a
critical
choice,
but
it's
not
an
irreversible
one.
A
lot
of
organizations
are
looking
at
kubernetes
as
a
portability
factor,
meaning
you
can
launch
on
AWS
kubernetes,
for
example,
and
if
it
gets
too
expensive,
you
could
switch
cloud
providers
or
you
can
bring
it
in
each
of
those
transitions
is
not
cookie
cutter,
but
it
is
relatively
easy.
A
We've
done
some
tests
also
showing
that
there's
no
loss
of
performance
in
multiple
cases,
for
example
running
on
a
cloud
provider
using
the
native
cloud
infrastructure
versus
running
kubernetes
on
the
cloud
provider
itself.
In
our
case,
we
found
that
the
networking
customization
that
we
had
to
do
in
each
case
made
it
not
specifically
worth
running
in
multiple
situations
that
we're
going
to
continue
on
premises
for
now,
but
the
door
is
always
open
beyond
the
organizational
challenges.
There's
a
lot
of
technical
questions
and
I
believe
these
are
some
of
the
most
critical
for
kubernetes.
A
There
are
also
questions
that
are
not
something
that
kubernetes
can
obviously
solve
directly.
There's
been
a
lot
of
talk
of
composability,
small
components
and
a
modular
functionality
from
kubernetes
itself
that
is
not
solved
does
not
change
how
fast
your
application
run.
For
example,
depending
on
your
uptime
requirements,
you
might
need
to
tune
heavily
to
get
the
performance
that
you
need
kubernetes
by
default
with
many
of
the
common
scripts
comes
with
an
extra
couple
of
hops
to
get
to
your
serving
containers.
A
If
your
app
takes
a
long
time
to
start,
you
really
need
to
worry
about
the
disposability
of
pods
and
implement
health
checks
very
carefully.
If
the
health
check
occasionally
returns
a
404,
you
may
be
killing
the
pod,
which
gets
very
very
expensive.
If
it's
a
long
restart
time,
we
had
an
example
where
a
legacy
package
manager
was
touching
a
lot
of
files,
and
so
when
you
would
try
to
start
many
pods
on
the
same
box,
you'd
end
up
with
lock
conflicts.
We
extend
experimented
a
lot
with
read-only
volumes.
We
tuned
the
application
itself.
A
We
tried
hacking
on
the
package
manager
and
ended
up
that
with
SSDs
to
get
over
the
critical
threshold
for
that
particular
application
as
another
example,
they're
very
successful
startups
out
there
that
are
running
BSE
and
obviously
this
is
not
for
you.
We
had
some
hosts
running
old
versions
of
Red
Hat
and
the
OS
is
incredibly
important.
We
we
spent
a
lot
of
time
catching
up.
There
is
work
to
do
there.
We
can't
be
infinitely
compatible.
It's
also
a
good
idea
to
stay
up
to
date.
A
As
another
example,
we
have
custom
data
centers
where
we
run
all
of
our
software
and
we
couldn't
easily
support
large
numbers
of
pods
because
of
limits
in
the
routing
tables
on
the
switches.
So
it's
little
things
like
this
that
turned
into
big
things.
You
can't
just
deploy
kubernetes
itself.
We
have
to
worry
about
how
we
get
enough
capacity
in
the
existing
systems
to
deploy
them
and
how
to
change
configurations
so
that
they
function
well.
A
I'm
sure,
because
your
coupon
you've
heard
a
lot,
you
understand
something
about
exactly
why
we
use
kubernetes,
but
graphs
like
this
show
us
the
motivation
for
them.
You
can
significantly
decrease
your
overall
capital
expenditure
and
operating
expenses
by
just
increasing
your
utilization.
A
little
bit
like
I
mentioned,
we've
seen
reductions
as
much
as
6
to
1
in
the
numbers
of
hosts
that
we've
been
able
to
run
exactly
the
same
workload
at
exactly
the
same
performance
just
by
migrating
on
to
the
kerbin
energies
infrastructure,
and
that's
for
a
few
reasons.
A
First
of
all,
sharing
compute
resources
with
multiple
applications
obviously
helps
a
lot,
especially
if
load
on
your
applications
is
not
tightly
correlated.
You
need
to
be
very
worried
about
whether
everything
is
going
to
spike
at
the
same
time
or
whether
you
have
distributed
workloads
that
can
be
allocated
together.
A
Everything
else
can
also
be
shared,
so
excess
hosts,
as
you
mentioned
here,
that
you
might
be
using
for
business
continuity
planning
for
reserve
that
may
have
very
low
utilization
in
the
normal
case.
Instead
of
having
many
separate
pools,
one
for
each
application
that
are
all
pre
deployed
and
just
waiting
now,
you
can
have
those
shared
waiting
in
the
wings
with
a
much
lower
number
of
total
Hardware.
A
Also
sharing
is
valuable,
but
it's
not
entirely
required.
So
with
1.6,
the
affinity
and
anti
affinity
have
gone
into
beta.
We
saw
some
demos
earlier
in
the
keynotes
that
showed
how
to
allocate
specific
workloads
to
specific
pools
of
machines
as
necessary,
and
the
auto
scaling
features
really
helped
with
that
in
a
way
that
you
can
have
critical
workloads,
for
example,
on
the
Yahoo
homepage.
A
So
there
are
a
few
other
things
beyond
just
performance.
There
are
a
lot
of
features
that
kubernetes
provides
that
allow
dynamic,
automated
management
of
large
production
clusters
and
great
tooling,
that
can
help
reduce
the
overhead
for
your
teams.
It's
this
wonderful
dichotomy
where
it
requires
some
expertise
to
run,
but
even
if
you're,
not
an
expert,
you
can
get
a
lot
out
of
the
box.
There
are
powerful
features
to
accomplish
great
things.
A
It
encourages
you
to
learn
and
it
points
at
what's
really
important
so
that
you
can
worry
about
innovating
on
your
own
software,
unified
catalog
of
infrastructure.
Here,
what
I
mean
by
that
is
that
there
are
side
benefits
that
you
might
not
think
about.
In
addition
to
smoothly
operating
the
software
that
you
have,
you
can
also
have
all
of
your
software
in
one
place.
The
API
server
allows
you,
for
example,
as
opene
I
mentioned,
to
inspect
everything
that's
running
and
build
dashboards
that
can
show
you
the
entire
list
of
dependent
operating
systems
for
security
scans.
A
A
So
the
second
thing
you
want
to
do
after
trying
out
the
demo
is
to
work
out
the
scope
of
your
kubernetes
goal.
It's
a
great
reason
to
clean
up
your
house.
So
if
you
can
launch
one
or
more
kubernetes
instances
with
the
right
safeguards,
then
you
have
this
blueprint
that
lets
you
deploy
new
applications
quickly,
based
on
the
current
state
of
your
infrastructure.
What
can
you
reasonably
expect
to
achieve
common
sense
is
that
stateless
micro
services
will
be
fast
and
easy.
A
So
rule
number
two
build
the
team
we
found.
The
kubernetes
is
easy
enough
for
either
developers
or
production
engineers
to
operate.
It
does
take
special
knowledge
to
really
do
a
good
job
specifically
in
these
three
areas
of
foundational
networking,
containerization
and
monitoring
and
logging
have
been
challenges
for
us.
A
So
does
your
team
understand
even
outside
of
kubernetes
what
l3
and
L
seven
are
and
how
many
hops
it
takes
to
get
through
a
dns
request?
Can
you
efficiently
process
large
volumes
of
logs?
Do
you
have
Splunk
or
fluent
already
installed
in
your
infrastructure?
It's
not
enough
just
to
read
the
documentation.
We
found
that
hands-on
practice.
Local
experts
and
lots
of
testing
like
anything,
makes
the
biggest
difference.
So
here
are
three
best
practices
that
I
would
really
encourage.
A
First,
it
can
be
tempting
to
take
your
team
and
encourage
them
to
all
go
read
everything
they
can
find.
This
is
not
efficient.
It
certainly
works,
but
finding
local
experts
delegating
individual
components
of
the
system
to
one
person
and
then
having
them
roll
that
knowledge
out
across
the
team
is
the
best
kubernetes.
The
hard
way
is
still
the
canonical
source.
Kelsie
Hightower
is
a
national
treasure.
A
A
A
We
initially
launched
with
a
1%
bucket
for
a
tiny
little
app
and
then
expanded
as
we
went
further
and
that
1%
makes
all
the
difference,
there's
a
huge
difference
between
being
100%
ready
with
zero
production
or
being
50%
ready
with
1%.
That's
giving
you
data
it's
surprising
to
me
how
often
you
see
questions
on
the
kubernetes
slacks
that
say
my
cluster
doesn't
work
and
you
say:
okay,
go,
run
general
control
and
tell
me
what
the
logs
say.
A
We
also
had
a
challenge
to
gather
support
from
management
in
order
to
build
the
best
possible
investment
case.
We
got
a
lot
of
data.
We
ran
jmeter
tests
that
validated
many
of
the
performance
claims
of
the
kubernetes
system
and
have
seen,
for
example,
odds
coming
up
in
seconds
with
thousands
and
thousands
of
pods
on
one
machine.
A
A
There's
also
constant
discussions
going
on
in
any
large
organization
about
whether
we
should
be
building
more
on-prem,
which
cloud
provider
we
should
be
choosing,
and
this
kind
of
ongoing
data
contributes
a
lot
to
that.
A
few
bumps
that
we
hit
in
the
road
I
mentioned
our
package
manager
gave
us
some
trouble
early
on
a
lot
of
the
application
level.
Details
and
containerization
really
caused
the
main
kinds
of
challenges
that
we've
experienced
versus
kubernetes
itself.
A
It's
important
to
set
up
defaults
in
your
system
that
don't
cause
disasters
that
something
goes
wrong.
If
you
run
delete
commands
manually
using
queue
control,
you
can
delete
objects
in
production.
If
someone
doesn't
have
their
configuration
set
correctly,
guess
how
we
discovered
that
if
you
haven't
already
set
up,
deploy,
automation
and
trained
yourself
to
always
always
use
it,
you
need
to
do
that.
A
First,
our
clusters
themselves
or
managed
using
chefs
and
then
applications
are
built
with
an
automated
production
pipeline
a
little
bit
like
Heroku
that
calls
to
control
to
deploy,
there's
also
helm,
for
example,
as
a
package
manager
which
helps
a
lot
with
that,
and
then
access
to
the
production
clusters
themselves
to
do
considered
harmful
is
problematic
for
the
same
kind
of
reason.
You
need
alternate
methods
for
debugging
and
maintenance,
with
kubernetes
queue,
control,
exec
and
describe
are
really
fantastic
and
then
tools
like
keeps
your
systick
Splunk.
A
You
should
remember
those
names
and
try
the
Nile,
okay,
so
rule
number
three
tune
for
the
real
world.
As
you
might
be
aware,
yahoo
has
had
some
security
challenges
in
the
past.
It's
really
we've
always
taken
it
seriously.
There
have
clearer
than
some
problems
that
are
being
solved,
but
also
it's
lit
a
fire
under
everyone
to
really
take
security
absolutely
seriously.
This
is
our
higher
than
number-one.
Priority
attaches
product
development
and
infrastructure,
so
this
morning
Clayton
mentioned
that
security
of
boring
and
and
depressing
and
I
think
that's
absolutely
the
opposite
of
what
it
is.
A
This
is
cloak-and-dagger,
spy
kind
of
things
you
can
think
of
yourself
as
a
secret
agent,
and
also
it
tends
to
be
something
really
interesting
about
the
infrastructure
that
you
didn't
understand
before.
There's
been
a
lot
of
discussion
lately
about
docker
networking
and
how
to
really
make
that
secure
by
default,
using
a
bridge
network
keeps
you
open
to
our
spoofing,
for
example,
in
certain
cases.
A
A
So
these
are
high-level
steps
in
our
pipelines
that
help
us
ensure
security
I'm
not
going
to
talk
to
every
single
one.
But
hopefully
recent
software
is
is
obvious:
we'd
love
to
get
to
unit
Carnales
to
minimize
exposure,
but
that's
still
a
long
way
away.
Deploy
automation,
maybe
a
little
bit
less
obvious,
but
it
does
encourage
good
practices.
It
allows
frequent
office
software
updates.
It
allows
people
who
are
not
on
the
team
necessarily
to
get
involved
in
security.
A
A
Image
scanning
I'm
surprised
how
few
teams
I've
talked
to
do
this.
There's
a
product
called
Claire
from
core
OS,
which
makes
it
easy.
You
really
want
to
know
where
your
images
are
coming
from
just
pulling
something
off
of
a
public
registry,
isn't
necessarily
going
to
be
any
good.
You
have
no
idea
what's
in
it
and
you
can
scan
for
CVS
using
a
bunch
of
of
available
tools.
A
Cluster
security
means
a
lot
more
than
just
odd
security
policies,
so
you
should
book
of
POD
security
policies
and
kubernetes.
That's
how
pods
interact
with
each
other,
but
also
think
carefully
about
your
networking
model.
It
also
means
that
some
of
the
kubernetes
components
are
not
secured
by
default,
and
that
was
mentioned
this
morning
as
well.
We
don't
want
to
break
existing
infrastructure
or
existing
deploys,
but
network
tackles,
like
IP
tables,
to
prevent
systems
from
talking
to
each
other
when
they
shouldn't
be,
is
very
important.
A
We've
locked
down
our
sed
servers,
so
only
the
master
can
talk
to
them.
Kubernetes
secrets,
you
may
or
may
not
know,
are
stored
in
plain
text.
So
if
you
want
to
do
anything
with
them,
it's
a
good
idea
to
add
an
extra
level
of
encryption
on
top
there's
a
kubernetes
dashboard
that
comes
by
default
with
the
system.
Although
it
allows
everyone
to
see
everything,
that's
going
on
in
the
cluster
and
depending
on
the
state
of
your
system,
you
may
not
want
that
and
then
basic
security
practices
running
SSL
everywhere,
for
example,
should
be
followed.
A
A
A
It
adds
a
lot
of
confidence
in
the
process,
okay
quickly
back
to
the
original
image.
So
these
are
the
components
that
we
have
added
in
our
system
and
I
couldn't
show
every
configuration
option
that
we've
added
and
that's
the
point
I
want
to
make
on
purpose
that
there
are
so
many
things
you
can
do
just
with
editing,
EML
files
and
then
obviously
checking
them
in
to
get
and
deploying
them
under
automation.
But
most
of
the
work
we've
done
like
quotas,
admission
controllers,
labeling
quads
for
allocation-
are
just
tuning
common
parameters
in
the
kubernetes
ecosystem.
A
We
have
the
luxury
of
our
own
data
centers,
which
may
customization
a
lot
easier.
One
IP
per
pod
means
lots
and
lots
of
routes
and
they're
changing
very
frequently
so
a
patchy
traffic
server
is
a
product,
that's
public
that
performs
really
well.
We
created
a
plug-in
that
will
watch
the
kubernetes
api
for
changes
and
update
routing
to
specific
ICS
as
a
load
balancer
to
send
traffic
directly
to
pause,
whether
than
going
through
intermediate
steps
and
then
there's
a
custom
ingress
provided
in
carbon
Eddy's
that
we
add
metadata
to
to
add
new
schemes.
A
Finally,
just
to
mention
the
authentication
tool
is
called
Athens
and
it's
public
on
github.
It
allows
you
to
encapsulate
user
management
externally
to
the
cluster,
so
our
back
is
coming
out
this
overlaps
with
some
of
that
functionality,
but
we
feel
that
there's
a
lot
of
value
to
have
authorization
and
authentication
in
one
system
and
to
have
it
external
to
the
clusters.
I
personally
feel
that
it
means
it's
easy
to
manage
cluster
access
and
namespaces
without
getting
involved
in
the
deploy
and
across
any
possible
installation.
A
There
are
three
essential
buckets
to
what
we're
doing
with
kubernetes
right
now.
One
of
them
is
our
main
serving
infrastructure.
One
of
them
is
feed
processing
and
the
last
one
is
a
build
system
that
we
use
internally.
That
was
recently
open
sourced.
We
still
have
a
very
small
rollout,
I
have
to
admit
there
are
thousands
of
RPS
coming
through
the
system
and
hundreds
of
thousands
of
more
that
we
are
working
on.
This
is
something
that
is
very
new
to
us
as
I'm
sure.
A
It's
me
to
to
many
of
you,
and
it's
been
successful
so
far,
so
we'll
see
how
it
goes
as
we
get
bigger.
You
should
also
check
out
screw
driver
CD
for
other
projects,
that's
something
that
we
use
successfully
and
can
improve
anyone's
infrastructure,
I'd
like
to
close
out
with
a
quick
list
for
kubernetes
based
on
my
own
personal
needs,
and
hopefully
all
of
you
can
just
deliver
that
for
me.
Thank
you
very
much.
A
We're
working
on
contributing
back
one
of
the
best
parts
of
the
ecosystem
is
the
incredibly
welcoming
community,
but
these
stuck
out
as
particularly
near-term
efforts.
That
would
make
the
experience
better.
That
we'd,
like
to
help
improve
first,
is
a
deeper
analysis
of
the
products
from
security
perspectives.
This
is
something
Clayton
also
mentioned
I'm
happy
to
hear
that
it's
a
community
effort,
siga
and
sig
networking
both
get
involved
in
security,
there's
a
lot
more
to
the
entire
infrastructure
that
needs
to
be
considered
for
really
hardened
systems.
A
There's
a
lot
of
knobs
that
you
could
possibly
get
wrong
and
we'd
love
to
have
an
easy
menu
of
recommendations
for
making
that
better.
That
means
cloud,
security
policies,
resource
quotas,
image
policies
and
so
on,
1.6
I
just
noticed,
actually
will
not
match
unauthenticated
users
anymore
with
the
star.
You
have
to
explicitly
allow
them
and
I.
Think
that's
great.
A
great
change.
Bluegreen
deploys
are
something
that
we
use
heavily.
We
like
a
lot.
It
means
that
you
can
push
an
entire
system
to
production
and
test
it
before
end-users
see
it.
A
This
is
something
that's
tightly
integrated
with
networking,
so
there
are
reasons
why
it's
not
part
of
kubernetes
yet,
and
in
fact
several
vendors
MDOT
to
openshift
can
provide
some
of
these
features.
I'd
love
to
be
able
to
see
this
in
more
open
source
products
as
well,
and
then
there
are
so
many
networking
tutorials.
It
makes
your
head
spin,
so
I
think
they're
all
great
for
their
own
reasons.
A
It
takes
a
lot
to
get
through
it
and
so
working
out,
which
ones
are
the
best
and
how
to
make
use
of
them
would
really
be
helpful,
especially
for
a
lot
of
developers.
Without
the
operational
background,
we
were
talking
around
the
water
cooler
the
other
day
and
thinking
that
we
need.
What
we
really
need
is
auto
scaling
for
engineers.
If
you
could
set
your
image
to
be,
you
know,
Kelsey,
flash,
Hightower
and
then
max
nodes,
40
or
something
that
would
really
make
a
big
difference
to
our
productivity,
I'm,
not
sure
that's
coming
anytime
soon.