►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Keynote: Kubernetes: Lessons from the Trenches – Arun Gupta
AWS launched Amazon EKS over a year ago, and it is now one of the fastest growing compute service in the AWS cloud. We will share the lessons learned running thousands of production clusters on behalf of our biggest accounts. We will talk about how we support enterprise scale clusters without compromising blast-radius, MTTR and MTBF, and testability. We have security practices, networking models, and permissions integrations that didn't exist a year ago. Building native -interfaces for tasks like log data analysis, in-place version upgrades, and cluster provisioning were large efforts requiring the technologists throughout the AWS team to gain expertise in Kubernetes. The attendees would learn about The Good, The Bad and The Ugly parts of building and managing Kubernetes clusters at scale at AWS.
https://sched.co/Nxxf
A
Thank
You
Vicki
I'm
very
excited
to
be
here
and
back
in
Shanghai.
My
name
is
Arun
Gupta
I'm,
a
principal
open
source,
technologist
working
the
open
source
team
at
Amazon,
Amazon
eks
is
our
managed
kubernetes
service
and
it
was
launched
over
a
year
ago.
It
is
one
of
the
fastest
growing
service
at
AWS.
In
this
talk,
I
will
share
some
lessons
for
running
tens
of
thousands
of
production
clusters
for
our
customers.
A
Amazon
has
14
leadership
principles
which
drives
what
the
company
does
and
customer
obsession
is
one
of
the
most
important
ones.
Security
is
job,
0
@
IW
s
and
then
operational
performance
followed
very
closely
by
it.
So
today
I
will
look
at
it.
You
know
kubernetes
from
those
lenses
as
we
were
operating
Amazon
eks
cluster.
How
did
we
see
through
customer
obsession,
security
and
operational
stability?
A
Everything
at
Amazon
that
we
do
is
starts
with
customers.
We
are
not
competitor,
driven
we're
not.
We
are
customer
obsessed.
We
look
at
the
customer
what
they
want,
and
then
we
start
from
there
and
we
work
backwards
from
there.
Roughly
90
to
95%
of
our
roadmap
is
driven
by
what
customer
want.
We
talk
to
customers,
we
understand
their
needs,
we
obsess
over
it
and
then
that's
how
we
define
what
our
roadmap
is
going
to
look
like
and
then
about
5
to
10
percent
is
about
the
strategy
interpretations
of
what
the
customers
would
really
want.
A
A
classic
example
of
what
the
customer
want
from
us
is.
They
said
you
know,
kubernetes
is
cool,
we
love
it,
but
running
of
kubernetes
cluster
is
hard
and
for
we
listen
to
our
customers
and
that's
where
we
launched
Amazon
eks
an
example
for
five
to
ten
percent
is
AWS
lambda,
which
is
where
we
thought
that
the
customers
would
like
this,
and
we
launched
that
over
four
years
ago
now.
A
A
So
let's
take
a
look
at
it,
as
we
were
talking
to
our
customers
in
our
journey.
What
were
there
asked
and
how
does
Amazon
eks
kind
of
map
to
it?
Essentially,
now
customers
wanted
to
leverage
everything
they
have
learnt
using
kubernetes.
Where
are
they
on
their
desktop
on
Prem,
wherever
they
are
running
it?
They
want
to
bring
that
expertise
but
on
to
the
cloud,
and
they
also
want
to
make
sure
that
this
is
done
in
100%
conformant
way
where
there
are
no
folks,
no
branches.
A
You
know
everything
is
done
in
upstream
compatible
way,
so
they
know
they
can
use
kubernetes
for
what
it
is
best
for
as
a
platform
for
platforms,
so
amazon
EK
as
one
of
the
core
tenets
is,
is
upstream
compatible.
There
are
no
internal
folks
or
branches
at
all
at
amazon
and
it
is
a
certified
kubernetes.
So
what
that
means
is
whatever
kubernetes
you
are
running
on.
Your
desktop
on
your
on-prem
on
in
the
cloud
is
all
the
same
version,
and
that
is
very
important.
A
So
what
that
means
is
your
existing
investment
works
out
of
the
box
with
Amazon
eks,
as
I
said,
open
source
contribution
is
a
core
tenet
for
Amazon
eks.
You
know
some
of
the
core
components
of
Amazon
eks
are
out
in
the
open
source
as
well.
Whether
it
is
the
cni
plugin
weather
is
the
Authenticator
on
how
we
authenticate
a
kubernetes
cluster
or
is
the
eks
army,
which
is
how
you
create
your
data
nodes
or
bring
your
data
plane
and
all
the
Packer
scripts
are
available.
A
So
some
of
our
partners
have
created
custom
armies
using
those
packer
scripts,
because
this
is
our
out
in
the
open
source.
We
also
contribute
to
a
wide
range
of
open
source
projects,
and
you
can
look
at
some
of
those
one
of
the
ones
that
I
want
to
call
out
specifically
is
eks
cuttle,
which
is
done
by
one
of
our
partners.
It's
a
very
easy
way
to
get
started
with
creating
your
Amazon
eks
cluster,
literally
eks
Caudill
create
cluster,
and
you
got
a
cluster
up
and
running.
A
Another
Astra
my
customers
was
control.
Plane
is
undifferentiated.
Heavy
lifting
I
want
to
run
a
kubernetes
cluster
up
in
the
cloud,
but
then
I,
don't
know
what
ec2
instance
is
to
use.
Should
my
EC,
my
XE
d
be
co-located
with
the
master,
should
it
be
secured?
How
should
it
be
secured
how
it
should
be
backed
up
what,
if
my
number
of
pods
on
my
control
plane
goes
high?
How
do
you
scale
that?
How
do
you
patch
that?
A
So
that's
a
very
common
ask
from
the
customer,
so
our
very
first
need
was
very
evidently
clear
that
all
we
want
to
do
is
provide
a
managed
control
plane
to
begin
with.
So
if
you
look
at
Amazon
eks,
it's
a
fully
managed
highly
available
control,
plane,
HCD
is
secured
and
backed
up
and
it
scales
based
upon
your
traffic.
A
Another
key
ask
from
our
customers
was
that,
as
you
are
taking
kubernetes
and
deploying
it
at
scale,
make
sure
you
test
it.
It's
very
important,
because
that's
what
kind
of
gives
you
that
conformance
feeling,
essentially
so
we
run
full
set
of
tests
now,
whether
it's
end-to-end
test
upgrade
tests,
and
now
we
launch
GPU
support
a
few
months
ago,
like
late
last
or
early
last
year,
and
all
the
performance
tests.
A
So
we
do
run
high
scalability
tests
as
part
of
each
eks
release
so
as
to
make
sure
that
your
workloads
are
very
well
supported
on
Amazon
eks
and
then
the
last
ask
really
was
that
kubernetes
is
cool.
I
love,
the
innovation
that
is
happening
in
the
community.
Give
me
that
integration,
if
I'm
an
AWS
customer,
give
me
the
seamless
integration
with
the
AWS
technologies,
but
don't
make
it
compulsory
for
me.
So
what
I've
listed
over
here
on
the
right
side
is
some
of
the
integrations
are
available.
So
things
like
you
know.
A
If
you
are
running
control,
plane
and
data
plane,
then
all
those
logs
go
to
cloud
watch.
If
you
are
doing
auditing,
it
goes
to
cloud
trail.
If
you
are
doing
authentication,
then
we
are
using
I
am,
which
is
how
you
do
security
on
AWS
platform.
So
all
of
those
integrations
are
available
to
you
out
of
the
box,
and
there
is
no
heavy
lifting
that
you
need
to
do
as
part
of
that.
A
One
of
the
most
exciting
things
for
me,
particularly
is
container
steam
is
really
going
out
there.
They
are
really
talking
about
having
a
transparent
roadmap.
This
works
across
all
container
services.
So
what
I'm,
showing
you
know
I,
would
encourage
you
to
go
to
this
URL
and
take
a
look
at
the
transparent
roadmap.
This
particular
one
is
with
the
eks
filter.
Essentially,
so,
as
you
can
see
on
the
top
right,
we
have
kubernetes
1:13.
A
The
support
came
out
a
few
days
ago
and
I'm
super
excited
to
announce
that
Amazon
eks
is
coming
in
ning
xia
region
and
the
Hongkong
region
in
the
next
few
weeks,
and
if
you
have
more
features,
support
more
features
that
you
want
to
ask
for
us.
I
would
encourage
you
to
file
a
bug
over
here.
This
is
exactly
how
you
know
we
are
customer
upset.
We
listen
to
customers,
we
listen
to
your
needs
and
that's
what
defines
our
roadmap
on?
What
are
the
next
set
of
features
that
we're
going
to
work
on
now?
A
Let's
switch
gears.
Let's
talk
about
security
a
little
bit.
Security
and
compliance
is
a
shared
responsibility
between
AWS
and
the
customer.
What
that
means
is
security
of
the
cloud
is
AWS
responsibility,
but
security
in
the
cloud
is
customer
responsibility
right.
Let's
take
a
look
at
it,
a
little
bit
more
in
detail.
A
You
know
down
to
the
physical
security
of
the
facilities
where
the
data
center
is
running,
so
you
don't
have
to
worry
about
that,
but
then
customer,
which
is,
you
assumes
the
responsibility
and
management
of
the
guest
operating
system,
the
patching
and
everything
that
comes
along
as
part
of
that,
and
we
provide
a
lot
of
abstracts
and
application
concepts
for
you,
so
that
your
data
is
secured
at
rest
in
motion
and
a
lot
of
different
tools
that
are
available
for
you.
Now.
A
Let's
see
this,
how
this
translates
to
from
kubernetes
perspective
now,
if
you
think
from
AWS
perspective
or
Amazon
eks
perspective
control,
plane
is
fully
managed
so
that
the
fact
that
it
is
highly
available
the
fact
that
HCD
is
secured
and
backed
up
the
fact
that
they
are
not
co-located,
that's
what
gives
it
high
availability.
Well,
those
are
all
our
responsibilities.
A
On
top
of
that,
what
is
the
army
that
you
want
to
use
for
your
data
plane,
whether
you
want
to
use
the
Amazon
Linux
to
default
army,
or
you
want
to
bring
a
bun
to
army
that
is
totally
up
to
you
now
you
can
use
our
bad
policies
on
top
of
that
or
you
want
to
make
your
API
server
endpoint
private.
That
is
completely
up
to
you.
We
provide
tools
for
all
of
that.
What
I
am
roles
you
want
to
use
for
securing
and
authenticating
your
cluster?
A
Those
are
completely
your
responsibility,
so
we
see
in
that
sense
it's
a
shared
responsibility
between
you
and
us,
and
we
would
love
to
work
with
you
and
understanding.
More
and
more
of
this,
let's
talk
about
the
last
aspect
of
it,
which
is
sort
of
the
operational
stability.
Okay,
now
Amazon
eks
is
available
in
13
regions
around
the
world,
with
support
for
more
coming
at
Amazon
scale.
How
do
we
roll
out
support
for
kubernetes
to
a
particular
region
or
if
we
were
to
patch
a
particular
reason
for
a
newer
version
of
kubernetes?
A
How
do
we
do
that?
Well,
the
concept
of
region
is
AWS,
has
multiple
regions
around
the
world,
and
this
is
a
physical
location
around
the
world
where
data
centers
are
co-located.
Now
each
region
is
distributed
across
multiple
Easy's
or
availability
zones,
and
that
gives
you
the
ability
to
build
applications
that
are
resilient
now.
What
we
do
is
for
deployment
purposes.
If
you
look
at
it,
you
know
these
are
showing
two
regions,
and
each
region
has
multiple
availability
zones
now
for
deployment
purpose.
A
Each
region
or
other
eat
availability
zone
is
split
up
into
multiple
cells
and
each
cell.
Essentially,
the
capacity
of
a
cell
is
defined
by
the
number
of
clusters
that
can
be
created
in
that
cell.
Ok,
so
now,
when
we
are
doing
a
deployment,
a
major
deployment
of
kubernetes
say
a
new
version
or
updating
a
new
version,
then
we
deploy
to
a
cell.
We
run
our
health
checks,
make
sure
the
clusters
are
up
and
running
make
sure
all
our
health
checks
are
passing.
A
If
that
is
successful,
then
we
move
on
to
the
next
cell
and
then
next
cell
and
then
next
cell,
so
on
so
forth.
So
that
gives
us
you
know
there
are
a
lot
of
advantages
for
that
sort
of
an
architecture
and
I'll
talk
about
that
in
a
second,
but
essentially
once
one
availability
zone
and
all
the
cells
in
that
availability
zone
are
updated,
then
we
move
to
the
next
availability
zone
and
so
on
so
forth,
and
then
the
next
region,
so
that
kind
of
gives
you
the
ability
to
really
do
a
cell
based
architecture.
A
Essentially,
so
what
are
some
of
the
advantages
of
such
a
cell
based
architecture?
Well,
the
first
advantage
really
is:
it
gives
you
a
lower
blast
radius
and
at
Amazon.
We
always
think
in
terms
of
plaster
ideas,
basically
that
if
there
is
a
problem
happening
somewhere,
how
big
the
blast
radius
is
how
much
of
an
impact
is
going
to
cause
to
the
customer.
A
Now
breaking
up
a
service
into
multiple
cells
helps
reduce
the
blast
radius,
because
cells
essentially
represent
the
bulkhead
that
provide
containment
for
many
common
failure
scenarios.
So
if
a
failure
really
happens,
know
within
a
cell
and
is
contained
within
the
cell,
so
that
the
other
cells
are
not
impacted
now
when
properly
isolated
from
each
other.
This
is
the
way
the
cells
are
being
done.
Cells
have
the
failure,
commit
commitments
similar
to
regions.
A
It
also
gives
you
higher
testability,
because
the
cab
size
of
cells
is
well
understood
and
tested
for
maximal.
You
know
behavior
that
how
much
it
can
scale.
So
we
know
exactly
how
much
it
can
be
tested,
because
it's
not
changing
on
a
regular
basis.
As
I
said,
the
capacity
of
a
cell
is
defined
by
the
number
of
clusters
that
are
deployed
into
it.
A
Another
key
part
of
the
cell
based
architecture
is
how
it
gives
you
haier,
MTBF
or
higher
main
time
between
failures.
The
probability
of
an
outage
is
also
lower
with
cells.
Now
they
have
a
consistent
cap
size.
So
every
day
is
not
a
new
adventure,
because
you
know
exactly
this
is
my
cell
size
and
you
keep
testing
that
again
and
again.
A
Now,
of
course,
you
have
a
system
with
n
cells,
so
possibly
n
errors,
but
then
one
n,
it
the
impact
and
but
the
higher
MTBF
and
a
lower
MTTR
really
gives
us
to
that
better
recovery
ability
and
overall,
a
higher
availability.
So
these
are
some
of
the
advantages
of
a
cell
based
architecture,
and
this
is
exactly
how
Amazon
eks
is
deployed.
You
know
when
you
are
rolling
out
kubernetes
to
a
region
or
upgrading
to
a
new
version.
A
When
we
are
testing
kubernetes,
you
know
upstream,
kubernetes
is
cool,
but
we
also
need
to
test
kubernetes
in
amazon,
eks
topology.
So
some
of
the
points
that
I'm
highlighting
here
are,
for
example,
in
our
case,
master
n
HCD
are
highly
available
and
they
are
not
co-located,
because
we
want
to
make
sure
that
the
master
stays
independent
and
HCD
is
a
stateful
database,
so
that
we
can
in
depend
in
the
indicate
upgrade
either
either
one
of
them
independently.
A
Of
course,
our
HDD
is
secured
and
backed
up
I
know
we
use
authentication
using
I
am
as
opposed
to
the
client
side
authentication.
We
also
have
the
cross
account
en
I,
because
the
control
plane
is
running
in
the
AWS
account
and
the
data
plane
is
running
in
the
customer
account.
So
you
want
to
make
sure
that
this
cross
account
en
I
wants.
We
also
have
a
CNI
plugin,
which
basically
give
you
a
native
AWS
networking.
A
So
we
want
to
make
sure
that
when
we
deploying
kubernetes
in
that
topology,
it
just
works
out
of
the
box
for
the
customers
and
a
wide
range
of
load.
Balancers
pick
what
you
like,
so
whether
it
is
lb
and
lb
or
lb,
we
also
have
the
ability
to
say
private
access
for
API
server,
endpoint
and
last,
but
not
the
least.
We
also
offer
99.9%
SLA
and
so
I
would
encourage
you
to
read
more
about
it
on
our
web
site
and
what
that
means
to
you.
A
So,
as
I
said,
you
know
when
we
deploy
kubernetes
as
part
of
Amazon
eks.
Those
are
the
three
factors
that
really
drives
us,
so
we
are
really
looking
forward
to
listening
from
you
more
and
more.
As
you
know,
what
what
features
would
you
like
to
us
now?
As
I
said
it's
already
caught,
Amazon
eks
is
already
coming
to
Ning,
Xia
and
Hong
Kong,
so
we're
very
excited
about
that
in
the
next
few
weeks,
security
is
job.
Zero
and
operational
stability
is
really.
You
know,
followed
very
closely
by
it.
A
One
of
the
things
I
want
to
highlight
also
is
this
is
the
book
that
one
of
my
colleagues
wrote
it
talks
about
open
source
in
the
enterprise.
This
book
is
translated
in
Chinese,
so
I
would
highly
encourage
you
to
come
down
to
the
booth
and
collect
your
copy
of
the
book.
I
am
part
of
the
open
source
team
as
well.
Now
we
have
about
200
plus
block
that
we
have
written,
which
talks
about.
You
know
how
open
source
works.
Now
in
all
of
the
customer
projects
that
are
used
using
open
source
at
Amazon.
A
All
of
those
blogs
are
actually
translated
in
Chinese,
so
I
would
love
for
you
to
subscribe
to
our
blog.
Read
those
blog
then
tell
us
what
other
content
that
you
would
like
to
see.
Once
again,
my
name
is
Arun
Gupta
and
I'm
going
to
be
hanging
out
at
the
conference
today.
If
you
have
questions,
come
talk
to
me,
please
thank
you.