►
From YouTube: Kubernetes on AWS: Practices & Opinions [I] - Arun Gupta, Amazon Web Services & Raffaele di Fazio
Description
Kubernetes on AWS: Practices & Opinions [I] - Arun Gupta, Amazon Web Services & Raffaele di Fazio, Zalando
A lot of progress has been made on how to bootstrap a cluster since Kubernetes' first commit. It is now only a matter of minutes to go from zero to a running cluster on Amazon Web Services. There are still many fundamental topics to take a simple setup to something that can be run in production in a large enterprise and it is easy to get confused by the number of options and customizations.
In this talk we will show both common practices for running Kubernetes on AWS and an opinionated view of those. Specifically, we will cover options and recommendations on how to install and manage clusters, configure high availability, perform rolling upgrades and handle disaster recovery, as well as continuous integration and deployment of applications, logging, and security.
At the same time, we will explain how those topics are addressed at Zalando, Europe's leading fashion platform, based upon their experience of operating tens of Kubernetes clusters in production on AWS.
About Raffaele Di Fazio
Raffaele works with the Zalando's Platform Engineering team in Berlin since 2015. There he is working on container technologies, currently focusing on Kubernetes and cluster orchestration. Over the years, Raffaele developed a genuine passion for simplicity and the Golang language. At Zalando, he also helps promoting open source.
About Arun Gupta
Arun Gupta is a Principal Open Source Technologist at Amazon Web Services. He has built and led developer communities for 12+ years at Sun, Oracle, Red Hat and Couchbase. He has deep expertise in leading cross-functional teams to develop and execute strategy, planning and execution of content, marketing campaigns, and programs. Prior to that he led engineering teams at Sun and is a founding member of the Java EE team.
He has extensive speaking experience in more than 40 countries on myriad topics and is a JavaOne Rock Star for four years in a row. Gupta also founded the Devoxx4Kids chapter in the US and continues to promote technology education among children. A prolific blogger, author of several books, an avid runner, a globe trotter, a Docker Captain, a Java Champion, a JUG leader, NetBeans Dream Team member, he is easily accessible at @arungupta.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
Today
we
are
here
to
talk
about
mastering
kubernetes
on
AWS.
We
know
it's
a
short
session,
we'll
try
to
do
as
much
justice
as
we
can.
The
slides
there
were
recommendations,
our
guidelines,
all
of
those
are
publicly
available
in
a
github
repo,
so
we'll
be
happy
to
share
that
towards
the
end
of
the
session
anyway,
with
you.
So
let's
get
started
with
that.
So
I
I
work
for
Amazon
I'm,
a
principal
open
source
technologies.
A
Italian,
you
love
Boeing,
yeah,
that's
a
fact
all
right
so
today,
instead
of
this,
we're
not
expecting
this
to
be
a
kubernetes
101.
Although
we
have
a
lot
of
content
around
that
topic
as
well,
but
we're
gonna
pick
three
topics
and
we're
gonna
deep
dive
into
those
topics
and
we're
gonna
share
our
recommendations
and
the
way
we
have
structured
the
session
is
I'll
talk
about
what
are
we
seeing
in
Amazon
how
our
customers
are
using
kubernetes
in
different
scenarios?
I
was
pretty
thrilled.
A
I
have
built
actually
a
personal
affection
for
Game
of
Thrones
now
because
it
runs
kubernetes
on
AWS,
so
that
is
pretty
cool.
I
learned
quite
a
few
things
from
that
session,
but
will
also
share
a
broader
perspective.
What
our
customers
are
doing
it
and
then
raffaeli
will
talk
about
it.
Exactly
practices
are
good,
but
here
is
how
we
run
it
in
opinion
advice.
You
know
we
hopefully
you'll
get
a
good
balance
to
be
of
it,
and
the
three
topics
we
chose
is:
how
do
you
set
up
your
cluster?
A
How
do
you
do
Identity
and
Access
Management
and
are
using
kubernetes
and
then
finally,
the
visibility
or
telemetry?
You
know
the
lots
of
names
for
this,
let's
get
started.
So
what
are
your
choices
if
you
are
running
button
from
a
voice
actually
been
talking
too
much
here
so,
but
what
are
your
choices
when
you
are
trying
to
set
up
a
company
this
cluster?
What
does
that
mean?
Really,
you
know
how
do
I
install
a
cluster?
How
do
I
operate
a
cluster?
How
do
I
upgrade
a
cluster?
How
do
I
deploy
in
applications
to
it?
A
What
are
my
choices?
Well
deploying
is
simple
because
as
Cube,
that's
your
CLI,
essentially
or
maybe
some
IDE
or
maven
package
and
all
but
we'll
focus
primarily
on
the
cluster
setup
itself.
Now,
if
you
are
a
developer,
I
know
you
want
to
set
up
a
single
node
kubernetes
cluster
mini
cube
is
a
credible
option.
Today
you
can
download
mini
cube,
it
starts
using
hypervisor
or
VirtualBox,
and
it
spins
up
a
single
node
single
worker,
single
master
cluster
on
your
machine,
and
you
can
interact
with
it
using
cube
cuddle.
All
of
that
works
pretty
good.
A
The
occur
at
darker
corn
also
announced
that
docker
desktop
will
also
have
integration
of
kubernetes
in
there
I'm
still
waiting
I'm
a
talker
captain
some
ways
still
waiting
for
those
to
come
into
my
hands
so
that
I
can
start
trying
it
out
as
part
of
docker
for
Mac.
That
brings
a
more
seamless
experience,
essentially
because
now
from
your
Java
files
or
whatever
your
language
files
are
you
go
build
a
docker
image
using
docker
for
a
Mac,
and
then
you
use
cube
curdle,
so
you
already
have
docker
for
Mac.
A
Why
download
a
new
tool,
that's
kind
of
cool?
On
the
community
side?
There
are
18
different
ways.
You
can
actually
deploy
a
kubernetes
on
AWS
I'm,
not
going
to
list
all
of
them,
but
one
of
the
favorite
one
that
we
have
seen
a
lot
of
our
customers.
Use
is
cops.
This
tool
is
actually
built
in
the
community
by
AWS
sig,
essentially
so
huge
props
to
just
in
Santa
Barbara
and
Chris
love
new
or
the
main
maintainer
and
a
bunch
of
other
people
as
well.
A
We've
been
maintaining
this
tool,
really
slick
and
easy,
and
also
an
example
of
how
a
cops
really
work.
If
you
are
looking
for
a
complete
list
on
the
18
different
ways
and
other
flavors
look
at
kubernetes
AWS
door,
I
owe
at
reinvent
last
week
we
introduced
eks,
which
is
elastic
container
service
for
kubernetes.
That
is
a
managed
service
from
Amazon.
You
go
to
Amazon,
console
or
AWS
console
say,
give
me
a
cluster
and
we
give
you
a
API
URL
and
that's
it.
Now
we
take
care
of
managing
the
cluster,
we'll
talk
about
that.
A
There
are
other
options
by
kauravas,
tectonic
and
Red
Hat
OpenShift,
which
runs
the
open,
YouTube
view
of
kubernetes,
because
it
provides
a
lot
of
tooling
on
top
of
communities
as
well.
Last,
but
not
the
least,
there
is,
of
course,
cloud
formation.
We
see
a
lot
of
customers
using
cloud
formation
because
they
really
want
a
handcraft
how
their
cluster
is
set
up,
how
their
networking
policy
looks
like
and
then
terraform,
of
course,
is
another
option.
A
So
these
are
some
of
the
options
that
we
see
that
our
customers
use
primarily
between
one
of
these
and
then,
of
course,
we
have
partners,
so
darker
hep
t
o
mesosphere.
They
all
provide
recommendations
on.
How
do
you
build
and
run
these
applications?
Now,
let's
take
a
look
at
it
cops
a
little
bit.
So
what
is
coughs
well
coughs
is
community
supported.
It
is
primarily
built,
as
I
said,
as
part
of
sig
AWS.
It's
a
top-level
kubernetes
project
so
get
up,
calm,
slash,
kubernetes,
slash
cops
is
where
you
get
all
the
details.
A
A
It
can
also
optionally
generate
cloud
formation
or
terraform
scripts
in
case
you
need
to
so
then
you
can
take
that
as
a
starting
point
and
then
deploy
your
cluster
using
that,
as
I
said,
this
is
a
top-level
project.
Let's
take
a
look
at
the
CLI
for
this.
Now.
How
does
others
work?
Well,
you
need.
You
are
deploying
your
cops
cluster
in
a
region
with
three
availability
zones.
You
know
you
are
looking
at
a
master
with
high
availability.
What
that
means
is
you
are
deploying
three
masters
and
each
master
also
has
a
co-located
HCD
with
it.
A
So
essentially,
what
I'm
saying
is
my
availability
zones
in
this
case:
R
Us
East,
one
B
1,
a
1,
a
1
B,
1
C
or
in
this
case
1
B,
C
and
D
cops,
also
stores
the
cluster
state
in
an
s3
bucket,
so
I'm
exploiting
that
variable
and
then
I,
say:
ok,
I'm
gonna
create
a
cluster
in
this
case
I'm
just
giving
the
cluster
name
as
cluster
dot.
Kate's,
not
local,
and
just
by
giving
Kate
Scott
local
I
don't
need
to
do
any
DNS
setup,
etc.
It
uses
gossip
protocol
to
discover
the
nodes
among
themselves.
A
It
uses
weave
networking
for
that
we've
matched
essentially
and
it
creates
a
cluster
and
then
I'm
specifying
my
master
count.
My
master
instance
type,
my
node
count.
My
node
instance
type
I,
can
provide
different
kinds
of
networking
availability
here,
in
this
case
I'm
using
calico,
but
there's
work
going
on
already
in
the
next
version
of
cops
by
which
you
can
actually
use
the
cni
plugin
that
we
released
last
week
and
then,
essentially,
you
say
now
create
the
cluster.
A
Let's
talk
a
little
bit
about
ECS
or
easier
East,
KS,
actually
elastic
container
service
for
kubernetes.
So
what
we
give
you
as
part
of
that,
is
essentially
a
managed
control
plane.
You
know
our
customers
have
come
to
us
and
told
us
run
kubernetes
for
me,
so
that
I
can
focus
on
my
applications.
So
and
that's
what
is
giving
you
we
you
come
to.
You
know
you
use
AWS,
console
or
CLI
and
say
create
a
cluster.
For
me.
We
create
a
cluster
now
and
we
give
you
API.
Url
worker
nodes
are
in
your
account.
A
The
master
is
in
our
account.
You
bring
your
own
worker
nodes.
We
give
you
cloud
formation
templates,
we
give
you
pre-built
armies,
you
take
those
armies
and
you
say
you
know,
bring
those
armies
and
connect
that
to
the
server
and
then
you're
good
to
go,
and
then
we
automatically
scale
the
master
for
you,
depending
upon
the
number
of
parts
number
of
requests
that
are
coming
in,
we
automatically
scale
the
cluster.
We
give
you
option
to
manually,
update
the
cluster,
upgrade
the
cluster
or
automatic
upgrade
in
that
cluster.
A
So
let's
look
at
the
core
tenets
for
what
are
the
core
tenets
for
eks?
Well,
the
first
thing
is
in
a
very
classical
Amazon
way:
availability,
reliability,
security.
All
the
ility
is
that
you
are
aware
of
that.
Amazon
gives
you
know
from
AWS.
That
is
what
we
give
you
exactly.
You
can
run
your
full
enterprise
grade
production
workloads
on
eks,
because
we
are
doing
all
the
heavy
lifting
for
you.
A
One
of
the
most
important
tenets
and
I
mean
they're
all
equally
important,
but
one
of
the
ones
that
I
am
super
excited
about
is
we're
going
to
provide
upstream
100%
upstream
experience.
What
that
means
is
you
have
your
kubernetes
cluster
running
on
premise?
You
build
the
application
over
there.
You
can
just
switch
your
cube
cradle
config
and
say
now
I'm
going
to
talk
to
this
cluster
and
deploy
my
applications
over
there.
So,
whatever
the
upstream
experience
is
going
to
be,
there
is
no
for
king.
A
We
will
provide
deep
integration
with
rest
of
the
AWS
stack,
so
I
am
integration,
for
example,
with
cue
card,
we'll
provide
deep
integration
with
that
I
am
integration
on
the
pod
runtime.
We
will
provide
with
that
with
cloud
watch
cloud
trail
with
x-ray.
All
of
those
integrations
will
be
there,
but
there
was
a
completely
optional
for
you
to
use.
You
know
I
mean
that's
what
our
customers
like,
so
we
provide
those
integrations.
If
you
want
to
use
them,
they
are
ready
for
you
now.
A
This
is
the
tenet
that
I
am
super
excited
about,
because
we
are
changing
the
company
culture.
Here
we
are
going
to
be
actively
contributing
to
kubernetes
project
and
what
that
means
is
all
the
work
that's
going
to
be
done
in
eks,
the
manage
control
plane
and
it's
going
to
be
ours,
but
literally
you
know
beak
full
e
compliant
and
done
100%
out
in
the
open
source.
So,
as
a
matter
of
fact,
we
built
a
CNI
plugin
which
allows
you
to
give
a
secondary
IP
address
from
your
V
PC
network
to
a
pod
that
work.
A
Let's
see
an
eye
plug-in,
is
fully
open
sourced.
So
essentially
you
can
take
that
see
an
eye
plug
in
plug
it
into
your
cluster
and
build
your
own
cluster.
That
way
simple
API,
we
can
say
we
can
say
AWS
eks,
create
a
cluster.
I
can
say,
give
me
the
cluster
name,
we're
going
to
provide
different
version
support
and
then
eventually
you
can
say
I
am.
I
am
role,
and
that
is
I
am
role
that
will
be
used
to
authenticate
with
the
cluster
and
then
can
propagate
all
the
way
to
the
pod.
A
So
how
does
it
work?
Well,
essentially,
if
you
come
to
us,
we
say
UK
give
me
a
cluster
today.
If
you
are
building
a
cluster,
you
saying
master
and
HCD,
and
then
your
worker
nodes,
which
are
in
different
availability
zones,
you
come
to
ek.
As
all
you
get
is
you
know?
Alright,
we
take
care
of
all
the
heavy
lifting
for
you.
We
give
you
a
control
plane
and
then
you
bring
your
worker
nodes.
So
the
responsibility
is
a
very
standard,
shared
responsibility,
model
of
AWS
and
then
those
worker
nodes
are
connected
to
the
control
plane.
B
Alright
so,
as
I
say,
the
MuRF
elephant
zalando,
so
so
now
you
heard
some
ways
you
can
actually
provision
or
the
clusters
what
I'm
gonna
tell
you
now
is
a
little
bit
like
how
we
do
this
as
Domino's.
Oh
really
do
pinyin
either
way
of
this
presentation
right
before
doing
that,
I
have
to
tell
you
what
Ando
is,
because
my
Peter,
you
don't
know
it
as
we
are
in
in
taxes
and
solando,
is
actually
the
largest
European
fashion
company.
This
means
we
sell
clothing
and
all
the
source
stuff
online.
B
So
it's
a
ecommerce
essentially
to
let
you
understand
the
scale.
The
company
I
just
show
you
some
some
numbers,
so
you
operate
only
Europe,
as
I
said
in
15
markets
we
have
some
six
fulfillment
centers
22
million
Arctic
customers
and
some
other
numbers.
But
what
is
really
interesting
is
that
we
have
1,100
employees
in
tech,
most
of
which
are
actually
engineers
and
they
want
to
deploy
their
applications.
B
Historically,
we
come
from
a
point
where
we
had
an
on-premise
infrastructure
based
on
a
bare
metal
setup,
in
which,
of
course,
we
had
a
limited
set
up
for
scaling
and
for,
for
example,
starting
new
projects
or
new
teams
and
studying
applications.
Moreover,
our
tooling
was
entirely
custom
based
on
some
software.
We
developed.
We
then
decided
to
migrate
to
the
cloud
specifically
to
AWS
to
make
sure
to
have.
You
know
this
high
speed
that
decide
velocity
for
developers
and
scaling
the
teams.
B
What
do
they
need
in
terms
of
resources
and
a
lot
of
other
things,
and
we
use
confirmation
to
the
pilot
sequence
after
that
we
decided
to
migrate
to
communities
which
allow
us,
first
of
all,
to
have
a
higher
densities
or
packing
all
those
workloads
together
and
to
give
an
API
to
actually
makes
more
sense
for
developers
and
is
not
and
remove
them.
You
know
this
need
of
understanding.
What
is
a
t2,
my
instead
of
and
for
Forex
large,
to
give
a
little
bit
more
context?
B
Just
so
I
said
we
had
the
smooth
lots
of
things
we
had
when
already
when
we
were
to
the
cloud
something
around
200
engineering
teams
and
each
team
was
Indian
responsible
as
well
for
for
the
CIC,
deep
art
and
all
these
kind
of
things
you
heard
this
morning
in
the
in
the
in
the
keynote
from
Kelsey.
Just
you
don't
want
to
deploy
from
cubes
video,
so
part
of
this.
B
All
this
whole
migration
to
communities
is
to
enforce
compliance
and
best
practices
by
means
of
you
know,
making
sure
people
can
only
deploy
it
to
clusters,
only
be
a
proper
CD
set
up
and
enforcing
these
ends
of
operations
by
not
having
them
directly
access
containers
and
do
random
stuff
with
them.
Okay,
so
let's
jump
at
the
at
the
kubernetes
cluster
set
up.
So
currently
we
have
multiple
AWS
account
and
we
provision
exactly
one
kubernetes
cluster
per
account
and
we
are
managing
something
around
50
covering
this
cluster.
B
So,
as
you
may
imagine,
this
is
not
a
terribly
use
huge
number.
If
you
are
Google
or
Amazon,
but
it's
pretty
pretty
big,
pretty
big
number
clustering
number,
if
you
are
a
fashion,
ecommerce
those
clusters
actually
made
designed
to
be
small,
such
that
we
can,
for
example,
limit
the
impact
of
possible
prominent
outages.
B
So
our
problems
with
those
clusters
and
as
well
kubernetes
what
kind
of
code
was
not
optimized
to
deal
with
a
blast
rate
limiting
I
think
about,
for
example,
mounting
EBS
volumes
was
not
used,
not
to
be
perfect,
is
improving,
but
they're
still
still
a
bunch
of
things
to
do.
Our
setup
is
strongly
based
on
on
cloud
formation.
B
We
use
confirmation
entirely
to
provision
those
clusters
and
we
chose
that
because
its
first
of
all
is
a
Douglas
native,
so
we
wanted
to
have
something
that
could
tailor
exactly
the
cloud
we
were
deploying
to
so
AWS
and
because
we
have
a
pretty
good
existing
experience
inside
the
company.
So,
with
this
array
take
the
tool
that
we
are
familiar
with,
our
setup
is
based
on
container
Linux.
B
We
don't
do
right
now,
any
my
customization,
so
we
use
whatever
you
can
use
as
well,
and
we
add
additional
decision
that
we
took,
for
example,
is
to
use
flannel
as
networking
to
support
more
than
50
nodes.
Amazon
now
is
jumping
on
the
hunter
ship,
so
is
helping
with
the
CNI
plugin,
but
it
used
to
be
quite
tricky
to
have
cluster
the
model
50
nodes
due
to
a
limitation
in
AWS
right
in
table
4.
B
So,
as
I
said,
we
have
all
these
clusters,
we
use
this
approach
of
immutable
infrastructure.
We
don't
do
updates
and
reboots
of
nodes.
Instead,
we
replace
them
in
a
rolling
fashion
across
the
set
up.
Looks
mostly
like
that,
so
we
have
to
auto
scaling
groups
the
master
spanning
tree
availability
zone,
as
well
as
the
worker
nodes,
as
the
work
level
is
able
to
scaling
groups.
We
run
everything
ourselves
so
also
hcd
hcd
runs
all
in
their
own
conformation
stack
so
outside
of
the
master
node.
B
So,
as
I
said,
we
have
50
clusters,
it's
quite
a
quite
a
big
number
and
we
have
to
find
a
way
to
operate
them
in
a
way
that
doesn't
require
us
to
do
this
manual
operation.
It's
nice
to
say
your
developers
cannot
deploy
to
production
manually
and
then
you
operate
the
cluster
manually.
This
doesn't
really
make
sense
right.
So
what
we
did
is
essentially
deploying
developing
two
tools.
B
One
is
the
cluster
registry,
which
contains
some
metadata
for
the
cluster,
so
really
information
descriptive
information
of
what
the
cluster
should
look
like
and
a
reference
to
the
configuration
of
this
cluster
which,
by
the
way,
is
open
source.
You
finally
get
calm
and
the
tool
that
just
watch
this
configuration
and
applies
this
by
means
of
clot
formation
stack
and
some
additional
AWS
resources.
So
I
am.
A
A
Essentially,
what
happens
is
there
are
two
IM
roles
created
for
you,
one
for
master
nodes
and
one
for
worker
nodes
and
the
policies
that
are
attached
to
them
are
a
bit
different
and
which
is
by
design,
because
the
capabilities
that
master
needs
to
have
are
a
bit
more
essentially
now
what
we
need
is
a
little
bit
more
granular
control
for
cue,
cuddle
and
parts,
because,
essentially
you
know,
if
you
think
about
it,
your
cue
card
all
relies
upon
cube
config
to
talk
to
your
master,
but
there
is
no
I.
Am
integration
over
there?
A
I
want
to
be
able
to
say
use.
This
I
am
to
authenticate
with
the
cluster
because
that's
sort
of
the
language
that
AWS
customers
are
used
to
and
then,
when
you
are
running
the
pod,
you
wanna
see
that
what
is
I
am
role
that
the
pod
can
take
at
the
runtime
so
that
the
policies
can
be
applied
accordingly.
A
There
is
a
kubernetes
cluster
running
and
then
in
AWS
environment
we
have
AWS
auth
service
running
here,
okay,
so
the
cue
card
all
passes
the
AWS
identity,
which
is
basically
an
I,
am
role
it
passes
it
on.
You
know
we
saw
the
role
or
name
so
we
pass
that
role.
I
am
role
to
the
kubernetes
cluster.
The
cooperate
is.
Excuse
me.
The
kubernetes
cluster
then
talks
to
the
AWS
auth
service,
not
okay.
This
is
the
I
am
role.
This
is
sort
of
what
policies
are
available.
A
It
gets
the
policies,
back,
authentication
is
done
using
AWS
auth
and
the
authorization
is
done
using
the
our
Mac
that
is
baked
into
the
kubernetes
cluster
itself
and
then,
accordingly,
the
action
is
allowed
or
not
allowed.
So
that's
sort
of
how
authentic
ater
model
works.
Let's
take
a
look
at
it
from
the
I
am
identity
for
the
pod
itself.
Now,
if
you
look
at
today
the
pod,
if
it
wants
to
do
something
within
the
ec2
instance,
it
will
talk
to
the
ec2
metadata
service
that
I
want
to
do.
A
This,
can
I
do
this
or
not
cube
I
am
is
a
project
that
has
been
extremely
popular,
which
basically
gives
you.
The
ability
to
assign
an
I
am
role
to
a
pod,
and
it
says:
okay
take
this
role.
So
let's
see
how
that
works.
Now
my
same
setup
here,
first
thing:
what
we
do
is
we
set
up
a
cube,
I
am
as
a
daemon
set
that
gets
set
up
now,
the
part
there's
a
security
token
service.
That
is
also
running
back
in
AWS
now
the
pod.
A
When
it's
trying
to
talk
to
the
ec2
metadata
service,
the
call
is
intercepted
by
the
daemon
set.
That
demon
said
then
guides
it
to
the
secured
token
service
figures
out.
Okay,
this
is
the
I
am
role
that
is
assigned
to
the
part.
What
are
the
policies
that
are
applicable
to
it,
and
then
it
makes
a
call
to
the
ec2
metadata
service,
so
the
whole
part
being
you
could
assign
a
part.
I
am
role
to
the
pod
at
runtime
and
you
can
use
an
I
am
role
at
the
cue
card
level
as
well.
A
Both
are
possible
and
the
way
I
am
role
is
assigned
to.
The
pod
is
literally
just
an
annotation
here.
So
I
have
my
deployment
here
and
in
here
all
I'm
saying
is
this:
is
my
I
am
role
I?
Give
me
I
am
role
are
in
here
and
there
it
goes
now
cube.
Who
I
am
is
a
great
project.
It
has
done
a
great
job,
but
there
are
certain
issues
with
it.
You
know
I
mean
it
requires
really
the
node
to
have
an
assumed
role
capability,
which
is
a
much
wider
capability.
A
A
Actually
that
which
allows
you
to
do
identity
propagation
across
different
clouds
across
different
container
orchestration
is
independent
of
the
container
runtime
and
we
are
particularly
excited
because
it's
not
really
tied
to
kubernetes
or
container
D
or
any
other
runtime,
because
then
it
can
bleed
into
other
CN
CF
projects
as
well.
So
the
possibilities
of
spiffy,
even
though,
is
still
a
futuristic
thing-
are
pretty
huge.
B
I'm
talking
about
I
am
at
Saranda.
We
actually
have
to
first
of
all
distinguish
between
two
different
systems,
so
we
talked
about
AWS
I
am,
and
we
talked
about
anion
system.
Then
we
use
in
internally
for
service
to
service
communication.
We
run
a
micro
service
infrastructure
and
employee
to
service
communication
regarding
AWS
I
am
a
use.
Coop
I
am
as
a
demon
set
in
a
very
similar
fashion
to
what
Aaron
already
shown,
and
but
an
interesting
part
is
actually
why
this
works
for
us
for
us,
so
why
this
is
actually
reasonable
for
us.
B
So
we
put
this
annotation
as
you've
seen
into
those
deployments.
So
why
do
we
trust
people
to
actually
do
that?
And
what
does
it
works
for
us?
It
works
for
this
idea
of
forest
principle.
So,
as
I
said,
no
deployments
happens
manually
to
production,
it
always
happens
through
a
version
llamo,
and
this
llamo
has
to
be
those
configuration
needs
to
be
approved,
and
this
means
whenever
two
engineers
actually
are
saying.
Okay,
this
is
right.
This
is
the
role
we
need.
This
is
fun
for
us
talking
about
the
Orion
system
so
which
you're
calling
platform
I'm.
B
As
I
said,
we
run
a
micro
service
infrastructure.
We
have
hundreds
of
micro
service,
they
talk
to
each
other,
providing
auth
tokens
as
well
as
employees.
They
also
use
auth
tokens
to
communicate
with
some
of
those
services.
We
wanted,
first
of
all
to
make
sure
that
employees
could
use
the
same
tokens.
They
were
used
to
use
in
this
infrastructure,
for
example,
to
to
talk
with
the
kubernetes
api
server.
We
did
that
by
a
natural
extension
points
of
the
kubernetes
api
server.
That
maybe
are
you
familiar
with,
which
is
the
web
book.
B
Essentially,
whenever
you
do
keep
CTL
whatever
you
do
get
parts.
If
you
pass
a
token,
this
will
be.
This
will
write
to
the
API
server
and
the
API
server
whenever
you
have
in
you
are
in
this
web
book
mode
for
authentication
authorization.
This
token
will
be
passed
to
a
web
book
which
is,
in
the
end,
custom
software
that
you
can
bright,
and
this
this
web
hook
in
our
case
is
responsible
for
validating
this
token
and
verifying
that
the
user
is
actually
authorized
to
do
anything.
That
needs
to
be
done
all
right.
B
So
this
is
for
users
talking
with
systems
right
so
with
api's.
But
what
about
those?
Those
application
needs
to
talk
with
each
other.
For
that
we
extend
the
communities
by
means
of
a
customer's
definition,
which
is
which
and
it's
a
small
systems
which
is
called
in
this
case
credentials
provider.
What
does
it?
What
the
system
do
is
in
the
end,
like
the
user,
will
just
create
this
customer
source
definition
and
say:
I
want
these
credentials.
B
This
tokens
for
my
for
my
application
and
the
system
would
just
read
from
the
canoes
API
this
desire
of
having
tokens,
and
it
will
talk
with
internal
I
am
infrastructure
that
we
had
before
kubernetes,
and
it
will
write
back
at
community
secrets
to
the
API.
This
means
in
this
case
that
we,
the
application,
can
really
use
a
native
kubernetes,
a
concept
which
is
a
secrets.
We
are
pretty
handy
to
to
mount
and
set
your
pods
and
with
that
we
are
integrated
and
they're
happy
to
just
use
the
tokens
all
right.
A
Let's
talk
about
visibility
in
a
cluster
I
mean
if
you
look
at
it,
you
know
in
terms
of
the
visibility
of
the
cluster.
If
I
look
in
the
top,
there
are
five
different
aspects
that
we
see
that
are
important
for
a
cluster.
You
want
to
see
the
end-to-end
logs.
You
know
log
coming
from
different
sources.
You
want
to
see
the
metrics.
You
know
that,
okay,
how
is
my
node
doing?
How
is
my
pod
doing?
You
want
to
take
a
look
at
events.
You
know
if
my
node
is
going
down.
A
My
cluster
is
going
up
what's
really
happening
over
there.
You
want
to
have
alerts.
You
know
you
wanna
have
some
alert
based,
auto
scalars,
you
know
triggering
up
and
then
eventually
you
want
tracing
which
is
end-to-end
tracing
of
an
application,
so
those
are
sort
of
five
different
visibility
aspects
that
we
think
about
it.
Now,
those
five
specs
are
really
coming
from
four
different
sources.
A
Now,
let's
take
a
look
at
logs.
Well,
the
logs,
of
course,
the
easiest
way
people
look
at
it
is
queue
catalogs,
that's
a
rather
easy
way.
That's
only
the
first
cut
of
it.
It
doesn't
give
you
all
the
details,
all
the
information
you
can
get
events
and
things
like
that
as
well,
but
queue
catalog
is
definitely
more
popular.
The
stack
that
we
see
a
lot
of
our
customers
using
is
the
F
stack
or
elastic
search
flow
and
D
and
Cabana.
So
let's
take
a
look
at
it.
A
How
that
combination
really
works,
but
essentially
what
you
have
is
a
cluster
and
this
cluster
is
running
in
a
region
and
and
to
separate
availability
zones.
Okay,
keep
feeding
in
and
out
very
good
okay
cool
yeah,
so
I
have
the
cluster
running
in
two
availability
zones
that
are
shown
in
horizontal
rows
and
then
I
have
auto
scaling
groups
as
well,
so
essentially
to
master,
which
is
not
a
good
design,
should
really
be
1
or
3
odd
number
of
masters,
and
then
I
have
a
number
of
workers
over
here.
A
So
once
it
goes
into
the
cloud
watch
logs,
you
have
an
ability
to
subscribe
to
a
elasticsearch
cluster
which
again
is
running
on
Amazon
itself.
So
you
have
an
elastic
search
service.
You
say
any
log
that
comes
to
cloud
watch
automatically,
send
it
to
elastic
search,
and
on
top
of
that,
essentially,
what
you
can
do
is
you
can
set
up
a
cabana
cluster.
So
that's
sort
of
a
F
stack
that
we
see
a
lot
of
our
customers
running
now.
There
are
a
lot
of
different
variations
for
it.
A
That's
not
the
only
way,
but
that
is
the
the
more
prominent
ones
that
we
see.
Take
a
look
at
metrics,
for
example.
What
kind
of
metrics
do
we
see?
Now?
There
is,
of
course,
node
that
gives
you
a
lot
of
metrics
node
exporter
is
a
common
tool
that
we
see
a
lot
of
our
customers
using
in
spinning
out
metrics
about
the
node
itself.
In
terms
of
pod
/
container,
you
can
look
at
cube
state
matrix,
you
know
that
is
actually
bundled
with
kubernetes
or
it
can
be
installed
depending
upon
the
configuration
see
it
wise.
A
There
is
a
pretty
popular
tool
that
developers
look
at
or
customers
look
at
and
then
from
the
applications
perspective.
Let's
say
if
you
are
a
Java
developer.
If
you
are
building
a
java
application,
you
can
start
exposing
your
metrics
that
were
done
using
JMX
using
slash
metrics
endpoint,
and
why
is
slash
metrics
endpoint,
because
essentially
what
you
need
is
a
cluster
wide
aggregator.
You
could
use
either
Prometheus
or
heap
store
as
one
of
those
sources.
A
You
know
to
be
a
plus
provide
aggregator
that
is
actually
pulling
the
data
from
all
the
scraping
the
data
from
all
slash,
metrics
endpoint.
Now
you
do
need
a
data
model
as
well.
Well,
Prometheus
is
a
time
series
database
as
well,
so
you
can
either
store
it
in
there
or
you
can
have
a
specific
influx
VB
or
a
graphite
database
where
your
data
is
stored,
you
know,
and
then
you
can
run
some
analyses.
On
top
of
that.
A
On
top
of
that,
you
would
like
to
build
like
an
alerting,
alerting
mechanism
where
okay,
now
I
got
all
the
data.
Now,
if
my
memory
threshold
goes
below
a
certain
number
trigger
an
alert
for
me,
so
you
can
build
either
alerting
your
own
or
there
are
plenty
of
tools
available
and
last,
but
not
the
least.
What
you
really
need
is
a
visualizer.
On
top
of
that,
there
is
a
dashboard
that
comes
as
part
of
kubernetes.
A
B
So
talking
about
logging
or
at
solando,
so
we
currently
use
a
centralized
login
solution,
so
we
don't
have
a
managed
start
like
that,
but
your
scaler
as
our
centralized
login
solution
and
we
deploy
a
daemon
set
so
a
pod,
then
each
of
these
nodes
they
actually
just
streams
this
log
to
the
software-as-a-service.
This
means
this
is
extremely
easy
for
developers
to
just
pour
their
application.
They
don't
have
to
hear
about
it.
We
have
this
piece
of
infrastructure,
there's
our
stream,
those
logs,
and
they
just
need
to
lock
two
standard
reports
on
their
reckoning
monitoring.
B
We
have
an
existing
existing
monitoring
solution
in
place
since
many
years
now
this
call
set
Mon,
it's
actually
open
source.
You
finally
get
up
and
what
we
did
essentially
when
we
started
migrating
to
kubernetes,
is
essentially
integrating
this
system
with
kubernetes,
so
making
some
of
the
native
kubernetes
resources
available
from
the
system
by
using
pro
meters,
not
exporting
to
get
instant,
metrics
and
hips
are
to
collect
pod
metrics.
What
is
really
important,
though?
It's
really
from
the
user
point
of
view
right
so
make
sure
that
the
user,
not
only
can
they
can't
deploy
right.
B
So
most
of
the
discussions
around
kubernetes,
our
cube
CTL
deploy
something
right,
that's
easy,
but
how
do
you
do
it
like
real
in
production
with
a
compliant
systems
when
a
way
that
people
can
actually
monitor
everything?
So
what
we
do
is
we
integrate
with
our
existing
systems
and
we
provide
some
default
checks
and
alerts
for
all
the
applications
are
deployed
that
the
teams
can
actually
just
just
use.
B
Additionally
to
that
we
have
an
English
controller
and
we
have
some
standard
metrics
regarding
latency
error
rate
that
people
can
just
reuse
to
build
some
proper
reporting
all
right.
So
all
of
this
data
will
not
be
interesting.
If
you
don't
have
a
way
to
actually
have
a
look
at
it,
we
have
some
custom
dashboards.
The
picture
is
quite
small,
but
this
is
actually
the
dashboard
that
we
use
to
monitor
all
these
classes.
B
So
this
was
just
just
look
at
some
of
the
important
things
that
we
learn
with
discover
and
we
recommend
regarding
kubernetes
on
ideas
in
this
case,
but
there
are
definitely
some
more
factors
to
consider.
For
example,
you
want
to
have
a
look
to
configure
some
same
defaults.
Kubernetes
will
not
limit
the
number
of
pots.
You
can
run
in
a
namespace
by
default,
so
you
want
to
put
a
quote
on
that
or
you
want
to
have
a
limit
range
to
have
some
default
resource
limits
for
your
application,
and
you
want
to
know
your
cluster
limits.
B
It's
easy
to
say:
I
can
run
ten
pods,
but,
as
you
can
see,
healthy
didn't
want
to
even
run
the
10
million
pods.
As
someone
asked
this
morning
in
the
keynote-
and
there
is
a
reason
for
that-
and
you
also
want
to
as
well
work
on
simplifying
the
user
experience
by,
for
example,
starting
using
things
like
ingress
or
external
DNS,
which
is
actually
an
open
source
project
to
get
some
nice
dns
name
for
apps,
which
we
developed
and
is
in
the
companies
incubator.
A
One
of
the
last
things
that
we
would
like
to
share
with
you
is
this
is
a
workshop
that
we
have
been
building
for
past
several
months
now.
This
is
a
workshop
that
we
are
sharing
with
our
customers
or
partners
or
developers.
If
you
are
new
to
kubernetes
and
you're
gonna
get
started
with
kubernetes
on
AWS.
A
You
know
you
can
literally
start
with
this
workshop
and
you
can
spend
a
few
days
literally
if
you
want
to
go
through
each
chapter
n-by-n
and
what
we
are
encouraging
is,
if
you
don't
like
something
about
the
workshop,
this
is
done
completely
in
an
open-source
way,
submit
an
issue,
send
a
pull
request
and
that's
what
we
are
looking
for
from
you.
We
are
really
trying
to
show
all
possible
ways
our
customers
are
using
it.