►
From YouTube: The Architecture of a Multi-Cloud Environment with Kubernetes [I] - Brian Redbeard, CoreOS
Description
The Architecture of a Multi-Cloud Environment with Kubernetes [I] - Brian Redbeard, CoreOS
Kubernetes is an orchestration platform that enables running distributed systems, which are designed with the philosophy of spreading wide to best prepare for outages. This is achieved by deploying your cloud applications at least across multiple hosts, and at best across multiple cloud vendors. Getting Kubernetes configured to run across multiple cloud environments, including on-premises, hybrid deployments, is a tricky undertaking. Hybrid deployments are a feature many organizations want to implement for a variety of reasons, including security over their data, reliability, and more.
Brian Redbeard, chief architect at CoreOS, will discuss the importance of using open source tools to prevent cloud vendors from locking their users into their walled gardens, and will explore the challenges of making Tectonic, CoreOS’s Kubernetes implementation, able to run on multiple cloud platforms.
About Brian Redbeard
Brian Harrington, also known as Redbeard, is chief architect at CoreOS. He is a developer, hacker, and technical writer in the areas of open-source development and systems administration. His time spent in both defensive and offensive computing have combined with his readings of classical anarchism to present new ideas in organizational hierarchies for software development. He has been featured on Al Jazeera as an expert in the field of computer security, and has been seen and heard on Bloomberg Television and National Public Radio. He currently resides in Oakland, CA and was grudgingly elected president of the hackerspace HacDC.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hello,
everyone,
so
we
are
coming
up
right
on
the
tail
end
of
the
day.
Here
it
is
4:35
and
I
have
35
minutes
to
go.
So
let's
get
this
show
on
the
road
so
to
jump
into
things.
I'm
red
beard,
I'm
with
core
OS
I
talk
about
weird
things.
Apparently
we're
talking
about
making
cloud
native,
weird,
so
I'm
in
the
right
place,
jumping
into
it.
I
am
here
to
tell
you
a
whole
bunch
of
lies.
A
Now
these
are
going
to
be
lies
in
the
sense
of
they
are
lies
like
high
school
physics,
where
I
tell
you
that
the
smallest
particle
in
the
universe
is
an
atom,
and
then
you
learned
that
there
are
quarks
and
antiquarks
and
positive,
spin
and
anti
positive,
and
all
of
that,
so
as
with
any
good
presentation,
we
should
be
able
to
answer
a
number
of
questions
by
the
end
of
this,
and
this
presentation
will
not
be
any
different.
If
you
can
answer
all
of
these
questions
by
the
time
we're
done,
then
I've
succeeded.
A
If
you
cannot
answer
them,
then
I
have
failed
so
first
off.
Who,
who
is
the
target
audience
of
this?
This
is
going
to
be
organizations
who
are
looking
to
run
kubernetes
in
some
redundant
manner.
What
are
we
going
to
be
talking
about
in
the
words
of
Mike
Tyson?
Everybody
has
a
plan
until
they
get
punched
in
the
mouth.
A
So
what
we
are
going
to
talk
about
are
the
considerations
that
you
have
to
make
to
be
able
to
build
out
these
types
of
multi
environment,
possibly
multi
cloud
systems
and
making
sure
that
you've
kind
of
planned
the
steps
and
that
the
execution
makes
some
sense
and
what
are
the
common
misunderstandings
there?
So
when
are
we
going
to
do
this?
Well,
we're
gonna,
let
you
know
when
everything
is
going
to
be
generally
available
within
kubernetes
I
will
be
talking
at
once.
Mall
point
about
tectonic,
but
this
largely
has
very
little
to
do
with
tectonic.
A
I
am
a
huge
believer
in
making
sure
that
you
know
they're
part
of
the
reason
why
you're
looking
at
kubernetes
is
to
make
sure
that
things
are
as
agnostic
as
possible
within
some
realm
of
general
influence.
So
we
are
going
to
make
sure
that
these
are
applicable
across
whatever
district
of
kubernetes.
You
happen
to
be
using
the
wear
of
this.
Are
you
know
what
are
the
considerations
kind
of
broken
down
by
the
compute
environment?
A
Are
there
specific
things
that
you
have
to
worry
about
on
digitalocean
versus
packet
net,
you
know:
is
there
anything
that's
exclusive
to
bare
metal
versus
AWS
and
finally
getting
into
the?
Why
can
you
answer
why
you
want
to
do
this
because
is
not
an
answer
to
that
or
at
least
a
valid
answer,
so
we
want
to
make
sure
that
you're
able
to
break
some
of
this
down
and
then
with
the
last
one.
How
do
you
actually
achieve
these
goals
so,
as
I
like
to
say
kind
of
in
a
from
the
trenches
type
perspective?
A
Let's
jump
step
one
into
the
past,
so
we
need
to
make
sure
that
we
don't
forget
the
past
and
one
of
my
colleagues
coined
the
term
cloud
kids
as
a
pejorative
for
himself.
You
know
he
back
when
I
ran
the
infrastructure
team
at
core
OS.
He
said
you
know
that
there
were
a
whole
lot
of
things
that
just
weren't
possible
and
that
you
know
he
as
a
cloud
kid
because
he
had
ran
and
where
he
was
working
on
an
infrastructure
team
and
had
never
touched
a
physical
server.
A
That
blew
my
mind:
an
infrastructure
s
re-type
individual,
who
had
never
grabbed
a
piece
of
metal
and
kind
of
worked
with
it.
So
for
the
cloud
kits
in
the
room,
this
tidbit
is
for
you
for
everybody
else,
we'll
move
through
it
real
fast,
so
to
understand
where
we're
at,
let's
analyze
an
example
from
the
past
which
for
this
one
I,
will
pick
Oracle
real
application
clusters
so
to
use
Oracle
RAC,
you
had
to
have
a
sand
and
to
understand
what
that
starts.
To
look
like.
A
Let's
look
at
this
very
pretty
picture,
so
in
this
picture
we
have
to
compute
nodes.
We
have
two
disks
you'll
notice,
that
each
of
those
disks
is
connected
to
both
of
the
servers
concurrently.
This
is
not
something
where
you
know:
you're
kind
of
doing
an
e
BS
kind
of
magically
attaching
and
detaching.
Both
servers
can
access
both
of
the
disks.
At
the
same
time,
that's
part
of
the
basis
behind
us
and-
and
it
is
up
to
the
things
that
consume
those
resources
as
block
devices
to
be
able
to
intelligently
consume
them
now
for
Oracle
RAC.
A
A
This
ugly
thing
is
a
worldwide
name,
so
to
make
it
just
a
little
bit
more
understandable,
we're
gonna,
reformat
it
and
we'll
reformat
it
again
and
we'll
keep
reformatting
it
and,
as
you
go,
you
notice
that
it
starts
to
look
like
a
MAC
address.
The
reason
for
that
is
is
because
it's
basically
a
MAC
address
for
a
disk,
the
I
Triple
E
issues.
A
This
will
make
more
sense
in
a
moment
as
to
why
I'm
talking
about
this.
But
this
lack
of
knowledge
about
things
like
sands
and
everything
was
largely
driven
by
the
cloud,
and
you
know
nobody
needed
to
really
know
what
a
LUN
was
anymore
or
what
it
meant
to
be
zoned
into
a
host.
But
your
sand
doesn't
limit
you
to
those
things
as
well
and,
as
I
said,
we
should
not
get
bogged
down
by
what
the
cloud
can't
do,
because
we
will
come
back
to
that.
So
moving
into
a
bit
more
of
the
present.
A
A
Hopefully
not
hopefully
you're
trying
to
get
to
lower
latency?
But
you
know
is
it
that
you
are
trying
to
increase
throughput?
Is
it
that
you're
trying
to
ensure
that
your
data
locality
puts
your
services
to
as
close
to
your
end
users
as
possible?
Thus,
you
know
a
user
accessing
your
resource
in
Singapore
gets
sent
to
a
data
center
in
Singapore
and
a
user
accessing
your
resource
in
London
gets
sent
to
a
data
center
in
London.
Are
you
trying
to
optimize
for
other
types
of
resource
utilization
like
RAM
or
CPU?
You
know.
A
Are
you
trying
to
avoid
the
next
five
nines
attack?
Do
you
not
want
to
be
e
Corp?
You
know
kind
of
responsible
for
having
all
of
your
data
destroyed
because
it
was
in
a
single
location
or
especially
press
into
the
world
that
we
live
in
right
now.
Are
you
trying
to
avoid
disaster
through
global
thermonuclear
war?
A
Hopefully
this
is
not
something
that
you
are
trying
to
protect
against,
but
in
other
words,
what
we
are
talking
about
here
is
you
need
to
know
what
your
failure
domains
are.
You
need
to
understand
whether
you're
trying
to
protect
against
disk
loss
or
data
center
loss,
because
sure
you
may
be
provisioning
things
at
an
Equinix
data
center
in
Ashburn,
Virginia,
and
then
a
tornado
comes
through
and
takes
out
your
resources
and
s3.
A
A
Have
a
hint
for
you
so
now
that
you
have
at
least
thought
through
this
and
you're
starting
to
map
out
what
your
reasons
for
wanting
to
do.
This
are.
This
means
that
you
are
going
to
have
to
define
your
environment
via
some
configuration
manifest.
What
this
means
is
that
you're
probably
going
to
do
things
in
the
realm
of
configuration
as
code,
so
how
do
I
get
this
done?
This
is
where
things
start
getting
actually
interesting
for
folks,
because
I
dig
into
a
little
bit
more
concrete
ideas.
A
Even
if,
in
a
strange
way,
this
version
control
system
is,
you
are
stuffing
them
as
objects
into
Etsy
D,
where
with
Etsy
D,
you
can
then
go
back
through
and
play
back
the
index
over
time
to
be
able
to
recover
things
now
in
reality,
you
probably
are
not
going
to
do
that,
because
we
have
a
much
better
system
for
versioning
text
documents,
and
that
is
git,
but
you
must
have
something
like
this.
If
you
do
not
have
this,
you
are
doing
it
wrong.
A
On
top
of
that,
I
use
this
beloved
piece
of
software
to
react
to
the
changes
in
those
repositories
and
monitor
and
take
action
when
things
are
occurring.
Now,
if
I'm
doing
this
with
a
bunch
of
terraform
definitions,
there's
probably
going
to
be
secrets
involved
in
some
form
now
because,
like
I,
said
I
used
to
run
the
infrastructure
team
at
core
OS
we've
been
doing
this
for
quite
a
while,
and
all
of
the
pieces
of
software
that
are
available
today
were
not
necessarily
available
in
the
past.
A
So
the
way
that
we
historically
got
this
done
was
using
a
utility
called
git,
crypt
and
I'm,
going
to
actually
give
you
a
snippet
of
Jenkins
pipeline,
showing
exactly
how
we
use
it.
It's
not
complex
but
get
crypt
gives
you
a
GPG
based
mechanism
for
pulling
secrets
out
of
encrypted
at
rest
storage
using
them,
while
you're
done
and
then
purging
them
from
memory,
but
just
because
I
am
doing
it.
The
with
these
tools
does
not
mean
that
you
have
to
do
it
using
these
tools.
A
There
are
other
things
that
can
be
used
in
similar
fashions,
like
you
can
use
terraform.
You
can
use
things
like
cops.
You
can
use
Kubb
spray.
All
of
these
are
mechanisms
for
kind
of
bringing
up
your
underlying
compute
you'll
also
talk
about
here.
In
a
moment,
you
can
also
use
things
like
helm,
charts
to
actually
manage
the
lifecycle
of
your
applications
and
for
anybody.
Who's
sat
through
Vic's
talk
right
before
this.
You
kind
of
were
kind
of
handed
some
direct
design
patterns
on
how
to
successfully
use
these.
A
Yeah
there's
a
whole
lot
of
feelings.
I
have
about
that,
but
you
know
just
use
perforce.
If
you
are
one
of
those
dyed-in-the-wool
like
Jenkins
is
garbage
and
we
have
successfully
migrated
everything
to
drone
I/o
or
to
Travis
great.
You
have
a
solution,
use
the
solution
you
have
and
the
one
you
understand.
These
are
ideas,
and
these
are
design
patterns
that
you
use
to
build
these
systems
in
this
way.
So
continuing
to
move
on.
A
Yes,
Amazon
will
let
you
use
the
same
10.0.0.0
slash
16
in
every
V
PC
you
set
up,
it
does
not
mean
you
should
do
that.
You
know
back
in
the
day
the
kind
of
state-of-the-art
system
for
IP
address
management
or
I
Pam's
was
an
Excel
spreadsheet
that
sat
on
a
shared
drive.
You
know
the
synchronization
problems
with
that
were
definitely
non-trivial.
A
So
this
does
not
seem
like
a
big
deal,
but
just
do
it
and
if
you
need
some
help
with
this,
I
had
an
intern
last
summer
write
a
utility
to
make
this
easier
to
do
and
I'm
going
to
move
on
to
an
example
of
this.
But
we
will
have
this
back
up
here
in
a
moment.
So
let's
actually
look
at
this,
and
for
this
I'm
going
to
try
to
refresh
it
there
we
go.
So
what
we've
got
going
on
is
I
run
the
CLI
utility
and
I
have
two
different
failure.
A
Domains
and
an
arbitrary
address
range
that
I
pick
I
pick
it
saying
that
we
have
a
slash
20,
that
we
want
to
use
for
the
whole
environment
and
it
breaks
down
what
the
ranges
should
be
to
have
a
public,
a
private
and
a
protected
network
as
well
as
extras,
I
change
it
to
a
different
subnet
size.
It
does
all
the
math
for
me
and
fixes
everything.
A
A
Maybe
I
need
a
terraform
definition
for
bringing
this
up
on
AWS,
so
it
renders
the
terraform
definitions
giving
me
the
nat
gateways,
and
all
of
these
other
things
gives
me
the
routing
table,
and
yet
I
know
that
everybody
is
not
using
terraform.
So
maybe
you
want
a
cloud
formation
and
we
can
just
do
this
and
render
a
cloud
formation
as
well.
These
are
common
design
patterns
that
everybody
should
not
be
reinventing
the
same
wheel,
and
that
is
the
value
of
open-source
utilities
like
this.
A
So
if
you
need
something
like
this,
we
have
the
ability
to
make
something
like
this
available
so
continuing
on
here.
Digging
in
there
we
go
perfect.
So
the
next
step,
knowing
what
your
storage
is,
stop
freaking
out
about
storage
seriously,
I
have
sat
through
enough
talks
here
and
enough
lightning
talks
and
everything
of
being
folks
going.
Why
are
we
going
to
do
about
storage?
A
When
is
storage
going
to
be
a
solved
problem
in
kubernetes?
Guess
what
if
you
are
running
any
reasonable
size?
It
is
a
solved
problems.
Let's
talk
about
it.
What
is
your
platform
are
using
AWS
cool
you've
got
EBS
volumes
are
using
GCP,
you've
got
persistent
disks.
Are
you
on
Azure?
Well,
they
have
a
whole
matrix
of
managed
and
unmanaged
and
premium
and
standard,
and
all
of
that,
but
it's
more
important
to
think
about.
A
What's
generically
happening
here
under
the
hood,
so
we'll
walk
through
an
example,
so
I
have
two
servers:
server,
green
and
server,
blue
and
I
have
an
EBS
volume
attached
to
server
green
and
then
that
tornado
strikes
and
server
green
goes
down.
It
comes
unavailable
and
the
whole
point
of
my
cluster
scheduler
in
knowing
about
these
remote
storage
objects
is
that
it
intelligently
D
schedules
that
disk
from
being
attached
to
server
red
and
attaches
it
to
server
blue
and
tada.
A
My
stateful
application
still
has
all
of
its
data
and
is
able
to
migrate
to
a
different
system
now.
The
whole
point
here
is
that
kubernetes
is
giving
you
shoulders
to
stand
on
the.
What
are
we
going
to
do
about?
Storage
is
a
concept
of
what
are
we
going
to
do
without
the
concepts
of
persistent
volumes
or
persistent
volume,
claims
or
storage
classes,
or
all
of
these
other
things
by
using
these
primitives
that
are
built-in
and
saying
I
have
defined
a
storage
class.
A
That
is
the
default
for
this
cluster,
meaning
that
if
I'm
on
AWS,
my
default
storage
class
is
just
use,
EBS
volumes
and
if
I'm
on
packet
net,
it
is
please
just
give
me
their
remote
block
storage.
Then
you
are
able
to
make
better
and
more
intelligent
decisions
here
that
whole
concept
that
I
just
showed
you
is
the
read/write
read/write
once
mechanism
and
you'll
notice
that
that
process
of
migrating
a
disk
from
one
machine
to
another,
that
is
the
missing
piece
in
historic
compute
systems.
A
When
you
have
a
SAN,
because
you
needed
some
mechanism
to
intelligently
go
hey.
This
host
has
gone
down,
reschedule
that
workload
on
a
different
host
and
attach
the
disk.
There
were
entire
mechanisms
like
what
are
the
tools
there
is
CL
VMD,
which
is
like
a
whole
cluster
Dell
VM
system
that
uses
multicast
addresses
to
be
able
to
dynamically
change
a
instance
from
B
or
a
disk,
that's
mounted
from
being
read
only
to
read/write,
but
that
brings
up
a
very
good
point.
A
A
One
you're
able
to
immediately
see
that
data
on
machine
number,
two
or
possibly
you
just
use
the
RBD
mechanism
from
seth,
where
you're
still
back
to
just
the
vanilla
kind
of
EBS
type
mechanism
of
I.
Have
some
storage
object?
I
want
it
to
be
available
as
a
block
device.
Please
make
that
available.
But
the
big
thing
is,
you
just
need
an
API
for
your
storage
engine
and,
if
you're
already
paying
for
NetApp,
they
give
that
to
you
if
you're
already
paying
for
EMC.
A
A
How
does
support
I
scuzzy
is
able
to
just
take
these
block
devices
and
plug
them
in,
and
you
don't
have
to
go
to
the
effort
of
thinking
about
all
of
this
now
getting
back
into
this
I
promised
some
more
thoughts
about
setup.
So,
let's
take
a
moment
and
let's
put
all
of
these
pieces
together.
So
what
do
we
need
to
worry
about
here?
Well,
hint.
A
Good
luck
with
that.
But
when
you
are
just
at
the
realm
of
getting
started
here,
this
is
going
to
work
just
fine.
You
are
also
going
to
be
wanting
to
do
reasonable,
monitoring
and
alerting,
because
you
need
to
measure
the
performance
and
alert
on
all
of
your
problems.
That
means
that
you
will
want
to
have
Prometheus
running.
You
will
want
to
get
alert
manager
configured
and
you
will
want
to
have
Jager
in
place
so
that
you
can
actually
trace
the
statistics
of
your
application.
A
This
is
not
rocket
science.
This
is
just
making
sure
that
you
are
doing
the
same
things
on
all
of
your
clusters
so
that
you
have
central
places
of
being
able
to
go.
I
go
to
my
tracing
dashboard
and
I
am
able
to
trace
components
of
the
various
applications
that
I
have,
regardless
of
where
they
happen
to
be
at
the
big
one.
A
Here,
it's
always
a
DNS
problem,
I
make
every
single
person
who
works
for
me,
read
the
first
three
chapters
of
DNS
and
bind,
and
they
laugh
about
that
and
then
I
tell
them
I'm
serious
and
they
will
read
the
first
three
chapters
of
DNS
and
bind
because
it
is
always
a
DNS
problem.
So
this
means
properly
federates
your
DNS
when
I
meet
when
I
say
that
I
mean
DNS.
Has
a
global
scope.
A
Do
not
pretend
that
it
does
not
have
a
global
scope,
because
when
you
do
that
you
are
doing
yourself
a
disservice
when
I
logged
into
our
developer,
AWS
account
and
I
saw
two
hundred
tectonic
dot
local
zones
come
on
folks.
What
are
you
doing?
There's
there's
not
two
hundred
concepts
of
tectonic
dot,
local,
that's
the
whole
point
of
name
spacing
we
have
this
global
lee
names
based
resource
called
DNS.
A
That
means
that
if
you
are
doing
things
just
think
about
it,
because
if
you
have
an
LDAP
system
for
example-
and
you
want
to
be
able
to
talk
to
or
fail
through
a
number
of
LDAP
systems,
one
SRV
records
already
handled
that
for
you.
But
if
you
don't
necessarily
want
to
go
that
route,
let's
think
about
it.
A
So
this
means
that
you
can
either
utilize
components
like
core
DNS
with
your
cluster,
to
handle
some
of
this
type
of
Federation
and
to
be
able
to
expose
both
the
cluster
dot
local
type
concept,
as
well
as
federating
that
out
to
your
external
DNS
systems,
but
it
also
means
you're
just
going
to
use
your
external
DNS
infrastructure.
Do
not
reinvent
this
wheel.
A
It
was
just
available
because,
especially
when
you
start
building
micro
service
type
applications,
you
don't
really
care
what
the
host
name
is
because
it's
always
going
to
be
behind
some
other
API
proxy
or
in
the
case
of
kubernetes
ingress
controller,
or
something
like
that.
Finally,
you
are
going
to
want
to
sync:
your
are
back
configuration
generally
across
your
clusters.
This
means
today
our
trusty,
Butler
or
spinnaker
can
handle
some
of
these
things.
A
It's
the
concept
of
continuous
deployment,
where,
as
you
are,
making
changes
to
those
configuration
files
that
you
have
in
revision,
control
that
Jenkins
is
monitoring
that
it
is
able
to
pull
down
the
credentials
from
get
Crypt
or
vault
and
then
be
able
to
apply
those
changes
to
the
cluster
traffic
distribution.
Cluster
traffic
needs
redundancy.
This
is
to
say
that
utilize,
either
your
cloud
load
balancers.
A
It
does
not
mean
that
you
will
always
have
the
most
optimal
traffic
path
resolution.
If
you
are
doing
static
routes,
we've
got
BGP.
You
can
do
that,
but
it
means
that
now
you
don't
have
to
worry
about
having
separate
pieces
of
hardware
which
you
have
to
buy
two
of
for
everything
and
then
setting
up
heartbeat
connections
to
them
and
then
learning
their
API
and
everything
else,
because
in
this
case,
BGP
is
a
simple
enough
API
to
do
this.
A
What
this
means
is
that
when
you
are
taking
all
these
components,
you
need
to
demand
the
having
network
api's,
because
in
distributed
systems,
everything
becomes
a
network
resource,
whether
it
is
storage
or
whether
it
is
a
database
as
a
service
or
whether
it
is
something
else.
So
now
we
move
in
to
the
execution
of
this,
and
we
discuss
how
we
begin
to
take
action
so
step.
One
cluster
provisioning
understand
that
there
are
separate
stages
here
you
deploy
hosts
and
those
hosts
have
some
manifest
and
you
have
clusters
and
that
cluster
has
some
manifest
configuration.
A
It
then
means
that,
as
you
do
that
you're
going
to
normalize
your
temp
normalize
and
template
eyes,
your
host
can
fix
that's
very
easy.
To
do
with
Linux.
Same
manifests
can
be
used
generally
for
both
bare-metal
machines
and
cloud,
and,
if
you're,
using
something
like
kickstart
and
cloud
configs
in
separate
systems
it.
May
you
just
want
to
break
things
down
to
kind
of
the
the
minimum
state
or
kind
of
use
ansible
to
manage
that
and
avoid
static
configs
at
all
cost.
If
you
think
you
need
static,
IP
addresses
you
probably
don't.
A
There
are
cases
where
you
may
need
that.
But
again,
if
your
answer
is
well,
it's
because
we've
always
done
it.
That
way.
That's
the
wrong
answer.
Things
like
DHCP
exist
for
a
reason,
and
things
like
DHCP
have
been
made
redundant
for
a
reason.
These
are
things
that
you
need
to
begin
exploring.
So,
let's
get
into
the
cluster
configuration,
so
the
things
that
change
over
time
are
the
couplet
flags
ensure
that
everything
is
as
we
like
to
call
it
a
core
OS
under
management.
A
A
So
this
is
an
example
of
like
the
crazy
matrix
build
that
we
have
that
deploys
things
to
different
clusters
in
different
regions
and
does
it
automatically
based
on
if
it's
a
production
resource
or
it's
a
development
resource,
and
when
you
start
getting
into
that,
let's
talk
about
this
step-by-step,
so
first
you're
going
to
unlock
your
credentials
using
greek,
crypt
or
vault.
This
is
where
we
pull
a
key
from
escrow.
A
It's
pretty
simple,
literally
a
bash
script
that
knows
how
to
call
get
crypt
and
is
using
variables
embedded
into
Jenkins
and
some
of
the
identity
and
secret
management
built
into
Jenkins
from
there.
It
just
gives
us
the
ability
to
pull
this
in
as
a
pipeline
stage.
So
I
can
unlock
my
credentials.
Then
I
do
a
get
clean
and
a
terraform
validate
pretty
straightforward
against
calling
these
things
as
bash
scripts.
A
In
a
Jenkins
pipeline,
the
validation
script
itself
runs
terraform
validate,
but
then
a
lot
of
you
are
also
go
developers,
and
you
know
that
you
should
be
running.
Go
fump
thanh
your
code,
you
should
be
running
to
terraform
thumped
and
we
have
components
here
that
exit
non
zero,
because
this
is
the
Van
Halen's
brown
M&Ms
moment.
It
shows
whether
or
not
you
actually
read
through
that
contract
and
made
sure
that
you
were
doing
all
of
the
things
that
were
required.
A
After
that
we
plan,
we
send
a
message
over
slack,
asking
a
human
being
to
confirm
that
the
work
should
be
done
and
then
we
deploy
it
with
terraform
apply
so
working
in
a
cluster.
This
way,
adding
a
new
cluster
is
as
easy
as
defining
an
environment
variable.
Then
you
have
to
maintain
what
you've
got.
So
we
also
use
a
similar
set
of
Jenkins
jobs
to
manage
these
environments.
A
Again,
we
define
the
environments,
we
give
them
names,
they
have
paths
to
the
location
of
those
configs
within
the
various
repositories
and
we
use
a
similar
set
of
processes,
get
crypt,
get
clean,
helm,
lint
and
helm
upgrade,
and
thanks
very
much
to
Brad
Ison
from
the
infrastructure
team.
Who
did
a
lot
of
this
work
and
like
really
insisted
on
making
that
the
way
that
we
did
everything
so
then
we
have
manual
operations,
don't
do
them
seriously,
don't
do
them?
Okay,
fine,
if
you're
going
to
do
them
at
least
make
sure
that
you
export
things.
A
So
thanks
to
Duffy
Cooley
for
this
one
I
had
no
idea
about
this
until
he
pointed
out
that
there's
an
export
flag
that
you
can
use
of,
coop
CT
I'll
get-
and
this
removes
all
specific
information
that
is
to
that
cluster
from
the
resource.
So
then
you're,
just
a
coops
ETL,
apply
away
from
mucking
up
all
the
work
that
your
robot
is
done
so
finally,
disaster
recovery
things
will
go
wrong.
This
is
LA
yesterday.
This
is
the
405
freeway
going
down
by
the
Getty
Center.
A
So
for
all
those
folks
who
came
from
Los
Angeles
have
fun
going
home,
have
a
plan
and
get
prepared
to
be
punched
in
the
mouth
and
test
the
plan.
Do
things
like
backing
up
Etsy
d
and
running
multiple
clusters
in
different
failure,
domains
and
use
certain
purpose-built
tools
like
hep
do
arc,
which
is
utility
for
managing
disaster
recovery,
specifically
for
your
kubernetes
cluster.
So
who
is
you
what
is
making
sure
that
you
know
how
to
build
out
a
multi
cloud
and
multi
kubernetes
system,
I'm
19
seconds
over,
which
is
why
I'm
rushing?
A
So
what
are
the
common
misunderstandings?
You
know
understanding
or
thinking
that
clouds
are
beautiful
and
unique
snowflakes
and
you
have
to
do
things
differently.
They're,
not
you
just
have
to
think
about
them
differently.
All
of
these
features
are
available.
Now.
This
is
not
things
that
you
have
to
wait
on.
You
can
take
this
and
do
it
all
today.
So
knowing
why?
How
yeah?
We
can
do
all
this
so
tectonic
you
can
free
for
up
to
10
nodes.
It
has
hooks
for
doing
a
lot
of
this
stuff
automatically.