►
From YouTube: How DoD Uses K8s & Flux to Achieve Compliance & Deployment Consistency - M. Medellin & G. Tillman
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How DoD Uses K8s and Flux to Achieve Compliance and Deployment Consistency - Michael Medellin & Gordon Tillman, Department of Defense
Like many other organizations, the DoD also started the journey to K8s and had its own challenges. Due to the nature of DoD systems, there are applications that are deployed in a more relaxed environment such as AWS GovCloud and also in a more restricted air-gapped environments with no internet or external connectivity. In the beginning, it was all about manual deployment and operations. By introducing Helm and Flux, DoD moved to a more declarative model where everything is version controlled and deployed by Flux thereby reducing manual operations, improving deployment consistency and also bringing more compliance with regard to change management and application life cycle. This session will walk through the migration steps, what it takes to operate Flux in an air-gapped environment and how we achieved parity when applications are deployed to environments with different constraints.
A
A
Awesome.
Thank
you
gordon
today,
we're
going
to
spend
about
20,
25
minutes
or
so
talking
to
everyone
about
how
the
department
of
defense
is
using
kubernetes
and
flux
to
achieve
our
compliance
and
deployment.
Consistency
that
we're
looking
for
in
our
efforts
to
develop
the
capabilities
that
our
users
are
looking
from
us
from
us
on
a
day-to-day
basis.
A
Just
you
know
I
you
know
coming
into
context.
I
want
to
make
sure
I
can
explain
a
bit
more
about
what
customer
running
is
and
what
we're
doing
our
vision
is
to
deliver
combat
capability
that
can
sense
and
respond
to
conflict
in
any
domain
any
time
and
anywhere.
We
are
part
of
the
mission
just
as
much
as
the
exercises
and
operations
that
go
on
with
flying
planes
around
the
world
or
flying
aircraft,
or
you
know,
driving
aircraft
carriers
around
the
world.
We
are.
A
So
if
you
dive
into
it,
what
really
is
castle
run
kessel
run
is
an
acquisition
development
environment.
So
traditionally,
within
the
government
we
have
acquisition
units
that
are
going
out
and
awarding
contracts
to
software
engineering
teams
and
other
teams
to
in
other
companies
to
actually
build
capability.
Much
like
you
would
go
out
and
build
an
airplane
or
a
ship
or
some
some
type
of
capability
or
for
the
department
of
defense.
A
You
know
in
kind
of
the
buzz
wordy
term
we're
we're
doing
devsecops
we're
trying
to
unify
a
way
to
ship
software
to
our
users
around
the
world.
Do
it
securely,
do
it
reliably
and
do
it
continuously
and
kind
of
achieving
that
loop,
so
a
bit
more
about
kind
of
custom,
lens
mission
and
kind
of?
What's
our
our
main
focus
from
you
know
what
we're?
What
are
we
actually
trying?
Who
are
what's
our
mission?
A
What
are
we
actually
trying
to
solve,
and
you
know
I
we
have
to
talk
a
bit
more
about
kind
of
the
air
operation
center
and
how
we
do
command
and
control
operations
for
the
united
states
air
force
and
our
coalition
forces
around
the
world
back
when
this
you
know
when
air
forces
kind
of
first
started.
This
was
very
much
manual
process,
working
with
maps,
communicating
over
radios
and
telephones
to
to
direct
resources
and
and
manpower
to
actually
affect
capability
around
the
world.
A
Based
on
the
mission
that
we
were
being
asked
to
serve
naturally
over
time,
this
gotten
this
became
more
software
driven.
So
in
the
1990s
you
saw
the
introduction
of
computers
and
digital
technology
to
help
coordinate
resources.
This
is
very
much
kind
of
a
normal
enterprise
resource
planning
problem.
It's
like
how
do
you
get
people
in
in
in
air
assets
to
the
location
best
necessary
to
suit
the
mission
and
actually
execute
the
mission,
and,
as
of
today,
this?
This
is
more
digitally
driven
right.
A
A
More
so
folks
have
an
understanding
of
the
problem
space
that
we're
working
with.
So
we
kind
of
got
the
mission
laid
out
we're
trying
to
solve
the
command
and
control
problem
for
our
users
who
are
around
the
world.
But
if
we,
if
we
step
back
a
bit,
you
know
actually
delivering
that
command
and
control
capabilities
is
actually
fairly
difficult.
A
We
have
multiple
air
gap
networks
that
we
have
to
support
so
dealing
with
deploying
our
clusters
and
our
applications
and
our
services
and
configuration
do
controlling
all
that
configuration
controlling
all
of
that
deployment
uniformly
across.
All
of
these
networks
that
are
aircraft
is
a
very
difficult
challenge.
It
sort
of
leads
us
to
kind
of
some
of
the
solutions
that
we're
going
to
be
talking
about
a
bit
later
on
in
the
presentation.
A
We
also
have
to
support
commercial
and
on-prem
infrastructure,
we're
using
commercial
cloud
we're
using
our
own.
You
know
globally,
distributed
on-premise
infrastructure
that
we
need
to
be
able
to
support
that
we're
putting
kubernetes
clusters
on
that
we're,
putting
applications
and
services
on
and
distributing
that
workload
around
the
world
to
be
able
to
serve
the
mission.
A
We
also
need
to
be
able
to
support
on-demand
operational
and
exercise
and
test
environments.
There's
always
exercises
and
tests
being
conducted
with
our
software
that
we
don't
want
to.
We
don't
want
to
kind
of
battle
test
our
production
or
operational
environment.
We
actually
want
to
use
separate,
isolated
environments
to
be
able
to
test
and
execute
exercises
without
interfering
with
normal
operations
in
in
an
actual
real
world
use
case.
A
A
Basically,
every
system
or
information
system
inside
of
the
department
of
defense
legally
needs
this
authority
to
operate,
to
be
able
to
take
on
operational
missions
whether
it's
you
know
managing
employee
payroll
data
to
actually
conducting
air
missions.
All
of
those
systems
have
to
have
an
authority
to
operate
that
we've
reduced
the
risk.
A
We've
proven
that
the
model
is
secure
and
that
we've
done
the
controls
necessary
to
get
to
mitigate
as
much
risk
as
possible
before
we
field
this
system
in
production,
and
this
process
is
very,
very
tedious
previously,
and
so
it's
led
us
kind
of
what
we
call
the
continuous
authority
to
operate.
That
I'll
kind
of
briefly
touch
on
here
to
set
the
stage
for
gordon
to
talk
a
bit
more
about
what
we're
doing
with
flux.
A
So
traditionally
the
ato
was
was:
was
you
know
it's
something
that
you
achieved
and
got
you
know
you?
You
achieved
your
ato.
You
earned
your
ato
in
a
month
to
years
time
frame
depending
on
the
complexity
of
the
system.
It's
very
paperwork
driven,
very,
very
tedious.
Major
architectural
changes
were
kind
of
out
of
the
question
because
it
would
impact
the
ato,
but
that
ato
would
kind
of
last
three
years.
A
So
then
you
know
kind
of
where
we
are
now
in
this
phase.
One
of
continuous
ato
is
that
kessler,
one
kind
of
pushed
the
ball
forward
here
and
said
we're
going
to
reduce
that
time
to
get
an
ato
down
the
days
and
months.
We're
going
to
leverage
rapid
assessment
with
application,
security
teams
and
assessment
teams
and
we're
going
to
implement
guard
rails
and
static
scanning
tools
to
help
provide
and
identify
vulnerabilities
in
our
code
bases
before
we
actually
ship
them
to
production
and
we're
going
to
leverage
best
practice.
A
How
can
we,
how
can
kestrel
intake
what
it
did
originally
and
pushing
the
balance
of
the
continuous
authority
to
operate
forward
and
do
the
next
best
thing
and
the
next
great
thing
for
application,
security,
devsecops
and
the
dod,
and
this
is
kind
of
what
we're
talking
about
today?
How
can
we
reduce
it
from
from
days
to
months
to
shift
code
to
production,
but
in
hours
or
days?
How
can
you
show
up
on
your
on
your
first
day
to
kessel,
run
and
submit
a
bug
change
to
production?
In
that
same
day?
A
How
can
you,
how
can
we
use
techniques
like
policy
as
code
and
configuration
management
tools
like
flux
and
and
and
other
types
of
git
ops
patterns,
to
help
drive
the
types
of
compliance
and
and
configuration
management
expectations
that
we
have
in
the
platform
for
our
customers
and
our
authorizing
officials?
On
the
platform?
A
Can
we
leverage
open
source
tool
chains,
products
like
kubernetes
products
like
flux
in
our
tool,
stack
to
help
enable
more
transparency
and
more
security
within
our
infrastructure,
because
I'm
fully
I'm
on
board
with
being
able
to
leverage
open
battle,
tested
systems
that
have
had
you
know,
security
and
open
penetration,
testings
and
vulnerability
assessments?
How
can
we
leverage
more
of
open
source
contributions
in
our
system
and
actually
and
then
turn
contribute
back
to
the
open
source
community
as
we
kind
of
move
forward
down
this
path?
A
B
B
It
also
includes
maintaining
the
appropriate
access
to
the
cluster
itself,
as
well
as
applications
that
are
deployed
in
it.
One
common
factor
in
all
this
is
the
use
of
flux.
Next
slide.
Okay,
so
why
flux?
There
is
some
measure
of
additional
security
which
fits
in
nicely
with
our
devsecops
model.
Nothing
outside
the
cluster
can
update
what
is
running
in
the
cluster
flux
verifies
and
will
reject
any
git
commits
that
are
not
signed
properly.
This
prevents
unauthorized
code
from
being
applied.
B
Obviously,
we
have
a
full
audit
history
of
everything
that
is
applied
to
a
cluster
via
git
and
we
eliminate
the
configuration
drift
that
can
happen
when
you
let
folks
manually,
apply
or
tweak
things
in
the
cluster
that
are
running
in
the
cluster
next
slide:
okay,
starting
with
infrastructure,
we
are
using
cluster
api
to
manage
the
deployment
of
work
load
clusters
themselves
now.
Cluster
api
is
a
kubernetes
project
that
provides
declarative
apis
for
managing
the
life
cycle
of
their
worker
clusters.
B
B
Now
before
we
move
on,
I
would
like
to
emphasize
that
cluster
api
does
more
than
just
deploy
worker
clusters.
It
provides
no
downtime
security,
patching
and
kubernetes
upgrades.
It
can
change
instance
types,
so
it
can
scale
workers
both
horizontally
by
scaling
the
number
of
nodes
and
vertically
by
upgrading
them
to
use
larger
instance
types.
In
addition,
as
a
safety
measure
fields
in
the
cluster
resources
that
could
cause
a
cluster
failure
if
they
were
changed
accidentally
are
immutable,
so
it
prevents
that
from
happening
next
slide.
B
Okay,
so,
let's
just
take
as
an
example:
what
happens
if
we
want
to
do
if
we
want
to
deploy
a
brand
new
kubernetes
cluster?
Well,
we
kick
off
a
gitlab
pipeline
and
it
first
runs
some
terraform
that
creates
the
networking
itself.
If,
for
example,
we
were
deploying
in
something
like
aws,
this
would
include
the
vpc
itself
and
cluster
api
can
actually
do
this
step,
but
we
wanted
an
easy
way
to
encapsulate
all
of
the
best
practices
from
our
cloud
team
based
on
the
various
security
audits
they
had
to
go
through.
B
For
example,
it
deploys
a
bastion
instance.
That's
running
a
hardened
image.
It
deploys
endpoints
that
allow
one
cluster
to
talk
to
another
one,
perhaps
or
to
talk
to
various
internal
or
external
services
and
depending
on
where
it's
deployed
it
may
configure
access
to
the
environment
via
something
like
zero
trust
and
lots
of
other
things
as
well.
B
It
then
reads
the
output
from
terraform
outputs
from
terraform
and
generates
the
custom
resources
that,
when
that
are
required
for
a
new
worker
cluster,
these
are
committed
into
git
and
we
have
a
flux
instance
running
in
that
same
management
cluster.
That
will
apply
these
new
custom
resources
and
trigger
the
cluster
api
controller,
to
deploy
them
all
right
and
ending
up
in
a
new
worker
cluster.
Next
slide,
please,
okay,
so
the
workload
cluster
is
up.
We
have
a
running
workload
cluster.
B
Now,
since
we
did
not
let
cluster
api
deploy
the
networking,
we
did
that
with
terraform.
It
cannot
automatically
associate
the
control,
plane,
nodes
and
worker
nodes
that
it
created
with
the
appropriate
load
balancer
targets.
So
we
have
a
kubernetes
cron
job
running
in
the
management
cluster,
that
watches
for
new
nodes
and
does
that
for
us,
then
a
post
workload,
cluster
deployment
hook
does
the
following.
Well,
first
of
all,
it
populates
some
secrets
in
vault
that
we
need
for
the
new
cluster.
B
B
So
among
other
vault
secrets
that
are
pre-populated
are
the
private
key
and
signing
key
that
the
bitnami
sealed
secrets.
Controller
is
initialized
with
this
lets.
Our
follow-up
processes
that
we'll
talk
about
here
generate
sealed
secrets
that
can
be
committed
to
get
safely
for
subsequent
application
by
flux
and
it's
able
to
decrypt
them
successfully
all
right
next
slide.
B
Okay,
so
let's
talk
about
the
baseline
services
I
mentioned
earlier.
This
part
is
actually
kind
of
kicked
off
by
a
human
currently,
because
different
clusters
have
different
requirements,
but
it's
not
very
involved.
We
have
a
repository
that
manages
all
of
what
I
call
all
of
what
I
call
baseline
services
and
to
register
a
new
cluster.
We
just
add
it
to
a
manifest
that
specifies
what
we
want
to
be
deployed.
Some
examples
of
that
may
be
a
logging
stack.
B
This
is
a
standard
efk
stack
based
on
open
distro,
maybe
monitoring
this
is
based
on
cube,
prometheus
stack.
We
can
deploy
various
host-based
threat,
detection
agents,
enginex
ingress,
everybody
needs
an
ingress
and
potentially
a
lot
of
other
things
that
kind
of
fall
in
this
category
of
baseline
services.
That
is
things
that
aren't
deployed
by
by
the
teams
themselves.
B
When
this
is
committed
in
gitlab,
it
kicks
off
a
pipeline
that
results
in
the
creation
of
some
directories
in
git
that
are
monitored
and
applied
by
flux.
Next
slide,
please,
okay!
So
here's
a
tiny
sample
of
what
that
monitor
deployment
directly
but
looked
like
I've
deleted
a
lot
of
stuff
just
to
make
it
fit,
but
I
want
to
emphasize
that
not
only
are
we
able
to
configure
and
deploy
whatever
helm
releases
that
are
are
required,
but
we
can
also
handle
other
customizations.
B
For
example,
we
can
do
our
back
that
limits
the
scope
of
what
a
given
team
can
access
in
the
cluster,
and
this
doesn't
whether
accessing
with
cube
coop,
cuddle,
a
rancher
dashboard
or
a
service
like
kibana
or
grafana.
We
can
pre-initialize
namespaces
for
various
team
applications.
We
can
customize
the
appropriate
host
names
to
use
based
on
the
region
for
things
like
git
lab,
the
various
private
docker
registries
that
we
host
nexus
repositories
for
helm,
charts
and
other
artifacts.
B
B
This
is
just
a
tiny
sample
of
a
flux,
v1
config,
we're
actually
in
the
middle
of
upgrading
to
flux,
v2
and
the
new
helm
operator
from
the
get
ops
tool
kit,
but
I
showed
rancher
dashboard
in
an
earlier
slide.
Well,
one
of
the
things
that
likes
to
do
when,
when
it's
monitoring
clusters
is
it
likes
to
add
labels
and
annotations
to
namespaces.
B
Normally,
what
it
does
is
when
it's
doing
its
thing,
it
runs
all
the
generator
commands
concatenates
the
output
and
applies
it,
but
the
first
command
you
see
here,
for
example,
generates
nothing
to
standard
output.
Instead,
it
runs
a
script
from
the
flexpod
that
will
create
all
of
the
required
namespaces
if
they
do
not
exist,
but
otherwise
leaves
them
alone.
So
we're
able
to
let
up
control
the
initial
deployment
of
the
namespaces
and,
at
the
same
time
let
rancher
do
its
thing.
B
Okay,
next
slide,
please
all
right!
I
mentioned
team
applications
earlier
well,
we
have
a
tool
called
rad
which
stands
for
release
and
deployment
dashboard.
It
is
our
internal
self-service
dashboard
for
teams
to
use
so
behind
the
scenes
rad
will
generate.
Kubernetes
manifests
that
are
committed
to
a
specific
git
repository
for
a
given
cluster
flux
is
running
in
the
cluster.
It
tracks
these
changes
and
will
apply
them,
and
it's
been
a
great
help
with
our
application
developers
as
they
transition
from
pivotal
cloud
foundry
to
native
kubernetes
deployment.
B
Now,
as
part
of
this,
we
include
a
platform
manifest.
Let's
call
it
the
declares
various
resources
that
may
be
required
by
an
application
like
databases,
caching
services,
whatever
we
have
a
controller
running
in
the
cluster
that
will
parse
this
process,
it
and
it'll
deploy
those
various
required
resources
automatically
we'll
talk
about
them
more
in
a
sec.
B
A
rad
is
an
abstraction
point
in
the
truest
sense,
in
that
it
is
able
to
evolve
from
a
user
iteration
perspective
by
completely
abstracting
the
deployment
environment
and
soon
it
will
also
handle
deployment
to
our
air
gap.
Environments.
It'll
package
up
the
appropriate
art,
appropriate
artifacts,
it'll,
securely
transmit
them
to
the
appropriate
environment
and
we'll
be
good
to
go
over
there
next
slide,
please,
okay.
B
So
when
rad
deploys
this
manifest
along
with
the
other
ones,
the
controller
sees
this
parses
it
and
will
automatically
deploy
this
and
any
other
dependencies
that
are
requested
here.
So
it's
very
easy
for
teams
to
get
up
and
running
in
kubernetes
with
this,
I
think
on
the
next
screenshot.
Yes,
okay,
so
here
we
have
a
somewhat
redacted
screenshot
that
shows
the
development
releases
of
an
application.
We
call
krid
and
I
think,
on
the
following
screenshot,
if
I
can
see
it.
B
Yes,
this
one
again
redacted
which
showed
the
production
deployments,
including
what
versions
are
deployed
where
we
can
have
different
versions
of
a
given
application
deployed
in
different
environments.
Staging
production,
whatever
and
rad,
will
also
report
the
results
of
a
deployment
to
an
environment.
To
tell
you
if
it
was
successful
or
not,
it
will
not
only
tell
you
that
the
pipeline
succeeded,
but
also
whether
or
not
the
pods
came
up,
etc
and
michael
I'm
going
to
throw
it
back
to
you
for
the
summary.
Please.
A
A
A
To
help
drive
forward
really
really
interesting
solutions
to
really
really
complex
problems,
and
with
that
I
really
appreciate
you
guys,
letting
us
tell
you
a
bit
more
about
the
problems
that
we're
solving
and
why
these
types
of
technologies
are
very
useful
to
us
in
solving
those
problems,
and
with
that
we
will
take
questions
and
we'll
be
around
here
for
the
next
10
to
15
minutes.
I
think
thank
you
very
much.