►
From YouTube: GitOps, Kubernetes, and Secret Management: Don't Bake in a Tent! - Kara de la Marck, CloudBees
Description
GitOps, Kubernetes, and Secret Management: Don't Bake in a Tent! - Kara de la Marck, CloudBees
GitOps uses Git as the “single source of truth” for declarative infrastructure and enables developers to manage infrastructure with the same Git pull requests they use to manage a codebase. Having all configuration files version-controlled by Git has many advantages, but best practices for securely managing secrets with GitOps remain contested. Join us in this presentation about GitOps and secret management. Attendees will learn about the pros and cons of various approaches and why the Jenkins X project has chosen to standardize on Kubernetes external secrets for secret management.
Introduction: (0:00)
Why Git?: (1:00)
Why Kubernetes ?: (2:50)
What's a Secret?: (5:13)
Security Concern in Git: (6:58)
Strategies for Managing Secrets: (8:58)
To Avoid Storing Secrets in Git: (11:25)
Rules: (13:16)
External Secret Management Systems: (14:34)
Using Jenkins X: (17:23)
Jenkins X 3 Alpha and Community Links: (18:45)
Wrap-up: (19:15)
Q&A: (20:40)
A
Hi
welcome
to
my
talk,
get
ups
kubernetes
and
secret
management.
Don't
bake
an
intent,
I'm
cara
de
la
milk.
I
work
at
cloudbees,
mostly
with
the
jenkins
x,
open
source.
Community
jenkins
x
is
a
ci
cd
platform
on
kubernetes,
with
githubs
built
in
and
committed
to
facilitating
github's
best
practices
in
this
talk
we'll
be
discussing
some
of
the
difficulties
of
secret
management
with
git,
ops
and
kubernetes.
A
A
This
means
all
infrastructure,
configuration
and
application
code
are
stored
in
git
repositories.
Infrastructures
code
isn't
unique
to
git
ops,
but
is
foundational
for
get
ops
with
good
ops.
The
entire
state
of
your
system
is
described
using
declarative
specification
for
each
environment,
and
these
are
start
and
get
so
a
key
part
of
get
ops.
A
Is
this
idea
of
environments
as
code
describing
your
deployments
declaratively
using
files
such
as
kubernetes
manifests
stored
in
a
get
repository
and
this
having
infrastructure,
config
and
deployments
declaratively
described
in
files
and
get
raises
issues
or
secret
management
and
secret
management
with
gaps
in
kubernetes?
Is
this
talk,
but
if
you
would
like
to
learn
more
about
get
ops
get
ups,
the
term
was
coined
by
weaveworks
and
recommend
going
and
looking
at
their
writing
on
the
subject.
A
A
The
other
part
of
get
ops
which
is
important
to
keep
in
mind
is,
is
that
it's,
you
have
to
be
able
to
observe
the
actual
state
of
your
cluster
and
the
state
of
your
system
and
reconcile
that
actual
state
to
your
desired
state,
as
defender
can
get
so.
Kubernetes
helps
a
lot
with
this.
It
handles
active
reconciliation
within
your
cluster,
so
kubernetes
was
open
source
from
google
and
is
now
the
most
popular
container
orchestration
platform.
A
A
First,
automation,
it
automates
the
process
of
applying
changes
correctly
and
in
a
timely
manner,
and
then
with
convergence
kubernetes
will
keep
trying
to
update
until
the
update
is
successful
in
item
projects.
Multiple
applications
of
convergence
have
the
same
outcome.
Conversion
actions
can
be
applied
multiple
times
without
changing
the
result
beyond
the
initial
desired
outcome.
A
A
A
This
is
an
example
of
a
simple
kubernetes
secret
is
a
simple
data
structure
composed
of
three
main
pieces
of
information,
the
name
of
the
secret,
the
type
of
the
secret
which
is
optional,
and
a
map
of
field
name
to
sensitive
data
encoded.
In
base
64.,
when
you
first
see
a
kubernetes
secret,
you
may
be
tempted
to
think
that
its
values
are
projected
by
encryption.
A
A
The
reason
kubernetes
does
basic
c4
encoding
is
so
that
secrets
can
store
binary
data,
and
this
is
important
for
things
like
storing
certificates
as
secrets,
you
do
not
want
to
store
kubernetes
secrets
in
get
the
basic
c4
encoding
is
like
plain
text.
So,
as
we
can
see
here,
the
secret
data
easily
unencoded
is
reading
username,
let's
all
join
and
password.
I
need
cake,
so
you
do
not
want
your
secrets
laid
out
and
get
like
an
open,
buffet
get
ops.
A
Now
there
are
additional
reasons:
it
is
a
security
concern
to
store
secrets
and
get
git
was
designed
as
a
collaborative
tool,
making
it
easy
for
many
people
to
view
and
review
each
other's
code.
It
was
designed
to
enable
access
to
other
people's
code,
and
for
these
reasons
it
is
very
dangerous
to
use
get
to
hold
secrets.
A
So
when
you're
dealing
with
secrets,
you
ideally
should
be
grinding,
read
access
to
secrets
on
a
need
to
know
basis
and,
for
example,
if
you
have
a
temporary
worker,
you
would
want
to
give
that
new
user.
The
least
amount
of
access
to
the
sensitive
data.
But
if
you,
but
with
git
repos,
it's
all
or
nothing.
So
if
they
have
access
to
your
git
repo,
they
have
access
to
everything
in
that
get
repair
also
distributed
gear
requests.
A
So
with
git
ops
team,
your
members
are
going
to
locally
clone
the
get
repo
to
their
laptops
and
workstations,
and
this
would
mean
the
proliferation
of
secrets
to
many
different
systems
and
opening
up
the
surface
of
attack.
So
if
any
of
those
laptops
were
stolen
or
workstations
compromised,
then
your
secrets
would
be
open
to
attack
okay,
so
our
one
rule
for
get
outs,
kubernetes
and
secret
management
is
don't
store,
kubernetes
secrets
and
get,
but
we
are
trying
to
do
git
ups
and
have
the
entire
state
of
our
system
again.
A
Remember
right
now
we
have
one
rule
for
get
ups
and
kubernetes
and
doing
secret
management,
okay,
which
is
do
not
store
kubernetes
secrets
and
get
to
help
us
with
the
conundrum
of
what
we
do
with
our
secrets.
There
are
a
number
of
different
strategies
for
secret
management
and
they
have
trade
us
around
flexibility,
security
and
manageability
at
the
end
of
going
through
these
different
strategies,
we'll
look
at
the
secret
managing
strategy
that
jenkins
x
has
adopted.
So
the
first
first
strategy
of
secret
management
is
to
break
the
rule.
A
A
A
Okay,
it's
kind
of
like,
if
you
break
to
your
chocolate
cake,
all
right
and
then
the
next
morning.
You
want
to
have
cake
with
your
coffee,
but
you
want
a
vanilla
cake.
Well,
you
can't
just
take
the
chocolate
over
the
cake.
It's
done.
You
have
to
go
bake.
A
vanilla
cake
and
similar
third
problem
is
that
the
secrets
contained
that
are
based
in
container
tend
to
be
less
flexible.
A
A
So
now
we
have
kind
of
two
rules
for
don't
bake
in
it.
Cake
and
a
tent
are
rules
for
get
ups
kubernetes
and
secret
management,
one
don't
store
raw
kubernetes
secrets
and
get
and
two
don't
bake,
sensitive
data
directly
into
your
container
images.
Okay
and
really.
The
second
rule
here
is
more
of
a
guideline
because
you
can
do
it,
but
there
are
a
lot
of
trade-offs
around
security,
flexibility
and
convenience.
A
Okay,
so
really
we
just
have
one
rule,
which
is
don't
store,
raw
kubernetes
secrets
and
get
and
as
you've
seen
using
secrets
is
optional
kubernetes,
but
it
is
much
more
convenient,
flexible
and
secure
than
the
other
techniques,
such
as
baking.
The
sensitive
data
directly
into
your
image,
since
secret
object
is
convenient
to
use
as
well.
It
provides
a
declarative
api
that
makes
it
easy
for
application
pods
to
access
the
data
without
any
special
code.
A
So
there's
a
lot
of
convenience
with
using
kubernetes
secrets,
and
so
we
want
to
use
kubernetes
cigarettes,
but
how
to
do
so
securely
now.
One
approach
is
to
say
outsource:
all
your
baking
needs
to
those
with
access
to
a
professional
kitchen
rather
than
trying
to
bake
in
a
tent
and
in
the
world
of
secret
management.
This
means
using
an
external
secret
management
system,
and
these
professional
bakers
can
do
a
lot
for
you.
A
They
can
securely
store
manager
secrets,
you
have
identity-based
access
to
sensitive
information
data,
encryption
in
flight
and
at
rest,
and
there
are
a
variety
of
external
secret
management
systems
that
exist.
One
of
the
most
popular
is
hashicorp
vault
and,
in
addition,
the
individual
cloud
providers
also
offer
secret
management
services.
A
So
we
have
like
google
seeker
manager,
aws
secret
manager,
xeros
keyvault,
alibaba's
clouds,
secret
manager,
and
they
might
differ
a
bit
in
features,
but
the
general
principles
are
the
same.
So
in
this
strategy,
rather
than
using
native
kubernetes,
features
to
store
and
load
secrets
into
a
container,
the
application
containers
themselves
retrieve
the
secret
values
dynamically
at
runtime.
At
the
point
of
use,.
A
However,
when
using
an
external
secret
store,
is
the
responsibility
of
the
application
to
retrieve
the
secrets
from
the
store
securely
and
often
teams
develop
custom
solutions
for
accessing
secret
data.
The
result
is
that
there
are
overlapping
efforts
for
developing
these
solutions
and
because
security
is
vital.
In
this
context,
each
individual
effort
can
require
substantial
engineering
time
and
so
all
that
blank
gray
space
in
the
diagram.
A
An
external
secret
declares
how
to
fetch
the
secret
data,
while
the
controller
is
responsible
for
fetching
secret
data
from
external
providers
and
converts
all
external
secrets
to
secrets.
The
conversion
is
completely
transparent
to
pods
that
can
access
the
secrets.
Normally
kubernetes
external
secrets
supports
the
external
secret
management
systems
of
all
the
major
cloud
providers
as
well
as
hashicorp
okay.
A
Then
the
controller
upsorts,
those
secrets
and
the
pods
could
then
access
the
secrets,
normally
or
kind
of
give
that
away,
but
jenkins
x
is
using
is
recommending
usage
of
kubernetes
external
secrets
number
one.
We
obviously
don't
want
you
to
commit
raw
kubernetes
secret
yaml
into
get,
and
you
can
use
the
old
secrets,
but
we
do
highly
recommend
using
kubernetes
external
secrets,
which
means
you
can
check
in
external
secret
resources
into
git,
which
are
a
reference
to
the
actual
secret
values
from
the
sun
provider.
A
Okay
and
jenga
zach
supports
the
major
external
secret
management
providers,
including
hashicorp
vault
amazon,
secret
manager,
google's
secret
manager,
alibaba
cloud
secret
manager
as
your
keyboards
okay.
So
the
jing
contacts
team
are
big
fans
of
kubernetes
external
secrets
and
to
what
we
developed
jx
secret,
which
is
a
small
command
line
tool
for
working
with
kubernetes
external
secrets
that
integrates
really
nicely
with
the
jenkins
x
workflow
for
continuous
delivery
on
kubernetes.
A
So
jing
index
has
chosen
a
strategic
management
approach
that
adds
an
abstraction
layer
above
secret
management
solutions
from
external
providers.
So
users
can
choose
where
to
where
they're
going
to
store
the
source
of
their
secrets,
and
this
will
be
preferably
outside
of
their
kubernetes
cluster,
and
this
is
very
good
practice
for
disaster
recovery
scenarios.
A
So
here
are
some
more
links
to
discover
more
about
jenkins
x3
alpha
how
it
differs
from
jenkins
x2,
some
of
the
architectural
changes
that
have
been
made
and
why
go
check
it
out.
It's
great
project,
and
also
here,
are
some
links
for
finding
out
how
to
become
more
involved
with
the
changing
sex
community.
You
can
join
us
for
our
bi-weekly
office
hours,
our
slack
channels,
discourse
we
have
sigs
and
also
we're
going
to
pursue.
We
are
participating
in
oktoberfest,
okay,
so
very
quick
wrap
up.
A
A
A
Kubernetes
external
secrets
does
a
great
job
of
securely
solving
this
problem
now,
above
all,
our
number
one
rule
for
secret
management
with
good
ups
and
kubernetes
is
if
our
don't
bake
in
the
tank
guides
is
don't
store
raw
kubernetes
secrets
and
get
if
you
take
nothing
else
away.
That
is
the
rule
to
remember.
A
Thank
you
for
listening
and
watching
this
talk.
Thank
you
all
for
your
for
your
nice.
Compliments,
that's
very
kind,
and
I'm
not
sure
everyone
here
knows,
but
on
the
slack
channel,
there's
there's
myself,
but
there's
a
number
of
people
who
are
really
involved
in
the
jenkins
x
project
as
well.
So
if
you
have
any
questions
about
either
get
ups
secret
management,
kubernetes
or
specifically
with
jenkins
x,
this
is
a
really
nice
time
to
ask
them.
If
you
would
like.
A
So
we
have
some
some
comments
around
some
of
the
security
issues
of
this,
for,
for
example,
baking
secrets
is,
is
can
really
bite
you,
which
it
definitely
most
definitely
can
and
then
also
a
question
on.
How
does
the
external
sequence
controller
know
when
to
fetch
and
update
secrets
and
james
jackson's
answers?
It
pulls
by
default
and
then,
after
the
pulling?
Yes,
it
varies
from
the
back
end
and,
and
so
that
varies
from
from
your
back
end
surface,
that
you've.
A
A
Okay,
what
what
you
definitely
want
to
do,
though,
if
you're
rotating
them
in
certain
ways,
maybe
your
secret
manager
can
help
you
with
that
when
you're
rotating
the
secrets,
if
you're
doing
it
manually-
and
you
have
those
secrets
baked
into
your
image-
it's
going
to
be
a
lot
of
work
for
you.
You
have
to
rebuild
your
images.
A
Rakesh
and
and
james
strachan
are
having
a
very
good
conversation
actually
in
the
chat
I'll
just
relay
it
to
the.
If
this
is
being
recorded
or
not,
james
has
said
that
it's
very
good
to
try
and
minimize
per
environment
changes
so
using
a
canonical
secret
name,
vault
location,
which
is
relative
to
the
environment,
minimizes,
getting
the
wrong
locations
and
then.
A
Steven
has
a
question
of
jx
secret:
is
it
a
jx
plugin
which
indeed
it
is,
and
it
actually
points
to
the
fact
that
we
have
a
new
or
sort
of
an
evolved,
extended,
plug-in
architecture
with
jenkins
x3,
which
is
is
actually
so?
The
core
of
jenkins
x3
is
actually
quite
small
and
most
a
lot
of
additional
functionalities
provided
with.
A
Plugins
I'll
just
I'll
just
say
this.
James
rollins
has
added
that
for
google
secrets
manager,
external
secrets
can
also
pull
secrets
from
different
gcp
projects.
So
using
things
like
workload,
identity
to
to
control
your
access
to
retrieve
the
source
of
the
secret
as
well
to
control
access
to
retrieve
the
source
of
the
secret
is
also
well
controlled.
A
A
Okay,
thank
you
all.
Do
you
check
out
the
links?
I
think
you
have
the
slides
in
the
the
top
format.
Maybe,
but
it
would
be
great
to
have
to
see
you
at
the
jenkins
ex
office
hours
or
on
our
slack
channels
or
even
in
one
of
our
sigs.
If
that
interests
you
and
yeah
get
involved,
make
some
pr
sector
professors
happening,
but
but
in
general,
be
great
to
have
your
contributions,
so
I
hope
I
see
more
of
all
of
you
and
thank
you
for
coming
to
this
talk.