►
From YouTube: Flux Security Deep Dive - Stefan Prodan, Weaveworks
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Flux Security Deep Dive - Stefan Prodan, Weaveworks
In this session Stefan will go deep into the security aspects of Flux v2. We'll start by explaining the Flux authorization model and how it relates to Kubernetes RBAC and account impersonation. Then we'll compare the soft and hard multitenancy models from a GitOps perspective. We'll explore the configuration options on how platform admins can lockdown Flux on multitenant environments and how they can onboard tenants onto clusters using the Flux CLI and Git. Finally we'll talk about the Flux roadmap for 2022.
A
So
my
name
is
bartek
and
I
will
be
your
moderator
today.
We
have
stefan
with
us
talking
about
flux
and
security.
I
would
like
to
remind
you
about
code
of
conduct
and
that
we
should
have
mask
unless
you
are
speaking
for
drinking,
and
I
would
like
to
say
also
that
we
have
two
seats
there
as
well.
For
two
people:
let's
go.
A
B
Welcome
everyone,
I'm
very
excited
to
be
here
two
years
of
virtual
conferences
here
wearing
me
down,
and
I
saw
so
nice
to
see
everyone
face
to
face.
So
for
those
that
don't
know
me,
I'm
stefan
prodan.
I
work
at
weworks,
I'm
a
flux
and
flagger
maintainer
for
five
years
now,
and
I'm
very
happy
to
you
know
talk
about
you
about
flux
and
the
security
aspects
of
flux.
But
first
of
all,
if
you
want
this
shirt,
please
come
by
the
flux.
B
B
Okay,
so
a
little
introduction
to
the
flux
project.
So
we
have
a
organization
called
flag
cd
which
is
under
cncf.
We
are
an
incubation
project
we
applied
for
graduation,
and
this
is
going
well
so
hopefully,
by
next
cubecon
will
be
a
graduated
project.
The
main
projects
in
the
in
the
flux
organizations
are
flux,
2.
Why
is
flux
2?
Because
when
we
started
flux
nearly
six
years
ago,
there
was
no.
You
know
there
were
no
crds.
There
was
no
operator
idea
inside
kubernetes,
upstream
so
flux
version.
B
B
Two
years
ago
we
started
working
on
flux
version
two
and
we
have
split
flux
version,
one
into
flux,
originals,
like
a
monolith
to
do
all
things
like
talking
to
git,
applying
things
on
the
cluster
and
so
on.
In
version
2,
this
repo
holds
the
cli
and
references
to
all
the
other
flux
components.
So
flux
is
now
joined,
a
micro
services
idea
where
we
have
these
specialized
controllers,
for
everything,
customer
resources
and
so
on
and
we'll
talk
about
it
a
little
bit.
B
Another
project
inside
the
flux
organization
is
flagger,
which
does
progressive
delivery
for
you,
so
it
what
it
does.
It
allows
you
to
decouple
the
deployment
process
that
flux
does
or
any
other
continuous
deployment
tool
from
the
release
process
right.
So
you
you
flagger
works
with
any
other
kubernetes
continuous
delivery,
or
you
can
also
do
I
don't
know
cattle
apply
from
your
jenkins
job
or
whatever
flagger
works
with
any
of
that.
So
flagger
is
not.
You
know
dependent
on
on
flux.
B
It
works
great
with
flux,
so
flagger
works
a
little
bit
with
with
networking,
so
you
have
to
have
some
kind
of
ingress
control
or
service
mesh
and
based
on
that
flagger
is
able
to
route
traffic.
Slowly
expose
the
new
version
of
your
app
to
your
users
and
the
magic
behind
it.
You
can
deploy
on
fridays
and
if
it
fails,
flux
roll
it's
back.
So
that's
that's
the
gist
of
it.
B
B
B
You
don't
reach
out
to
flux,
you
tell
flux
what
to
do
by
committing
yamls
into
different
sources
and,
of
course,
being
github.
We
started
off
with
git
as
a
main
source
of
truth
for
your
whole
infrastructure
or
your
clusters
that
make
up
your
fleet
and
so
on,
but
we've
expanded
flux
over
time
to
different
type
of
sources.
So
it's
not
only
about
git
and
flux,
is
moving
to
new
territory,
and
I'm
going
to
talk
about
that
a
little
bit.
B
So
the
main
sources
that
we
currently
support
today
are
git
our
helm,
repositories,
oci
container
registries
and
s3
buckets.
So
if
you,
if
you
see
s3
there
and
everything
that's
compatible
with
the
s3
apis,
for
example,
you
can
use
your
menial
cluster
stored
there.
Your
desired
state
and
flux
will
do
exactly
the
same
thing
right.
So
it's
not
only
about
git
flux
is
no
longer
only
about
git.
B
The
idea
here
of
a
continuous
delivery
solution
is
that,
okay,
you
have
the
sources,
but
you
can
also
send
events
to
flux.
So
flux
is
reactive.
It
reacts
to
all
sorts
of
events,
so
it's
not
only
about
time
pushing
something
to
git.
Then
I'm
going
to
wait
for
flux
to
notice
that
change,
you
can
ping
flux,
we
have
a
notification
controller
and
you
can
from
your
ci.
B
B
So
it
has
this
even
listener
where
you,
you
can
tell
it
to
do
things,
but
flux
will
not
do
anything
besides.
What's
describing
it
right,
if
you,
even
if
you
ping
flux,
I
do
this
stuff
flux
will
do
whatever
you
said
in
git,
so
you
cannot
control
it
from
outside.
Through
events,
you
can
only
tell
it
to
do
faster,
whatever
you
configure
it
to
do
and
another.
B
The
other
side
of
events
is
flux,
emits
events
for
everything
it
does,
and
we
have
a
component
which
issues
events
inside
the
clusters,
kubernetes
events,
but
also
outside
the
cluster.
So
in
a
way
you
push
changes
to
your
sources,
flux
does
something
with
that
change
and
the
result
of
that
change.
Flux,
can
you
know
post
it
back
to
you,
maybe
on
your
slack
channel
microsoft
teams,
page
of
duty,
whatever
even
prometheus,
alert
manager
and
so
on.
So
the
idea
here
is
that
you
have
this
continuous
loop
feedback
loop
right,
it's
not
just
okay!
B
B
So
that's
part
of
our
observability
story.
We
also
expose
prometheus
metrics,
we
give
you
graphana,
dashboards
and
so
on.
So
we
have
a
let's
say,
a
good
observability
story
outside
of
a
dedicated
ui,
and
what
I'm
going
to
talk
about
more
today
is
about
how
you
can
extend
flux
across
clusters
and
how
you
can
make
flux
aware
of
tenants
running
on
those
clusters
and
isolating
them.
So
today
we
are
going
to
talk
about
how
is
flux
made,
which
are
the
things
that
make
up
flux,
how
we
release
flux.
B
How
is
flux
dealing
with
secrets,
because
if
you
store
secrets
in
in
sources
in
git,
repos,
s3
buckets
and
so
on?
Those
secrets
might
must
be
stored,
encrypted
and
secure
and
we'll
see
how
flags
deal
with
that.
And
finally,
we
are
going
to
talk
about.
Multi-Tenancy
is
kubernetes,
is
really
multi-tenant
or
not
we're
going
to
see
how
how
flux
works
with
different
multi-tenancy
models
and
finally,
are
going
to
talk
about
what
things
are
needed
for
flux
version
2
to
reach
ga
like
a
2.0.0
release.
B
Okay,
let's
start
with
what
is
flux
made
of
I
I
said
a
couple
of
things
at
the
beginning,
but
there
are
many
many
things
making
up
flux.
So,
of
course,
we
have
kubernetes
api
extensions
called
crds
right.
You
flux
has
no
http
api.
You
everything
you
do.
You
have
to
do
it
through
the
kubernetes
api
in
a
declarative
model
by
storing
these
custom
resources
in
some
sources,
or
you
can
directly
apply
them
on
the
cluster.
B
Of
course,
all
these
custom
resources
have
to
have
something
that
operates
on
them.
So
we
have
we've
built
the
kubernetes
controllers,
powered
by
kubernetes
controller
runtime,
which
is
a
great
sdk
for
building
for
extending
kubernetes.
If
you
are
thinking
about
building
your
own
controller,
I
definitely
recommend
control
runtime
and
builder.
They
are
great
tools
made
by
the
kubernetes
community
and
they
work
really
great.
So
we
based
flux
version
2.
Only
on
upstream
kubernetes
components.
Our
controllers
are
built
with
controller
runtime.
B
B
The
command
line
tool
has
many
functions
from
generating
yamls
the
custom
resources
of
flux.
If
you
don't
want
to
type
your
yammers,
you
want
the
the
command
line,
has
an
export
create
and
an
export
function,
so
you
can
give
it
a
command
line
arguments
and
it
will
write
the
yaml
somewhere
on
disk
or
it
can
apply
direct
on
the
cluster
and
we've
seen
a
lot
of
people
building
all
sorts
of
utilities
around
it.
You
can
it's
easily.
B
B
It
works
with
the
github
gitlab
bitbucket
apis
to
allow
your
team
members
access
on
those
repos
and
it
exposes
a
simple
command
which
is
called
flux,
bootstrap
and
we
all
have
time
to
demo
it
where
you
say:
flux
bootstrap,
this
particular
repository,
and
it
sets
everything
up
now,
that's
the
first
time
and
the
last
time
the
flux
cli
has
to
touch
your
cluster.
So
after
you
run
bootstrap
what
what
that
command?
Does
it
installs
flux
on
your
cluster?
Then
it
sets
up
the
git
repo
in
such
a
way
that
flux
can
clone
it.
B
B
And
finally,
flux
is
made
up
of
many
go
and
some
c
libraries
we
we
use
all
the
kubernetes
things
we
adopted
k-status,
which
is
a
library
in
kubernetes
for
determining
the
the
readiness
of
a
resource.
So
that's
why
flux
is
able
to
after
it
applies
something
on
your
cluster
is
able
to
monitor
what's
happening
with
those
resources.
B
Okay,
I've
modified
the
deployment.
But
what
if
the
deployment
fails
with
those
changes
right
flux
is
able
to
monitor
that
using
this
library
and
will
report
back.
I
know
I've
changed
10
deployments,
one
is
failing
and
flux
will
tell
you
which
one
is
failing
and
why,
with
for
dealing
with
git,
we
in
flux
version
1,
we've
been
shelling
out
a
lot
to
the
git,
cli
and
so
on
and
from
a
security
point
of
view.
Shelling
out
to
other
binaries
is
a
very
bad
thing
right.
B
You
need
to
mount
files,
ssh
keys
and
so
on.
So
in
version
2
we
decided
that
flux
should
be
all
the
flux.
Components
should
be
a
single
process
demon.
They
shouldn't
execute
anything
they
shouldn't
have
to
have
a
shell
even
installed
in
the
container
so
for
dealing
with
git,
especially
this
is
quite
challenging
if
you
don't
use
the
cli,
so
we
had
to
use
two
libraries,
one
is
called
goget,
which
is
a
pure
go
implementation
for
the
git
protocol
version
1
and
for
the
git
protocol
version
2.
What
azure,
devops
and
aws
code
commit
uses.
B
We
had
to
use
a
libgit2,
which
is
a
c
library
made
by
the
git
team,
and,
yes,
that's
why
we
we
have
to
statically,
build
the
flux
now
and
and
and
include
all
these
c
libraries,
which
also
include
open,
ssh,
libras,
h2
and
so
on,
for
making
flux,
integrating
with
other
tools
in
the
in
the
cncf
landscape
like
customize,
helm,
menial,
oci,
encryption,
decryption
and
so
on.
We
are
also
using
only
go
libraries
of
all
the
sdks
of
this
tool,
so
we
we
don't
shell
out
to
anything.
B
We
don't
use
the
helm
binary
or
whatever
we
use
the
helm
sdk
we
use
the
customize,
sdk
and
so
on,
and
we
try
to
contribute
back
upstream
our
changes,
our
improvements
and
also
for
integrating
flux
with
cloud
providers
with
all
sorts
of
services
like
okay.
I
want
to
reach
out
to
some
container
registry,
ecr
or
whatever
right.
We,
we
use
the
upstream
sdk
for
that,
so
we
always
stay
on
the
latest
version
of
those
sdk
and
we
try
to
maintain
them
all
being
every
time
on
the
latest
stable
release
right.
B
So
we
have
all
those
things.
So
the
question
is:
how
can
a
handful
of
people
maintain
hundreds
of
dependencies
tens
of
integrations
of
across
all
the
cloud
vendors?
And
the
answer
is
quite
complicated,
so
yeah
we
dedicate
a
lot
of
time.
We
we
have
seven
core
maintainers
18
maintainers
across
all
the
all
the
ripples.
We
have
a
very
helpful
community.
Every
time
we
do
a
flux,
patch
release.
If
something
goes
wrong
in,
like
five
minutes,
someone
will
post
on
slack
or
operation.
Hey.
B
I've
tried
this
out
on
my
staging
cluster.
Something
breaks
so
yeah.
The
community
has
been
very,
very
helpful.
Slack
is
crazy,
but
yeah
there's
life
in
open
source
and
even
with
a
helpful
community.
Even
if
you
are
like
very
careful
whatever
you
do
without
automation
is
almost
impossible
to
ship
something
stable
every
time
we
do.
We
roughly
do
a
release
every
two
weeks
across
all
the
controllers,
so
there
are
like
six
controllers:
12
custom
resource
definitions.
B
This
is
flux,
release
pipeline
for
the
libraries,
the
controllers
and
the
cli.
We
we
have
unit
testing
and
we
use
amp
test
from
from
the
controller
runtime
library,
which
is
allows
you
to
run
a
kubernetes
api,
an
etcd
instance
inside
your
your
testing
process.
So
it
runs
those
processes
locally.
You
don't
need
a
container
or
anything
like
that.
So
our
unit
testing
are
targeting
m
tests,
so
we
can
test
against
different
versions
of
the
kubernetes
api,
but
is
not
a
full-blown
cluster,
so
you
don't
have
you
cannot
run
deployments
there.
B
What
you
can
do
is
test
that
your
custom
resources
and
the
logic
of
your
controllers,
the
reconciling
loop,
are
yeah
according
to
the
spec
we've
also
integrated
with
google
oss
files.
So
our
controllers,
our
libraries,
are
continuously
fuzzed
by
by
the
google
project
and
I'm
very
happy
about
it.
We
found
some
issues
that
way.
B
Of
course,
we
we
build
everything
in
all
our
all
our
controls
and
and
cli,
and
target
all
the
osc's
out
there,
all
armed
64
amd
64
arm
v6
for
raspberry,
pi
and
so
on.
And
finally,
we
do
a
full-blown
end-to-end
test
for
everything
and,
of
course,
if
all
goes
good,
we
release
either
a
library
or
no
controller,
but
that
doesn't
mean
when
we,
when
we
release
a
controller
that
doesn't
mean
you,
you
can
actually
deploy
it
very
easily
on
your
cluster.
Why?
B
Because
we
have
all
these
controllers,
but
if
we
test
each
controller
independently,
then
how
can
we
make
sure
they
work
nice
together
right?
So
that's
where
the
flux
to
repo
comes
into
play
and
we
have
we.
We
have
a
github
bot
and
what
it
does
it
assembles
all
the
controllers.
So
once
they
are
released,
the
bot
assembles
all
the
controllers,
put
them
put
them
together
in
the
flux
to
repo,
and
then
we
have
this
release
pipeline
of
the
of
flux
2
as
a
distribution
of
our
controllers.
B
Where
here
we
integrate
with
security
scanning
github
called
ql
and
and
sneak,
then
we
run
end-to-end
tests.
So
we
test
all
the
controllers
together,
the
bootstrap,
the
cli,
everything
inside
github
actions
and
we
use
the
linux
and
the
mac
os
runners.
Now
the
issue
here
with
github
actions,
it
doesn't
have
support
for
arm
64..
You
can
have
runners
only
for
amd
and
we
applied
for
credits.
B
Cncf
helped
us
a
lot
and
we
we
are
now
we
have
a
testing
grid
on
equinix
metal
that
we
can
use,
and
there
we
have
our
github
runners
deployed
there
hosted
on
on
these
bare
metal
machines,
and
there
is
where
we
we
make
sure,
and
we
run
all
them
to
enters
for
arm
64..
We
have
a
large
user
base
that
runs
flux
on
graviton,
2
and
other
on
economics
and
other
r64
clusters.
So
for
us,
r64
is
as
important
as
md64.
We
make
no
difference.
B
We
want
to
make
sure
everything
works
on
both
and,
of
course,
we
have
a
lot
of
users
home
users
which
are
running
flux
on
their
raspberry
pi's,
which
is
arm
v7,
and
for
those
we
couldn't
find
the
provider
that
will
just
give
us
raspberry
pi.
So
there
are
communities
raspberry,
pi
communities
which
help
us
they
run
their
own
tests
and
if
they
find
problems
they
come
back
to
us.
But
in
the
last
year
we
didn't
had
any
issues.
B
We
had
a
couple
of
issues
at
the
beginning
with
with
our
vixes
v6,
due
to
the
c
libraries
that
we
use
and
so
on,
but
those
are
sorted
right,
so
we
all
these
tests
are
running
using
kubernetes
kind
and
kubernetes
kind
is,
is
a
great
way
of
running
end-to-end
tests
because
it,
it
runs
a
whole
cluster
in
a
single
container.
So
you
need
a
docker
or
something
like
that
spin
up
a
whole
cluster
and
you
can
test
on
different
kubernetes
versions
right
and
we
support.
B
For
example,
flux
works
from
now
20
to
24.,
so
we
can
test
on
all
all
the
minor
versions
and
make
sure
we
didn't
break
something
because
the
kubernetes
api
changes
a
lot
and
the
client
go
changes
a
lot,
all
the
go
libraries.
So
we
we
really
try
to
make
everything
possible
to
catch
any
kind
of
issues
there.
B
And
finally,
we
do
our
cloud
end-to-end
testing
we
on
on
azure
eksgk
and,
at
the
release
part
everything
everything
that's
part
of
the
release
is
sine
with
cosign
and
we
also
publish
a
software
bill
of
materials,
and
this
is
these
are
the
release
artifacts.
This
is
what
happens
at
every
flux
release
we
have
multi-arch
container
images
signed
images,
checksums
deployment
manifests.
B
We
also
publish
open
api
specs
in
the
json
format
for
all
custom
resources
and
you
can
import
those
in
your
id
like
I
know,
intellij,
golang,
visual
studio
code
and
so
on,
and
you
have
when
you
type
the
ammos
you'll
see
you
get
auto
completion.
You
get
validation
right
into
your
id
for
all
the
flux,
custom
resources
we
publish
the
cli
binary
with
packages
homebrew
arch
and
so
on.
B
Okay,
so
back
to
security,
what
makes
the
flux
controller
secure
first,
as
I
said
before,
we
don't
shell
out
third-party
binary,
so
we
and
we
don't
depend
on
os
packages
and
so
on,
so
we
yeah
the
code
is
like
whatever
we
wrote
there.
It's
only
that
that
runs
in
your
cluster
and
then
because
of
that
we
can
seal
flux
with
we
are
dropping
all
linux
capabilities
inside
the
container.
B
The
root
file
system
is
read
only
we
use
second,
the
second
profile,
the
default
one.
The
runtime
default,
we
don't
run
as
root
our
controllers
and
we
use
kubernetes
impersonation
api
every
time
we
do
something
on
the
cluster
and
going
to
talk
about
that
later
on,
on
the
multi-tenancy
side,
now
flags
versus
the
competition
right,
as
I
said
most,
I
think
most
continuous
delivery
products
out
there.
They
allow
you
to
you
know
plug
in
some
binary
some
script,
and
this
is
how
you
you
drive
continuous
deployment
from
a
security
point
of
view.
B
We
felt
that
we
don't
want
that
into
flux
and
in
a
way
it
limits
flux
features.
We
have
to
implement
any
kind
of
integration,
because
you
cannot
just
throw
a
script
there
or
a
binary
in
the
in
the
flux
container
and
make
flux
run
that
so
basically
flux,
statically
build
no
shell
out
to
anything.
No
git,
no
cube
cattle,
no
hell!
No
nothing!
There
is
no
http
api.
B
You
can
only
control
flux
to
the
kubernetes
api,
so
our
back
and
everything
else
applies
to
everything
that
flux
does
all
the
actions
that
flux
performs
for
everything
it
does
it's.
It
creates
a
kubernetes
event.
So
if
you
enable
the
audit
log,
if
you
store
these
events
in
some
external
storage,
you
have
a
full
trace
of
what
flux
does,
and
you
can
compare
that
with
your
git
history
right
in
the
github,
so
they
say:
hey
the
orbit.
B
Log
is
the
git
history,
but
a
small
changing
git
can
reflect
in
hundreds
of
actions
on
your
clusters
right.
So
you,
if
you
store
both
both
things,
you
aggregate
them
somewhere.
Then
you
have
a
single
blast,
a
glass
of
paint
where
you
see
everything,
that's
happening.
This
git
action
triggered
all
these
events.
A
B
Is
no
way
to
extend
flux
through
plugins
or
scripting.
How
you
extend
flux,
is
you
build
new
kubernetes
controllers
and
we
we
have
a
sdk
called
the
github
toolkit.
You
can
you
get
started
with
queue
builder
and
control
runtime?
Then
you
import
our
libraries
in
your
own
controller,
and
that
makes
you
part
of
flux,
and
you
can
easily
extend
flux
layout
and
one
example
is:
weworks
has
created
a
terraform
controller
which
allows
extends
flux
to
the
terraformatory.
So
you
push
terraform
code
to
your
git
repo.
B
B
Okay,
who
trusts
flux
from
what
what
I
say
like
we
flux
is
embedded
in
a
in
other
distributions
like
asia,
arc
ukc,
anywhere
vmware,
tanzu
iq.
Also,
the
u.s
department
of
defense
and
u.s
air
force
are
using.
Flux
is
part
of
the
platform.
One
deutsche
telekom
has
a
thing
for
a
platform
for
deploying
kubernetes
at
the
edge
for
5g
and
that's
also
also
invest
flux
into
it,
and
there
are
many
more
examples
like
sap
here,
fidelity
and
so
on
so
yeah.
B
A
lot
of
platforms
opted
to
include
flux
in
their
own
platform,
because
flux
is
not
openated,
you
can
pick
and
choose
controllers.
Maybe
you
won't
only
want
to
use
flux,
source
controller
and
the
customized
one
you
your
platform
doesn't
deal
with
helm,
so
you
can
pick
and
choose
any
flux,
controller
and
it's
easily
integrated
in
your
platform,
and
I
think
that's
that's
our
strongest
point.
We
don't
have
opinions,
we
don't
say:
hey
you
only
like
this,
you
can
install
it
or
only
like
this.
B
You
can
use
it
and
that's
why
we
we
see
adoption
here,
okay,
how
securities
flux
so
people
trust
us,
but
how
secure
is
it
in
we?
We
had
the
security
audits
sponsored
by
cncf
made
by
the
logix.
They
discovered
some
things.
We
we
had
a
great
collaboration
with
them.
We
we
address
all
the
security
issues
that
they
found
and
based
on
that
audit,
we
put
in
place
an
rfc
process
and
everything
that
touches
a
flag
security.
Posture
like
we
want
to,
you
know,
add
a
new
custom
resource
that
can
have
some.
B
You
know
it
changes
the
security
model.
You
have
to
go
through
this,
this
rfc
process
right
that
means
flux.
Changes
are
not
as
fast
or
new
features,
don't
end
up
as
fast
as
you
may
want,
but
we
think
that
everything
that
you
know
goes
into
touches.
The
security
aspect
must
undergo
this
process
and
where
all
the
flux
core
maintainers,
they
have
to
prove
it
and
so
on.
So
we
have
a
process
in
place
which
is
detected
by
our
governance.
B
This
year
the
flagsteam
has
focused
a
lot
on
security
hardening.
We
have
done
our
own
internal
audits.
We
we
found
vulnerabilities
in
the
in
our
multi-tenancy
model.
We
we
patched
them.
We
improved
a
lot
the
secrets,
management
management
and
the
decryption
on
on
multi-tenant
environments
and
lots
of
other
things,
and
by
the
end
of
this
year,
cncf
tax
security
will
look
again
at
flux
and
maybe
then
we'll
release
flux.
Ga
we
really
need
to
have
our
our
story
straight
before.
That
is
flux,
bulletproof.
Of
course
not.
B
There
is
no
such
thing
as
bulletproofing
software
right.
We
even
if
I
know
some
software
product
has
no
cvs.
It
probably
it
has
cvs,
but
no
one
has
found
them.
So
since
we
launched
flags
version
2,
we
have
these
four
high
and
critical
cvs
that
are
affecting
multi-tenancy.
So
there
are
ways
of
there
were
ways
of
elevating
privileges
from
a
tenant
to
the
cluster
admin,
and
those
are
more
about
hey.
B
We
use
cubeconfigs
and
cubeconfigs
allows
you
to
shell
out
to
something
right,
so
we
have
to
strip
down
the
cube,
configs
and
disable
any
kind
of
shell
out.
Even
if
our
code
doesn't
shell
out
some
libraries
that
we
use
do
this
shell
out,
so
we
we
found
those
things
and
we
we
remove
them
and
we
sanitize
all
the
all
the
inputs
that
are
now
coming
into
into
and
make
flux
multi-tenancy.
B
How
do
you
keep
flux
up
to
date?
We,
as
I
said
we
release
very
often
every
two
weeks,
every
three
weeks,
and
it's
really
painful.
If
you
do
this
manually
so
because
flux
is
able
to
upgrade
itself,
you
can
use
our
github
action
if
you
are
using
github
what
that
github
action
does
every
time
we
release
a
new
version
of
flux,
the
github
action
will
open
a
pull
request
on
your
repo,
where
you
boost
reflex
with
the
new
flux
version.
Once
you
merge
that
flag
sees.
B
Oh,
there
is
a
new
version
of
me,
let's
upgrade,
and
if
you
are
using
gitlab,
bitbucket
or
other
solutions,
the
people
from
renovatebot,
they
created
the
an
extension
for
flux
and
it
does
the
same
thing
at
our
github
action
and
you
can
keep
flux
up
to
date,
really
easy
with
no
effort.
So
please,
if
you
are
using
flux,
consider
this
strategy
to
you
know
being
up
to
date
and
our
apis
are
stable.
They
are
at
beta,
but
we
we,
we
said
user.
B
B
Okay,
so
from
a
github's
perspective,
when
you,
when
you
have
git
or
any
kind
of
source
involved
in
your
continuous
delivery,
there
are
challenges
that
come
with.
First
are:
how
do
you
keep
your
secret
safe?
You
cannot
store
the
secret
in
plain
text
in
your
repo.
How
do
you
restrict
access
to
sensitive
data?
What
happens
if
someone
steals
your
github,
your
git
hub
token,
for
example,
they
can
log
in
right.
B
They
can
do
things
on
your
behalf
on
github
and
what
how
flux
protects
you
from
that
flux
works
with
open
pgp
and
you
can
tell
flux,
don't
trust
the
committer,
even
if
someone
has
right
access
to
a
repo.
Only
these
people,
only
these
keys,
only
the
commits
signed
with
these
keys
are
allowed
to
make
changes
on
the
cluster.
B
So,
even
if
someone
takes
over
your
github
account
by
some
means,
even
if
you
know
they
commit
as
you
any
hub,
if
you
have
tall
flux,
your
pgp
key
that
commit
will
be
verified
by
the
key,
and
if
that
verification,
failed
flux
will
not
apply,
the
modification
will
issue
an
event.
An
error
event
will
say:
hey
this
commit
is
not
authorized.
B
Okay,
so
are
my
secrets:
saving
it.
We
flux
works
with
any
kind
of
controller
or
operator
that
deals
with
secrets,
but
we
felt
like
we
have
to
come
up
with
the
solution
where
you
don't
have
any
kind
of
dependencies
to
other
controllers,
like
secrets,
hash,
core,
volt
controller
or
anything
so
we've
built
into
flux,
secrets
decryption
and
we
choose
to
integrate
with
mozilla
soaps.
B
We
are
committed
to
the
stops
development
and
maintenance.
We
are
now
we
have
made
a
couple
improvements
inside
flux
for
the
swaps,
integration,
and
now
we
are
backporting
it
to
the
upstream
and
because
of
soaps.
Basically,
the
soaps
is
a
cli.
You
use
it
to
encrypt
your
secrets
and
then
you
configure
flux
to
decrypt
the
secrets,
and
this
works
with
static
keys
for
with
age
encryption.
B
For
example,
one
of
my
favorite
tools
in
in
the
encryption
landscape,
which
is
a
great
replay
replacement
for
open
pgp,
but
we
also
work
with
open
pgp
and
what
subs
gives
us
is
the
power
of
integrating
with
all
the
cloud
vendors.
So
we
have
azure,
google,
aws,
kms
or
also
hash,
corp
integration
and
so
on.
Then,
if
sops
adds
a
new
provider
for
us,
it's
very
easy
to
extend
flux
and
and
and
integrate
with
that
provider
as
well.
B
This
is
how
secrets
operation
looks
like
have
a
public
key.
You
can
store
the
public
in
your
git
repo.
You
can
encrypt
that
you
push
it
to
the
git
and
then
flux
talks
to
either
takes
the
static
key
from
a
kubernetes
secret.
If
you
use
something
like
age
or
calls
out
to
the
cloud
kms
using
workload
identity,
so
you
don't
store
it
anywhere,
the
the
master
key
and
it
decrypts
the
secrets.
B
Okay
and
finally,
is
covenant
through
multi-tenant
right,
yes,
but
so
multi-tenancy
is
really
hard
hard
multi-tenancy,
it's
easy.
If
you
have
the
right
to
link
so
install
multi-tenancy
right,
multiple
tenants
share
the
same
cluster,
so
there
are
global
objects
like
custom
resource
definitions,
namespaces
and
so
on.
B
You,
you
cannot
let
your
tenants
control
that,
so
you
have
to
have
the
separation
of
platform
admins
or
cluster
admins,
and
tenants
and
tenants
have
to
be
restricted
to
name
spaces
in
higher
multi-tenancy
is
where
tenants
get
dedicated
clusters
and
a
dedicated
flux
instance
or
you
can
run
flux
on
your
management
cluster,
for
example,
where
you
spin
up
your
where
you
have
your
cluster
api
providers,
you
can
run
flux
there
and
flux
talks
to
the
listens
to
the
cluster
api
events.
It
detects
how
a
new
cluster
has
been
created.
B
It
takes
that
cube
config
and
provisions
all
the
tenants
workload
from
the
management
cluster
to
the
tenant
cluster.
So
we
have
two
ways
of
doing
higher
multi-tenancy
you
either
give
your
tenant
full
control
over
flux
or
the
tenant
has
no
idea.
Flux
is
even
running
there
because
flux
is
running
on
a
management
cluster
and
you
that's
how
you
peer
a
git
ripple
owned
by
the
tenant
on
their
cluster
and
about
10
isolation
boundaries.
B
So
there
are
these
two
things
and
because
flux,
github
storage
works
with
sources,
kubernetes
has
a
tenancy
model
sources
and
git
has
its
own
tenancy
model
and
flux
is
the
one
that
bridges
the
gap
between
the
two
right.
So,
as
I
said,
you
have
dedicate
dedicate
cluster
and
dedicated
report
stories.
If
you
do
hard
multi
tenancy
and
if
you
do
solve
multi-tenancy,
then
you
have
to
deal
with.
You
have
to
create
dedicated
namespaces
role,
bindings,
no
things
and
so
on.
B
Okay-
and
this
is
how
it
looks
like
you
have
sources
for
admins,
which
can
run
run
as
cluster
army
on
all
the
clusters,
and
then
you
have
git
repos
for
your
tenants
and
what
flux
does
it
uses
the
impersonation
api
and
reconciles
all
the
tenants
workloads
under
their
own
accounts?
So
they
are
highly
restricted.
They
cannot
touch
other
name
spaces
or
global
objects,
and
so
on
and
no
time
for
demo
flag.
Ga
you
can
come
by
the
booth
and
I
can
give
you
the
demo
there.
So
it's
really
easy
right:
okay,
flax,
ga!