►
Description
Solving Environment Promotion with Flux - Sam Tavakoli & Adelina Simion, Form3
Without a doubt, Flux is now one of the most popular tools for GitOps. Form3 have been using Flux extensively for PR based operations of our Kubernetes clusters, which has resulted in a great developer experience for their growing engineering teams. However, Flux Kustomize overlays have proven insufficient for Form3's complex business needs, which involve multi-cloud workloads and controlled releases between environments. As a result, the Tooling team at Form3 have written their own tool, k8s-promoter. It automates the promotion of workloads by creating pull requests which copy manifests to the target cluster. Then, Flux reconciles manifests from the directory structure and correctly promotes workloads. This talk will cover: - How the typical commit/deploy flow at Form3 looks, as well as how they use GitOps - Why the tooling Team have decided to use promotion via duplication - Architecture and design details of k8s-promoter - A demo of the deploy flow using k8s-promoter - Lessons learned and future improvements for the project Join this talk to learn from the journey of solving the problem of environment promotion at Form3!
A
Thank
you
so
much
for
coming
to
our
session.
We
really
appreciate
your
time
so
today
we'll
be
talking
about
our
current
solution
for
environment
promotion
with
flux,
so
I'm
adelina
simeon
and
I'm
a
technology
evangelist
at
form3
and
I'm
joined
here
on
stage
by
sam
tavakoli,
who
is
a
technical
architect
working
at
form
three
as
well,
and
what
we'll
be
talking
about
today
is
the
cades
promoter
project,
which
is
open
source
we've
recently
open
sourced
it.
A
A
A
Our
platform
has
a
requirement
to
deploy
on
multi-cloud,
so
our
teams
are
building
multi-cloud
stacks
and
we
needed
something
that
would
work
for
the
multi-cloud
project
and
furthermore,
this
was
something
that
we
had
to
deliver
to
unblock
the
teams.
So
we
wanted
to
build
a
simple
solution
and
then
iterate
on
it.
A
We
had
some
requirements
around.
We
wanted
to
manage
cluster
state
and
git,
and
also
we
needed
to
enforce
release
order
as
well
as
easily
diff
the
differences
between
environments.
And
finally,
we
of
course,
because
it's
such
an
important
piece
of
software,
we
wanted
to
have
end-to-end
testing
of
our
promotion
flow
as
well
all
right.
So
let
me
just
set
a
few
main
concepts
before
we
really
show
you
our
environments
and
go
into
it.
So
first
off
we
have
jurisdictions
which
are
fully
separate
accounts
for
for
data,
isolation
and
location.
A
So
our
environment
setup
is,
as
you
see
it
here,
we
have
stacks,
which
are
deployed
on
multi-cloud
on
aws,
gcp
and
azure,
and
promotion
goes
in
order
from
dev
to
test
to
prod.
There
is
a
one-to-one
relationship
between
an
environment
and
a
stack,
so
the
same
stack
cannot
exist
in
multiple
environments
and
we
have
multiple
jurisdictions
that
can
you
know
as
many
as
we
need.
So
we
are
talking
about
managing
many
multi-cloud
stacks,
but
this
talk
is
more
about
like
what
happens
after
the
ci
flow.
So
rci
flow
is
is
standard.
A
Our
engineers
raise
prs,
we
have
the
code
review
and
then,
at
the
very
end,
of
course,
you
have
the
ci
doing
the
triggering
the
build
and
testing
it
and
at
the
end
we
have
a
new
image.
That's
pushed
the
registry,
but
how
do
we
move
all
of
these
changes
in
all
of
the
environments
that
we're
talking
about
the
team
considered
three
main
solutions
which
you
see
here
so
first
we
could
have
used
customized
overlays,
we
could
maintain
separate
branches
for
each
cluster
or
we
could
go
for
a
directory
structure.
A
So
the
solution
that
we
decided
to
go
for,
based
on
our
experience
in
our
circumstances,
was
to
go
with
a
directory
structure.
Use
and
use
it
together
with
manifests
manifest,
would
give
us
an
immutable
artifact
that
could
we
could
then
promote
across
environments,
and
we
didn't
consider
templating,
because
we
wanted
to
keep
a
low
barrier
for
adoption.
As
we
said,
we
wanted
to
unblock
the
teams
and
then
iterate
on
another
on
on
this
full
solution.
A
So
this
is
the
hierarchical,
the
hierarchical
directory
that
represents
our
environments,
so
the
manifests
folder
is
the
source
of
truth
and
changes
to
the
manifest
folder
will
trigger.
The
promotion
and
changes
in
here
could
be
either
done
by
pr
or
using
flux's
image
or
update
automation,
and
each
cluster
has
its
own
has
its
own
directory
from
which
it
reconciles
from,
and
then
you
also
have
stacks
that
have
their
workloads
in
their
states.
In
this
directory
and
sam
will
show
you
more
about
the
directory
structure
and
how
it
works.
A
A
B
No
okay,
oh
no
right!
So,
as
edina
said,
we
wanted
to
unblock
the
internal
multi-cloud
efforts
very
early
on
and
build
on
top
of
it.
We
didn't
want
to
invest
in
changing
the
whole
ci
flow.
We
didn't
want
to
invest
in
replacing
that
with
anything
new
in
that
way,
so
we
kept
the
developer
experience
of
our
old
approach,
which
was
the
pr
approach
and
quite
familiar
to
us.
B
We
also
wanted
to
codify
our
release
and
conventions,
as
in
the
ordering
of
what
environment
needs
to
get
targeted,
and
we
wanted
to
make
sure
of
the
changes
that
we
do
since
the
sort
of
template
of
the
way
that
we
do
releases
wasn't
very
specified
the
early
days.
We
wanted
to
be
able
to
move
quickly
and
change
how
we
worked
by
being
able
to
test
our
promotions.
B
So
as
part
of
this
tool,
we
also
wrote
a
wrapper
one
go
get
to
implement
the
http
backend,
so
we
can
kind
of
visualize
not
visualize,
but
assert
on
git
state
over
the
http
wire
using
our
own
sort
of
tests
and
the
tool
works
with
single
and
multi-tenant
setup,
and
we
will
focus
mostly
on
the
single
tenant
approach.
So
I
will
go
through
the
sort
of
flow
with
case
promoter
at
the
moment
that
we
have
and
then
what
we'll
show
a
little
bit
overview
of
how
the
two
works.
B
We
also
have
flux,
image,
automation,
controller
that
sits
here
scanning
our
oci
registries
and
bumping
the
versions
of
flux
manifest,
and
this,
of
course
doesn't
go
through
a
pr
process,
but
it's
merging
to
master.
But
then
it
triggers
aci
flow,
which
the
review
process
here
will
do
as
well
after
it's
been
merged.
B
So
once
the
pr
has
been
merged
in
the
tenant
repo,
it
will
kick
off
the
ci
job
and
it
will
essentially
have
three
tasks:
promoted
development,
promote
the
test
and
promote
the
production
which
will
kick
off
the
case
promoter,
given
these
parameters,
so
the
parameters
being
to
what
target
environment
you
want
to
promote
when
the
tool
runs.
The
first
thing
it
does,
though
it's
it
clones
or
actually
checks
that
master
on
another
repo,
which
is
infrastructure,
cluster
config,
which
is
essentially
where
we
have
a
a
cluster
definition
across
our
fleet.
B
B
So
when
the
binary
runs
for
promotion
to
dev,
it
will
group
all
the
changes
for
the
dev
clusters
in
one
go
and
raise
a
pr
where
the
author
proves
it
for
the
test
test
task.
What
it
will
do,
it
will
observe
that
there
hasn't
been
any
promotion
to
development
then.
Hence
it
won't
actually
do
any
promotion,
so
it
just
stops
execute
executing
there.
B
The
same
happens
for
promotion
and
then
once
that
pr,
the
phrase
to
development
has
been
raised,
it
will
and
once
it's
been
merged,
it
will
trigger
another
merge,
ci
job,
which
will
then
again
raise
three
tasks
with
for
the
case
promoter
and
once
and
as
it
has
observed,
the
change
has
now
been
done
in
development.
It
won't
do
anything,
however,
for
the
test
clusters,
as
we
have
requirements
on
being
able
to
raise
on
conducting
the
changes
one
by
one
in
that
environment.
B
It
will
raise
three
individual
prs
where
the
author
has
to
then
merge
them
and
then
wait
for
the
production,
and
this
is
essentially
repeat,
repeats
for
production
once
all
of
those
changes
have
been
rolled
out
to
test.
So
if,
if,
for
instance,
only
one
test
cluster
had
been
merged,
it
wouldn't
raise
the
production
pr
until
it's
observed
the
changes
across
the
whole
state
given
infrastructure
cluster
config.
B
So
I
will
go
to
the
brief
overview
of
the
tool
itself.
The
two
most
important
aspects
is
the
fact
that
which
target
environment
is
and
the
commit
range.
It
goes
through
the
clone
step,
which
is
essentially
getting
the
state
of
the
tenant
repository
and
then
detect
what
changes
has
been
happening
to
the
workload
and
then
there's
a
filter
step
with
excludes,
given
the
worker
exclusions
and
then
also
checking
the
previous
previous
environments,
and
then
it
will
promote
it.
B
But
the
actual
step
of
the
detection
is
it's
using
gold
kit
and
we
do
diff
tree
over
the
commit
range.
We
then
get
the
sort
of
the
git
state
of
what
actually
has
changed.
We
infer
the
workload
changes
upon
this
to
figure
out
all
right,
given
certain
changes
to
manifest
inside
of
these
workload
directories,
the
following
actions
need
to
happen
and
we
take
those
sort
of
workload
operation
and
pass
it
on
to
the
filter
step,
which
then
observes.
B
Given
these
changes
to
the
workload
and
given
the
work,
exclusion
rules
should
it
be
allowed
to
be
promoted
upwards
or
not,
and
we
check
the
validate
the
previous
environment
as
well
and
as
part
of
the
promotion,
is
essentially
grouping.
The
cluster
checking
out
the
new
branch
and
applying
the
changes
commit
and
push
and
raise
the
pr
as
the
tool
does
for
us
and
that's
about
it.
So
as
part
of
the
demo,
let's
see
if
this
works,
I
pray
to
the
democrats.
B
Let's
see
I
raised
the
pr
here
where
essentially
it's
it's
quite
simple:
let's
see
if
I
can
show
the
yeah,
that's
not,
essentially
I'm
raising
a
change
to
flux.
Man.
Can
you
see
this?
Is
this
readable
logic,
okay,
cool
that
I'm
introducing
a
change
to
just
the
version,
it's
very
simple
to
the
flux,
manifest
demo
services
and
I
am
going
to
merge
this.
This
will
then
kick
off
the
jobs
that
I
mentioned
earlier
and
hopefully
github
will
be
responsive
and
we
can
see
for,
for
instance,
promoted
development.
B
Let's
give
you
two
more
sets,
I
think
the
other
ones
actually
have
gone
through
yeah,
so
for
the
promoted
development,
we
will
see
that
it
will.
These
are
the
options
going
into
it.
It
will
begin
the
promotion
and
we
are
promoting
the
four
clusters
in
one
pr
and
it
will
actually
raise
it
to
github
right
for
test.
What
it
will
say
is
it
will
observe
the
fact
that
it
will
drop
the
change.
B
It's
observe
the
change,
but
it
will
say
that
if
the
source
is
not
the
development
as
expected,
because
we're
promoting
to
test
and
therefore
it
won't
do
anything
and
the
same
goes
for
production.
So
if
we
now
go
to
the
pull
requests,
we
now
see
an
overview
page
which
essentially
links
to
the
actual
change
that
was
that
caused
the
promotion
to
kick
in.
B
We
have
an
overview
of
what
clusters
have
been
affected,
which
which
workload
it
is
and
if
we
had
more
changes
in
multiple
workloads,
we
would
actually
enumerate
them
as
new
columns
here,
and
we
also
introduced
to
ping
the
author
so
that
he
actually
sees
that
something
is
being
raised.
If
we
check
the
differences,
it's
essentially
the
same
version
change
being
promoted
to
all
of
the
clusters
that
we
can
see
right
so-
and
this
is
done
by
the
clusters.
B
If
it
doesn't,
then
essentially
I
would
just
have
to
show
previous.
Oh
there
we
go
so
here
we
can
see
that
it
is
beginning
the
promotion
it
started
to
promote
to
test
internal
aws.
It
raises
a
specific
protocol
for
that
and
and
then
it
goes
on
and
raises
the
other
pos
and,
as
you
can
see,
is
not
the
fastest
here,
but
this
is
something
that
we
introduced
to
be
able
to
rate
limits
on
the
apis,
and
I
will
explain
more
about
that.
B
B
If
some,
this
binary
is
doing
it
for
you,
if
you
leave
a
release
and
come
back
to
it,
it's
not
always
easy
to
understand
at
what
particular
state
it
is
in,
and
that's
one
aspect
that
we
have
problems
with
and
also
the
github
rate
limits.
As
I
mentioned-
and
this
is
due
to
the
fact
of
we
could
have
n
number
of
works
concurrently
running
and
we
wouldn't
be
able
to
control
that.
So
therefore,
this
causes
a
bit
of
pain.
B
We
had
to
artificially
induce
like
regulate
ourselves,
given
the
sort
of
responses
from
github,
and
also
the
fact
that
we
would
have
to
manage
the
maintenance
efforts
to
bump
the
actual
case
promoter
version
across
the
many
tenants
which
isn't
too
hard.
But
it
still
is
labor
and
we
don't
like
doing
labor.
B
So
what
we've
started
looking
at.
Well,
I'm
not
part
of
tooling
anymore,
I'm
in
a
market
team,
but
what
they're
doing
now
is
experimenting
with
pipelines
and
essentially
keeping
the
sort
of
initial
change
to
be
pr
by
the
cab
process
and
making
sure
that's
properly
reviewed.
But
past
that
point.
It's
essentially
is
governed
by
the
pipeline
itself.
B
So
you
don't
need
to
kind
of
review
the
state
of
the
prs
to
be
able
to
induce
what
sort
of
cluster
the
change
has
gone
on
to,
and
here
in
this
example
is
we
produced
a
pipeline
using
git
github
actions
and
the
sort
of
development
clusters
have
been
released
and
somebody
has
approved
to
deploy
it
to
a
specific
integration,
environment
and
yeah.
That's
pretty
much.
It
come
chat
to
us
at
our
stand.
We're
here
and
yeah
check
out
the
tool.
B
This
is
not
what
kate
promoter
would
do,
but
we
have
had
the
thought
of
having
essentially
the
next
iteration
of
it
would
be
the
pipelines
and
then
building
a
progressive
deliver
on
top
of
that.
But
no
it
would
it's
not
doing
that
right
now
for
what
you're
describing,
but
nothing
really
stops
us
from
being
like.