►
Description
GitOps Everything!? We Sure Can!, Ayelet de-Roos, AppsFlyer
Usually when developers mention GitOps, it is to describe yet another continuous delivery mechanism for their (micro-)services. There are developers that to some extent apply it to their infrastructure, but can it describe both? Can you also apply it to your SaaS solutions? At Appsflyer, with an architecture of over 850 micro-services, thousands of cloud resources and dozens of SaaS integrations, we strive to automate them all using GitOps workflow. In this talk, Ayelet de-Roos will present how AppsFlyer applies Terraform with GitOps to automate everything, how AppsFlyer keeps its deployables safe and easy to manage with Flux CD, how AppsFlyer identifies drifts in non-Kubernetes environments, and how to enforce company standards and policies using Open Policy Agent.
A
A
A
A
A
We
first
started
to
look
at
the
daily
operations.
We
found
that
they
are
very,
very
fragmented,
meaning,
for
example,
a
user,
a
developer
one
wants
to
commit
a
single
commit
to
get
developed,
develop
a
single
feature.
He
needs
to
go
through
the
git
through
a
cia
ci
system,
a
test
system.
He
needs
to
go
through
a
different
service
for
the
deployment
of
the
service
and
a
different
system
for
monitoring
his
service
and
production.
A
Very,
very
fragmented:
we
looked
at
the
developer
autonomy
as
I
said,
our
teams
work
in
squads
and
meaning
they
should
work
in
full,
full
autonomy,
and
we
saw
it's
not
really
the
case
a
lot
of
times
that
developers
need
help
and
assistance
from
the
platform
team.
They
don't
know
how
to
build
basic
infrastructures.
They're
missing
permissions.
I
can
say
as
a
platform
engineer,
that
we
get
multiple
tickets
from
developers
that
they
don't
know
how
to
do
something
or
they're
missing
permissions,
and
then
they
need
to
wait
for
us
to
give
them
permission
or
to
assist.
A
A
So
why
did
we
choose
git
ups
as
a
solution
to
approach
these
pain
points,
most
a
review,
or
probably
after
this
day.
All
of
you
know
that
the
four
goals
of
the
git
ups
is
that
she'll
be
declarative,
versioned
and
immunoable
pulled
automatically
and
continuously
reconciled.
We
should
always
get
to
the
desired
state
the
user
wants
for
his
service.
A
We
saw
that
the
goals
suit
our
needs
and
we
decided
to
base
our
solution
and
the
github's
principles.
We
created
our
own
solution
principles.
We
wanted
to
be
intuitive
very
easy
for
the
user
to
adapt
and
understand
how
to
use
everything
should
happen
in
a
single
source
of
truth.
In
the
same
place
with
not
not
fragmented
over
the
place
self-serve,
we
want
full
autonomy
for
the
user.
He
should
know
how
to
handle
a
service
by
himself.
A
A
We
looked
at
a
developer's
experi
day.
As
we
see
it
in
appsplatter
developers
flow
in
apply.
We
have
developer
a
and
developer
b
developer
a
he
commits
some
code
changes
and
declares
some
service
deployment
plan
and
committed
to
git.
You
see
gitlab
here,
because
this
is
what
we
use
in
appsplayer
and
developer
b.
She
commits
some
infrastructure
requirements
and
some
page
repeated
policies.
They
both
commit
and
push
to
the
same
git
repository
then
without
getting
still
into
details.
A
The
github's
workflow
comes
into
account
knows
how
to
take
all
the
changes
that
was
just
committed
to
the
git
repositories
and
apply
the
changes
and
communicate
it
to
all
of
our
resources
from
kubernetes
deployments
until
sas
integrations
everything
we
want
to
cover
everything.
Everything
should
be
automated
by
us
with
the
github's
workflow,
so
kubernetes
deployments
cloud
cloud
services
and
sas
integrations
like
datadog
monitors.
A
We
decided
to
create
an
emitted
at
a
folder.
Maybe
some
of
you
are
familiar
with
the
metadata
folder,
because
the
cncf
already
started
using
it,
but
we
created
also
our
own
unique
one
that
will
suit
our
developers
and
apps
players
needs.
As
you
know,
developers
don't
like
to
change
the
repositories
to
change
their
services,
so
we
wanted
to
create
something
very
clear,
very
easy
to
adapt.
A
So,
like
you
see
here
we
have
a
dot
af
folder.
Under
it
we
have
three
subfolders
actions,
deployments
and
environments
folders
in
the
action
folder
we'll
defined
in
each
subfolder,
a
different
action
for
the
service,
for
example,
a
build
folder
that
will
define
how
to
build
the
service.
Then
we
have
the
deployments
folder.
A
Each
service
can
have
multiple
deployments,
because
a
service
can
have
sas
deployments
like
data
talk,
monitors
or
or
page
or
duty
policies.
It
can
have
also
other
services.
It
needs
for
b
for
being
deployed,
so
every
deployment
will
have
its
own
subfolder.
That
will
define,
define
how
this
service
should
be
deployed.
A
And,
lastly,
we
have
the
environments
folder
that
will
define
defined
general
environments
for
the
service,
for
example,
and
which
region
to
deploy
which
availability
zone.
Like
you
see
all
the
files
are
terraform
files
declarative
language.
If
we
look
inside
one
of
them,
if
you
look
inside,
you
can
see
that
in
with
terraform,
we
can
also
provide
platform
models.
We
provide
for
the
developers
we
now
for
bringing
up
a
kafka
cluster.
The
user
doesn't
need
to
know
anything
ali
needs
to
say.
I
want
a
kafka
version
with
this
version,
and
that's
it.
A
A
A
What
can
go
out
and
by
that
we
came
and
found
open
policy
agent
oppa,
which
showed
that,
because
it's
a
very
has
a
very,
very
big
community,
specifically
in
the
cncf,
it's
policy
as
code
very
easy
to
test
and
there's
a
lot
of
policies,
best
practices
that
we
can
take
and
leverage
from
the
community,
and
we
can
also
contribute
our
own.
We,
for
example,
contributed
a
terraform
policy
how
a
deployment
and
terraform
should
look
like,
and
now
our
solution
looks
like
this
and
the
git
repository.
A
So
let's
stop
a
minute.
Look
at
our
toolbox.
We
have
the
git
the
kubernetes
for
deployment
form
for
the
desired
state
and
oppa
for
the
policy,
and
just
by
that
we
kept.
We
covered
all
the
principles.
It's
an
intuitive
solution,
single
source
of
truth,
editable
coming
from
git
telephone
is
a
very
self-serve
language.
A
A
A
You
may
say
we
have
a
terraform
plan
using
turf
we're
using
telephone
for
the
desired
state
and
running
telephone
plan
can
detect
the
drifts
in
our
service
right,
because,
when
running
telephone
plan,
we
can
see
drift
in
the
service.
What
is
missing,
what
was
already
deployed?
What
should
what
will
go
will
be
added
and
so
on.
So
why
aren't
we
using
that?
A
As
I
already
mentioned
a
few
times,
we
have
more
than
85
micro
services.
We
can't
run
every
because
we
want
to
detect
drifts
all
the
time
we
can't
run
telephone
plan,
every
few
minutes
and
9500,
microservices
and
and
thousands
of
resources,
it's
very,
very
expensive,
timewise
and
we're
doing
this
solution
to
improve
the
developers
experience.
We
wanted
the
experience
to
be
fast
and
to
be
clear
and-
and
this
didn't
suit
our
need,
so
we
started
to
search
for
a
good
solution
for
our
case
for
the
big
infrastructure
we
have.
A
After
investigating
and
looking
for
new
startups,
maybe
and
so
on,
we
found
a
firefly,
a
new
startup
in
israel
that
gives
us
the
ability
to
get
alerts
from
our
code
and
also
outside
our
code.
Without
us
doing
any
action
we
can
get.
We
just
get
alerts
from
outside
of
our
of
our
recording.
We
don't
need
to
run
any
command
like
a
terraform
plan
we
can
detect
without
integration
with
firefighting.
We
can
detect
code
changes
inside
our
code
inside
our
desired
state.
We
can
get
detect
drift
in
our
desired
state.
A
A
A
A
We
use
the
customized
controller
and
the
source
controller.
The
source
controller
will
reconcile
between
our
git
repositories.
It
will
look
at
all
of
our
services,
all
of
our
repositories
in
in
apps
in
the
gitlab
and
detect
changes.
If
it
detects
changes,
it
will
clone
the
repository
the
artifact
and
save
it
in
the
gate,
repository
crd
and
we
have
the
customized
controller
that
will
look
at
our
fleet
repository.
Our
fleet
repository
is
basically
a
set
of
yaml
files.
Each
yaml
file
will
describe
a
different
repository.
A
So,
by
that
we
have
detection
also
and
our
services,
we
need
to
manage
and
github
all
of
our
repositories.
Okay,
but
how
does
a
doll
connect?
How
do
we
actually
do
the
reconciliation
we
have
detections
from
the
cloud
inside
our
code?
We
know
which
repositories
repositories
to
manage.
How
do
we
reconcile
our
thousands
of
resources?
Hundreds
of
microservices,
sorry.
A
A
The
gate
repository
watcher
controller
will
reconcile
between
the
git
repository
crd
and
the
deployment
unit.
Crd,
the
gate,
repository
crd,
as
I
mentioned
before,
is
getting
updates
from
the
flux
controller,
about
repositories
that
were
changed
or
detected
as
new
okay.
Once
the
git
repository
watcher
controller
detects
the
change,
it
takes
all
the
deployments
from
the
service,
as
I
mentioned
before,
for
each
service
we
can
have
multiple
deployments.
A
Each
deployment
will
get
a
section
and
the
deployment
unit
crd.
Then
we
have
the
deployment
unit
controller
that
will
reconcile
the
deployment
unit
crd.
It
will
detect
new
deployments
that
needs
to
that
need
to
get
deployed
either
by
alerts
from
the
firefighter,
as
I
mentioned
before,
or
and
by
an
interval
of
10
minutes,
it
will
go
and
look
at
the
deployment
of
unit
crd
once
it
detects
a
deployment
that
needs
to
get
deployed.
A
It
will
send
it
to
the
deploy
tools.
Why
doesn't
the
deployment
unit
controller
do
the
deployment
by
itself?
We
saw
that
infrastructure
deployments,
for
example,
bringing
a
kafka
cluster
up
can
take
a
lot
of
time.
It
can
take
a
few
minutes.
10
minutes
and
the
controller
in
kubernetes
needs
to
take
to
be
fast
to
take
milliseconds,
and
this
is
why
each
deployment
will
get
its
own
kubernetes
job
and
the
deploy
tools
that
will
know
to
renter
from
plan
and
terraform
apply
and
then
later
on.
A
A
This
is
our
fula
reconciliation
solution.
Just
for
numbers.
With
this
solution,
we
managed
to
benchmark
a
thousand
deployments
reconciliation
deployments
under
four
minutes,
but
don't
think
everything
went
so
smoothly.
We
have
a
big
infrastructure
and
apps
player
and
we
need
to
know
how
to
handle
this
amazing
number
amazing
success.
A
I
can
tell
you,
for
example,
and
my
team
we're
in
charge
of
the
gitlab
and
ups
player
and
a
few
weeks
ago
the
whole
github
crashed,
and
we
didn't
understand
why
we
investigated
that
and
then
found
that
this
amazing
solution
sent
us
to
the
gitlab
4000
jobs
at
the
same
time.
So
we
need
to
still
work
in
progress
to
know
how
to
handle
this
amazing
numbers
amazing
success,
but
we're
getting
there,
and
this
is
our
solution.
We
have
the
terraform
for
desired
state.
A
A
Okay,
if
you
remember,
I
spoke
about
the
transparency
of
that.
We
want
the
developers
to
have
and
what
happened
in
their
code
and
now,
with
this
amazing
solution,
everything
is
happening
in
the
github's
workflow,
the
user,
the
developers
don't
know
what
happened
with
their
services.
Everything
is
happening
there.
They
only
describe
and
define
their
desired
state.
So
how
do
they
know
what
happened
in
their
service?
A
We
created
a
github
user
that
each
time
a
deployment
is
happening
or
a
change
is
happening
from
the
github's
workflow
in
our
github
workflow.
It
will
send
a
commit
to
the
git
club
repository.
You
can
see
our
terraform
plan,
teleform
pre-plan
plan,
apply
and
verify
telephone
verify,
you're
not
familiar
with,
because
it's
not
really
a
terraform
command.
We
created
created
it
for
for
this
scenario.
A
If
we
open
the
terraform
verify
that
we
created,
we
can
see
exactly
what
happened
in
the
telephone
deployment
and
which
product
was
deployed.
What
time,
if
it
was
successful
or
not.
You
can
also
see
here
a
link
to
a
data
dog
monitors
in
the
datadog
monitor.
We
can
see
exactly
what
happened
in
the
deployment.
You
can
see
that
the
last
log
is
that
the
job
was
finished
and
the
status
was
synced
and
by
that
we
solved
all
of
our
pain
points.
The
operation,
the
daily
operations
of
the
developers,
aren't
fragmented
anymore.
A
Everything
is
happening
in
the
gitaps
workflow,
the
developer.
Autonomy
is
very
high
because
they
can
define
what's
happening.
What
will
happen
in
the
service
and
the
desired
state,
which
is
very
self-serve
in
the
telephone?
The
setups
are
very
easy
to
bring
up
because
we
do
everything
for
them
in
the
gitaps
workflow
and
the
transparency
is
very
high,
either
by
the
editable
that
we,
that
is
coming
from
git.
A
A
A
D
So
I
saw
that
you
include
the
gatekeeper
in
your
gitops
workflow
and
I
was
wondering
where
do
you
check
if
those
policies
are
I
mean?
Where
do
we
enforce
that?
For
those
policies
I
mean
once
the
changes
are
committed
in
git
or
while,
for
example,
you're
opening
a
pull
request,
yeah
through
which
part
of
the
workflow.
A
A
D
Okay,
okay
and
another
question
actually
concerning
the
way
you
deploy
resources.
As
if
I
understand
correctly,
you
have
a
wrapper
around
the
configurations
around
whatever
resource
you
want
to
deploy
that
you
play
with
flux,
and
then
you
have
terraform,
which
does
the
actual
deployment
of
kubernetes
resources,
and
we.