►
From YouTube: We Have Always Done It This Way! Now Let’s Try Something Completely Different -Eliran Bivas
Description
We Have Always Done It This Way! Now Let’s Try Something Completely Different -Eliran Bivas, AppsFlyer
At Appsflyer, we have 400 engineers that write software in several programming languages and with an architecture of over 850 micro-services, thousands of cloud resources and dozens of SaaS integrations. We felt a change was needed. Can GitOps be that change? In this talk, Eliran Bivas, AppsFlyer's Cloud Native Leader, will present the challenges the department faced and still faces when adopting GitOps practices. How AppsFlyer’s Engineering Platform organization changed, how AppsFlyer uncovered the unknowns, and how AppsFlyer educated its Platform Group, and later the entire R&D organization to practice GitOps.
A
A
A
A
So
what
happened
a
few
months
earlier
to
that
declaration
by
reshef?
Well,
we
sat
down
as
a
platform
and
tried
to
figure
out
what
what
are
the
steps
we
need
to
take
to
actually
take
our
infrastructure
to
its
next
level.
But
where
do
we
actually
begin?
Do
we
even
have
issues
in
our
platform
or
in
our
infrastructure?
A
Simply
put
we
had
a
lot
of
unknowns
to
begin
with,
we
didn't
really
know
the
exact
scale
of
architecture.
Now
that
I
got
you
the
number
of
250
000,
we
really
didn't
know
that
back
then
we
didn't
even
know
what
our
developers
are
actually
doing
without
trying
to
achieve
it's
not
like.
We
wish
to
provide
them
how
to
walk
faster,
what
prevent
them
from
making
it
slower.
A
But
how
are
you
actually
going
to
start
to
explore
such
a
huge
infrastructure,
such
as
arsenal
as
an
organization
as
an
engineer
organization,
our
mission
goal
is
to
support
the
organization
towards
its
company
success
as
platform.
Our
customers
are
developers,
but
we
need
to
understand
what
are
their
daily
needs,
so
we
can
actually
provide
them
with
value
that
they
can
benefit
the
business
right.
This
is
the
end
goal
of
every
engineering
organization,
but
how
we
can
effectively
collect
feedback
from
all
of
this
engineering
organization.
A
Is
you
can't
simply
go
and
ask
each
and
every
one?
What
are
you
doing
today?
Am
I
doing
right?
Am
I
doing
wrong?
This
is
this?
Is
the
scale
to
compare
the
platform
compared
to
the
entire
engineering?
So
we
can't
really
go
and
ask
everyone
it.
It
wouldn't
scale
and
we
only
knew
when
there
were
problems,
so
they
will
reach
out
to
us.
Look,
I
got
a
broken
cluster
or
something
like
that.
A
We
looked
at
ourselves
as
a
platform.
We
had
issues
as
well
when
we
started
the
process,
we
were
split
into
seven
vertical
teams
and
that
were
handling
different
domains.
In
our
platform
we
were
working
mostly
in
a
survival
mode.
We
were
trying
to
put
our
head
above
the
water
trying
to
put
one
fire
after
the
other.
Maybe
a
kafka
cluster
would
break
down
a
spark.
Job,
didn't
didn't
complete
it
or
maybe
a
deployment
didn't
reach
a
certain
state.
A
Because
of
that
we
were
very.
It
was
very
hard
for
us
to
actually
evolve
our
system.
We
couldn't
really
go
to
the
next
spark
version
or
maybe
update
our
kafka
cluster.
We
were
trying
to
just
survive
our
infrastructure
and
it
was
very
hard
to
align.
We
were
operating
in
squads,
but
all
of
them
were
verticals
and
didn't
really
had
a
unified
experience
for
our
user
each
and
everyone
had
a
different
mechanism
for
operating.
A
So
it
was
very
difficult
to
actually
collect
feedback,
because
a
certain
feedback
would
fit
a
certain
team
and
it
wasn't
really
an
easy
task
to
collect
that
feedback.
If
you
look
at
the
diagram
by
hendrick
that
described
the
mode
of
operations,
we
were
walking
in
a
high
autonomy,
but
extremely
low
alignment.
Our
entire
culture
was
chaotic.
A
Our
road
ahead
was
unknown
as
I
mentioned
earlier,
but
we
had
to
start
collecting
the
information
because
we
had
to
understand
where
we
had
it.
We
first
wanted
to
focus
on
the
developer
experience
and
the
developer.
Experience
that
we
we
needed
to
learn
is
what
are
the
daily
activities
that
we're
trying
to
actually
boost?
What
are
the
developers
are
actually
doing
and
how
we
can
assist
them
and
what
prevents
them.
As
I
mentioned
from
moving
faster,
we
introduced
a
new
model
that
we
called
the
champions.
A
The
champions
were
essentially
software
developers
that
selected
from
the
teams
themselves
and
in
a
bi-weekly
meetings
with
the
platform
we
were
able
to
get
what
are
the
requirements
and
their
needs
when
operating
with
the
platform?
It
took
us
a
few
months
to
kick
off
this
entire
process,
but
we
learned
a
lot
from
it.
We
met.
We
went
from
a
model
that
usually
we
got
information
just
when
something
wrong
happened,
and
then
you
can't
really
know
what
they're
trying
to
achieve.
You
only
knew
that
you're
trying
to
walk
with
them
to
solve
their
problems.
A
It
wasn't
really
an
effective
way
to
collect
feedback.
We
went
to
a
model
that
actually
got
the
champions
as
a
representative
for
the
team
and
and
by
that
he
can
actually
explain
to
us
look.
This
is
what
we're
trying
to
achieve.
This
is
the
the
the
goals
that
we
are
trying
to
have,
and
these
are
the
the
paint
points
that
I
have
and
we
use
the
model
back
to
actually
test
out
and
validate
some
of
our
estimations
like
will
this
tool
be
effective
to
you?
A
They
had
too
many
systems
to
operate
if,
for
example,
to
to
push
a
single
committee
to
it,
took
you
like
seven,
a
different
system
just
to
push
it
into
production
and
it's
very,
very
tedious,
walk.
You
have
to
push
a
commit
to
git,
try
built
it,
maybe
in
jenkins,
try
and
test
it
in
a
different
system
and
deploy
it
in
a
different
one,
and
the
operation
goes
to
a
different
system.
A
The
entire
experience
was
unified,
everything
was
fragmented.
You
had
to
learn
how
to
operate
each
and
every
system,
and
we
looked
at
the
autonomy
of
our
developer
and
we
found
out
even
though
that
we
said
it
will
operating
in
in
squads.
We
didn't
really
provide
them
with
any
capability
to
actually
succeed
if
they
had
any
requirement
an
infrastructure
requirement,
they
had
to
come
to
platform.
If
the
infrastructure
was
broken,
they
had
to
come
to
platform
anything
that
they
needed
infrastructure
wise.
We
were
their
bottlenecks.
A
We
didn't
really
help
them
that
much
and
when
you
are
giving
an
estimation
for
a
project,
it
can't
really
give
valid
estimation
if
you're,
actually
relying
on
someone
else
platform,
we
looked
at
the
setups
and
found
out
that
they
are
mostly
fragile.
If
something
bad
did
happen,
we
didn't
know
how
to
bring
the
system
back
up
efficiently.
A
We
were
walking
on
a
lot
of
magic
there
and
last
but
not
least,
we
had
a
lot
of
issues
on
transparency
and
the
complete
auditability
of
our
system.
If
something
wrong
did
happen,
we
didn't
know
why
and-
and
it's
a
it's
very
very
it
was
it-
it
become
very,
very
tough
job
to
actually
learn
what
to
do,
and
and
for
that
example,
the
rds
cluster,
if
it
was
deleted,
who
deleted
it?
A
A
Now
here
comes
our
next
evolution,
the
3.0,
so
we
selected
two
developers
from
each
of
the
squads
inside
the
platform
and
form
what
we
called
a
cloud
native
topic
and
a
topic
is
a
mixture
between
a
chapter
and
a
guild
in
a
spotify
model,
but
you
can
simply
adopt
a
model
and
say:
look
I'm
doing
something.
We
did
something
different
because
of
our
understanding
of
our
requirements.
A
We
are
striving
to
push
for
technical
excellency
for
across
the
platform
and
by
providing
strong
alignment
with
for
the
entire
squads
in
the
platform.
We're
able
to
give
in
the
will
give
them
each
and
everyone
an
increased
autonomy
to
work
better,
to
increase
the
overall
impact
increased
relativity
as
as
a
squad
and
eventually
as
platform,
and
boost
our
growth,
both
in
infrastructure
and
human
resources
alike.
A
A
Our
infrastructure
was
hardly
managed.
We
didn't
employ
any
infrastructure
as
code
policies
pipeline.
Sorry,
we
didn't
employ
cicd
pipelines
correctly.
Even
we
had
too
many
internal
tooling
that
were
didn't
up
to
the
task
and
of
course
they
didn't
comply
to
any
community
standard
shocked,
but
we
still
had
to
shop
for
a
solution.
So
this
is
exactly
what
we
did.
A
We
went
around
into
the
community
and
shopped
for
a
solution,
and
this
was
our
supermarket,
so
the
cncf
landscape,
as
you
all
know,
composes
of
thousands
of
technologies
and
we
started
shopping
around
for
a
solution
and
testing
out
thousands
of
them
takes
time,
but
we
did
exactly
this.
We
invested
a
lot
of
time
about
three
four
months
process
and
we
created
proof
of
concept
of
many
many
technologies
that
we
assume
that
can
help
us.
A
We
not
we,
we
needed
to
learn
what
are
the
the
benefits
and,
of
course,
pitfalls,
of
each
of
these
technologies.
We
get
a
pocs
to
compare
kubernetes
and
nomad.
We,
we
create
pocs,
to
compare
argo
and
flux.
What
is
different
with
telephone,
pollumi
and
crossplane,
and
and
for
that
it
takes
a
lot
of
time,
but
you
learn
so
much.
A
We
met
with
thousands,
hundreds
of
them
and
and
and
my
daily
calendar
looked
like
five
minutes
a
day,
five
meetings
a
day
with
different
startups
and
was
amazing,
because
we
learned
a
lot
how
how
they
can
actually
boost
our
our
solution
into
its
actual
cloud-native
infrastructure,
how
we
can
get
cloud
native
solution
much
better.
We
actually
partnered
partnered
with
several
startups
to
actually
help
us
uncover
more
of
the
unknowns
that
we
still
didn't
know
about
the
architecture.
A
Once
we
completed
the
this
entire
evaluation
process,
we
set
out
to
have
this
solution
principle
for
ourselves
as
opsler
engineering,
we
said
that
solution
must
be
intuitive.
It
must
be
familiar
to
everyone
easy
to
use.
It
must
have
a
single
sort
of
truth.
We
don't
want
the
system
to
be
fragmented
anymore.
It
must
be
a
one-stop
shop
for
all
the
operational
information
that
can
be
close
to
your
code.
A
Why
not
have
a
self-serve
by
boosting
the
ownership
and
autonomy
of
our
developers?
We
want
the
system
to
be
declarative.
No
more
new
tool
to
learn
simply
tell
us
what
you
need
and,
of
course
we
wanted
to
boost
the
transparency
of
our
system
so
auditable,
so
it
could
be
easier
for
us
and
for
the
users
to
understand
what
happened
and
last
but
not
least,
one
of
the
solutions
to
be
community
driven,
no
more
investing
inventing
the
will.
A
You
wanted
to
leverage
the
existing,
tooling
and
frameworks
that
could
make
the
community
already
battle
tested
if
some
of
it
of
all
of
it
sounds
familiar.
You
probably
heard
it,
because
the
desired
side
of
the
github's
managed
system
is,
of
course,
to
be
declarative
version
put
automatically
and
continuously
reconcile.
So
we
figure
that,
if
we'll
simply
say
we're
gonna
adopt
git
ups,
we're
gonna
check
mark
three
of
our
items
and
we're
good
to
go.
A
We're
only
left
to
to
do
the
rest,
that's
all,
but
how
you
can
actually
adopt
gear
ups,
how
you
went
from
the
theory
of
it
into
real
practice,
how
you
actually
create
a
system
that
went
from
the
hello
world
of
it
to
the
large-scale
system
of
upsly
we're
trying
to
walk
with
each
and
every
technology
that
we
selected.
We
were
dogfolding
for
almost
six
months,
just
to
understand
how
to
operate
each
and
every
technology
and
framework
we
tested
out
several
frameworks
tested
several
tooling
and,
of
course,
several
flows
just
to
understand.
A
What
is
the
pain
points
when
using
gear
ups
and
we
landed
on
this
intuitive
flow?
What
we
believe
would
be
an
intuitive
flow
we
took
to,
let's
take,
for
example,
two
developers
in
abstract.
The
first
developer
is
doing
the
code
commits
for
the
service
and
also
declaring
the
service
deployment
plan.
The
second
developer.
She
is
declaring
the
infrastructure
requirement
for
that
service
and,
of
course,
also
declaring
the
pager
duty
policies
to
monitor
that
service.
Both
of
them
are
pushing
to
the
same
git
repository
and
the
declarative
format
that
we
selected
was
terraform.
A
We're
given.
We
have
given
our
user
great
power
now
and
with
that
come
great
responsibility.
We
didn't
want
them
to
lose
their
autonomy,
that
we
just
gave
them,
but
we
didn't
want
them
to
actually
help
themselves
or
others
in
the
process.
So
we
added
open
policy
agent
to
gather
our
flows
and
we
can
employ
now
policy
and
as
code
to
actually
provide
offslide
policy
into
the
process
and,
of
course,
leverage
the
community
policies
to
guard
out
the
to
guard
our
github's
workflow
and
make
it
a
lot
safer.
A
A
A
What
is
the
definition
that
we
as
upsly
want
for
githubs,
not
the
community
definition?
We,
we
tried
a
different
approach
here
of
actually
saying
that
get
ups
for
us
is
managing
workflow
for
the
application
infrastructure
alike
sitting
in
git,
and
we
focus
on
the
developer
experience
using
the
tools
that
you're
already
familiar
with.
No
new
tools
are
involved
in
here.
A
We
explained
them
how
this
solution
and
then
how
this
workflow
actually
solved.
The
pain
points
that
they
described
earlier
to
us.
The
system
is
no
longer
fragmented.
Everything
is
in
git,
you
have
increased
autonomy
because
you're
now
can
employ
several
self-service
flows
and
the
system
is
much
less
fragile.
A
A
It's
now
very
easy
for
for
our
user
to
understand
who
made
a
change
and
what
went
wrong,
but,
of
course
we
caught
go.
We
got
a
a
very
cold
response
and,
to
be
honest,
it's
totally
understandable
our
engineering
teams
and
other
things
on
their
mind.
They
were
pushing
for
the
for
the
success
of
our
of
the
business.
They
preferred
the
known
broken
flow
instead
of
working
with
a
new
integration
now,
but
we
didn't
really
back
down
on
it.
A
We
used
the
construct
that
I
explained
earlier
of
the
champions,
for
example,
that
we
use
the
champions
to
better
understand
what
are
the
fears
and
concerns
when
it
when
integrating
with
githubs
and
how
we
can
actually
provide
them
with
better
confidence
to
to
work
with
git
ups.
We
took
all
the
information
that
we
got
from
the
champions
back
to
the
topic
and
using
that
feedback
to
create
a
unified
messaging
and
communication
back
to
our
developer
teams,
and
we
capped
on
dog
footing
the
solution.
A
We
continuously
improved
the
integration
and
flow
to
better
assist
and
deal
with
the
concern
our
user
provided
us
so
where
we
had
a
where
are
we
today?
In
terms
of
managed
resources
in
our
cloud
accounts,
we
went
on
f.
We
went
up
from
three
percent
to
25
coverage
in
infrastructures
code
and
keep
in
mind.
We
are
managing
250,
000
cloud
resources
in
our
new
infrastructure.
We
are
la,
we
are
covering
97
now,
which
is
a
huge
improvement
for
us.
A
A
A
Our
overall
contribution
is,
as
an
organization
team,
as
a
platform
team
has
improved
tremendously
to
open
source,
because
we
took
the
knowledge
that
we
learned
from
using
those
tools
on
a
large
scale
system
such
as
upsly
and
couldn't
put
it
back
to
open
source.
We
didn't
keep
it
inside.
We
wanted
to
to
give
back
to
the
community
all
the
information
that
we
learned
and
we're
still
doing
it,
we're
still
in
a
continuous
effort
to
support
our
own
upslide
scale
in
our
telephone
controller
implementation.
That
already
we
contributed
to
flux.
A
We
have
an
improved
reliability
effort
to
for
our
drift
detection
selection.
Drift
detection
solution,
keep
in
mind
that
in
a
large
scale
like
this
most
most
of
the
drift
detection
solutions
out,
there
will
simply
won't
work,
so
we
partner
with
a
startup
that
actually
solve
it
for
us,
we're
still
evaluating
several
solutions
to
continuously
complete
our
git
ups
flow,
just
a
small
recap
on
what
we've
been
speaking
for
the
past
20
minutes,
or
so
we
operate
we're
hoping
in
this
mode.
A
In
about
18
months
now
we
started
with
a
lot
of
unknowns.
We
didn't
know
what
our
user
was
doing
and
we
didn't
know
what
infrastructure
is.
Actually
we
started
by
collecting
the
daily
activities
and
understanding
using
the
champions
model
to
get
their
feedback.
We
use
the
topic
to
create
a
unified
experience
of
as
a
platform.
A
We
invested
a
lot
in
evaluating
and
piercing
several
technologies,
and
we
kept
on
evaluating
through
ourself
with
dog
fooding
the
entire
solution.
We
still
are
dog
fooding.
This
is
one
of
the
I
think,
the
key
item
for
us
as
a
platform.
Above
all,
we
invested
a
lot
of
in
education
and
open
communication
of
any
change
that
we
as
a
platform
took
to
boost
the
the
operability
of
our
systems
and,
of
course,
learn
back
what
we
can,
what
we
can
have,
what
do,
what
we
can
modify
and
as
a
platform
we
kept
going.
A
A
A
Okay,
it's
actually
a
part
of
it.
It's
going
to
be
in
a
session,
so
I'm
briefly
going
to
answer
that.
The
reason
for
that
is
that
we
learned
that
the
community
for
teleform
is
much
much
louder.
Crossplan
might
be
the
future,
I'm
I'm
a
personal
fan
of
it,
but
it
doesn't
really
add
up
to
the
scale
of
upsly
and
and
for
the
entire
reconcile
process.
B
A
A
A
A
The
success
rate
for
the
topic
was,
at
the
beginning,
extremely
low.
No
one
actually
wanted
to
adopt,
but
using
a
lot
of
convincing
and
a
lot
of
management
support,
it
became
very,
very
easy
to
push
for
the
first
results
and
by
giving
the
first
results,
it
became
very
easy
to
push
for
the
next
milestones.