►
From YouTube: SneakOps – Getting Users To Use GitOps Without Them Even Knowing... - Éamon Ryan & Hedley Simons
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
SneakOps – Getting Users To Use GitOps Without Them Even Knowing About It! - Éamon Ryan & Hedley Simons, Grafana Labs
Speakers: Éamon Ryan, Hedley Simons
Most people are well-aware of the benefits of GitOps for various workflows especially as it relates to Kubernetes deployments - it allows easy integration of approvals, verifiable change history and automation hooks. However, getting users to adopt a different way of working can be challenging - especially if it involves tools and methods that would be convenient for you, the administrator, but inconvenient for the end-user. So, what do you do when your end-users are not directly using Git in their day-to-day work, but you still want to add the benefits of GitOps to your deployments without slowing them down? Simple - you implement GitOps but hide the entire process from them! In this session, Éamon and Heds will take you through how they took an internal Grafana environment that had grown increasingly messy and unreliable due to usage by a rapidly expanding internal team - and transformed it into a repeatable, promotable, process-driven well-oiled machine fueled by GitOps, Kubernetes, Terraform and more - all without the end-users having to learn or interact with Git at all!
A
Welcome
to
the
talk
it's
going
to
get
well
easier,
a
bit
okay,
it's
gonna
be
a
little
darker
in
here,
as
we
were
just
warned.
The
reason
for
that
is,
of
course,
we're
using
a
dark,
very
dark
slide
theme,
which
is
actually
our
company
slide
theme,
so
that's
just
kind
of
how
it
is
and
the
lights
are
kind
of
washing
it
out
so
pretty
enjoy
the
darkness
for
a
little
bit
commit.
You
can
have
a
nap
if
you
want
like
we're,
not
gonna
be
mad
about
it.
A
I
was
going
to
make
a
joke
about
it's
great,
to
see,
there's
so
many
people
attending,
but
now
I
can't
see
anybody
anymore.
So
you
know
I'm
just
going
to
assume
you're
all
still
here.
So
the
talk
is
snake
Ops
getting
youthers
you
use
get
up,
so
that's
even
knowing
about
it,
and
my
name
is
Eamonn
Ryan
I'm,
a
principal
field
engineer
at
confined
labs
and
I'm
joined
here
by
my
partner
in
crime.
C
Apps
we
ensure
that
they
are
able
to
get
the
the
technical
debt
that
they
need
for
some
of
their
engagements.
They
know
a
lot
about
the
Health
products
are
used.
We
know
a
lot
of
that
has
products
are
architected,
we
also
write
workshops,
we
write
an
open
material
and
we
also
built
and
maintain
the
demo
kit
and
we're
talking
about
in
a
little
bit
now.
A
So
this
is
kind
of
well
am
I
too
close
to
something.
I
can't
really
tell
is
it
you
know?
Is
it
this
one?
Okay,
all
right,
let's
use
better
okay,
so
this
is
a
very
basic
view
of
the
environment
that
has
mentioned
called
the
demo
kit
that
we
support.
It
essentially
is
a
gke
cluster
with
a
whole
bunch
of
stuff
deployed
on
top
of
it.
So
we've
got
a
large
buzzing
history
just
turn
this
one
off.
That's
all
it's
up!
Okay,
anyway,
so
it's
this
large
instance.
A
It's
got
grafana
in
there
as
I'm
sure
everyone
here
is
familiar
with
grafana,
maybe
not
familiar
with
all
the
other
stuff.
It
connects
to
lots
of
people
only
connected
to
one
or
two
things,
but
because
we
are
a
Groupon
labs
and
one
of
the
teams
we
support
is
involved
in
showing
demos
for
what
grafana
can
speak
to.
A
We
have
it
hooked
up
to
literally
everything
we
possibly
can
so
on
the
left
side,
it's
Prometheus,
as
well
as
all
the
other
projects
and
products
that
grafana
has
like
on-call
and
mimir
and
incident,
and
Loki
and
Tempo
and
ML
and
k6
on
the
right
hand,
side.
We
have
all
these
third
party
things
that
we
can
also
connect
up
to,
especially
on
like,
like
the
Enterprise
version
of
our
stuff,
like
servicenow
and
Oracle
and
Splunk
and
so
on.
C
C
B
C
A
Thank
you.
So
our
plan
for
to
to
solve
this
is
to
have
multiple
grafana
environments
instead.
So
the
first
is
a
Sandbox
complete,
Anarchy,
Mad
Max
looks
very
much
like
the
except
that
you
know
it's.
It's
intended
to
be
that
way.
Instead
of
having
evolved
that
way
over
the
let's
be
like
off
here
and
then
we
have
a
second
environment,
which
is
our
production,
pristine,
environment
and
the
way
to
keep
it
pristine
is,
of
course,
to
not
let
people
change
it
easily.
A
To
add
to
the
grafana
instance
sounds
great,
except
for
you
know,
grafana
is
like,
doesn't
have
a
totally
great
flow
for
automating.
This
in
itself,
you
have,
to
kind
of
you
know,
use
the
API
or
something
like
that.
It
doesn't
really
have
a
native
integration
for
this.
Yet,
but
in
theory
we
thought
this
would
help
and
it
would
give
us
a
nice
cleaner
setup.
C
B
B
C
The
Source
control
systems
that
we're
using
and
because
they're
constantly
meetings
they
don't
really
have
time
to
actually
go
away
and
learn
all
this
stuff,
all
the
processes,
all
the
flows
that
we
want
them
to
use
and,
of
course,
the
lack
of
use
of
it
and
the
lack
of
most
of
the
majority
of
the
solutions
Engineers
actually
putting
PRS
in
or
thinking
about.
This
means
that
the
mess
remains
it
just
continues.
A
We
actually
have
an
additional
problem,
which
is
that
as
we
as
we
as
we
add
more
grafana
instances,
so
we've
been
going
from
a
single
centralized
one
to
having
one
in
the
US
and
one
in
Europe,
and
one
in
in
the
asia-pacific
region
is
that
well,
the
SCS
are
adding
dashboards
in
in
the
main
one
and
then
they're
like.
Oh,
it's
not
in
the
other
one,
it's
not
in
the
other
one
so
that
they're
not
really
propagating
around
and
if
they
had
done
the
gift
process.
A
C
We're
gonna
so
we're
gonna
get
opposite
all,
so
we're
going
to
use
some
fairly
standard
tools,
but
we're
going
to
go
into
what
those
tools
are
and
a
little
bit
about
what
they
do
just
so
we
can
kind
of
set
the
scene
for
when
we
do
this
demo,
so
the
first
is
grafana
itself.
Grafana
obviously
has
many
products
in
this
case
it's
the
beautiful
dashboarding
tool
that
everybody
knows
and
loves,
but
it
also
has
a
full
rest
API
so
that
rest
API
acts
for
Json
data.
It
allows
you
to
upload
dashboards.
C
It
allows
you
to
upload
data
sources.
It
allows
you
to
pretty
much
do
whatever
you
want
with
any
of
the
the
resources
that
grafana
supports.
It's
full
crowd,
operation
supported,
but
in
this
particular
case
we're
going
to
use
it
to
upload
and
download
dashboards
from
one
grafana
instance
and
upload
them
into
our
production
instance.
Instead,
where
they
will
be
basically
locked
out
so
that
people
can't
start
mucking
around
with
them,
so
we're
going
to
run
a
little
demon
in
a
pod
on
our
kubernetes
cluster.
C
A
So
another
component
that
we're
using
here,
which
I'm
sure
lots
of
people
are
already
familiar
with
because
it's
been
around
for
years,
it's
terraform!
So
briefly,
if
you're
not
it
allows
you
to
do
infrastructure
as
code,
it
works
potently.
So
that
means
you
can
rerun
things
and
it
won't.
You
know
Stomp
all
over
everything,
you've
done,
Works
off
of
desired
versus
actual
State,
and
there
is
a
terraform
provider
for
grafana,
and
one
of
the
basic
resources
index
is
the
dashboard
resource.
So
you
can
add
dashboards,
no
problem.
A
You
can
remove
dashboards,
we've
actually
put
as
a
side
note
a
ton
of
Dev
effort
into
that
provider
this
year.
So
it
does
like
way
more
stuff
than
it
used
to
do.
But
obviously
it's
one
of
its
most
basic
features
is
put
dashboards
into
grafana,
so
that
works
really
well.
It
will
mean
that
you
know,
as
as
we
try
to
push
dashboards
into
grafana,
it
will
only
add
in
the
ones
that
it
needs
to
versus
ones
that
are
already
there,
so
it
just
works
on
like
the
diff
kind
of
basis.
D
A
C
So
this
is
where
Atlantis
comes
in,
so
obviously
terraform
doesn't
run
by
itself
we're
going
to
store
these
dashboards
inside
git
GitHub
in
this
case,
and
what
we
actually
want
to
happen
is
whenever
a
PR
is
created
and
the
demon
itself
is
going
to
create
that
PR.
For
us,
we
want
to
ensure
that
a
we
can
verify
the
dashboards
that
will
need
an
approval
process
from
somebody
that
is
responsible
for
the
production
environment,
but
when
that,
when
that
approval
has
been
given,
we
want
to
ensure
that
we
can
actually
do
something
with
it.
C
So
Atlantis
is
actually
a
way
of
allowing
you
to
run
terraform
via
git
pull
requests,
it's
a
very
fully
fledged
system.
It
allows
you
to
do
various
things
such
as
you
know,
only
apply
terraform
when
something
has
been
approved.
You
know
clean
up
on
and
emerge
all
of
that
kind
of
stuff
that
you
need
for
a
get
Ops
workflow.
So
the
general
idea
is
that
we
will
have
our
dashboards
in
GitHub.
C
We
will
create
a
new
PR
from
the
demon.
That's
actually
doing
all
the
work
pulling
these
in
from
our
development
grafana.
It
will
get
approved.
Sorry,
the
the
plan,
the
terraform
plan,
will
actually
be
run
at
the
point
where
that
PR
is
created,
so
that
we
can
actually
assure
ourselves
that
we're
not
going
to
break
anything.
If
we
go
ahead
with
actually
merging
it,
somebody
will
then
come
along
and
approve
it.
A
And
one
other
small
thing
on
on
this
side:
if
people
were
about
were
thinking
of
asking
the
question
of
hey,
why
didn't
you
use
the
terraform
controller
in
flux?
I
didn't
know
that
existed
until
this
Tuesday,
so
we'll
we'll
probably
go
and
look
at
it.
We
were
at
github's
Con
on
Tuesday.
Oh
that's
useful!
We
should
maybe
look
at
that,
but
yeah
literally
didn't
know
it
existed
until
this
week.
A
So
another
little
piece
of
glue
we're
using
is
GitHub
actions
so
allows
you
to
really
easily
run
little
workflows
inside
of
GitHub
things
that
can
be
triggered
by
PR
Creations
comments,
all
that
kind
of
stuff.
For
us
we
were
just
using
it
as
a
little
bit
of
glue
because
of
the
way
that
Atlantis
currently
works.
A
It
actually
requires
you
to
comment
on
the
pr
to
to
allow
it
to
actually
proceed,
but
we
wanted
that
to
happen
in
a
slightly
more
automated
fashion,
so
we
have
an
action
that
comes
in
and
like
automates,
that
like
a
little
bit
for
us,
so
that
we
don't
have
to
do
it
quite
in
as
manual
fashion
all
right.
This
brings
us.
Oh
no.
This
is
me
again:
okay,
by
the
way
of
artwork
we've
been
showing,
in
the
background,
are
just
fun.
A
Bits
of
you
know
all
the
different
AI
tools
for
generating
art
that
you've
seen
around
lately.
So
we
use
them
to
ask
it
for
things
like
GitHub
actions,
which
created
a
very
strange
image
here
with
like
a
fox
wave,
and
this
is
what
it
came
up
with
for
Atlantis,
even
though
it
very
much
looks
like
Rapture
from
BioShock
kind.
A
That,
honestly,
that's
pretty
cool.
This
is
one
of
the
ones
we
got
for
sneak
up,
so
we
got
this
very
character
in
your
data
center,
so
I
thought
that
was
perfect
and
there's
a
few
more
of
them
as
we
as
we
move
along
okay.
So
our
solution
is
we
take
our
thank
you
to
move
closer
because
I
can't
even
read
the
text.
Now
we
take
our
sandbox
grafana
here
and
down
here.
A
We
have
our
solution
to
engineer
and
he's
going
to
go
in
and
add
a
dashboard
which
will
have
like
a
bunch
of
characteristics,
as
you
might
imagine
so.
It'll
have
an
ID
it'll,
have
a
tags
associated
with
it
and
that
dashboard,
what
we're
going
to
have
them
do
is
save
it
into
a
very
specific
folder
calling
it
The,
Magic
folder
doesn't
really
matter
which
folder
it
is
as
long
as
it's
the
one
that
we
designate,
so
they
save
it
into
a
specific
folder.
A
A
It's
going
to
then
create
the
plan
to
run
this
against
the
production
grafana,
and
then
that
will
allow
us
to
get
as
many
dashboards
in
as
many
folders
as
we
actually
want
yeah
as
many
followers
as
we
actually
want,
and
that
will
mean
that
we
will
have
a
system
where
the
Aldi
person
does.
The
SC
in
the
bottom
left
is
save
the
dashboard
they
want
into
a
specific
folder
in
and
like
just
has
a
particular
set
of
tags
and
then
off
it
goes.
C
So
we
keep
talking
about
this
demon,
that's
going
to
work
with
Joshua's
ball.
Actually
is
it
so
this
is
what
our
sneak
Ops
demon
actually
is
it's
written
in
a
language
that
rhymes
with
toad
I'm,
not
proud
of
it.
It
is
going
to
be
written,
it
can
go
or
rust,
but
what
it
basically
does
is
look
in
this
golden
folder
in
the
development
environment
and
every
time
a
new
folder
appears
it
goes
away.
It
looks
exactly
what
the
folder
is.
C
It
looks
at
the
tags
that
are
attached
on
the
dashboard
to
determine
the
folder
in
the
production
environment
that
we're
going
to
put
it
in.
It
looks
at
a
version
tag
as
well.
It's
very
important
that
we
don't
allow
se's
to
create
new
dashboards
that
are
going
to
overwrite
production
instances
that
are
actually
already
useful.
That
would
get
very
messy
very
quickly,
so
it
does
a
load
of
different
checks
to
ensure
that
it's
actually
safe
to
provision
this
dashboard
to
the
production
environment.
It
raises
errors.
C
If
any
of
this
information
isn't
there
that
it
needs
or
something's
going
to
go
wrong
and
then,
if
everything
is
all
right,
it
will
raise
a
PR
for
it
in
our
GitHub
instance
to
ensure
that
we
can
then
go
through
the
the
entire
git
Ops
flow
to
ensure
that
we
can
kind
of
get
that
into
production
as
well.
If
there's
an
author
tag
on
it
as
well,
it
will
also
pay
you.
C
I'm
praying
to
the
Elder
gods
that
everything
is
going
to
work.
So
in
this
particular
case,
you
can
see
that
we
have
both
our
development
environment,
which
currently
has
a
couple
of
folders,
but
that
we
have
our
Wild
West
folder
here,
where
maybe
a
Solutions
engineer
is
working
with
an
amazing
demo
dashboard.
We
have
our
production
instance
as
well,
which
has
a
few
different
folders
that
people
are
using
at
the
moment
for
demos
and
what
I'm
going
to
do
is
I'm
going
to
go
away
and
open.
C
My
dashboard
doesn't
really
matter
what
the
dashboard
has
in
it,
but
I'm
going
to
save
this
now
into
the
folder
that
we
use
to
actually
pick
up
any
changes
to
go
into
production.
So,
let's
move
that
into
our
sneakups
folder
and
let's
save
it
now,
what's
going
on
at
the
moment,
is
the
way
that
our
Dash
demon
actually
works.
Is
it
pulls
every
10
minutes?
C
We
don't
want
it
to
poll
every
10
minutes
for
that
we
would
go
way
over
time.
So
it's
currently
running
in
a
little
demo
cluster
here
you
can
see
that
we're
running
Atlantis
we're
running
our
demon,
we're
running
a
few
things
that
we
need
for
observability
and
we're
running
both
our
grafana
development
instance
and
our
production
instance
as
well.
C
Hopefully,
by
the
time
we
go
back
to
slack,
he
says,
of
course,
we're
going
to
hit
the
either
we're
going
to
hit
30
seconds
by
the
way
there
we
go
so
actually
the
Dutch
team
has
told
us
that
we
can't
provision
that
we
actually
need
some
more
tags,
that
we
don't
have
the
prop
version
and
the
pro
folder
I've
not
told
it
where
to
put
the
dashboard
and
I've
not
told
it
what
version
it
is.
So
what
I
need
to
do
is
go
back
to
my
dashboard
now
change
my
tags.
C
We're
going
to
remove
the
tags
that
Dash
team
has
put
on
there.
That's
actually
to
help
us,
so
it
will
help
both
us
and
solutions.
Engineers
determine
why
the
dashboard
can
be
provisioned
and
it
will
give
an
Epoch
date
there
to
tell
you
when
it
happened.
So
let's
remove
those.
It's
dark,
of
course,
so
I
can't
see
properly
my
screen,
but
we're
gonna
we're
gonna.
Put
that
you
know.
Let's
call
it
amazing
demos
folder
we're
going
to
give
it
a
version
of
one,
so
that
will
allow
us
now
to
hopefully
try
again.
C
C
Am
there
we
go
so
now,
dash
demon
has
picked
this
up.
It
said:
okay,
yeah,
you've.
Given
me,
everything
I
need
to
actually
provision
this
folder
and
it's
also
very
handily,
given
us
the
link
to
GitHub
where
the
pr
has
been
created.
So
if
we're
going
to
have
a
look
at
that,
obviously
we
can't
show
you
our
production
or
demo
environment.
C
C
He
says
there
we
go,
it
has
actually
run
it's
just
GitHub
didn't
refresh,
so
we
now
have
an
output
here
and
Atlantis
has
run
the
plan.
Terraform
has
told
us
exactly
what's
going
to
change
we're
going
to
add
a
new
dashboard,
we're
going
to
add
a
new
folder
and,
at
this
point,
we're
in
a
good
position
for
somebody
who
maintains
the
production
environment
to
come
along
and
approve
the
changes
button
boy.
Do
your
thing:
I.
C
This
usually
wouldn't
allow
you
to
merge
either
it's
just
because
it's
a
private
repo
that
I'm
not
actually
paying
for
in
my
organization,
so
I've
been
able
to
turn
branch
of
protection
on
Eamon
has
come
along
and
he
said
a
few
things
in
the
comments.
Github
actions
has
now
discovered
that
there's
something
that
has
approved
or
sorry
an
approval
has
occurred
on
the
pr.
Because
of
that,
it's
now
gone
along
and
said
and
latences
apply.
So
Atlantis
is
now
going
to
do
its
thing.
C
It's
going
to
apply
terraform
on
rpr,
which
again,
hopefully
should
not
take
too
long
it
hasn't
there.
We
go.
It's
done
it.
It's
applied,
our
dashboard
changes,
it's
closed,
our
branch
and
if
we
now
go
back
to
our
production
instance
and
take
another
look,
we
can
see
we
have
a
Golden's
dashboard
and
our
dashboard
has
been
moved
across
fully
along
with
all
of
our
tags.
Now
the
SC
can
come
along.
They
can
demo
this
dashboard
and
we
know
we're
in
a
good
position
and
it's
a
dashboard
should
actually
be
there.
C
A
So
quick
recap:
so
our
environment
big
shared
grafana
instance,
tons
of
data
sources,
tons
of
Integrations
lots
of
users
using
it
our
whole
SE
team
are
maybe
the
primary
users
there's
dozens
of
them
less
than
100
more
than
50.,
but
it's
actually
open
to
everybody
at
grafana
and
it
actually
gets
used
by
engineering
teams.
It
gets
used
by
customer
support
teams
gets
used
by
Professional
Services,
it's
used
by
everybody,
so
lots
of
people
in
there.
C
So
the
situation
was,
we
have
Massachusetts
brawl.
We
have
lots
of
duplicated
dashboards.
We
have
dashboards
that
basically
don't
work
anymore.
There's
a
real
danger
of
somebody
going
and
saying:
okay,
I'm
going
to
show
you
this
lovely
demo
and
then
nothing
works
and
we
have
a
lot
of
old
dashboards
and
it's
just
not
a
situation
that
we
can
continue
with.
A
C
But
the
challenge
is
that
a
lot
of
our
users
don't
actually
use
git
on
a
day-to-day
basis,
so
we
need
to
ensure
that
we
did
something
that
would
allow
them
to
use
the
tools
that
they
know
and
love
very,
very
well,
obviously,
to
do
something
that
allows
us
to
still
coping
over
overview
and
kind
of
get
some
view
of
what's
going
on,
but
also
that
is
very
minimal.
Our
part
to
actually
do
all
of
this.
A
Well,
we'll
build
this
tool,
let
them
save
dashboards
to
a
magic
folder.
They
don't
ever
have
to
even
open
GitHub
or
a
git
or
even
know
how
to
run
get
commands.
That's
where
they're
put
out
by
the
grafana
API
gets
committed
to
a
git
folder
structure.
Atlantis
runs
a
terraform
script,
that
I
wrote
that
is
able
to
iterate
over
that
same
folder
structure,
so
a
folder
of
folders
of
dashboards
and
just
run
that
in
a
loop
and
just
drop
them
in
exactly
where
they
need
to
be.
A
Should
you
do
this,
so
this
is
a
custom
solution
to
a
specific
problem
that
we
had.
This
is
something
we
were
actually
rolling
out
for
real
in
our
internal
environment.
For
for
those
groups,
we're
not
necessarily
saying
everyone
should
do
it
exactly
the
way
that
we
did
it,
but
this
is
intended
to
be
an
example
of
say
you
have
something
and
you
want
to
get
opposite,
but
your
users
are
not
really
get
Savvy.
A
This
is
like
an
example
of
what
you
could
do
as
long
as
the
tool
that
you're
trying
to
get
UPS
has
some
kind
of
API
in
a
way
of
programmatically
speaking
to
it.
You
could
do
something
like
this,
even
if
it
wasn't
exactly
this,
the
tools
are
there.
You
just
need
to
stick
them
together.
Do
a
bit
of
work,
Bang,
Your
Head
on
the
wall,
because
it
didn't
work
17
times.
You
know
all
that
kind
of
stuff.
C
But
what
about
maintenance?
Well,
oh
excuse
me.
Well,
as
you
can
see,
we
try
to
pick
some
off-the-shelf
tools
to
make
it
very
easy.
The
only
thing
we
had
to
do
was
write
a
demon
to
ensure
that
we
could
actually
work
with
grafana
API
and
create
the
pull
requests
and
actually
having
looked
through
GitHub
actions.
A
bit
more
I've
slowly
been
learning.
It
I
think
we
can
do
an
awful
lot
of
it
without
actually
having
to
have
a
demon
itself.
C
So
if
you
can
pick
the
tools
that
you
need,
you
know
and
configure
it
in
such
a
way
that
you're
able
to
maintain
it
with
low
mate
with
as
low
an
effort
as
possible.
Then
actually
it
probably
is
a
really
good
thing
to
do.
As
Eamon
says.
If
you've
got
users
that
are
very
high
level,
they
don't
actually
know
the
tooling
underneath
this
could
be
a
really
really
good
way
of
ensuring
that
they
can
kind
of
self-regulate.
What
they're
doing.
A
And
on
the
future
roadmap,
so
this
is
also
apparently
commensured
to
the
inevitable
question
of.
Is
this
open
source
and
can
I
have
it?
We
didn't
actually
make
it
public
at
the
moment
we're
like
kind
of
on
the
fence
about
whether
we
should
but
the
it's
not
because
we
don't
want
to
give
it
to
people.
A
C
A
Don't
hold
me
to
it
but
yeah,
so
there
is
like
official
work
there
and
like
we're
like
hesitant
to
like
just
hand
it
to
people,
because
then
people
will
start
using
it
and
maybe
that's
awkward
when
the
new
version
of
grafana
comes
out.
So
we're
like
I,
don't
really
know
what
we
should
do
there,
but
within
the
tool
itself,
we
were
thinking
of
doing
things
like
adding
data
sources
and.
C
So
yeah
there's
I
mean
one
of
the
one
of
the
things
as
I
mentioned
earlier.
Is
you
you
get
very
good
at
reading
Json,
but
that's
that's
good
if
you've
been
reading
it
for
a
long
long
time,
but
not
so
good.
If
you
know
you're
just
doing
something
else
and
you're
there
just
trying
to
admin
and
maintain.
A
F
C
To
be
bluntly,
honest
part
of
the
problem
that
we
have
is
there:
we
have
a
very
small
field,
engineering
team,
so
we
try
and
do
things
like
this
in
bits
and
pieces
when
we're
not
talking
in
depth
to
customers
about
Tempo
Loki
or
you
know
mamir
that
kind
of
thing.
So
it
is
there's
a
lot
of
things
on
the
roadmap,
but
unfortunately
that's
not
one
of
them.
That's
there
at
the
moment.
A
There,
it
would
be
wouldn't
be
that
great,
that
much
work
to
add
that
bit
because,
because
the
actual
provisioning
piece
of
it
is
effectively
terraform,
if
we
just
change
it
so
that
if
something
is
removed
in
a
certain
way,
it
removes
it
out
of
the
git
folder.
Well,
terraform
will
take
care
of
it
without
running
a
terraform
destroyed.
The
apply
will
recognize
that
it's
not
there
and
we'll
actually
delete
it.
So,
like
half
of
it
is
done
by
Deep
by
nature
of
it
being
terraformed.
A
C
Yeah,
it
will
so,
if
you're
in
a
if
the,
if
the
pr
was
still
open
at
the
point
where
somebody
comes
along
and
they
actually
change
it
again,
the
polling
will
actually
go
and
look
at
all
the
PRS
that
have
been
created
in
the
demon
will
go
okay.
Well,
this
has
changed.
The
version
number's
gone
up
now,
I'm,
going
to
essentially
update
what
we've
got
here.
C
Do
it
with
the
tags
and
ensure
that
the
plan
will
run
again
so
that
we're
not
in
a
situation
where
maybe
somebody
else
has
come
along
at
the
same
time
done
the
same
thing.
If
again,
this
goes
back
to
checking
to
make
sure
there
are
no
conflicting
versions
with
the
same
folders.
The
same
dashboards,
Etc.
D
A
E
A
G
A
D
A
Yeah,
the
way
that
Atlanta
seems
to
work
is
that
you
comment
Atlantis
apply
and
then
it
applies
it
and
then
the
pr
is
still
open
and
then
you're
supposed
to
merge
it
afterwards
and
it's
kind
of
weird,
because
you
sort
of
feel
like
you
should
merge
it
and
then
that
should
result
in
the
apply,
but
it
seems
to
be
just
that's
the
way
Atlantis
works.
I
wasn't
able
to
find
if
you
could
reconfigure
it
to
work
a
different
way.
That's
also
why
we
had
the
GitHub
action
that
runs
the
comments.
A
A
So
if
somebody
you're
saying
if
somebody
changed
the
dashboard
in
the
production
grafana,
so
they
can't,
because
we're
literally
disallowing
them
from
doing
that,
so,
like
the
only
people
who
could
do
it
would
literally
be
our
team
of
four
people,
because
we'll
be
the
only
people
with
permissions
to
do
that.
If
that
was
to
happen,
I
guess
you
just
delete
it
out
of
production,
then
just
re-push.
It.
C
From
obviously
telephone's
gonna
provision,
everything
again
the
thing
about
grafana
now
is
from
grafana
nine.
We
put
in
a
very,
very
detailed
fine
level,
grained
access
control
system,
so
we
can
actually
give
permissions
to
lots
of
people
to
do
everything
they
need
to
do,
but
not
edit
the
things
we
don't
want
them
to
edit.
So
actually
griffano
has
given
this
net.
Thankfully
grafana
has
the
ability
to
do
all
of
that
very
very
easily
without
having
to
worry
about
it.
C
A
Was
quick
and
easy
to
set
up
like
well?
No,
so
the
other
thing
that
you're
getting
for
not
directly
running
it
automatically
in
a
CI.
Is
you
get
to
see
it
before
it
runs?
So
it
comments
in
the
pr
shows
you
the
whole
plan
lets
you
see
what
the
diff
will
be
so
that
people
in
our
team
can
go
and
look
and
say.
Oh,
it's
going
to
do
this.
That
looks
okay,
we
can
let
it
run
instead
of
if
you
run
in
CI.
A
It's
always
assuming
everything
is
okay
and
everything
may
not
be
okay,
something
something
weird
could
be
happening
and
you
could
you
could?
You
might
never
see
the
diff
and
all
of
a
sudden
terraforms
wiped
out
every
dashboard
you
have
so
it's
kind
of
so
you
can
see
what
it's
going
to
do
and
save
yourself,
hopefully
from
some
kind
of
Destruction.