►
From YouTube: Peribolos: How Kubernetes Uses GitOps to Manage GitHub Communities at Scale - Christoph Blecker
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Peribolos: How Kubernetes Uses GitOps to Manage GitHub Communities at Scale - Christoph Blecker, Red Hat & Erick Fejta, Google
With 6 organizations, 150+ repositories, 400+ GitHub teams, and 800+ unique members, the Kubernetes project has a massive GitHub footprint. Scaling this footprint without excessive manual toil or impacting project velocity is a challenge. We addressed this challenge by defining our GitHub organization as configuration in a GitHub repository. We wrote tooling, which we called peribolos, that continually applies this config, including whenever we merge changes to it. Peribolos is built on top of prow, the CI system the Kubernetes project built and runs on. We will show how it works, discuss some of its design decisions around delegation and safety and help you decide whether it can benefit your organization.
https://sched.co/MPZA
A
So
I
wanted
to
kind
of
to
give
you
a
frame
of
like
the
problems
that
we
were
trying
to
tackle.
I
wanted
to
walk
through
one
of
those
manual
processes,
so
they
have
created
a
new
repo
and
I
want
to
create
a
github
team.
How
does
the
maintainer
--zz
forever
so
I
opened
up
my
org
and
I
click
new
team?
This
brings
up
a
new
screen
where
I
fill
out
details
and
creative,
now
github
by
default,
puts
the
person
that
creates
the
team
in
the
team.
A
So
if
I
don't
want
that,
I
need
to
go
and
remove
myself
by
clicking
here
and
here
and
here
and
here
now,
I'm
finally
ready
to
start
adding
people
to
this
team.
Let's
start
by
clicking
here
which
brings
up
this
screen
where
I
fill
out
a
name
and
click
invite
one
by
one
repeat
until
done
or
fingers
are
bleeding.
A
As
you
can
see,
any
manual
process
of
this
kind
is
going
to
be
time-consuming,
requires
clicking
through
multiple
screens
many
checkboxes
to
be
able
to
do
something
that
is
very
routine.
It's
also
error-prone
and
many
actions
require
owner
privileges
to
the
org,
to
do
very
simple
actions
outside
of
audit
logs
that
are
only
visible
to
those
people
with
the
owner
privilege.
There
was
no
visibility
into
who
made
what
change
and
when
we
needed
something
that
allowed
us
to
in
a
simple
and
clear
way,
take
actions
against
all
of
our
orgs.
A
We
wanted
to
provide
that
visibility
and
audit
trail
to
anyone
working
in
the
project,
even
those
outside
of
our
project,
similar
to
the
kubernetes
model.
We
wanted
to
reconcile
the
actual
state
with
our
source
of
truth
and
make
whatever
changes
we
needed
to
get
to
that
desired
state
we
needed
to
be
able
to
delegate
privilege
in
order
to
expand
the
pool
of
people
who
are
able
to
take
on
community
management
responsibilities.
And,
lastly,
we
needed
to
accomplish
this
all
in
a
safe
way.
A
B
So
the
you
know
basic
way
that
this
works
is
by
having
some
sort
of
desired
state
that
you,
then
you
know,
compare
that
with
the
actual
state
of
the
world
and
taking
any
actions
that
are
necessary
to
drive
towards
the
goal
state.
So,
for
example,
if
you're
familiar
with
kubernetes
deployments,
you're
going
to
declare
something
like
I,
you
know
want
my
pod
to
have
these
containers
and
these
number
of
replicas
the
these
arguments.
B
Whatever
you
declare
that
you
then
tell
the
API
server,
that
is
what
you
want,
and
then
it
will
notice
if
there's
any
discrepancies,
it'll,
add
replicas
or
remove
the
replicas
or
deletes
and
recreate
a
pod
with
a
new
image,
you've
declared
and
eventually
over
time,
and
repeating
that
you
will
eventually
get
to
the
state
that
you
want.
This
works
really
well
well
for
us
and
we
have
elsewhere
in
the
project.
B
Use
this
to
do
things
like
manage
github
labels,
it's
how
we
declare
what
branches
we
want
to
protect
on
various
repos
and
are
various
automation
tools.
We
use
similar
things
to
upgrade
from
version
X
to
X,
plus
one.
You
know
it's
also
nice,
because
any
errors
given
with
this
process
errors
are
corrected
over
time.
Whether
you
have
an
intermittent
error,
that's
giving
you
a
500,
or
maybe
you
have
miss
configuration.
The
way
that
you
handle,
that
is
by
you,
know,
either
retrying
or
else
fixing
the
configuration
to
what
you
actually
want
and
re
running
them.
B
So
applying
this
to
our
org
management,
you
know,
essentially,
you
will
declare
and
some
file
the
set
of
members
that
you
want
and
what
access
levels
you
would
like
them
to
have.
Then
the
tool
will
read
the
actual
members
that
you
have
in
the
organization
and
do
whatever
actions
are
necessary
to
go
to
the
state
that
you
want,
such
as
inviting
any
members
that
you
want.
B
That
aren't
there
right
now,
you
can
invite
them
to
the
work
if
they
have
the
wrong
level
like
if
they
are
a
regular
member
and
you
want
them
to
become
an
administrator,
you
can
promote
them
to
admin
and
if
they're
no
longer
part
of
the
or
you
can
it'll,
then
delete
them.
And
you
know
the
same
process
will
apply
for
the
various.
A
B
Types
of
metadata
or
data-
that's
like
metadata,
the
teams,
such
as
what
teams
exist,
who
are
in
the
teams
etc.
So
the
name
that
we
have
for
this
is
parabola,
and
that
is
a
I
guess.
The
name
comes
from
a
Greek
word
for
a
quart
enclosed
by
a
wall,
especially
one
surrounding
a
secret
area.
This
is
a
picture
of
the
parable,
is
outside
the
temple
of
apollo
and
obviously
not
the
kubernetes
community.
B
We
really
like
our
community
so
figured
potentially
a
good
name
also
the
general
design
is
and
our
expectations
of
how
we
think
or
expect
people
to
use
this.
Is
you
sort
of
take
your
current
state
of
the
world
and
dump
that
into
a
file?
Then
you
make
any
changes
that
you
want
to
make
it
more
correct
across
time
like
if
new
people
come,
you
can
add
those
we,
you
know
try
to
make
it
easy
for
you
to
make
those
changes.
B
Then
you
run
the
program
which
will
do
the
reconciliation,
loop
and
there's
a
variety
of
security
mitigations
to
make
sure
that
it
is
safe
for
you
to
run
it
without
running
with
that
bad
situation,
where
you're
locked
out
of
your
organ,
so
you
know
we
try
to
make
this
easy
to
run
and
play
around
with
so
right
now
or
whenever.
If
you
want
to
get
started,
it's
pretty
easy
to.
Do
you
don't?
Actually,
if
you
want,
you
don't
even
need
to
install
anything,
you
can
just
run
the
image
we
publish
images
on
any
commits.
B
B
So
the
again,
the
first
thing
we
would
expect
someone
to
do
is
dump
the
current
configuration.
So
we
have
a
convenient
way
to
dump
the
org
and
that
will
run
logs,
like
you
see
on
the
right
side
of
the
screen,
and
then
it
will
output
the
yamo
file.
That
is
expecting,
which
you
can
kind
of
see
an
example
of
what
that
might
look
like
on
the
left.
B
And
so
then
we
would
expect
you
to
you're
going
to
use
your
favorite
yamo
editor
to
make
any
changes
so
had
any
new
users
who
aren't
there
create
any
teams
that
are
missing
whatever
make
the
file?
How
you
want
it
to
be,
and
then
you
know
the
tool
is
a
little
bit
flexible
like
maybe
you
want
parables
to
manage
the
teams
that
exist,
but
you
don't
necessarily
want
to
control
like
who
is
in
each
team,
so
their
flags
to
control
those
things.
B
So
you
know
you
could
have
the
fixed
teams,
but
not
the
fixed
team
members
or
whatever
there's
sort
of
a
bunch
of
different
options.
You
know
to
control
that,
and
so
then,
once
you
are
happy
with
the
configuration
you've
declared,
we
would
expect
you
to.
You
know,
run
the
command
to
apply
it
and
that's
going
to
do
the
whole
read
the
configuration
you've
declared
compare
that
with
github
and
make
all
the
changes,
so
we
make
it
safe
by
default,
so
it'll
first
by
default.
B
B
Also,
you
know
we
try
to
make
design
choices.
That's
are
convenient
for
people
rather
than
the
machines,
since
machines
tend
to
be
good
at
you
know,
transforming
data
so
like
one
good
example
is
github.
Team
names
are
not
necessarily
unique,
so
the
org
can
have
a
bunch
of
different
teams
named
editor
and
those
are
all
going
to
be
backed
by
a
unique
ID.
B
But
it's
not
very
easy
to
write
a
config
that
to
declare
that,
since
you
don't
know
what
the
ID
will
be
ahead
of
time,
so
we
you
know
designed
it
so
that
we
will
essentially
require
you
to
have
a
unique
name
and
we'll
find
that
ID
and
then
use
that
if
they're
extras
will
delete
the
extras.
Likewise,
you
know.
Maybe
there
are
potentially
fields
that
you
do
want
manage
and
other
fields
you
don't
like
manage
like.
Maybe
you
don't
want
to
have
the
private
billing
email
for
your
organs?
B
And
so,
if
you
just
delete
that
from
the
config,
then
we
will
ignore
that
we
won't
care
what
the
value
is.
We
won't
try
and
drive
that
towards
a
particular
goal
and
then
also
an
important
thing.
You
know,
given
that
we
have
a
larger
organism,
want
to
allow
delegating
parts
to
different
SIG's,
which
is
sort
of
how
we
have
divided
our
org
into
sub
sub
groups
or
sub
teams
and
yeah.
B
We
want.
You
know
it's
important,
also
that
we
want
things
to
be
safe,
so
we
want
problems
to
be
more
on
the
funny
side
of
things
rather
than
you
know,
tragic.
So,
for
example,
a
you
know,
a
more
inconvenient
thing
would
be
if
every
time
you
run
it
because
of
some
weird
casing
issue,
we
wind
up
removing
you
from
the
org
and
then
five
minutes
later
re,
adding
you
that
obviously,
is
less
than
ideal,
but
substantially
less
tragic
than
if
a
bug
wound
up.
B
B
There
are
some
rate
limiting
you
know,
problems
with
github
that
we've
had
to.
You
know,
make
sure
that
we're
not
bossing
ourselves,
and
we
also
want
to
make
sure
that
we
run
this
in
a
safe
location.
That
isn't,
you
know
potentially
running
untrusted
presubmit
code,
so
that's
sort
of
how
the
Tool
Works
and
but
the
more
fun
thing
is
rather
than
just
you
know,
having
a
file
on
your
desktop
somewhere.
We
want
to
you,
know,
drive
this
with
get-ups
and
Christoph
is
gonna
talk
about
that.
A
A
A
So
it's
it
split
up
into
all
these
modular
components
and
parabola
is
actually
a
perfect
fit
for
this
system.
As
far
as
how
we've
designed
it
in
our
organization.
So
now
we've
got
this
flow
where
you
have
a
configuration
that
is
checked
into
a
get
repo
prowl
ends
up
listening
for
events
and
web
hooks
that
are
coming
from
github
yeah
and
is
able
to
process
and
respond
to
those.
So
we
use
this
again
in
lots
of
different
ways.
We
have
chat
off
scanned
to
to
take
actions
against
an
issue
or
a
PR.
A
We
also
listen
for
push
events
on
PR
branches.
We
listen
for
merge
events
on
like
our
master
branch
and
are
able
to
take
action.
Prowl
in
this
case,
we'll
can,
with
a
configuration,
spin
up
a
kubernetes
pot,
because
prowl
is
built
on
kubernetes
and
you
raise
so
we
can
spin
up
a
job
pod
with
the
kubernetes
pods
back
and
take
actions
when
we
hear
certain
events
in
the
case
of
parabola,
we
listen
specifically
for
when
a
PR
to
your
master
branch.
A
So
we
have
a
very
simple
kind
of
job
spec
that
looks
very
similar
to
a
kubernetes
pods
back
to
kick-off
a
job,
and
this
will
check
out
that
that
github,
that
git
repo
and
run
pair
bows
with
the
same
flags
as
you
saw
earlier
anytime,
something
merges
to
that
repo.
So
this
is
the
glue
that
then
takes
that
you've
got
now
an
authority
of
config
in
a
github
repo
and
when
something
new
merges,
you
want
to
take
action
right
away,
and
you
want
to
change
that
in
your
state
of
the
world.
A
So
I
went
I
started
off
with
a
workflow
of
like
creating
github
team
and
how
many
different
check
boxes
there
were.
So
what
is
that
same?
Workflow,
look
like
under
parables,
so
I
can
now
check
out
this
get
repo
I
copy
and
paste
a
little
bit
of
amyl
in
the
in
the
definitions
and
I
change
that
yeah
mole
to
define
this
new
team
that
I'm
creating
and
then
I
open
a
pull
request.
A
Just
like
we
do
for
any
piece
of
code
that
lands
in
kubernetes.
We
run
a
whole
bunch
of
go
line.
Unit
tests,
we
run
linters.
We
make
sure
the
change
is
safe.
That
change
gets
reviewed
by
other
people
in
the
the
organization
and
when
all
of
that
is
ready,
it
merges
to
master
branch.
A
parabola,
struttin
is
is
kicked
off
immediately
and
those
changes
take
immediate
effect.
A
So
because
you
express
the
desired
state
and
let
the
software
do
what's
needed
to
get
you
there,
the
workflow
is
more
simple
and
straightforward.
We
don't
need
to
train
folks
on
the
right
buttons
or
checkboxes
to
use,
as
the
software
takes
care
of
all
of
that.
We're
also
able
to
drive
our
own
standards
and
policy
with
golang
unit
tests
that
catch
out
of
policy
changes
before
they
take
effect.
B
So
it
hasn't
always
been
smooth
sailing,
and
in
case
you
haven't
had
enough
nautical
puns
this
week,
but
one
potentially
one
early
issue
that
we
had
was
around
token
usage.
So
that's
when
you
run
parabola
seeds
time
it
has
to
pull
down
the
entire
state
of
what
exists
in
github.
All
your
organization's
members
penny
members,
teams,
etc.
B
That
can
take
a
lot
of
API
tokens
and
github
has
rate
limits
in
place
to
prevent
this
abuse,
some
of
which
they
publicize
other,
which
are
sort
of
secret
and
internal,
and
this
you
know,
protects
their
service
from
you
know:
rogue
actors
harming
it,
and
so
when
we
first
turned
it
on
they
liked,
one
visible
limit
is
5,000
requests
per
hour
and
parabola
star
org
is
large
enough.
Where
we
quickly,
you
know,
consumed
all
that.
B
Obviously
that
is
not
good,
especially
because
at
that
you
know
we
were
using
the
single
token,
so
it
was
shared
from
the
rest
of
our
CI
system.
So
not
only
did
the
parabola
strum
fail,
but
then
you
know
pre
submits
that
prow
was
running
elsewhere,
we're
failing
which
you
know,
makes
people
unhappy.
So
we
saw
that
you
know
a
couple
ways
one
is
we
implemented
a
tuning
throttling
system
in
our
code?
B
That
way,
you
know
you
can
tell
what
the
what
the
limit
is
per
hour
and
per
run
and
how
quickly
you
wanted
to
consume
those
tokens
and
so
yeah.
So
again,
following
our
you
know
safe
by
default,
we
have
it
be
some
reasonable
standard
and
we
can
increase
it
or
decrease
it
as
as
required,
and
so
that
way
we
know
that,
even
if
you
know
you're
using
the
same
token,
through
multiple
parts
of
your
system,
that
one
part
isn't
going
to
consume
all
of
them
and
so
it'll
slow
it
down.
B
But
you
know,
then
that's
better
than
having
something
to
be
broken.
The
second
thing
which
potentially
more
interesting
is
we
have
a
optional
caching
layer
as
part
of
prowl
called
gh
proxy
that
exists
in
a
repo,
and
that
allows
us
to
cache
the
response
as
we
get
from
github
and
revalidate
things
that
get
back
from
github.
So
we
cache
the
calls
that
we
make
and
then
we
do
a
no-cost
validation.
B
B
Then
we
see
that
hey,
Alice
Smith
is
now
gone
from
the
repo,
so
we
attempt
to
add
her
back
to
the
org,
but
that
no
longer
exists
and
so
parabola
fails
and
blows
up
and
whatever
other
actions
you
know
it
might
take.
You
know
it
then
finishes.
Luckily,
you
know
this
doesn't
tap
in
super
often.
So
our
solution
you
know
is,
we
expect
someone
to
go
in
correct
the
name
to
say
Alice.
B
Instead
of
Alice
Smith
and
then
we
run
it
and
the
next
time
that
we'll
fix
itself,
but
that's
obviously
something
we
want
to
improve.
So
some
other
things
we
want
to
improve
is
we'd
like
to
extend
parables
to
manage
repose.
So
there's
a
similar,
you
know
amount
of
annoying
button
clicking
you
have
to
do
to
create
new
repose
and
so
we'd
like
to
eliminate
that
laborious
clicking.
We
also
would
like
to
delegate
do
the
same
delegation
to
SIG's
to
where,
where
SIG's
can
create
all
the
teams
they
need.
We
would
like
to.
B
We
would
like
to
enable
SIG's
to
create
whatever
repose
they
need
and
do
this
with
the
same
pattern.
That's
been
working
well
for
us
for
the
or
configuration
there's.
Also
some
interesting,
you
know
edge
cases
around
concurrency
where
maybe
like
we
mentioned.
However,
there's
both
will
both
kick
off
a
will,
kick
off
the
job
when
you
merge
a
commit,
but
we
also
have
a
time
thing.
Those
could
potentially
line
up
or
maybe
there's
multiple
commits
that
happen
really
quickly.
B
So
you
could
wind
up
in
a
weird
state
where
there's
you
know
three
versions
of
the
job
that
are
running
slightly
different
have
slightly
different
goals,
so
obviously,
eventually
that
will
correct
itself,
but
there
could
be
some
weird.
You
know
case
where,
for
a
few
hours
will
like
invite
you
to
the
org
or
create
a
team
and
five
minutes
later
delete
it.
That's
you
know
not
good
we'd
like
to
make
that
better.
We
also
want
to
do
better
delegation.
B
In
addition
to
delegating
the
repo
stuff
we
want
to
right
now
we
sort
of
depend
on
basil
to
stitch
together
the
various
files
into
the
one
big
file-
that's
parabola
so
actually
reads,
and
we
would
like
to
remove
this
dependency
on
basil
and
just
have
it
work.
You
know
purely
with
go
code
yeah,
so
that's
so
yeah.
B
So
we'd
really
like
you
to
you,
know,
try
this
and
either
by
contributing
to
adding
some
contributions
to
how
we
use
it
in
the
kubernetes
org
or
if
it's
applicable
to
where
so
orgs
that
you
interact
with
use
it
there.
You
can
check
out.
You
know
here's
the
command
from
earlier
to
try
running
it
yourself.
Here's
a
link
to
our
source
code
on
github
there's
also,
you
can
look
at
our
public
declaration.
B
So
if
you're
wondering
who
is
in
the
kubernetes
org,
you
can
read
that
repo
and
figure
it
out
and
if
you
want
to
add
yourself
to
a
team,
you
can
create
a
pull
request
to
add
yourself
to
that
team
and
once
you
know
merges
you
can
see
all
the
exciting
things
happen
and
yeah
we're.
You
know
a
bunch
of
friendly
people
where
you're
going
to
contact,
Christophe
and
I
on
slack
or
github,
then
there's
also
the
contributes
cig
testing
and
proud
channels.
On
slack,
that's
we're.
You
know,
happy
and
friendly.
B
A
We
we
looked
at
that
when
we
were
originally
trying
to
to
find
a
solution,
but
at
the
time
there's
a
bunch
of
like
extra
things
with
terraform
like
we
needed
to
find
a
way
to
store
state.
We
needed
to
there's
a
whole
bunch
of
problems
that
terraform
would
have
introduced
that
weren't
conducive
to
our
model.
So,
as
Eric
was
was
saying
earlier,
we
were
looking
to
the
kubernetes
model
of
hey.
Can
we
design
something
where
we
just
have?
This
is
desired?
A
This
is
real-world,
do
whatever
you
need
to
drive
forward
to
that,
without
needing
to
maintain
that
third
kind
of
post
of
what
was
that?
What
was
the
state
of
the
world
the
last
time,
terraform
Ram,
so
it
just
wasn't
fitting
to
what
we
needed
to
do
specifically
as
well
as
the
the
way
we
write
it
like
we're.
Writing
everything
in
yeah
mole
as
far
as
most
of
our
configurations.
A
Right
now,
I
think
it
is
hard-coded
in
in
how
it's
set
up
that
were
hard-coded
against
the
github
API,
but
we
have
we
have
some
there.
There
is
some
ongoing
work
in
prowl
in
general
to
make
it
more
flexible
and
to
work
with
github
Enterprise.
So
it's
definitely
something
that
we
would
be
open
to
is
like
a
configurable
flag
to
change
or
to
change
the
host
to
be
able
to
talk
to
a
github
Enterprise
instance.
All
the
same
API
is
that
we're
using
are
available
on
github
Enterprise
as
well.
There.
B
Are
flags
to
sort
of
say
what
the
end
point
is,
but
I
there
seem
to
sometimes
be
some
edge
cases
like
we
don't
typically
use
github
enterprise,
but
we
there
are
users
of
prowl
who
use
that,
and
so
we've
been
receiving
some
pull
requests
to
enable
it
more
of
Prowse
functionality
for
that
and
yeah.
So
I'm
not
sure.
By
the
way
we
also
have.
If
anybody
wants
a
prowl
sticker,
you
can
come
up
afterwards
and
get
them.