►
From YouTube: Managing GitHub At Scale - GitHub Universe 2017
Description
Presented by Alvaro Ramirez del Villar & Frank Lamar, Home Depot.
As engineering teams move toward different build systems that rely on polling, it will be essential for GitHub Enterprise admin to know how to identify, educate, and help remediate the build system. If they can’t, their instance will become unstable and ultimately fail. When it happened to us, our team came up with the best solution: We diagnosed and created the automation required for monitoring and remediating the problem. And we gave back to the open source community in the process. Now, we can help others navigate such a tricky problem by not only showing them what we did, but also by giving them monitoring and remediation tips.
About GitHub Universe:
GitHub Universe is a two-day conference dedicated to the creativity and curiosity of the largest software community in the world. Sessions cover topics from team culture to open source software across industries and technologies.
For more information on GitHub Universe, check the website:
https://githubuniverse.com
A
Computer
color
matching:
how
long
did
you
say,
computers
right,
our
optical
computer
will
match
any
color
perfectly
in
just
60
seconds.
Well,
that's
great,
but
are
we
talking
abyss
brands,
we'll
use
our
very
best
paint
Behr
premium
plus
just
bring
us
any
settle
anything
and
we'll
guarantee
a
perfect
match.
Absolutely
free
at
Home.
Depot
low
prices
are
just
the
beginning.
B
C
C
B
Right
so
we
came
today
to
talk
to
you
about
our
experience
managing
github
at
scale,
so
we're
enterprise
customers,
so
we're
not
using
github
com.
We
have
our
own
on-premise
installation
and
we
just
wanted
to
share
some
of
our
experiences,
joys
and
challenges
about
what
that
looks
like
all
right.
We're
gonna
cover
managing
github
at
scale,
we're
migrating
from
SVN
to
github,
which
was
our
original
version
control
system.
C
So
to
understand
this
whole
journey
I
think
what
you
need
to
understand.
First,
is
kind
of
the
environment
prior
to
all
of
this
transition.
So
before
we
transition
to
github
and
we
did
the
whole
digital
transformation,
we
had
kind
of
a
central
governance
for
all
of
our
developers.
So
that
means
that
we
had
managers
and
admins
making
decisions
about
what
kinds
of
languages
developers
could
use.
We
had
decisions
around
what
kind
of
build
systems
they
could
use
and
we
had
strict
and
strong
controls
over
all
of
those
things.
C
Now
that
doesn't
sound
like
the
kind
of
collaboration
and
whatever
else
we
that
went
on
next
door
here.
So
let's
talk
about
how
we
got
there
right,
so
this
wasn't
some
evil
plot
right.
We
we
had
a
lot
of
bad
a
lot
of
management
and
admins
saying
we
don't
want
our
developers
to
have
to
worry
about
how
do
I
build
the
software.
How
do
I
deploy
the
software
into
our
complicated
environment?
C
So
what
we
did
was
we
said:
alright,
we're
going
to
go
ahead
and
create
tools,
we're
going
to
create
some
structure.
Some
things
like
this
that
will
allow
developers
to
just
worry
about
developing
all
they
have
to
do
is
worry
about
the
business
processes,
the
different
business
logic
that
they
need
to
write
and
things
like
that.
They
don't
have
to
worry
about.
How
do
I
get
this
deployed
to
our
QA
environment?
How
do
I
get
this
deployed
to
our
production
environment?
So
we
took
care
of
all
that
for
them.
C
The
only
issue
with
that
is,
as
that
kind
of
self
engrained,
it
became
a
really
tight
noose
around
our
developers
right.
So
we
had
very
strict
controls.
We
didn't
let
our
developers
choose
the
language
that
was
appropriate
for
the
software.
They
were.
Writing
we
told
them.
This
is
the
language
you
need
to
write
in.
This
is
the
build
tool
you
need
to
use.
This
is
how
we
progress
from
one
stage
to
another
and
that
you
know
kind
of
made
management.
C
Look
a
little
bit
like
this,
and
eventually
you
know
we
kind
of
develop
our
sauce
more
like
this.
We
have
this
power,
but
we're
not
using
it
correctly.
We
look
kind
of
like
buffoons
right
and
so
somewhere
along
the
line.
We
can't
everybody
that
was
kind
of
giving
that
direction
realized
wait
a
minute.
This
isn't
the
right
way
to
do
it.
We
need
to
loosen
our
rein,
so
we
need
to
let
go
of
a
little
bit
of
this
control
and
give
the
developer
a
lot
more
power.
B
All
right
so
when
we
gave
developers
that
freedom
they
had
an
initial
reaction,
they
were
excited.
They
were
excited
to
see
the
culture
of
the
Home
Depot
changing
generally,
enterprises
move
really
slow,
especially
enterprises,
but
they
were
really
excited.
They
were
getting
this
same
opportunity
to
move
fast
more
like
a
traditional
software
company.
In
addition
to
taking
the
reins
off,
we
also
allow
you,
whatever
language,
we
were
no
longer
just
a
Java
shop.
If
you
thought
Python
was
the
right
choice,
then
by
all
means
used
Python.
B
He
thought
JavaScript
was
the
right
choice
by
all
means:
use
JavaScript.
If
Jenkins
was
too
heavy
weight
of
a
build
tool
for
what
you
need,
it
use.
What
you
thought
was
best.
If
you
felt
like
maven
was
better
than
Gradle,
then
you
had
the
freedom
to
choose.
You
had
the
freedom
to
do
that.
We
also
integrate
in
the
season.
B
We
also
implemented
a
lot
of
things
from
agile
manifesto
as
well
as
extreme
program,
and
we
introduced
retros,
where
teams
got
a
chance
to
voice
what
they
felt
like
the
team
was
doing
right
and
the
team
was
doing
wrong.
They
had
more
control
over
their
destiny,
and
this
guy
seems
really
excited
about
code.
B
Developers
could
then,
whether
you
were
doing
a
greenfield
project
or
a
legacy
project?
You
had
the
opportunity
to
assemble
your
code,
assemble
your
tools,
the
best
way
that
you
saw
as
a
part
of
that
transformation
developers
requested
a
more
modern
version.
Control
system
and
github
seemed
to
be
the
de
facto
standard,
and
they
were
really
excited
about
using
github,
we
became
github
enterprise
customers.
B
But
we
became
get
hub
many
of
our
customers,
but
the
engineering
teams
had
shot
first,
some
of
them
already
had
Roget
labs
and
other
bitbucket
servers
out
there.
But
once
we
said
again,
a
github
Enterprise
was
coming.
Developers
were
happy.
We
created
the
instance
and
we
set
it
up
initially
with
small
group
of
developers.
A
couple
teams
were
allowed
to
test
it
out,
and
then
we
open
it
up
to
the
enterprise
at
large
and
everyone
was
really
excited.
We
gave
teams
an
entire
year
to
migrate.
B
So
if
you
had
a
legacy
project
on
our
old
version
control
system
that
would
still
be
available
to
you
for
one
year
and
at
the
end
will
be
turn
that
off
and
we
expected
everything
to
be
in
github
Enterprise
by
that
time,
if
you
were
having
a
greenfield
project,
you're
starting
brand
new
courage
to
start
right
from
with
github
from
the
beginning.
So
how
do
you
think
that
turned
out
you
think
everybody
was
ready
to
go
by
the
time,
the
cutoff
date
exactly?
No
one
was
ready.
It
was
like
hurting
octo
cats
all
right.
B
So
all
these
new
changes
going
on
another
thing
that
we
embraced
was
see,
ICD
and
see.
Icd
depend
on
who
you
ask
and
have
different
definitions
and
can
mean
different
things
to
different
people.
The
definition
that
we
settled
on
was
continuous
integration,
continuous
delivery,
because
we're
Enterprise
we're
not
moving
as
fast
as
other
quote.
Unquote.
Software
companies
we
need
to
bundle,
features
together
and
then
come
out
with
releases
depend
on
what
your
sprint
was
every
two
weeks
or
every
week.
Okay.
B
So
this
is
a
kind
of
outline
of
what
the
CI
CD
process
is
I'm,
not
sure
how
technical
the
room
is,
but
basically
the
development
is
ongoing
and
since
you
get
a
new
set
of
features
that
you
want
to
build,
you
build
MVP
release
that
as
soon
as
possible,
you
iterate
over
it,
get
it
into
your
hands
of
your
users.
You
get
feedback
and
you
continue
to
iterate
on
that
over
and
over
again,
so
that's
kind
of
kind
of
what
we
were
doing
in
the
continuous
integration,
continuous
delivery
of
them.
B
They
created
a
template
project
that
other
the
posts
to
study
and
learn
how
CI
CD
will
work
and
then
build
their
own
C,
ICD
pipelines
from
that.
So
did
they
study
it?
No,
they
just
copied
and
pasted
they
copied
and
pasted
this
project
into
theirs
to
create
their
pipelines
without
fully
understanding
what
it
meant
and
what
the
ramifications
would
be.
C
C
Happened:
yeah.
Well,
we
fell
over
right,
so
we
were
unaware
of
it
at
the
time
right
so
Oh,
most
of
the
concourse
sea-ice
talk
that
we
just
gave
we
kind
of
learned
about
after
this
terrible
thing
happened
right.
Our
instance
fell
over
and
as
the
good
software
engineers
that
we
were,
we
increased
memory.
We
increased
everything.
Cpu
memory
made
a
huge
instance
because
we
were
running
out
of
bandwidth.
C
We
were
running
out
of
resources,
so
we
just
increased
everything
now
that
had
some
secondary
side
effects
inside
the
VMware
space,
where
we
were
being
V
motioned
and
that
the
the
instance
was
unstable
here,
I'd
like
to
do
a
quick
shout
out
to
the
github
support
folks,
who
helped
us
figure
out
a
lot
of
what
was
happening
with
our
instance.
At
this
point
they
were
actually
the
ones
that
suggested
for
us
to
look
at
the
clones
because
they
said
wait
a
minute
guys.
You
have
tons
of
floor
requests
coming
in.
C
You
need
to
go,
look
at
that
and
figure
out
what
is
going
on
there,
and
this
is
what
they
gave
us.
They
said
here.
You
have
we're
seeing
a
lot
of
clones.
You
can
use
this,
this
grep
against
the
babel
dialog
to
see
the
counts
for
you
and
figure
out
where
they're
coming
from
what
they're
doing
so.
This
was
our
first
attempt
at
kind
of
mitigating
the
problem
before
we
even
knew
what
it
was
of
what
a
wire
instance
was
so
unstable.
C
So,
as
you
can
see
here,
we're
just
hitting
the
babble
D
log
we're
grabbing
for
get
upload
pack,
which
is
what
happens
when
you
do
a
git
clone
right,
any
good
clone
triggers
this
we're
separating
it
by
repo
and
then
we're
sorting
it
and
then
we're
writing
it
to
a
file.
Now
this
was
our
kind
of
first
attempt
at
how
to
figure
out
one.
What
was
going
wrong
with
our
instance
and
then
to
how
do
we
get
the
word
back
out
to
the
developers,
the
users
to
say:
hey
guys,
you're
killing
us?
C
We
need
you
to
help
right,
so
this
created
a
file.
We
use
that
file.
We
grabbed
it
from
our
instance.
We
we
would
put.
We
posted
that
file
using
slack
our
interoffice
messaging
software.
We
wrote
emails
to
the
highest
level
users
and,
in
addition,
since
we
were
sorting
on
repo,
it
kind
of
gave
us
an
indication
that
what's
happening
here,
is
that
concourse
CI
right
is
that
the
new
build
system
is
pulling
too
much,
and
it's
not
really
working
out
for
us.
C
So
after
all
of
this
research
that
we
did
and
notifying
the
users
you
know,
what
would
you
expect
would
happen
in
an
organization
that
you
know
of
responsible
people
right?
No,
no,
nobody
did
anything,
nothing
changed.
We
sent
emails
we
sent,
but
there
was
no
there's.
No
there
wasn't.
There
wasn't
really
understanding
of
what
the
issues
were.
Who
was
doing
it?
C
We
had
to
enter
into
kind
of
aggressive
negotiations
right,
so
we
had
these
high
intensity
users
that
were
assaulting
us
unbeknownst
to
them
a
lot
of
times
and
what
we
did,
what
we
decided
to
do
in
order
to
avoid
this
situation
right,
which
was
concourse
trying
to
Deathstar
our
poor
github
Enterprise
instance,
was
we
created?
Oh.
C
No
all
right,
so
what
we
did
was
we
started
posting
in
slack
in
an
automated
fashion
right,
so
we
we
would
post
the
top
50
users-
and
we
said
hey
guys
here,
the
people
that
have
started
doing
kind
of
too
much
poling
right
and
so
that
public
venue
of
posting
really
kind
of
started
turning
the
tide
a
little
bit
so
that
we
could
avoid
the
disaster.
That
would
be
this
so.
B
We
also
had
to
educate
our
development
community
since
they
had
adopted
conch
or
CI,
and
that
was
the
CI
tool
of
choice
we
kind
of
had
to
educate
them
on
how
it
worked
like
concourse.
These
green
squares
and
rectangles
represent
tasks,
and
a
task
is
like
a
single
unit
of
work,
and
it's
excuse
me
other
jobs
and
then
inside
a
job.
You
have
tasks
and
task
or
a
single
unit
of
work.
The
black
rectangles
that
you
see
on
the
outside
or
resources.
So
the
resources
are
things
that
you
pull
into
your
project.
B
B
B
All
right
so
without
consulting
our
HR,
we
decided
we
got
to
really
get
these
guys
attention.
So
we
created
shame
by
shame.
Bot
was
a
bot
that
was
written
as
a
Python
slack
bot
that
we
used
to
communicate
to
our
developers
and
really
like
kind
of
post
it
on
the
wall.
Here
are
a
high
clone.
Users
here
are
high
intensity
users,
so
the
private
emails
didn't
work.
B
The
the
private
slack
messages
didn't
work,
but
once
it
became
public
and
everyone's
that
when
the
entire
development
community
was
aware
of
who
was
doing
the
high
clone
and
who
was
doing
the
high
using,
then
people
were
coming
to
us.
How
can
I
lower
my
clone
you?
How
can
I
get
out
of
this?
People
did
not
want
to
be
put
out
as
the
people
who
were
responsible
for
the
instability
that
we
were
seeing
in
github.
B
C
How
didn't
rewrite
shame,
but
obviously
pair
programming
right
here
we
go
perfect
example:
we
got
together
on
our
team
collaborated
a
little
bit
using
github
and
figured
out
the
best
way
to
get
Shane
bot
going
that
initial
post
of
Shane
bot
to
our
public
to
our
public
get
slack
channel.
Well,
enterprise-wide
really
got
people
to
notice.
We
got
contacted
by
managers
of
these
different
users,
saying
hey.
C
C
So,
let's
take
a
look
at
it.
This
is
an
initial
version
of
our
Shane
bot.
I
tried
to
get
our
current.
What
our
current
looks
like,
but
surprisingly
PowerPoint
is
not
the
best
way
to
show
people
code.
So
when
you
have
a
lot
of
it,
it
starts
looking
like
gobbledygook.
So
what
I
did
was
this
is
kind
of
our
first
temped,
which
shows
in
very
basic
fashion
what
we're
doing
so.
We've
iterated
on
this
and
made
it
a
little
bit
better,
but
the
the
bones
are
here
right.
So
what
do
we
do?
C
C
C
At
that
point,
this
one
is
a
function
that
processes
our
user
counts.
So
we
have
the
input
of
that
log
that
has
all
of
our
pull
requests.
What
we
do
is
we
then
pull
out
just
the
user
name,
which
you
can
see
that
log
entry
is
keyed
is
a
key
value
pair
log.
So
we
we
look
for
the
user
item
and
then
grab
the
next
one
after
splitting
it
on
all
the
equal
signs.
C
So
once
we
grab
that
user,
the
babble
D
log
has
a
wealth
of
information,
but
it
stores,
depending
on
how
you
did
the
clone,
either
with
a
login
or
with
touken
or
with.
However,
it
stores
the
user
information
slightly
differently,
sometimes
so
here
that
second
dock
is
we're
doing
the
translation
based
on.
If
you
use
the
token
or
not,
then
what
we
do
is
we
just
count
all
the
names
and
we
display
them
next
to
the
count
by
name
and
then
we
sort
it
in
reverse
order.
C
Then
we
run
our
daily
slack
job
that
we'll
post
this
information
to
slack
for
us
our
original
our
original
thing,
as
you
saw
outputted
a
file,
and
we
started
going
down
that
road
to
see.
If
we
could
programmatically
post
a
file
to
slack
that
posed
some
problems.
There
was
not
really
some
good
documentation
there,
so
we
decided
instead
to
just
post
text,
so
here
this
function,
what
it
does
is
does
a
lot
of
formatting
around
how
we
want
to
display
that
text
so
that
users
can
consume
it,
and
it's
not
just
some.
C
You
know
random
smattering
of
names
and
numbers,
so
you
can
see,
we
use
shame
bought
there.
We
post
it
to
whatever
channel
we
want,
and
then
it's
the
gate,
usage
and
the
report
type
is
either
clones
or
pushes
so
here
we're
doing
top
50
most
frequent
clones
by
either
user
by
org
by
repo
we
break
it
down.
C
The
commands
are
very
similar
based
on
what
you're
matching-
and
this
is
a
lot
of
that
formatting
here
in
the
middle-
that
I
was
talking
about
to
try
and
get
it
to
align
nicely
when
you
display
it
on
slack,
which
is,
you
know,
doesn't
have
the
the
same
type
space
and
then
in
the
bottom.
There
we
we
just,
we
curl
it
against
the
slack
web
book
URL,
and
so
what
this
does
for
us
is.
C
It
displays
as
a
message
the
the
top
50
most
frequent
clones
by
user
by
repo
that
and
then
so
people
can
see.
Oh
yeah,
that's
my
username
I
know.
I,
don't
want
to
be
on
this
list.
That
has
you
know
the
shame
bot
list.
So
you
know
this
helped
us
kind
of
get
the
word
out
there
and
have
people
understand
that
you
know
what
they
were
doing
and
how
they
were
impacting
our
instance.
B
Alright,
so
that
was
shame.
Bot
was
one
of
the
tools
that
we
use,
that
we
have
built
around
our
github
Enterprise
instance
to
make
our
lives
as
github
admin
easier.
We're
a
small
team,
we're
a
team
of
less
than
six
and
we're
supporting
over
2,000
developers
and
github
Enterprise
is
not
the
only
tool
that
we
support
so
along
with
Shane
bot.
B
One
thing
we
created
was
github
suspender,
so
some
people
still
refuse
to
change
either.
They
did
not
know
how
to
adjust
their
CI
CD
pipelines
to
poll
less
or
they
were
just
unwilling
to
they
didn't
care.
We
created
our
own
suspender
and
what
our
own,
what
our
suspender
would
do,
would
look
at
github
uses
activity
and
suspend
based
on
certain
criteria.
Another
challenge
that
we
faced
was
was
managing
the
seats
within
github
when
you
personally
github
Enterprise
license
you
get
so
many
number
of
seats.
B
So
we
had
a
lot
of
people
who
were
close
to
the
code
but
weren't
necessarily
committing
on
a
regular
basis,
so
like
project
managers,
designers,
business
analysts,
who
would
also
sign
up
for
github
log
in
one
time,
look
at
something
and
then
never
log
in
again,
but
they
were
still
taking
up
a
seat.
It
got
so
bad.
So
one
point
I
think
we
were
down
to
like
20,
20
or
so
seats
left
and
the
very
next
week
we
had
like
a
hundred
and
ten
intern
starting
our
IT.
So
it
was
like
okay.
B
Well,
we
can't
go
in
and
keep
on
suspending
people
manually,
and
that
was
where
github
the
spinner
was
born.
So
it
was
suspend
for
bad
behavior
as
well
as
if
you
were
inactive
in
the
instance
for
X
number
of
days.
So
I
think
we
did
like
30
years
or
60
days
and
it's
in
the
github
documentation
on
how
you
know
how
to
go
in
and
do
that
and
if
you
suspend
a
user
all
their
stuff
is
still
there
their
repos
their
commits
their
pool
requests.
B
Everything
remains
the
same
it
just
they
can't
access
github
and
there
license
becomes
free
for
someone.
Who's
gonna
be
no
more
active
in
it.
Alright,
so
we
skinned,
we
started
suspending
people
for
various
reasons.
Now
we
started
getting
a
slew
of
emails,
hey
my
account
is
suspended
and
I
need
to
go
back
in,
and
this
is
now
taking
up
a
large
portion
on
our
day.
B
So
we've
decided
to
create
our
own
self-service
portal,
where
users
could
go
in
and
unsuspended
themselves,
so
as
long
as
they
still
had
access
to,
they
had
their
proper
credentials
to
access
github.
In
the
first
place
they
could
go
in
and
unsuspended
a
gentile
out
to
the
github
team
who
provided
us
with
our
very
own
octocat
for
our
unsuspended
that
we've
used
there
and
a
couple
of
other
places
on
the
developer.
B
C
We
create
a
shame
bot.
We
started
posting
to
slack
a
lot,
but,
as
you
guys
know,
once
you
see
a
notification
enough
times,
you
start
ignoring
it
right.
It's
like
oh
yeah,
it's
all
right,
9
a.m.
every
day
doesn't
matter
nobody's
looking
at
this,
it's
not
really
important,
so
we
decided
that
we
needed
a
shame.
Bot
2.0,
let's
give
shame,
bought
some
real
teeth
right.
Let's
take
advantage
of
some
of
these
tools
that
we've
written
for
other
things
and
really
integrate
that
into
our
shame,
bot
approach.
So
what
were
we
were
going
to
do?
C
C
Necessarily
the
best
thing
don't
want
to
be
told
that
they're
not
doing
the
right
thing.
They
don't
want
to
be
shamed
and
they
don't
want
to
be
suspended.
So
shame
about
2.0
got
a
bit
of
a
reworking.
Some
people
call
to
our
attention
that
maybe
the
name,
shame
bot
wasn't
the
best.
You
know
we
meant
it
as
a
joke
is
kind
of
some
some
happy
times,
but
some
people
were
like
you
know,
we
shouldn't
be
shaming
our
associates.
C
That's
not
really
part
of
our
values
wheel
right
so,
instead
of
shame,
bot
2.0
it
became
do
the
right
thing,
facilitator,
real
nice
and
you
know
it
it
kind
of
made.
You
feel
good
about
it
right
so
and
instead
of
suspending
people,
what
we
agreed
upon
with
the
community
at
large
was
that
we
would
email
the
developer
that
was
responsible
for
these
clones
and
if
they
didn't
respond
in
a
week,
you
just
escalated
it
to
also
emailing
their
manager.
And
then,
if
they
didn't
respond,
you
could
escalate
it
even
higher.
C
B
B
So,
as
far
as
like
onboarding
new
associates,
we
want
to
insert
ourselves
into
the
process
of
teaching
them
and
educating
on
how
to
use
github,
along
with
the
rest
of
our
tools,
the
right
way,
so
they're
good
stewards
of
the
instance
from
the
very
beginning,
and
also
we
created
using
github
pages.
We
created
our
own
set
of
Home
Depot
flavor
usage
guides
for
github
Enterprise,
as
well
as
all
of
our
other
tools.
So
we
hosted
all
this
on
github
pages
like
these.
B
This
is
the
best
way
to
use
this
particular
tool
with
the
Home
Depot
stack.
This
is
the
best
way
to
be
a
good
steward
of
the
instance,
and
we
want
to
make
that
collaborative.
We
open
source
that
we
gave
it
to
the
community
so
if
they
could
edit
those
pages
to
reflect
the
things
that
they
learned
out
in
the
trenches,
because
we're
weird
github
admin
we're
not
necessarily
building
apps
that
our
customer
faces.
B
Most
of
the
tools
that
we
deal
with
our
internal
facing
and
the
other
developers
are
our
customers,
but
they
may
pick
up
on
practices
or
ways
that
we
can
use
the
tools,
that's
better
for
the
community
at
large.
So
we
open
sourced
that
within
the
Home
Depot
and
allow
them
to
contribute
and
give
back
to
the
education
to
that
process
as
well.
C
Just
just
one
thing
to
add
in
that
vein
of
us
being
more
github
admins
and
we're
not
really
as
in
touch
with
say,
like
how
Concourse
you
know,
cuz
a
lot
of
teams,
we're
kind
of
building
their
own
concourse
I
want
to
call
out
that
before
we
came
up
with
the
shame
bot
idea,
there
was
a
specific
fix
to
the
problem
of
cloning
right
initially,
concourse
did
not
offer
web
hooks
right,
Home,
Depot,
open
source
and
collaborated
back
to
that
project
and
added
web
hook.
Support
to
concourse.
C
So
at
that
point
is
when
we
said
all
right
guys
before
there
wasn't
a
way
to
do
it,
and
if
you
needed
to
get
your
build
every
two
seconds,
then
you
had
to
pull
but
now
with
web
hooks,
which
I
think
is
even
kind
of
the
suggested
approach
from
a
github
site.
Anyway.
Now
that's
available
to
all
the
build
systems,
so
you
really
don't
need
to
be
pulling
unnecessarily
just
for
some,
some
quick
numbers
on
the
poling
versus
the
pushing
side
that
we
did.
Shane
bought
Fame
bought
yes.
C
This
morning,
when
the
report
came
out,
we
had
the
top
repo
on
the
push
side
was
like
a
hundred
or
so,
and
the
top
repo
on
the
pull
side
was
a
hundred
thousand.
So
let's
say
that
a
hundred
things
were
pushed
and
needed
to
be
pushed
out
to
deployed
with
the
CI
CDA
mindset.
We
still
pushed,
we
still
pulled
a
thousand
times
more
often
than
we
needed
to,
and
that's
the
kind
of
behavior
that
we're
trying
to
encourage
users
to
not
do
that's
just
a
waste
all.
B
Right,
so
that
was
pretty
much
our
this
portion
of
our
experience
being
github
enterprise
customers
and
managing
in
github
at
scale
coming
into
some
problems
and
the
creative
solutions
that
we
had
to
implement
going
forward.
So
after
that
I
don't
think
that's
pretty
much
all
we
have
and
we're
open
for
any
questions
that
you
guys
may
have
for
us.
C
C
C
Good
question
LDAP
synchronization
actually
caused
us
some
headaches
at
the
onset
of
our
suspending.
So
our
initial
approach
was
act,
so
we
we
use
a
a
single
access
group
to
manage
access
to
github
on
the
whole,
so
our
initial
approach
was
to
actually
take
users
out
of
that
group,
and
that
was
our
only
way
of
suspending
because
LDAP
sync,
as
you
know,
by
default
it'll,
say:
oh
this
synced
correctly
and
you're
suspended
I.
C
Don't
know
why
I
let
Sun
suspend
you
and
so
that
kind
of
takes
away
from
what
we
were
trying
to
do
so
they
they
github
Enterprise
has
an
option
that
you
can
check
or
something
you
have
to
do
on
the
console
that
turns
off
that
auto
on
suspend
when
you're
using
LDAP
sync,
because
what
we
we
didn't
want
was
because
LF
sync
runs
every
you
know
ten
minutes
or
something
or
however
often
thirty
minutes.
However,
often
it
is
the
the
the
things
that
we're
trying
to
do
when
we're
suspending.
C
C
B
C
Unfortunately,
we
are
not
there,
yet
we
so
it
drops
the
lists
realistically
speaking
drops
off
at
around
200
or
so
is
really
the
high-intensity
repos,
and
those
are
repos
that
you
know
like
Frank
was
saying
there
that
were
created
by
a
team
to
use
as
a
template
right
and
so
there's
also
some
repos
in
there
that
kind
of
have
some
common
code
for
the
concourse
setup
that
you
that
they
offered
as
a
template.
So
those
are
the
ones
that
are
being
hammered
a
lot
around
200.
C
C
C
B
C
Our
user
base
is
south
of
3000
and
we
are
sized
for
what
they
recommend
for
a
user
base
of
over
10,000,
and
that
was
because
initially
we
didn't
know
why
we
were
falling
over,
so
we
just
threw
Ram
at
it.
Cuz
it
looked
like
it
was
running
out
of
resources.
Hopefully,
with
this
we
may
be
able
to,
we
probably
won't
bring
it
down.
I
mean
we've
already
increased
it,
we'll
probably
just
leave
it
where
it
is,
but
it
should
run
better
once
we
get
people
using
it
responsibly.
C
So
we
at
Home
Depot
have
a
couple
of
data
centers
in
the
u.s.
one
in
Atlanta,
one
in
Austin,
where
people
mostly
work.
We
have
a
single
instance
with
replicas
that
single
instance
serves
everybody.
We're
looking
into
some
of
the
some
of
the
new
stuff
with
211
sounds
interesting,
so
we're
seeing
if
that
makes
sense
for
us
but
yeah
as
of
now.
That's
how
we're
that's,
how
we're
doing
it.
So
our
single
instance
supports
our
entire
company.
C
So
luckily
we
haven't
gotten
to
that
point
yet
where
it's
really
detrimental
other
than
the
initial
fall
over,
but
once
it
fell
over
and
we
added
more
to
it
and
then
figured
out
our
VMware
stuff,
it's
it's
been
running
pretty
stable,
II.
What
we're
worried
about
now
is
we
were
seeing
a
trend
of
increase
right,
because
at
this
point
you
know
the
movement
is
still
picking
up
speed.
C
People
are
still
adopting
this
Concours
pipeline
and
some
people-
don't
really
you
know
they
just
grabbed.
Oh,
this
code
looks
good.
This
looks
and
then
and
then
maybe
that
causes
problems
for
everybody.
So
we
haven't
gotten
to
the
point
where
we
need
to
really
target
disabling
things,
but
we
could
probably,
if
it
comes
to
that
I
think
what
we
would
probably
do
is
suspend
users
so
that
they
couldn't
hit
the
instance
anymore.
The
only
thing
you
really
have
to
worry
about
is
the
great
amount
of
logging
that
github
does
so
even
as
you're
getting
requests.
C
B
B
Our
cluster
is
cold,
cold,
hot,
warm
and
I.
Think
on
our
roadmap.
We
didn't
put
that
on
the
slide
about
romance
we're
trying
to
move
to
a
hot,
hot,
warm
cluster,
and
we
want
to
automate
the
fellow
a
process
between
the
two.
So
hopefully
it's
not
like
we're
gonna
begin
phone
calls
waking
us
up
in
the
middle
of
night
to
do
that.
We
want
that
process
to
happen
automatically.