►
From YouTube: Managing GitHub at Scale - GitHub Universe 2016
Description
Todd O'Connor, SCM Architect & Engineer at Adobe, describes how managing a rapidly growing, large scale, GitHub instance provides many challenges to balance end user productivity with enterprise security requirements and server administrative needs.
Adobe's GitHub Enterprise server is a single instance serving over 7,700 users. Todd will touch upon the needs of Adobe's site admins (concerned with server health, bad behavior detection), managers (concerned with metrics and reports), the security team (concerned with log analysis and access), and end users (integrations, user management).
About GitHub Universe:
GitHub Universe is a two-day conference dedicated to the creativity and curiosity of the largest software community in the world. Sessions cover topics from team culture to open source software across industries and technologies.
For more information on GitHub Universe, check the website:
https://githubuniverse.com
A
My
name
is
Todd
O'connor,
I'm
a
senior
sem
engineer
with
Adobe
and
today
I'd
like
to
talk
to
you
a
little
bit
about
what
we
do
and
what
we've
learned
in
managing
github
at
scale.
To
be
clear.
This
is
github
enterprise,
so
I'm
pretty
sure
everyone
is
going
to
be
from
it
with
that,
rather
than
github
com.
A
So
a
couple
slides
briefly
on
what
Adobe's,
up
to
these
days,
I've
been
with
a
company
over
10
years
9
when
I
first
started,
it
was
all
about
shrink-wrap
apps
in
the
Bay
Area
you
go
down
to
frys
you'd,
have
your
big
shelf
full
of
nice,
shiny,
adobe,
apps
and
boxes
about
that
big
as
well?
Well,
now
it's
all
about
digital.
It's
about
streaming,
the
big
games
to
your
devices.
It's
about
online
ad
campaigns,
it's
about
analytics!
A
Here's
a
little
information
on
what
the
company
looks
like
these
days.
The
company
was
founded
in
nineteen
eighty-two.
One
of
the
points
I'd
like
to
draw
your
attention
to
to
keep
in
mind
for
later
on
in
the
slides
will
have
a
tie-in
is
in
the
upper
left.
The
number
of
employees
about
15,000
employees
worldwide,
41
different
countries,
we'll
see
how
that
ties
in
a
little
bit
later
on.
A
Adobe
acquisitions,
through
the
years
as
many
companies
doing,
that
they
grow
through
acquisition.
Three
that
I'd
like
to
point
out
for
us
that
were
major
events,
was
2000
to
excel
yo.
That
was
our
document
services.
The
document
cloud
in
2005
macromedia,
you
probably
all
have
heard
of
this
little
application
called
flash,
so
that
was
macromedia
2009
omniture.
A
So
github
through
the
years,
what
I'd
like
to
talk
about
today
is
some
of
the
things
that
we
do
to
help
out.
Three
segments
of
our
user
base
at
Adobe.
I
want
to
show
what
we
do
to
help
out
the
users
I
want
to
show
what
we
do
to
help
out
the
managers,
and
I
want
to
show
what
we
do
to
help
myself.
The
site
admin.
A
This
is
one
of
my
favorite
quotes
and
one
of
the
ones
I
use
with
my
boy
scouts
that
kind
of
scratch
their
head
like
mr.
O'connor.
What
does
that
mean
experiences?
That
is
what
you
get
when
you
don't
get
what
you
wanted,
maybe
in
github
terminology.
The
angry
unicorn
is
what
you
get
when
you
don't
get.
What
you
wanted.
The
increase
unicorn
is
not
my
friend.
We
have
a
lot
of
experience.
A
We
got
a
lot
of
experience
in
the
early
days
with
with
github
Enterprise
Server
sizing
processes
being
killed,
and
we
don't
know
why
disks
grow.
You
know
filling
up.
Why
did
the
disk
fill
up?
We
got
a
lot
of
experience
so
back
in
the
early
days.
What
did
our
server
look
like?
Well,
we
had
a
we
started.
A
You
know
we
had
frequent
events
adding
more
RAM,
adding
more
CPU,
adding
more
disk
space.
Those
are
all
frequent
events
for
us.
So
what
does
github
look
like
today
today,
github
just
works.
You
know,
I
talk
a
lot
of
folks.
Yesterday,
I
talked
a
lot
of
folks
at
the
conference
today
and
I'm
asking
them.
What
do
you
guys
do
for
your
github
Enterprise
monitoring?
What
do
you
do
for
this?
Or
what
do
you
do
for
that
and
they're,
like
you
know,
do
anything
we
just
turned
it
on
and
github
just
works.
A
So
I've
heard
that
those
three
words
a
lot
over
the
last
couple
days.
Github
just
works,
but
I
know
you
guys
are
technical
crowd.
You
want
specifics
so
here
today
by
the
numbers
is
what
our
server
looks
like
four
and
a
half
years
later,
widely
adopted
across
the
entire
company.
We
have
43,000
plus
repos
over
1,100
orgs,
the
server
specs.
You
can
see
they're
single
VM,
still
16
CPU
160
gig
ram
160
gig
brute
Drive.
A
The
root
drive
is
an
important
one
to
make
really
big,
because
really
of
the
capacity
that
you
have
allocated,
you
only
have
half
of
that
available
to
the
vm.
The
other
half
has
kept
virtual
for
when
you
do
an
upgrade.
The
the
upgrade
process
wants
to
use
that
virtual
disk
for
its
stuff.
So
if
you
only
allocate
a
40
gig
drive,
you
only
have
20
gig
usable
experience,
6
terabytes
of
SSD
storage.
We
have
a
replica
server
same
specs.
A
We
have
a
stage
server
same
specs,
highly
highly
recommended
to
have
a
staging
environment
that
that
mirrors
what
your
primary
production
environment
looks
like.
So
you
can
test
your
upgrades,
so
you
can
test
web
hooks
or
whatever
invasive
things
that
you
want
to
try
try
that
on
your
stage,
environment
first,
our
our
landscape
also
includes
a
backup
server,
simple,
Linux,
VM,
lots
of
storage.
You
want
to
be
able
to
have
it
close
to
your
primary,
so
the
backups
are
fast
and
you
want
to
keep
a
lot
of
backups.
A
Some
folks
have
asked
me
earlier
today:
it's
like
well,
how
long
do
your
backups
take,
or
you
know
our
backups
take?
You
know,
amount
of
hours.
What
about
your
backups?
Well,
I!
Do
my
backups
often
and
I
do
them
frequently.
So
that
way,
the
deltas
are
very
small,
so
I'll
do
a
backup,
like
maybe
six
eight
ten
times
a
day.
So
if
we
turn
to
it
really
really
quickly
back
up
so
small
I'll
have
a
lot
available
at
my
fingertips
in
case
I
need
it.
A
So
talking
about
those
three
points:
users,
managers
and
admins-
let's
talk
about
the
self-service
portal.
This
is
something
that
we
put
together
to
help
our
user
base.
I
help
them
do
the
common
routines,
maybe
some
difficult
routines.
But,
more
importantly,
it's
not
a
wrapper
around
github
Enterprise.
We
built
a
self-service
portal
as
a
complement
to
github
Enterprise.
We
built
some
services
and
and
and
have
these
options
available
for
users
for
those
difficult
things
or
things
that
we
want
to
hide
from
them.
A
A
Github
Enterprise
to
the
adobe,
completely
self
service
users,
login
users
create
their
orbs
users,
create
their
teams
and
users
create
their
repos.
The
org
owners
are
responsible
for
their
access
management.
It's
completely
self
service.
Anybody.
The
company
can
use
it
our
generic
users.
Sometimes
you
might
call
them
your
BOTS.
Your
service
accounts,
your
role
based
accounts,
your
build
account,
your
automation
accounts,
your
Derek
users
have
difficulty-
or
at
least
ours,
with
logging
into
the
browser.
Because
of
our
two-factor
authentication.
The
generic
accounts
don't
have
a
pin
plus
OTP.
A
They
just
have
a
regular
password,
so
we
had
to
build
a
solution
for
that.
For
our
end,
users,
that
managed
and
generic
accounts
users
need
to
set
up
integrations
Jenkins
fisheye,
whatever
integrations
they
want
to
set
up.
Some
of
these
integrations
have
secret
Keys
involved.
I,
don't
want
to
just
hand
off
by
secret
keys
and
I
hear
you
know,
please
don't
share
this
or
post
it
on
your
Facebook
account
right.
A
So
we
wanted
to
hide
that
behind
the
scenes
and
give
them
a
one
button,
push
set
it
up
for
them,
make
it
really
simple
and
then
a
self-service
report.
You
know
we
have
all
of
these
users
that
are
setting
up
orgs.
We
have
all
of
the
you
know:
they're,
creating
a
bunch
of
repose
they're,
creating
a
bunch
of
teams.
A
They
don't
know
what
the
breakdown
is
of
access
control
for
that,
so
we
want
to
give
them
the
capability
to
for
this
org
show
me
all
the
teams
show
me
all
the
members
of
the
team's
show
me
the
access
of
those
teams
and
show
me
what
those
teams
have
access
to
repo
wise.
So
it's
a
nice
report
and
those
folks
can
use
that
report.
For
you
know,
maybe
an
audit
internal
audit
of
their
own
or
something
like
that.
So
it
really
makes
it
simple.
We
just
use
a
series
of
API
calls
to
do
that.
A
A
Techniques
so
to
keep
our
licensing
counts,
manageable
and
cost
manageable.
We
auto
suspend
people,
we
have
a
little
cron
job
that
runs
and
a
an
account
and
a
utility
that
goes
out
there
and
and
checks
all
the
accounts
across
github
Enterprise,
and
certainly
we
want
to
suspend
those
folks
that
have
left
the
company
right.
They
don't
need
access
anymore,
suspend
them.
We
also
suspend
folks
that
are
not
active
in
a
certain
time
period.
Enterprise
has
a
built-in
dormant
report,
but
their
default
is
30
days.
A
A
We
leverage
the
custom
sign
in
and
suspended
messages.
This
has
been
a
really
really
big
win
for,
for
us
and
I
want
to.
You
know,
give
a
big
shout
out
to
github
for
implementing
this
feature.
Thank
you
very
much
when
these
accounts
are
suspended
and
they
try
to
log
back
into
enterprise.
They'll
see
our
nice
little
custom
message:
sorry
user.
You've
been
suspended,
go
to
this
address
and
you
can
unsuspend
yourself
our
self-service
portal
before
that
before
we
had
this
custom
sign
in
massive
this
suspend
message,
I
was
answering
probably
several
emails
per
day.
A
Todd
my
account
suspended
yes,
I
know
this
is
what
you
need
to
do
right
now.
This
is
great.
I
can
focus
on
other
things
and
the
tool
can
handle
this
for
the
users,
so
Thank
You
github.
For
this
feature
again,
the
self-service
portal
can
help
with
the
integrations
setting
up
that
Jenkins
with
or
the
juror,
but
the
or
the
or
the
fish
hi,
with
that
special
secret
key
that
they
don't
need
to
know
about.
A
All
right
self-service
portal:
what
does
it
look
like?
I
can't
give
you
a
full-blown
demo,
but
when
you
log
into
the
self
self
service
portal-
and
you
drill
down
into
the
the
github
option
as
a
user
you
are
given
presented
right
now
with
three
options,
you
can
suspend
or
unsuspend
your
account,
and
in
this
case
you
can
enable
fisheye
or
you
can
manage
any
of
the
generic
accounts
that
you
are
an
owner
for
identified
as
an
owner
for
in
our
ldap
system.
So
you
might
be
thanking
you
know,
perish
the
thought,
Todd.
Why?
A
Why
do
you
want
to
suspend
your
account
right?
Aren't
we
all
wanting
to
use
github?
Well,
you
know,
maybe
it's
a
good
use
case
person's
going
to
go
on
vacation
they're,
going
to
go
on
sabbatical
they're
not
be
at
the
company
for
a
while,
so
they
can
suspend
their
account.
They
can
be
proactive
and
suspend
their
account
while
they're
out,
we
mainly
use
it
for
testing
purposes.
So
I'll
suspend
my
account.
Try
some
stuff
make
sure
our
processes
are
still
working
unsuspend,
my
account
so
99%
of
the
actions
for
that
button
or
unsuspend
actions.
A
We
have
very
few
people
that
we
know
that
actually
suspend
all
right.
So
now,
let's
take
a
look
at
a
little
bit
more
detail
here
on
the
fisheye
integration.
So
if
I
were
to
log
in
behind
the
scenes,
we
discover
what
repos
I'm
an
owner,
for,
I
might
be
my
own
personal
repos
or
I
might
be
an
org
owner
so
that
I
have
access
to
all
the
repos
within
that
org.
A
So
we
present
the
user
with
all
of
the
repos
that
they're
an
owner
for,
and
maybe
the
organization
or
maybe
it's
my
personal
account-
that's
the
owner
column,
and
in
this
case
we
have
two
examples.
We
have
the
first
one,
that's
not
enabled
we've
got
a
blue
button,
you
can
add
two
fish
eye
or
the
second
one
is
highlighted,
yellow
that's
already
enabled.
Maybe
they
don't
need
it
anymore?
They
can
click
the
button
and
they
can
remove
that
integration.
Pretty
simple,
really
easy.
A
Managing
generic
accounts,
a
lot
of
our
teams
have
build
automation,
as
many
of
you
probably
do
as
well.
You
don't
want
to
embed
your
credentials
into
the
build
system,
so
you
have
a
generic
count
to
do
that.
Well
now
you
need
to
manage
that
account
and
that
account
needs
to
log
into
github,
but
that
account
has
difficulty
logging
into
github.
So
what
do
you
do?
A
We
have
this
port,
this
section
of
the
portal,
where
again
we
have
some
options:
the
account
owner,
let's
say
myself:
I-
can
create
my
generic
account
within
github
Enterprise
through
a
series
of
API
calls
the
person
right
now,
if
you
log
into
github
Enterprise
for
the
first
time
the
account
is
actually
created
automatically
for
you,
but
for
a
user
that
can't
log
in
the
account
can't
be
created.
So
what
do
you
do?
We
do
it
through
the
API
here
we
allow
them
to
upload
SSH
keys
for
those
users.
A
A
lot
of
our
build
systems,
connect
over
ssh,
need
to
upload
and
manage
ssh
keys.
If
we
ever
do
an
ssh
key
audit,
we
allow
them
to
go
in
here
and
validate
those
SSH
keys
and
again
we
have
the
familiar,
suspend
unsuspend
button,
very
simple
interface.
This
one
here
is
just
creating
the
account
I
type
in
the
account
name.
A
lot
of
heavy
lifting
behind
the
scenes
does
Todd
own
it.
You
know
what
does
the
account
already
exist?
A
This
is
uploading,
SSH
keys,
again
simple
interface,
give
your
SSH
key
a
name
or
give
it
the
username
and
give
it
the
title
that
you
want
to
call
it
and
then
paste
in
your
SSH
key.
The
really
cool
thing
here
is
that
github
enterprise
already
does
the
ssh
key
validation,
so
I
can't
upload
a
key
that
somebody
else
is
using.
So
we
get
that
for
free,
which
is
really
nice.
People
can't
share
their
keys
or
upload
their
keys
to
multiple
accounts.
So
that's
really
nice.
A
Managing
genetic
accounts
again
so
this
is
the
actually
managing
of
your
SSH
keys
type
in
your
generic
user
click
load
or
retrieve
ssh
keys.
The
dialog
down
here
on
the
lower
left
will
or
the
screen
down
here
on
the
lower
left
will
come
up
and
it'll
show
you
all
of
your
ssh
keys
the
little
button
in
the
upper
right.
You
can
switch
from
a
table
view
to
the
more
icon
view
over
there
on
the
right,
so
it
just
gives
the
user
something
you
know
cool
to
be
able
to
format
their
display.
A
But
here
you
see
the
machines,
you
see
the
keys
and
these
are
already
verified.
Had
they
not
been
verified,
there'd
be
a
button
or
a
toggle
there
that
you
could
check.
So,
yes,
I
want
to
continue
to
use
this
key
now
continue
to
use
it.
If
you
don't
need
that
key
anymore,
you
can
go
ahead
and
delete
it
all
through
the
self-service
portal.
A
A
All
right,
metrics,
so
I've
shown
some
implement
some
screens
and
some
things
that
we
do
to
help
out
the
users
at
Adobe.
Now,
let's
talk
about
the
managers
and
the
metrics,
everybody
wants
data.
Why
do
you
need
this?
Justify
that?
So,
let's
take
a
look
at
some
metrics,
some
of
the
things
that
we
report
on
space-
that's
kind
of
fitting
here
for
the
universe,
right:
space,
user
growth,
user
locales.
Where
are
my
users
coming
from?
Were
they
using
github
Enterprise
from
and
then
you
also
have
available
to
today
all
these
site
admin
reports?
A
So
if
you're
an
enterprise
you
can
go
into
the
site,
admin!
Look
at
the
reports.
You
can
get
all
your
users,
you
can
get
your
active
users,
organizations
repositories
download
a
CSV
file.
Then
you
can
slice
and
dice
it
and
put
it
into
your
favorite
graphing
system
or
do
whatever
you
need
to
do
with
it.
So
that's
pretty!
That's
pretty
pretty
nice
to
have.
A
All
right,
let's
talk
about
the
first
one
space,
so
this
is
a
spit.
This
is
a
graph
charting.
Our
space
usage
looks
like
it's
probably
about
a
year
and
a
half
or
so
some
of
the
things
I'd
like
to
point
out.
Well,
we
have
two
thresholds
on
there.
We
have
a
seventy
percent
yellow
threshold.
That's
your
warning
threshold!
If
you
start
getting
up
around
yellow
warning,
you
probably
want
to
start
think
about
adding
more
space.
A
A
The
first
event
is,
we
were
getting
up
around
that
ninety
percent
threshold.
We
need
to
do
something
right
now
we
add
a
terabyte
great
dropped
us
back
down
the
users
continued
using
the
product
space
continues
to
grow.
Why
does
the
space
continue
to
grow?
This
was
pre
I
think
we
round
to
16
at
this
time.
So
this
growth
here
is
attributed
to
Forks.
A
We
did
it.
The
second
event
is,
we
did
an
upgrade.
We
upgraded
to
225
actually
github
in
version
2
to
3,
introduced
space,
optimization
back-end
database,
optimization
and
D
duped
all
the
forks
we
saved
about
a
one
and
a
half
terabytes
of
space,
just
by
getting
rid
of
all
of
our
Forks,
so
big
feature
implementation,
big
win
for
us
again,
another
shout-out.
Thank
you
to
github.
For
that
appreciate,
users
started
using
the
product,
had
a
little
blip
when
over
70
came
back
down,
probably
some
log
rotations,
or
something
like
that.
A
Maybe
some
may
be
some
repos
that
were
in
purgatory
finally
got
purged
space
went
along.
We
had
the
opportunity
we
were
given
some
more
space.
Okay,
we'll
take
it.
So
we
added
more
space,
went
on
for
quite
some
time
very
flat
kind
of
growth,
and
then
we
migrated
to
a
new
server.
We
added
much
more
space
to
give
us
much
more
future
growth,
and
then
you
can
see
now
we're
about
the
forty
five
percent
usage
I'm,
pretty
comfortable
with
where
we
are
today
on
our
space
usage.
A
But
those
are
the
four
major
events
or
the
four
major
events.
I'd
like
to
point
out
here
and
what
we
do
for
space
monitor,
and
you
know
graphing
and
tracking
our
space.
If
you're
not
doing
this
for
your
organization,
it
probably
would
go
a
long
way
when
you
have
to
go
request
more
space.
It's
like
well
Todd.
You
just
came
to
me
a
week
ago
and
wanted
more
space
where's.
All
the
space
going
was
like
well
I!
Guess
what
I
have
a
graph
here
and
it
shows
my
users
are
using
it.
A
Managers
want
to
know
adoption
rate
Todd
you
keep
asking
for
more
space.
How
many
users
do
you
have
well?
This
is
how
many
users
I
have.
I
have
almost
8,000
users
and
we're
averaging
about
160
new
users
per
month.
So
and
this
isn't
only
in
four
and
a
half
years,
so
it's
widely
adopted
lost
the
entire
company
rapid
growth.
A
lot
abuse
manages
then
take
that
and
say:
okay!
Well,
can't
you
graph
that
a
little
bit
different
way
I'd
like
to
see
it
presented
some
what's
slightly
different?
Okay.
A
A
You
know
what
product
are
they
working
on
you
know
all
sorts
of
other
information
is
available
than
just
what
I'm
you
know
showing
you
here,
but
it
really
goes
a
long
way
for
our
managers
to
be
able
to
slice
and
dice
that
dashboards
if
you're,
not
building
dashboards.
These
would
be
great
additions
to
your
dashboards
so
that
people
to
quick
glance
can
say
space
user
growth,
where
they're
coming
from
those
sorts
of
things.
Keep
that
in
mind.
A
Monitoring
alright,
so
we
help
the
users
we've
helped
the
managers.
Now,
let's
help
me
help
myself
with
monitoring
I,
don't
want
to
be
kept
up
all
night
long.
You
know
worrying
about
the
health
of
my
of
my
system,
some
of
the
things
that
we
monitor.
Obviously
in
your
landscape,
you
want
to
monitor
all
the
servers
in
your
landscape.
I,
don't
monitor
my
stage
server,
it's
a
stage
server,
but
certainly
you
want
to
monitor
your
primary
we're
again
only
in
a
single
server
setup
here.
A
A
You
can
use
that
if
you
don't
have
SNMP
set
up
or
you're
not
familiar
with
it.
It's
simply.
If
you
have
a
trusted
machine,
you
just
ssh
to
your
github
enterprise
appliance,
run
a
DF.
Now,
that's
that
would
be
a
simple
way
to
grab
your
space.
If
you
want
to
just
have
an
external
monitor
that
just
says
in
my
service
up
or
down
something
really
simple,
just
run
this
simple
curl
command,
the
host
name
or
the
address
of
your
appliance
/
status
will
just
return
a
number.
A
Are
we
up
or
are
we
down
something
really
simple
log
forwarding?
This
is
a
technique
that
there
was
a
new
feature
that
just
came
out
and
a
later
version
of
enterprise,
which
is
really
great
I,
can
afford
my
logs
to
multiple
locations.
So
if
you
don't,
if
you're,
not
a
site
admin
that
has
privileged
access
to
your
enterprise
appliance,
how
do
you
get
access
to
the
logs?
What
we
want
to
afford
those
logs
somewhere
forward
them
to
splunk
for
them
to
log
stash?
A
That
would
be
one
way
for
a
team
that
maybe
does
some
log
analysis
to
have
access
to
those
logs,
because
now
you
can
do
multiple
destinations.
Maybe
you
have
a
security
team,
or
maybe
you
have
another
team
that
needs
the
logs
for
whatever
reason,
comma
separated
list
add
another
address:
send
the
logs
off
to
their
server
as
well.
A
Let
them
you
know,
slice
and
dice
it
and
look
it
up
and
do
whatever
they
need
to
do,
but
definitely
get
your
logs
off
of
your
enterprise
appliance
and
put
them
into
a
long-term
storage
system,
something
that
other
users
outside
of
your
group
would
have
access
to
look
at
your
replica.
What
do
you
do
to
monitor
your
replica
JT
ripple
status
gives
you
a
simple
output,
you
could
go
verbose
or
you
could
go
very
verbose
either
way.
A
It's
going
to
tell
you
if
your
replica
is
up-to-date,
how
far
behind
it
is
and
those
sorts
of
things
so
we're
monitoring
our
replica.
Sometimes
I
get
an
alarm:
hey
the
replicas
behind
it
eventually
catches
up,
but
at
least
I
know.
What's
going
on
collect
d,
so
enterprise,
if
you're
a
site
admin
you've,
probably
already
gone
into
the
management
console.
You
see
some
of
the
monitoring
graphs
great
set
of
graphs.
You
might
again
have
a
team
that
doesn't
have
privileged
access
to
get
to
that
area.
A
Send
that
information
off
of
your
enterprise
appliance
to
a
collect
d
server,
and
then
you
can
graph
at
your
off.
So
that's
really
really
nice
to
have
and
then
out
and
get
github
com
we've
got
getting
on
the
git
repository
monitor.
You
might
investigate
that.
Maybe
you
could,
you
know,
contribute
take
that
code
tweak
it
around,
maybe
make
it
work
for
you.
A
So
those
are
some
of
the
techniques.
Here's
that
management
console
that
I
mentioned
earlier.
So
here
we
have
enabled
SNMP
check
it
and
you
put
in
your
community
string
production,
environment
stage,
environment,
whatever
open
up
any
ports
that
need
to
be
opened
up,
and
then
you
can
grab
the
information
on
that
log.
Forwarding
very
simple,
very
simple
interface,
very
clean,
very
nice
check
it
enable
it
add
the
address
and
port
number
of
your
server
and
then
the
protocol
we
choose
UDP
I,
think
it's
recommended.
A
If
you're
in
a
environment
where
maybe
network
latency
is
a
little
bit
of
an
issue,
you
could
go
UDP
or
you
go
TCP.
If
there
are
any
network
failures,
I
think
tcp
has
some
issues
might
cause
some
interruptions.
Udp
is
a
sin
and
forget
it
so
I
I
go
UDP,
I,
go
lowest
common
denominator;
I,
don't
want
any
impact,
my
enterprise
appliance,
then
the
bottom,
when
the
air
is
collect,
d,
forwarding
check
that
go
ahead
and
set,
and
you
put
in
your
settings.
A
Alright
polling
so
now
I'm
going
to
take
a
step
and
back
and
we're
going
to
dig
down
into
a
specific
example
of
some
things
that
that
keep
me
up
at
night
or
some
things
that
you
might
have
experience
with
your
enterprise
appliances
like
something
just
happened.
What
just
happened?
Why
did
we
get
that
little
blip
or
what
just
went
bump
before
I
jump
in
and
dive
in
I
want
to
give
a
big
shout
out
to
get
up
support.
A
They
helped
me
a
lot
with
these
slides,
specifically
the
commands,
so
if
it
weren't
for
them
providing
some
of
these
commands
I
working
with
them.
You
know
over
the
years
and
a
lot
of
you
know
analysis
of
my
system,
so
I've
come
to
get
a
lot
of
commands
from
them.
So
I
want
a
big
shout-out
to
github
support
on
that.
Thank
you
very
much.
So
polling
polling
is
one
of
those
things
that
it's
easy
for
a
user
to
set
up
a
machine
and
it
just
starts
pulling
your
system
right.
A
So,
let's
take
a
little
bit
of
in
depth.
Look
what
we
really
wanted
to
do
is
we
want
to
discover
doing
the
polling,
I
want
to
engage
them
and
say:
hey
did
you
know
you're
pulling
and
then
I
want
to
educate
them
on
a
better
best
practice?
Hey,
you
know
what
we've
got
web
hooks.
That's
the
better
thing
to
do
so.
The
problem
polling
in
general
is
just
is
just
not
good.
We
just
don't
like
pulling
the
user
assumption
and
I
actually
contacted
a
user
on
their
polling
activity.
A
Like
hey,
Todd
I
thought
there
were
unlimited
resources
on
the
server.
No,
no,
we
only
have
one
server
and
you're
on
it
with
eight
thousand
other
people.
What
you're
doing
is
impacting
everybody
else.
You
don't
have
unlimited
resources
solution
is
to
use
web
hooks,
so
we
need
to
find
out
what's
going
on,
we
need
to
change
their
behavior
and
if
you
end
up
helping
one
person,
you'll
probably
end
up
helping
it'll
be
a
trickle-down
effect.
You'll
end
up
helping
everybody
on
your
instance,
because
then
everybody's
going
to
feel
the
benefit
of
that.
A
So
some
of
the
things
that
we're
going
to
look
at
we're
going
to
look
at
a
series
of
searches,
we're
going
to
look
in
to
specific
log
files,
I
have
privileged
access
to
my
enterprise
appliance,
so
I'm
going
to
dig
right
down
into
the
log
files.
If
you
have
these
sent
off
to
your
splunk
system,
you
could
probably
look
at
them
there
as
well,
and
we're
also
going
to
use
the
built-in
sessions
graph.
So,
let's
jump
in
here's
this
built-in
sessions
graph.
That
should
probably
be
your
first
stop.
A
Github
has
all
these
great
graphs
that
you
could
take
a
look
at
this
is
one
in
particular.
Now
we
want
to
look
in
the
case
of
polling.
We
want
to
look
at
a
week's
worth
of
data.
We
want
to
look
at
something
that's
a
little
bit
more
meaningful.
So
in
this
case
this
server
looks
pretty
good.
We
have
a
lot
of
activity
during
the
business
week
and
then
the
low
is
the
weekend
little
to
no
activity.
That's
great!
That's
what
we
would
expect.
We
like
to
see
that
this
is
a
good
server.
A
I'd
take
this
server!
This
server!
On
the
other
hand,
has
consistently
high
traffic
the
entire
week
or
requests.
I'm.
Sorry
has
consistently
high
requests
all
week,
long,
that's
probably
even
on
the
weekends.
So
this
is
a
pretty
good
indication
that
you
have
somebody
out
there
pulling
or
maybe
many
systems
pulling
so
first
thing
when
you
get
home
check
out
your
enterprise,
your
set
your
graphs.
You
want
your
server
to
look
like
this
and
not
like
that,
and
if
it
looks
like
that,
let's
dig
down
in
and
find
out
who's
the
culprit.
A
So
the
first
thing
we
want
to
look
at
is
polling
searches
in
10-minute
increments.
So
here
we've
done
a
search
on
the
Babel.
We
have
consistently
high
numbers,
8,000
9,000,
okay,
so
we
probably
have
some
polling
going
on,
but
all
of
a
sudden
at
10,000
there's
a
spike.
That's
a
pretty
good
indication
that
there's
polling,
probably
a
bill
job
woke
up.
You
know
it's
the
end
of
the
day.
It's
lunchtime
something
just
happened,
started
hitting
the
system.
Is
there
anything
to
build?
A
A
Let's
take
a
look
at
for
the
top
ten
owners
of
repositories.
So
again
we're
going
to
look
in
the
Babel
d
log
and
what
I
find
is
there?
Is
this
user
called
polling
user
96,000
requests
hmmm,
probably
going
to
look
at
him
a
little
bit
more
in
detail?
We
have
some
other
high
numbers
and
then
it
kind
of
drops
off
into
the
you
know
the
teens.
A
Now
for
the
users,
the
top
10
users
doing
get
operations,
hey
look
coincidence,
there's
our
polling
user
at
the
top
of
the
list.
Again
110,000
requests
we're
going
to
have
to
dig
down
to
this
little
bit
more
detail
and
find
out
what's
going
on
all
right,
taking
that
user
record
the
271
and
then
their
user
ID
repping,
for
that
in
the
Babel
d
log
now,
I
can
find
out
there
per
hour
access
rate
very
high
numbers
here,
6,000
at
the
peak
we
had
111
accesses
per
minute,
almost
2
per
second,
that's,
probably
a
good
indication.
A
So
now,
let's
find
the
repositories
that
that
user
is
polling
for
so
here
we've
taken
that
user
information
again.
Looking
the
Babel
d
log
adjusting
some
of
the
search
strings
right
now
we're
looking
for
repo.
We
can
see
that
they're
hitting
this
repo
called
big
repo
9696
thousand
times,
and
that's
their
big
one
that
they're
hitting
because
everything
else
drops
off
after
that
they're
only
hitting
the
other
repost
three
thousand
times.
So
this
repo
is
really
popular
for
this
user.
A
Doing
requests
almost
two
per
second
now
armed
with
all
of
this
information,
I
go
and
I
talked
to
the
end
users,
I
say:
okay,
I
found
out
you're
doing
this
much
polling.
This
is
what
you're
hitting,
and
this
is
how
often
you're
hitting
and
a
lot
of
the
users
have
been
hey.
Todd,
that's
great
information!
Thank
you
because
we
didn't
know
we
had
this
Jenkins
box
that
we
forgot
about
somebody
turned
it
on
and
it
just
went
crazy,
we'll
go
turn
it
off.
A
Knowing
the
repo
that
they're
hitting
has
helped
some
of
my
users
actually
troubleshoot
and
find
out
where
that
traffic
is
coming
from
just
asking
them
to
say:
hey,
can
you
stop
polling?
Well,
where
am
I
pulling
from?
All
of
this
information
helps
you
to
help
them
to
help
the
health
of
your
github
Enterprise
appliance.
A
Now
you
can
also
look
for
the
15
most
active,
SSH
keys.
That's
another
way
to
look
at
it
now
we
happen
to
use
SSH
access
a
lot
so
here
this
particular
fingerprint
is
being
used,
55
almost
56
thousand
times
so
what's
nice
is
you
can
take
that
fingerprint
go
to
your
site,
admin,
your
search,
dialog
you
could
put
that
in
and
then
you
can
find
the
user
that's
associated
with
that
fingerprint.
Then
you
can
go!
Ask
them!
It's
like
hey,
you're,
hittin
this
here
and
getting
hub
56
thousand
times.
What
do
you
do?
A
You
have
a
lot
of
build
machines?
Do
you
have
a
lot
of
builds
on
one
machine?
You
know
start
a
dialogue
and
find
out
what
they're
trying
to
do
maybe
help
change
the
behavior
a
little
bit
all
right.
So
some
other
considerations.
So
we've
looked
at
helping
the
users,
we've
helped
looked
at
helping
our
managers
and
we
looked
at
helping
the
site
admins.
So
what
are
some
of
the
other
considerations
when
you're
setting
up
or
managing
your
github
enterprise
appliance
so
performance
previous
ebooks?
Something
new
that
just
came
out
previously.
Books
are
great
to
help.
A
You
enforce
your
business
policy,
but
because
the
way
that
they
run
they
run
on
every
push.
You
really
need
to
make
sure
that
they're
super
small
super
lightweight
and
run
super
fast,
because
if
not,
you
can
start
to
drag
down
your
system
and
the
the
hook
itself
is
actually
checked
into
a
repo
on
your
enterprise
appliance.
So
it
actually
runs
from
the
server.
A
So
you
could
imagine
that
if
you're
doing
something
heavyweight
trying
to
analyze
a
push,
it's
going
to
drag
down
the
performance,
you
got
to
make
sure
this
thing
is
super
lightweight
and
super
fast,
because
most
of
us
have
not.
All
of
us
are
probably
running
our
appliance
in
a
vm
versus,
like
maybe
the
cloud
if
you're
on
a
vm,
you
know,
are
you
on
a
shared
host
or
on
a
dedicated
host?
A
I
know
most
companies
like
to
just
say:
you
know
we
have
hosts
out
there
and
we're
going
to
john
here
with
who-knows-who
right,
try
to
get
yourself
on
a
dedicated
host
or
a
host
with
as
little
other
customers
or
tenants
as
possible.
Make
sure
you
have
the
most
performance
large
files.
Large
files
is,
is
very
interesting.
Not
only
are
they
large
files,
but
they
can
be
binary
files?
Do
you
allow
that,
in
your
do
you
love
that
people
to
push
those
to
your
enterprise
appliance
right?
Are
you
letting
people
push
big
files?
A
Are
using
lfs
the
built-in
get
lfs
large
files
support,
because
if
you
just
toggle
the
switch
and
say
okay
yeah
use
lfs,
unless
you're
you
pushing
your
lfs
files
to
an
external
source,
those
files
are
still
going
to
go
onto
your
enterprise,
appliance
for
storage,
going
back
to
the
point
of
space
monitoring,
so
people,
if
you're,
going
to
allow
people
to
check
in
mega
Giga,
Byte
worth
of
files
and
they're
being
stored
on
your
appliance,
you're
gonna
have
to
monitor
your
space
very
very
closely.
Now.
A
What
do
you
do
about
people
that
want
to
push
big
files
in
the
admin
Center?
There's
the
limits
github
likes
to
recommend
100
megabytes
profile
and
a
gigabyte
per
repo
right.
Now
we
don't
have
any
controls
on
repo
size.
People
can
push
push,
create
branches,
create
revisions
of
vials.
We
can
check
the
sizes
of
those
repos
and
we
can
send
them
a
nice
reminder.
Email,
hey,
you're
exceeded
a
gigabyte,
your
exceeded
2
gigabytes.
A
You
might
want
to
think
about
slimming
down
the
one
thing
that
we
can
control
is
the
file
size,
so
the
default
is
100
megabytes
per
file.
If
you're
a
little
bit
more
generous,
you
can
give
your
your
users
a
little
bit
more
space.
I
would
definitely
recommend
not
going
unlimited,
because
you
never
know
what
you're
going
to
get,
especially
if
somebody
pushes
a
really
big
file,
it's
in
a
repo,
it's
pretty
much
going
to
be
there
forever
people
start
changing
that
file
or
adding
new
versions
of
that
file.
A
Some
things
you
want
to
think
about
when
you're
setting
up
your
enterprise
appliance
is
your
recovery
point
objective
and
your
recovery
time
objective.
So
there
are
two
things
to
help
you
achieve
those
there's
high
availability
and
then
there's
also
disaster
recovery,
github
definition
of
a
che.
The
replica
is,
in
the
same
data
center
very
low
to
no
latency,
probably
in
the
rack
right
next
to
your
primary
instance
right
and
it's
used
when
the
primary
instance
fails,
for
whatever
reason,
a
lot
of
various
reasons
for
that.
A
Unfortunately,
there's
only
one
to
one
mapping,
one
primary
one
replica
dr:
that's
something
you
want
to
think
about
long
term.
You
want
to
maybe
have
at
an
external
site
somewhere
a
distant
data
center,
some
different
geo,
maybe
a
different
office
in
a
different
state,
maybe
outside
the
state
of
california,
somewhere
else.
The
issue
with
this
is
the
time
it
takes
to
build
it
because
it's
built
off
of
a
restore
mechanism,
so
you
have
to
first
take
a
backup
of
your
primary
if
that
backup
takes
6
10
12
hours.
A
Now
you
have
to
take
that
back
up
and
you
have
to
restore
it
over
the
wind,
because
you
don't
want
your
dr.
In
the
same
data
center,
you
wanted
at
a
distant
site
now
you're
going
over
the
land
with.
However
much
data
has
been,
you
know,
backed
up
in
6
10
12
hours
by
the
time
that
restores
done
your
maybe
24
hours
down
the
road
and
now
you're
that
farther
behind.
A
So
your
recovery
point
is
going
to
be
further
behind
with
the
dr
than
it
is
with
the
h,
a
replica,
but
your
recovery
time
in
both
scenarios
is
about
the
same.
I
use
the
replica
a
lot,
especially
if
I'm
setting
up
a
brand
new
server
I'll
set
up
a
replica
and
that
thing
keeps
up-to-date
and
going
back
to
one
of
my
previous
slides.
It
just
works
for
me.
It
works
really
well.
A
I
can
switch
over
to
a
replica
and
probably
you
know,
10
to
15
minutes
if
I'm
taking
my
time,
you
know
you
stop
your
primary
put
in
the
maintenance
mode,
make
your
dns
change,
spin
it
up
and
you're
good
to
go.
So
if
you're,
not
using
replica
today,
you
might
want
to
think
about
it
really
great
for
h
a
situation
in
case.
Your
primary
Abra
has
a
problem.
A
Enterprise
Support
I'll
just
say
that
you
know
I've
covered
a
lot
of
stuff
today,
but
if
you
ever
have
any
doubts
about
the
health
of
your
system,
you
know
what
was
that
thing
that
just
went
bump
in
the
night?
Why
did
we
just
have
this
spike
contact?
The
experts
at
github
for
help
I've
had
nothing
but
excellent
support
from
the
folks
at
github
support.
Very,
very
talented
group
of
folk
they've
been
top-notch
and
again
I
want
to
thank
github
for
that
and
I've
been
in
the
industry.
A
Quite
a
while
and
I've
dealt
with
a
lot
of
products.
Github
support
is
right
up
there
at
the
top.
So
I
want
to
thank
those
guys.
What
I
like
to
do
is
perform
a
periodic
health
check.
Just
like
going
to
see
your
doctor
annually.
If
maybe
enterprise,
isn't
your
primary
job
duty
right,
get
a
support,
bundle,
send
it
into
github
and
let
the
experts
analyze
it.
They
can
look
to
see
if
somebody's
pulling.
They
can
look
to
see.
A
If
you
have
my
sequel
issues,
they
can
look
to
see
if
you
need
to
add
more
RAM,
more
CPU
to
your
systems.
They
are
that
they're
the
knowledge
experts
on
this
so
when
in
doubt,
if
you
have
any
questions
by
all
means
contact
the
folks
at
support
and
with
that
we've
got
about
two
minutes-
left
Q&A,
oh
well,
I
want
to
thank
you
any
questions.
A
I'm
sorry,
what
was
the
quote
sites?
You
know
I,
don't
I
don't
so
the
question
was:
how
do
I
deal
with
remote
sites?
Do
I
have
a
cache
server,
and
so,
as
I
mentioned,
you
know,
we've
got
about
half
of
over
half
of
the
company
using
github
Enterprise.
You
know
my
earlier
slide
was
15,000
employees.
I've
got
about
8,000,
github
enterprise
users,
I,
don't
have
any
complaints
either.
We
really
have
a
good
network
connections
or
the
way
that
github
is
architected
with
how
they
clone
and
having
all
of
your
repos
local.
A
When
you're
working
on
them
and
you're
only
pushing
the
things
that
need
to
change.
You
know:
I've
have
other
products
in
the
house
that
you
know
we
have
to
have
a
lot
of
fine
tuning
and
hand-holding
to
make
it
work
really.
Well,
this
thing
just
works
for
us.
You
know
the
folks
in
Europe
that
use
it
the
folks
in
in
India
that
use
it
the
folks
around
the
world
that
use
it.
It
just
works
really
well
for
them.
So
I
don't
have
any
local
caches
or
anything
like
that.
A
A
The
question
is:
did
we
do
any
customizations
to
our
appliance
and
the
answer
is
no,
it's
a
I,
consider
it
and
the
way
that
I
advertise
it
to
my
internal
folks.
Is
it's
a
black
box
use
it,
as
is
we
don't?
You
know,
I
get
on
there
to
look
through
the
logs,
but
I'm,
not
tweaking
any
sis
controls
I'm,
not
tweaking
this
I'm,
not
tweaking
that
we
use
it,
as
is
as
provided
by
github
one.
You
know,
we'd
want
to
void
the
warranty
right,
so
I
don't
want
to
break
anything.
A
You
know
it's
like
Oh
Todd,
you
no
warranties
off,
because
you
touch
this
file
over
here.
No,
we
just
use
it
as
is,
and
it
works
really
well.
What
we
have
had
to
do,
though,
over
the
years
is
add
more
space,
add
more
RAM
at
more
CPU.
You
know
to
make
it
work
really
well,
but
once
we
get
those
numbers
like
I
said
earlier,
we
have
our
new
server
spec
that
new
server
specs
been
working
really
well
for
us,
and
a
lot
of
those
problems
have
been
alleviated.
A
A
So
as
soon
as
the
release
comes
out,
I'll
put
it
on
my
stage
environment,
then
we'll
let
that
bake
for
maybe
two
three,
maybe
four
weeks
because
I
know
a
very
brand
new
release,
like
27,
usually
has
a
little
bit
of
iteration
a
dot
1
a
2
and
a
dot
30
comes
out,
but
then
it
kind
of
settles
down.
At
that
point,
it's
been
about
a
month,
maybe
two
months,
we
were
confident
that
a
stable
we've
been
running
it
in
our
stage
environment.
Then
I'll
schedule
an
upgrade
and
go
from
there
question.
A
What
kind
of
question
is?
What
kind
of
down
time
do
we
have
during
our
upgrade?
That's
a
good
question
because
a
lot
of
folks
have
been
asking,
though
we
have
an
environment
where
we
we
have
zero
downtime.
We
can't
do
upgrades
I
have
the
luxury
of
being
able
to
coordinate
with
all
my
end
users
around
the
world
inside.
You
know
what
I
need
down
time
on
saturday
at
5am
and
when
I
do
my
upgrade
the
upgrades
right
now
we're
really
fast.
A
A
A
So
the
question
is
ssh
key
management.
Do
you
have
any
requirements
that
your
ssh
keys
be
a
certain
length?
No,
we
haven't
gone
down
to
that
detail.
Yet
people,
just
you,
know
ssh-keygen
our
SI
habla
go
from
there.
They
add
the
man.
We
do
some
periodic
auditing
where
we'll
burn
all
the
ssh
keys
everybody's
got
to
regenerate
them,
or
at
least
they
have
to
audit
them
and
say
yes,
I'm
still
using
that
ssh
key
right
all
right
last
last
question:
then
we're
done.
A
Have
we
ever
had
the
question
is:
have
we
ever
had
to
exercise
or
disaster
recovery,
in
instance,
or
roll
back
an
upgrade?
The
answer
is
no
I
have
tested
rolling
between
my
stage
environment
and
the
replica,
and
then
flipping
back
and
forth.
I
could
do
that
really
rapidly.
I
have
tested
the
when
I
upgrade
to
a
new
server
going
to
that
new
one
that
we
just
spectate
earlier.
A
That
actually
was
a
replica
and
then,
when
we
did
the
cut
over
and
say
hey
we're
flipping
over
to
the
new
server
I
made
that
one
the
primary
and
it
worked.
So
definitely
you
want
to
test
that
test
it
with
your
stage
environment.
It's
really
fun
to
play
with
and
go
back
and
forth
and
and
make
sure
that
your
process
works.
A
You
definitely
want
to
make
sure
your
process
works
document
it
so
that
not
only
you
know
it,
but
everybody
else
on
your
team
knows
how
to
do
it
and
that
they
know
how
to
do
it.
Well,
because
you'll
never
know
when
that
you
know
you'll
get
that
call
at
3am
on
a
Saturday.
Something
just
happened,
and
those
are
the
things
that
keep
us
up
at
night.
Thank
you
very
much
and
I'll
be
up
at
the
front.
If
you
have
any
questions
and
enjoy
the
conference.