►
From YouTube: Commit Virtual 2020: Scaling DevOps at the NSA
Description
Like many large public sector organizations, the National Security Agency (NSA) had a volunteer-led source code repository running on old hardware. Through a team of passionate developers, the NSA developed a public-private partnership with GitLab and Amazon Web Services (AWS). Through this partnership, NSA migrated from on-premises hardware to AWS. This effort involved refactoring a non-version controlled environment to a version controlled, fully automated deployment strategy using Infrastructure-as-Code. The migration leveraged AWS managed services to drastically improve the reliability of the service, in order to provide the performance and stability the NSA’s mission requires. Join this talk to learn about NSA’s DevOps journey and the technology they used along the way.
Speaker:
Eric Mosher - Technical Lead, NSA
Get in touch with Sales: http://bit.ly/2IygR7z
A
B
Hello
and
welcome
to
scaling
devops
at
the
nsa,
I'm
eric
moser
and
I'm
here
from
the
national
security
agency,
and
this
is
the
story
of
how
we
upgraded
our
git
lab
instance.
Four
major
versions:
in
one
day
we
upgraded
from
version
seven
to
eleven,
I'm
eric
moser
and
I
am
part
of
a
group
called
devx
at
nsa,
and
that
is
improving
the
developer
experience.
B
I
lead
a
team
called
the
devops
pipeline
and
that's
the
team
that
runs
get
lab
within
nsa
I've
been
at
nsa
for
eight
years.
I've
had
a
variety
of
technical
roles
and
I
started
my
career
with
front-end
work,
doing
data
visualizations
and
that
sort
of
led
me
to
being
interested
in
the
server
side
and
issues
of
size
and
scale,
and
this
ultimately
led
me
to
cloud
technologies,
cicd,
devops
and
things
of
that
nature.
B
Okay,
so
I'm
here
from
nsa,
let's
get
it
out
of
the
way
up
front.
Some
things
are
classified
some
things
I
can't
talk
about
in
general.
Numbers
are
not
something
I
can
talk
about,
but
what
I
can
say
is
as
far
as
number
of
users,
I
can't
say
the
number
of
users
we
have
in
our
gitlab
instance,
but
I
can
say
that
issues
of
size,
scale
and
number
of
users
are
concerned
for
architecture.
B
B
I
can't
say
exactly
when
we
upgraded
from
version
7
to
11,
but
I
can
say
that
we
did
it
in
one
day.
It
was
a
long
day
and
we
didn't
and
we
learned
a
lot
in
that
process.
I
can't
say
number
of
servers,
instances
or
compute
that
we
use.
However,
I
can
say
that
we've
we've
deployed
a
highly
available
elastically
scalable
architecture
in
aws
and
we'll
talk
a
little
bit
more
about
that
later.
B
B
The
second
is
cyber
security,
that's
all
about
protecting
networks
and
sensitive
information,
and
behind
these
two
missions
are
many
developers,
system,
administrators
network
administrators
and
other
technical
folks
that
have
to
run
the
capabilities
that
power.
These
two
missions
we're
part
of
the
intelligence
community
and
the
department
of
defense,
and
we
are
the
largest
enterprise
in
the
ic
and
because
we're
such
a
highly
technical
agency,
we
do
a
lot
with
open
source.
B
Before
I
talk
about
the
actual
upgrade
and
and
the
things
we
learned
there,
I
think
it's
helpful
for.
I
think
it's
helpful
to
understand
the
constraints
that
we're
under
the
first
is
that
we
are
in
an
air
grabbed.
That
means
that
we
don't
have
internet,
there's
no
internet
access,
so
many
capabilities
might
assume
that
it
can
reach
out
to
an
internet
url
that
doesn't
work
for
us.
The
second
is
our
ato
process
authority
to
operate.
B
This
is
our
security
process,
and
this
leverages
additional
requirements
that
we
have
on
us,
and
you
can
think
of
this.
This
process
is
more
rigorous
than
hipaa
compliance,
so
gitlab
at
nsa.
How
did
it
get
started?
It
started
with
a
source
code
install
in
about
2013,
and
it
was
a
modified
version
of
that
source
code.
It
was
started
by
volunteers
because
there
was
no
solution
for
git
version
control
at
the
time
and
it
came
out
of
a
team
whose
job
was
something
else.
B
The
service
grew,
the
number
of
users
grew,
but
the
team
did
not.
The
team
didn't
was
not
able
to
scale
fast
enough
to
meet
the
demand,
and
so
the
instance
the
instance
suffered
as
a
result
in
the
service
suffered,
as
a
result
fast
forwarding
a
little
bit
to
mid
2018
that
this
is
what
gitlab
and
nsa
looked
like
when
I
first
got
involved
with
it.
It
was
a
single
server
with
aging
on
aging
hardware,
and
it
was
because
it
was
that
modified
source
code.
B
It
was
a
long
process
to
update
to
the
latest
version,
so
this
meant
because
it
was
that
modified
source
code.
We
had
to
kind
of
own
the
entire
code
base.
Even
though
we
were
only
changing
certain
parts,
it
was
version
seven
and
because
it
was
version
seven
there
was
no
get
lab
ci
cd
and
because
there
wasn't
this
support
for
the
instance,
there
was
a
lack
of
trust
in
the
instance
and
the
service
that
it
provided
again
in
mid
2018.
B
Something
else
started
to
change
in
nsa,
and
that
was
something
called
devex,
which
was
improving
the
developer
experience
and
because
of
devx
things
started
to
change
a
little
bit
with
our
get
lab
instance.
The
first
part
was
that
devx
took
on
support
of
the
gitlab
instance,
along
with
some
other
developer
tools
that
were
out
there.
Devex
began
a
lot
of
cultural
shifts
that
continue.
B
Today
we
do
things
like
postmortems,
we
read
and
talk
to
people
about
devops
and
the
principles
there,
and
we
do
a
lot
of
coaching
on
ci
cd
practices
as
well,
and
these
continue
today-
and
I
think
the
gitlab
instance
has
enabled
some
of
these
to
to
catch
on,
particularly
in
the
cicd
realm.
We
took
an
inventory
of
the
number
of
gitlab
instances
at
nsa
and
the
reason
we
did
this
is
we
started
to
because
there
wasn't
that
support.
B
In
version
seven,
we
compiled
a
list
of
changes
that
we
made
to
the
gitlab
source
code,
and
this
was
particularly
helpful
as
we
went
into
these
discussions
with
the
gitlab
public
sector
team,
because
this
enabled
us
to
more
smartly
determine
what
was
important
and
what
was
not
important.
As
far
as
those
changes
and
the
last
thing
is,
we
developed,
we
assembled
a
passionate
team
to
run
this
instance.
B
This
kind
of
began
our
public-private
partnership.
We
approached
the
get
lab
public
sector
folks,
with
a
few
goals
in
mind.
We
had
these
were
kind
of
rough
goals.
We
didn't
know
all
these
things
going
in,
but
over
time
in
these
discussions
this
began
to
develop.
We
knew
we
wanted
to
upgrade.
We
were
on
version,
seven
version,
11
was
the
latest
at
the
time,
so
we
knew
we
wanted
to
upgrade.
B
We
knew
we
wanted
to
get
off
of
that
single
server
architecture,
but
we
weren't
quite
sure
how
and
over
time
we
found
that
aws
was
a
possibility
for
us,
and
that
was
part
of
the
reason
we
went
with
this
high
availability,
elastic
scaling
architecture.
We
wanted
to
have
a
more
sustainable
upgrade
path
in
the
future.
We
know
that
get
lab
commercially
releases
the
22nd
of
every
month,
so
we
wanted
to
be
within
a
few
weeks
of
that
commercial
release
and
we
knew
we
wanted
to
automate
the
server
as
it
stood
was
not
version
controlled.
B
B
We
have
a
project
classification,
which
is
the
ability
to
label
a
project
based
on
the
sensitivity
of
that
code,
and
we
have
external
authorization.
We
require
both
authentication
and
authorization,
and
the
authorization
is
done
with
a
call
to
an
external
service,
and
this
external
authorization
is
is
done
with
that
call
to
an
external
service
and
checks.
What
the
person
has
access
to
the
path
toward
version
11,
because
we
had
those
upstream
changes
made
to
the
source
code.
B
We
were
able
to
stop
modifying
it,
and
this
has
been
one
of
our
core
things
that
we've
we've
kept
up
with
is
not
modifying
the
source
code.
We've
moved
fully
to
omnibus
and
that's
enabled
us
to
keep
up
with
these
versions,
because
this
was
also
a
cloud
migration
baked
into
the
upgrade
as
well.
We
debated
doing
a
lift
and
shift
to
the
cloud.
B
A
lift
and
shift
would
be
if
we
took
what
was
on
premises
and
moved
it
to
the
cloud
without
any
real
changes,
and
I
think
in
some
of
these
early
conversations
we
leveraged
that
aws
solutions
architect
knowledge
to
really
ensure
that
we
were
taking
advantage
of
of
the
good
things
that
are
available
in
the
cloud,
and
so
we
decided
against
that
lift
and
shift.
We
decided
it
was
risky
enough
to
do
this
upgrade
and
we
could
only
improve
the
service
by
by
doing
this,
we
scripted
the
entire
process
of
version
7
to
11..
B
We
used
rsync
and
pg
dump
to
sync
our
sync,
the
data
to
the
cloud,
so
the
get
data
and
the
database
we
needed
to
get
that
stuff
over
to
the
cloud
and
that
enabled
us
the
day
of
the
upgrade
to
only
sync
a
couple
hours
of
data
rather
than
the
entire
data
set.
We
went
big
on
those
cloud
resources
to
start
with,
because
we
didn't
have
a
great
idea
of
what
our
size
and
scale
would
be,
and
we
knew
we
we
suspected
anyway,
that
things
would
catch
on
and
and
more
usage
would
happen.
B
I
think
this
did
did
something
to
to
build
that
trust
in
this
team,
because
we
were
communicating
so
much
that
we
were
able
to
answer
many
of
the
questions
that
our
customers
had
and
all
of
this
stuff
enabled
us
to
practice,
and
that
was
one
of
the
things
that
enabled
us
to
not
roll
back
the
day
of
we
practiced
everything
from
the
infrastructure.
As
code,
we
stood
everything
up
and
tore
it
all
down.
We
practiced
the
process
of
version
seven
to
eleven.
B
We
practiced
that
with
data
without
data
with
practice,
data
with
the
real
big
data
set
all
all
of
that
stuff.
We
practiced
that
and
the
day
of
the
upgrade
we
had
a
corrupted
pg
dump
and
we
knew
we
knew.
We
had
seen
that
and
that
enabled
us
to
go
fix
it
and
to
move
forward
rather
than
having
to
roll
back
for
the
upgrade
itself.
We
started
with
known
technologies,
and
what
I
mean
by
known
technologies
is
that
we
had
some
level
of
expertise
that
we
could
leverage
there.
B
B
The
second
technology
was
salt
stack
and
going
back
to
the
idea
of
known
technologies.
We
chose
this
technology
because
I
personally
had
about
two
years
of
experience
with
it
and
that
enabled
us
to
move
very
very
fast.
I
was
able
to
answer
questions.
I
was
able
to
do
a
lot
of
it
myself
and
that
that
helped
us
move
quickly
because
moving
quickly
with
this
upgrade
was
one
of
our
big
concerns.
B
The
last
piece
was
we
use
gitlab
ci
cd,
to
tie
everything
togeth
together
and
to
automate
that
automation-
and
this
mirrors
how
gitlab.com
is
run
gitlab
uses
gitlab
to
build,
get
lab.
It's
a
chicken
and
egg
problem,
but
this
this
enabled
us
to
do
things
like
use,
merge,
request
to
have
a
code
review
and
then
that
would
trigger
a
pipeline
which
would
create
infrastructure
post
upgrade.
B
We
continued
with
those
frequent
blog
posts,
particularly
in
that
first
one
to
two
months
where,
where
we
had
at
least
some
stability
issues,
and
those
were
most
of
those
were
resolved
really
quickly.
But
this
communication
went
a
long
way
to
answering
the
questions
and
saying:
hey
we're
on
this.
Here's
what's
going
on
here's
what
we're
doing
about
it
and
all
that
we
learned
a
tremendous
amount
and
we
performed
a
post
mortem
and
shared
that
internally.
B
And
while
I
can't
share
a
ton
of
details
from
that,
what
I
can
say
is
that
it
really
came
down
to
testing.
We
didn't
know
the
right
kind
of
testing
to
do
and
we
needed
some
smarter
automation,
and
so,
after
the
upgrade,
we
made
sure
to
build
in
more
of
this
automated
testing,
because
we
had
a
better
idea
of
what
we
needed.
And
then
we
also
added
that
smarter
automation,
which
is
instead
of
focusing
just
on
the
creation
of
the
infrastructure.
B
We
responded
to
user
feedback
and
after
we
upgraded
one
of
the
first
things
we
heard
is
when's
pages
coming,
so
we
got
on
that
pretty
quickly
and
within
a
couple
months
we
were
able
to
deploy
it,
and
the
last
thing
here
is
that
we
moved
from
nfs
to
italy,
and
this
did
a
tremendous
amount
for
error
rates
on
pushes
and
polls.
The
original
upgrade
included
nfs.
B
That
was
part
of
the
recommendation
from
gitlab
public
sector
and
then
moving
to
getaly
enabled
us
to
to
reduce
those
error
rates
and
to
kind
of
mirror
what
was
going
on
with
gitlab.com
so
post
upgrade.
Since
we
went
in
with
these
known
technologies
to
start
with,
we
knew
we
wanted
to
come
back
and
revisit
that
after
we
were
upgraded
and
stable.
B
The
first
thing
we
did
is
we
moved
from
cloud
formation
to
terraform
and
one
of
the
big
reasons
we
did.
This
is
because
cloud
formation
restricts
you
to
aws
only
and
we
don't
have
any
immediate
need
to
move
from
aws,
but
moving
to
terraform
has
already
enabled
us
to
talk
to
a
couple
other
things
that
we
we
might
need
that
are
on
the
side
that
are
that
are
separate
from
that
aws
infrastructure,
and
so
this
was
this
was
our
reasoning
for
moving
the
terraform.
B
Yes,
salt
can
be
run
without
a
master,
but
we
felt
that
ansible
kind
of
simplified
that
workflow
for
us,
and
so
that
that
was
the
the
next
technology
we
changed
and
get
lab
ci
cd.
We
kept
that
as
tying
everything
together
and
using
gitlab
to
build
gitlab
and
that
that
leads
in
well
to
the
next
slide.
So
this
is
our
architecture.
B
What
I,
what
I
can
show
of
our
architecture
and
starting
at
the
bottom,
there
you'll
see
something
called
what
we
call
get
lab
ops
and
our
deploy
runner,
which
is
a
shell
runner
and
that's
the
get
lab
that
we
use
to
build
and
configure
gitlab.
And
so
the
rest
of
the
architecture
is
what
what
actually
runs
the
service
that
people
use
and
all
of
that
is
driven
through
our
terraform
and
ansible
code,
along
with
the
cicd
pipelines
that
talk
to
aws.
B
So
the
separation
there
is
terraform
talks
to
aws
and
then
ansible
configures
those
machines
after
they're
created,
and
the
second
thing
that's
important
to
point
out
here.
If
you
look
up
in
the
middle,
are
at
our
auto
scaling
layer
and
then
in
front
of
pages,
there's
two
things
in
red:
there
you'll
see
gem,
which
is
an
omni
auth
gem
that
we
we
pair
along
with
get
lab
omnibus
and
then
nginx
and
node.js
app
in
front
of
pages.
B
That's
proxying
pages
for
us
to
do
that.
Authorization
that
I
discussed
earlier
and
these
two
things
in
red
are
the
only
custom
things
we
have
and
that's
why
we're
able
to
move
quickly
here
is
that
we've
cut
down
on
any
custom
work
that
we're
doing
and
those
those
custom
things
are
very,
very
stable
and
they
integrate
well
with
with
gitlab
so
outcomes
here.
B
This
was
a
great
public
private
partnership
between
nsa
gitlab
and
aws,
and
I
can't
stress
enough
that
involving
these
other
organizations
was
one
of
the
key
things
to
our
success.
Here
we
learned
a
lot
and
we
leveraged
the
expertise
of
gitlab
and
aws
to
make
this
happen,
and
that
was
one
of
the
one
of
the
cool
things
to
see,
because
I
think
it's
that's
fairly
rare
in
government
spaces
that
we
move
this
quickly
and
and
do
things
as
well.