►
From YouTube: Provisioning 100 GitLab Spot Runners on AWS in Less Than 10 Mins via Less Than 10 Clicks For $5/hr
Description
Whether you want 1 runner that is scheduled to run only two days a week, or 150 that service your main GitLab load and are easy to update - you've found the automation you are looking for ;)
Links:
- https://gitlab.com/guided-explorations/aws/gitlab-runner-autoscaling-aws-asg/-/blob/main/easybuttons.md
Playlists:
- GitLab and AWS (https://youtube.com/playlist?list=PL05JrBw4t0Ko30Bkf8bAvR-8E441Fy2G9)
- GitLab Alliances Tech (https://youtube.com/playlist?list=PL05JrBw4t0KqaWCrU4avIY6TQveKTDMso)
- GitLab Reference Architecture, Performance Testing, Scaling and HA (https://youtube.com/playlist?list=PL05JrBw4t0KqgVugqCld2fUvrBulanchq)
- Guided Explorations (https://youtube.com/playlist?list=PL05JrBw4t0KruBVtI-Ynb3mCHQTkh2eBY)
- CI/CD Tips Tricks and Hacks (https://youtube.com/playlist?list=PL05JrBw4t0Kr92rF9z0PEuOtQnH89VD7n)
- GitLab and Windows (https://youtube.com/playlist?list=PL05JrBw4t0KqOJeQ1y5wSBxArdTEtmnxb)
- GitLab and PowerShell (https://youtube.com/playlist?list=PL05JrBw4t0KrklJ2kDTl2qXNaGSlzKeoW)
- DarwinTheorizes (https://youtube.com/playlist?list=PL05JrBw4t0KoH74CcAtP0Ze7JXjX7zT6g)
A
A
So
I
want
to
show
you
today
something
called
the
get
lab
scaling
runner
vending
machine
for
aws.
It
is
enablement
automation
that
should
make
this
a
lot
easier
for
you.
The
actual
automation
itself
is
very
simple:
to
use
for
simple
use
cases,
but
is
quite
elaborate
and
sophisticated
under
the
covers.
So
if
you
need
to
do
more
elaborate
things,
it's
a
great
place
to
start.
Even
if
you
want
to
customize
it,
but
there's
a
lot
of
functionality
provided
out
of
the
box
in
the
parameters
that
are
given.
A
So
let's
take
a
look
at
this
we're
going
to
try
to
deploy
100
runners
in
less
than
10
clicks
and
less
than
10
minutes
on
aws
spot.
So
let's
take
a
look
at
that,
so
here's
the
repository
and
when
we
look
in
the
readme
file
there's
a
lot
of
information
here
about
the
various
different
things
it
can
do,
we're
going
to
actually
go
to
the
root
of
the
repository
to
the
readme,
and
when
we
look
in
here
we
have
something
called
the
easy
buttons.
A
A
A
Okay,
so
changed
regions,
so
for
our
first
click
we
had
the
loading
the
template.
Second,
click
is
doing
that
copy.
Third
click
is
to
paste
it,
I'm
not
counting
switching
tabs,
because
I
could
have
this
open
on
another
browser.
Fourth,
click:
fifth:
click
to
paste
that
I'm
gonna
go
down
here.
Most
of
this
can
stay
the
same.
My
size
is
going
to
be
100
runners,
oops
and
then
we'll
go
down
down
down.
So
that
should
be
oh
boy.
I
might
have
lost
clown.
Six
clicks,
seven,
eight
nine!
A
A
A
This
sub
document
discusses
a
lot
of
the
different
features
that
are
in
here.
So
this
is
really
designed
around
scaled
out
runner
operations.
If
you
have
it,
if
you
don't
have
it,
these
features
are
still
convenient.
So
one
of
the
things
that's
built
in
is
maintenance,
upgrading,
downgrading
and
scaling
and
we'll
actually
see
that
at
the
end,
we're
going
to
try
to
do
upgrade,
or
actually,
in
this
case,
a
downgrade
of
100
runners
and
see
how
long
that
takes.
A
A
If
you
happen
to
be
running
self-managed,
it
also
does
get
lab
runner,
naming
that
includes
the
the
amazon
account
id
and
the
instance
id,
and
the
reason
for
this
is,
if
you
have
a
lot
of
amazon
accounts,
and
you
maybe
even
have
more
than
one
git
lab
instance,
it
can
get
very
confusing
to
simply
find
the
runner
out
there
in
amazon
land
or
vice
versa.
You
found
a
runner
out
in
amazon,
land
and
you're
like
where
do
I
find
this
in
gitlab,
so
the
tagging
information
on
both
sides
help
you
locate.
All
of
that.
A
It
also
supports
windows
and
linux
out
of
the
box
with
parity,
so
you
can
do
docker
windows
runners
if
you
want,
or
shell
windows,
runners
and
docker
linux
or
shell
linux
runners
as
well.
It's
also
extensible
in
in
that
particular
regard.
So
if
you
look
in
the
repository
files
here
under
runner
configs,
we
have
four
shell
scripts
for
the
two
platforms.
If
you
wanted
to
create
a
different
one,
you
could
create
a
different
one.
A
Let's
go
back
to
our
features
document,
some
of
the
other
things
that
are
built
in
so
it
has
high
availability
built
in.
So
if
you
deploy
it
with
one
runner,
it
has
warm,
which
means,
if
the
instance-
that's
that
is
your
runner
terminates,
it
will
self-spawn
a
new
one
and
you'll
have
a
short
brief
window
of
downtime.
A
While
it
starts
a
new
one,
if
you
do
two,
then
you'll
have
no
downtime,
although
you
might
have
reduced
capacity
if
one
of
them
terminates
and
then,
if
you
do
three
or
more
you're,
just
basically
doing
manual
scaling,
which
also
has
with
it
as
well
and
then
finally,
you
can
schedule
so
you
can
schedule
the
entire
asg
to
shut
down
and
start
up
at
a
certain
date
and
time
based
on
the
built-in
amazon
scaling
cron
schedule.
So
all
of
this
is
exposed
to
you
in
the
parameters.
A
In
fact,
when
we
blew
through
the
parameters
there,
there
were
scheduling
parameters.
We
could
have
gone
ahead
and
used
in
order
to
cause
time-based
scaling
to
happen
for
for
scheduling
that
runner
also.
We
have
cost
and
budget
tagging.
So
when
you
use
one
of
the
easy
buttons
we
tag
all
the
runners
and
if
you
have
a
cost
management
tool
for
your
entire
amazon
installations,
then
all
of
that
can
flow
up
and
you
can
see
what
runners
are
costing
you
across
the
board,
wherever
they're
running
in
whatever
accounts.
A
Now
we
support
spot
and
on-demand
compute.
The
one
we're
spinning
right
now
is
100
spot
and
you
could
spin
a
different
runner
asg
with
100
on
demand
and,
importantly,
the
information
about
whether
it's
a
spot
runner
or
an
on-demand
runner
bubbles
right
up
into
the
tags
in
gitlab,
and
this
lets
you
select
your
type
of
compute
based
on
what
kind
of
job
you're
running.
So
you
can
say
hey.
A
You
know
what
I'm
testing
I'm
doing
mass
scale
testing
if
a
few
of
the
nodes
terminate
doesn't
really
matter
so
I'll,
run
that
on
spot
compute,
where
it's
cheap,
you
might
say,
hey
this
job
is
going
to
sit
here
and
pull
for
status
and
might
sit
there
running
for
a
day.
Looking
for
status
from
some
long
running
process,
I
don't
want
that
on
spot
compute
if
you're
not
familiar
spot
compute
terminations
are
not
that
frequent
and
you
can
pick
within
this
quick
start.
A
You
can
pick
a
scheduling
methodology
that
reduces
them
even
further,
so
they're
quite
low
as
it
is
also
it's
got
a
lot
of
security
capabilities
built
in
it
will
create
an
im
instance
profile
for
you.
If
you
have
your
own,
you
can
simply
override
it
and
peg
it
to
that.
It's
also
least
privileged,
so
it
always
gives
the
least
amount
of
privileges
for
everything
and
least
config.
So
if
you
don't
use
something
in
it,
it
won't
configure
the
item
at
all.
A
With
this
approach,
what
we
do
is
we
rely
on
amazon's
scaling,
appliance
ware
to
do
the
actual
runner
scaling
and
then
within
git
lab
each
of
those
runners
registers
independently
and
git
lab
then
distributes
the
jobs
to
them.
So
it
works
a
little
bit
differently.
You'll
see
a
lot
more
runners
in
here,
because
the
automation
is
using
both
a
termination
hook
as
well
as
a
launching
hook.
A
A
We're
getting
a
lot
of
our
success
signals
in
looks
like
we're
complete
okay.
So
let's
subtract
our
times
here
we
have
32
minutes
51
and
we
started
at
25.49.
So
if
my
math
is
right,
that's
about
seven
minutes.
So
we
have
100
runners
in
seven
minutes,
let's
refresh
here,
to
see
if
all
100
are
registered
yet-
and
it
says
we
have
100
runners
registered
so
there
you
have
it
100
spot
runners
in
less
than
10
minutes
and
less
than
10
clicks.
A
So
that's
pretty
cool.
However,
it
doesn't
stop
there.
So
this
is,
as
I
mentioned,
engineered
for
long
term
maintainability
as
well
a
lot
of
times
when
you
run
into
automation,
it's
designed
to
get
you
up
and
running,
but
then
in
six
months,
you're,
like
oh
man,
all
these
runners
are
too
old.
How
do
I
update
them?
All
it's
going
to
take
me
forever.
Do
I
have
to
destroy
everything
and
start
over?
So
in
this
case,
you
do
not
we're
going
to
go
into
our
stack
now.
A
It
started
as
a
root
stack,
which
called
the
sub
stack,
and
that
was
so.
We
could
hide
some
of
the
complexity
of
the
parameters
that
are
available
in
the
underlying
automation.
So
I'm
going
to
go
into
my
sub
stack
where
the
actual
instances
respond
up
and
I'm
going
to
hit
update
over
here.
Now,
it's
going
to
try
to
recommend
me
to
go
to
my
root
stack
and
in
most
cases
that
would
be
the
right
thing
to
do,
but
my
parameters
that
I
want
to
access
are
not
exposed
there.
A
So
I'm
going
to
say
no,
I
do
mean
to
do
this,
we're
going
to
leave
the
template
the
same
then
we
go
in
and
we
can
see
all
of
the
parameters.
So
this
is
how
many
parameters
were
really
in
use
in
the
background,
and
we
hid
some
of
those
on
the
front
end.
So
what
we're
going
to
do
is
update
this
stack
now,
I'm
going
to
emulate
an
upgrade
by
actually
downgrading
it.
You
can
see
here
we
could
pick
linux
or
windows.
We
can
pick
the
version
of
this
particular
automation.
A
A
We
also
can
control
the
job
currency,
which
was
hidden
from
us
before
and
then
we're
going
to
go
down
to
some
special
update
settings
so
all
the
way
to
the
bottom.
Here
it's
asking
me
about
specific
settings
to
do,
and
so
what
we're
gonna
do
is
we're.
Gonna
leave
the
instance
deployment
strategy
rolling
through
instances,
I'm
gonna
cheat
a
little
bit
and
be
fairly
aggressive
here,
so
are,
if
you
were
really
doing
this
in
production
you
might
out
of
our
100
or
what
will
end
up
being
80.
A
You
might
only
want
to
do
two
or
five
at
a
time
so
that
your
capacity
isn't
impacted
too
much,
but
what
I'm
going
to
do
is
make
it
go
a
little
faster
and
we'll
do
20
at
a
time
and
then
the
minimum
number
of
instances
in
service
at
any
given
given
time.
So
this
is
like
the
low
end
threshold
of
the
fewest
number
that
you
would
want
running.
We
could
have
done
stuff
like
peg
our
army.
We
could
be
upgrading
our
army,
we
could
be
going
forward
or
backward.
A
So
if
this
all
works
correctly,
what's
going
to
happen,
is
it
will
roll
through
our
asg
and
we'll
see
it
terminating
instances,
starting
new
instances
with
the
new
configuration
in
the
gitlab
panel,
because
we
are
doing
the
termination
hook,
it
will
deregister
the
current
ones
as
they
are
taken
out
of
service
and
it'll
re-register,
the
new
ones
and
we'll
start
to
see
the
version
of
you
know.
A
portion
of
our
runner
fleet
will
be
at
different
versions
at
any
given
time.
I'm
just
going
to
refresh
this
here.
A
A
We
should
have
20
of
them,
terminate
fairly
quickly,
go
down
to
25
here,
everything's
still
at
13
11.,
so
we
haven't
really
gotten
significantly
underway.
Yet
I'm
going
to
sort
on
instant
state
and
we'll
go
to
our
second
page
here.
So
here
you
see
initializing,
so
we
have
1
2,
3,
4,
5,
6,
7,
8,
9,
10,
11
and
12.
A
new
instances
initializing.
So
it
is
starting
up
new
instances.
Look
here.
We
have
83
instances,
so
it
terminated
20
right
away
because
we
changed
the
total
number.
So
we
don't
need
to
upgrade
those
20
just
to
terminate
them.
If
we
go
back
to
get
lab
and
refresh,
we
had
81
there
for
a
second
now
it's
got
60
so
we're
in
an
in
intermediary.
So
we
also
said
that
we're
we're
going
to
process
20
at
a
time.
So
what
has
happened?
A
A
A
page
six.
We
start
with
the
the
the
you
know:
we've
done,
we
finished
20
and
now
it's
going
to
start
cycling
on
another
20.,
so
you
can
see
that
the
maintenance
built
in
is
also
really
handy
to
build
and
now
manage
this
runner
fleet
going
forward.
It's
also
actually
very
easy
to
just
destroy
the
whole
thing
and
set
up
a
new
one.
If
you
need
to
also
the
vending
machine,
you
can
do
it
as
many
times
as
you
want
so
think
about
some
of
the
runner
situations
that
you
might
have.
A
A
Let's
say
that
you
want
special
runners
for
deployment
and
they
only
need
to
be
running
on
the
weekend
because
you
only
do
deployments
on
the
weekend
and
you
need
one
per
account.
So
you
could
use
this
to
vend
a
runner
to
each
of
those
accounts
and
put
it
on
a
schedule.
So
it's
only
running
during
the
weekend,
then
let's
say
you
have
runners
per
each
development
team
and
some
development
teams
are
not
global
they're
in
a
certain
set
of
time
zones
and
they
only
work
during
weekdays.
A
So
you
could
set
that
one
up
and
schedule
it,
so
it
only
runs
during
weekdays.
Any
of
these
can
be
run
in
spot
that
you're
comfortable
with
running
on
spot.
So
I
have
a
hundred
instances
running
on
m5
large,
and
maybe
you
can
guess
how
much
it
would
cost
me
to
run
these
for
an
hour
if
you
guess
around
five
dollars,
you're
right.
So
it's
not
going
to
cost
me
more
than
five
dollars
to
run
these
100
instances
for
an
hour.
A
A
A
If
we
get
back
to
page
one,
so
we're
just
chunking
through
the
asg.
If
we
wanted
to
get
a
better
details
on
where
we're
at
we
can
go
to
our
auto
scaling
groups,
we
can
dig
into
the
auto
scaling
group.
You
can
see
here,
83
instances
now
desired,
min
and
max,
and
then,
if
we
dig
in,
we
can
see
activity
and
we
can
actually
watch
what's
going
on
here.
So
we
can
see
how
many
are
being
terminated,
what
life
cycle
section
they're
in
let's
see
if
we
get
right
to
the
end
here.
A
And
if
we
had
any
that
were
in
a
status
of
either
their
their
pre-launch
hook
or
their
terminating
hook,
we
would
be
able
to
see
that
as
well,
so
you
can
check
it
all
out
and
what
we'll
do
is.
I
should
probably
refresh
this
actually
once
we
have
all
of
them
cycled.
I
want
to
take
a
peek
at
the
time
and
see
what
it
took
us
to
cycle
through
all
of
them
page
15.
A
Create
progress,
so
this
is
just
a
running
log
of
what's
been
going
on
in
our
cloud
formation,
the
cool
thing
about
doing
it,
confirmation
I
could
have
gone
into
the
asg
and
altered
it
directly,
but
the
problem
is
like
if
I
want
a
different
gitlab
version.
Well,
I
couldn't
do
that
in
the
asg.
But
basically,
even
if
I
alter
the
number
in
here,
then
the
stack
keeps
the
state.
A
Let
me
just
quickly
take
you
over
and
see
if
there's
any
other
things
that
we
haven't
talked
about
so
for
linux
we
support
arm.
So
you
can
do
everything
on
arm.
If
you
want
to,
we
also
use
amazon.
Linux
2
for
the
runners
and
that's
because
amazon,
optimizes,
amazon,
linux
2
to
run
on
their
system
so
just
running
the
regular
distro
on
a
full-on
operating
system
is
more
lean
and
then
on
top
of
it
they
have
the
newer
version
of
the
kernel
so
technique.
A
It's
the
centos
7
binary
compatible
distro,
but
they
have
the
newer
kernel,
and
so
they
can
use
the
newer
docker
overlay
drivers
which
speeds
up
docker
a
bit.
So
there's
a
lot
of
benefits
along
that
line.
I
didn't
mention
earlier
that
we
support
mixed
instances,
policy,
which
is
one
asg
that
has
some
spot
and
some
on
demand.
So
you
can
say,
give
me
a
50
spot
50
on
demand
and
then
each
of
those
instances
is
self-identifying
for
those
tags.
We
didn't
look
at
the
tags.
A
So
if
we
go
up
to
the
top
here,
the
tags
show
us
the
os.
They
show
us
the
get
lab
executor
in
this
case.
It's
docker
and
the
compute
type.
So
it
says
dash
spot.
If
these
were
non-spot,
it
would
say
dash
on
demand
and
then
amd
64..
So
it's
telling
us
what
the
hardware
architecture
is,
so
that
if
this
was
arm,
we
would
see,
I
believe,
arm
64
there.
So
you
can
target
jobs
based
on
those
attributes
of
the
actual.
A
Compute,
that's
underlying
double
check
here
again
on
our
status
and
we'll
refresh
our
console
here,
65
occasionally.
While
it's
doing
this,
it's
it's
not
too
worried
about
a
few
extra
instances
hanging
around
while
it
is
cycling
through
so
on
page
17,
we
have
13.7
page
19,
page
20.,
so
we
should
be
done.
A
A
A
And
update
complete
47
minutes
43
seconds
so
just
over
10
minutes,
so
this
has
been
100
spot
runners,
provisioned
in
under
10
minutes
and
under
10
clicks
and
100
spot
runners
updated
in
about
10
minutes
and
about
3
clicks.
So
I
hope
that
this
automation
is
something
that
you
will
find
valuable.
You
can
find
it
out
on
guided
explorations,
aws,
so
gitlab.com
guided
explorations,
slash,
aws
and
then
gitlab
h,
a
scaling
running
vendor
machine
for
aws.