►
Description
Presented by Stuart Davidson, Senior Engineering, Mana
Skyscanner practice Continuous Deployment wherever possible and have a robust deployment system in place – but it’s not been easy and there’s been plenty of bumps and unintended consequences along the way. Tools that work at a smaller scale become strategic roadblocks as you try to move your team into a faster and safer path to production. Stuart will hope to dispel the myth that you need to be "all-in" from the start and walk through how Skyscanner got to where they are, one step and one transition at a time.
About GitHub Satellite 2019
A community connected by code
Explore our interconnected community—and how collaboration turns ideas into innovations.
Join us in November at San Francisco's Palace of Fine Arts for GitHub Universe - https://githubuniverse.com/
A
Right,
a
couple
of
services:
there,
public
service
announcements
before
we
get
started,
and
you
know
there's
a
much
better
talk
going
on
upstairs
right,
yeah,
okay,
I'm
assuming
this
is
the
lunchtime
effect
everybody's
had
a
huge
lunch
and
can't
be
bothered
walking
up
the
stairs.
Thank
you
all
for
coming
along,
though
I
really
appreciate
it.
I
got
a
new
watch
recently,
and
it's
got
this.
This
heart
rate
monitor
on
it
and
for
the
last
five
minutes
has
been
going
up
and
up
and
up
and
up
if
it
starts
bleeping,
I'm,
probably
clinically
dead.
A
As
long
as
I'm
moving,
don't
panic.
Okay,
it'll
be
fine.
Final
thing:
I've
sent
this
link
to
the
live
stream
to
my
mum
and
dad
and
my
wife,
so
hello,
mum
and
dad
hello,
wife,
let's
just
to
prove
that
all
the
hours
that
I
spent
in
front
of
a
computer
instead
of
being
outside,
have
actually
come
to
something.
There's
actually
look
at
this
or
it
gave
up
satellite.
It's
amazing,
it's
a
meeting
for
a
venue
and
a
keynote,
and
all
it's
been
awesome.
So
far
right
have
we
started
the
timing
hasn't
started
yet.
A
We've
started
right:
good,
okay,
so
my
name
is
Stuart
Davidson
I'm
here
to
talk
to
you
about
continuous
deployment
as
Skyscanner
I'm,
not
a
vendor
I'm,
not
someone
who's
here
to
sell
you
things,
I'm,
hoping
to
walk
you
through
some
stories,
some
anecdotes
of
how
Skyscanner
got
to
a
place
of
continuous
deployment.
It's
not
been
easy.
There's
been
a
lot
of
a
lot
of
mistakes.
Basically
with
me
that
we've
made
a
lot
of
mistakes.
A
I
would
use
stronger
language,
but
we're
in
a
public
setting
and
hopefully,
by
doing
this
talk
I
can
share
the
learnings
that
we've
had,
so
our
mistakes
aren't
nearly
as
bad.
If
you
take
something
away
from
this,
then
there's
value
to
the
mistakes,
and
we
don't.
We
don't
feel
nearly
as
bad
right,
so
continuous
deployment,
Scott
yeah
I
meant
to
introduce
myself
I'm
Stuart
Davidson.
There
was
a
big
picture
of
me
on
previously.
My
my
handle
is
spec
Arts,
page
Twitter,
github
LinkedIn.
A
If
you've
got
any
half
of
the
reason
that
I
come
and
do
this
is
to
hear
where
you
are
with
your
organization
and
what
you're
doing,
because
we're
just
one
small
organization,
who's
managed
to
do
this,
we're
quite
were
relatively
small
compared
to
some
of
the
companies
out
there.
I
was
talking
to
a
lady
that
was
working
at
SAP.
Recently,
that's
a
big
company.
So
coming
talk
to
me
about
where
you
are,
what
you're
doing
I'm
sure
I
can
learn
something
from
you,
so
I'm
really
keen
that
it
continues.
A
I'm
the
tribe
leader
of
a
tribe
in
Skyscanner
called
the
platform
tribe.
We
run
all
the
infrastructure,
so
we've
got
enormous
big
kubernetes
clusters
kicking
around
the
globe.
We
do
all
the
the
routing
we
do.
Our
EWS
account
management.
All
that
sort
of
stuff
and
yeah
get
hub
have
asked
me
to
come
and
talk
about
this
journey
before
we
get
into
the
governs
of
it.
I
just
wanted
to
very
quickly
discuss
the
difference
between
continuous
delivery
and
continuous
deployment.
A
But
there's
there's
a
manual
step
between
getting
your
code
from
your
package
management
or
whatever
you
store
it
into
production,
whereas
continuous
deployment
every
commit
that
is
merged
into
your
master
branch
ends
up
in
production
right,
it's
all
automated!
So
there's,
no
one
that
there's
no
one!
That's
going
boom,
this
sort
of
fuels
the
right
weight.
You
know
like
these
random
things
that
we
think
about
these
arbitrary
metrics
that
we
decided
that
a
release
is
good
enough
or
not.
That's
that's
the
big
difference.
A
So
I'm
going
to
talk
about
continuous
deployment,
we
throw
all
of
our
marriages
straight
into
production,
most
of
our
merges
into
production
right,
we're
speaking
the
the
conversations
I've
had
over
even
just
last
last
night,
the
three
things
people
want
to
know
and
I
hope
I
cover
these
and
if
I
don't
please
please
come
and
talk
to
me.
How
did
Skyscanner
get
to
continuous
deployment?
What
impact
did
it
have
on
our
business
and
what
lessons
did
we
learn?
A
I
felt
I
should
call
these
owners
now,
because
the
rest
of
this
talk
is
just
stories
and
so
yeah.
Well,
hopefully,
I
cover
these
three
things.
If
I,
don't
please,
please
get
in
touch
right,
let's
star
once
upon
a
time
cast
your
mind
back
we're
we're
talking,
2016
end
of
2015
start
2016
I'm
in
a
squad
called
release
engineering,
there's
about
six
of
us
and
most
of
our
tank.
We
run
our
source
control.
A
We
have
an
enormous
team
city
instance,
and
then
we
have
a
package
management
tool
and
we
don't
really
focus
on
the
source
control.
The
package
management
because
most
of
our
time
is
spent
administrating.
This
massive
team
and
their
team
city
instance
and
I
see
massive.
We
have
like
50
agents
but
we're
perpetually
working
on
trying
to
upgrade
these
agents
to
cover
all
of
the
use
cases
in
our
business.
We've
got
a
backlog
that
obviously
that
would
choke
a
horse.
A
That's
a
that's
a
Scottish
freeze
to
see
it
was
a
very
large
backlog
of
work
to
do
and
we
were
getting
absolutely
nowhere.
And
then
we
got
this
new
CTO,
a
guy
called
Brian
dove
and-
and
he
started
to
talk
to
us-
and
that
was
odd.
Like
see
see
X
always
talking
to
you
know,
sis
admins,
it
felt
a
bit
weird,
but
he
was
he
knew
what
he
was
talking
about
right.
He
was
clever
and
he
was
asking
good
questions
and
he
was
listening
and
we
started
to
feel
like
people
and
it
was.
A
It
was
weird,
but
it
was
a
good
feeling
right.
We
were
getting
attention
if
only
we'd
seen
the
signs
earlier
right
because
and
I've
picked
a
bad
picture
of
him
specifically
because
of
this
question.
He
he
came
down
I,
remember
really.
Clearly
he
came
down
one
one
day
and
we
were
all
sitting
in
our
desks
and
he
came
and
he's
a
higher
IQ
my
folks
days
ago,
on
the
other,
and
he
goes
I
want
you
to
create
a
platform
that
will
deploy
10,000
changes
ID
to
our
website.
A
Right
now,
I
want
to
start
off
by
saying
this
is
all
poetic
license?
Okay,
yes,
there
was
this
conversation,
but
there
was
also
lots
of
emails
and
your
plans
and
PowerPoint
presentations,
but
they're
not
nearly
as
interesting
for
a
talk
like
this,
so
this
is
kind
of
how
it
went
right.
I
want
you
to
create
a
platform
that
will
deploy
10,000
changes
a
day
to
our
website.
Now
we
we
weren't
sure
if
he
was
kidding
or
not,
we
weren't
sure
if
he
was
making
it
up,
and
so
we
we
can
I
think
you
mean
10.
A
Now
these
characters,
if
you're,
not
wrecking
Marty
fans,
these
are
the
meeseeks
and
there
there
are
I
guess
a
race
of
beings
that
are
dedicated
to
helping
other
people
and
their
downfall
was
trying
to
teach
people
how
to
play
golf
as
a
Scotsman
that
it
just
it
was
the
best
episode
in
the
world,
and
we
look
a
lot
like
that
with
the
ginger
hair
and
the
blue
skin,
because
we
see
we
have
lost
the
daylight.
So
this
is.
This
is
almost
real
life.
A
If
you
can
imagine
that
as
a
Skyscanner
office,
we
think
you
mean
10,
he
clarified
no
I
actually
mean
10,000.
Maybe
he's
got
his
his
reference
of
data
wrong.
Maybe
he
means
a
year.
No,
he
means
10,000
deployments
parody
every
day,
so
we
thought
we'd,
maybe
ask
him
well
how,
because
he's
obviously
seen
this
this
great
grand
idea,
I'm
not
willing
to
suggest
we're.
So
how
do
we
go
about
this
Brian?
You
worked
and
what
do
we
do?
A
He
said
well,
I,
don't
really
know,
but
that's
what
we
pay
you
for,
but
if
you
need
anything
support
messaging,
that
sort
of
thing
you
let
me
know
and
as
he
walked
away
I
asked
for
gin,
but
it
didn't
really
happen
and
he's
yet
to
actually
give
me
any
sort
of
beverage.
So
I'll
follow
that
up
afterwards,
but
this
was
this
souse
into
a
flat
spin
right.
We
were
late.
How
do
you
even
go
about
doing
something
like
this
we'd
heard
stories
right?
You
hear
these
big
internet
economy
companies
Facebook,
Amazon
Google.
A
You
know
these
these
companies
that
have
like
40,000
people
doing
what
well
in
our
company
at
the
moment
it
would
be
grant
Kaitlyn,
Fergal
and
nobuta
right
they
and
Carla's.
So
that's
five
individuals
trying
to
trying
to
do
the
same
thing
like
how
do
you
get
to
there?
How
do
you
do
that
and
then
we
realized?
Actually
this
set
us
up
to
investigate
other
things.
A
He's
he's
cleared
all
the
assumptions
that
we
had
before
about
how
we
had
to
continue
to
do
things,
and
we
started
to
look
at
our
tooling
in
a
different
way,
because
we
knew
we
absolutely
knew
that
the
tools
that
we
were
using
were
never
going
to
scale
to
where
we
wanted
to
get
to.
So
we've
got
the
CTOs
permission
to
start
changing
things
right.
So
the
first
thing
that
we
we
investigated
was
our
CI,
tooling
and
I.
Call
this
the
Jenkins
paradox
because
more
people
know
about
Jenkins
than
teamcity,
but
it
still
stands.
A
A
If
you
wanted
the
the
latest
version
of
Java
C
you
you
have
to
get
on
the
build
agents
and
to
get
on
the
build
agents
you
had
to
talk
to
us
so
just
depending
on
you
know
how
much
time
we
had
depended
on
what
the
technology
choices.
We're
of
the
whole
company
and
so
yeah.
How
do
we
find
out?
They
were
in
a
strategic
enabler.
How
did
we
find
out?
A
We
were
this
force
multiplier
for
good
in
the
company
when
we
realized
that
we
were
this
massive
strategic
roadblock
to
getting
anything
done
in
the
company,
so
yeah.
The
Jenkins
paradox
is
this:
this
I
don't
know
I've
been
trying
to
trying
to
get
a
good
way
of
explaining
it.
But
basically
you
have
this
expectation
that
every
single
bit
of
code
that
you've
maybe
written
in
the
last
six
months
and
that's
generous
some
companies-
think
the
last
three
years.
You
should
be
able
to
rebuild
and
put
into
production
right.
So
you
need
the
stability.
A
You
need
the
ability
to
to
build
code
at
any
point
in
the
day,
because
there's
a
24/7
thing
you
might
need
to
suddenly
push
out
a
change,
so
you
need
a
stable
and
you
need
the
the
actual
installation
of
code
on
your
agents
to
be
stable
so
that
you
can
re
release
that
code.
So
that's
that's
one
force,
but
the
other
force
is
all
the
new
stuff
that
people
want
to
try
out.
I.
Don't
want
to
try
out
like
Kotlin
and
Lanigan
and
java
11,
and
I
want
to
try
all
these
new
things.
A
A
So
we
started
to
look
at
how
our
CI
system
worked
and
we
discovered
an
open-source
project
called
drone.
Drone
CI
and
we've
been
using
it
for
about
18
to
24
months.
Now
it's
an
excellent
project
and
I'm
so
glad
about,
like
the
github
sponsorship
thing
that
we're
gonna
have
to
send
a
few
critters,
we
cuz
he.
It
was
Brad
visits
key.
He
was
a
single
contributor
to
this
project
and
he
created
this
open-source
CI
tool
called
drone.
A
It's
it's
not
that
unusual
know
how
drone
works,
but
fundamentally
it's
based
in
containers.
These
are
ephemeral
containers
and
what
happens
is
you
have
a
container
with
get
in
it
and
it
it
spins
up
and
it
takes
the
cord
into
into
Europe.
Sorry
start
again:
apologies
you've
got
a
scratch
container,
you
put
a
get
container
on
top,
the
get
container
pulls
your
cords
into
the
scratch
container
and
then
it
disappears.
A
Alright,
then-
and
this
is
the
critical
one
you
put
an
another
container
on
top-
that
contains
your
build
environment,
and
then
you
build
the
code
this
in
your
scratch
container,
and
then
you
get
rid
of
that
container.
That
is
an
absolute
game.
Changer
that
allows
each
squad
to
decide
their
own
build
environment
that
takes
all
the
responsibility
away
for
agents
fro
from
our
squad.
We've
suddenly
got
like
70%
more
time
back
as
crazy,
I
haven't
finished
here.
A
Sorry,
you,
you,
then
put
a
container
with
like
artifact
or
like
artifactory
plugin,
or
something
like
that
and
take
your
code
and
push
it
to
artifactory,
get
rid
of
your
factory
container
and
then
that's
the
work
done.
So
you
get
rid
of
your
ephemeral,
your
scratch
container,
it's
all
ephemeral.
Now
my
team
only
needs
to
worry
about
the
infrastructure
of
running
drawn
and
not
the
build
agents
and
not
the
different
software
tooling.
That
was
it.
A
I
can't
describe
how
big
a
deal
this
was,
and
we
had
such
a
quick
uptick
because
squads
across
the
company.
They
all
wanted
this.
They
all
wanted
to
control
their
own
destiny.
We
were
working
like
a
squads
and
trades
model
and
Skyscanner,
so
each
squad
wants
autonomy
and
they
can
now
run
what
they
want.
A
This
led
to
our
an
amazing
discovery,
and
it
was
not
intentional,
but
I
see
it
now,
because
I
think
this
is.
This
is
maybe
a
really
big
key
learning
for
a
company.
That's
trying
to
get
into
this
area.
We
had
inadvertently
trained
at
least
one
person
in
every
squad
how
to
use
containers
right
in
docker
because
they
needed
it
for
their
build
environment
and
they
were
really
empowered,
like
they
really
really
wanted
to
get
into
this.
A
So
we
had
trained
every
every
engineering,
squadron
Skyscanner
how
to
use
containers
so,
and
that
led
us
to
think
right.
Well,
we've
got
the
spare
time
and
we've
got
all
these
educated
engineers.
Now,
what
are
we
gonna
do
with
it?
Here's
the
big
plan
right.
So
again,
this
isn't
like
kubernetes.
This
isn't
massive
this.
This
is
a
very
simple
diagram
of
what
we
did
in
ECS
right,
AWS
ECS.
We've
got
a
Bluegreen
deployment
system
right:
okay,
Bluegreen
methodology
with
an
ELB
and
a
route
53
in
front
18
months
ago.
This
was
amazing,
amazing.
A
So
what
we
could
do,
while
there
was
traffic
coming
in,
was
deploy.
It
was
to
see
this
traffic
coming
into
blue.
We
add
one
container
and
and
then
the
traffic's
flowing
between
three
containers,
and
then
we
got
rid
of
the
blue
one,
and
that
mean
we
were
on
sort
of
50
50
50,
because
we'd
have
that
one
and
that
one
so
that's
a
split
over
and
then
we
added
another
green
container
and
then
we
removed
the
blue
one
and
the
you
know
we
still
dealt
with
production
traffic.
A
It
was
coming
in,
but
we
upgraded
our
software.
We
didn't
need
to
have
outages
anymore.
What
like?
That's,
that's
a
huge
thing,
but
it
was.
It
was
a
pretty
simple
thing
to
orchestrate
right
and
it's
n
containers
and
the
more
containers
you
have,
the
more
granular
you
can
be
with
your
traffic.
We
didn't
try
and
control
the
traffic
at
the
elbe
layer
or
anything
clever
like
that.
We
just
dealt
with
it
as
a
container
level.
We
still
do
in
some
of
the
places
right.
This
still
works.
A
This
still
makes
money
for
us,
but
it
was
an
absolute
game
changer.
Then
we
thought
because
it
started
to
get
ad
option
right.
This
was
good,
but
people
were
a
little
nervous
about
what
was
actually
going
on
with
their
containers
and
the
traffic
that
was
coming
in.
So
we
took
the
blue
container.
Your
awesome
idea
right
and
it's
never
finished
so
it's
version
all
one
and
then
all
the
traffic
coming
into
this
container.
What
we
did
again
not
like
this,
isn't
this
isn't
rocket
science?
This
isn't,
you
know,
visualizing
a
black
hole.
A
This
is
really
simple
stuff.
What
we
did
was
we
put
a
Nietzsche
proxy
in
front
of
it.
So
because
we
controlled
the
deployment
mechanism,
we
could,
we
could
add
a
site
car
container
of
a
Nietzsche
proxy
container
and
have
the
traffic
go
through
it
and
then
what
we
did
was
we
put
all
of
the
traffic
or
all
the
stats
and
the
traffic
into
open,
TS,
DB
and
visualized
it
through
graph
Anna.
Alright,
that
blew
people's
minds.
They
were
like
wow,
the
visibility
we
have
on
our
services.
All
of
a
sudden.
A
We
can
start
to
automate
our
are
met.
We
can
start
to
do
metrics
checks
on
our
containers
as
we
rolled
it
out,
so
we
could
assess
if
we
had
a
spike
in
500s
and
then
roll
back
our
services.
If
we
had
that
spike,
because
the
data
was
an
open,
TS
DB,
but
we
did
it
just.
It
was
like
a
change.
Over
a
day,
somebody
puts
980
proxy
container
boopadoo,
got
the
config
bush
and
happens,
and
then
what
we
did
was
we
put
a
log
forwarder,
the
other
end,
we
thought
well,
this
was
popular
enough.
A
Let's
do
it,
let's
do
the
other
reason
we
put
all
the
logs
into
into
their
network
stack
this
like
this.
Was
this
got
Brian,
seeing
things
like
we
can
deploy
an
idea,
and
so
I
was
a
terrible
American
accent.
I'm,
sorry,
especially
with
all
the
Americans
and
the
audience
he
was
like
yeah
we
can.
We
can
take
an
idea
and
put
it
into
production
and
have
production
like
monitoring
and
logging
within
half
an
hour
like.
A
A
Assumed
it'd
be
like
seeing
Baltimore
in
a
Harry
Potter
conference
right
he's
like
Gil,
get
labs
there
and
then
our
continuous
integration
pipeline.
Well,
we
had
drawn,
we
built
our
containers
and
then
we
pushed
it
into
artifactory,
J
frog.
Then
our
continues
deployment
pipeline.
We
called
our
system
slingshot
and
slingshot
was
an
epi
that
dealt
with
a
system
called
simple
workflow
in
EWS.
A
No,
this
is
a
service
in
AWS
that
few
people
have
heard
of,
but
it's
a
speed
engine,
essentially
people
I
believe,
don't
hold
me
to
it,
but
I
think
people
do
all
of
their
transactions
using
SWF
right.
So
we
had
an
epi
that
pushed
the
state
of
the
deployment
intense
WF.
We
said
we
want
to
start
a
deployment
and
SW
have
said
you
want
to
start
deployments.
The
API
said
write
about
it,
deploy
a
better
fire
route,
53
and
say:
okay,
you
want
to
deploy
route
52
right.
The
speech
was
held
in
SWF.
A
A
The
field
halfway
through
and
the
service
owner
asked
well
how
do
I
fix
this
because
I'm
kinda
halfway
through
a
deployment,
we
don't
know
what
the
state
is,
and
we
said,
don't
worry,
just
fully
uninstall
your
service
and
then
fully
reinstall
your
service,
and
that
wasn't
cool
because
there
was
production
traffic
coming
in
and
it
was
making
money
and
things
so
and
yeah.
We
we
quickly
moved
away
from
that
model
and
that's
a
key
learning.
Okay,
don't
orchestrate
your
deployments
using
a
batch
script,
SWF
ste
engine,
we
use
DC
s
as
our
scheduler.
A
Again,
it's
not
as
glamorous
as
kubernetes,
but
we
learned
an
awful
lot.
It
was
there.
It
was
ready
to
go.
We
don't
need
to
administrate
it
very
much
it's
running
in
AWS,
and
so
we
really
thought
like
the
benefits
we've
got.
We
still
run
a
massive
workload
in
ECS.
We
are
moving
a
lot
of
our
stuff
to
kubernetes,
but
I
feel
we're
getting
more
out
of
kubernetes,
because
we
now
understand
the
benefits
that
certain
parts
of
kubernetes
brings.
You
don't
need
to
go
straight
into.
A
Yeah
and
and
we
started
because
we
could
doing
containers-
we
actually
started
doing
lambdas
as
well,
and
that
what
again
not
a
huge
change
to
what
we
were
doing
and
yeah
you
go
from
source
code
to
your
service,
which
give
the
the
really
management
phrase
of
from
source
to
service
and
zero
clicks
all
right.
So
yeah,
our
CTO
you'll
love
that
phrase,
and
he
used
it
all
the
time
and
we
got
to
stand
in
front
of
our
investors
and
see
things
like
that
and
they
loved
it
as
well
right.
A
So
this
was
our
in
that
that's
the
end
of
2017.
Obviously,
we've
progressed
since
then.
I
did
want
to
talk
a
little
bit
about
some
of
the
benefits
of
continuous
deployment,
because
the
initial
reaction
to
the
idea
of
every
commit
going
into
production
is
one
of
sheer
terror
right.
It's
like
you,
can't
you
put
what
if
we
think
with
the
QA
Department
and
the
Excel
spreadsheet,
how
do
you?
How
do
you?
How
do
you
mitigate
the
risk?
This
is
a
massive
risk.
You're
imposing
every
commit
is
going
into
production
without
a
human
verifying
it.
A
A
The
would
you
rather
would
you
rather
take
all
of
the
commits
that
you've
made
and
your
squad
has
made,
and
potentially
your
tribe
has
made
over
the
course
of
the
week
and
then
throw
it
on
to
production
and
see
what
happens
right
or
would
you
rather
take
each
change,
individual
change
to
do
on
to
production
and
see
what
happens
see
what
the
effect
is
take
another
one
put
on
to
production.
Take
another
one
put
on
to
production.
A
I
would
rather
see
what
each
change,
what
affects
what
regressions
it
had
on
my
platform
I
would
rather
do
them
one
by
one
that
seems
less
risky.
That's
mitigating
the
risk
right,
so
continuous
deployment
is
potentially
less
risky
than
bundling
up.
You
know
months
of
people,
effort
and
throwing
onto
production
and
just
seeing
what
happens
right,
the
other
one
is,
if
you
notice,
there's
a
problem
on
that
big
week's
worth
of
of
effort.
What
do
you
do
you
roll
back,
a
week's
worth
of
effort?
A
So
that's
put
you
back
a
whole
week
right
and
not
only
that
all
of
your
developers
are
a
week
from
there.
Maybe
even
two
weeks,
further
on
right,
you've
got
a
week
of
your
release
that
was
going
to
production.
You've,
maybe
got
a
week's
worth
of
effort
in
your
QA
department,
and
then
your
developers
are
potentially
a
week's
worth
of
development
ahead.
You
might
be
two
three
weeks
behind
by
rejecting
that
release
or
what
you
could
do
is
put
one
small
change
on
your
platforms,
see
if
it
breaks
anything.
A
If
it
doesn't
you
roll
back
one
change
and
one
change
alone,
that
developers
may
be
gone
for
coffee
right,
they've
come
back
and
they've
said
that
didn't
work,
but
I
know
why,
right?
Okay,
the
context
is
still
in
their
heads.
The
impact
isn't
nearly
as
big,
because
it's
just
a
very
small
change
and
if
it
is,
then
you
know
exactly
the
change
that
caused
the
issue
rather
than
digging
through
a
week's
worth
of
effort
and
trying
to
discover
what
went
wrong
right.
This
is
it's
more
risk
mitigation.
A
The
third
thing-
and
this
is
something
that
maybe
drives
me
to
get
up
in
the
morning,
but
when
I
was
a
graduate-
is
to
work
for
a
company
and
I
won't
mention
the
company,
but
I
used
to
work
for
a
company
and
I
was
involved
in
the
release.
Engineering
area
and
I.
Remember
distinctly
it's
3:00
in
the
morning
getting
up
to
do
a
production
release
right
so
my
girlfriend
at
the
time
and
my
wife,
my
wife,
and
was
really
pretty
angry
at
me
for
disturbing
her
sleep
at
3:00
in
the
morning.
A
So
already
I've
got
that
angst.
I
get
my
laptop
out
and
I
set
it
up
on
my
dining
table
and
I'm
really
tired,
because
I've
had
a
busy
day
before
and
I'm.
Looking
at
this
document,
this
word
document
describing
all
of
the
steps
that
are
meant
to
be
there.
But,
of
course,
no
one
updates
a
Word
document
when
they're
updating
the
code
right.
So
there's
a
bit
of
ambiguity.
A
If
this
is
accurate
or
not
and
I'm
taking
these
steps
in
this
word
document
and
I'm
typing
in
config
into
text
boxes
that
aren't
doing
any
validation
or
any
sort
of
things
like
that,
and
then
I
hit
go
and
it
broke.
The
system
write
production
failure
outage.
Yes,
it
was
at
3
in
the
morning.
Yes,
there
were
no
customers
affected,
but
I
had
no
support.
I
had
no
one
was
awake
to
help
me
out.
A
I
had
no
idea
of
what
to
do
next,
because
there
was
everybody
was
asleep
right
and,
as
a
graduate
are
you
gonna
go
and
wake
up
your
boss
because
you've,
you
know,
stumbled
in
a
you,
know,
config
mistake
and
have
to
admit
to
him
that
you
give
or
or
heard
that
you've
made
a
terrible.
You
know
mistake:
that's
just
distress
that
you're
putting
on
your
people.
How
dare
you?
How
dare
you
through
that
stress
on
individuals
like
that?
That's
not
cool
right!
So
you
know.
Mental
health
is
a
massive
issue
in
in
software
engineering.
A
These
days
and
I
think
that's
a
real
aspect
of
what
we're
doing
here.
We
can
deploy
during
the
business
day
everybody's
fresh,
everyone's
alert.
Everyone
knows
what's
going
on
and
if
there
is
a
problem,
we
fix
it
and
there
are
ways
of
mitigating
your
blast.
Radius
right,
you
can
work
in
different
regions
rather
than
have
only
one
server
you
can.
You
can
do
all
sorts
of
things
to
mitigate
your
risk,
but
don't
put
that
risk
on
these.
These
pure
individuals
that
are
just
trying
to
meet
your
company
work
wow
that
got
a
bit
deep
dinner.
A
Sorry
right,
where
was
I
yeah
right,
yeah,
okay,
container
registries,
right
another
another
learning,
key
learning
we
got
into
containers
pretty
heavily.
We
started
using
them
in
production
when
others
didn't
right.
You
would
go
to
these
conferences
and
it's
all
like
who's
using
containers,
who's
using
the
reproduction
boom
right
so
and
we
started
using
them
in
production,
and
we
we
discovered
very
quickly
that
one
of
the
most
important
parts
of
your
infrastructure
is
your
your
container
registry.
A
And
so,
if
there's
anybody
from
the
packaged
registry
team
here
I
would
I
would
strongly
advise
to
understand
the
the
nuance
between
the
two.
Our
code
packages
were
stored
in
our
two
Factory
and
it
didn't
need
to
be
up
24/7,
because
if
we
couldn't
build
code
that
wasn't
necessarily
a
big
problem,
I
mean
it
was,
but
it
wasn't
if
your
container
registry
is
down
and
you're
trying
to
auto
scale
or
you're
trying
to
do
a
deployment
or
your.
A
A
If
you
thought
that
come
and
talk
to
me
because
we're
hiring
right
cuz,
that
was,
that
was
not
something
we
thought
about.
Then
we
had
to
do
on
the
fly
we
had
to
put
like
a
proxy
in
aha.
That
was
a
bad
48
hours.
That
was
sad,
so
yeah.
This
is
a
big
warning.
When
you
need
it,
when
you
really
need
it,
it
will
fail
on
you.
It's
waiting
for
a
chance
to
get
you
source
control.
The
reason
I'm
here
the
reason
I'm
here
I've
just
seen
the
next
slide.
A
I
apologize
for
the
animation
is
all
gone,
weird,
sorry,
source
control.
We
started
to
move
fast.
We
started
to
make
lots
and
lots
and
lots
of
changes
on
our
platform
and
our
source
control
system
started
to
creak
at
the
edges.
Now,
I'm,
not
sure
how
much
depth
I
can
go
into
here,
but
I
will
say
that
github
was
tremendous
and
Vitor
who's
sitting
over
the
front
and
Marco
who's
sitting
somewhere.
A
The
two
of
them
put
together
such
a
great
story
about
github
enterprise
server
and
our
adoption
was
slaw
of
github
Enterprise,
because
the
operation
model
of
and
in
for
a
github
enterprise
server.
You
just
spin
up
a
single
ec2
instance.
Right
all
of
the
storage
is
local.
The
elasticsearch
stuff
is
all
in
there.
They've
got
loads
of
scripts
that
copy
the
data
over
to
another
ec2
instance,
whereas
with
the
other
vendor,
we
had
to
run
her
own
NFS,
we
had
to
run
her
own
elasticsearch.
A
We
had
to
run
a
whole
lot
of
other
things
like
this
was
amazing
and
another
enabler
for
our
ops
teams
and
our
release
engineering
teams
to
just
get
a
bit
more
time
in
the
day.
So
we
ran
this
massive
migration
and
if
I've
got
time,
I'll
talk
through
some
of
the
bits
and
pieces
I'm
a
bit
nervous,
but
I
know
I've
only
got
five
minutes
so
yeah
we
migrated
8500
Reapers
over
ten
weeks.
A
That
was
a
massive
logistic,
endeavor
I
think
it
went
well
I
think
it
went
too
well
because
no
one's
banked
to
me
yet
and
I.
Imagine
like
that's
a
good
thing
right.
If
you're
working
as
a
sysadmin,
you
make
such
a
fundamental
change
and
everybody's
like
yeah
go
out
the
next
day
and
you
know
we
were
in
github
Rob
and
get
lab
and
the
CI
worked,
and
you
know
you
know
all
of
the
issues
and
like
PRS
and
stuff
got
migrated
over
and
we
didn't
really
notice
yeah.
A
Well,
you
know
good
job,
you
like
that's,
that's,
what's
being
a
systems,
admin
is
right.
You
only
you
only
get
noticed
when
things
go
wrong,
but
yeah
massive
reduction
in
our
ops
overhead,
one
of
the
big
things
as
we
were
going
to
continuous
deployment.
We
need
to
really
consider
our
automated
testing
and
move
using
checks.
This
is
an
API
that
get
help
have
that.
A
Maybe
you've
not
looked
into,
but
it's
really
cool
you
can
you
want
to
move
your
testing
as
far
left
as
you
can,
so
you
can
use
your
checks
when
you're
doing
an
mr
something
like
that
you
can.
You
can
look
at
your
code
at
that
point
and
run
up
a
smorgasbord
of
checks
at
that
point.
Yet
don't
wait
till
you're
in
production,
that's
a
bad
thing,
move
them
move
them
lengths.
In
fact,
you've
got
BOTS.
No,
that
will
look
at
your
your
code
reviews
and
give
you
comments.
A
We've
moved
so
far
left
or
actually
into
the
developer.
So
yeah,
that's
that's
cool!
That's
that's!
A
really
appropriate
use.
So
when
sorry
for
the
the
animations
2018
we're
using
github
we're
using
ECR,
so
that's
EWS
is
hosted
container
registry
drone,
still
darker,
still
continuous
deployment,
still
using
slingshot,
simple
workflow,
ECS
and
kubernetes.
No
yeah
I,
don't
know
why.
I'm
there,
okay
I,
don't
work
in
PowerPoint.
Sorry
infrastructure
is
code.
Is
the
next
big
thing
for
us?
A
We're
we're
like
it's
not
that
we've
solved
the
software
deployment
model
it
like
we're
still
the
things
to
do,
but
we're
iterating
on
it
right.
We're
learning
things,
but
infrastructure
is
code.
Is
the
next
big
thing
for
us?
How
do
we
keep
up
with
the
kubernetes
release
cycle
of
like
a
major
release
every
ten
weeks?
That's
fast
when
it
comes
to
infrastructure,
there's
not
enough
people.
A
Thinking
about
that
as
a
problem
and
how
to
actually
do
it
deployments
at
that
piece
in
an
environment
where,
if
you
get
a
cluster
deployment
wrong,
that
has
a
big
blast
radius,
that's
not
just
an
API!
That's
you
know
all
of
these
posts
sort
of
thing
right,
so
yeah
we're
working
heavily
on
that
sort
of
area
at
the
moment
and
we're
doing
around
21,000
deploys
a
month
now,
yeah
wow,
that's
huge!
No!
It's
not
that's
the
point
right
we're
for
a
text,
box
change
or
a
you
know,
a
spelling
error.
A
We
make
a
change
right,
so
we
can
make
21,000
distinct
changes
a
month
on
our
platform,
so
it's
just
a
different
way
of
approaching
it,
but
our
risk
and
the
amount
of
outages
we
have-
and
things
has
just
so
substantially
dropped
and
Skyscanner
I'd
have
to
deal
with
weird
and
wonderful
geopolitical
situations
like
the
the
president
of
Kazakhstan
at
left,
his
post,
he
retired,
and
he
changed
the
name
of
the
capital
after
him,
you're
like
okay.
So
if
you're
doing
a
flight
search
in
sky
Scout,
that's
a
big
deal
right.
A
We
have
to
change
things
with
like
IATA
codes
as
well:
Chucky's
Airport
changed
their
IATA
code.
One
of
these
static
bits
of
data
that
never
changes.
It
changed
right
on
fact.
No
the
IATA
code
didn't
change
the
airport
dead
right.
They
moved
that
they
changed
the
airport.
They
had
to
move
enemy
so
being
able
to
change
quickly,
safely
being
able
to
react
to
situations
that
we
can't.
We
have
no
control
over,
but
we
can
have
control
over
how
quickly
we
can
react
to
it.
It's
a
it's.
A
A
big
deal
for
us
is
a
big
game-changing
deal
when
our
companies
really
invested
in
this
sort
of
work
right,
one
yeah,
okay,
so
next
things
that
we're
looking
at
continuous
deployment
of
kubernetes
I've
talked
about
package
management.
I
may
be
one
of
the
only
people
in
the
world
to
be
really
excited
about
package
management,
except
for
people
that
github
now
the
announcement
of
this
was
was
I've,
been
talking
about
the
fact
that
package
management
needs
to
evolve
over
the
last
like
18
months
to
two
years.
A
If
you
look
at
my
github
I
had
I
built
one
in
go
because
I
felt
there
needed
to
be
a
cloud
native
solution
for
this
and
there's
a
killer
feature
because
of
the
number
of
builds
that
we
do
getting
the
packages
onto
the
ephemeral,
build
agents
that
you
know
we're
off
to
scaling
we're
using
spot
instances.
These
build
agents
are
disappearing
and
starting
up
disappearance,
so
caching,
the
artifacts
in
the
right
place
at
the
right
time.
A
It's
going
to
be
critical
to
us
going
forward
and
scaling
as
a
company,
so
I'm
really
into
again
put
a
bit
pressure
and
get
our
bomb
here.
So
this
is
that's
something
I
really
want
to
see
from
package
management
working
harder,
reducing
technical,
obsolete.
We
build
a
lot
of
stuff,
so
there's
no!
This
tree
of
you
know
systems
that
never
happened
so
yeah
we're
working
to
improve
that
Oh
seconds
left
right:
okay,
big,
deep
thoughts,
things
that
I
have
learned
and
observed
over
the
last
little
while
enable
Minh.
A
If
you
have
a
team
that
focuses
on
the
enablement
of
others,
source
control,
CI
things
like
that
enablement
isn't
doing
things
for
other
people,
it's
enabling
it
to
do
them
to
do
it
for
themselves.
So
don't
focus
on
what
you
can
do
for
a
squad.
What
can
a
squad
do
for
themselves?
Right?
That's
a
huge
distinction
in
how
you
run
this
software.
A
If
you're
experimenting
with
a
Toula
process,
try
something
tricep,
don't
don't
think
about
a
six-month
Gantt
chart
with
like
kubernetes,
try
something
and
you
will
learn
and
you
will
iterate
and
then
you
can
pick
something
that's
more
relevant
rather
than
just
hypothesizing
about
it.
I
don't
think
we
would
have
got
to
where
we
are
today.
Had
we
gone
in
with
kubernetes
or
measles,
we
just
went
for
ECS
because
it
was
there
and
we
tried
it.
We
learned
and
we
iterated
there's
a
lot
of
change
in
the
ecosystem.
A
At
the
moment,
lots
of
tools
coming
out
lots
of
new
things
embrace
this
as
software
engineers
right
embrace
this
as
engineers,
because
that's
one
of
the
reasons
you
got
into
this
industry,
there's
lots
of
change.
There's
lots
of
interesting
things.
Try
not
to
get
too
hung
up
on
all
tools
and
old
ways
of
doing
things.
That's
part
of
the
joy
of
what
you
do
and
the
journey
of
a
thousand
miles
starts
for
the
single
step.
We
didn't
think
where
we
were
going
to
be
in
eighteen
months.
A
We
just
had
a
vague
idea
of
where
we
wanted
to
head
to
and
we
iterated-
and
we
did
one
small
thing
after
another
and
I
would
advocate
to
anybody.
That's
trying
to
take
that
path.
To
do
exactly
the
same.
Thank
you
very
much,
I
appreciate
your
time.
Everybody
and
if
anybody's
got
any
questions,
just
please
reach
out
to
me
in
the
social
media
Cheers.
Thank
you.