►
From YouTube: Building K8s on AWS & Enabling Developers to Deliver. What to Do Better - Greg Sansom & Fahad Tariq
Description
Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Building K8s on AWS and Enabling Developers to Deliver. What to Do Better - Greg Sansom, Zip & Fahad Tariq, CMD
Zip’s been on a journey of discovery, migration and adoption and meeting strict demands. This is the journey Greg Sansom and Fahad Tariq would like to share with you. An infrastructure team usually has brilliant ideas on how development should happen. So engineers plan and they build and they usually get eighty percent right. This talk is about the twenty they don’t, and why.
The company recently took advantage of an opportunity to develop a brand new platform with aim of vastly improving deployment capabilities. With the help of a partner, 143 git repositories, 16000 commits from 54 people, there’s been a few lessons learnt. Having developers involved from the start and deploying to the environments meant fast feedback, a reorganisation of priorities and occasionally, a validation or two.
This talk is about the challenges faced, compromises made, promises kept and lessons learned.
A
Hey
guys
welcome
and
thanks
for
coming
I'm
Greg
Sansom
I'm,
the
head
of
infrastructure
at
zip,
we're
a
Sydney
based
FinTech
disrupter.
We
offer
fare
fast
and
seamless
payment
solutions
online
and
in-store.
We've
experienced
a
huge
amount
of
growth
over
the
past
few
years,
we're
moving
from
being
a
startup
into
a
scalar
and
we've
got
got
our
global
ambitions
so
heaps
going
on
cool.
B
A
So
we
had
a
few
earlier
in
the
year
we
had
quite
a
few
major
projects
which
we
had
to
get
completed,
leading
up
into
the
Christmas
big
trading
period.
The
opportunity
here
was
to
deliver
these
projects
before
people
started
shopping
for
Christmas.
If
we
didn't
deliver
on
time,
we'd
miss
the
opportunity
and
the
initiatives
would
lose
their
value.
A
We
had
three
engineering
teams
chomping
at
the
bit
to
start
deploying
to
the
new
platform,
and
we
had
external
partners
ready
to
start
integration
testing,
and
this
was
before
we
even
started.
So
we
were
under
HIPAA
pressure
to
get
things
to
get
things
done.
The
delivery
timeline
that
we're
working
to
included
development
and
production
izing
of
a
number
of
new
micro
service
applications,
as
well
as
what
we
focused
on,
because
the
core
infrastructure
build
building
out
the
platform
making
sure
it's
reliable,
scalable,
secure,
monitoring,
instrumentation,
alerting
all
the
nuts
and
bolts
and
all
of
us.
A
B
Once
we
had
all
those
in
place,
we
led
kubernetes
clusters
on
top
using
eks
and
then
the
various
components
that
went
on
top
of
that
and
by
the
end
of
it
it
was
absolutely
huge,
as
you
can
imagine
so
that
was
the
that
was
the
platform.
And
after
that
we
had
the
same
thing
out
in
multiple
environments.
So
going
all
the
way
from
dev
to
prod.
We
had
multiple
environments
up
in
a
couple
of
weeks
and
being
actively
used.
B
A
So,
as
we
were
running
through
the
project
for
hard
and
I
were
constantly
talking
about
what
we
could
do
to
streamline
the
delivery
of
this
and
ensure
that
the
project
was
successful
once
we
completed
we
kind
of
continued
that
discussion
and
started
working
on
a
list
of
key
steps
and
measures
that
we
could
implement
to
ensure
similar
success
for
future
projects.
This
is
what
we're
going
to
be
talking
about
today,
I'll
run
through
those
steps.
A
B
So
laying
the
foundation
getting
the
foundation
right
is
very,
it's
very
important.
It's
the
key
to
everything.
So
we
started
out
picking
all
sorts
of
all
sorts
of
services
that
we
wanted
in
I'm
sure
we've
all
heard
of
one-way
door
in
two-way
door
decisions.
Sorry
we're
getting
a
bit
of
feedback,
one
way
to
all
of
us
to
wade
our
decisions
when
you're
moving
into
a
Cuban
ad
space,
especially
I,
think
it
pays
dividends
to
be
pragmatic
about
it
and
take
a
pragmatic
approach,
there's
decisions
and
applications.
B
You
can
walk
back
from
and
replace
quite
easily
and
then
there's
some
that
you
can
replace,
but
it
takes
significant
amount
of
effort,
so
everybody's
seen
the
CNC
F
landscape,
it's
quite
large
by
lost
count.
There
are
more
than
1,200
applications
there
and
it's
a
fast-moving
space.
So
what
should
the
criteria
be
to
choose
the
right
set
of
applications
that
that
suits
your
business
today
and
that's
really
it?
The
criteria
should
be.
What
helps
your
business
today?
B
What
helps
your
developers
get
your
software
into
production
as
quickly
as
possible,
safely
reliably,
that
has
to
be
the
main
driving
criteria
behind
the
choice
of
those
tooling.
We
basically
figured
out
that
a
light
and
highly
decoupled
decoupled
platform
is
a
good
platform
to
aim
for
and-
and
that's
that's
that
retrofitting
so
avoid
retrofitting.
Where
possible,
identify
your
heavily
integrated,
complex
services
up
front
as
quickly
as
you
can
things
that
will
take
a
long
time
to
to
retrofit
if
you've
made
the
decision
further
down
the
track
to
implement
so,
for
example,
secrets
management.
B
Tooling,
if
you
don't
implement
something
today
and
you
have
developers
start
to
build
out
on
the
platform
and
they
need
secrets
well,
they'll
just
work
around
whatever
solution
you
plan
to
put
in
they'll
start
using
a
double
your
secrets,
manager
or
parameter
store,
whereas
maybe
you
might
have
gone
with
hashing
walk
down
the
line
so
avoid
retrofitting
as
much
as
possible.
Yes,.
A
So
the
way
we
avoid
retrofitting
pretty
simple
way
make
sure
that
all
your
non-functional
requirements,
whatever
they
are
built
into
the
platform,
at
least
as
an
MVP
before
you
start
coding,
while
you're
laying
the
foundation
bring
the
developers
in
but
not
on
the
critical
path,
get
them
to
start
testing
and
providing
feedback,
see
what
works.
What
doesn't
work?
A
A
Yeah
so
DevOps
Inception
was
the
next
item
on
the
checklist
and
make
sure,
while
we've
been
busy
building
infrastructure,
the
product
and
development
teams
are
busy
dreaming
up
new
features
and
building
new
products
and
services
to
make
customers
happy.
That's
really
what
everything
is
all
about.
So
we
need
to
get
that
right.
A
Yes,
I
drew
during
the
product
delivery
phase,
there'll
be
a
period
of
discovery
and
requirements
gathering
usually
conducted
as
a
projecting
exception.
We've
decided
that
it's
good
to
have
a
DevOps
inception
for
every
single
product
development
project,
and
this
is
where
we
sit
with.
The
developers,
spend
a
good
chunk
of
time
with
and
figure
out
what
our
non-functional
requirements
are.
What
are
our
must-haves?
What
can
we
negotiate
on
find
any
pain
points
that
developers
might
experience
as
they're
trying
to
deploy
applications
into
our
into
our
infrastructure
platform?
A
B
Can
help
each
other
do
an
early
skills
assessment?
Essentially,
what
is
it
that
your
developers
and
your
DevOps
team?
Actually
you
know
about
the
alphabet
soup
of
applications
you're
going
to
throw
them
into
so
it's
very
easy
to
say:
I
want
one
kubernetes
please,
but
do
they
actually
know
how
to
use
it,
identify
where
they're
lacking
and
try
and
focus
your
training
on
those
specific
things
where
they
would
actually
add
value?
So,
for
example,
if
you've
got
a
very
strong
build
and
deployment
pipeline,
your
developers
really
need
to
know
a
cute
couple.
B
So
emphasize
dollars
sharing
as
much
as
possible
if
you've
got
a
really
cool
solution,
maybe
using
graphs
and
global
accelerators
and
load
balances
and
multiple
ingress
controllers
and
some
really
clever
route,
53
routing
to
glue
it
all
together.
It
doesn't
really
mean
anything
unless
one
it's
running
in
production
and
to
somebody
else
knows
how
to
support
it
or
extend
it
moving
forward.
So
putting
emphasis
on
knowledge
sharing,
make
sure
you
have
brown
bags
and
showcases
regular
brown
bags
and
showcases
have
people
attend
them
and
actually
pay
attention
and
that's
harder
than
people
think
it's.
B
It's
very
difficulty
to
have
that
happen.
Trying
to
incentivize
people
to
come,
give
sneak
peeks
about
what
you're
going
to
be
talking
about.
Make
sure
you
put
aside
time
to
prepare
for
the
showcase.
So
maybe
once
a
week,
once
every
other
week,
just
put
aside
half
an
hour
an
hour,
maybe
two
hours
to
prepare
for
a
showcase,
that's
decent
quality
and
and
and
incentivize
people
to
come,
like
I,
said
if
nothing
else
works
just
order,
pizzas
and
drinks
and
everyone
will
be
there,
hopefully
have
a
shared
living
ways
of
working
document.
B
So
we
found
this
really
important,
a
ways
of
working
document
that
had
procedures
on
how
you
raise
merge,
requests
or
how
you
get
code
into
production
or
what
your
branching
strategy
is
once
you've
got
this
semi
ready
and
throw
it
over
to
a
team
in
your
business
that
has
a
lot
of
time
on
their
hands
and
I'm
sure
there's
plenty.
They
don't
all
need
to
be
developers
or
Ops
people
once
it's
with
them,
throw
over
some
work
and
see
how
they
go.
How
long
does
it
take
them
to
raise
a
PR?
B
B
For
example,
if
you're
choosing
a
technology
in
the
container
scanning
space
like
artifactory
x-ray,
make
sure
you
put
down
the
constraints
that
you
had
around
that
and
possibly
the
objectives
and
the
pros
and
cons
while
you're
actually
choosing
it,
because
something
down
something
further
down
the
line.
Something
else
might
come
along
and
you
need
to
have
a
really
good
criteria.
What
to
assess
it
by
so
a
tooling
decision
record
because
gives
you
that,
and
unless
it's
done
while
you're
deploying
it
it'll
never
get
done
so
we
can
guarantee
that
pretty
much
cool.
B
So
the
third
point
here
was
shared
ownership
of
delivery,
Kuban
Eddie's,
perhaps
more
so
than
other
other
other
other
spaces
relies
and
implies,
and
perhaps
even
relies
on
a
strong
DevOps
focus.
But
how
do
you
actually
get
your
dev
and
ops
teams
aligned?
So
we
found
that
having
a
shared
objective
was
was
really
important.
How
do
you
get
software
into
production
and
running
on
in
a
scaleable
reliably
and
secure
way?
So
if
that
focus
is
there,
then
your
DevOps
teams
can
be
aligned
to
address
it.
B
If
the
focus
is
and
there
you
could
have
your
ops
or
your
infra
guys
over
engineering,
a
solution
for
a
problem
that
your
developers
don't
really
have
have
right
now,
so
try
and
work
out
what
those
problems
are.
Is
it
a
speed
of
deployment
issue?
Are
they
having
issues
with
the
reliability
of
the
platform?
Have
they
correctly
identified
all
the
risks
in
deployment
once
you've
worked
out,
what
those
issues
are
address
them
and
then
move
on
to
other
things.
Without
this
alignment,
infrastructure
guys
do
end
up
over
engineering
solutions.
B
Containers
also,
naturally
allow
you
to
have
a
micro
services
architecture,
but
with
micro
services
comes
a
lot
of
complexity,
so
have
your
dev
ops
team
constantly
chase
up
and
work
out
whether
you
are
utilizing
one
of
the
benefits
of
micro
services,
which
is
releasing
often?
Are
you
really
seeing
multiple
times
a
day,
or
are
you
just
doing
it
once
every
couple
of
days
make
sure
your
developers
have
the
autonomy
and
the
confidence
to
actually
push
to
production
and
have
the
ops
guy
as
a
system
with
all
hands,
if
necessary,.
A
B
Is
sorry,
okay,
the
other
thing
you
can
the
other
way
the
alignment
happens
naturally,
is
through
service
champions,
so
service
champions
are
essentially
developers
who
who
have
taken
ownership
of
a
product
and
and
and
and
know
how
to
use
it
very
well.
How
do
you
make
this
happen?
Well,
Deb's.
We
find
that
having
devs
part
of
the
design
and
QA
for
something
that
they're
going
to
be
consuming
a
service
that
the
infrastructure
team
is
coming
up
with,
is
really
important.
That
way,
you
have
developers,
perhaps
even
and
there's
an
example.
B
Greg
will
talk
about
later,
where
they
get
involved
with
the
process,
perhaps
even
enough
to
help
you
build
it
and
it
allows
them
to
take
ownership
of
it
and
then
communicate
it
to
other
devs
and
what
we
found
is
developers
to
speak
to
them
properly.
You
need
to
be
a
developer
resort,
so
have
your
champions
communicate
the
correct
way
to
use
a
particular
service
or
a
feature
that
you've
come
up
with
and
another
way
is
to
expose
a
service
or
expose
a
service
to
a
limited
number
of
developers
very
early
on.
B
We
had
had
an
interesting
problem
where
we
had
a
lot
of
clusters,
but
a
centralized
logging
solution
so
from
the
ops
team's
perspective.
That
was
the
challenge
for
us.
How
do
we
get
all
the
logs
from
these
clusters
to
elasticsearch
in
a
secure
way
where
we
have
authentication
on
the
endpoints
and
and
everything
that
was
the
challenge
we
were
addressing
when
we
saw
fluent
but
was
able
to
forward
logs
to
elasticsearch
we
and
we
could
search
them
in
Cabana.
B
We
consider
that
as
job
done
and
handed
it
over
to
the
devs,
all
the
devs
simultaneously
found
out
that
the
logs
were
practically
unusable,
so
fluent
bid
had
a
field
that
was
being
set
that
conflicted
with
the
field
that
was
coming
from
their
applications,
so
it
caused
them
a
bit
of
pain
and
it
could
have
been
avoided
had
we
had
them
involved
from
the
start.
So
just
as
every
feature
has
an
owner
that
takes
that.
That
owns
the
proper,
ongoing
maintenance
of
a
particular
feature.
B
A
Really
important
to
make
sure
that
your
developers
are
enabled
and
supported
in
building
infrastructure.
If
you
find
through
your
project,
you
have
developers
asking
a
core
infrastructure
team,
but
to
build
something
and
the
infrastructure
team
builds
the
thing
then
hands
it
over
to
the
developers
and
then
they
start
to
see
problems
with
it.
You
need
to
start
re-evaluating
new
ways
of
working
as.
A
A
Yeah
we
had
a
good
success
story
around
this.
If
I
had
alluded
to
it
before,
but
a
team
of
developers
came
to
us
and
said:
look,
we
need
an
RDS
database.
Could
just
stand
it
up
and
we
said
sorry,
we
don't
have
a
template
ready
just
yet
we
don't
have
the
capacity
to
do
it
right
now.
We
pointed
them
towards
some
documents
and
reference
template
that
they
could
use
in
sirs
for
a
terraform
module.
A
They
looked
at
that
went
away
and
built
a
highly
opinionated
specific
artiest
terraform
module,
which
included
sensible
defaults
like
multi,
AZ,
encryption
and
multi-region
deployment
once
they've
done
that
they
shipped
it
over
to
us
gave
us
look,
we
qada
and
said:
yep
looks
good,
it's
solid,
its
robust
satisfies
the
security
requirements.
Go
ahead
from
there.
They
deployed
their
database
cool,
they
weren't
satisfied
just
yet.
They
then
did
a
showcase.
A
This
is
a
developer
team,
doing
a
showcase
around
how
they
built
any
usable
infrastructure
component
shared
that
rid
the
rest
of
the
zip
engineering
team,
follow
it
from
there
I
think.
Over
the
next
few
days,
we
had
five
new
RDS
databases
spun
up
and
all
the
teams
are
happy
because
now
they
could
have
their
databases
and
we
were
happy
that
they
met
the
right
criteria
that
we
had
set.
A
We
talked
a
bit
about
two-way
value
transfer,
and
this
is
where,
by
having
the
infrastructure
specialists
support
the
developers
to
do
their
thing,
everyone
really
wins.
The
developers
are
able
to
move
faster
and
we
have
well
for
one
less
work
to
do,
but
also
satisfaction
that
the
developers
are
implementing
best
practice
standards
and
sensible
defaults
as
a
common
practice.
A
So
what
we've
been
busy
working
on
the
building
it
build
out
of
the
infrastructure
and
getting
the
developers
aligned
and
supportive
and
helping
us?
It's
also
really
important
to
focus
on
the
developer.
Experience
spend
some
time
if
you're
an
infrastructure
or
an
Operations
guy,
spend
some
time
sitting
with
the
developers
become
a
developer,
understand
their
experience
and
feel
their
pain.
A
You
don't
really
know
how
the
develop.
How
well
with
the
system,
is
working
as
a
whole
until
you've
actually
sat
with
the
developer
and
gone
through
the
house.
This
actually
work
an
example.
Here
we
saw
some,
we
heard
anecdotally
some
reports
around
poor
performance
source
slowness
of
pipelines,
which
we
kind
of
didn't
take
a
lot
of
notice
of
initially.
A
At
some
point,
we
sat
down
with
the
developer
for
some
not
related
reason
and
said:
well,
this
pipelines
too
slow.
It
was
only
a
few
minutes,
but
it
was
really
running
hundreds
of
deployments
per
day.
It
was
enough
that
it
was
causing
a
significant
problem
and
it
was
big
enough
that
we
really
needed
to
get
onto
it.
So
once
we
knew,
we
understood
that
developer
experience,
we're
able
to
prioritize
it
and
pretty
quickly
implement
a
couple
of
fixes
which
improved
the
overall
experience
and
productivity
for
the
whole
engineering
team.
B
So
I'll
just
quickly
explain
what
the
issue
was
because
I
think
we
only
have
a
couple
of
minutes.
The
we
were
using
gitlab
CI
with
docker
executives
and
the
deployment
pattern
we
were
using
had
three
musketeer
so
make
docker
compose
and
docker,
and
the
initial
docker
container
that
spun
up
on
every
pipeline
run
spun
up
more
containers.
Now
this
led
to
the
containers
without
having
a
shared
cache
being
downloaded
all
the
time
for
every
pipeline
execution.
B
A
In
the
same
way
that
product
team's
product
owners
go
and
talk
to
their
customers
and
find
out
how
that
product
is
working
out
in
the
wild
and
look
for
opportunities
to
make
it
better
or
provide
new
services,
we
need
to
be
treating
the
infrastructure
as
a
product.
Infrastructure
is
our
product
and
the
developers
are
our
customer.
So
it's
really
important
that
we
make
them
happy.
B
Scaffolding
templating,
so
how
do
you
release
new
features
to
to
developers
when
they,
when
you
have
implemented
them
with
logging
and
monitoring?
It's
quite
easy
comes
bundled
as
part
of
the
platform,
but
how
about
code
in
container
scanning
to
pipelines
that
are
being
managed
by
developers?
Well,
there's
a
couple
of
options
in
the
space
we
could
go
with,
having
all
the
features
ready
for
devs
before
they
even
start
using
the
platform.
Obviously,
that's
not
possible,
and-
and
in
one
of
the
key
things
about
successful
delivery
is,
is
having
fast
feedback.
B
Can
you
get
none
of
that?
If
you
have
everything
implemented
before
dev
start
using
it?
So
not
a
recommended
solution.
The
other
option
is
to
have
a
way
to
introduce
these
features
through
the
use
of
templates,
so
git
lab
has
and
include
remotes
that
we
that
we
could
use
in
this
particular
scenario
and
just
quickly
on
the
local
development
when
you're
moving
into
a
new
space.
B
It's
really
important
to
have
your
developers
and
both
your
dev
and
Ops
teams
get
really
comfortable
with
the
tooling
that
they'll
be
using
moving
forward
and
one
of
the
really
easy
ways
we
found
this
happens
is
through
having
a
having
the
same
tools
that
you're
using
in
production
mimicked
locally.
So
we
have
and
there's
no
single
tool
that
does
all
of
this
for
you.
So
we
you
walk
into
this.
With
a
couple
of
objectives
in
mind,
you
want
developers
to
be
able
to
build
push
and
deploy
containers
to
your
cluster,
whether
that's
local
or
remote.
B
You
want
developers
to
be
able
to
interactively
debug
those
containers
inside
running
pods
and
thirdly,
you
want
to
give
them
some
equivalent
services
which
they
can
use
to
mimic
what's
running
in
production
like
Kafka
or
or
databases.
So
the
way
we
did
that
one
of
the
ways
we've
we've
used
that
that
this
is
possible
is
through
the
use
of
a
couple
of
these
tools,
specifically
jib
scaffold
and
a
mini
cube,
and
combining
all
of
these
jib
gives
you,
the
Java
image.
B
Scaffold
continuously,
deploys
it
to
or
to
the
cluster,
and
the
cluster
is
provided
through
three
mini
cube
and
we
could
find
helm.
Charts
that
were
usable
through
Kafka
has
one
sorry.
Confluent
has
one
for
Kafka
and
bit.
Nami
has
one
for
Postgres,
so
combining
all
of
these
we
were
able
to
give
devs
a
really
rich
local
experience
that
they
could
test
their
ideas.
Out
of
him.
A
All
right
so
just
summary
summarizing
what
we've
been
talking
about.
The
four
things
we've
figured
out
are
really
important
to
focus
on.
If
you're
doing
this
sort
of
project
lay
the
foundation
try
and
get
as
much
as
reasonable
or
possible
into
place
before
you
start
placing
work
real
workloads
on
it,
do
this
off
off
the
developers
critical
path
as
much
as
you
can,
but
get
them
involved,
DevOps
inception
for
every
new
product
project,
try
and
make
sure
that
your
developer
and
operational
and
infrastructure
and
security
teams
are
all
aligned
and
working
towards
common
goals.
A
That
leads
to
shared
ownership
of
delivery,
which
is
again
around
common
goals,
make
sure
that
your
developers
are
comfortable
working
with
infrastructure
and
vice
versa,
big
focus
on
the
development
experience.
This
is
really
important.
The
more
developers
you
have
the
more
import
this
becomes
if
developers
have
any
pain
at
all,
even
if
they're
not
aware
of
it.
What
I
notice
that
it's
important
that
we
understand
that
we
can
find
opportunities
to
fix
things
if
we
sit
down
and
understand
the
developer,
experience
and
start
thinking
like
a
developer
all
right.
Thank
you
thanks.
A
Hope
you
found
this
useful
while
we've
got
a
roomful
of
Engineers
I
just
want
to
call
that
zip
is
growing.
We
want
to
move,
intend
to
keep
growing
if
you
want
to
join
the
get
on
the
train
check
out
our
careers
page
on
the
zip
code.
Find
me
on
LinkedIn
come
and
talk
to
me,
while
we're
here
find
a
way
to
get
in
the
door.
There's
plenty
of
opportunity
there
and.