►
From YouTube: Release Engineering as a Service - The Art of Integrating with K8s - Maryam Labib & Ramya Shenoy
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Release Engineering as a Service - The Art of Integrating with K8s - Maryam Labib & Ramya Shenoy, Pivotal
We're missing opportunities to improve and optimize the software integration process. This lack of focus can decrease code quality, mask incompatibility between components, and frustrate users that seek reliable release cadence. In this talk, two creative engineers will tell the story of how their team built ‘Release Engineering as a Service’ in order to tackle slow release cycles and endless bug reports. The aim of the talk is to help open source contributors understand how small changes to their integration process can have an enormous impact on how fast their favorite features get delivered. Maryam and Ramya will discuss: - What is a release engineering team in the context of Kubernetes? - How should we use CI/CD to integrate software consistently? - How can we improve feedback cycles using automation, parallelization, and reutilization?
https://sched.co/NrmU
A
So
what
is
release
engineering
in
our
context,
so
our
product
is
made
up
of
many
different
teams,
and
each
of
these
teams
contributes
specific
parts
of
the
code
to
us
and
then
the
release
engineering
team
takes
the
scored
and
runs
them
through
pipelines
which
are
called
test.
Suites
also
called
a
stress
test
Suites
and
then
base
them
into
a
product
and
serves
it
to
the
customer.
A
What
does
this
mean
in
terms
of
kubernetes?
We
are
talking
about
different
integrations
like
harbor,
at
CD,
&,
Flu,
nd,
etc.
So
teams
integrate
with
these
and
share
code
with
us,
and
then
the
rail
and
team
runs
them
through
pipelines
or
test
Suites,
an
example
of
which
is
on
the
slide
there
to
create
a
product
called
pivotal
container
service,
and
then
that's
the
one
that
we
shipped
to
customers.
A
Few
things
before
we
proceed.
What
is
concourse
concourse
is
a
open
source
tool.
It's
a
way
in
which
you
can
write
tests
into
different
jobs.
You
can
combine
them
into
different
jobs
and
then
create
a
pipeline.
This
is
an
example
of
a
pipeline
and
you
can
have
different
jobs
sequenced
one
after
another,
and
you
can
have
light
resources,
resource
types
and
inputs
and
outputs
to
this
job.
So
it's
just
a
way
of
having
your
test
suite.
It's
anonymous
to
that,
and
it's
a
pretty
cool
tool.
A
You
should
check
it
out
at
Concord,
CI
dot
all
will
be
referencing
this
frequently
through
our
talk
few
more
terminologies
that
we
will
be
using
interchangeably
in
our
talk.
When
we
save
fly,
we
mean
running
a
pipeline
or
running
a
test.
Suite
a
pipeline
is
equal
into
a
test.
Suite
and
a
lock
implies
an
environment
or
a
test.
Bed
and
a
pool
is
a
group
of
environments
which
are
configured
the
same.
B
B
So
naturally
we
asked
ourselves.
Why
is
this
the
state
that
we're
in
right
now?
Well,
one
of
the
things
is
that
all
of
the
scripts
and
things
that
we
were
using
had
worked
at
a
certain
point,
but
the
teams
that
were
integrating
with
kubernetes
have
been
growing
and
we've
been
supporting
more
and
more
integrations.
The
size
of
this
rel
inch
team
itself
had
stayed
the
same,
so
we
no
longer
had
enough
people
to
keep
up
with
all
of
the
work,
and
we
had
done
a
lot
of
things
super
like
duct
tape.
B
So
now
we
had
a
lot
of
technical
debt
that
we
had
accrued,
and
this
was
resulting
in
us
releasing
slowly,
and
so
our
customers
were
unhappy.
We
broke
it
down
into
these
particular
issues
that
we
had.
We
had
written
a
lot
of
scripts
that
were
not
being
tested.
We
had
a
lot
of
llamó
that
was
duplicated
everywhere.
Our
environments
that
were
being
used
for
tests
were
in
high
demand
and
we
didn't
have
enough
of
them.
Our
pipelines
were
taking
way
too
long.
A
So
the
first
thing
that
we
want
to
talk
about
is
no
testing
one
of
the
problems
that
we
had.
So
why
didn't?
We
have
any
testing
in
place
when
we
look
closely,
we
found
that
we
had
a
lot
of
batch
scripts
written
in
order
to
do
our
testing
and
bash
is
really
hard
to
you
know,
test
drive
or
even
just
to
write
tests
even
after
you
write
the
script,
it's
also
intermixed
with
Linux
commands
here
and
there.
So
it's
hard
to
notice
bugs
it's
hard
in
general
to
just
program
in
in
bash.
A
So
we
decided
to
tackle
that
first
and
as
a
team,
we
decided
to
change
it
into
programming
languages.
So
what
could
we
go
for
it?
The
two
options
were
either
Ruby
Ruby
and
go
lang,
and
the
reasons
for
these
were
because
the
team
prior
experience
in
these
areas.
There
are
a
lot
of
Ruby
experience
in
the
team
and
there
were
a
lot
of
go
line.
Experience
in
the
team
also
Ruby
is
more
testable
or
test-driven,
and
golang
is
also
useful
for
concurrency
and
parallelism,
which
we
could
really
make
use
of.
A
So
here
is
an
example
of
how
our
scripts
looked
before.
For
example,
we
had
an
environment
metadata
file
and
let's
say
we
had
to
connect
to
this
environment
right,
so
we
had
all
of
these
properties
in
maybe
like
Yammer,
and
we
had
to
read
through
it
and
find
out
the
proxy
and
the
username
and
password
in
order
to
connect
to
it,
and
this
is
how
we
were
writing
it
in
bash
and,
as
you
can
see,
there's
like
yq--
or
awk
I
said,
and
there
are
other
commands
that
we
were
using
interchangeably.
A
This,
as
you
can
see,
is
untestable
because
we
have
to
test.
We
have
a
marker,
a
lot
of
things
for
these,
so
then
we
decided
to
do
bash
in
Ruby,
and
this
was
an
intermediate
step.
So
what
we
were
doing
was
reading
the
environment
metadata
by
opening
files
reading
it
into
a
hash
and
then
getting
properties
out
of
it
doing
whatever
processing
we
wanted
to
do
and
then
closing
the
file-
and
this
was
happening
pretty
much
everywhere
in
wherever
we
were
referencing
the
environment
for
testing.
A
This
is
somewhat
testable,
like
you
can
still
mark
out
a
few
things
and
get
away
with
it,
but
not
completely
testable
like,
for
example,
there's
like
splitting
on
colon
and
then
expecting
that
the
first
array
of
an
item
is
going
to
be
present,
etc.
So
it's
quite
error-prone,
so
we
decided
to
completely
change
it.
A
ruby
and
what
we
mean
by
this
is
try
to
think
of
the
the
things
that
you
manipulate
in
your
test,
Suites
as
objects
that
you
pass
around.
A
For
example,
the
environment
environment
metadata
could
actually
be
a
lock
file
class,
which
hide
properties
that
you
can
fill
in,
and
this
is
super
testable
in
Ruby.
So
we
try
to
think
of
it
as
in
an
abstract
way
and
started
creating
classes
for
log
files
and
pipelines,
and
it
was
easier
this
way
to
test
driver.
B
So
why
is
duplicate
llamo
a
problem,
so
we
were
getting
a
lot
of
llamo
from
kubernetes
configuration
in
general
and
our
CI
CD
tool
that
we
were
using
Concours
is
completely
configured
by
llamo.
So
we
were
dealing
with
a
lot
of
llamo
that
looked
very
similar
and
was
slightly
different
and
the
process
of
looking
at
a
bunch
of
files
and
trying
to
figure
out
what
was
different
between
these
files
that
created
a
lot
of
cognitive
overhead.
The
second
part
is
that
we
were
as
part
of
our
process.
B
We
were
copying
and
pasting
a
lot
of
files,
and
then
we
would
just
change
one
or
two
keys,
and
sometimes
our
team,
which
is
human,
like
forgot,
to
go
back
and
change
these
keys.
So
we
had
like
mistakes
like
that.
The
team
was
very
unhappy
anytime,
we
needed
to
add
pipelines,
modify
pipelines
to
do
anything.
Everyone
got
very
sad
about
it
and
it's
always
important
to
try
to
have
happy
tubs.
B
B
B
But
after
this
we
decided
to
use
ARB,
so
er
B
is
basically
a
way
of
using
Ruby
in
yellow.
So
this
is
an
example
where,
for
as
we
were
doing
testing-
and
we
were
testing
kubernetes
against
all
these
different
infrastructures,
and
so,
instead
of
just
repeating
AWS
AWS
AWS
gcpd,
CP
DCP,
we
now
can
just
use
like
these
four
loops,
and
so
that
was
really
helpful
and
it
really
dried
up
our
mo.
B
So
we
decided
we
wanted
to
try
something
else,
and
this
time
we
went
with
anchors
and
aliases.
So
if
any
of
you
are
familiar
with,
the
MMO
itself
has
this
notion
of
an
an
anchor
and
when
you
define
an
anchor
you
can
just
reference
in
other
places,
so
we
would
start
having
one
resource
type
defined
in
one
place
and
then
we
would
just
use
it
in
a
lot
of
other
pipelines.
B
So
this
did
a
really
good
job
of
drying
up,
but
we
noticed
that
it
wasn't
taking
care
of
the
cognitive
overhead.
We
were
still
spending
a
lot
of
time,
opening
up
a
lot
of
different
files
to
figure
out
what
we
were
actually
including
in
the
pipeline
and
specifically
for
us.
We
were
seeing
that
certain
parts
depended
on
other
parts,
but
we
couldn't
even
tell
that
they
used
each
other,
because
now
we
dried
up
everything
so
much
that
we
couldn't
tell
what
was
even
failing
or
why
was
failing.
B
We
decided
for
attempt
number
three
that
we
were
just
looking
at
it
the
wrong
way.
We
were
splitting
this
up
based
on
how
it
was
configured
in
Concours,
and
what
we
should
do
instead
is
just
look
at
how
our
pipelines
are
split
up.
We
noticed
that
there's
a
lot
of
parts
of
our
pipeline
where,
like
at
the
top,
we
would
dry
up
something
and
that
thing
was
referencing
something
at
the
bottom
and
in
this
case
resource
type.
B
That's
being
used
by
a
job,
so
we
were
looking
closely
at
our
pipelines
and
sometimes
we
would
have
a
configuration
like
this.
We
had
the
infrastructure
layer,
so
vSphere
6/7
in
this
case,
and
then
we
had
the
networking
layer
which
is
NSX,
and
then
we
had
the
version
of
whatever
software
we're
testing
and
then
the
scenarios
like
in
this
case
an
install
and
we
decided
to
have
these
templates,
these
templated
files
and
we're
going
to
just
have
all
of
these
dependencies
in
one
file
and
then,
when
you
actually
come
to
make
a
new
pipeline.
A
Ok,
so
the
next
problem
we
had
was
environment
contention
now
to
explain
what
we
mean
by
environment
contention.
We
as
a
relaunch
team,
we're
testing
on
many
different
guises.
Some
were
public
like
AWS,
Azure
and
GC
P,
and
then
we
also
had
like
a
private
ones
that
we
make
sure
to
test
on
like
we
sphere
and
then
NSX
pipelines.
A
Nimbus
here
is
an
internal
tool
that
VMware
uses
to
spin
up
NSX
environments.
So
basically
we
cover
all
of
this
in
our
testing.
So
why
is
this
a
problem?
If
you
don't
have
these
environments
around?
First
of
all,
for
public
I
asses
like
we
had
many
different
pipelines,
but
the
number
of
environments
that
we
had
was
very
little,
so
the
ratio
was
really
skewed.
A
We
had
50
plus
testing
configurations
to
test,
but
there
was
no
automation
around
environments,
so
we
were
basically
going
to
the
console
and
then
by
creating
them
manually
and
like
making
all
these
configurations
and
forgetting
some
parts
of
it
for
privatizes
there
is.
There
were
other
issues
like
Nimbus,
like
I
explained
before,
is
based
on
a
quota
system.
So
whoever
requests
for
an
environment
first
gets
that
environment
and,
let's
say
the
quota
for
an
entire
organizations
over.
A
You
wouldn't
be
able
to
request
for
another
one,
so
you
just
have
to
wait
till
someone
cleans
it
up
and
then
you
have
access
to
it.
So
basically
we
were
really
sad
again.
This
was
causing
a
lot
of
human
errors
and
configuration
errors
and
made
us
sad
like
the
dog
of
the
picture.
So
how
do
we
fix
it?
A
We
used
terraform,
which
is
an
open-source
tool
which
can
create
environments
on
the
fly,
so
we
started
having
a
pipeline
for
creating
an
environment
and
destroying
an
environment,
and
this
was
third
eye
as
so,
it
was
like
a
one-click
solution
and
we
just
automated
it.
So
we
never
had
a
bother
about
like
creating
it
again
unless
just
going
to
a
pipeline
and
clicking
on
a
button.
Also
the
pipeline's
were
recycled.
Sorry,
the
environments
were
recycled
after
24
hours
if
they
weren't
used
for
the
private
I
asses.
That
is
Nimbus.
A
Specifically,
our
infrastructure
team
from
VMware
came
up
with
this
tool
called
Shepherd,
which
pre-made
environments
and
then
preserved
it
for
the
Vilanch
team,
and
we
basically
started
having
a
shared
pool
for
all
the
teams
that
were
going
to
test
on
vSphere
NSX.
So
this
way
we
had
on-demand
environments
and
lesser
contention,
because
now
we
were
at
least
we
were
high
priority.
B
So
we
were
watching
a
lot
of
our
deadlines
pass
us
by
and
we
were
missing
them
and
were
like.
Why
are
we
always
missing
our
deadlines
and
also
I
totally
screwed
up
the
formatting?
And
all
of
these?
It's
fine.
You
didn't
notice,
you
didn't
notice,
so
we
looked
at
pipelines
that
are
time
consuming
and
we
saw
that
they're
time
consuming
for
two
reasons.
The
first
is
that
they're
really
slow,
and
the
second
is
that
a
lot
of
them
were
really
flaky,
so
we
wanted
to
look
at
the
ones
that
were
slow
first.
B
Another
thing
that
we
wanted
to
tackle
is
that
in
a
lot
of
our
test,
Suites,
we
were
spinning
up
a
cluster
and
then
we
were
running
tests
against
that
workload
and
then
we
were
destroying
the
cluster
and
then
this
was
happening
also
in
serial.
So
we
decided-
let's
just
have
one
job,
create
a
cluster
and
passed
the
information
for
that
cluster
to
all
the
jobs
that
need
it
and
have
those
jobs
run
in
parallel.
So
this
this
resulted
in
our
pipelines
being
sped
up
by
four
times.
B
The
next
thing
is
we
wanted
to
look
at
our
pipelines
that
were
flaky,
so
we
knew
that
this
was
the
flow
that
we
were
experiencing.
That
was
causing
us
pain.
We
would
have
a
PR
get
merged
and
we
would
build
the
product
and
then
we
would
try
to
run
it
through
almost
39
pipelines
against
different
infrastructures,
and
we
would
have
to
trigger
this
many
many
times
same
exact
inputs
until
it
went
green
and
we
could
ship
it.
B
What
we
really
wanted
is
to
be
able
to
merge,
it
run
it
through
all
the
pipelines
once
and
then
see
it
go
green.
So
one
person
on
our
team
was
like
licious,
run
a
pipeline.
Let's
just
call
it
run
run
run
and
this
pipelines
gonna
do
exactly
what
it
sounds
like
it's
just
going
to
run
over
and
over
again
with
the
same
commits
and
we're
just
going
to
keep
this
pipeline
around
and
use
it
to
just
get
a
sense
of
how
flaky
certain
jobs
are.
B
So
we
did
112
runs
took
about
four
weeks
and
we
realized
that
our
pipelines
were
succeeding.
Fifty-Five
percent
of
the
time
so
after
some
very
complicated
analysis
is
very
small.
Here
we
actually
realized
that
there
were
certain
jobs
that
were
failing
a
lot,
certain
jobs
that
weren't
failing
as
much,
and
we
decided
to
prioritize
these
jobs
by
failure
rate.
B
This
was
actually
really
successful
because
we
discovered
a
lot
of
random
errors
that
really
hadn't
been
looking
into
apparently
running
out
of
disk.
Apparently
sometimes
we
were
retrying
and
it
was
failing
and
by
being
able
to
really
look
for
these
were
able
to
look
for
those
specific
teams
that
were
in
charge
of
wherever
that
integration
was
failing
and
they
were
able
to
fix
these
things.
And
this
made
a
big
difference
to
her
speed
overall.
A
So
the
next
problem
we
had
was
no
integration
process.
So
at
this
point
we
had
all
these
pipelines
all
the
teams
had
access
to
these
pipelines
and
we
had
no
idea
who
was
triggering
what
pipeline
at
what
time.
So
this
was
the
like
really
initially
before
we
had
any
processes
in
place.
We
would
come
to
work
in
the
morning
one
day
and
find
pipelines
past
or
pipelines
run
our
pipelines
deleted
and
also
teams
had
no
way
to
test
their
code.
A
They
were
pushing
changes
to
master
directly
and
actually
debugging
after
shipping
it
because
they
had
no
way
of
knowing
where
their
tests
were
failing,
or
they
had
no
way
to
run
their
tests
in
the
first
place.
So
we
could
fix
some
of
these
issues
like
restrict
our
concourse
access.
Like
we
created,
we
made
it
more
private
and
only
the
rail
ang
team
could
access
it
and
we
started
introducing
code
reviews
and
PR
processes,
but
it
still
doesn't
fix
the
issue
of
the
teams
not
having
a
way
to
test
it.
A
A
So
this
is
where
Relan
as
a
service
comes
in.
So
basically
what
the
rail
ang
team
did
was
created
an
exact
same
concourse
that
was
available
to
the
teams
and
each
of
them
had
their
own
separate,
buckets
and
own
separate
places
to
host
their
code.
So
basically
what
they
would
do
was
they
had
access
to
our
code,
so
they
would
run
those
pipelines
in
the
separate
concourse
and
they
could
also
run
only
what
was
relevant
to
them
like
let's
say
they
wanted
to
test
only
vSphere
and
AWS.
A
They
had
access
to
the
amyl
files
and
they
all
they
had
to
do
was
run
it
in
this
concours
and
test
it
out.
So
this
way
we
just
had
a
script
which
made
this
configuration.
The
teams
were
changed,
the
location
of
their
buckets
and
their
repositories
run.
The
pipeline
in
the
separate
Raskin
course
fail
and
see.
If
it
fails,
if
any
job
failed,
they
would
fix
it
and
then
PR
it
to
us,
and
then
we
would.
We
would
have
it
green
in
our
production
concours.
So
this
way
both
sides
had
more
confidence.
A
The
teams
had
more
confidence
because
they
tested
their
code
and
we
had
more
confidence
because
we
know
that
they
had
a
way
to
test
it.
The
last
part
is
open
source
licensing.
So
at
this
point
we
are,
we
have
pretty
much
sped
up
most
of
our
processes,
but
there's
still
one
last
hiccup.
So
whatever
we
are
shipping
is
this
legal
and
one
of
the
biggest
problems
of
shipping
open
source
software
is
knowing
whether
it's
legal
or
not,
whether
it
have
you
acknowledged
for
every
open
source
software
that
you
have
included
so
who's
responsible
for
this.
A
At
this
point
it
was
the
rel
inch
team.
So
what
does
this
mean
when
we're
talking
about
open
source
licensing?
And
why
is
it
hard?
We
were
talking
about
kubernetes,
especially
we're
talking
about
containers
and
containers,
are
made
up
of
software
packages.
They
are
made
up
of
base
OSS
and
it's
pretty
early
right
now
for
us
to
know
what
is
included
or
how.
How
do
we
scan
these
containers
is
hard.
A
Containers
can
either
be
on
the
internet
or
they
can
either
be
created
by
the
teams
themselves
and
we
need
to
know
what's
included
in
each
of
those
containers,
so
we
started
scanning
them
by
ourselves.
We
take
each
of
the
we
take
each
of
the
component
teams.
Call
we
scan
them.
We
wrote
scripts
to
scan
these
containers
and
find
what
software
was
included
in
that
find.
A
software
packages
find
their
licensing
through
open
source
tools
and
then
create
OSL.
A
We
also
became
smarter
on
what
to
OSL
and
what
not.
For
example,
we
started
asking
the
teams
to
do
their
own
part
for
the
OSL,
because
the
teams
know
what
containers
they
include.
We
also
did
OSL
only
for
GA
bills
and
not
for
only
for
general
availability
bills
or
not
for
all
dev
bills
that
we
were
like
producing.
We
also
started
doing
this
much
earlier
in
the
process
when
we
knew
that
something
was
a
candidate
for
a
build
and
also
started
providing
teams
with
early
access
to
the
bills.
B
B
At
the
beginning
of
this,
when
the
team
had
started,
we
were
shipping
every
six
months,
and
now
we
were
shipping
and
you
build
every
10
days.
Our
pipelines
went
from
12
hours
to
four
hours,
a
large
part
of
that
was
parallelization
and
we
were
able
to
even
have
a
turnaround
for
bugs
see
bees
patches
of
48
hours.
A
I
release
time
for
one
of
our
builds,
one
300
was
35
hours
from
start
to
finish.
That
was
from
testing
to
OSL.
We
got
it
down
for
release
1
for
0
to
about
10
hours.
Now
the
OSL
is
not
included
in
that
word,
but
yeah
OSL,
actually
always
is
not
OSL
is
not
included.
Is
this
is
just
testing
part
I'm?
So
sorry,
it.
B
A
So
we
develop
some
scripts
internally,
we
used
bash,
but
it's
actually
a
really
hard
process.
We
did
not
find
any
tools
that
are
reliable
in
the
market,
so
we
had
to
develop
it
in-house,
but
like
the
basic
idea,
it's
it's
it's
little
manual
at
the
moment,
but
at
least
mostly
roll
it
in
bash
yeah.
We
didn't.
We
be
used
for
actually
scanning
the
rest
of
the
product.
We
used
for
Saleh
G
and
forget
the
other
one
yeah
license
finder
and
pathology,
but
they
don't
do
container
scanning
for
containers.
We
did
it
ourselves.
Okay,.
D
B
So
we
have
a
github
repo
that
has
the
actual
lock
files
in
them.
The
credentials
are
stored
elsewhere,
but
it
basically
the
act
of
claiming
an
environment.
You
actually
move
it
from
one
folder
in
the
github
repo
to
another
folder.
So
we
look
at
the
time
that
it
was
claimed
that
it
was
basically
moved
to
that
folder
and
that's
the
time
that
it
was
claimed.
So
we
have
a
pipeline
that
runs
every
hour.
B
It
looks
at
all
of
the
locks
that
were
claimed
from
that
repo
within
the
past,
like
24
hours
or
more,
and
then
it
moves
them
into
the
destroy
pool
so
like
every
we
basically
use
github
and
like
we
look
at
commits
everything's
like
automated
through
commits.
So
that's
how
we
figure
out
how
long
in
environments
being
used
great.
D
E
I
mean,
since
you
project
rely
on
some
sort
of
party
dependencies
either
Java
also
IPM
fails
or
some
other
fails.
If
you,
if
your
project,
has
lots
of
micro
services
what
projects
all
of
these
stories
should
rely
on
some
sort
of
party
dependencies,
then
how
do
you
project
all
these
project
managed
sort
party
dependencies.
B
B
It's
a
good
question,
so,
okay,
so
what
we
do
on
our
team
is
every
time
we
make
a
change
to
any
integration
in
kubernetes.
What
we
do
is
we
build
our
product
with
that
change
and
we
have
to
get
an
environment
that's
configured
with
this
configuration,
so
what
you're
looking
at
here
right
now?
These
are
all
configurations
and
for
vSphere
like
for
VC
or
6/7,
for
example,
like
we
actually
have
our
own,
like
set
of
servers
locally,
that
we're
using
to
test
for
nsx
we're
actually
like
deploying
that
networking
layer
on
top.
B
We
do
this
for
public
infrastructure
as
well
like
for
GCP,
for
example,
but
with
GCP
we
would
just
do
flannel
for
the
networking
layer
and
the
OEM.
That
part
is
sort
of
a
that's
sort
of
a
product.
That's
like
we're.
Also
testing
against
and
the
scenarios
are
just
install,
is
like
we're
just
trying
to
see
that
we
can
get
it
deployed,
get
a
cluster,
get
it
to
pass
against
tests
for
upgrades,
we're
just
trying
to
make
sure
that
we
can
go
from
one
version
to
the
version
after
it.
B
B
Okay,
like
just
for
these
environments,
I
just
wanted
the
ones
on
the
right
hand,
side.
Those
are
all
public
infrastructures,
so
we're
using
like
terraform
to
be
able
to
deploy
everything
on
those
environments
for
the
left
hand,
side
we
have
like
internal
tooling
and
we're
using
that
to
be
able
to
specify
like
which
configure
like
which
specific
in
integrations.
We
want
in
it.
If
we
like
have
I,
don't
know
like
Viera
lie
or
harbor
or
something
then
we're
specifying
it
that
way,
but
I
guess
a
lot
of
these
are
right.