►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Streamlining Kubernetes Application CI/CD with Bazel - Gregg Donovan, Etsy.com, Inc. & Chris Love, CNM Consulting
Creating a repeatable Kubernetes application deployment pipeline with the Bazel build system offers a number of advantages over traditional Kubernetes application CI/CD approaches. We will describe how Etsy uses Bazel, along with its container and Kubernetes support, to deploy many different services to multiple different clusters and environments. Using a repeatable build process maximizes container layer caching and takes advantage of Kubernetes built in pod template hashing. With this caching in place, it becomes feasible to deploy all services in a large repo reliably without under-deploying -- not deploying changed components, leaving them unvalidated on master -- or over-deploying, waiting for a potentially expensive rollout despite not having made changes. Bazel's rules_docker also allow you to build deterministic container images without the Docker daemon, simplifying CI/CD.
https://sched.co/MPYv
A
A
A
A
B
B
B
So
Etsy
is
a
large
international
and
rapidly
growing
marketplace.
We
have
nearly
40
million
buyers
and
over
2
million
sellers
around
the
world.
We
also
have
offices
around
the
world.
The
closest
one
to
us
today
is
our
engineering
office
in
Dublin,
and
we
are
constantly
trying
to
improve
the
product
to
keep
up
with
our
buyers
and
sellers
needs,
so
that
makes
development
velocity.
Our
number
one
concern
so
I
work
on
search
at
Etsy,
which
is
an
especially
fun
problem.
B
As
the
handcrafted
items
we
sell
are
each
unique
and
so
search
combines
large-scale
machine
learning
problems,
systems,
challenges,
real-world
impact
and
lots
of
interesting
products
to
think
about,
and
with
problems
of
this
scale
come
the
challenges
of
scale.
Among
those
are
the
challenges
of
scaling
out,
build
test
and
deployment.
We
have
thousands
of
tests
in
a
large
multi-language,
Monterrey
Bois,
complex,
stateful
applications
to
build
and
deploy,
and
we
have
dozens
of
engineers
working
on
our
stack,
who
need
to
rapidly
test
changes
and
validate
the
correctness
and
performance
of
those
changes.
B
A
And
why
don't
we
start
off
with
Blaise
and
I'll?
Take
that
clicker
from
you
clicker?
Thank
you.
So
why
don't
we
start
off
talking
about
a
bit
about
blaze?
Blaze
is
actually
the
predecessor
yeah.
You
know
that
slide.
Sorry,
somebody's
somebody's
already
giving
me
spunk
from
the
front
row.
So,
let's
talk
a
bit
about
place.
Blaze
is
actually
the
predecessor
to
basil
it's
an
internal
tool
within
Google.
It
had
to
build
it
because
of
running
into
slight
googly
problems.
A
Basil
is
now
the
open-source
version
of
blaze
and
they
had
to.
They
had
to
make
a
Google
had
to
make
their
own
build
tool
because
they
had
because
they
had
to,
they
have
a
very
large
code
base.
They
also
have
a
testing
or
a
standard
of
testing
within
Google.
You
have
to
test
your
code.
You
have
to
test
your
code
a
lot
and
then
you
have
to
test
somewhere
and
they
have
a
lot
of
code.
A
So,
let's
talk
Linux
kernel,
pretty
much
runs
the
internet
right
talking
about
two
hundred
thousand
lines
of
code.
I
think
we
know
this
project
all
right.
It's
the
largest
actually
open
source
project,
go
project
in
the
world.
We're
talking
right
about
two
million
lines
of
code
that
are
some
of
its
generated
talk
about
Windows,
it's
50
million
lines
of
code,
approximately
of
course
it's
closed
source,
and
then
we
have
Google
yeah,
I,
messed
up
a
scale.
Let
me
see
if
I
can
fix
that
nope.
Let
me
see
if
I
can
fix
that
again.
Yeah.
A
Just
look
at
that
number
for
a
second.
So
every
day
they
have
to
build
that
codebase.
They
build
it
multiple
times
a
day.
They
have
60
approximately
60,000
changes
to
their
codebase
a
day.
These
numbers
are
about
four
or
five
years
old
by
the
way
ever
released.
So
we
don't
want
to
talk
about
what
they're
at
now.
A
So
when
you're
talking
about
code
at
this
size,
you're
talking
about
big
code
when
you
have
a
lot
of
code
base
in
your
mono
repo
you're
talking
about
huge,
builds
and
then
we're
talking
about,
let's
test
it,
which
gets
really
fun,
so
you
might
be
going
right
now.
I,
do
not
have
that
many
lines
of
code
in
my
organization,
I,
understand
I,
don't
have
a
mono
repo
I
understand
Basil's,
not
just
if
you're
at
Google,
all
right,
let's
be
clear,
I,
don't
know
anybody
else
that
has
that
many
lines
of
code
we're.
A
So
we
all
use
build
tools
here.
I
took
a
quick
straw.
Man
pull
of
who's
really
happy
with
their
build
tools
and
I
got
chuckles.
You
know,
and
very
few
people
left
lifted
their
hands
who
who
here
can
guarantee
that
they
can
rebuild
a
commit
that
was
put
in
two
weeks
ago,
guarantee
yeah
I
like
that
who
here
has
ever
had
an
issue
with
rolling
back
where
they
couldn't
build
or
commit?
Oh,
come
on,
be
honest,
be
honest!
It's
okay,
we've
all
been
there.
A
So
one
of
the
key
things
about
basil
is
being
deterministic.
I'm,
not
gonna.
Go
into
this
big
long,
wonderful!
You
know,
you
know,
sentence
structure
about
determinism.
This
is
what
we
all
strive
for.
Our
built.
An
old
build,
can
be
rebuilt,
so
we've
always
had
you
know
we
install
the
new
language
library
on
our
build
box.
All
of
a
sudden.
We
try
to
do
an
old
build
and
there
goes
the
build
baby.
Basel
strives,
if
you
using
in
the
manner
it's
set
up
to
basel,
strives
to
have
reproduce,
ibill
and
builds
have
to
be
hermetic.
A
So
what
that
means
is
that
we've
got
system
libraries
on
our
bill
box.
We've
got
a
jdk
in
our
bill
box.
We've
got
python
on
our
bill
box.
We
have
Ruby
on
our
bill
box.
Your
bill
should
not
be
using
those
tools.
Your
bill
should
be
completely
encapsulated
in
its
own
little
bottle.
It
should
be
hermetic.
A
So
let's
talk
about
some
future
functionality
as
well.
Try
testing,
don't
repeat
yourself,
you
tested
it
and
you
didn't
change
that
code.
Why
are
you
testing
it
again
with
the
most
modern
unit,
testing
tools,
you
have
to
run
the
entire
test
suite
again,
even
if
you
didn't
change
any
code
around
the
around
it
parallelism
we're
talking
at
times
about
a
lot
of
code,
we're
talking
about
wanting
to
be
able
to
deploy
three
hundred
three
thousand
times
a
day.
A
In
order
to
do
that,
we
have
to
do
a
lot
of
builds
in
order
to
do
a
lot
of
builds.
The
builds
have
to
be
fast
with
parallelism
and
with
also
with
pooling
and
caching
you're
able
to
get
the
type
of
performance
you're.
Looking
for
as
well
as
as
a
developer,
who
likes
waiting
for
their
builds
to
be
done.
A
A
It
allows
you
to
have
a
pool
of
workers
in
the
cloud
locally
if
you
maintain
it
and
with
RBE,
which
is
a
Google
cloud
product
you're
able
to
plug
right
into
basil
and
fire
up
a
worker
pool
and
you're
able
to
not
only
have
the
horsepower
of
your
build
machine
but
you're
able
to
have
a
dynamic
cache
as
well
as
dynamic
horsepower.
Behind
your
build
from
that
process.
A
A
So,
let's
just
get
a
quick
background
of
what
basil
is
so
within
there
you've
got
a
workspace
file.
You
open
up
a
basil
project,
you
had
a
workspace
file
and
you
have
a
workspace.
It's
basically
the
root
component
of
your
directory
structure
within
the
workspace
file,
you're
able
to
define
lots
of
different
things.
Typically,
you
have
workspace
rules,
you'll
notice
that
we
have
we're
pulling
in
a
dependency.
We're
naming
our
project
there,
so
we're
pulling
in
go
you'll
notice
that
we're
also
pulling
in
the
SHA.
A
So
not
only
are
we
downloading
that
from
github
or
your
location
of
choice,
we
are
checking
that
it's
actually
the
download
that
you
should
be
getting
kind
of
important,
also
Bill
files.
So
most
directories
within
a
basil
project
have
a
build
file
which
is
named
Bill,
dot,
basil
or
bill,
depending
on
which
day
it
is
you'll
notice.
Within
here
we're
we're
pulling
in
we're
loading
components
that
are
actually
starl
art
components
pulling
in
what's
called
a
cave.
Okay,
it's
object.
A
We'll
talk
about
more
later
on
how
kubernetes
interacts
with
basil
we're
able
to
alias
those
objects.
So
if
you
notice,
we
pull
it
in,
it's
called
objects.
Basil
now
we're
gonna
call
that
Kade's
objects.
Then
we
define
some
stuff.
One
of
the
interesting
things
about
basil
is
the
path
notation
you'll
have
to
get
used
to
so
you're.
Looking
at
a
line,
this
is
actual
there.
It
says
forward,
slash
forward,
slash,
jeaious
clients
forward,
slash
source
blah
blah
blah.
So
what
it
is
is
a
file
structure.
Ford,
slash
means
the
directory
for
slash
forward.
A
C
A
B
B
Reproducible
builds
especially
for
container
layers
and
programmatic
control
over
the
generated
llamó
for
each
different
environment,
and
this
approach
really
shines
with
a
multi
application,
mono
repo,
when
you
change
shared
libraries,
deb
files,
tooling,
etc,
base
images,
you
need
to
know
which
components
need
to
be
redeployed
and
which
don't
so
in
a
simple
application.
You
might
be
able
to
keep
this
all
in
your
head,
but
at
scale.
You
need
automation,
so
I'm
going
to
talk
about
why
we
use
basil
rules
kubernetes
and
rules
docker
and
how
it
gives
us
fast.
B
Correct,
deploys
I'm,
also
gonna
offer
some
general
kubernetes
and
basil
tips
based
on
our
two
years
of
working
with
basil
and
two
years
of
working
with
kubernetes
and
doing
a
big
migration
to
gke.
So
when
we
started
using
basil,
one
of
the
best
features
we
noticed
is
that
Basil's,
reproducibility
and
caching
change
our
entire
local
development
flow.
Instead
of
working
on
your
code
and
then
re
running,
the
specific
tests
or
tests
that
you
think
is
affected
by
your
changes.
B
B
So
when
we
saw
this,
we
wondered
if
we'd
be
possible,
to
bring
similar
benefits
to
deployment.
Can
we
make
make
code
changes,
deploy
everything
and
then
let
basil
and
kubernetes
figure
out
what
actually
needs
to
be
redeployed,
so
this
is
sort
of
our
ideal
deployment
work
flow.
We
have,
you
know
dozens
of
deployments,
hundreds
of
kubernetes
objects,
we
don't
to
have
keep
in
our
heads,
which
one
of
them
has
actually
been
affected
by
a
given
commit.
B
After
all,
there
are
edge
cases.
You
might
have
a
shell
script
that
it's
depended
on
by
multiple
app
containers
but
transitively
by
a
sidecar
container.
It's
very
difficult
to
reason
about
this
stuff,
so
achieving
this
idealized
workflow
is
quite
challenging.
Now,
unless
we
have
a
reproducible
build
every
time
you
do
a
new
build,
it
will
appear
as
if
there
are
new
artifacts
to
deploy.
B
If
our
build
artifacts
have
changed
by
a
single
bit,
it
will
appear
as
they
are
new
and
they'll
be
need
to
be
rolled
out
so,
and
these
are
just
some
of
the
challenges
to
reproducibility.
For
example,
most
file
systems
do
not
guarantee
that
listing
files
in
a
directory
will
always
result
in
the
same
order.
Thar
and
jar
file
archives
will
by
default,
include
creation
timestamps
all
of
these
break
reproducibility,
but
basil
is
designed
to
handle
these
challenges
so
Basil's
rules.
B
Docker
is
a
set
of
basil
rules
for
pulling
down
base
images,
augmenting
them
with
build
artifacts
and
assets
and
building
container
layers.
It
also
offers
language
specific
rules
that
make
it
easy
to
go
from
a
binary,
say,
Java
or
go
main
into
a
container
image.
The
interesting
thing
is
that
you
do
not
need
docker
to
run
the
docker
rules,
so
if
you've
run
into
issues
with
docker
and
your
CI
RCD
environments
or
running
docker
within
a
docker
container
to
build
images.
This
removes
that
problem
from
the
equation
in
Basil's
rules.
B
Kubernetes
is
a
set
of
basil
rules
that
enable
deploying
templated,
llamó
and
container
images
to
do
kubernetes.
It
allows
you
to
do
authentication,
publish
containers,
publish
your
container
digests
container
gamal,
do
full,
basically
full
application
lifecycle
and
choose
your
own
templating
technology
Etsy.
We
use
custom
Python
code,
but
we
also
use
JSON
net
for
other
applications
and
you
could
use
Helen
if
you
like,
and
with
this
approach
we
avoid
two
problems.
B
You
know
under
deploying
not
to
point
components
that
have
changed,
leaving
them
unvalidated
on
master
and
over
deploy
waiting
for
a
potentially
expensive
rollout,
despite
having
not
made
changes,
so
we
can
radically
simplify
our
CI
HCD
flow.
We
can
have
a
single
pipeline
for
building
testing
and
redeploying
all
of
our
applications,
relying
on
basil
to
negotiate
what
actually
needs
to
be
deployed
with
the
container
registry
in
kubernetes.
B
So
how
does
this
actually
work?
So,
let's
dive
into
some
of
the
details?
How
does
this
negotiation
between
your
build
and
the
container
registry
and
kubernetes
actually
work
in
practice?
So,
fortunately,
kubernetes
is
smart
enough
to
only
deploy
what
you
change
and
it
does
this
by
hashing
its
inputs
much
like
basil,
so
these
hashes
include
both
the
llamo
and
the
exact
versions
of
the
containers
that
you're
deploying.
So
in
this
case,
you'd
see
spec
template
is
hashed
in
the
shot
256
they
stored
along
with
your
deployment.
B
So
kubernetes
knows
if
it
sees
that
same
shot,
256
of
spec
template
it
doesn't
need
to
to
do
a
rollout
of
your
deployment
and
taking
advantage
of
incremental
deployment
in
kubernetes
requires
that
you
can
deterministically
generate
all
of
your
inputs,
including
your
containers.
Otherwise,
every
time
you
do
a
build,
you'll
generate
a
new
container
version,
even
if
it
hasn't
changed
in
a
meaningful
way.
Rules
docker
provides
rules
for
generating
container
images.
Deterministically
allows
you
to
build
a
single
repeatable
package
from
base
container
to
your
own
application
code.
B
In
the
example
above
you
can
see
that
basil
has
helpfully
stripped
out
dates
from
our
containers,
and
it
seems
a
bit
odd
that
code
you
wrote
today
appears
to
be
from
1970,
but
it
is
very
helpful
to
get
repeatable
images.
Otherwise,
just
the
simple
change
of
dates
will
cause
kubernetes
to
redeploy
them
and
then
rules
kubernetes
builds
on
top
of
this.
It
takes
as
a
set
as
input
a
set
of
docker
images
that
you
want
to
build
and
what
are
their
there
basil
build
targets
for
generating
them,
and
it
takes
an
input.
Kubernetes
llamó.
B
We
have
placeholders
for
the
images
so
unresolved.
It
builds
your
containers
pushes
them
to.
The
container
registry
gets
the
shot
256
of
each
container
and
then
it
substitutes
those
shot.
256
is
for
the
placeholders
you
had
for
your
images,
and
so
the
the
final
thing
that
gets
sent
to
Coop's
ETL
ply
his
your
your
generated
gamal
with
those
sha-256
s
appended
to
each
of
your
container
images,
so
digging
even
deeper.
Just
what
is
a
container
so
container
is
an
ordered
list
of
tar
balls
representing
filesystem
layers.
Each
layer
itself
is
represented
as
a
shot.
B
256
of
the
uncompressed
contents,
the
overall
image
is
a
shot
256
of
the
layers
and
the
metadata
so
rules
docker
builds
these
layers
with
custom
tool
chains
written
in
Python
and
star
LARC,
a
Python
dialect
used
by
custom
basal
code
and
Basil's
rules.
Docker
removes
the
complexity
of
actually
running
docker
in
CI
CD
and
allows
Basil's
parallelism
and
caching
to
speed
container
layer
building
and
it
built
on
the
standard
container
registry
v2
API,
which
provides
caching
from
local,
build
to
the
remote
container
registry
and
rules.
B
Docker
uses
its
HTTP
head
request,
support
to
check
for
an
existing
sha-256
of
an
early
cached
container
image
or
the
shot
256
of
a
container
layer.
This
removes
the
need
to
re-upload
container
images
and
layers
that
are
already
present
in
the
remote
registry.
So
if
you
do
build
and
nothing
has
changed,
all
of
these
head
requests
are
going
to
see
an
existing
cache
layer.
The
existing
cached
image
and
it'll
be
basically
no
op
between
your
local
build
and
the
registry,
so
very
fast.
B
Finally,
kubernetes
deployment
hashes
the
pod
template
to
determine
if
your
application
needs
to
be
redeployed
and
if
the
pod
template
hash
is
unchanged,
there's
no
need
for
a
rollout,
and
so
with
this
you
have
the
benefit
of
basil's
reproducibility
from
your
source
at
the
bottom,
to
your
kubernetes
deployment,
sort
at
the
finish
line
and
rule
script,
Davies
offers
a
number
of
other
features
and
interesting.
One
that
we've
really
taken
advantage
of
is
the
ability
to
provide
your
own
coop
CTL
version.
There's
some
weird
edge
cases
here
and
there.
B
B
So
we
have
reusable
layers
for
things
like
instrumentation
libraries,
g-cloud,
GS,
util
other
reusable
things
we
might
want
to
add
to
application
containers,
but
not
have
to
layer
them
in
and
so,
rather
than
stuffing
everything
into
a
single
bloated
base
layer
we
can.
We
can
assemble
these
Lego
bricks
and
so
a
few
tips
from
our
sort
to
year,
migration
from
on-premise
infrastructure.
So
we
we
previously
ran
our
own
data
centers
and
ran
kubernetes
on
Prem
before
moving
things
to
gke,
and
so
one
one
thing
we
recommend
is
the
district's
container.
B
So
a
lot
of
a
lot
of
your
application
containers,
you
deployed
kubernetes.
They
don't
need
the
full
operating
system
so
Google
as
this
disrelish
project
that
provides
just
the
bare
minimum.
You
need
to
deploy
an
application.
So
given
your
your
main
method,
so
for
Java,
that's
G,
Lib
C,
SSL,
the
JVM,
it's
very
minimal,
so
you
get
these
nice
small,
nice,
small,
docker
images
with
a
very
limited
attack
surface
and
as
I
mentioned,
sha-256
is
or
I
think
a
best
practice.
B
So
even
if
you're,
not
using
rules,
kubernetes
or
rules,
docker
helm
supports
these
scaffold
supports
these
a
lot
of
the
different
communities.
Workflow
tools
support
them.
It
avoids
the
the
violation
of
immutability
you
get
if
you
use
a
tag,
so
a
tag
represents
a
potentially
moving
target
and
many
companies
usually
will
eventually
have
this
giant
wall
of
Yama.
So
for
the
complex
Yama
building
use
cases
we
have
at
Etsy,
we
use
put
custom
Python
code
and
the
client
API
is
very
nice
for
this.
B
If
you're,
if
you're
use
cases
are
simpler,
you
know,
customize
is
great.
There
was
a
good
tools,
but
don't
hesitate
to
use
the
kubernetes
client
API
is
if
you're
the
complexity
of
your
yanil
generation
is,
is
really
overwhelming
and
sort
of
pushing
past
the
limits
that
these
templating
tools
are
designed
to
deal
with
and
tilt
tilt
is
a
great
tool
that
we've
been
using
to
try
to
understand,
what's
happening
with
kubernetes
deployments.
B
They're
very
nice
I
recommend
you
go
talk
to
them
and
using
crts
for
cloud
resources,
so
kubernetes
custom
resource
definitions
are
very
powerful
to
share
a
lot
of
different
talks
about
these,
but
the
different
cloud
operators
are
adding
C
RDS
for
their
cloud
resources
that
will
allow
you
to
use
this.
Your
same
cube,
CTL
apply
for
things
like
GCS
buckets
or
s3
buckets,
and
you
know
we
invest
a
lot
of
time
in
these
kubernetes
CI
CD
pipelines.
B
A
Freddy
and
I
want
to
I
want
to
talk
about
one
thing
that
Etsy
does
that's
really
powerful.
You
know
that
one
of
the
main
complexities
that
were
running
into
his
developers
and
his
operators
of
kubernetes
is
ya,
know
it's.
It
can
get
just
to
be
a
mess.
You
know
400
yamo
files,
all
over
the
place.
If
you
take
it,
you
put
it
into
code
you're,
not
only
able
to
get
rid
of
some
of
that
complexity,
but
you're
also
able
to
test
it,
which
is
huge
all
right.
So
the
the
demo
is
available.
A
Open
source
I've
got
the
shortened
link
there.
If
you
want
to
take
a
picture
of
that,
it's
a
small
web
application
that
has
two
different
components
in
it.
First
is
angular
a
web
front-end
and
then
you've
got
a
Java
rest
service
for
one
of
the
angular
pages
or
tabs
or
whatever
you
call
it.
In
angular,
we've
got
multiple
languages,
so
we've
got
Java,
we've
got
typescript,
we've
got
JavaScript,
and
if
we
call
HTML
and
CSS
languages,
we
got
those
too
we're
doing
unit
testing,
so
we're
running
j
unit,
we're
also
using
protractor
and
Jazmyn.
A
A
Seven
five:
there
we
go
so
clean
command,
basically
cleans
out
the
cache
local
cache.
That
is
so.
We
now
have
as
soon
as
it's
done
here.
We
have
we're.
Gonna
have
nothing
on
disk,
that's
specific
to
our
build.
We
will
have
some
dependencies
on
disk.
Now
we
are
using
RBE
with
this
build,
so
we've
got
a
pool
of
workers
sitting
around
that
have
already
got
the
components
cached.
A
So
we're
looking
at
targets,
here's
the
big
number
and
it
keeps
going
up-
and
we
also
notice
that
we
have
a
bunch
of
stuff
happening
locally.
In
remote,
we've
got
tars
being
built,
we've
got
JavaScript
being
compiled,
we've
got
tests
being
run,
we've
got
all
kinds
of
stuff
happening
and
you'll
notice.
There's
forty
three
thousand
five
hundred
and
thirty
three
targets
in
this
build.
This
is
a
small
application,
has
no
js'
so
that
that
really
helps
us
we're
compiling
Cooper,
Nate
or
coop
control.
So
that
really
helps
us
as
well.
A
Get
us
a
couple
more
targets
going
on,
but
you'll
notice
that,
yes,
the
JavaScript
roll
up
does
take
a
while,
but
nothing
was
cached
locally
on
this
machine
and
you're
noticing
that
we're
even
pulling
in
remote
tarball
layers
as
greg
was
describing
earlier.
Eh
eh
containers
broken
up
into
very
stars,
so
we're
able
to
package
each
one
individually.
A
So
he
ran
four
of
our
unit
tests,
which
hey
they
all
passed,
and
we
looked
at
about
43
seconds
of
compile
time
and
51
51
seconds
of
compile
time
with
approximately
43,000
is
that
right,
number
yeah,
forty-three
thousand
different
objects
that
basil
touched
looked
up.
You
know
compared
the
hashes,
make
sure
that
nothing
has
changed
all
right.
Let's
make
a
test
and
break
stuff.
A
We've
got
a
controller
and
instead
of
saying
here,
I'm
gonna
say
haiku
con
all
right.
This
is
just
a
Java
controller
with
spring
boot.
All
right.
Let's
run
that
test
again
now
this
time
it's
gonna
run
a
bunch
faster,
because
we've
already
got
stuff
cached
locally
and
I
wrote
a
test
for
that
controller
and
unfortunately,
I
forgot
to
update
my
test
all
right.
So
let's
go
in.
C
A
All
right,
let's
run
the
test
again
make
sure
I
didn't
touch
anything
else.
You
notice
that
I've
got
a
bunch
of
different
command-line
arguments
on
here.
I've
got
what's
called
a
basel
RC
file
really
important.
It's
about
three
hundred
lines
of
different
configurations,
especially
with
RB
e,
comes
a
little
bit
complex,
but
fortunately,
we've
got
the
RB
folks
give
us
the
full
belt.
You
know,
90%
of
the
build
file
and
you'll
also
notice
that
I'm
defining
a
couple
of
things.
A
If
you
look
at
I'm
defining
a
cluster,
so
that's
actually
my
context
in
coop
control
and
I'm,
also
defining
the
container
registry
that
I'm
using
and
those
definitions
are
dynamic
at
built.
So
say
we
have
dev.
We
have
tests.
We
have
different
container
registries
that
we're
using.
We
can
define
those
at
the
time
of
the
build
so
that
we
have
containers
over
here.
We
have
clusters
over
there
and
we
made
everything
together.
So
tests
passed
great.
So
there's
some
irony
here:
Basil's
replacement
for
make
and
I'm
running
a
make
command
to.
A
Alright,
everybody
notices
the
coop
control
apply
commands,
but
what
you'll
notice
is
that
you
can
see
the
path
that
cout
control
is
actually
running.
We've
compiled
cout
control
from
kubernetes
source,
so
we're
guaranteed
that
I
have
kuben
control.
Thirteen
five
or
whatever
version
I
specified
in
my
workspace
file.
I
have
run
into
so
many
challenges
with
the
API
when
we
get
a
couple
versions
out
and
outside
the
standard
of
the
API
server.
A
Do
yourself
a
favor
even
if
you're,
not
using
basics,
basil,
maintain
the
version
of
coop
control
very,
very
strongly
in
your
organization,
so
you'll
notice
a
couple
things
here:
you'll
notice
that
the
service
that
we
applied,
so
our
manifests,
have
a
service
in
them.
That's
unchanged
will
notice
that,
though,
that
the
spring
boot
server
our
spring
boot
containers
changed.
So
we
modified
that
deployment
and
then
what
great
also
talked
about
is
you
know
it's
it's
really
getting
powerful
when
you
don't
have
to
redeploy
angular.
So
we
up,
we
updated
our
micro
service.
A
A
So
see
if
the
refresh
the
page
works
right
alright,
so
we
have
a
very
simple
application.
As
I
mentioned,
we
have
the
rest
controller.
That
goes
on
the
to-do
page.
So
we
can
add
a
two
here
say:
hi,
everyone
say
hi
to
everyone
had
the
to
do
very
simple
controller:
no
database
back-end,
it's
an
ArrayList,
that's
in
memory.
So
when
you
redeploy
your
deployment,
there
goes
our
to
dues,
but
so
we've
gone
from.
We
go
back
to
this
we've
gone
from
source
code.
A
We
define
our
source
code,
we
define
our
dependencies
in
basil,
we
define
our
tests.
We
then
test.
We
have
the
test
run.
Then
we
apply
the
manifests
and
in
between
all
of
that,
the
source
code
is
built.
The
binaries
are
built,
we
got
multiple
languages,
possibly
cross
builds
happening.
We
then
have
containers
and
everything
built.
We
have
manifests
that
are
being
modified
generated
and
then
it's
deployed
to
kubernetes
and
I.
Did
it
by
doing
this?
A
C
A
C
B
That's
a
that's
a
very
good
question,
so
the
question
is
so
we're
building
our
own
coop
CTL
from
source.
Do
we
also,
then
compile
the
compiler
from
source
and
the
compilers
toolchain
from
source,
etc,
etc?
So
within
Google
I
think
the
answer
is
yes:
everything
gets
compiled
from
source
at
Etsy.
We
cut
it
off
at
the
dependency
at
coop,
CTL
and
then
I
think
we
used
up
the
binary
of
the
the
go
compiler,
but
it's
a
it's.
A
Long
as
your
source
dependencies
have
basil
laid
out
already
you're
able
to
then
include
them
and
then
with
the
proper
chip
tool.
Change,
you're
you're
able
to,
for
instance,
run
GCC
a
compiled
version
of
GCC,
and
if
that
changes,
and
then
basil
knows
that
the
dependency
tree-
and
it
goes
all
the
way
back
and
every
project
that
depends
on
that
version
of
GCC
is
then
recompile
and
retest
it.
It's
pretty
powerful.
Other
questions.
D
A
A
It
doesn't
support
Canaries
out
of
the
box,
but
what
it
does
support
is
star
lark,
which
is
basically
Python
ish.
So
it
gives
you
a
capability
of
doing
programmatic
deployments.
Where
you
have
dependencies,
you
have
weight
States,
you
have
okay,
this
has
to
happen.
First,
you
have
dependency
management.
We've
actually
worked
with
at
C
in
terms
of
tuning
their
elasticsearch
deployment,
so
that
we
have
stuff
like
rollovers.
Failover
upgrades
are
all
coming
actually
out
of
Basel,
so
you're
able
to
chain
stuff
together.
No,
it
takes
some
programming
knowledge.
A
B
We
so
we
an
FC
as
Chris
said
we,
we
use
a
lot
of
star
lock
code,
so
star
lock.
Is
this
this
Python
like
dialect
that
you
can
use
to
do
custom
things
with
your
build
and
for
something
like
Canaries?
We
don't
use
like
in
place.
Canaries
like
multiple
deployments
in
a
namespace.
We
use
up
just
a
lot
of
namespaces
and
then
having
traffic
go
between
different
namespaces.
So
we
have
a
single
place
where
we
define
all
of
the
different
applications,
all
of
the
different
clusters.
B
We
use
multiple
clusters
and
all
of
our
namespaces,
and
then
we
generate
communities,
objects
for
sort
of
to
come
to
set
of
those
the
power
set
of
all
of
those
things,
and
so
we
might.
We
might
test
different
different
container
versions
in
different
namespaces.
Different
branch
shows
things
like
that.
So
one
question
right
there
in
the
middle.