►
From YouTube: One Chart to Rule Them All: Continuous Deployment with Helm at Ticketmaster - Michael Goodness
Description
One Chart to Rule Them All: Continuous Deployment with Helm at Ticketmaster - Michael Goodness & Raphael Deem, Ticketmaster
As Kubernetes continues to mature, it's increasingly hard for users to keep track of the latest resource types, much less the best way to employ them. ReplicationControllers and Services were easy enough. Then came Deployments and Ingresses. Now we have PodDisruptionBudgets, ClusterRoleBindings, and HorizontalPodAutoscalers. Luckily, we also have Helm to package and deploy these various components (and more) as a single unit.
In this talk we'll dissect the single, flexible Helm chart Ticketmaster developed for use by multiple product teams. We'll show how we use just a handful of variables to enable log collection with Fluentd, metric scraping with Prometheus, and automatic scaling of pods. Then we'll demonstrate the GitLab CI workflow through which we deploy multiple builds of an application to multiple Kubernetes clusters running both on-prem and in AWS.
About Raphael Deem
Raphael is a Systems Engineer at Ticketmaster and open source contributor. He is a relative newcomer to the community, having started working with Kubernetes within the last six months. Prior to working at Ticketmaster, he was a remote engineer for Platform.sh, a Paris-based PaaS built with Python and LXC containers running on AWS. He was (briefly) a high school math teacher, and is a casual juggler.
About Michael Goodness
Mike is a Lead Systems Engineer on the Kubernauts team at Ticketmaster and a CNCF Ambassador. He began working with Kubernetes in late 2015, and quickly became an avid member of the community. While primarily involved with day-to-day cluster operations, he is also keenly interested in helping developers use cloud native technologies to deploy scalable, reliable applications. He is a co-maintainer of the Helm community charts repository and has contributed several popular charts. Hailing from Wisconsin, Mike is a co-organizer of DevOps Days Madison, a dedicated Milwaukee Brewers fan, and a cheese curd aficionado.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
Secondly,
I
want
to
apologize
again
to
everyone.
Who's
watching
and
I
was
hoping
to
see
some
sweet
Lord
of
the
Rings
memes.
Given
the
title
of
the
talk,
you're
gonna
be
sorely
disappointed,
although
I
could
say
a
Ticketmaster
one
does
not
simply
cube
cuddle,
create
a
little
background.
I
have
a
couple
of
years
of
production
experience
with
kubernetes,
both
at
Ticketmaster
and
at
my
previous
company
I'm,
a
Helmut
arts
contributor
and
Co
maintainer
recently
became
a
CNC
F
ambassador
and
I
co
organised
the
DevOps
days,
Madison
conference.
A
The
the
code
there
is,
you
know,
be
gentle.
This
is
what
we're
going
to
talk
about.
I
gonna
keep
it.
You
know,
kind
of
nice
and
tight.
Hopefully
we
are
going
to
kind
of
assume
some
familiar.
Some
familiarity
with
helm.
I
will
cover
a
few
of
the
basics,
but
you
know
I'm
also
going
to
be
picking
apart
or
kind
of
dissecting
a
few
sections
from
our
our
web
service
chart
and
it's
going
to
assume
some
knowledge
of
how
helm
works
and
kubernetes
manifests,
but
I.
A
A
A
So
you
take,
you
know
all
of
those
all
of
it.
All
of
that
ya
know
file
that
goes
into
building
an
application,
deploying
an
application
to
kubernetes.
You
multiply
that
by
you
know
the
number
of
clusters
that
you
may
have
in
your
environment,
you
know,
and
and
each
one
has
a
certain.
You
know
certain
configuration
points
that
you
know
that
are
different
than
the
others.
A
You
need
to
deploy
different
versions
of
the
same
application
and
each
you
know
some
of
them
have
I
mean
at
the
very
least,
your
docker
image
tag
is
going
to
be
different.
So
that's
you
know,
that's
a
different
configuration
point.
You
know
the
problem
with
kubernetes
as
it
is
today
is
there's
no
real
way
to
bundle
all
of
those
applicable
all
of
these
resources.
All
these
manifests
together
into
a
into
a
single
unit.
You
know
into
what
we
really
think
of
as
a
deployable
application.
A
We
have
labels
and
that's
you
know,
that's
the
native
mechanism,
but
there's
there's
no
real
enforcement.
You
know
you,
you
other
than
manual
manipulation,
making
sure
that
you
know
each
manifest
has
a
label
you
know
to
to
create
that
Association.
There's,
no
other
real
mechanism
for
it
and
that's
where
helm
comes
into
play
so
helm
again
at
a
high
level,
allows
you
to
treat
your
kubernetes
application
as
a
single
unit.
It
provides
a
rendering
engine,
most
mostly
focused,
on
go
template,
though
that
is
pluggable
with
a
great
deal
of
difficulty.
A
It
provides
a
package
and
application
manager
is
that
butcher
did
I,
hear
butcher
laughs,
no
all
right,
so
it
can
act
as
your
application.
You're
a
package
manager
akin
to
you
know
the
way
app
to
yum
or
apk
works
for
your
Linux
distribution,
and
then
it
also
provides
a
release
manager
so
that
if
you're
deploying
the
same
application
multiple
times,
you
can
track
that
release.
You
can
you
can
you
can
track
those
those
related
manifests
or
those
related
resources
and
alter
them
as
a
single
unit?
A
Some
terminology
really-
and
you
know,
I-
should
have
put
this
build
a
point.
Last
obviously,
when
we
refer
to
helm
we're,
you
know
we're
going
to
be
referring
to
the
complete
application,
but
it's
also
specifically
the
client-side
tiller
is
the
server-side
application
that
actually
lives
in
your
cluster
and
communicates
with
the
API
server.
A
B
B
So,
as
you
can
see,
that's
quite
a
few
clusters
there.
We
also
have
one
namespace
per
team,
so
every
team
gets
their
own
namespace.
This
is
facilitated
through.
We
have
a
tool
called
namespace
creator,
internally
kind
of
like
how
github
was,
if
you
went
I
heard
that
in
the
keynote
they
have
something
similar.
B
So
the
team,
basically
a
product
team
that
wants
to
use
kubernetes
cluster,
will
fork
a
repository
that
has
a
list
of
the
enabled,
namespaces
and
add
their
product
code
to
it,
and
then
that
will
perform
some
validation
to
make
sure,
for
instance,
that
they
have.
One
thing
that
we
enforce
is
that
they
have
a
contact,
technical
contact
filled
in
another
system
so
that,
if
their
stuff
goes
down,
we
know
who
to
reach
out
to.
B
Let's
see
the
namespace
creator
also
provides
role
based
access
control
via
Active
Directory
groups.
So
if
you
know
you
want
to
give
somebody
access
to
that
namespace,
you
add
them
to
an
Active,
Directory
group,
and
it
also
enforces
resource
quotas
on
the
entire
namespace
and
deploy
is
tiller,
the
server
side
of
helm
for
each
namespace
to
isolate
things,
the
resources
and
the
creation
of
them.
Everything
is
separate.
Finally,
another
detail
here
is
that
we
have
multiple
as
a
result
of
having
the
hybrid
cloud.
B
We
have
AWS
ingress
controller,
which
uses
the
application
load,
balancing
and
gorse
controller,
which
we
developed
jointly
with
Kouros
and
then
on
Prem.
We
just
use
the
standard
nginx
ingress
controller,
but
as
a
result
of
all
these
complications-
or
you
know
different
details,
that's
essentially
why
we
developed
the
web
service
chart.
A
A
A
There
are
some
pros
and
cons
as
there
are
to
everything
to
that
approach.
The
pros,
as
I
mentioned,
is
that,
or
you
know
the
the
single
Pro
really
the
biggest
one
is
that
that
chart
is
tuned
to
specifically
to
that
application.
There
are
no
extra
configuration
points
to
confuse
a
team
or,
to
you
know,
to
conflict
or
just
to
you
know,
cause
problems.
It's
it's
very
purpose-built.
A
The
cons
are
that
you
know
there's
no,
there's
no
commonality
between
those
charts.
So
when
team
a
you
know,
discovers
a
new
way
to
do
something:
team
that
that's
not
easily
communicated
or
shared
with
team
B,
you
know,
there's
no
shared
best
practices
or
you
know
not
it
at
least
not
easily.
You
know
requires
some
manual
intervention
people.
Actually
you
know
talking
to
each
other
which
yeah
you're
right
yeah.
A
We
then
need
to
communicate
those
changes
back
to
the
teams
again,
communication,
alright
or
when
there's
an
upstream
bug
that
is
either
discovered
or
fixed.
That
requires
a
change
and
chart
functionality
that
needs
to
be
shared.
So
it's
just
you
know
it's
it's
all
about
sharing
and
the
difficulty
therein.
So
a
few
months
ago
we
decided
to
kind
of
flip
that
and,
as
you
can
see
from
the
bullet
points,
like
really
flip
it
so
now
the
coup
Bernards
team
maintains
that
web
service
chart
manually
and
we
share
it
across
all
the
teams.
A
So
the
pros
again,
as
you
can
see,
are
again
flipped.
You
know
we
have
some
there,
there's
no
need
for
commonality,
because
there
is
only
one
chart
team,
a
deploys
that
chart
using
the
same
values
or
the
same.
You
know
the
same
options
are
available
to
team
a
as
are
available
to
team
B.
If
a
team
discovers
a
new
better
way
to
do
it,
they
can
submit
those
improvements
via
PR.
A
Likewise,
when
we
change
something
in
the
in
the
cluster
or
there's
an
upstream
change,
we
can
integrate
that
into
the
cluster
cut
a
new
version,
and
you
know
send
that
out
to
the
teams
who
you
know
then
update
their
pipelines
so
that
they're
deploying
using
that
new
version
of
of
the
chart.
This
has
worked
out.
A
So
basic
structure.
Here
we
have
the
usual
usual
components
of
a
helm,
our
chart,
yeah
Mel,
you
know
metadata.
We
have
the
templates
folder
and
then
our
values
files
so
I'm
going
to
dig
into
the
values
file
and
because
I'm
using
a
large
terminal.
This
is
going
to
be
lots
of
scrolling.
As
you
can
see,
this
values
file
is
251
lines,
long,
60,
256
characters.
So
to
say
we
provide
a
lot
of
knobs
and
dials
is
putting
it
somewhat
mildly.
A
A
few
I'll
go
through
a
few
of
these
options,
so
we
have
things
like
you
know:
a
configurable,
AWS
region,
I
am
role
revision,
history
limit,
let's
get
to
some
more
interesting
ones
of
affinity.
If
a
team
and
most
teams
eggs
have
actually
adopted
this,
we're
probably
going
to
flip
this
bit
because
most
teams
do
want
to
have
some
anti
affinities
so
that
their
pods
get
spread
across
availability
zones.
A
Here
we
have
that
set
to
false
just
because
that
is
kind
of
the
you
know:
it's
not
it's
not
out-of-the-box
behavior,
but
again,
we're
probably
going
to
flip
that
pretty
soon
a
couple
of
things
that
I
will
mention
in
just
a
few
minutes,
our
replicas
count
and
max
replicas
counts.
So
these
come
into
play
when
creating
the
deployment
replicas
count.
A
If
you
used
a
community
chart
before
you're
gonna
recognize
some
of
these
values,
because
we've
identified
those
as
best
patterns,
so
you
know
being
able
to
specify
the
service
account
name
being
able
to
add
custom,
pod,
annotations
and
labels.
We
recognize
that
you
know
we
don't
want
to
provide
values
for
everything,
so
some
of
these
are
relatively
relatively
freeform,
where
they,
you
know,
they're
just
expecting
lists
or
whereas
you
know
others
are
expecting
actual
objects,
ingress
settings
so
again,
a
62%
of
the
way
through
and
I've
scrolled
enough.
A
A
But
you
know
it's
a
it's
a
standard
manifest
that
we
just
have
made
very
configurable.
One
particular
item
of
interest.
Here
is
what
we've
done
with
this
conditional
around
our
rolling
update
strategy.
We've
we've
said:
if,
if
we're
only
deploying
one
replica,
we
want
to
set
the
maximum
unavailable
to
zero
so
that
when
we
do
a
rolling
update,
we're
sure
that
we
still
have
a
pod
running.
A
You
know
during
that
update,
so
it'll
bring
a
pod
up
rather
than
killing
the
old
pod
and
bring
you
know
before
and
then
not
waiting
until
the
new
one
is,
our
you
know,
bring
your
pod
up
and
then
not
waiting
until
the
old
one
new
one
is
ready.
It
makes
sure
that
at
least
one
pod
is
running
during
the
rolling
update,
so
that
is
actually
something
we
did,
that
one
of
our
teams
discovered
they
were
having.
A
You
know
they
were
having
outages
with
any
time
they
were
deploying
you
know
best,
but
like
best
best
practice
would
be
don't
run
one
replicas,
but
you
know
we
try
to
accommodate
we've
added.
Let's
see,
I
am
role
so
this
again.
This
is
a
pattern
that
use
in
the
community
charts
repo
for
AWS,
specific
applications
being
able
to
add
the
I
am
role
as
an
annotation
and
it
containers
there's
a
there's,
a
bug
in
versions
of
kubernetes,
less
than
1.8
that
ignore
the
standard.
A
A
Let's
see,
there
are
here's.
The
anti
affinity
rule
that
I
mentioned
in
the
values
file
I
think
what's
a
particular
interest
in
what
we
actually
kind
of
called
out
in
our
summary
of
the
talk
is
the
ability
to
add
side
cars
to
your
applications,
so
we've
recently
started
pursuing
Jaeger
for
tracing
distributed
tracing,
so
we
added
a
Jaeger
dot
enabled
flag
to
our
values
file.
A
If
you
said
Jaeger
not
enabled
to
true
the
Jaeger
agent
gets
added
as
a
sidecar,
and
then
we
provide
other
configuration
points
so
that
if
a
team
is
testing
a
newer
version
or
a
different
version
of
the
image
they
can
plug
that
in
resources.
Also,
this
is
a
pattern
that
that
Vic
mentioned
yesterday
and
just
kind
of
FYI.
A
You
should
definitely
when
that
video
is
available
check
that
out,
because
if
you
contribute
to
the
community
repo,
we
will
be
very
grateful
if
you
follow
those
patterns
like
day
one,
but
in
this
case
I'm
referring
to
the
resources
section
here
where
you
plug
in
your
resources,
because
you're
you
know,
different
teams
are
going
to
discover
different
resource
needs.
So
this
is
how
we
accommodate
that
fluent
D.
Some
of
our
teams
are
using
fluent
D
for
log
collection
and
forwarding.
A
251,
you
know
lines
and
our
values
file
is
you
know
pretty
good,
we
don't
need
to
add.
You
know,
options
for
every
possible
sidecar
that
somebody
might
want
to
deploy.
So
we
have
here
just
a
plane
if
you
want
to
provide
a
custom.
Sidecar
just
give
us
the
ammo
and
we
inject
that
right
into
the
manifest,
and
then
that
gets
added
to
your
pod.
A
A
Just
like
we
want
to
be
able
to
enable
sidecar
containers,
we
want
to
enable
Prometheus
like
OneTouch
Prometheus
metrics
scraping,
so
we
have
a
metrics
enabled
value.
You
set
that
to
true
and
helm
adds
these
annotations
to
the
service
and
then
our
standard
Prometheus
configuration
starts
scraping.
Those
services
scraping
the
metrics
off
of
those
services
automatically.
A
Let's
see,
then
the
other
thing
I
wanted
to
point
out
in
the
service
is
right.
Here
we
have
a
conditional.
We
wrap
our
service
type
in
a
conditional
here.
So
when
we
deploy
to
AWS,
we
set
the
platform
value
to
AWS
and
if
we're
requesting
an
ingress,
we
said
ingress
dot
enabled
equals
true
the
way
the
alb
ingress
controller
works.
Is
that
it?
You
know
it
requires
a
node
port
to
be
exposed.
It
attaches
to
the
node.
A
A
We
have
discovered
that
this,
it's
not
quite
a
binary
thing,
so
this
is
probably
something
we
need
to
revisit
soon.
We
have
services
that
are
not
deployed
to
AWS
and
do
need
to
be
note.
Ports-
and
you
know,
they're
not
always
going
to
be
cluster
IP
services.
You
know,
even
though
these
conditions
aren't
met.
So
again,
that's
that's
something.
We
learn
and
as
we
do,
we
make
changes
and
everybody
benefits.
A
Speaking
of
ingress.
Let's
take
a
look
here.
The
entire
manifest
is
wrapped
in
a
conditional.
So
if
ingress,
if,
if
an
egress
isn't
required,
we
don't
bother
creating
the
resource,
you
know
pretty
straightforward,
but
here
again,
if
we're
deploying
to
AWS,
then
we
need
to
add
a
set
of
annotations
that
the
alb
ingress
controller
uses
to
provision
those
application
load
balancers,
and
there
are
quite
a
few
of
them.
Aws
provides
a
wealth
of
options.
Configurability
and
the
alb
ingress
makes
does
really
does
a
good
job
of
supporting
them.
A
A
We
have
a
helper
template
that
lists
our
ten,
our
subnet
names
based
on
the
type
of
sub
based
on
the
type
of
alb
we're
asking
for.
If
we're
asking
for
an
internet
facing
ingress,
then
it
puts
the
security
groups
and
the
subnets
that
correspond
to
our
public
facing
infrastructure
in
those
annotations.
If
they
are
internal,
only
alb
endpoints,
then
we
use
a
different
set
of
security
groups
and
subnets
so
that
that
kind
of
covers
how
we
deploy
or
how
we
handle
the
AWS
case.
If
we're
at
the
point,
you
are
unplanned
premises,
its
Rafael
mentioned.
A
We
use
a
shared
nginx
ingress
controller
in
those
clusters,
so
we
so
we
don't
need
the
AWS
annotations.
We
just
need
one
that
sets
the
class
to,
in
our
case,
shared
nginx,
and
then
here
we
have,
you
know
a
few
of
the
usual
suspects.
You
know
annotations
and
we
can
set
hostname.
We
actually
do
have
another
helper
template
that
will
provide
a
default
fully
qualified
domain
name
that
can
be
overridden
by
teams
if
they
want
to
use.
A
You
know
their
own
custom
hostname
as
promised
I'm
not
going
to
go
into
config
Maps
they're,
really
pretty
pretty
bare-bones
for
the
most
part,
fluent
E
is
just
kind
of
a
default
configuration
config
map.
The
config
map,
yeah
mo,
is
basically
empty.
All
we
do,
is
you
feed
in
you
feed
in
a
key
value
of
you
know?
What's
your
what's
your
config
name,
your
config
map
key
should
be,
and
then
you
feed
in
the
data
and
we
just
stick
it
right
in
the
manifest
it's
it's
super
elegant.
A
What
I
will
show,
though,
because
I
mentioned
it
earlier
is
the
horizontal
pod
autoscaler.
So
this
is
where
those
two
values
max
replicas
replicas
count
and
max
replicas
count
come
into
play.
If
you
provide
both
of
them
and
met
your
specify
a
higher
max,
then
you
know
kind
of
min
I
guess
we
could
have
called
it
min.
Then
it
will
create
this
manifest
and
it
will
use
those
values
in
the
in
the
appropriate
field.
So
in
the
spec
you
have
max
replicas
and
you
have
min
replicas.
A
You
know
anybody
who
knows
the
HPA
realizes
that
that
it
is
if
you,
if
you
have
the
custom
metrics
api,
enabled
you
can
scale
based
on
you
know:
Prometheus
metrics,
for
example,
and
this
this
that's
something
that
we
really
like
to
offer
like
soon
to
our
dev
teams.
They
are
really
keenly
interested
in
being
able
to
scale
based
on
things
like
requests
per.
Second,
you
know
being
able
to
spin
up
new
pods
when
you
hit
a
threshold.
A
For
now,
though,
we
are
using
just
the
base
functionality
which
is
CPU.
So
with
that
what
the
HPA
does
is
it
looks
at
the
average
CPU
usage
across
pods
and
it
when
it
when
it
exceeds
a
given
threshold,
it
will
create
more
pods
up
to
your
maximum
until
that
threshold
is
no
longer
being
met,
so
a
little
bit
yeah
inside
kubernetes
and
not
really
inside,
but
bonus
kubernetes
information.
The
last
one
I'm
going
to
show
is
pod
pod
disruption
budget.
This
is
a
relatively
new
ish
resource
that
we
are
also
finding
pretty
convenient.
A
This
is
when
we're
doing
when
we're
doing
manual,
node
native
maintenance
say
we
need
to
take
a
node
down
for
whatever
reason,
when
you
do
a
cube
drain,
its
first
going
to
look
at
your
pod
disruption
budget
and
make
sure
that
you
keep
that
it
that
the
scheduler
keeps
your
max
unavailable,
pods,
that
number
of
pods
it's
going
to
affect
no
more
than
the
head
number
of
pods
during
a
maintenance
during
maintenance.
So
this
is
again
something
that
was
highlighted
by
an
application
team.
They
were
wondering
interactive.
You
know
why
all
of
their
pods
are.
A
Why
would
they
were
returning
five
hundreds?
You
know
at
the
same
time
that
we
were
taking
a
pod
down
for
maintenance,
and
it's
because
of
a
few
things,
I
mean
you
know
all
of
their
pods
got
Co
scheduled,
which
you
know
oops,
but
something
like
this.
You
know
the
pod
DB
is
meant
to
address
that
so
that,
even
if
you
do
get
close
scheduled,
this
will
make
sure
that
you
know
that
X
number
of
pods
will
be
will
be
up
and
running
before
it
kills
the
rest
of
them.
So
it
comes
in
very
handy.
A
You'll
notice,
we
have
a
conditional
around
this
manifest
as
well.
This
so
if
release
dot
is
install,
is
a
helm
condition
that
says
you
know
we're
only
going
to
create
this.
If
this
is
an
initial
installation,
the
reason
is
that
the
pdb,
at
least
as
we
understand
that
the
pdb
is
currently
immutable.
So
if
you
try
to
change
a
pod
disruption
budget
after
it's
been
created,
your
helm
is
gonna
flip
out
a
little
bit.
You're
gonna
get
an
error
message
that
it
you
know,
resource
already
exists,
or
so
their
you
know,
field
cannot
be
changed.
A
B
All
right,
you
guys
hear
me
all
right,
so
I'm,
just
gonna,
run
really
quickly
through
an
example
of
deploying
our
deployment,
essentially
with
the
web
service.
Chart.
Let's
see
if
you're
interested
the
code
is
available
on
my
github
brothels,
hello
cube
con,
it's
pretty
minimal,
it's
just!
It's
not
actually
going
to
include
the
web
service
chart
or
anything,
but
if
you
want
to
go
and
look
at
the
files
in
more
detail
that
I'm
pulling
up,
that's
where
it
is.
B
What
the
example
does
just
basic
go:
application
like
hello
world,
it's
gonna
run,
go
to
us,
it's
gonna,
build
the
application,
push
the
container
image
to
our
container
registry,
which,
in
our
case
this
ECR,
amazonia
CR
and
then
finally
deploy
the
application
using
the
web
service.
Chart
and
helm.
Hope.
Thank
you.
So
this
is
the
app
in
question.
As
you
can
see,
just
as
hello,
a
cube
con,
I'm
gonna
just
show
you
the
files
here
really
quickly
simple,
go
app!
Here's
the
test.
My
hands
are
shaking
like
I
was
hoping
they
wouldn't.
B
B
So
that's
probably
the
most
important
thing.
This
is
shared
values
that
will
be
used
both
for
AWS
and
on-prem.
If
we
look
at
one
of
the
others,
you'll
see,
I
also
filled
in
the
fully
qualified
domain
name,
and
the
platform
is
on-prem
so
yeah
you
can
specify
multiple
values,
files
for
helm
and
it
will
use
a
you
know
they
can
overwrite
each
other
and
stuff.
These
are
completely
not
overlapping.
So
that's
not
even
the
consideration
anyway.
So
we
can
go
to
the
website.
I
already
deployed
this,
so
we
wouldn't
have
to
wait
for
the
whoops.
B
So
you
know
the
whoops
we
have
to
be
on
the
these
are
internal
only
or
so
you
have
to
get
on
the
VPN.
So
might
I
yeah
we're
intended
to
do
that
earlier,
so
yeah,
hello,
Keep
Calm
that
would
be
showing
up
there,
but
but
yeah
it's
only.
It's
not
exposed
externally.
We've
had
to
configure
that
separately.
I
did
not
put
that
annotation
on
there
or
up
yeah.
So
we
won't
be
able
to
see
the
hello
Keep
Calm,
but
you
can
use
your
imagination.
It's
just
regular
text.
B
B
B
But
yeah
that
so
you
know
the
point
is
that
once
we
do
this,
I
didn't
go
over.
Actually
what
get
lab
runners
are,
but
you're
probably
familiar
with
them
in
some
flavor
or
another.
It's
a
lot
like
Jenkins
or
Travis
CI,
and
essentially
we're
just
running
some
code
in
a
you
know
to
find
the
container
of
choice,
and
so
you
know
whoops,
oh
yeah.
Well,
we
need
to
be
on
the
VPN
to
push
as
well,
so
there
we
go
those
but
yeah.
Next,
you
would
have
seen
that
the
message
changed
after
a
moment
yeah.