►
From YouTube: Operating Tekton Efficiently and Securely for multi-team use cases - Wendy Dembowski, Google
Description
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
Operating Tekton Efficiently and Securely for multi-team use cases - Wendy Dembowski, Google
Do you operate a Tekton Pipelines installation? Do you need to support multiple different teams using Tekton for CI/CD? In this talk I’ll discuss strategies for operating Tekton with workload isolation and user latency in mind. We’ll go into detail on strategies to avoid startup latency for Tekton PipelineRuns and TaskRuns while minimizing the number of nodes in your cluster to increase resource efficiency. We’ll also explore using the kubernetes autoscaler, caching strategies when the Docker sidecar prevents a Tekton Task from taking advantage of the Kubernetes node cache, and other cluster operation optimizations to support multi-team Tekton installation.
A
A
So,
let's
get
started,
oh
and
I'm,
going
to
talk
about
operating,
tecton,
efficiently
and
securely
for
multi-team
use
cases.
So,
for
the
purpose
of
this
talk,
we
have
this
company
called
things:
dogs,
love
and
things.
Dog
love
has
three
team
leads.
One
of
them
is
Peanut
and
he's
our
tecton
operator,
Sadie
she's,
an
application
engineer
on
the
blankets
team
and
Charlie
is
an
application
engineer
on
the
treats
team.
A
Peanut
wants
to
be
able
to
operate
tecton
securely
and
simply
as
simply
as
possible.
He
wants
one
installation.
He
doesn't
want
to
deal
with
multiple
clusters.
First
teams
and
it's
important
to
note
that
Charlie
really
loves
to
use
the
advanced
features
of
docker
that
are
in
the
experimental
areas,
so
that
will
come
up
later,
but
calling
it
out
now.
A
So
we
have
some
cicd
goals.
The
main
goals
are,
as
I
mentioned
before,
to
share
common
infrastructure
across
the
company,
builds
should
produce
provenance.
So
we
talked
earlier
about
salsa,
we
want
builds
to
produce
provenance
in
order
to
achieve
the
salsa
level
too
requirements
and
the
security
team
has
really
asked
peanut
to
work
towards
salsa
level.
A
Three,
so
tecton
chains
isn't
quite
there
yet,
but
there
are
also
some
things
about
how
you
run
your
CI
CD
system
and
how
builds
are
executed
that
help
to
fulfill
salsa,
3
requirements
and
and
part
of
that,
is
that
one
one
pipeline
run
in
tecton's
key
shouldn't
be
able
to
influence
other
concurrently
running
pipeline
runs
and
then
finally,
the
infrastructure
must
balance
team
resource
consumption.
Compute
costs
should
be
as
low
as
possible
and
pipeline
run.
Latency
should
be
low,
so
we
don't
get
inpatient
application
Developers.
A
You
can
see
that
he
isolated
the
tecton
pipelines
and
change
controllers
in
their
own
namespace
and
in
their
own
node
pool.
This
is
just
to
ensure
that
the
developer
workloads,
who
don't
run
in
the
same
on
the
same
machine
or
in
the
same
tecton,
namespace
or
sorry
in
the
same
kubernetes
namespace
as
the
controllers,
because
we
don't
trust
the
application
developer
code.
So
we
want
to
keep
them
as
far
away
from
our
our
control
plane
code
as
possible.
A
This
thin
control
plane
API,
is
so
that
peanut
can
authenticate
the
workloads
that
come
through
and
then
append
the
node
selectors
to
the
developer
team's
pipelines
so
that
they
get
directed
to
the
right.
Namespace
and
node
pool
important
to
note
that
the
tecton
control
namespace
would
be
tainted
and
the
only
things
tolerated
to
run.
There
would
be
the
the
tecton
operator
workloads,
which
is
controller
and
chains
or
pipelines
and
chains.
A
So
we
are
generating
provenance
with
tecton
chains,
and
there
is
one
problem
with
the
setup
on
the
previous
page
and
that's
that
textile
chains
will
output
provenance
to
a
single
location.
So
what
we've
decided
to
do
is
run
run
each
app
Dev
workload
in
a
separate
namespace
as
well,
and
then
each
chains
instance.
We
install
two
chains
controllers
instead
of
just
one
and
each
of
those
watches
the
corresponding
active
namespace
and
can
write
provenance
to
the
place
configured
by
the
aptaf
team.
A
So
you
know
Charlie's
team.
The
treats
team
is
working
toward
a
new
feature,
launch
and
they're
they're,
just
pushing
stuff
out
like
very
rapidly,
and
it's
Sadie
comes
over
to
peanut
and
lets
him
know
that
she's
not
getting
any
resources
for
her.
Her
builds
and
she
doesn't
understand
why
she
has
these
high
there's,
there's
very
high
latency
for
her.
It's
a
new
problem
today,
so
peanut
does
some
investigation
and
he
sees
that
the
church
team
is
just
taking
all
the
resources.
A
In
order
to
address
this,
he
puts
the
the
two
apt
Dev
teams
into
separate
node
pools
where
we
can
configure
the
minimum
number
and
maximum
number
of
nodes
allocated
to
those
teams
separately.
And
that
means
that,
no
matter
how
busy
the
treats
team
is,
the
blankets
team
will
always
have
resources,
and
this
also
offers
another
layer
of
isolation
between
the
builds
such
that
the
the
treats
pipeline
runs
and
the
the
blankets
pipeline
runs
will
never
end
up.
Running
task
runs
on
the
same
machine.
A
So
let's
talk
more
about
isolation.
We
currently
have
pod
level
isolation
between
our
pipelines
and
that
that's
just
offered
by
tecton.
The
pipelines
from
separate
teams
now
run
in
separate
node
pools
and
in
separate
namespaces,
but
multiple
pipelines
from
a
single
team
can
still
run
on
the
same
node.
A
A
All
right
now
we
get
over
to
the
compute
cost.
We
don't
want
idle
nodes
running,
it's
not
great,
so
we
set
up
Auto
scaling
and
in
this
case
we
have
a
maximum
and
minimum
number
of
nodes
per
node
pool
and
we'll
scale
up
as
resources
are
required
right
now
the
minimum
is
run
one.
So
there
is
always
one
node
ready
to
run
but
scaling
when
a
new.
When
a
new
pod
needs
to
be
scheduled,
then
we
have
to
bring
up
another
node.
A
A
So
peanut
finds
a
solution.
There's
a
blog
post
at
the
bottom
of
here.
That's
very
tiny,
but
if
you're,
if
you're
interested
in
reading
more
I
suggest
going
to
that,
so
we
have
this
concept
of
balloon
pods.
So
you
can
see
on
these
slides
I'm.
Sorry,
it's
a
little
hard
to
read.
The
pink
pods
are
balloon.
Pods
and
the
blue
pods
are
our
tecton
task
runs.
A
So
what
is
a
balloon
pod?
It's
a
pod
that
we
just
schedule
when
we
bring
up
the
node
pool
at
a
negative
10
priority.
So
as
soon
as
we
need
more
resources,
the
balloon
pod
gets
bumped
and
then
it
becomes
a
pending
pod
and
the
application
pod.
So
the
task
runs
are
able
to
be
scheduled
within
a
second
rather
than
50
seconds,
which
is
pretty
exciting.
A
And
then,
while
the
application
pods
are
running,
the
balloon
pod
will
be
scheduled
and
capacity
will
be
added,
while
no
one's
waiting
for
it.
This
is
a
great
concept
and
it
decreased
the
latency
startup
time
during
scale
up,
and
it
also
makes
the
the
workload
the
timing
of
the
the
pipeline
run
execution
it's
more
predictable,
because
now
you
know
you're
not
going
to
have
occasional
spikes
in
in
the
time
that
things
run.
A
So
the
other
issue
we
have
is
around
Docker
usage.
So
right
now,
the
way
that
you
run
Docker
builds
in
tecton.
Is
you
attach
a
Docker
and
Docker
site
card
to
a
task
and
and
then
all
the
the
builds
that
require
Docker
to
build
run
in
that
Sidecar
and
the
issue
here
is
the
side.
Cards
are
ephemeral
per
task
run
and
their
storage
is
also
ephemeral.
So
we
don't
benefit
from
the
the
node
cache
or
anything
else.
We
we
just
have
to
pull
images
fresh
every
time.
A
So
what
we
do
here
is
we
create
a
new
image
pool
job
and
we
provision
two
persistent
volumes
that
are
going
to
pull
images
that
the
application
developer
specifically
ask
for.
So
those
this
doesn't
solve
all
of
our
caching
problems.
But
if
there
are
common
images
that
the
users
are
using
in
their
task
steps,
we
can
cache
those.
A
So
each
each
application,
developer
team
gets
to
indicate
the
list
of
images
that
they
would
like
to
have
cached,
and
then
we
set
this
image
pool
job
up,
which
is
just
a
crown
job
that
runs
periodically
and
we
have
a
cash
expiry
time.
So,
at
the
cash
expiry
time
we
start
to
mount
the
the
inactive
persistent
volume
to
all
new
pods
that
are
that
require
the
docker
and
Docker
sidecar,
and
this
is
mounted
through
our
own
custom,
Docker
and
Docker
image.
A
So
it's
it's
mounted
as
the
storage
on
that
Docker
and
Docker
image
and
then,
after
the
cash
expiry.
When
we're
we,
we
switched
the
inactive
one
to
active,
and
then
we
ensure
that
all
that
the
the
previously
active
pod
is
no
longer
being
used
by
any
any
task
runs
before
we
Mount
that
to
the
image,
pool,
job
and
refresh
the
cache,
and
so
that's
essentially
it
gives
us
better.
A
Caching,
it
doesn't
solve
all
the
problems
as
I
mentioned,
because
if
you,
if
you
build
an
image
that
you
want
to
use
later
on
in
your
task,
runs,
you
always
have
to
pull
that
from
the
registry.
So
there
is
a
proposal
for
a
pipeline
level
sidecar
in
the
techtone
community
and
that's
the
type
of
thing
that
we
would
need
to
really
help
with
caching,
along
with
some
some
sort
of
solution
like
this.
That
has
the
built-in
cache
for
things
that
you
just
don't
want
to
pull
all
the
time.
A
We
talked
about
sharing
infrastructure
across
teams
in
a
secure
and
balanced
way,
generating
provenance,
resource
isolation
and
reducing
compute
costs
and
latency,
and
that's
all
I
got
I.
Think
I
won't
take
questions
in
the
to
honor
our
time.
I
actually
have
time.
Okay,
all
right
all
right!
Any
questions
go
ahead.
A
They
they
end
up,
they're
they're,
not
automatically
created,
they
have
to
be
provisioned
per
application
developer
team,
so
we
just
generate
like
when
we
configure
our
cluster
for
a
new
application
developer
team.
We
we
added
namespace
for
that
team
and
then
within
that
thin
API
server
like
in
here.
We
we
ensure
that
the
namespaces
are
added
to
the
tecton
requests
there
and
we
have
like
a
mapping
of
like
you
know
it's
so,
based
on
the
user
submitting
it.
We
know
what
team
they're
on
yeah
exactly
yep.
A
A
Yeah
yeah,
so
the
question
was
whether
we
find
the
taints
and
tolerations
burdensome
for
for
managing
I.
Think
there's
a
good
summary
yeah,
so
I
think
with
with
enough
teams
like
it's,
it's
all
just
code
right,
so
you
just
have
the
mappings
of
like
this
team
gets
this,
and
so
the
the
part
is
the
initial
configuration.
But
once
that's
set
up
it's
it's
just
like
quick
lookups
into
tables
to
indicate
what
gets
added
to
which
request.
A
If
you're
not
used
to
stuff,
so
we
we've
considered
things
around
how
to
like
do
like
right.
Now,
the
the
node
types
are
configurable
by
the
Optive
team
based
on
what
they
think
their
workloads
will
look
like
I,
don't
think,
we've
specifically
considered
the
did.
You
say
spot
instances:
okay,
yeah,
we
haven't.
We
haven't
considered
that
something
to
look
into
foreign.