►
From YouTube: When the Tekton Cat Misbehaves – Pitfalls, Mistakes, and Lessons Learned - Mark Nuttall, IBM
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
When the Tekton Cat Misbehaves – Pitfalls, Mistakes, and Lessons Learned - Mark Nuttall, IBM
The Tekton project provides Kubernetes resources for declaring and launching CI/CD pipelines. You and your team's skills in working with Kubernetes can be applied directly to building and maintaining your CI/CD system. This is great once it's all set up and working - but a new kitten will sometimes misbehave! Tekton's design, and its Kubernetes-native implementation can make it fail in peculiar, distinctive ways. In this talk we'll show how Tekton can go wrong, explain why, and demonstrate how to fix it. We'll also use these failure modes to talk more generally about the strengths and weaknesses of Tekton's custom resource-based approach. We'll highlight the best methods and tools for debugging in this environment, and look forward to the key improvements needed - and planned - to raise a truly well-behaved CI/CD system.
https://sched.co/ZenL
A
A
So
several
of
the
ways
in
which
tecton
used
to
lead
you
down
strange
paths
or
misbehave
on
you
are
won't,
say
not
possible
anymore,
but
much
harder
to
get
to,
because
the
changes
that
are
made
in
v1
beta,
so
I'm
kind
of
splitting
up.
It
can
still
do
these
things
to
you.
These
things
are
much
harder
to
see
anymore
because
because
of
the
changes
that
were
made
in
beta
and
then
finally,
we'll
have
a
look
at
okay,
tecton's,
a
system
built
out
of
custom
resources.
A
A
A
And
you
know
where
you
run,
it
doesn't
affect
where
it
where
it
deploys
to.
Obviously
it's
got
an
increasingly
rich
catalog
of
pre-built
tasks
and
that's
only
become
more
true
as
we've
as
we've
gone
through
this
year.
So
it's
got
lots
of
primitive
building
blocks
for
doing
all
kinds
of
useful
and
common
tasks
that
you
want
to
automate,
particularly
in
sve
icd
space.
A
A
A
This
is
really
showing
that
it's
not
just
a
linear
flow,
but
an
acyclic
cyclic,
basically
directed
graph.
I
think
it's
the
right
term,
essentially
a
set
of
tasks
that
run
in
an
order
where
some
may
need
to
finish
before
others
can
need
to
run
some
can
run
in
parallel.
Some
are
going
to
be
sequential,
so
building
blocks.
A
A
A
You
can
kick
them
off
manually.
Typically,
these
things
are
going
to
be
running
dynamically
from
from
web
hooks
or
driven
by
other
events
or
triggers
and
techton
has
a
triggers
subproject
that
specifically
deals
with
okay.
What
might
have
happened?
What
events
might
have
happened
that
you
want
to
use
in
order
to
generate
pipeline,
runs
and
create
templated
instances
of
them.
A
A
So
in
the
previous
slide
we
talked
about
how
pods
can
share
these
persistent
volumes
and
that's
a
common
assumption
in
the
in
the
old
days.
If
you've,
if
you
use
tacton
before
it
went
into
beta,
there
were
ways
in
which
these
persistent
volumes
could
be
created.
Implicitly
for
you,
and
so
you
may
not
be
kind
of
warned
that
persistent
volume
claims
would
be
created.
A
A
And
finally,
if
we
pick
through
the
yaml
in
that
description
of
the
pod,
we
see
that
yep
this
pipeline
run
here
is
mentioned.
It's
trying
to
mount
some
persistent
volume
that
isn't
happening,
and-
and
that's
because
because
we
said
that
there
were
multiple
tasks
in
the
pipeline,
we
provided
an
output
from
the
from
the
git
resource.
A
It's
less
easy
to
run
into
that
problem,
because
the
beta
version
of
tekton
introduces
these
idea
of
workspaces,
which
are
an
analog
of
a
shared
file
system
and
workspaces
need
a
volume
claim
template.
You
need
to
say
this
volume
this
workspace
needs
a
volume
claim.
So
the
fact
of
having
to
specify
a
volume
name
can
alert
you
or
remind
you
that
oh
yeah,
I
need
dynamic
provisioning
in
my
cluster.
Otherwise
this
thing
isn't
going
to
work.
A
Okay,
so
having
got
past
that
your
pipelines
are
up
and
running,
but
the
next
thing
that
we
tend
to
run
into
is
problems
with
credentials
so
well,
let's,
let's
do
something.
What
we're
going
to
do,
we'll
clone
a
repo?
Do
a
docker
build
docker,
push
see
what
happens
so
we're
doing
a
docker
build
in
this
case,
I'm
using
builder,
you
might
use
canico.
There
are
several
other
ways
of
building
images
on
kubernetes
clusters.
A
A
So,
essentially,
you
need
to
create
secrets
with
annotations
on
that
tech
tool
will
use
list
them
in
your
service
accounts
and
then
finally,
reference
them
from
your
pipeline
runs,
and
it's
these
annotations
here
that
say
what
these
secrets
are
for.
So
this
is
a
docker
secret.
We
have
this
docker
annotation
on
if
it
was
a
github
secret,
we'd
have
a
git
secret
here,
so.
A
Okay,
so
having
created
the
secrets,
we
then
frequently
talk
about
patching
service
accounts,
because
you
may
well
use
a
cube,
ctl
patch
to
add
it
in
so
all
right.
Fine,
if
we
apply
this
docker
secret
and
patch
it
onto
the
service
account,
which
is
adding
this
extra
line
on
the
service
account
to
say
that
the
secret
is
now
in
use
there
oops
sorry.
A
A
A
A
A
A
A
So
what
does
that
tell
us?
Well,
the
consequences
of
this
are
that
if
I
need
to
have
multiple
credentials
for
a
single,
a
single
repository,
a
single
resource
like
github
or
docker
hub,
you
essentially
need
to
patch
them
onto
different
service
accounts,
and
so
you
need
to
choose
the
service
account.
That's
got
the
credentials
on
that.
You
want
the
alternative.
Is
that
you
simply
don't
do
that
and
you
end
up
with
using
functional
identities
rather
than
those
pretending
to
be
particular
users.
A
All
right,
so
we
said
dynamic
provisioning.
We
typically
need
that
to
be
enabled
we
need
to
patch
our
sequence
onto
service
accounts,
and
this
idea
that,
if
for
as
a
whole
of
github,
you
can
only
put
one
credential
onto
one
service
account
so
techton
typically
does
it
as
somebody
it
tends
to
run
on
behalf
of
somebody,
so
you
tend
to
creating
an
identity
that
tecton
will
use
for
dealing
with
all
the
repositories
that
you're
interested
in
for
a
particular
service
account.
A
That's
sort
of
a
consequence
of
the
way
that
that
secret
handling
works
and
we've
shown
that
the
tkn
command
line
both
for
starting
pipelines
following
their
logs
looking
at
their
logs.
It's
particularly
helpful.
It's
it's
quicker
to
get
information
now
using
the
tkn
than
what
you
can
dig
out
with
coop
ctl,
so
it's
very
useful
command
line
and
one
that's
well
worth
using
all
with
total.
A
A
Things
that
were
abstractions
for
external
systems
and
objects-
it
has
ones
for
get
pull,
requests
images
that
you
might
build
clusters,
you
might
deploy
to
storage
systems,
cloud
events,
these
things
haven't
been
ported
to
beta
they're,
all
still
alpha
level
and
there's
a
good
link
here
that
talks
about
why
those
things
haven't
been
brought
forward.
Essentially,
they
each
have
different
requirements,
different
side
effects,
there's
not
a
lot
of
consistency.
A
Each
thing
makes
sense
in
itself,
but
as
a
set
of
pipeline
resources,
they're
neither
complete
nor
intuitive
to
understand,
and
so
what
they've
done
instead
upon
me,
is
to
create
a
set
of
tasks
in
the
task
library
that
you
can
use
instead
of
using
pipeline
resources,
and
the
tasks
are
really
the
way
to
go.
If
you're
used
to
using
pipeline
resources,
I
would
advise
you
move
to
using
tasks
instead
from
the
task
library.
They
are
all
of
them
excellent
replacements,
and
it
reduces
the
number
of
different
types
of
thing
you
have
to
think
about.
A
In
the
old
days
you
just
have
to
copy
between
resources.
We
would
use
these
pipeline
resources
and
each
individual
task
would
have
to
say
right
now.
Copy
everything
I've
done
into
my
output
pipeline
resource,
so
the
next
thing
in
the
chain
pick
it
up
this
added
a
lot
of
sort
of
boilerplate
and
execution
time
to
your
tasks.
A
A
A
A
Finally,
that
old,
git
pipeline
resource
would
implicitly
create
persistent
volumes.
You'd
have
to
explicitly
copy
the
results
that
you've
done
with
it,
and
because
this
is
a
this
important
point
because
it
ran
in
an
init
container,
you
couldn't
mount
additional
secrets
in.
I
worked
with
a
customer
only
some
weeks
ago
that
were
trying
to
use
self-signed
certificates
on
their
kubernetes
cluster
and
the
tecton
git
pipeline
resource
wasn't
happy
with
that
now.
The
solution
for
them
was
to
move
forward
to
beta
so
that
they
could
use
the
kit.
A
A
They
behave
in
the
way
that
you
would
expect
them
to
do
all
the
skills
that
you've
got
of
working
with
kube
in
general.
Role-Based
access
control,
kubectl
those
things
just
behave
very
intuitively
for
you-
there
are
some
oddities,
particularly
around
pipeline,
runs.
If
you
want
to
see
the
logs
of
a
pipeline
run,
those
are
in
the
pipeline
runners,
so
you
need
to
get
the
logs
out
of
them.
So
you
end
up
with
these
resources
left
lying
around
after
you've
finished.
A
And
similarly,
if
you
delete
that
pipeline
run
results,
then
you've
lost
your
logs
frequently,
you
may
want
to
persist
the
logs
of
your
build
for
much
longer
than
the
metadata
about
it.
You
know
how
many
containers
what
was
mounted
in
what
the
the
kubernetes
description
of
it
and
that's
something
I
touch
on
in
the
next
in
the
next
point.
I'll
come
back
to
that.
The
the
other
thing
that
we
see
is.
A
So
if
you
want
to
have
your
build
logic
in
your
git
resource,
you
end
up
with
this
problem
of
double
cloning.
You
have
to
clone
it
once
to
get
the
descriptions
of
the
custom
resources
that
you
want
to
apply
so
that
you
can
now
get
clone
and
use
it,
and
so
it
leads
to
this
point.
The
tactile
isn't
in
a
way
travis
or
jenkins.
A
A
A
tekton
pipeline
can
be
a
load
more
yaml
than
somebody
if
you're
used
to
travis,
you
might
end
up
with
four
or
five
lines
of
a
travis
description
that
could
turn
into
a
lot
more
a
lot
more
yammer
for
a
teton
pipeline,
and
that
leads
to
this
point
that
what
teton
really
is
very
good
at
is
a
cloud
native
micro
service
factory,
I.e.
If
I
have
a
lot
of
repositories
whose
build
is
functionally
identical,
they
all
look
the
same.
I
want
to
do.
A
A
Okay.
So
I've
covered
what
tekton
is
some
of
the
gotchas
that
you'll
see
if
you
just
download
it
and
start
trying
to
to
get
the
hang
of
how
to
use
it.
We've
looked
at
how
it's
evolved
this
year
and
how
it's
really
emphasizing
tasks
and
de-emphasizing
pipeline
resources.
It's
consolidating
on
fewer
concepts
and
this
growing
library
of
tasks
are
where
the
exciting
changes
in
tecton
are
and
where
the
growth
is
and
where.
A
I
would
encourage
you
to
focus
your
attention
if
you're
picking
up
and
using
it
and
again
be
aware
that
tecton
isn't
an
off-the-shelf
product,
it's
a
set
of
piece
parts
that
you
can
use
for:
building
cloud-native
ci
cd
systems
and
that,
if
you're
looking
to
build
particularly
a
lot
of
microservices
in
exactly
the
same
way,
then
yes
time
might
be.
What
you're
looking
for
at
this
point,
if
it
wasn't
just
a
recording,
I'd,
be
stopping
and
taking
questions.
A
A
A
Okay,
so
hopefully
that's-
hopefully
you
can
still
hear
me
so
I
talked
about
at
the
end.
What
tecton
is
best
at
and
it
is
that
microservice
factory.
It
is
that
idea
of
building
a
lot
of
repositories
that
have
functionally
the
same,
build
logic
for
them
and
so
that
the
idea
of
microservice
factory
is
what
we've
seen
taped
on,
be
the
most
used
for
and
again,
if
you've
got
in
in
jenkins
in
jenkins
like
world.
A
A
A
A
Know
that
just
because
you
built
it
yesterday,
you
want
to
be
able
to
build
it
again.
Tomorrow,
text
on
is
a
much
cleaner
design,
a
much
more
kind
of
serverless
design,
if
you
like,
and
you
don't
see
that
kind
of
plug-in
creep
that
that
you
can
see
with
tactile.
That's
certainly
true,
however,
if
you're
using
jenkins
and
what
you've
got
works
and
it
isn't
broken.
Why
would
you
consider
moving
to
tecton
or
anything
else?
A
A
Lack
of
groovy
support,
lack
of
a
polished
ecosystem
well,
certainly,
work
on
the
ecosystem
is
where
a
lot
of
activity
on
tecton
is
certainly
groovy
languages
for
creating
pipelines
as
code
yeah,
true
tacton
doesn't
have
a
dfl,
and
I
don't
think
it's
going
to
have
one.
You
you're
writing
tasks
in
whatever
languages
you
want
to
write
your
tasks
in.
A
Does
many
millions
of
jobs
impact
a
kubernetes
server
or
are
there
ways
to
archive,
builds
different
companies
that
use
techton
in
their
builds
will
do
that
logging
and
archiving
for
you?
I
know
firsthand
that
ibm's
techton
based
build
pipeline
in
our
public
cloud.
Yes,
we
do
that
work
of
archiving
those
logs
for
you
so
that
they
do
persist
beyond
the
duration
of
your
pipeline
runs,
but
if
you're
building
your
own
tank
on
base
system,
that's
a
problem.
That's
left
for
you
to
solve,
so
you
need
to
most
people.
A
A
Jenkins
x
uses
it
under
the
covers.
I'm
not
an
expert
in
it.
It's
been
some
time
since
I've
used
jenkins
x,
jenkins
x,
has
a
lot
of
opinions
and
you
will
accept
all
of
its
opinions
for
none
of
them.
So
you
need
to
look
very
carefully
at
jenkins
x.
If
you
agree
with
it
in
every
particular,
then
it's
going
to
give
you
a
lot
of
value.
If
you
disagree
with
it
anywhere,
you
need
to
move
on.
A
A
Really
teton
runs
on
kubernetes,
only
it's
very,
very
coop
native.
I
don't.
I
don't
think
I'm
gonna
possibly
properly
address
that
slide
piece.
The
slide
should
be
up
ping.
Me
ping
me
on
slack.
If
you
need
a
copy
of
the
slides
and
you
can't
get
them,
how
would
you
debug
a
pipeline
in
case?
It
fails
the
so
often
if
your
pipeline
is
going
really
badly
wrong.
You
end
up
putting
great
big
long
sleeps
in
your
code
and
you
coop
ctl
exec
into
the
pods
and
dig
around
to
see
what's
going
on.
A
A
It's
quite
unusual
for
things
only
to
go
wrong
once
you
put
them
into
tecton,
contain
into
pods
and
run
them
on
tactile,
but
it
does
happen
and
yeah
sleep
and
shell
in
is
very
frequently
the
case.
If
the
logs
aren't
giving
you
what
you
need,
does
tetron
work
well,
alongside
prow,
we
do
use
both
techton
and
pro
in
the
tecton
project.