►
From YouTube: Beyond Kubebuilder - Generating Entire Kubernetes Controller Implementat... Amine Hilaly & Jay Pipes
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Beyond Kubebuilder - Generating Entire Kubernetes Controller Implementations - Amine Hilaly & Jay Pipes, Amazon Web Services
"Tales from the Kubernetes controller factory floor" If you have used the Kubernetes controller-tools and kubebuilder projects to create a custom Kubernetes controller, you already know the immense power of these tools. However, a significant amount of work remains to implement a controller once kubebuilder has produced API types and basic controller scaffolding. What if you had to build dozens of controllers managing thousands of resources? You'd need a factory to produce full controller implementations from API model schemas. Amine and Jay happen to work in such a Kubernetes controller factory! In this talk, they will give you a tour of the Kubernetes controller factory, showing you how to use the API machinery and what dangers linger on the factory floor. They will teach you how to be the most productive worker in the whole factory by building on top of controller-tools and kubebuilder functionality. You will be introduced to open source tools and strategies that make Kubernetes controller factory life safe and enjoyable!
A
Thank
you
for
coming
to
our
our
talk
today.
My
name
is
Jay
pipes
for
the
purposes
of
this
particular
session,
I'll
be
playing
the
role
of
Bob
the
Builder,
and
this
is
Amin
hilali
who's
going
to
be
playing
the
role
of
Spud
the
Scarecrow
and
we're
going
to
talk
a
little
bit
about
Beyond
Cube
Builder,
generating
kubernetes
custom
controller
implementations
go
ahead,
so
we
have
because
of
a
variety
of
needs
of
this
set
of
kubernetes
controller
projects
called
ack.
A
We've
had
to
build
our
own
Factory
for
producing
kubernetes
controllers
and
we've
done
it
over
the
last
couple
years
and
we've
learned
a
few
lessons
over
the
last
couple
of
years
and
wanted
to
share
with
you
some
of
the
tools
that
we've
we've
built
to
help
generate
controller
implementations
and
maybe
give
you
some
inspiration
for
how
you
can
Implement
your
own
custom.
Kubernetes
controllers,
so
come
on
in
there's
lots
of
seats
over
here,
so
yeah
go
ahead,
go
ahead,
I
mean
so,
just
like
any
other
Factory.
You
will
need
hard
hats
and
I'm.
A
Not
gonna.
Put
this
on
I
already
look
like
too
much
of
a
dork,
but
anyway,
we'll
be
giving
these
away
at
the
end
of
this
particular
session,
with
lots
of
signatures
from
kubernetes
community
members
and
all
the
stickers
and
stuff
like
that,
so
yeah
I
again
will
not
be
wearing
my
Bob.
The
Builder
Hat.
A
Like
I
said,
we
had
to
build
a
a
factory
that
produces
kubernetes
controllers,
so
most
of
you
are
probably
familiar
with
with
coup
Builder
right,
which
is
the
the
Upstream
project,
that
sort
of
wraps
a
bunch
of
code
generation
tools
and
sort
of
templates
and
sort
of
cookie
cutter
stuff
for
creating
kubernetes
custom
controllers.
So
we
have
had
to
go
beyond
two
Builder
and
build
a
whole
bunch
of
Automation
and
additional
code
generation
tools
to
deal
with
our
specific
problem
that
we
had
in
the
ack
land.
B
Yeah,
so
before
talking
about
building
a
kubernetes
controller
Factory,
let's
talk
about
what
is
a
controller,
so
a
as
almost
all
of
you
know.
Right
now,
controllers
are
processes
or
programs
that
constantly
or
actively
try
to
reconcile
kubernetes
objects.
It's
an
infinite
Loop,
for
example.
It's
going
to
go
and
query
what
is
the
desired
State?
What
is
the
current
state
and
it's
going
to
make
actions
to
make
the
changes
and
move
our
current
state
towards
the
desired
state?
B
We
have
a
lot
of
controllers
and
kubernetes.
Almost
everything
in
kubernetes
is
just
controllers.
It's
a
smelting
pot
of
controllers.
Each
one
of
them
is
reconciling
a
different
kind
of
resources.
A
good
example
is
deployments
and
eight
points.
For
example,
whenever
you
create
a
deployment,
you
have
a
deployment
controller,
creating
a
deployment
objects
and
the
weakasset
objects,
and
then
behind
the
scenes
you
have
a
lot
of
other
controllers
that
will
go
and
create
your
objects,
including
the
pods.
We
have
also
job
nodes
and
even
name
spaces
are
created
or
managed
by
kubernetes
native
controllers.
B
And
then
you
can
also
build
your
own
custom
controllers.
For
example,
you
can
control
your
Home
Smart
bulbs.
You
can
have
a
crd
that
will
have
in
a
spec
whether
a
lamp
is
turned
on
or
turned
off,
and
you
can
manage
that
through
your
kubernetes
cluster.
You
can
also
order
pizza
shout
out
to
Michael
hasenblast
for
creating
the
pizza
controller
so
right
now
there
is
a
controller
where
you
can
order
pizza
using
crds.
B
You
could
actually
find
that
and
also
you
can
manage
Cloud
resources,
for
example
the
AWS
S3
or
Google
storage,
Google,
Cloud,
Storage
or
Azure
blob
storage.
You
can
manage
those
through
kubernetes
crds
and
even
you
can
manage
kubecon
cfps,
because
why
use
uis
when
you
have
Cube
cuddle?
It's
always
better.
So
if
you're
interested
into
learning
how
to
manage
cfps
through
kubernetes,
please
go
see
the
tutorial:
how
to
build
controllers
at
fortree,
430,
pm
and
yeah.
Next
is
your
toolbox?
B
If
you
ever
want
to
build
your
own
controllers,
you
for
sure
gonna
hit
or
gonna
see
some
one
of
the
known
tools
in
the
kubernetes
world.
One
of
them
is
Q
Builder
queue.
Builder
is
a
tool
that
helps
you
in
scaffold
in
your
controller
project
and
behind
the
scenes
it
uses
tools
like
controller
gen
and
controller
tools
and
other
examples
like
operator
framework
and
controller
runtime.
B
B
Work,
use
good
luck
with
that
fun,
facts,
deployment,
controller,
andropic,
asset
controller,
I,
think
also
the
stateful
set
one
are
all
written
using
these
basic
components.
They
are
not
built
using
controller
runtime
and
let's
one
of
the
latest
news,
I
asked
one
of
the
people
from
the
Sig
app
whether
they
want
to
move
to
what
controller
runtime
or
not.
They
will
not.
They
will
stay
in
this
library,
but
if
you
want
to
build
your
own,
it's
better
to
go
to
use
controller
runtime,
because
it
does
a
lot
of
work
for
you.
B
Okay,
so
now
we
know
the
tools,
let's
define
the
limits
or
the
boundaries
between
what
developers
need
to
do
and
what
the
libraries
and
the
tools
do
for
you.
For
example,
the
tools
and
the
libraries
will
help
you
generate
the
go
types,
the
crds,
which
will
also
help
you
generate
the
co
clients.
If
you
want
to
interact
with
those
with
API
server
to
query
or
modify
those
objects,
it
will
give
you
good
libraries
to
handle
login
and
leader
election
rate
limiting
web
Hooks
and
stuff
like
that.
B
However,
on
the
other
hand,
as
a
developer,
you
still
have
to
write
your
own
reconciliation
logic.
You
will
still
have
to
write
the
validation
and
the
mutation
webhooks.
You
also
have
to
write
units
and
end-to-end
tests
and
from
time
to
time,
you
will
have
to
fight
with
your
own
robots,
because
robots
are
not
good.
B
The
best
you'll
also
have
to
maintain
control
images
and
maintain
Helm
shots
and
documentation,
and
so
you're
going
to
come
back
and
fight
with
your
own
power
bots
and
do
a
lot
of
Slash
retests
and
slash
LG
GM
at
the
end.
So
as
a
developer,
there's
a
lot
of
logic
to
to
write
and
to
implement
and
yeah.
This
is
just
a
global
overview
and
what's
the
difference
or
what
what
are
the
boundaries
between
what
the
tools
do
and
developers
have
to
do.
A
A
Where
that
well
apis
change,
they
evolve
over
time.
They're
not
static
right,
and
so
we
needed
a
way
to
integrate
changes
to
the
Upstream
service
apis
in
a
smooth
and
consistent
and
reliable
fashion
right.
We
realize
that
we
could
go
ahead
and
generate
a
controller,
a
full
controller
implementation
for
something
like
S3
or
RDS.
But
next
month
the
RDS
team
or
the
F3
team
can
come
out
with
a
new
field
in
a
particular
resource
in
their
API
or
a
new
resource
entirely
right.
A
They
could
also
change
the
behavior
of
you
know
how
things
happen
on
the
service
side
and
that
behavior
change
would
then
roll
out
and
and
cause
conflicting
damage
to
all
of
the
controllers
that
were
generated
with
a
past
schema
or
model
for
that
API.
And
so
we
had
to
build
a
bunch
of
Automation
and
code
generation
tools
to
continually
keep
the
controller.
The
controllers
that
are
generated
up
to
date
with
those
evolving
API
schemas.
A
So
we're
gonna
well
amine
is
going
to
give
a
tour
of
our
little
controller
Factory
the
inputs
to
this
sort
of
like
controller
Factory
are
the
AWS
API
models
right,
the
the
definitions
of
the
apis.
They
contain
things
like
descriptions
of
the
operations
that
each
of
the
service
apis
have.
You
know
for
S3
like
create
bucket
Etc
and
then
also
descriptions
of
the
shapes
right.
A
You
can
kind
of
think
of
there
are
these
things
called
the
coral
API
models
in
AWS,
but
you
can
kind
of
think
about
it
like
an
earlier
version
of
open,
API
schema
right,
but
it's
just
sort
of
like
AWS
specific
anyway.
These
API
models
and
particular
generation
or
generator.yaml
configuration
file
contains
some
instructions
to
our
code
generation
framework
and
tools
to
how
to
generate
the
controller,
how
to
identify
what
resources
are
in
the
API
and
that
kind
of
thing.
But
the
end
output
in
our
Factory
is
a
full
controller
implementation
right.
A
So,
if
what
you
think
about,
if
you're
used
to
cube
Builder,
what
you
get
when
you
generate
a
controller,
is
basically
just
a
stub
right,
the
the
reconciler
logic
and
everything
you
still
have
to
go,
and
and
do
your
do
your
thing
you
have
to
implement
it.
We
didn't
want
to
do
that
for
200
controllers,
and
so
we
had
to
build
something
that
could
do
that
for
us
right.
B
So
so,
the
first
step
in
this
controller
Factory
after
giving
the
API
schema
and
the
generator
that
yaml
configuration
file
is
finding
what
are
the
resources
we
should
be
managing,
for
example,
on
the
right
here.
You
can
see
that
we
have
a
small
snippet
from
the
API
schema
for
dynamodb.
There
is
a
an
operation
called
create
stable.
So
the
first
thing
to
do
to
we
do
to
find
the
resources
is
to
find
every
operation
that
starts
with,
creates
strip
the
create
and
keep
the
rest,
that's
our
resource.
B
A
I'll
just
interrupt
and
say,
ameen
said
it's
just
as
simple
as
that:
I
wish
it
was
as
simple
as
that
any
of
you
are
familiar
with
AWS
apis
know
that
they
suffer
from
a
glorious
gloriously
like
idiosyncratic
inconsistencies
across
those
service.
Apis
right,
especially
like
the
behavior
of
like
update
code,
pass,
and
things
like
that,
because
you
know
all
AWS
Services
they're
like
little
startups
inside
of
AWS,
and
so
they
all
kind
of
do
their
own
little
thing.
So
anyway,
it
may
look
simple,
but
it's
absolutely
not.
B
So
after
inferring
that
table
as
a
resource
now,
we
need
to
find
what
are
the
fields
of
this
table,
resource
which
Fields
go
to
the
spec
and
which
Fields
go
to
the
status,
and
for
that
it
is
not
as
easy
as
go
into
the
create
table
inputs
and
create
table
outputs,
and
whenever
we
see
a
field
appearing
in
both
of
the
operations
we
know
this
is
spec.
Whenever
we
see
an
a
field,
that's
only
appearing
on
the
response.
We
know
the
status,
for
example,
table
sages.
B
It's
going
to
be
on
the
stages,
because
we
do
not
give
that
field
in
the
input
we
only
receive
it
in
the
outputs,
however,
table
name
it's
going
to
be
in
the
spec,
because
we
see
it
on
the
inputs
and
the
outputs.
At
the
same
time.
This
is
a
the
the
default
way
of
interfering.
What
should
go
in
the
spec
and
we
should
go
into
the
stages.
B
So
after
knowing
what
are
the
resources
and
what
are
the
fields,
we
write
all
this
information
into
apis,
V1,
Alpha
One,
and
you
can
see
here
that
we
have
inline
24
tables,
back
161
table
status
and,
of
course
the
object,
kubernetes
or
cube
builder
needs
to
know
that
these
are
kubernetes
objects
is
table
and
table
lists.
You
can
see
that
we
have
a
objects,
metadata
aspect
and
the
status.
B
Okay,
so
now
we
have
the
objects,
their
definitions,
we're
gonna,
do
a
quick
round
of
controllers
and
feet
are
queue
Builder.
So,
as
you
see
here
before,
we
have
a
few
markers,
key
Builder
objects
and
sub
resource.
Those
comments
will
be
used
with
control
in
controller
gen
to,
for
example,
know
how
to
how
to
input
the
printer
columns
and
the
validation
patterns
and
the
roll
scoping.
We're
also
going
to
leverage
controller
gen
to
generate
like
files
like
ZZ,
deep
copy.
B
So
at
this
stage
we
did
all
this
to
only
have
the
controller
skeleton.
There
is
no
reconciliation
Loop.
There
is
no
logic
at
all.
It's
just
the
definition
of
of
the
objects,
and
this
is
almost
at
the
same
level
where
Cube
Builder
is
going
to
put
you
it's
going
to
help.
You
only
manage
those
crds
or
like
the
types,
the
spec
and
the
stages.
A
So
earlier,
I
I
showed
you
a
little
bit
of
some
Snippets
about
those
API
model:
definitions
that
we
have
for
the
the
AWS
service
apis
and
how
we
take
those
model
definitions
and
determine
what
are
the
custom
resource
definitions
that
that
particular
controller
is
going
to
expose
and
what
are
those
custom
resource
definitions?
Look
like
right.
What
fields
are
in
the
spec
and
the
status
and
all
that
kind
of
stuff?
A
We
then
use
controller
gen
and
some
of
the
other
like
deep
croppygen
and
some
of
those
tools
to
generate
some
of
the
API
objects
and
infrastructure.
But
now,
like
Amin,
said
we're
at
the
stage
where
coup
Builder
basically
has
left
you.
You
know
to
implement
your
own
controller.
So
that's
where
the
fun
starts
for
us.
We
need
to
go
ahead
and
develop
a
translation
layer
between
the
kubernetes
representation
of
that
resource,
which
is
the
CR
the
custom
resource
and
the
AWS
SDK
go
representation
of
that
same
resource
right.
A
So
much
of
our
controllers
are
just
translating
between
the
kubernetes
representation
of
something
and
the
AWS
representation
of
something
as
you
go
ahead
and
develop
your
own
custom
controllers.
What
a
lot
of
times
you'll
be
thinking
is
there's
a
back-end
integration
API
and
you
need
to
sort
of
map
between
representation
of
in
of
resources
in
that
back-end
API
and
the
kubernetes
API.
A
A
So
what
I
want
to
show
here
and
it's
a
little
difficult
to
see
I
understand
you-
can
go
download
the
slides
afterwards,
but
all
of
this
code
that
you
see
is
entirely
generated
right.
We
have
a
method
in
our
resource
manager,
interface
called
SDK
find
and
it
essentially
Maps
between
the
kubernetes
resource
and
the
AWS
SDK
go
resource.
All
the
code
that
you
see
on
the
right
is
incredibly
tedious
code
to
hand
write
right.
It
also
is
extremely
error
prone
if
humans
are
like
manually
custom.
A
Like
writing
this
stuff
right,
it's
super
annoying
to
read
and
that's
why
we
code
generate
everything
that
we
can.
What
you
see
on
the
right
is
essentially
processing
a
response.
An
HTTP
response
from
the
AWS
SDK
go
finding
which
field
in
that
response
matches
the
field
in
the
spec
or
the
status
of
the
CR,
and
then
setting
that
value
right,
and
it's
super
annoying
to
have
to
handwrite
all
this
kind
of
stuff
trust
me.
We
tried
first
and
yeah.
A
A
A
B
So
one
of
the
functions
we
also
generate
is
called
manager
for
so
what
you've
seen
before
the
Delta
function
and
the
SDK
client
functions
are
just
methods,
part
of
an
object.
We
call
the
resource
manager
and
the
resource
manager
is
just
a
ghost
track
that
helps
the
controller,
manage
resources
in
the
cloud.
So
this
object,
AWS
resource
manager,
has
all
these
eight
or
nine
methods.
It's
got
instructions
on
how
to
read
a
resource.
How
to
create
One
update
one
delete
one
and
a
bunch
of
other
stuff.
B
A
That
that
reckon
that
common
run
time,
this
ack
runtime,
what
it
does
is,
makes
the
reconciliation
Logic
for
all
resources
and
all
of
the
controllers
consistent
right
and
that
logic
is
I
mean
to
simplify
it
a
little
bit.
It's
like
just
you
get
the
resource
from
the
kubernetes
API
server.
You
determine
whether
or
not
it's
you
know
a
new
resource
or
whether
it's
being
modified
or
deleted
and
all
that
kind
of
stuff.
A
B
Also
with
that,
we
have
a
metrics
common
library
that
reports
metrics
to
whatever
your
your
tool
is
used
to
collect
metrics.
We
have
also
like
common
error
handling
Library.
We
have
a
common
utility
for
conditions
and
we
have
some
common
crds
that
we
use
across
all
the
controllers.
For
example,
whenever
you
want
to
adopt
a
resource,
you
use
the
same
crd
across
all
the
controllers.
A
Yeah,
so
we
have
right
now
we're
we've
generated
21
controllers
that
are
actually
published
on
our
our
ECR
public
repository,
11
of
which
are
in
or
maybe
it's
12
that
are
in
G8
stage.
We
have
quite
a
few
more
controllers
that
are
up
there
in
our
in
our
GitHub
repository
that
are
in
the
process
of
being
built
out
by
AWS
service
teams.
A
A
Our
goal
is
to
have
these
resource
managers
that
we
sort
of
showed
you
some
code
for
earlier
to
have
those
be
go
packages
that
can
be
then
imported
by
cross-plane
or
terraform
right
that
all
they
do
is
just
handle
appropriately
in
a
very
declarative,
kubernetes-like
way,
the
interfacing
between
the
kubernetes
universe
and
the
AWS
universe
and
translating
between
those
things.
So
it
is
our
goal
not
to
be
like
you
know,
only
we're
only
going
to
do
AWS
stuff
or
whatever.
A
A
So
the
the
ECR
public
is
a
registry
that
can
house
any
oci
artifact,
so
we
automatically
publish
we
automatically
construct
the
helm
chart
for
a
particular
controller
and
all
of
the
yaml
manifests
that
go
along
with
it
and
publish
that
to
ECR
public,
along
with
repositories
for
all
of
the
controller
images,
the
container
images
themselves
and
also
automated
all
the
like
olm.
The
operator
lifecycle,
management,
stuff
and
Publishing
to
operator
Hub
and
all
that
kind
of
stuff.
A
We
have
a
a
test
infrastructure
repository
that
contains
virtually
all
of
our
automation,
and
so,
if
you're
interested
in
I
don't
know
getting
some
inspiration
to
automate
your
own
controller
building,
I
just
want
to
grab
some
code
that
you
think
will
be
will
be
useful.
I
have
some
links
at
the
end
of
these
slides.
Where
you
can,
we
can
go
and
see
and
just
grab
some
inspiration
or
grab
the
code
so.
B
And
for
those
of
you
wondering
what
is
the
two
percent
left
of
the
non-generated
code?
The
code
we
wrote.
The
reason
for
that
is
that
the
AWS
apis
are
like
writing
code
that
generates
code
that
handles
a
lot
of
resources
across
all
the
AWS.
Services
is
almost
impossible,
and
sometimes
we
have
to
manually
write
some
custom
bits
of
code
that
will
like
get
injected
in
some
specific
places
in
the
SDK
or
the
Delta.
For
example.
Let's
say
you
want
to
compare
a
map
of
a
structs
of
arrays
of
maps.
B
A
C
C
A
Want
to
open
up
the
runtime
and
show
where
the
finalizer
is
added,
so
in
that
common,
ack
runtime
as
part
of
the
reconciliation
logic
right,
the
reconciliation
Loop,
that's
common
to
all
the
controllers.
We,
the
very
first
thing
we
do
to
put
the
resource
under
management-
is
add
a
finalizer
to
the
custom
resource
right
which
essentially
protects
it
from
being
deleted
by
anything
else.
Until
we
remove
that
finalizer
right.
A
Child
resources
right?
So
we
have.
We
have
a
system
called
resource
references
where
you
can
reference
a
a
custom
resource
from
another
custom
resource,
and
we
have
this
resolve
references
functionality
that
will
wait
to
delete
or
update
a
parent
object
until
all
of
its
child
resources
get
to.
One
of
our
conditions
is
called
resource
synced,
and
we
can.
We
can
control
how
the
resource
manager
sets
or
determines
whether
a
resource
is
in
resource
synced
by
instructing
the
code
generator.
Okay
for
dynamodb
tables.
A
B
D
A
I
wish
so
yeah
it's
a
great
question,
so
I
mean
the
question
is
look
the
AWS
apis
are
not
declarative.
For
the
most
part.
Right
I
mean
it's
a
set
of
imperative.
Api
calls
right,
create
bucket,
put
life
cycle
policy
like
that
that
kind
of
stuff.
It's
not
the
patch
and
apply
type
behavior
that
we've
come
to
expect
from
kubernetes.
A
A
I
mean
if
you're
asking
like
what's
the
solution
specifically
for
AWS
Service
apis,
you
kind
of
got
to
take
them
one
at
a
time
and
figure
out
what
the
behavior
of
the
back
end
service
is
some
of
the
services
some
of
the
service
apis.
You
need
to
to
call
various
update
operations
in
a
specific
order,
and
if
you
don't
it
gets
mad,
others
you
can
call
in
parallel.
You
know
multiple
attribute
update
apis,
so
it
just
kind
of
depends
on
on
the
service
API.
A
E
A
Question
so
right
now
we
have
a.
We
have
a
sort
of
half
and
half
system.
We
have
this
thing
called
controller
bootstrap
which,
which
will
generate
all
of
our
our
end-to-end
tests,
are
written
in
Python,
there's
a
python
testing
framework
and
our
controller
bootstrap
code
generation
project
generates
the
basics
of
that.
But
then
it's
up
to
the
service
team
and
us
to
go
and
write
specific
end-to-end
tests
for
those
resources.
I
would
I
would
love
to
get
to
a
point
where
we
can
take
a
look
at
the
API
model.
F
F
A
I
mean
if
you're,
if
you're
asking
do
do,
I
do
I
view
a
future
where
we're
completely
co-generated,
where
we
can
take
an
API
model
definition
and
a
generator
configuration
file
and
just
let
it
go
yeah
I
can
see
that
future,
maybe
not
for
the
tests
right
now,
but
certainly
for
the
controller
implementation.
What
we
do
is
we
look
for
patterns
right
of
how
does
the
AWS
service
function
in
comparison
to
another
one
right?
How
does
RDS
and
elastic
cache
and
Amazon
mq
have
they
have
certain
patterns
of
API
usage?
A
B
G
First,
thank
you.
That's
really
cool
for
you
guys
to
share
this,
so
we
can
see
like
what
you're
doing
the
question
is
since
you're
talking
about
centralizing
a
lot
of
your
code
and,
like
it
looked
like
you
know,
20
controllers
so
far
bring
it
into
a
library,
and
everything
like
have
you
know.
Have
you
seen
a
noticeable
impact
from
centralizing
that
code?
Have
you
noticed
like
any
sort
of
reduction
in
like
bugs
across
all
the
different
controllers
or
whatnot
or
yes,.
A
Yeah,
we
absolutely
have
some
of
our
controllers
actually
started
out
life
as
hand
built
right,
like
the
sagemaker
operator,
I
started
out
life
as
the
kubernetes
sagemaker
operator,
something
like
that
and
a
whole
class
of
bugs
and
user
pain
and
inconsistencies.
Just
we
just
wipe
that
off
right,
because
when,
when
you
start
co-generating
everything
from
a
model
definition
you're,
not
handwriting
thing,
you
just
kind
of
remove
a
whole
classification
of
common
failures
and
bugs
so
yeah.
We've
definitely
seen
an
improvement
in
that
that's
really
cool.
Thank
you.
Yeah.
A
Sorry
we
we
do
want
to
give
away
our
Bob,
the
Builder,
hats
and
well.
I
had
a
trivia
question.
First,
one
to
raise
their
hand,
gets
it,
or
at
least
first
one
raise
your
hand
that
I
can
see.
Okay,
so
here's
the
question:
what
is
the
the
name
of
the
code
generation
tool
that
generates
the
runtime
object,
copy
and
copy
into
functions
for
crds.