►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Apiserver 生成器：通过聚合 Apiserver 扩展 Kubernetes 系统 | Apiserver Builder: Extending Kubernetes via Aggregated Apiserver - Min Kim, Ant Group
目前有两种可插拔的方式来扩展集群的自定义 Kubernetes 应用程序接口：自定义资源定义（也被称为 CRD）和 Apiserver 聚合（也被称为 AA）。经过多年的发展，自定义资源定义 (CRD) 现在在几个版本的 GA 阶段处于稳定工作状态，但另一方面，我们可能会发现自定义资源定义由于可扩展性有限，并不总是最佳选择——例如，我们将不得不引入多个网络钩子 apiservers，以便在新的资源类型上建立准入/转换。或者，为了保持良好的兼容性，在多个版本之间切换可能在技术上很困难。因此，如果我们希望以编码复杂度为代价实现软件开发工具包框架级的扩展，我们可以将 Apiserver 聚合作为自定义资源定义的替代方案。我作为这个特定的官方 SIG 子项目的维护者已经 3 年多了，我将通过一个名为 apiserver-builder 的强大命令行工具，与观众分享构建您自己的聚合 apiserver 的实用方法。
Currently there're two pluggable ways of extending custom Kubernetes API for your cluster: Custom Resource Definition (also known as CRD) and Apiserver Aggregation (also known as AA). After years' of evolving, CRD is now stably working in GA phase in steady state for several releases but on the other hand we may find that CRD is not always the best option due to its limitted extensibility --- e.g. we will have to introduce multiples webhook apiservers in order to build up admission/conversion over the new resource types. Or it can be technically tough to rolling between multiple versions for with compatibility well-preserved. So we may consider AA as an alternative of CRD if we want sdk-framework-level extensibility at the cost of coding complexity. Being the maintainer of this particular official SIG subproject for over 3 years, I will be sharing audience the practical way of building your own aggregated apiserver by leveraging a powerful command-line tooling named apiserver-builder.
A
A
My
whole
presentation
is
going
to
start
with
a
very
detailed
introduction
about
the
technique
of
api
server
aggregation
per
se
and
in
the
next
one,
I'm
going
to
expand
the
details
about
the
subprojects
named
api
server
builder
and
also
it's
a
successor
news.
A
project
named
api
server,
runtimes
and
in
the
end,
they've
also
been
qa
sections
in
which
I
will
be
addressing
all
the
questions
from
you.
If
there's
any
and
the
time
is
limited,
so
we
can
probably
extend
our
q
a
sections
to
the
slack
channels
over
there.
A
You
can
just
dm
me
or
whatever
you
want,
and
okay,
I'm
gonna
start
with
a
brief
introduction
about
myself.
My
name
is
mein
jin
and
I'm
a
software
engineer
working
at
alibaba
clouds
and
and
I'm
also
a
kubernetes
machinery
working
in
siege
api
machinery,
I'm
the
project
owner
of
multiple
kubernetes
subjects
and
I'm
also
the
speaker
of
kubica
2019,
europe,
2019
2019,
north
america,
2020,
north
america
and
besides
kubernetes,
I'm
also
actively
running
the
project
of
open
class
manager.
A
A
Okay,
let's
move
on
to
the
second
part
of
to
start
the
first
part
of
my
presentation.
It's
the
background
about
api
server
obligation
over
the
years
of
community
support.
I
found
many
developers
are
stuck
in
a
builder
complaining
that
a
technique
is
too
complicated
to
understand,
and
most
of
them
are
starting
at
the,
for
example,
the
installation
of
api
server
applications
or
the
the
management,
the
management
of
certification
and
or
something
like
that.
So
today.
A
A
Then
the
kubernetes
cluster
will
automatically
resolve
the
endpoints
for
you.
As
for
how
to
build
an
a
server,
there
are
basically
three
physical
approaches.
The
first
one
is
just
copy
and
pasting
from
another
official
project
named
sample
api
server,
and
but
the
project
is
actually
hard
coded,
so
you
will
have
to
edit
the,
for
example,
the
resource
name
of
the
the
servers,
and
the
second
approach
is
extending
the
api
servers
where
the
scaffoldings
provided
by
the
api
server
builder,
which
will
be
on
similar
to
the
user
user.
A
Experience
of
how
cookie
builder
works
and
the
the
third
phase
by
the
way
will
be
just
importing
the
the
libraries
of
api
server
runtime,
which
I
will
elaborate
later.
You
can
just
initialize
a
brand
new
column
project
and
just
add
the
api
server
runtimes
into
the
dependencies
and
the
magic
will
happens.
A
As
far
as
for
the
insights
of
a
api
server
applications,
we're
gonna
expand
the
the
internal
technique
from
four
aspects
of
the
authentication
and
authorization,
emission
controls
and
class
panelists
the
api
priority
firmness
by
by
leveraging
the
technique
of
aaa,
you
can
basically
do
any
kinds
of
customization
customizations.
If
you
want,
for
example,
you
can
even
replace
the
persistent
layer
for
an
api
server,
the
normally
the
storage
layer
of
an
api
server
is
supposed
to
be
etc.
A
But
you
can
you
by
by
leveraging
a
you,
can
just
replace
it
with
any
kinds
of
other
storages
you
want,
and
this
picture
shows
how
na
works
in
the
architecture
so
and
a
server
is
a
standalone
api
server
outside
of
google
api
server
and
by
the
spec
of
api
service.
You
can
make
the
traffic
going
through
the
qbf
server
successfully
proceeded
to
to
the
to
the
downstream
of
a
servers,
and
you
can
even
develop
your
custom
controllers
working
based
on
the
au
servers.
A
The
the
custom
controllers
are
supposed
to
connect
a
servers
while
the
proxy
of
2d
ips
server.
Instead
of
requesting
direct
speed,
however,
direct
requesting
will
also
be
working,
which
I
will
also
be
explaining
later
so
before
we
dive
into
the
authentication
of
apis
of
application.
We
have
to
understand
how
the
trs,
the
general
technique
of
tis
works.
Basically,
there
are
two
kinds
of
drs
in
in
in
the
world.
A
Subject:
alternative
names,
which
is
a
list
of
domain
names
or
ip
addresses,
so
so
the
the
essay
basically
restricts
the
usage
of
of
server
certificates
and
as
for
the
mutual
trs,
in
addition
to
vanilla,
tls
of
the
mutual
trs
as
another
round
of
verifications,
which
is
the
server,
will
also
verify
the
the
client
certificates
from
from
the
clients.
So
so
there's
a
there's
a
trusted
authorities
in
memory.
A
Inside
of
servers,
the
servers
is
supposed
to
reject
any
any
requests
that
doesn't
doesn't
contain
doesn't
have
the
client
certificate
signed
by
that
ca.
So
this
is
basically
how
the
mutual
tis
works
in
the
indian
there's
also
an
a
optional
step
of
verifying
the
concepts
in,
but
actually
in
the
kubernetes
native
libraries
is
skipped
by
defaults.
A
Okay,
after
after
knowing
how
tis
works,
let's
take
a
look
at
how
authentication
works.
This
picture
is
the
typical
case
when
a
client
is
trying
to
request
a
a
server,
the
proxy
of
kubi
api
server.
So
the
normal
authentication
happens
between
that
between
the
kubic
console
and
the
could
be
api
server.
The
the
qb
api
server
will
will
check
whether
the
client
certificate
is
signed
by
a
ca
authorities
which
is
specified
by
a
flag
named
the
desktop
client
ca
file.
A
And,
however,
this
is
rather
simple.
But,
however,
let's
take
a
look
at
the
authentications.
The
mutual
gos
happens
between
the
kubiak
server
and
the
aggregated
api
server.
The
a
server
is
actually
it's
sorry,
it
could
be
a
data
server.
It's
actually
is.
Is
the
client
when,
by
processing
the
request
to
the
a
servers,
so
the
client
verification
happens
when
the
a
server
returning
the
server
certificate
back
to
the
kubi
api
server.
A
So
the
decoupage
server
will
check
whether
the
server
certificates
of
a
server
is
signed
by
the
dca
bundles
specified
in
this
api
service,
and
the
a
server
will
also
perform
another
round
of
server
verifications
by
checking
whether
the
client
certificates
from
the
queue
api
servers
are
checking.
The
client
certificates
is
defined
by
the
ca
authorities
specified
by
the
dash
dash
to
crash
charter,
ca,
client
file,
the
flag
and
optionally.
A
If
the
a
servers
are
deployed
inside
of
kubernetes
clusters,
you
can
also
looking
for
a
config
maps
under
the
namespace
of
kuby
system
and
then
the
extension
api
server
authentications.
There's
a
key
named,
requested
cva
file.
Then
the
content
will
be
on
the
expected
signer
of
client
certificates,
and
this
is
the
this
is
the
indirect
approach
when
a
client
is
accessing
any
servers
where
the
crosshair
communicator
server.
A
However,
client
can
also
access
a
a
server
directly
without
the
result
proxy
by
the
qb
server,
or
in
this
case
the
case
can
be
different
when
the
clients,
if
a
client
says
requesting
using
a
signed
certificate,
then
there
could
be
aps.
Then
the
aggregated
api
server
will
just
checking
whether
the
the
decline
certificates
by
by.
A
So
this
is
how
the
authentication
works
between
kubi
api
server
and
na
server,
which
is
a
bit
complex.
Then
then
they
are
density
of
simple
cookie
cutter
on
could
be
api
server
authentication
then,
after
the
authentication,
the
the
left
ddd
directly.
The
next
thing
will
happen
is
the
of
the
authorization
the
authorization
works
inside
a
server
is
pretty
easy.
First
of
all
for
clarification,
there
could
be
api
server,
we're
just
forwarding
all
the
requests
to
any
servers
without
performing
any
kinds
of
authorization
check.
A
So
all
the
authorization
checks
are
happening
inside
of
a
servers,
but
dda
server
doesn't
contain
any
logic
of
authorization.
So
again
the
a
servers
will
be
delegating
the
authorization
back
to
include
the
api
server.
They
are
in
dedicated
api
named
subject,
access
review
and
the
results
of
authorization
check
will
be
cached
inside
of
a
servers
and
and
also
we
can
also
wipe
off
the
authorizations
in
the
a
servers.
A
A
This
is
how
authentication
works.
It
just
works
in
the
flow,
the
visual
tis
between
clients
and
qb
of
data
server.
Then
the
mutual
tiers
between
the
could
be
ipa
server
and
the
a
servers
and
then
upon
authorization.
The
a
servers
will
be
delegating
the
authorizations
back
to
kubiak
server
and
returning
the
responses.
A
A
A
Okay,
after
understanding
the
details
of
implementation
of
a
servers,
let's,
let's
take
a
step
back
and
take
take
a
look
at
the
crds
in
a
higher
level.
Let's
take
a
comparison
between
crds
and
a
the
crd
is
basically
a
declarative
api.
However,
the
aaa
is
implantative,
though,
by
by
the
declarative
api,
we
can
easily
define
the
schema
of
cr
crds
by
using
the
grammar
of
oas
or
the
old
swagger,
and
but
there
are
some
inconvenience
when
we
are
trying
to
to
develop
a
multiversion.
A
A
So
when
you
are
trying
to
extend
your
arbitrage
of
resources,
then
it
will
be
probably
something
you
are
looking
for
and
the
technique
a
can
help
us
to
replace
the
etcd
as
the
persistent
layer
we
can.
We
can
build
dna
working
based
on
mysql,
other
kinds
of
sql
storages
or
even
you
can
be
working
on
a
local
file
in
spots
on
the
dark
slide
of
a
is
that
we're
going
to
define
our
schemas
in
golang,
and
it
requires
of
actual
work
of
coding
and.
A
The
process
of
resources
is
already
used
in
the
in
the
kubernetes,
such
as
pulse
proxy
service
proxy.
So
if
you
are
looking
for
a
custom,
proxy
implementation
or
other
any
kinds
of
some
resources,
then
you
should
probably
try
a
and
also,
if
you
are,
if
you
want
to
get
rid
of
ecd
storage,
I
think
the
a
will
be
the
only
choice
to
go
so
by
replacing
the
storage
layer
of
the
etc.
A
So
in
the
past
they
are
a
bigger
general
debates
between
crds
and
na
and
the
at
that
time
the
crds
na
they
are,
they
are
neither
mature
and
developers,
are
having
trouble
getting
on
board
any
any
any
of
techniques.
So
the
query
builder
is
actually
trying
to
providing
users
a
converging,
abstraction
layer
that
eliminates
the
gaps
because
there
it
is
in
a
so.
The
users
will
be
just
facing
running
running
on
the
obstructions
provided
by
the
computer
builder
or
guest
over
builders
and
make
let
the
builders
do
all
the
rest
to
do
that.
A
To
do
the
dirty
things
for
you,
the
the
second
goal
will
be
scaffolding.
The
scaffolding
basically
covers
first
of
all,
is
the
templating
of
your
projects.
No
matter
is
some
controller
or
in
a
servers,
and
also
we
need
a
scrapbooking
to
setting
up
the
local
testing
environment
in
the
past,
setting
of
having
a
local
environment
that
you
say
actually
luxury
things,
because
it
involves
too
many
dependencies.
A
So,
as
you
can
see
in
the
peak,
actually,
the
cookie
builder
is
forks
from
the
api
server.
In
the
very
beginning,
the
picture
is
the
very
first
valid
commit
of
kobe
builders,
so
the
original
idea
is
that
the
the
kuby
builders
is
supposed
to
learning
or
the
defects.
Learning
the
fading
lessons
from
the
api
servers,
then
I'll
have
a
better
building
toolings
for
the
developers
and
after
the
kobe
builder,
then
we
have
the
controller
runtime.
A
A
The
sample
controller
is
very
well
documented,
but
it's
not
user-friendly
for
developers,
because
you
will
need
to
re-read
the
internals
and
every
every
developers
may
have
their
own
understanding
of
how
a
controller
works.
After
a
period
of
time,
we
extracted
the
gold
template
from
the
sample
controller.
So
we
don't.
We
don't
have
to
manually
copy
and
paste
it
from
the
simple
controllers
anymore,
but
it's
still
not
user
friendly,
because
we
don't
have
the
unified
obstruction
of
controller.
A
So
in
the
end
we
had
so
so
so
we
initiated
the
project
control
runtimes.
If
you
notice
many
packages
inside
of
controller
runtimes
are
are
defined
as
golang's
internal
packages
and
that's
the
point
of
hiding
the
internals
hiding
the
underneath
details
from
the
users
as
much
as
possible,
and
the
controller's
controller
and
times
have
have
a
few
have
its
standardized
interface
of
the
controller.
A
And
it's
a
no
easy.
It's
an
easy,
sdk
framework
for
developing
and
testing.
Yes,
the
testing
is
also
covered
in
the
control
runtime.
However,
however,
their
the
dark
side
of
controller
runtime
is
that
some
packages
is
conflicting
with
the
native
libraries
from
the
kubernetes,
but
in
sometimes
the
diverging
can
be
misleading,
because
sometimes
the
abstractions
in
the
control
runtimes
are
just
not
correct.
A
So
I
think
you
you,
you
can
get
it
the
first
draft
level
api
server
runtime
is
drafted
in
2018
by
by
the
founder
of
controller
runtime,
soli
and
the
source
is
still
available
in
his
repositories
and
now
it's
incubating
into
a
official
kubernetes
subject,
which
is
now
publicly
available
here,
and
the
different
thing
comparing
to
the
controller
time,
is
that
the
api
server
runtime
is
completely
honoring.
The
original
implement
original
implementation
of
sample
api
server.
A
Overall,
the
api
servers
will
be
a
project
of
doing
scaffoldings
or
api
server
runtime,
which
is
similar
to
the
relationship
between
code
builder
and
controller
runtime,
and
also
the
api
server
builder
provides
users
a
set
of
useful
utilities,
the
command
lines
to
operate.
The
ai
servers
inside
your
kubernetes
clusters,
as
for
templating
or
the
code
generations,
note
that
the
api
server
builders
will
only
do
those
one
shot
code:
generations
for
you,
including
the
project
initializations
or
adding
your
custom
resources.
A
All
of
these
work
will
just
happen
once
and
it's
not
consistent
and
as
for
the
scope
of
api
server
runtime,
it's
aiming
at
being
a
user-friendly
library
for
building
aggregated
server,
and
it's
also
providing
one
light,
switch
two
links
for
code
generations,
which
is
the
consistent
code
generations
not
comparing
to
the
ips
server.
The
api
server
does
one-time
things,
but
in
the
api
runtimes,
the
decode
generation
is
actually
referencing.
A
A
Then,
let's
take
a
look
at
how
the
original
sample
api
server
works,
the
code
from
the
code
codewise
in
the
code.
You
can
see
that
the
rust
storage
implementation
is
initialized
is
instantiated
manually
in
the
code.
You
need
to
organize
or
organize
all
the
resources
into
a
into
a
golan
map
which
is
a
bit
hard
to
understand.
Then
you
can
just
install
the
api
groups
into
generic
api
servers,
but
ideally
we
hope
users
not
to
touch
any
internal
details
like
this.
A
A
This
is
pretty
common
in
the
in
the
java
world,
so
we
found
this
it's
really
useful,
so
we
just
decided
to
moving
moving
it
to
the
golden
world.
So,
as
you
see,
we
can
just
build
our
own
custom
servers
in
one
line
by
just
by
registering
the
resources
and
tuning
a
few
configurations,
and
we
can
also
adjust
the
flag
registrations
or
other.
A
A
If
you
are,
if
your
ide
is
supported,
you
can
just
research
your
custom,
api
types
with
a
resource,
store
objects,
interface
and
assert
your
this
type
with
resource
store,
object,
list
interface.
Then
then,
after
you,
you
make
the
interface
session
works.
Then
new
kubernetes
resources
are
supposed
to
looking
like
a
charm
and
optionally.
This
is
the
mandatory
the
required
step
of
developing
a
custom
resources,
but,
however,
there
are
some
optional
steps
of
advanced
on
customizations,
for
example,
the
validators.
A
You
can
assert
your
custom
type
with
a
interface
named
validator,
then
in
the
in
the
interface
in
the
implementation
of
validator,
you
can
do
advanced
logics,
like
crossfield
validations
or
any
kinds
of
non-trivial
validations.
If
you
want
and-
and
this
picture
also
shows
you
the
how
to
develop
a
non-native
cd
api
servers
using
the
apis
or
runtimes
ddr,
as
you
can
see
in
the
independent
flow,
we
can
call
without
eccd
methods,
then
this
will
be
completely
disabling,
etc.
A
A
As
you
can
see
in
the
picture,
I
asserted
the
subresource
with
an
interface
named
connector,
then
the
implementation
of
connectors,
the
connect
method
will
just
returning
a
http
docs
handler.
The
registered
api
is
under
this
group.
After
all,
after
setting
up
the
aad
connection
between
kubernetes
and
servers
and
a
then
the
qb
api
servers
will
be
understanding.
A
First
of
all,
I'm
gonna
show
you
how
to
crawl
prop
the
healthiness
of
cluster
without
there
using
the
process
of
resource
okay.
This
is
how
the
process
sub-resource
works.
The
process
of
results
will
be
directing
redirecting
order
requests
under
the
under
under
the
proxy
subresources
to
the
to
the
actual
real
cluster.
Okay.
I
will
leave
my
time
to
the
last
q,
a
sections.