►
From YouTube: The Future of Your CRDs – Evolving an API - Stefan Schimanski, Red Hat & Mehdy Bohlool, Google
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
The Future of Your CRDs – Evolving an API - Stefan Schimanski, Red Hat & Mehdy Bohlool, Google
CustomResourceDefinitions (CRDs) are a cornerstone of many extensions built on-top of Kubernetes. They allow to define custom API objects and the API server will handle their life-cycle. Most APIs start with one version, but sooner or later need to evolve while keeping compatibility with current clients/users. Kubernetes standard types do that through versioning and conversion between versions. This talk covers the corresponding evolution path for CRD based APIs. The talk will cover: - Why Kubernetes has multiple versions - New feature: CRD Versioning with Conversion - Round-Tripability Concept - How to update schemaless API objects to schemaful - Pruning and how to enable it - Defaulting - How to plan for Master Rollbacks when writing a CRD - Storage version and storage migration - Failure modes - Monitoring and managing a cluster with multi-versions - Kube builder support and Demo
To Learn More: https://sched.co/GrTo
A
So
good
morning,
everybody
we
are
pretty
impressed
by
the
number
of
people,
cadsr
developer
topic
and
to
see
so
many
people
interested
in
that
it's
just
amazing.
We
were
also
surprised
about
the
comic
I
think
everybody
has
seen
that
in
the
in
the
keynote
this
morning,
fantastic,
comic
and
yeah.
We
couldn't
stop
to
foe
it
as
a
first
slide.
Today,
yeah,
my
name
is
Jeff
Rancher
Manske
I'm.
Looking
for
that,
it's
part
of
my
work
is
on
CDs.
A
B
B
So
first,
let's
set
up.
Let's
set
some
terms
here.
We
are
talking
about
custom
resources.
There
are
two
ways
in
kubernetes
to
define
custom
resources.
One
is
true
custom
resource
definitions
that
probably
most
of
you
heard,
but
the
other
one
is
aggregated
api
server
and
yes,
our
documentation
called
those
resources.
Custom
resources.
B
Here
we
are
focusing
on
CR
DS,
we're
going
from
now
on.
When
we
are
talking
about
C
ours,
we
are
talking
about
defined
using
C
or
DS,
and
there
are
you
know
there
are.
Sometimes
these
terms
goes
wrong,
that
we
call
it
C
or
D,
versioning
and
C
or
D
conversion,
but
it's
actually
CR
versioning
and
see
our
conversion
to
c
rd,
so
c
rd
before
111
had
one
version
field
that
you
specify
your
version
and
group
and
coin
in
111.
B
We
start
adding
this
versions
list
that
is
going
to
actually
replace
the
version
field
version
field
was
required.
Now
it's
optional
and
if
you
set
it,
it
should
be
the
first
item
in
this
versions
list
and
if
it's
not
such
that
that's
fine,
it
will
be
defaulted
to
the
first
item
in
the
versions.
This
the
simplest
version,
definition
is
actually
named,
served
and
storage
for
each
version.
B
Name
is
obviously
name
of
the
version
that
going
to
be
using
the
in
the
URL
to
or
anywhere
else
we
are
referring
to
the
version
servlet
you
actually
disable
and
enable
this
version
and
I
will
talk
more
about
that.
Why
why
we
just
don't
remove
it?
Sometimes
you
can
just
remove
a
version
and
then
storage
is
specifying
which
version
is
going
to
be
used
when
we
are
storing
see
our
days
in
at
CD,
only
an
own.
Only
one
of
these
versions
can
be,
can
have
this
flag
on.
B
Only
one
version
should
be
a
specified
up
as
a
storage
version
other
than
those
three
simple
fields.
We
actually
starting
113
be
moved
other
CRD
fields
into
the
versioning,
remove
them
perversion.
First,
one
is
validation.
We
had
a
validation,
fill
that
top
level.
Now
we
have
a
schema
filled
at
each
level
as
each
version,
each
version
can
have
its
its
own
schema,
because
we
fought
for
backward
compatibility
and
for
to
make
sure
that
we
can
do
upgrade
the
dungarees.
We
did
some
validation
here.
B
If
you
set
perversion
fills
any
of
these,
you
cannot
set
the
top
level
one
so
either
top
the
other.
One
should
be
set
or
perversion
fields.
If
top
level
one
is
said,
also
use
for
all
versions.
If
perversions
fields
are
set,
the
top
level
should
be
empty.
Because
of
some
we
actually
look
at
many
upgraded
Ungar
in
scenarios.
There
are
scenarios
that
actually
an
all
the
operator
may
not
know
about
perversion
fields
and
may
try
to
patch
the
C
or
D
to
set
a
top-level
field.
B
In
that
case,
we
will
also
clear
perversion
fields
if
the
top-level
said
top-level
one
is
set,
but
other
than
that,
you
should
make
sure
that
either
top-level
or
perversion
fields
are
set
for
sub
resources.
We
did
the
same,
so
we
can
define
sub
resources
perversion.
This
is
very
actually
useful
example,
for
example,
you
want
to
add
status
to
a
new
version.
B
Adding
status
to
the
only
version
you
have
would
be
would
have
problems
because
you're
all
controller
may
not
work
status,
changes
the
behavior
of
of
your
object,
so
you
can
actually
define
a
new
version
and
enable
status
in
that
version.
Also
for
additional
printer
columns,
you
can
define
them
perversion
and
again
the
same
rule.
If
you
set
the
table
level,
one,
it
will
be
used
for
all
of
them,
and
these
are
serving
alpha
in
113.
I
will
hand
you
hand
it
to
Stefan
yeah.
A
A
bit
back
point
about
versions,
versions,
are
representations
of
the
same
resource
so
on
the
slide
here,
I
have
one
object,
that's
called
your
your
resource
and
it's
actually
the
same,
just
three
different
representations.
So
when
you
create
an
object
using
v1,
you
can
get
the
same
object
again
in
V
1,
beta
1
or
V
1
I.
For
one
this
example.
So
it's
important
it's
one
object,
just
representation,
nothing
else,
and
to
do
that,
we've
object
in
a
different
version.
A
Of
course
we
need
conversions,
and
this
is
the
topic
of
the
talk
and
you
can
imagine
that
maybe
your
controllers
creates
a
v1
and
the
user
uses
V
1
beta
1
and
another
controller
once
I
have
one
it's
very
important
that
you
don't
lose
information.
So
this
this
conversion
must
go
back
and
forth
between
versions
and
there
wouldn't
be
any
data
loss
because
it's
representation
and
it
it
must
be
yeah
without
any
information.
Was
this
conversion
is
very
deep
in
in
the
API
server.
A
So
you
can
imagine
that
we
saw
something
actually
when
we
saw
something
at
CD.
You
remember:
MIDI
shows
the
storage
version,
just
just
one
store,
which
version,
which
means
when
the
user
sends
a
v1
object
and
you
have
a
storage
version
for
B
1,
Peter
1,
so
version
was
the
object
is
converted
before
so
you
have
convert
from
basically
everywhere
and
if
you
look
at
this
picture
here,
it
can
be
that
in
on
one
request,
you
have
2
or
3
conversion
chords.
A
So
this
is
central
and
the
reason
again
why
you
have
B
to
be
very,
very
careful
to
to
have
no
data
loss
when
converting
for
native
resources.
This
is
implemented.
Obviously
in
go
so
what
conversion
it's
done
in
go
in
substitute,
ABI,
server,
4c
ADIZ!
We
couldn't
do
that
for
obvious
reasons,
so
we
needed
something.
Yeah
put
those
conversions
in
and
what
we
did.
We
proposed
to
use
the
rap
hoop
for
that.
A
B
We
talked
about
the
cell
phone
talked
about
the
conversion.
Well,
how
we
did
it
in
C
are
these
so
starting
from
111
when
we
introduced
the
versions,
we
assume
that
there's
no
conversion
in
all
the
object
would
be
the
same.
That's
the
first
version.
We
wanted
to
get
the
versioning
out,
so
people
can
try
it
in
that
one.
We
call
it
a
non
conversion.
That
means
the
only
thing
the
conversion
will
do
is
to
set
the
API
versions
of
the
crs
when
you
go
from
one
version
to
the
other
and
assume
the
schema
of
that.
B
Cr
is
staying
the
same
so,
for
example,
in
that
Windows
version.
If
you
want
to
add
a
field,
you
need
to
add
it
to
all
versions.
You
can
add
it
only
to
one
or
if
you
want
to
rename
it,
you
need
to
rename
it
everywhere
you
can.
The
schema
cannot
be
different.
If
it's
a
non
conversion,
then
we
actually
start
discussing
of
how
how
we
want
to
do
the
conversion
in
in
C
or
DS.
We
had
a
long
discussion
on
declarative
conversion,
how
to
define
it.
What
is
good,
what
is
good
operator?
B
B
B
This
is
in
113,
and
this
is
alpha
so
how
we
define
that
in
the
spec
of
the
C
or
D,
we
added
a
section
called
conversion.
The
first
fill
in
conversion
is
strategy.
It
will
be
defaulted
to
num
if
you
don't
have
this
field.
Yet
if
you
set
it
to
webhook,
that's
the
only
other
value
that
is
allowed.
If
you
set
it
to
webhook,
you
should
define
a
verbal
client
configuration
that
basically
will
point
the
API
server
to
the
service
that
implements
the
web
hook.
B
B
B
B
B
No
Jason
could
be
decoded,
that's
good,
so,
but
bills
keep
that,
for
it
should
be
something
there,
but
I,
don't
know
why.
So
we
try
to
actually
create
another
C
or
D
another
CR.
This
CR
is
actually
has
a
different
version.
V2
and
API
server
will
return
and
or
or
say,
I
can
I
can
convert
it
because
you
know
that
we
didn't
actually
deploy
our
web
cookie
F.
So,
let's,
let's
create
our
web
book
for
every
need.
B
Some,
our
back
rules
to
let
the
web
hook,
access
to
it
or
I
need
to
talk
about
this
verbal,
the
one
that
I
implemented
here.
There
is
one
actually
a
good
example
of
web
hook.
That
is
in
our
end-to-end
test.
You
can
look
at
that
is
simply
do
the
conversion
between
house,
port
and
host
and
port
it
divided
host
port
into
host
and
port
in
one
version
to
the
other.
This
one
I
implemented
here,
because
we
did
a
lot
of
you
know.
We
spend
a
lot
of
time
on
declarative
conversion.
B
I
I
told
we
can
I
can
actually
implement
a
web
hook
that
do
the
declarative
conversion
for
us,
so
the
one
that
I
implemented
is
a
declarative
conversion
based
on
this
schema.
So
basically
it
will
listen
to
the
C
or
DS
it'll
watch
the
RDS
and
cache
this
scheme
of
different
versions
of
c
or
DS,
and
there
are
some
tags
in
the
schema
that
tells
tells
the
converter
how
to
convert
from
one
version
to
another.
So
if
anybody
just
want
to
rename,
for
example,
that's
the
only
operation
is
supports
right
now
it
doesn't.
B
B
B
Okay
works,
so
let
me
just
follow
the
law.
So
basically,
what
what
is
sent
to
the
API
to
the
web
hook
is
a
web
hook.
Request
that
is,
has
a
desired
version.
Aps
ever
wants
a
v2
and
the
object
is
passing
is
actually
a
v1.
It
has
a
field
call
server
and
then
the
conversion
actually
renamed
that
to
pass
and
and
I
can
actually
show
you
the
CRD
to
see
how
I
defined
it
the
schema
of
the
v1
in
the
description
it
has
something
like
plus
conversion
from
stable,
v2
renamed
the
host
to
this
field.
B
So
let's
go
back
to
the
demo.
So
now,
if
I
create
cr2,
that
is
in
version
2,
it
will
create
it,
it
will
convert
it
to
be
one
and
a
story:
I
can
actually
get
it
in
v1
or
v2.
If
you
see
the
v1
has
server
and
the
v2
has
Haas
in
it
here
and
here,
I'm,
sorry
and
here
and
it's
fun,
let's
create
a
third
one
and
then
listlessly
stir
everything
when
you
list
it.
Actually
we
listed
in
v1
when
you
listen
v1,
because
the
storage
is
we
want.
B
B
For
this
conversion,
you
see
actually
that
a
list
of
objects
is
sense
to
webhook
for
conversion.
Three
objects,
sensible
for
conversion
and
three
object
returned
in
v2.
It's
very
important
to
keep
in
mind
that
this
this
list,
API
server,
can't
send
any
list
and
it
doesn't
need.
It
doesn't
mean
that
a
user
sent
at
least
request
that
we
are
sending
a
list
of
objects
to
webhook
for
any
kind
of
optimization.
We
can
send
more
than
one
object
to
the
verb
book
for
conversion
and
the
webhook
should
convert
them
independently.
A
A
When
we
add
a
new
birth,
that's
what
we
do
here
in
the
first
box
on
the
left.
The
question
is:
when
will
you
when
will
you
change
the
storage
River,
so
V
1,
V,
2
1
is
new,
it's
not
the
storage
version,
so
we
we
keep
storage
in
V,
alpha
v,
1
I
for
one
and
at
some
point
we
want
to
switch
as
the
question
is
when
so
I
switch
here.
So
we
want
everyone
get
sports
me
one
vita
one
gets
to.
A
This
will
not
have
any
effect
on
objects
in
HC
d,
so
at
CD
it
has
objects
in
V,
1
I
for
one
and
they
stay
there,
so
just
on
white.
So
if
you
update
an
object
or
create
a
new
object,
then
the
storage
version
has
an
effect
and
everything
which
is
written
to
a
CD
from
this
point
on
will
be
in
the
new
storage
version.
A
At
some
point
you
have
to
do
migration.
Obviously,
to
make
this
easier,
we
have
in
the
status
of
the
CAG
a
field
called
stored
version.
So
we're
like
what
all
versions
which
have
in
storage
version
at
some
point
in
time
and
you
can
use
that
yeah
to
know
when
it's
necessary
to
to
convert
all
objects
which
are
maybe
in
the
old
version
and
eventually,
when
you
decide
everything
is
converted
migrated-
and
you
don't
have
me
whenever
one
at
all
in
the
system
anymore.
So
it's
it
is
completely
updated.
A
A
It's
very
important
to
really
think
about
when
to
do
those
steps.
So
in
kubernetes,
when
we
introduce
a
new
version,
we
wait
quite
some
time,
usually
at
least
in
the
release.
Maybe
two
releases
until
we
switch
storage
versions,
so
easy
simple
users
have
to
be
able
to
hold
back
and
hold
back
means.
You
installed
server,
binary
or
controller
by
new
whatever,
and
it
doesn't
know
the
new
version,
but
there
might
be
versions.
Objects
in
the
in
the
system
of
the
new
version,
which
was
just
introduced
with
last
lease.
So
always
wait.
A
A
Storage
migration
I
sketched
it
already.
It's
very
important
to
keep
this
in
mind.
Changing
the
storage
work
in
the
CID
does
not
mean
at
CD
changes.
Everything
at
least
stays
the
same,
so
you
always
have
to
do
migration.
Migration
means
you
have
to
write
every
object
to
a
CD,
which
means,
as
a
simplest
case,
what
we
do
in
in
who
Anita
specific
resources
we
just
do
an
empty
patch,
empty
patch
means
nothing
changes
but
TSM
I
to
a
CD,
so
just
list
all
objects
patch,
all
of
them
without
any
change.
A
They
have
written
back
to
a
CD
and
you're
fine,
so
migration
is
done
at
the
moment.
This
must
be
done
by
the
developer
of
the
controller
of
the
CID
there's
a
tool
in
the
works
by
some
colleagues
and
API
machinery
for
generic
mechanism.
For
that.
But
it's
not
ready
I
think,
but
this
will
come
so
I
will
be
in
generic
official
way
to
do
that
at
the
moment,
just
be
prepared
to
do
migrations
in
your
own
code,
I.
A
Mentioned
already
a
couple
of
times:
this
is
about
data
of
the
user,
so
conversion
means
you
change
representation
of
data
and
do
not
lose
anything
lot
of
yield,
not
a
comma.
It's
really
important
to
get
conversions
right.
It's
not
easy!
Everybody
who
has
done
that
outside
of
CDs
in
native
resources
to
get
some
whites
might
take
days,
many
reviews
and
many
tests.
Here's
one
pattern,
which
is
it's
separate
in
upstream
in
Internet
resources
we
call
it
our
tube
tests.
So,
as
idea
is
very
basic,
very
simple,
we
start
with
a
v1
object.
A
In
this
example,
we
put
in
just
random
data,
so
we
have
something
that's
called
a
father
or
fussing,
so
we
have
a
function
which
can
further
object
with
random
swings
and
all
the
structures
to
us
were
randomized.
Then
you
convert,
for
example,
to
be
1.
Peter
1
to
this
middle
one
here
and
you
come
to
convert
back
to
V
1,
so
back
and
force
for
born
were
v1
to
another
version
and
back
to
everyone,
and
if
conversion
is
correct,
it
should
be
the
same
object.
So
this
test
can
be,
of
course,
repeated.
A
We
do
that
a
thousand
times
with
random
objects,
and
if
this
survives,
you
conversion
survives
this
test.
You
can
be
pretty
sure
that
it
works,
so
our
tube
testing
is
very
important.
There
are
many
many
examples
in
kubernetes:
it's
a
source
code.
You
can,
you
can
follow.
We
also
have
some
some
library
kind
of
testing
Bible
on
that.
So
just
take
a
look
at
the
types
we
have
in
kubernetes
for
examples.
B
So
again,
as
Stefan
said
round,
trip
ability
is
very,
very
important
and
I
will
discuss
it
before
that.
I
was
just
going
back
to
what
I
showed
you.
Actually
the
served
version
and
I
told
you
can
just
remove
a
version.
That's
because
in
the
upgrading,
your
storage,
when
you
are
done
with
a
version
and
you
want
to
remove
it,
you
should
first
set
the
flag
to
false.
B
You
cannot
remove
it
because
the
version
is
still
available
in
s
CD
and
a
PI
server
will
not
to
let
you
do
that,
because
the
stored
version
of
the
status
has
the
version
and
it
will
not
allow
you
to
remove
the
version.
So
what
you
should
do
is
disable
the
version,
so
no
other
new,
a
request
comes
for
that
version.
Upgrade
your
storage
update
the
status
stored
version
to
remove
that
version
from
it
first
and
then
remove
it
from
there
version
space.
So
what
can
go
wrong?
B
Your
webhook
may
stop
working
for
any
reason.
You
know
there's
no
connection
between
a
PI
server
and
webhook.
What
will
happen
is
very
important
to
understand
the
web.
Who
calls
is
on
demand,
so
if
they
are
failing,
everything
else
works
just
that
C
or
D
that
C
or
that
you're
trying
to
access
an
e
and
needs
conversion
will
fail.
So
other
types
works.
If
you
request
for
the
C
are
in
the
same
version
as
a
storage,
it
will
work
if
you
try
to
store
it
in
the
same
version
as
storage.
B
It
will
work,
but
if
it
needs
a
conversion,
it
will
fail.
So
for
emergency
you
can
actually
try
to
get.
The
C
are
in
different
versions
to
see
which
version
is
the
storage
version,
even
if
you
don't
know
that
sometimes
looking
at
C
or
D
doesn't
help
you,
because
you
don't
know
when
they
see
are
when
they
see
or
is
stored.
B
What
storage
version
was
at
that
time,
but
you
can
just
try
a
different
version
to
see
which
one
it
working
and
patch
it
or
do
whatever
we
want
with
it
to
make
it
work,
but
you
should
make
the
web
book
works
as
soon
as
possible,
and
conversion
can
fail
for
a
subset
of
C
or
sued.
Your
web
book
may
still
work.
For
you
know
the
connection.
Can
be
available
between
a
PI
server
and
the
web
book
what
the
vehicle
conversion
may
fail
for
some
CCR's.
The
same
concept
only
fails
for
those
ers,
nothing
else.
B
The
the
last
one
is
actually
webhook
returns
that
conversion
here,
it's
something
that
is
not
drawn
three
pible
and
again
putting
a
stress
on
this,
because
this
is
a
failure
that
you
don't
see
if
something
goes
wrong.
This
is
a
logical
failure.
You
don't
see
an
error
from
cue,
cuttle
or
an
api
things
will
work,
seems
to
be
working
normally,
but
you
will
have
data
loss
users.
Data
does
so
very,
very
important
to
run
tests
on
run
trip
ability,
because
this
is
not
something
that
you
can
catch
in
production
easily.
B
So
we'll
have
more
more
about
this
in
capi
machinery,
deep
dives,
if
you
are
interested,
please
come
and
available
to
discuss
more
implementation,
details
of
that.
What's
beyond
this,
what
are
we
going
to
work
on
in
2019?
So,
basically,
the
plan
is
to
make
the
C
or
D
conversion
SCR
these
generally,
as
a
ga
feature.
They
are
a
couple
of
you
know,
blocking
issues
that
we
are
working
on.
Cr,
D
conversion
and
versioning
is
still
alpha.
B
It
should
be
better
in
1:14
and
it
should
be
GA
version
or
two
after
that
there
is
an
issue
586
a
2
if
you're
interested
just
look
at
that
issued
or
list
of
things
that
we
are
working
on
these
stuff
blockers
and
lists
of
things
that
we
wish.
We
want
to
work
on,
and
some
people
like
to
work
on,
but
the
main
ones
that
blocks
GA
is
generating
of
an
API
spec
for
C
or
DS,
actually
open.
If
you
see
are
these
needs
to
actually
provide
their
schema.
B
True
through
the
validation
schema
that
we
are
calling
a
schema
perversion
right
now,
but
we
are
not
serving
it
if
you're
providing
the
schema
is
not
served
for
kube,
Cottle
and
other.
You
know
other
client
generator,
for
example.
So
that's
something
that
we
are
working
on
it
barely
actually
missed.
1:13
is
going
to
be
in
114.
Kota
support
is
very
important,
defaulting
and
pruning,
something
that
we
have
a
designer
for,
and
the
conversion
conversation
is
actually
going
on
on
it
and
there
are
a
number
of
smaller
issues,
but
these
are.
A
This
issue,
which
is
listed,
there,
has
many
many
more
topics
and
some
days
ago
it
was
updated.
So
we
have
discussion
what
is
really
needed
for
GA
and
as
media
showed
those
those
other
points
here
there.
Many
more
and
some
of
them
are
completely
undecided,
which
means
there's
no
consensus.
Most
of
them
have
no
proposal
attached.
So
I
listed
some
here.
If
there
are
some
which
are
really
critical
for
you
come
to
us
come
to
the
sick
meeting.
A
If
you're
able
write
a
proposal
for
that
starts
a
discussion,
we
know
we
cannot
do
all
of
that
in
to
2019.
That's
why
we
have
selected
so
was
here.
Those
are
we
leave.
We
won't
go
to
GA
before
those
are
sort
of
that's
basically
the
plan
or
what
you
want
to
do
ya
see
a
conversion,
so
this
is
kind
of
a
take
away
slight,
so
they're
books,
that's
what
we
have
now.
It's
IFR
this
means
in
1:13.
You
have
to
enable
it
manually,
there's
a
feature
I
get
for
that
you
can
play
with
it.
A
A
B
What
I
you
know
discussed
what
most
of
the
discussion
was
around
the
declarative
conversion,
the
people
that
like
that
was
mostly
because
of
the
round
trip
ability,
because
we
can
actually
make
sure
a
declarative
conversion
is
round
trip
able
but
Vanessa
web
book.
We
as
API
server
as
kubernetes,
cannot
have
any
guaranteed
as
we
are
calling
into
a
web
hook
and
the
conversion
is
happening
there.
So
it's
up
to
the
web
developer
to
make
sure
that
it's
renewable
and
the
conversion
is
logically
correct.
B
A
Yeah,
so
we
have
admission
mutating
admission:
webhooks
can
be
used
for
defaulting
today,
so
there
is
a
way.
Admission
is
not
really
where
you
should
do
that.
I'm
a
I'm,
a
few
point
of
you.
The
proposal
we
have
is
about
declarative,
so
they
have
in
the
open,
API
validation.
You
get
just
a
field
default
and
you
want
to
apply
that.
A
B
A
It
yeah
it
depends
a
lot
on
the
operation
you
use.
All
operations
should
be
should
have
an
inverse
right.
This
can
be
easy
in
some
cases,
rename
can
be
in
the
inverted
of
obviously,
but
there
are
cases
where,
as
it's
not
possible,
there
are
long
long
discussions
for
native
types.
If
this
happens,
sometimes
one
has
to
accept
it.
That
something
wakes
can
also
happen
in
the
example
is
host
port
so
having
host
and
port
in
one
version
and
having
one
string
proposed
in
the
other
one.
It's
also
important.
A
He
talked
about
we
only
one
top-level
schema
or
especially
Birds
in
schema,
not
goes.
We
cannot
mix
them.
There's
a
long
discussion
about
that.
Maybe
it's
it's
nice
to
read
that
to
really
understand
how
to
solve
this
case.
So
it's
complicated
and
one
has
to
put
invariance
I
want
two
types
like
this
is
a
top
level
or
version
to
make
it
a
little
bit
so.
B
It's
important
to
look
at
it
this
way.
Yes,
the
CR
DS,
if
you're
talking
about
the
conversion
of
see
ours
here
but
C
or
D,
is
an
object
in
our
system
and
that's
we
need
to
be
around
three
people,
but
when
we
store
it
still
in
the
old
version
and
run
going
from
old
version
to
new
version
and
coming
back
to
the
old
version
is
always
round-trip
ago.
B
Right
now,
when
we
change
the
storage,
then
we
need
to
actually
keep
her
versions,
even
if
many
we
are
going
to
v1,
even
if
you're
doing
that,
we
need
to
somehow
keep
the
information
to
make
it
round
three
people,
but
right
now
we
are
storing
in
in
all
versions.
So
we
don't
have
the
round
strippable
problem
yet,
but
if
we
store
in
new
version,
that's
yes,
maybe.
A
The
answer
for
your
question,
so
we
have
we
had
cases
many
cases
where
we
use
annotations,
so
we
move
new
fields
into
annotations
in
the
old
version.
That's
one
solution:
it's
not
secure
because
those
are
not
protected,
sometimes
for
native
resources,
so
there
are
many
consequences
in
other
cases.
We
also
edit
the
new
fields
in
the
old
position
just
to
makes
us
want
to
probability
possible.
So
you
change
the
old
version.
Clients
will
maybe
just
to
observe
you
I
mean
the
feeds.