►
From YouTube: Building a Day1/Day2 Application Operations Platform On CNCF Projects - Alois Reitbauer & Alex Jones
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Building a Day1/Day2 Application Operations Platform On CNCF Projects. - Alois Reitbauer, Dynatrace & Alex Jones, Canonical
Effectively delivering and operating large and complex cloud-native applications becomes more and more important as companies move an increasing number of applications to Kubernetes. Most companies are building self-service platforms which can be used by individual teams while enabling companies to drive company wide practices. The cloud-native ecosystem provides a large number of projects that help with different aspects of building these platforms. In this talk we will cover all major aspects of the application lifecycle from build, test over to provision, delivery and release all the way to operational management and showcase different tools and how they can be used and combined together. After the talk you will be able to answer all the below questions and more: How can I best build cloud native applications? What are the best approaches to provide standard components like databases, etc? How can I provision infrastructure following the same cloud native approach I use for my application? How can best manage the deployment and rollout process? How can I seamlessly integrate practices like chaos testing? How can I automate the setup of operations requirements like security, observability, …? How can I automate day2 operations at an infrastructure and application level? We will focus on sharing concepts combines with small examples which help illustrate how different aspects can be done with different tools.
B
B
A
Is
maybe
a
bit
different
mouthful?
Let's
start,
first
with
a
quick
introduction
to
the
two
of
us,
my
name
is
always
I'm
working
for
Diana,
trace
and
advantageways
being
responsible,
amongst
other
things,
for
everything
we
do
open
source
wise.
All.
The
number
of
cncfs
projects
like
open
feature,
open,
Telemetry
and
Captain
I'm,
also
co-chair
of
tech,
app
delivery
in
the
cncf
and,
like
a
lot
of
this
motivated
Target
I'm
having
or
that
we
are
having
here
today
and
now.
B
Yeah,
so
my
name
is
Alex.
You
can
see.
I
put
my
full
name
there
to
distinguish
myself
from
certain
news
headlines:
I'm
an
engineering
director
at
canonical,
so
I
work
on
micro,
Kate's,
charm,
Cates.
We
build
Upstream
kubernetes,
but
I
also
on
the
governing
board.
Now
for
open
feature
and
a
contributor
there,
and
also
work
with
elois
on
Tag
app
delivery.
A
So
before
we
get
started
here,
there's
a
slider
link,
you
can
actually
use
the
slider
link.
We
will
have
a
couple
of
polls
as
part
of
the
slider
and
you
can
then
also
interact
with
as
well
as
light
on
top
of
the
chat
as
well.
So
yeah
just
take
your
phone
I'll
leave
it
here
for
a
second
yeah
yeah,
give.
A
A
A
They
finished
the
deploy
their
operators,
maybe
that's
why
okay
all
right
and
one
more
yeah.
So
this
is
like
an
open-ended
question
now,
if
you
think
about
day,
one
and
day
two
operations
just
put
in
some
of
the
challenges
that
you
had
so
far
so
day,
one
we
really
talk
about
deployment,
we'll
go
into
bit
more
details
and
day.
Two
is
like
really
running
them
in
products.
It
doesn't
throw
in.
B
A
A
Education
is
good,
awesome
awesome.
This
is
very
useful.
Thank
you.
Everyone
for
participating,
yeah,
I,
think
it.
Obviously
you
can
keep
adding
to
this,
and
so,
but
now,
let's
get
started.
Let's
start
with
some
things
you
probably
already
know,
or
at
least
familiar
with.
That's
also
why
we
were
asking
those
questions.
Obviously,
if
we
talk
about
the
two
operations,
we
have
a
couple
of
Concepts
that
you
should
be
more
or
less
familiar
with
we'll
dive
into
details
there
as
well.
A
One
is
that
the
the
declarative
and
extensively
API
that
kubernetes
is
offering
like,
because
you
can
use
The,
Primitives
and
kubernetes.
You
can
extend
it
with
your
own
crds.
You
can
use
composability
I.
Think
that's
what
pretty
much
everybody
here
is
familiar
with
we're
asking
about
github's
githubs
as
a
concept
of
model
syncing
your
environment
here
and
matching
the
system
state
with
the
desired
state
of
your
environment,
using
it
to
update
and
manage
your
environment.
We
had
quite
an
interesting
distribution.
There
I
think
it's.
A
We
I
think
we're
going
to
be
pretty
much
on
point
because
we
won't
be
talking.
We
put
this
in
deliberately
like
this
is
the
like
the
basic
drawing
of
githubs.
Maybe
what
you
would
put
like
on
a
box
of
corn
flakes
Alex
today
will
go
much
deeper
into
what
we're
going
to
do
there
and
the
third
one
is
like
really
about
operators
and
series
for
custom
Logics.
A
So
if
you
at
least
have
some
understanding
of
operators
in
series
and
how
they
help
to
build
out
certain
things,
especially
when
we
move
more
towards
the
day
two
piece,
you
will
see
how
operators
and
controllers
play
player
over
here,
so
I
think
everybody's
quite
familiar
enough,
but
we
will
dive
into
some
of
those
topics
more
in
detail.
So
when
we
talk
about,
they
wanted
to
do
operations,
and
there
was
the
word
backup
on
there
and
backup
is
actually
one
of
the
things
we're
not
going
to
cover
today.
A
But
we're
really
focusing
on
is
really
looking
at
an
application
like
full
stack,
all
the
way
down
to
infrastructure
from
your
intent
to
deploy
until
you
retire
it
from
production
and
one
of
the
distinctions
I
really
like
about
day.
One
and
day
two
is
using
an
analogy
from
another
area.
In
this
case,
it's
space
travel
not
that
it's
rocket
science-
maybe
sometimes
it
is
a
bit
of
rocket
science,
but
I
like
to
compare
the
one
operations
with
launch
control
when
you're
launching
a
rocket
so
in
launch
control.
A
What
you're
actually
doing
you're,
obviously
trying
to
get
this
rocket
off
the
ground
into
space
right
think
about
launch
control
is
that
everything
on
launch
control
is
focused
on
whether
getting
that
rocket
up.
There
is
something
that
you
want
to
do
announce
and
usually
your
process.
If
something
is
not
right,
is
abortion
or
rollback
oriented
so
a
lot
of
them?
They
want
if
it
doesn't
work.
We
roll
back
to
previous
state
that
was
working
to
allows
no
good
configuration,
and
the
interesting
fact
is
that
this
is
the
same.
A
Actually
what
you
do
like
in
space
travel.
However,
there
is
this
point
when
the
rocket
goes
up
and
it
like
reaches
like
that's
roughs,
that
it
goes
that
it
has
multiple
cheese
going
there.
So
you
can't
eject
that
capsule
on
top
anymore,
which
you
could
usually
explode
off,
but
suddenly
you
can't
anymore,
so
you
have
to
continue
doing
what
you're
doing
so.
A
So
the
main
differentiation
is
really
here.
This
is
okay,
we
are
bored
if
it
doesn't
work
day,
two
operations.
We
really
keep
the
system
running
and
modify
so
just
keep
running
again,
like
think
again
about
space
travel.
You
can
tell
the
astronauts
who
might
be
in
space
right
now.
We
don't
know
how
to
bring
you
back
to
Earth,
but
we're
going
to
launch
another
crew
tomorrow
and
see
how
it's
going
welcome
being
part
of
our
cannabis
based
reality.
B
You
know
it's
a
substrate
that
you
can
effectively
deploy
out
many
different
types
of
intraoperable
components
and
it's
very
extensible
by
its
nature.
So
it
really
makes
it
a
perfect
environment
to
use
as
your
mission
control,
because
you
can
deploy
it
on
top
of
any
other
infrastructure
substrate
and
effectively.
The
API
is
extensible
enough
that
you
can
start
to
Define
custom
types
of
behavior
and
custom
types
of
Primitives.
That
mean
something
to
you.
B
The
last
thing
is
that
it
really
takes
us
in
the
direction
of
being
a
declarative
process,
in
that
we
write
something
we
deploy,
that
we
reconcile
it,
and
that
gives
us
the
day
one
day.
One
is
where
you
start
your
journey
in
this,
so
starting
out,
what
does
it
actually
look
like
well,
I?
Think
most
folks
will
be
familiar.
We
had
sort
of
a
median
distribution
there
most
folks
were
familiar
with
guitars,
but
here's
a
little
crash
course
reminder
that
effectively.
You
can
achieve
get
UPS
in
a
very
simple
way.
B
You
know
it's
a
repository,
that's
sinking
back,
and
you
have
this
set
of
activities
that
allow
you
to
use
the
fundamental
components
of
git
in
a
way
that
can
provide
you
a
lot
of
value
directly.
You
know
so
authentication
also
having
checks
and
tests
as
part
of
that
gating
system.
In
this
example
here
this
could
be
a
repository
that
has
an
application
workload.
It
might
be
an
infrastructure
workload,
we'll
describe
what
that
actually
means
later
on,
but
it's
some
type
of
declarative
configuration
now
in
reality,
it's
typically
a
lot
more
complex
than
that.
B
This
illustration
shows
what
actually
looks
like
when
you
start
to
put
this
in
practice.
So,
as
you
see
here,
we
have
this
idea
of
managed
services,
so
these
could
be
your
kubernetes
cluster
in
production.
But
how
do
we
go
through
this?
How
do
we
get
to
this
point
of
having
all
these
fancy
services
on
the
right?
B
Typically
speaking,
the
thinking
right
now
is
that
we
want
to
have
git
Ops
to
reconcile
not
only
the
infrastructure,
declarative,
config,
but
also
the
application
workloads.
But,
of
course,
there's
a
challenge
getting
there,
because
our
infrastructure,
declarative
configuration,
might
have
several
different
topologies
that
it's
deployed
on
whether
it's
openstack,
whether
it's
you
know
directly
onto
some
bare
metal
provider,
whether
it's
in
a
cloud
provider,
and
so
we
need
a
way
of
describing
that
that's
reproducible
and
a
way
that
we
can
template
to
go
forward.
B
So
you'll
see
that
one
of
the
ways
of
thinking
is
to
actually
have
some
sort
of
management
control
cluster,
where
you
are
saying,
here's
the
desired
intent.
I
need
you
to
think
about
the
components
that
I
need
to
get
there
we'll
come
on
to
talk
about
some
cncf
projects
later,
but
you
can
probably
summarize
theorize
that,
if
you're
familiar
with
these
you'll
see
where
I'm
going
with
this
in
that
the
infrastructure
Primitives
that
might
be
give
me
a
load.
B
Balancer,
give
me
a
managed
object,
storage
things
that
you
can
describe
as
the
declarative,
configuration
and
they're
then
provided
by
a
cloud
infrastructure
provider,
and
so
where
does
it
all
lead
to?
Well,
when
you
start
to
build
these
managed
kubernetes
clusters,
the
idea
is
that
you
can
provide
the
application
repositories.
So
folks,
who
want
to
just
deploy
a
workload
on
top
of
them,
you
can
give
them
a
batteries
included
model.
So,
let's
walk
through
how
this
actually
works.
B
In
this
sort
of
scenario,
you
have
a
declarative
configuration
for
say
some
infrastructure
I
need
an
API
Gateway,
a
piece
of
object,
storage
and
some
other
bits
and
Bobs
and
they're
all
the
primary
substrate
for
this
application.
Repository
I
might
have
a
WordPress
application
that
needs
all
those
things,
but
to
get
there,
I
need
to
have
a
certain
amount
of
things
pre-provisioned,
and
this
is
the
real
heart
of
the
day.
One
activities
it's
provisioning
and
deployment
you'll
see
that
what
happens
in
this
example
is
that
we
have
a
management
kubernetes
cluster.
B
That's
created
that
management
cluster
contains
all
sorts
of
Primitives
that
describe
these
types
of
managed.
Services
right,
I
want
an
object,
storage
of
a
certain
size
in
a
certain
place,
with
a
certain
set
of
controls
on
it.
Equally
I
also
want
to
create
multiple
kubernetes
clusters.
I
mean
the
trend,
we're
trending
towards
having
ephemeral
clusters
that
are
short-lived
rather
than
large,
node
clusters.
B
Storage
is
I
understand
what
an
API
Gateway
is,
and
it
will
be
contextual
because
it
will
understand
the
annotations
of
that
particular
cloud
service
that
you
are
correlating
to
and
once
you
get
there,
that's
not
really
where
the
journey
ends,
because
if
you
deploy
out
like
that
you're,
it's
pretty
much
like
firing
a
gun,
it's
a
black
box.
So
really
you
just
need
to
start
thinking
about.
How
do
you
aggregate
your
observability
between
application,
Level
and
infrastructure
level?
How
do
you
differentiate?
B
This
is
where
we
start
to
see
day.
One
operations
has
a
lot
of
toil,
because
what
we're
seeing
companies
do
is
just
use
the
fire
hose
of
putting
all
logs
or
messages
so
from
the
entire
node.
All
the
way
up
to
the
application,
on
top
into
their
observability
systems
and
sort
of
just
saying
that's
great-
we've
got
observability,
but
not
really
knowing
how
to
then
use
that
as
a
control
and
a
feedback
loop
for
slos
and
other
governance.
B
And
so
in
a
nutshell.
One
of
the
key
abilities
that
you
get
from
following
this
kind
of
pattern
is
declarative
reproducible
ways
to
build
that
infrastructure,
but
also
when
you
start
to
implement
the
feedback
loop
of
the
observability
being
aggregated.
You
can
start
to
modify
the
declarative
config
and
go
further
into
the
remediation
side
of
things.
A
First
of
all,
we
have
to
extend
a
bit
of
our
mindset
that
Alex
talked
about
from
day
one
to
day
two
what
we
want.
We
talked
about
infrastructure
that
we're
refining
as
we
talked
about
application
workloads,
and
we
talked
about
like
the
synchronization
that
the
operated
pattern
provides
between
a
desired
state
and
the
actual
state
of
the
system.
So
what
we
have
to
do,
we
have
to
extend
our
desired
state
so
that
the
system
four
date
two
operations
actually
knows
what
to.
B
A
And
there's
a
couple
of
ways
to
do
that.
So
if
you
look
at
the
idea
of
alerting
or
modifying
State,
you
have
three
options.
The
first
one
is
anomaly:
detection
which
more
or
less
means
every
system
is
behaving
not
as
it
is
supposed
to
be.
This
can
mean
that
you
have
response
times
that
you're,
usually
50,
milliseconds
they're,
not
suddenly
like
120,
but
your
slos
or
your
slas
might
still
be
at
a
couple
of
seconds.
So
it's
technically
fine,
but
the
system
is
not
behaving
as
it
is
usually
behaving.
A
A
Two
operations
actually
wants
to
do
that
so
actively
configure
the
cluster,
and
obviously
you
have
at
some
point,
you've
been
leaving
the
imperative
mode
because
you're
moving
to
an
imperative
mode.
You
have
to
tell
the
system
what
it
should
do
when
it
is
in
a
certain
situation.
You
technically
have
this.
Even
it's
usually
hidden
very
well.
If
you
look
at
classical
operations,
that's
what
you
would
have.
B
A
Like
your
remediation
workflows
and
the
scripts
that
you're
usually
running,
but
you
also
need
to
provide
this
ability
to
Excel
native
system
as
well
what
you
need
forever,
especially
in
a
cloud
native
environment
where
we
run
a
lot
of
small
composed
Services,
it's
a
slightly
different
approach.
That
was
a
traditional
application.
B
A
There's
always
this
term
coming
up
about
AI
ups
and
I
offices,
a
lot
of
things
to
a
lot
of
people,
and
it
is
more
advances-
that's
on
this
slide,
but
they
still
want
to
share
the
basic
idea.
The
basic
problem
that
you
need
to
solve
is
we
want
to
make
this
actually
work
in
in
massively
distributed
system.
B
A
The
result
out
of
this
is
obviously
that
you
will
get
alerts
for
all
of
those
three
services
without
context,
so
you
would
not
know
how
to
react
on
this
situation,
like
which
service
should
you
remediate
in
the
worst
case,
you
would
just
try
to
change
all
the
three
services.
Well,
there's
actually
one
that
actually
needs
to
be
changed
this
by
the
way
also
has
impact
on
how
you
model
operational
workflows.
You
don't
model
them
on
the
entire
system,
but
you
need
to
model
them
based
on
specific
services.
A
A
If
then,
on
top
you're,
adding
like
temporal
information
like
what
component
failed,
first
plus
additional
information
like
how
things
can
be
related
to
each
other,
like
how
affiliate
mode
can
be
related
to
a
Slowdown
or
how
certain
failures
at
the
application,
Level
can
be
related
to
infrastructure,
you're,
actually
arriving
at
something
like
this,
which
can
certainly
differentiate
the
impact
of
a
failure
mode
to
the
actual
cause
or
root
cause.
I
know
not.
Everybody
likes
the
term
root,
cause
I,
think
you're
kind
of
getting
to
it.
A
So
what
we
actually
did
here,
we
used
traditional
monitoring,
combined
it
with
tracing
and
some
additional
knowledge.
We
have
about
our
system
to
to
come
to
a
way
better
way
of
actually
alerting
and
defining
these
kind
of
things.
So
that's
very
high
level
what
you
really
want
to
get,
and
so
this
is
way
more
important
being
hungama,
whether
it
is
AI
whether
this
is
machine
learning
or
whatever.
It
is
a
system
that
can
reconstruct
based
on
context.
A
So
what
we
now
need
to
do
in
order
to
move
to
a
day.
Two
operation
scenario
is
actually
a
couple
of
things.
Alex
talked
about
like
this
whole
gitups
configuration
that
we
have
for
day
one
and
we're
now
starting
to
extend
it
for
day
two.
The
first
thing
we
do
as
we
start
everything
the
clarity.
First,
we
are
adding
the
declarative
monitoring
in
there,
so
your
monitoring
configuration
should
also
be
part
of
your
system.
Configuration
this
today
and
I.
A
A
You
want
to
have
this
version
control
and
linked
to
the
version
that
you're
actually
deploying
because
as
you're
rolling
forward
with
new
deployment,
you
want
your
remediation
logic
to
roll
forward
with
this
as
well,
and
this
logic
can
be
basically
everything.
It
can
be
just
a
link
to
a
container
image.
It
can
be
a
script
that
you're
providing
there.
Whatever
is
kind
of
needed
for
the
environment
that
you
want
to
work
with,
and
these
logic
blocks
are
usually
very
simple.
A
So
they're
not
like
super
complex
workflows,
as
you
might
be,
used
to
from
traditional
applications
where
I
feel
like
workflow,
with
15
20
or
more
steps,
as
they
obviously
are
always
about
like
one
single
service,
they're,
usually
more
or
less
little.
If,
if
than
else
statements
like,
if
this
happens,
then
modify
the
service
this
way
and
usually
they
come
from
developers,
because
developers
know
how
you
can
reconfigure
a
system
properly
and
then
they're
composed
at
runtime,
depending
what
we
learned
before
the
AI
upside,
how
things
actually
work.
A
So
what
we're
now
doing
as
we're
doing
our
GitHub
sync
we're
not
just
thinking
our
declarative
configuration,
how
the
application
should
be
deployed
and
build.
We're
also
thinking
over
a
still
thinking
over
all
declarative
monitoring
configuration
and
a
remediation
logic,
obviously
doing
this
per
se
once
it
was
a
lot
of
good,
because
kubernetes
would
know
what
to
do
with
this
new
crds
that
we
just
built.
A
B
A
Monitoring
configuration
and
pushing
it
towards
your
observability
system,
so
the
idea
should
be
your
observability
system
is
not
configured
out
of
band
and
it's
like
split
brain
type
of
situation,
but
you
actually
let
kubernetes
really
be
the
control
plane
here
to
also
take
control
over
the
observability
system.
So
it
uses
this
configuration
and
then
deploys
it
to
your
Prometheus
or
whatever
solution,
you're
kind
of
using
there,
including
creating
alerts
creating
metrics
that
you
want
to
calculate.
A
For
example,
if
you
have
an
SLO
definition,
there's
a
project
called
like
open,
slos
and
and
other
projects
like
Captain
Hefner
as
low
definition
as
well.
They
create
the
proper
metrics
to
do
the
metrics
calculation
for
you,
so
that
you
don't
have
to
take
care
of
them
and
then
what
you
need
is
actually
somebody
watching
over
this,
like
what
does
the
observability
platform
tell
us
about
what's
happening
here,
and
this
is
in
our
case
is
our
day
two
application
controller.
This
is
either
push
or
pull
the
way.
A
How
it's
interacting
with
the
monitoring
system,
and
this
more
or
less
depends
on
security
boundaries
or
what
you
feel
more
comfortable.
In
many
cases
it
is
actually
more
a
pull
because,
depending
on
whether
you
have
an
on-prem
monitoring
system
or
it's
entirely
a
separate
component.
So
whether
the
monitoring
system
is
allowed
to
send
modifications
into.
B
A
Cluster
or
you're
actually
fetching
them
so
once
that
your
day,
two
app
controller
actually
detects
that
there
is
a
problem
happening
in
the
system.
What
they
do
is
they
look
up
this
remediation
logic,
which
is
technically
the
CRT
that
they
are
working
with
again,
it's
assume
our
example
where
we
have
if
the
load
on
a
certain
system
is
too
high
and
response
times
going
down,
then
scale
to
cached
content
delivery.
That's.
A
It
would
find
it's
most
likely
if
metrics
are
worth
or
it's
a
certain
type
of
alert
arises
then
modify
a
feature
flag
or
whatever
you
want
to
do
in
this
case.
So
what
this
then
does
it
spins
of
a
remediation
job
and
job
in
this
case
is
really
a
job
in
kubernetes
we
most
likely
model
this
as
a
job,
because
once
it
has
finished
running,
you
can't
get
rid
of
it.
A
Changing
some
system
configuration
modifying
it
in
some
ways,
maybe
even
talking
to
some
of
the
cloud
apis,
but
even
more
importantly,
and
that
honestly
right
now
is
a
bit
of
the
controversial
thing.
The
remediation
jobs
might
actually
actively
change
your
declarative
configuration
so
they're,
intercepting
with
your
github's
workflow
and
in
a
GitHub
scenario.
A
That's
currently
the
only
way
you
can
actually
change
this,
because
if
it
would
change
it
right
in
the
cluster,
the
github's
controller
would
revert
your
change
because
suddenly
the
desired
State
and
the
actual
state
is
out
of
sync,
and
it
doesn't
know
the
desire
to
say
that
it
would
have
in
there.
So
this
is
kind
of
a
bit
of
an
unsolved
problem
in
here,
but
it
should
give
you
an
idea
on
how
you
like
to
build
it
through
the
two
operations
pieces
in
there.
A
Sorry
something
about
anomaly
detection
using
really
slos
to
inform
you
about.
What's
going
on
doing
this
correlation
and
then
really
doing
this
context,
every
management
so
adding
additional
context,
management
to
the
workloads
that
you're,
deploying
and
I
think
this
brings
us
to
how
we
built
this
on
top
of
CNF
project.
B
So,
really
to
just
build
upon
the
previous
slides.
There
are
several
projects
within
the
cncf
ecosystem
which
I
picked
at
random
because
they
fulfill
the
requirements
here,
but
what
they
all
do
is
they
enable
you
to
build
that
workflow
that
we've
just
described
so
being
able
to
use
flux
or
Argo
or
any
number
of
GitHub
space
tools
is
extremely
convenient
when
you
need
to
be
able
to
create
a
management
cluster
that
represents
the
infrastructure
and
the
workloads
that
you
want
to
apply
on
that
infrastructure.
B
One
of
the
things
that's
really
interesting
to
me,
and
this
came
through
Tag
app
delivery
was
cross-playing
and
we
were
doing
the
due
diligence
for
that,
because
cross
plain
is
a
very
interesting
project
in
that
it
represents
all
the
resources
in
Cloud
providers,
which
isn't
something
that
traditionally
something
like
cluster
API
would
do,
and
in
fact
you
can
create
load
balances
and
block
storage
and
all
those
other
artifacts
that
are
convenient
and
even
more
what's
useful,
is
I.
Think
as
I
described
in
the
previous
slide.
B
You
have
the
app
repositories
and
then
you
have
the
infrastructure
repositories.
You
can
marry
the
two
together.
You
can
have
a
singular
repository
that
describes
the
dependencies
of
your
application
in
the
same
repository
in
a
yaml
definition,
and
that
brings
us
a
lot
closer
in
the
day,
one
to
being
able
to
get
to
a
point
of
a
very
ubiquitous
cycle
throughout
this
infrastructure.
Life
cycle,
and
what's
interesting,
especially
I,
think
is
that
I
mentioned
cluster
API,
not
having
the
same
capabilities
of
cross-plane.
B
But
of
course
that
is
now
a
subset
called
by
cross
plane
and
you
can
use
cluster
API
directly
if
you
want
to
cut
out
that
particular
process.
What's
nice,
I
think
is
that
we
have
a
lot
of
Lego
bricks
inside
of
the
cncf
ecosystem
that
are
convenient
for
you
to
choose
your
journey
and
I
always
also
mentioned
using
tooling
such
as
tracing
with
jaeger.
You
know
one
of
the
Ambitions
of
the
open
feature
project
is
to
add
baggage
so
that
you
can
see
the
feature
flag.
B
That's
not
that's
not
represented
any
more
so
than
in
the
captain
project,
which
is
also
a
new,
a
newly
thought
out
project
in
terms
of
building
some
components
that
are
really
innovating
in
this
space.
You
know
looking
at
things
like
slos
and
using
those
in
a
way
that
you
can
actually
make
changes
to
enable
that
day-to
Journey,
if
you
haven't,
checked
out
Captain
I,
think
that
I
would
do
it.
B
Miss
Justice
now
to
try
and
explain
it,
but
Captain
very
much
follows
that
path
of
being
able
to
give
you
a
greater
picture
on
the
context
or
application
right.
So
if
you
have
a
particular
workload,
that's
running
in
kubernetes,
you
can
also
see
those
relationships
and
the
dependencies
and
manage
the
slos
in
a
really
convenient
way
and
I
think
this
sets
a
tone
for
tooling
of
the
future.
In
that
day,
two
is
a
problem
that
we're
starting
to
get
closer
to
solving.
B
B
So,
in
summary,
we
try
to
distill
I.
Think
the
key
tenets
of
what
we
would
like
you
to
hopefully
take
away
here
is
that
data
operations
is
very
much
about
trying
to
remove
the
humans
out
of
the
equation
here
and
to
rely
on
automation.
You
cannot
rely
on
automation
unless
you
have
the
right,
obserability
signals
being
correctly
remediated
and
acted
upon
good
Ops
kunet
is
an
operators
are
just
enablers
for
this
pattern.
Really,
you
know
they're
convenient
enablers.
This
is
a
pattern
of
operation.
B
This
is
how
you
manage
the
life
cycle
of
your
digital
estate
and
slos
and
error
budgets.
We're
starting
to
see
now
more
than
ever
are
becoming
the
inputs
for
those
corrective
actions
for
operators.
Those
are
becoming
the
driving
force
behind
what
how
an
operator
makes
that
decision,
how
the
remediation
job
actually
starts
to
build
that
picture
of
what
it's
going
to
transmit
back
to
the
declarative
configuration.
A
Yeah
and
I
think
what
we
could
actually
show
here.
You
could
probably
use
like
The
Core
Concepts
of
kubernetes
as
a
capital
plane
to
build
all
of
them
and
like
really
stick
to
this
declarative
and
operator
model.
It's
also
why
we
went
into
some
more
details.
I
could
build
day
two
and
also
for
the
day,
one
bits
and
pieces.
A
It
is
actually
a
passive
and
you
don't
need
to
accept,
like
out
of
band,
to
fall
back
for
non-non
cloud
native,
tooling,
I
think
also
you
see
how
like
projects
in
the
cncf
fall
together
nicely
once
you
stick
to
this,
like
Core
Concepts
that
we
have
defined
for
for
kubernetes
as
a
control,
plane
and
and
how
well
they
are
they
connect
and
playing
with
each
other.
Obviously
you
find
our
Twitter
handles
here
for
any
questions
or
how
it
will
engage
with
us.
A
That
QR
code
is
a
different
one,
that
for
the
working
group
on
platform,
that's
part
of
that
Tech,
app
delivery,
so
Tech
app
delivery
has
a
working
group,
the
now
platform
working
group.
It
used
to
be
called
Cooperative
delivery,
which
kind
of
was
doing
the
same
thing,
but
people
feel
more
comfortable
these
days
talking
about
platform
and
platform
engineering.
A
It's
part
of
our
work
so
I.
We
definitely
invite
you
to
join
working
group
platforms
here.
If
this
is
something
that's
interesting
to
you,
if
you
have
an
opinion
on
it,
if
you
have
questions
on
it,
because
that's
what
we'll
be
focusing
on
a
lot
within
Tech
app
delivery
to
help
you
build
those
platforms,
bring
projects
together
and
building
the
seamless
experience.
So
how
like
the
input
was
useful.
B
B
Catalog
and
you've
got
like
five
things
you
can
use
in
there,
and
it's
really
tough
I
think
that
it's
a
wider
question
right
in
terms
of
the
governance
audit
Cuts
several
ways
right,
whether
that's
looking
at
the
security
protocols
in
place,
whether
you're
looking
at
fips,
CIS,
et
cetera,
Etc
or
whether
you're
looking
at
you
know,
stock.
Two
and
more
of
the
controls
that
you've
got
on
your
pii
Etc
I.
Think
what's
interesting
in
this
respect.
B
Is
that
because
you're
able
to
effectively
represent
your
entire
digital
estate
right
in
a
series
of
repositories
that
get
pulled
into
reality,
it
makes
it
very
easy
to
find
one
place
where
you
run
the
audit
scanners
over.
So
as
an
example-
and
this
might
be
only
a
subset
to
the
answer
of
your
question
but
as
an
example,
we
would
at
Bank
scan
the
repository
to
check
where
you
know,
simple
things
like
images
were
coming
from.
B
Right
was
it
coming
from
the
wild
or
was
it
coming
through
a
prop
a
squid
proxy
did
the
did
the
permissions,
for
example
like
to
use
host
Network.
You
know
simple
misconfiguration
issues,
kind
of
what
I
had
in
that
check
and
test.
You
could
extend
that
to
have
a
suite
of
audit
tests
in
there
so
that
when
it
gets
to
dual
production,
the
security
team,
then
don't
remind
you,
you've
made
a
problem.
I
think
also.
You
know
to
that
end
with
the
git
Ops
process.
B
There's
nothing
to
say:
you
can't
deploy
Out
Security
scanners,
like
trivia
Etc
into
your
environment
and
use
the
observability
for
trivia,
gives
you
Prometheus
metrics.
It's
the
same
thing
right.
It's
just
another
signal
that
feeds
back
into
the
remediative
action.
What
you
could
do
in
that
case
and
I'll
demo
this
on
Friday
in
the
open
feature
talk
is
that
you
can
use
feature
flagging
to
actually
actively
in
the
cluster,
get
the
API
service
turn
capabilities
off.
B
So
when
you
detect
say
from
your
security
scanner,
there's
a
Prometheus
metric,
that's
returning
a
one,
meaning
there's
a
cve
in
it.
You
can
turn
off
the
admission
controller
that
will
stop
pods
being
created.
It
basically
puts
the
cluster
into
a
shutdown
mode,
so
there
are
all
these
interesting
capabilities
that
we're
starting
to
build
on
and,
as
I
said,
Shameless
plug,
but
open
feature.
Plus
these
kind
of
practices
go
hand
in
hand
for
that
area.
A
Maybe
the
champion
here.
Actually
this
is
the
most
audit
friendly
thing
that
you
would
have
find,
because
everything
is
run
where
I
get
then
you
could
even
run
and
get
blame
like
most
operational
processes.
That
you're
running
today
rely
on
people
knowing
what
they're
doing
and
like
documenting
it
actively.
This
is
actually
all
documented.
It's
all
in
git,
you
have
every
single
change.
You
have
every
single
action
in
there.
A
You
have
access
control
on
all
different
levels,
enabled
it
looks
like
a
bit
scary,
because
the
system
is
doing
something
by
itself,
but
it's
never
doing
anything.
You
don't
want
it
to
do.
You
can
always
add
manual
guard
rails,
and
you
know
exactly
every
single
state
transition
of
that
system
that
is
more
auditing
than
most
systems
today
actually
have.
A
If
you
really
stick
to
the
model,
it
only
breaks
and
that's
the
important
thing,
if
you're
doing
something
out
of
band
and
that's
why
it's
so
important
to
really
design
around
those
prints,
at
least
in
principle,
and
that
like
Alex,
mentioned
open
feature.
If
you
feature
management
system
uses
to
be
a
separate
system
that
handles
feature
Flags
separately,
you're,
suddenly
breaking
that
entire
audit
chain,
because
you
have
one
system
deployed,
you
have
to
audit
log
for
this
one
and
then
suddenly
an
out-of-band
system
changes
something
on
those
systems.
B
Much
better
than
as
an
anecdote,
you
know,
I've
seen
things
like
service
now
being
used
to
create
temporary
ephemeral
accounts
to
then
make
a
manual
imperative
change
within
a
cluster.
That
then,
is
completely
unregulated
and
unordered,
whereas
this
way
the
principle
would
be
that
you
have
a
git
group
of
Administrators
within
your.
You
know,
VCS
that
can
make
a
change
in
a
PR
merge
that
change
in
and
it's
far
more
traceable.
B
You
know,
there's
no
real,
differentiating
you
I
guess
what
you're
driving
at
is
the
whole
infrastructure,
provisioning
piece
so
with
cross-plane
and
with
Cappy.
You
have
two
components:
right:
it's
a
bootstrap
provider
which
gives
you
the
flavor
of
kubernetes,
there's
a
there's,
an
infrastructure
provider
and
there's
about
15
different
Cloud
providers.
What's
interesting
about
this,
is
it's
a
it's
a
democratization?
So
if
I
have
a
startup
and
I
have
a
rack
and
I
want
to
show
it
to
the
world?
B
I
might
write
my
own
infrastructure
provider
and
then,
when
somebody
writes
a
piece
of
cluster
yaml
and
they
put
Alex's
company,
the
provider
will
start
the
Pod
and
it
knows
how
to
reconcile
that
and
it
connects
to
my
API
in
in
London
right.
So,
what's
really
interesting
about
the
cncf
projects
and
I
mentioned
posix,
is
it's
a
great
leveler?
It's
been
creating
a
ubiquitous
platform
that
you
can
start
building
towards
so
for
bare
nettle.
B
A
Yeah
I
think
the
key
Point
again
is
they're
not
doing
it
out
of
band
I.
Think
that's
what
they
keep
hearing
us
repeating
over
and
over
once
we
move
say:
okay,
we
have
our
terraform
scripts,
nothing
against
terraform
at
all.
But
if
we're
not,
then
like
not
integrated
into
this
process
and
they're
handling
a
lot
of
these,
you
will
suddenly
use
this
connection
or
connective
tissue
that
you
want
to
have
in
there.