►
From YouTube: A Containerd And Friends Update: What’s New...Phil Estes, Mike Brown, Maksym Pavlenko, Michael Zappa
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
A Containerd And Friends Update: What’s New In Runtimes? - Phil Estes, AWS; Mike Brown, IBM; Maksym Pavlenko, Apple; Michael Zappa, Microsoft
After five years as a CNCF project, containerd is still actively growing in contributors and maintainers who are busy working on interesting features and capabilities in the core and non-core containerd projects. During this project update from maintainers you'll learn about the latest work in containerd, including our recent addition of sandboxes, a handful of CRI and CNI improvements, as well as various improvements to the architecture and services that drive containerd's use by other projects and platforms. Outside of the core containerd project, our "non-core" projects have grown in number, including several Rust-based projects, new snapshotter implementations, and the increasingly popular client-focused project, nerdctl. Come join us for a fast-paced update on all these areas and to ask your containerd questions with the handful of on-site containerd maintainers.
A
A
It's
like
inviting
a
bunch
of
friends
over
and
having
a
plumber
come
talk
about
the
plumbing
in
your
house
and
how
interesting
the
pipes
are,
but
you
know
we're
we're
a
pretty
core
foundational
run
time,
but
but
maybe
not
super
exciting
in
the
sense
of
like
tons
of
new
features,
but
we're
going
to
try
and
cover
a
little
bit
about
where
the
project
is
going.
You
heard
from
Derek
if
you
saw
that
video
kind
of
our
release
cycle
updates.
A
So,
as
I
said,
I'm
going
to
kind
of
Rapid
Fire
go
through
a
few.
Hopefully
interesting
updates.
One
is
that
we
currently
have
we're
about
to
have
three
release
lines
under
development:
one
five
would-be
end
of
life,
but
we've
just
extended
that
into
January.
So
a
few
more
months-
and
we
just
made
a
release
to
catch
a
few
back
ports
of
bug
fixes
so
folks
that
are
still
using
one
five
there's
still
a
few
months
of
support
for
that
1
6
is
kind
of
the
biggest
news
we
just
made.
A
A
readme
update
and
I
have
a
slide
about
that
in
a
minute,
but
we
just
made
our
latest
release
just
a
few
days
ago
of
1.6,
that's
probably
kind
of
the
the
most
stable
and
used
version
of
containerdy
at
the
moment.
That's
what
a
lot
of
projects
are
importing
and
then,
of
course,
there's
our
main
branch
where
we've
been
working
towards
our
one
seven
release
and
we
just
released
the
beta
zero
for
that
yesterday.
A
Derek
mentioned
that
in
his
video,
and
so
that's
where
a
lot
of
the
the
new
functionality
and
kind
of
our
stepping
stone
towards
2.0
that
that
Derek
mentioned
so
a
bit
more
about
the
LTS
release
our
long-term
support.
So
we
know
a
lot
of
folks
again
like
containerdy
because
of
stable
and
kind
of
slow
moving
in
a
sense,
and
so
we
wanted
to
take
the
release.
A
1
6
Branch
again,
because
one
seven
is
kind
of
a
stepping
stone
towards
2-0
and
some
deprecations,
and
we
want
to
make
sure
people
have
a
well-supported
container
need
Branch.
They
can
use
for
many
years,
and
so
we've
extended
that
out
through
2025
and
I
won't
read
this
paragraph.
You
can
go
to
the
releases.md
in
our
repo
and
read
what
we're
trying
to
to
Define.
As
what
does
it
mean
that
we're
going
to
have
long-term
support?
What
does
it
mean
for
backboarding
bug
fixes
what
what
does
it
mean
for
CRI
changes
and
kubernetes
releases?
A
And
if
you
see
something
there
that
doesn't
sound
right
or
sounds
like,
we
haven't
really
defined
it.
Well
enough
we're
happy
to
take
input
open,
an
issue,
and
let's
talk
about,
you
know
how
we
can
properly
Define
what
it
means
that
we're
going
to
have
this
LTS
release
and
again
1
6
is
going
to
be
that
for
now
a
couple
Community
updates,
one.
A
A
This
is
a
feature
you
can
access
today
in
preview,
but
I
think
one
of
the
main
benefits
is
that
a
lot
of
the
Innovations
in
container
D
around
snapshotters
will
now
be
able
to
be
exposed
up
through
Docker
as
well,
instead
of
having
their
own
push
and
pull
and
image
management
and
graph
drivers.
So
that's
a
that's
something.
We've
been
looking
for
forward
to
for
a
long
time.
A
Also
many
of
you
know
that
the
docker
shim
deprecation
has
finally
come
to
pass,
and
so
there's
been
a
lot
of
talk
about
migration
and
using
container
D
and
we've
definitely
seen
an
uptick
in
contributions
and
interests
and
users
which
has
again
been
great
for
our
community
we've.
We've
got
a
lot
more
contributions.
I
put
some
stats
there
on
growth.
It's
always
a
little
bit
hard
to
find
out.
You
know
very
specific
data
on
on
user
growth.
A
We
don't
have
Telemetry,
we
don't
know
about
downloads,
but
definitely
there's
some
different
reports
that
come
out
every
year
that
have
shown
pretty
significant
growth
for
containerdy.
As
a
kubernetes
runtime
as
well
so
I'm
going
to
touch
on
a
few
more
sub-projects
before
I
turn
it
over
to
Maxim
nerd
CTL,
a
very
popular
client
akihero
is
here
the
creator
of
that
has
grown
into
an
amazing
Community.
They
just
released
their
v1.0
released
just
a
few
days
ago.
A
A
Based
on
that,
a
lot
of
others
have
been
coming
up
with
new
Innovations
and
ideas.
Around
lazy
loading,
our
team
at
AWS
came
up
with
Sochi
seekable,
oci
format,
and
so
we've
open
sourced
that
snapshotter
this
summer
and
looking
for
interested
parties
who,
who
are
would
love
to
collaborate
around
Sochi.
A
That's
a
block-based
lazy
loading
snapshotter
that
that
relies
on
a
few
other
projects,
like
the
overlay
block
device
driver
that
they've
written
so
again,
a
lot
of
them
Innovation.
There.
A
lot
of
interest
around
that
and
it's
cool
to
see
these
Innovations.
Basically
coming
that
don't
aren't
tied
to
container
D's
life
cycle,
these
can
be
developed
separately,
have
their
own
communities
and
integrate
with
container
d.
B
Thank
you
so,
a
couple
years
ago
we
we
got
brass
version
of
ttrpc,
a
lightweight
communication
protocol
for
that
continuity,
demand
uses
to
talk
to
its
streams,
and
these
years
this
year
we
have
added
a
few
more
additions.
Java
rust
family.
We
have
open
source
trust
crates.
These
are
great
that
let
you
write
plugins
and
extension
for
continuity,
demon.
B
Obviously
in
Rust
language
and
currently,
we've
got
seven
different
crates.
We
have
grpc
client,
so
you
can
interact
with
continuity
demand
from
your
rust
program.
There
is
also
snapshot
crate,
so
candid
has
the
concept
of
proxy
snapshoters.
These
are
snapshot
that
can
be
communicated
via
jrpc
protocol.
So
basically,
this
creates
lets.
You
write
custom
snapshotters
by
just
implementing
a
single
interface,
and
everything
else
is
taken
care
of
by
the
library
and
also
we
offer
a
full
set
of
crates
to
implement
custom
shims
if
you
want
to
benefit
from
rust,
speed
and
safety
features.
B
So
this
streams
stream
Library
aims
to
provide
same
seamless
experience
as
Go
version.
So
basically
you
just
Implement
a
task
service
interface
that
manages
container
life
cycle
and
everything
else
is
hidden
from
the
user.
So
the
library
will
take
care
of
parts
and
command
line
arguments
a
relating
environment
run
into
their
business,
server
behind
and
so
on,
and
also
it
provides
both
sync
and
async
versions
of
the
code.
B
In
addition
to
shame,
Library,
there
is
a
rust
version
of
Francie
shrim.
This
work
mainly
serves
two
goals,
so
first
one
we
wanted
to
provide
a
reference
implementation,
so
people
have
an
example
how
to
write
custom
shims
end
to
end
and
how
to
work
with
the
library.
Second,
one:
it's
a
fully
featured
production
radiation
implementation.
We
run
some
integration
tests
on
OSI
against
the
shim,
and
originally
it
was
adopted
by
people
from
Huawei
cloud.
They
run
it
in
production
on
more
than
5000
nodes
and
they've
they've
got
quite
decent
resource
utilization.
B
So
on
decent
environments,
you
can
expect
more
than
twice
better
memory.
Consumption
now,
what's
coming
in
1.7
release
we're
introducing
two
new
apis,
the
first
one.
We
continue
to
work
on
Sandbox
API
to
quickly
recap
its
a
new
set
of
apis
that
had
a
notion
of
group
of
containers
and
by
group
we
we
mean
really
anything
here.
B
We
can
substitute
this
with
something
else
like
hypervisor
sandbox
implementations
will
be
provided
by
contain
Editions,
so
shim
expected
to
implement
this
new
controller
interface
that
will
manage
sandbox
lifecycle,
in
addition
to
Containers
life
cycle,
and
this
interface
is
going
to
be
optional.
So,
when
implemented,
continuity
know
that
this
stream
supports
sandboxes,
otherwise
we'll
default
to
current
implementation
with
containers,
the
demon
itself
will
remain
implementation,
agnostic,
so
we'll
have
no
knowledge
of
what
kind
of
sandboxes
we
are
running
in
future
once
fully
migrated
to
sandbox
API.
B
So
it's
not
enough
to
force
him
to
just
Implement
sandbox
interface.
Someone
has
to
call
it,
and
this
is
where
our
Seri
plugin
comes
in
the
picture
in
current
implementation,
when
we
schedule
the
code,
our
CRI
layer,
managers,
ports
and
blocks
directly,
and
instead,
the
goal
for
1.7
is
to
add
extra
level
of
wind
direction.
So,
instead
of
managing
parts
and
boxes,
the
CRI
will
be
calling
our
sandbox
API
instead
and
in
this
case,
in
this
case,
Watson
box
will
become
one
of
controller
implementations.
B
Another
new
addition
that
we
plan
to
get
in
in
1.7
is
transfer
service,
so
high
level,
it's
a
new
API
to
transfer
artifacts
from
source
to
destination
and,
in
our
case,
source
and
destination,
will
be
plugin
plugins.
So
we
can
request
a
particular
login
when
we
initiate
new
transfer
session,
so
this
table
on
the
slide.
It.
B
Current
functionality
supported
by
continuity,
but
today
all
of
this
currently
happening
on
client
Side
by
passing
continuity
demand.
So
every
user
of
this
client
defines
how
they're
going
to
use
it.
There
is
no
central
place
to
configure
it.
For
instance,
today
a
continuity
demon
does
not
participate
in
pool
operation.
It's
all
happening
on
client
side
with
transfer
API.
This
will
be
moved
to
demand
side
and
will
will
be
implemented
as
a
source
and
destination
plugins.
B
So,
let's
take
as
an
example
this
full
operation,
so
we
will
be
using
a
registry
as
a
source
plugin,
so
we
will
be
downloading
a
container
image
from
oci
compatible
registry
and
we
use
Image
store
as
destination,
so
continuity
will
be
saving
this
image
locally.
However,
now
with
transfer
API
instead
of
registry,
we
can
request
a
different
plugin.
For
instance,
it
could
be
S3
storage,
and
this
way
we
can
pull
directly
from
S3
to
and
download
images
locally.
B
And
there
are
some
new
use
cases
that
we
want
to
cover.
In
future,
we
plan
to
implement
new
plugins
plug-in
types
in
contain
to
decor
for
image,
sign-in
and
validation,
we're
also
exploring
better
support
of
credentials,
image,
encryptions,
image,
encryption
and
so
on.
Some
further
words
that
we
planned
in
sandbox
API
is
in
other
work
that
we
planned
this
inbox
API
integration,
so
can
any
D
streams
will
be
able
to
provide
its
own
plugin
implementations
and
this
unlocks
a
lot
new
use
cases
such
as,
for
instance,
confidential
content
confidential
computing.
B
C
D
Right
yeah,
I'm
gonna
be
talking
about
the
kubernetes
interfaces
updates,
which
will
just
include
the
cni
Mike
Brown
will
do
the
CRI
and
NRI
changes
sweet.
So
let's
discuss
the
cni.
This
should
be
a
come
at
no
surprise
that
the
Dr
Shim
is
no
more
and
a
few
people
asked
me
to
clarify
where
the
cni
is
being
invoked
and
we'll
get
that
clarified.
For
those
that
don't
know,
the
cni
is
what
is
responsible
for
the
setup
and
cleanup
of
network
resources
and
kubernetes,
and
with
that
we
have
our
updates.
D
There
is
a
big
change
around
the
Run
pod
sandbox
CRI
method,
which
is
pretty
much
the
glue
magic
of
the
Pod
setup.
We
had
to
do
some
reordering,
which
was
to
fix
a
bug
that
surfaced
itself
as
IP
address
exhaustion.
If
the
container
exited
with
an
error,
we
captured
some
future
work
to
improve
this
Cleanup
in
the
event
of
failure.
D
So,
look
for
that
the
scene
I
maintainer
is
actually
intend
on
deprecating
the
loopback
plug-in.
It's
actually
not
a
really
true,
cni,
plug-in
and
funny
enough
that
there's
actually
a
surprise
that,
regardless
of
your
cni
of
choice,
you
still
need
the
loopback
plugin,
so
we're
we
intend
on
making
that
transition
rather
seamless.
You
should
even
know
it
was
there
and
you
probably
won't
even
notice
the
change
after
it
happens.
If
you're
curious
of
the
whys
and
hows
just
come,
find
me
with
the
removal
of
the
doctor.
D
Shim,
your
plug-in
and
network
configure
configurations
are
configurable
and
the
config
Tamil
and
the
defaults
are
the
same.
Opscene
I've
been
for
binaries
SCC
and
I
netd
for
your
network
configurations,
and
just
dovetailing
off
of
that.
Please
test
your
updates
and
not
in
production.
There's
a
few
hiccups
when
that
happened.
D
If
you
have
any
specific
questions,
please
come
find
me,
but
just
for
where
everything
starts
is
everything
starts
once
the
Pod
is
binded
via
the
Kube
scheduler
and
the
kublet
we'll
use
is
the
sync
pod
method
to
reach
out
to
the
API
server
to
determine
if
it
needs
to
schedule
a
pod
on
the
specific
node.
Once
this
happens,
the
kublet
uses
the
Ron
pod
sandbox
CRI
method
to
call
container
D
in
the
order
of
operations
from
a
high
level.
D
R,
the
sandbox
container
is
created,
the
network,
namespace
is
created
and
then
setup
is
called
and
setup
is
what
actually
calls
cni
ad,
but
that's
actually
a
we
use
a
go.
Our
own
wrapper
called
ghost,
unite
that
wraps
lib
cni
and
that
calls
add,
Network
lists,
execute
plug-in
with
result
and
then
exact
plug-in,
and
that's
actually.
D
What
calls
like
you
know,
cni
ad
and
then
at
that
point
the
sandbox
container
is
now
started
and
what
actually
starts
the
regular
containers
is
the
create
container
CRI
method
and
that's
called
by
the
Kubla
as
well
right
after
the
Run,
pod
sandbox
and
that's
what
starts
the
ephemeral
container
starts
the
knit
containers
and
then
starts
the
normal
containers.
Now
many
you
have
and
something
that's
often
missed
is
status.
So
if
you
don't
actually
have
your
cni
set
up
correctly
AKA,
you
don't
have
a
network
configuration
your
binaries
are
missing.
D
Status
will
actually
call
back
up
to
the
Kubla
and
say
you're,
not
ready,
so
you
won't
be
able
to
actually
schedule
pods
on
that
node
until
you
do
so
so
that's
often
missed
when
a
pot
is
actually
terminated.
The
kill
pod
function
is
called
is
was
responsible
in
the
kublet,
and
this
is
what
caused
the
stop
pod,
sandbox,
CRI
method
and
from
a
high
level.
D
This
stops
the
containers
and
then
stops
the
sandbox
containers
and
then
actually
calls
the
remove
function
in
go
cni
and
that
actually
calls
the
Dell
network
list
which
executes
your
cni
plugins
in
reverse.
And
then
you
call
the
exec
plugin
without
result,
exact
plug-in
and
then
the
network
namespace
is
deleted.
D
Eventually,
once
the
garbage
collection
kicks
in
and
that's
via
the
run
method
and
garbage
collection
is
run
on
a
timer,
that's
actually
what
calls
the
remove
pod
sandbox
and
if
this
pod
sandbox
is
not
stopped,
we're
actually
going
to
go
ahead
and
call
the
stop
pod
sandbox
and
then
that's
what
removes
the
containers.
Then
the
removes
the
sandbox
container
state
and
then
the
Pod
is
officially
gone.
C
C
Source
Community
right
we
work
together
with
the
kubernetes
teams.
We
work
together
with
the
cni
groups.
We
are
always
looking
for
new
reviewers
new
maintainers
Mr
Zappa
is
a
new
maintainer
for
for
containerdy
and
reviewer
he's
been
doing
a
great
job
for
us
around
cni.
He
called
me
up
about
a
year
ago.
I
think
it
was,
and
we
were
chatting
in
slack
saying
what's
going
on.
C
How
does
this
thing
work
and
the
next
thing
you
know
you
know,
he's
doing,
pull,
requests
and
and
doing
designs
and
making
it
all
work
better
and
helping
with
the
cni
group
as
well
and
I,
know
now
he's
working
with
Microsoft
on
the
same
same
topic
right.
We
we
see
a
lot
of
our
maintainers
move
to
different
companies
and
everything,
but
it's
it's
great
that
they
they
never
seem
to
leave
we're
keeping
our
maintainers.
We
enjoy
the
feedback
that
we're
getting
from
you
guys.
We
like
working
with
I'm
a
it's
a
node
member.
C
Try
to
get
you
know
into
all
of
the
calls.
If
I
can
we
work
with
the
cryo
team
and
other
container
run
times,
other
providers
of
the
apis
down
in
a
lower
level
and
and
the
CRI
this
container
runtime
interface
is
that
is
that
API
that
sits
between
kublet
and
or
container
runtimes?
That
are
you?
Can
we
call
them
container
runtime
integration?
You
know
proof
points.
We
we
make
sure
that
your
pots,
your
containers,
run
the
same
on
all
of
your
nodes
on
all
of
your
clouds
on
all
of
your.
C
C
If
you
will,
you
know
to
the
cry
structures
and
apis
and
right
now
we're
at
the
o25,
which
is
you
know,
a
mirror
of
what
the
current
version
of
kubernetes
is
at,
where
we
bring
in
these
we've
entered
this
stuff
in,
and
we
make
sure
that
the
apis
work
and
and
we're
always
looking
for
help
and
adding
new
features.
For
example,
there's
been
some
issues,
you
know
over
the
last
few
years
around
isolation
and
security.
Well,
we
we'd,
like
this
username
space,
support
to
be
one
way
that
you
can.
C
We
we
we
have
an
open
issue
and
we're
still
working
on
PR
details
to
be
worked
out
on
the
container
D
side,
but
we
know
what
need
what
we
need
to
do.
Checkpoint
restore.
This
is
a
interesting
effort.
That's
been
going
on
for
a
long
time
by
Adrian,
Rieber
and
and
a
bunch
of
people
down
in
Linux
I.
Don't
know
if
you
know
about
cry
you,
but
basically
the
the
idea
is.
C
If
you
freeze
your
container,
make
a
copy
of
everything,
including
memory
storage
on
disk,
and
then
you
could
should
be
able
to
debug
the
container
and
resume
that
container
and
restore
it.
But
we
don't
have
it
working
with
podjet
that
work
is
underway
and
we
currently
don't
support,
freezing
and
restoring
containers.
You
know
by
default,
there's
a
feature
Branch.
You
have
to
turn
it
on,
and
it's
mostly
focused
on
the
forensic
effort
right
now.
C
We're
going
to
do
things
like
secure
the
where
we're
storing
your
you
know
your
your
checkpoint,
some
stuff
going
on
there
registry
configuration
so
apologies
here,
it's
hard
to
configure
container
D.
Admittedly
we
we
do
support
configuring
to
point
to
your
Registries,
but
I
think
it's
important
to
point
out
that
we
don't
own
the
job
of
actually
doing
that
it
we
share
it
with
kubernetes.
Kubernetes
has,
as
you
may
know,
image
pool
secrets
and,
and
they
they
own.
The
mission
Kubla
does
specifically
of
pulling
your
your
images
through
the
cry
API.
C
So
when
a
new
pod
is
coming
up,
what
they'll
do
is
they'll
ask
the
container
runtime.
What
is
the
status
of
the
images
for
all
these
containers
that
I
need
to
run
in
this
pod
and
then
based
on
that
point,
they'll
start
calling
down
into
the
Container
runtime
in
an
imperative
way
asking
you
know
us
to
pull
those
images
and,
of
course
we
have
to
do
that
resolve
and
you
may
have
seen
some
earlier
apis
that
we're
gonna
we're
gonna
do
to
make
this
better.
C
Mike
Michael
put
together
an
interesting
project,
called
called
NRI.
It's
been
one
of
our
sub
projects
and
we're
improving
it
to
support
actual
running
programs
that
could
use
the
ttrpc
API.
To
request
for
changes
to
your.
You
know:
specs
your
oci
specs.
This
is
to
be
able
to
support
additional
device
types.
The
Intel
team
has
been
doing
a
lot
of
work
here.
They
would
like
to
be
able
to.
C
Oci,
of
course,
oci
is
the
it's
the
base
spec
for
containers,
we're
we're
working
continuously
with
the
oci
team,
we're
actually
maintainers
as
well
with
oci.
There
has
been
a
new
work
group
that
was
successfully
bringing
forth
artifact
support
into
into
the
spec
and
we're
working
out
some
details,
but
basically
we
we
would
like
to
be
able
to
not
just
pull
container
images,
but
also
artifacts
from
your
Registries
that
are
associated
with
those
container
images,
things
like
s-bombs
and
your
image,
scanning
results
and
and
other
types
of
projects
that
are
underway.
C
A
So
the
cncf
asked
us
to
put
up
this
QR
code
only
scan
that
if
you
think
we
did
a
great
job
and
you
want
to
submit
feedback,
no,
you
can
provide
any
feedback
on
the
talk
there.
I
don't
know
how
I
don't
see
Mike
set
up
in
the
room,
but
we
do
have
five
minutes,
I
think
before
the
official
end
of
the
session.
A
A
A
Yeah
good
point
in
yeah,
so
in
Prior
talks
you
know,
there's
the
CERN
folks
that
have
the
cvmfs
snapshotter
there
there
may
be
others,
hey
apologies.
There
are
a
lot
of
interesting
Innovations
and
snapchatters
yeah.
In
this
talk,
we've
only
mentioned
lazy
loading,
but,
like
Maxim
said
you
know
the
ability
to
have
a
proxy
snapshotter
means.
You
know.
As
long
as
you
handle
the
the
simple
snapshot
or
interface,
you
can
write
that
on
top
of
a
distributed
file
system
or
whatever
interface
you'd
like
yep
yeah,
what
what
was?
A
You
know
how
do
I
debug
so
yeah,
there's
a
few
great
blog
posts
that
have
been
written
since
the
1.24
release
on
some
of
those
topics,
and
you
know
I'm
sure
we
can't
exhaust
it
here,
but
one
is
that
there
are
some
nice
tools
that
people
have
written
to
detect
pods
that
are
trying
to
mount
like
the
docker
socket.
So
you've
got
containers
within
a
pod
that
are
actually
trying
to
do
something
interact
with
Docker.
That's
going
to
be
an
immediate
issue:
containerd
doesn't
offer
a
Docker
socket,
not
the
docker
API.
A
So
that's
kind
of
probably
the
biggest
thing
we've
seen.
People
have
issue
with
is
is
I
I
had
something
depended
on
on
Docker
I
had
something
that
mounted
expected
the
docker
interface
for
debug.
You
know
obviously
there's
a
there's
a
lot
of
improvement
in
tools
that
are
using
the
CRI,
so
CRI,
CTL
or
Kettle
or
whatever
you
want
to
call
it.
A
You
know
that
that
can
talk
to
any
CRI
runtime
and
do
a
lot
of
the
same.
You
know
list
containers
list.
Pods,
stop
start,
you
know
list
images,
so
some
of
the
same
command
line
tools
are
available.
Again.
You
know
you
lose
some
of
the
muscle
memory
of
well
I
knew
the
docker
commands
a
lot
of
those
similar
things
are
available.
We
have
a
simple
CTR
tool,
there's
nerd
CTL,
so
you
know
some
of
these
tools
are
filling
in
the
gaps
of
you
know.
I
can
still
do
poke
around
see,
what's
happening.