►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Kubernetes Conformance Deep Dive – William Denniss, Google (Intermediate Skill Level)
About William
William is a Product Manager at Google on Google Kubernetes Engine. He chairs the Kubernetes Conformance working group, and has a passion for interoperability and developer experience. Previously he worked in the OAuth community, authoring RFC 8252 and creating AppAuth, the leading open source OAuth client for native apps. In 2017 he presented talks at DockerCon, Google Cloud Next Tokyo and KubeCon Austin.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
B
C
B
Any
day
now,
so,
in
this
deep
dive
session,
we're
actually
gonna
be
going
into
the
weeds
and
we're
gonna
be
talking
about
testing
and
improving
our
testing
looking
at
our
coverage
and
all
the
goodness
that
that
kind
of
creates
the
they
kind
of
polymers
suite.
That
has
we
don't
and
without
further
ado.
Let
me
hand
it
over
to
Hackett.
D
Hello,
I'm,
a
hippie
hacker
and
Rowan
and
I
came
over
from
New
Zealand
to
talk
about
api's
new
API
snoop
is
a
collection
of
log
analysis
tools
and
ways
to
collect
logs
for
our
cake
mix
capex
or
our
kubernetes
api
consumers.
We
have
some
approaches
based
on
cube
admin.
We
get
some
improvements
on
and
we'll
share
a
little
bit
about
that
so
communities,
API
consumers
or
capex.
We
want
to
focus
on
looking
at
what
our
AP
is,
where
our
applications
are
using
in
the
field,
so
that
we
know
what's
important
for
us
to
test.
D
First,
how
do
we
get
access
to
these
logs?
We
tried
some
different
approaches.
We
started
off
with
some
man-in-the-middle
approaches:
side,
carring
things
in
IP
tables
and
redirecting
through
losing
envoy
or
SEO,
but
it
was
a
bit
complex.
So
we
looked
into
web
hooks
as
well.
We
added
support
for
kubernetes
Kubb
admin
so
that
you
can
deploy
your
API
servers
with
all
the
correct
parameters
to
enable
the
audit
policy
and
to
add
a
web
hook
destination
so
that,
as
every
API
flows
through
the
API
server,
we
have.
D
We
can
collect
it
without
needing
to
be
on
the
API
server
itself.
We
worked
on
some
workarounds
for
mini
Cuban,
cops
that
weren't
let
complete
in
the
future.
We
may
be
able
to
use
the
cluster
API
to
reconfigure
our
API
servers
to
have
information
that
needs
to
send
webhooks
for
our
audit
information.
D
D
So
now,
we've
got
these
great,
wonderful
snoops
on
our
API.
Why
don't
we?
What
can
we
do
with
that
information?
And
that's
where
our
analysis,
tools
that
we're
prototyping
and
putting
through
into
this
repo
are
starting?
We
have
a
log
review
that
can
load
audit
files
and
those
are
those
raw
logs
that
come
from
the
API
server,
we're
just
tailing
them
and
putting
into
a
file
and
loading
them
up
into
an
SQLite
DB.
The
remaining
commands
access
that
DB
and
output
them
to
various
CSV
files.
D
We
have
one
that
we
imported
the
Google
sheets
document
that
was
there
prior
with
all
the
information
on
what's
currently
being
audited
or
being
being
tested
for
conformance.
We
have
another
one
for
all
of
the
charts,
each
of
the
charts
that
we
tested.
We
basically
went
through
quite
a
few
of
the
stable
helm,
charts
spun
them
up.
We
look
for
ones
that
had
our
back,
because
we
knew
if
they
had
our
back
they're
going
to
be
capex
and
not
just
an
application
running
on
kubernetes.
D
D
Might
be,
we
do
something
similar
to
what
we
do
with
the
kubernetes
conformance
use,
API
snoop
in
a
way
similar
to
sonobuoy,
to
collect
the
information
for
that
application
and
certify
it
for
one
version
if
it's
stable
or
one
version,
if
it's
alpha
and
beta
and
the
version
and
the
next
version.
If
it's
a
hundred
percent
stable.
D
We
just
didn't
get
around
to
adding
the
ratio.
This
is
just
moments
ago,
and
we
came
up
with
this
this
approach,
so
we
have
all
this
data
now
and
we
have
it
for
each
of
our
applications
and
it's
not
just
the
metrics
on
which
API
call
we
used,
but
all
the
parameters
and
the
responses
for
that.
If
we
look
at
the
containers
or
in
the
and
the
images
we
pull
in,
it
might
be
possible
to
go
through
and
find
the
source
code
for
those
applications
and
Link
the
particular
call
to
our
data.
D
That's
making
the
call,
so
we
can
do
some
just
say
interesting
things
that
we
don't
know.
Yet.
This
is
a
graph
of
scaffold.
If
you
go
to
API
snoop
Diet
coop,
you
can
click
on
the
Apps
and
check
out
or
click
on
this
link
within
the
slide
deck.
It
shows
that
we
have
a
certain
percentage
of
stable
and
a
certain
percentage
of
beta
and
within
those
all
of
the
API
groups
within
kubernetes.
D
Sure
there's
two
things
I
wanted
to
highlight
here:
one
is
we
have
the
kubernetes
conformance
with
its
we
passed
and
we're
good
and
the
concept
of
possibly
having
kubernetes
Capek,
certifications,
conformance
and
if
it's
super
stiff,
it's
solid
on
that
version
or
stable.
Then
you
know
that
it's
going
to
work
on
the
next
release.
E
Sorry,
I
probably
should
have
given
the
content.
I
I
did
ask
Chris
to
do
this
work
so
I'm.
If
you're
unhappy
with
that,
you
should
blame
me
not
him,
but
most
certification
programs
tend
to
tend
to
certify
both
sides,
both
the
platform
and
the
application.
That's
running
on
it.
One
of
the
pieces
of
really
initial
feedback
that
I'd
love
to
get
is
that
I'm,
under
the
assumption
that
there's
not
a
lot
of
interest
in
certifying
applications
in
the
sense
of
I
want
to
run
WordPress
and
MySQL.
On
my
kubernetes
cluster,
everybody
agrees.
E
If
you
containerize
that
and
write
some
llamo,
it
should
run
on
a
certified
kubernetes
cluster.
So
there's
nothing
that
should
be
certified
there.
So
what
we
wanted
to
do
is
get
across
this
idea
of
oh,
but
there
are
a
set
of
tools
that
are
interacting
with
the
kubernetes
api
in
a
more
sophisticated
way,
and
so
scaffold
raft
brigade
case
on
it.
E
Misty,
oh
and
those
are
all
open
source,
but
there's
also
a
bunch
of
closed
source
ones
like
an
aqua
or
a
twist
lock
on
the
security
side
that,
at
least
in
principle,
any
those
tools
should
ideally
run
on
any
certified
kubernetes.
So
we
just
invented
this
term
yesterday
afternoon
and
that
you
could
talk
me
out
of
which
is
a
kubernetes
api,
consumer,
yeah,
I.
A
Was
a
question
because
are
you
are
you
measuring
the
api's
that
are
being
used
for
what
purpose?
What
just
just
fad
of
interest,
see
who's
using?
What
or
are
you
gonna
use
it
to
get
to
some
kind
of
checklist
or
something
at
some
point,
and
the
other
question
I
have
is
why
you
said
we're
not
going
to
do
conformance
testing
of
applications?
E
E
No
no
I
understand,
but
that
was
the
distinction
we
didn't
do
it.
We
definitely
were
missing
two
interest
slides
here
that
I
apologize
for,
but
we're
making
this
distinction.
It
may
be
over
the
artificial
between
just
an
application,
which
is
just
a
container
and
something
that's
exercising
the
kubernetes
api
and
anything
that's
exercising
the
kubernetes
api
is
something
that
you
would
run
and
run
this
tool
on
and
make
sure
hey.
Am
I
only
using
conformant
api's
that
are
available?
The
one
other
piece
that
I'll
just
add
in
and
then
I'll
get
off.
E
The
stage
is
the
other
sense
of
this
tool
is
even
if
we
never
go
forward
with
a
parallel
certification
program.
We
thought
it
would
be
very
useful
to
look
at
real
tools.
What
api's
are
they
exercising
today
that
we
don't
have
conformance
test
for
and
as
we
work
with
Globen
and
look
to
prioritize
those
that
it
would
be
nice
to
prioritize
the
important
ones
that
don't
have
coverage
yet.
D
Thanks
for
that,
then,
thanks
for
all
the
clarifications,
this
next
is
a
prioritized
list
of
the
times
that
scaffold
used
these
api's.
They
are
untested.
I
think
the
scaffold
team
would
probably
really
be
happy
if
these
were
in
conformance
case
on
it
uses
less
API.
So
you
can
see
that
it
is
pretty
beta,
though,
and
the
api's
that
are
stable
and
I
put
one
beta
in
there.
These
are
the
ones
that
the
case
on
a
team
would
be
really
interested
in
getting
into
the
conformance.
Suite
draft
was
another
one.
D
We
looked
at,
it's
about,
50/50
uses
a
lot
of
app
area
and
here's
it's
prioritized
list
of
what's
important
to
them.
This
is
the
sonobuoy
you'll
notice,
the
big
gray
portions.
There
that's
what
we're
missing.
This
is
the
it.
From
our
perspective,
we
have
a
little
different
results
than
amici
and
we'll
figure
out
what
those
why
those
are
different
at
some
point
later,
there's
the
whole
list
for
all
of
our
applications.
This
is
the
prioritized
list
for
all
the
applications
together,
so
secrets
service,
config
map
services,
pods.
D
Yes,
and
we
need
to
try
to
drive
those
tests
harder
and
I,
think
that's
what
this
slide
says:
I
need
to
figure
out
how
to
get
helm
test
or
project
specific
IDI
test
didn't
do
everything
that
that
application
needs
to
do
with
the
API.
This
allow
us
to
do
some
analysis
of
their
patterns
instead
answer
your
question.
D
This
is
the
the
external
analysis
we
captured
at
the
API
level.
So
if
you
can
get
the
application
to
do
everything
it's
supposed
to
do
we'll
prep,
you
know
we
will
know
what
that
output
is.
That's
it.
If
you
want
to
look
at
some
of
these
slides,
API
snoop
co-op
has
links
to
visuals,
then,
when
you
and
move
your
mouse
over,
you
can
see
the
different
statistics
on
that,
including
the
the
Excel
spreadsheet.
That's
been
updated.
They
have
all
of
these
applications,
whether
they're
using
that
API
or
not
and
I'll
hand
it
over
yep.
D
D
There's
several
ideas
that
I
have
that
we
haven't
flushed
out
yet
around
looking
for
patterns
with
all
of
the
applications
as
they
flow.
So
it's
the
the
data
data-driven
use
of
the
API.
So
it's
not
just
this
API
it'll
be
the
things
that
start
and
Flair's
gonna
be
constant
flows
and
those
are
probably
the
the
user
stories
we
need
to
do.
We
need
to
test
within
our
EDT
test
and
conformance
Suites,
but
that's
going
to
take.
G
Thanks
guys
quick
show
of
hands
how
many
of
you
were
in
the
intro
on
Wednesday,
okay,
that,
okay,
that
leaves
like
a
lot
of
people
who
are
in
there
so
I'm
quickly,
going
to
run
oh,
go
over
the
overview
and
then
deep
and
then
we'll
do
a
deep
dive
into
the
test.
Sic
testing
efforts
that
we
are
doing
here.
G
G
It's
this
about
a
year
back,
CN
CF
in
the
Kuban
it
is
community.
They
started
the
software
conformance
certification
program.
What
it
is
is
it's
a
quality
gate
for
cloud
providers
to
become
Kuban.
It
is
certified.
So
in
basically
it's
we
have
a
conformance
test
suite
which
have
set
of
golden
end-to-end
test
cases
which,
if
the
cloud
provider
passes,
then
the
customers
can
be
quite
confident
that
they
provide
the
goals
of
consistency,
Portability
and
all
support
all
the
features
that
come
with
that
version
of
communities.
G
It
was
launched
with
version
1.8,
with
32,
confirm
in
platforms
and
as
for
the
certification
policy
itself,
the
providers
need
to
get
certified
for
each
release
of
Kuban
IDs,
especially
running
that
tests
that
come
with
that
version,
and
they
also
need
to
renew
their
certification
every
year
to
ensure
that
they
stay
conferment
quick
overview.
The
program
itself
is
run
by
the
community
for
the
community
under
the
ages
of
cnc.
G
G
If
you
need
more
info
on
this,
we
can
always
discuss
it
offline
in
interest
of
time,
and
we
also
have
some
tooling
around
publicly
publishing
these
results
and
working
closely
with
slick
testing,
as
I
said,
to
drive
and
adopt
reliable,
tooling
and
proactively,
reaching
out
to
providers
to
get
them
to
certify
and
again.
The
reason
for
doing-
and
the
second
mission
is
to
increase
the
conformance
test
coverage
itself
right
now.
These
are
our
coverage
numbers
in
v110.
We
only
have
about
18%
of
the
ape
stable
API
covered.
G
While
this
is
a
good
trend,
still
the
baseline
is
pretty
low
and
we
want
to
get
it.
We
want
to
drive
the
coverage
number
high
to
be
able
to
confidently
certify
any
provider,
and
our
goal
is
to
increase
this
coverage
by
40%
in
q3
and
for
the
top
three
API
is
that
will
be
identified
using
API,
snow
or
other
API
coverage
tools.
We
want
to
drive
that
coverage
to
80%
and
Metra
would
talk
about
how
we
plan
to
do
this.
C
C
You
have
to
join
the
weekly
working
group
meetings
and
the
slack
Channel,
and
basically
also
the
sig
testing,
has
weekly
meetings
and
slack
channels
as
well
and
how
to
increase
coverage,
basically
prioritizing
API
endpoints,
which
ap
hacker
went
over
and
a
really
good
overview
of
how
to
prioritize,
and
we
were
working
with
a
contracted
engineering
group
from
globe
and
who
are
vendors,
who
were
actually
actively
working
on
writing
these
tests
right
now
and
enlisting
sick
participation
both
for
reviewing
these
tests,
as
well
as
like
approving
all
the
changes.
The
globin
vendors
are
working
on.
C
These
changes
and
we'd
love
for
people
to
step
up
in
the
community
to
start
reviewing
their
peers
and
some
guidelines
on
how
to
write
tests
so
I'm
going
a
little
faster
in
the
interest
of
time,
but
basically
working
with
the
SIG's
to
identify
all
the
test
gaps
and
then
figure
out
like
what
tests
to
be
added
to
the
conformance
test
suite.
Sometimes
there
may
be
existing
tests.
C
That
would
be
a
confirmands
test,
but
we
are
simply
not
aware
or
have
not
added
it
to
the
suite
and
review
the
end-to-end
test
scenarios
to
add,
and
we
should
be
doing
the
test
by
user
journey
and
using
conferment
sit
instead
of
the
ginko
eight
label
to
describe
the
tests
adding
very
like
less
flaky
or
non
flaky.
Tests
to
the
suite
is
ideal
and
verifying
that
they
pass
on
multi
node
clusters,
as
well
as
single
node
clusters,
especially
for
networking.
Yes,.
H
I
C
Think
we
would
like
it
for
folks
to
start
adding
it
as
they
write
to
the
test
as
well
right
now
and
not
added
in
retrospect,
as
we
did
for
the
first
round.
Yes
exactly
and
yeah,
likewise
for
the
PRS,
when
they
come
in
also
versioning
them
with
the
release
number.
Those
should
all
be
requirements
going
forward
and
not
like
post
requirements
and
yes,
yes
and
cigarette
texture
we'll
be
reviewing
and
approving
the
PRS,
but
these
are
the
things
that
they
will
be
looking
for.
C
Two
week
is
the
SLA
for
reviewers
and
ready
to
focus.
Basically,
these
are
some
overall
guidelines
for
where
to
focus.
Attention
like
the
key
api's
are
very
important,
but
like
focusing
on
behaviors
that
are
likely
to
be
swapped
out,
focusing
on
part
feature
testing
and
the
open,
API
spec
deep
dives
are
needed
for
certain
SIG's
and
certain
applications
as
well
and
making
sure
we
test
at
C
D
at
a
holistic
level.
At
the
API
level
and
not
internal
implementation.
Details
test
shouldn't
be
hitting
a
at
C
D
directly.
C
We
have
a
lot
more
best
practices
and
things
to
focus
on
I
have
a
document
that
I
can
share
out
more
widely,
so
people
know
what
ready
to
focus
their
attention
when
they
are
writing
these
tests,
and
we
also
have
been
developing
more
of
a
hermetic
conformance
testing
tool
with
darker
and
darker.
There
are
some
caveats:
it
does
not
work
with
multi
node
networking,
but
then
for
a
lot
of
conformance
tests.
This
should
provide
a
way
to
be
able
to
run
these
tests
extremely
quickly
and
write,
like
conformance
tests
that
are
not
flaky
and
virtualizing.
C
The
cluster
on
one
machine
is
what
we
are
doing
and
it
can
be
easily
tear
torn
down,
because
it's
darker
and
darker
like
when
you
tear
down
the
master,
docker
gets
destroyed,
many
swappable
components
can
be
tested,
and
these
are
some
of
the
limitations
of
darker
and
darker.
We
don't
expect
it
to
be
the
test
Hall
solution,
but
for
a
large
case
of
test,
this
might
make
it
extremely
faster
and
so
I'll
give
it
to
ash
to
talk
about
reporting,
test
results.
G
So
I
touched
upon
at
this
point
briefly.
One
of
the
things
that
the
confirm
is
working
group
is
looking
at
is
to
provide
a
easy
way
for
contribute
for
providers
to
contribute
their
test
results
for
for
it
to
be
published
in
a
public
space
where
everybody
has
visibility
to
it.
There
are
multiple
reasons
to
it.
The
way
the
conformance
program
works
right
now
is
the
providers
run
the
tests
on
their
own
and
they
submit
the
artifacts
for
review
and
certification
by
CN
CF.
G
While
this
works,
it
doesn't
there's
no
gear,
there's
no
way
for
us
to
verify
that
the
providers
did
run
it
and/or.
They
ran
it
on
the
right
version
unless
the
users
can
run
the
test
on
their
own.
So
the
so
publishing
it
to
a
common
place.
Kind
of
raised
raises
awareness,
a
two
level
one,
the
the
providers
themselves
can,
you
know,
can
be
alerted
to
signal
early
signals
of
when
a
test
fails
and
number
two.
G
If
there's
some
other
regression
elsewhere,
we
can
know
the
impact
on
the
scope
of
which
providers
it
affects
so
right
now.
We've
we
kind
of
the
test
grid
is
the
chosen
tool
for
dashboarding
and
monitoring
these
results.
This
is
not
something
new
for
those
who
are
familiar
with
testing
in
the
open
source.
World
tester
is
been
the
monitoring
and
reporting
tool
for
hundreds
of
end-to-end
tests
that
are
run
both
and
pre
submits
NCI
pipeline.
G
Currently
so
in
the
last
couple
of
months
on
suggesting
developed
an
upload
tool
which
basically
consumes
the
artifacts
that
come
out
of
the
conformance
test
or
into
intense
and
it
uploads
the
art
it
parses
it
and
uploads
it
in
a
format
from
which
tests
great
can
pull
the
results
and
display
it.
So
this
this
tool
is
super
easy
to
use
both
manually
and
to
be
plugged
into
the
providers
CI
pipeline,
and
it
integrates
well
with
existing
conformance
tools
like
any
test.
G
If
your
test
rigorous
cube
test,
also
no
boy,
it
kind
integrates
well
with
both
and
here's
a
quick
overview
of
how
it
works.
So
there
are
multiple
cloud
providers
they
run.
They
trigger
end
to
end
tests
using
this
job.
Ideally,
and
this
test
run
on
there,
they
okay,
it's
cluster.
It
takes
the
test
from
the
version
of
the
kids.
Will
they
trying
to
certify
and
the
test
produces
the
e
to
eat
logs
and
the
J
unit?
The
tools
then
consumes
these
inputs
and
provides
basically
a
couple
of
output
files.
G
The
timestamp
to
JSON
VEX
start
in
the
stop
time
and
the
run
result
the
overall
success
and
then
pushes
it
to
GCS
buckets
specific
to
that
particular
cloud
provider
and
then
for
each
provider
they
are
given.
They
provision
themselves
in
test
grid
with
a
dashboard
and
the
specific
dashboard
gets
access
to
the
appropriate
GCS
bucket
from
which
test
grid
is
able
to
pull
the
results
on
a
periodic
basis
as
and
when
they
are
available.
So
this
is
the
current
conformance
dashboard.
G
So,
as
you
can
see,
we
have
others
say
end-to-end
test
being
published
there
as
well,
so
we
have
a
specific
tab
for
conformance
and
we
can
see.
We
have
all
the
conformance
results
under
that.
So
currently
we
have
OpenStack
as
one
of
the
provider
who
is
been
contributing
results
to
us
on
a
regular
basis.
So
yeah
we
have
different
versions
of
tests
that
we
run
for
them
and
we
can
see
that
we
are
able
to
monitor
their
conformance
pretty
closely
I'll
hand
it
over
to
Christy
for
some
new
framework
that
she's.
B
J
Okay,
okay
cool-
this
is
gonna,
be
really
brief:
I'm
Kristy
I'm,
a
software
engineer
at
Google
and
I
want
to
talk
about
the
potential
for
a
new
conformance
test
framework.
So
if
you
take
a
look
at
the
existing
conformance
test
framework,
this
is
where
it
lives,
and
so
I
took
a
look
at
it,
and
the
number
of
commits
that
were
in
that
directory
is
nearly
1500
and
those
1500
commits
came
from
a
hundred
and
eighty
five
different
people
and
then,
if
you
start
looking
at
the
email
addresses
that
they
used,
there
are
33
different
domains.
J
So
that's
not
necessarily
33
different
companies,
but
it
actually
might
even
be
more
because
Gmail
like
who
knows
what
companies
those
people
came
from.
So
what
we're
saying
is
that
the
end-to-end
framework
has
grown
organically
over
nearly
1,500
commits
by
nearly
200
people
at
potentially
more
than
20
different
companies,
which
I
think
raises
some
interesting
questions
like
how
do
you
maintain
things
like
quality
consistency
and
standards
over
that
many
contributions,
I
think
within
one
company,
it's
hard
to
do
that.
So
how
are
we
going
to
do?
How
did
we
do
that
with
the
test
framework?
J
Also
I
think
an
easy
answer
to
that
is
well.
The
reviewer
is
going
to
is
going
to
enforce
those
things,
but
that
many
commits
that
many
pull
requests
I
think
it's
probably
too
much
to
expect
the
reviewers
to
be
focusing
on
the
particular
standards
of
the
conformance
test
framework.
So
the
next
question
is
ginko.
J
So
if
you're
familiar
with
these
end-to-end
tests
at
all,
you'll
know
that
they
were
built
with
ginko
and
after
looking
into
it,
we've
come
to
the
conclusion
that
ginko
is
the
best
bdd
framework
and
go,
but
then
we
have
to
ask:
do
we
really
want
to
be
using
BDD
when
we
have
all
these
contributors
coming
in?
Do
we
want
them
to
have
to
learn
all
the
best
practices
for
BDD
in
order
to
contribute?
So
we
think
it's
time
to
revisit
this
framework
and
prefer
a
few
things.
We
want
go
style
in
our
tests.
J
We
want
contributors
to
be
able
to
ramp
up
quickly.
We
want
to
be
able
to
debug
failures
quickly
and
we
want
to
take
advantage
of
the
hermetic
test
environment
that
was
just
mentioned,
and
another
thing
I
think
the
community
is
hungry
for
a
end-to-end
testing
framework
that
they
can
use
for
other
kubernetes
based
testing,
so
in
inclusion.
I
think
that
it's
time
that
we
take
a
look
at
having
a
new
framework
and
that'll
mean
better
contributions
and
I
mean
we
have
more
contributors,
and
that
is
it
for
that.
I
will
have
a
proposal.
J
I
I
I
J
C
I
B
Thanks:
hey
Ron,
if
you
know
what's
interested,
that
was
a
huge
amount
of
content
and
absolutely
amazing,
just
love,
seeing
all
the
work
everyone's
doing
in
this.
If
people
have
time,
let's
just
continue
the
conversation
outside
the
room
and
the
next
calls
I
think
in
three
weeks
so
should
be
a
lot
to
discuss
them.
Thanks,
Aaron.