►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Promoting Kubernetes CI/CD to the Next Level - Tim Pouyer, IBM
Many companies and organizations have adopted CI/CD processes in order to help deliver applications running on Kubernetes quickly, transparently, and with automated tests. While this is a desirable goal, it gets more complex when developing a management layer on top of k8s, especially when both images and helm charts are involved. In developing IBM Cloud Private, we have implemented a CI/CD process that automates promotion through a series of quality gates where we 1) Ensure that all charts (43) and images (135) are in sync across three supported architectures 2) Deploy instances of ICP clusters with different topologies, management services, and infrastructure 3) Run automated and manual functional and security regression tests against those clusters. Since implementation, have we have been able to iterate more quickly by discovering issues earlier in the development process.
https://sched.co/Nrma
A
A
So
it's
about
how
we
actually
build
and
test
and
deploy
full
kubernetes
clusters
right,
not
just
applications
into
a
running
cluster
right.
So
I
think
this
is
very
applicable
to
anybody
that
I
know,
there's
not
a
lot
of
people
that
that
actually
build
their
own
kubernetes
distribution,
but
I
think
the
automation
practices
that
we
kind
of
I'll
talk
about
here
are
very
applicable
to
anybody.
A
That's
adopting
kubernetes
I
come
from
a
very
DevOps
background,
been
doing
DevOps,
I
guess
for
about
14
years
now,
and
I
will
tell
you
that
that
the
principles
for
how
we
deploy
applications,
which
kubernetes
takes
care
of
a
lot
of
that
for
us
right.
They
with
readiness,
probes
and
liveness
probes.
We
can
start
treating
our
applications
like
cattle
rather
than
pets.
Right,
that's
a
very
famous
term.
A
You
know
talk
a
little
bit
about
Who
I
am
what
my
background
is,
what
IBM
cloud
private
is
and
how
we
got
to
where
we
did
and
then
we're
gonna
start
looking
at
DevOps
principles
or
shrink-wrap
software,
and
this
term
might
actually
date
me
a
little
bit
for
the
younger
crowd.
Shrink-Wrap
software
is
the
software
that
used
to
be
sold
in
a
box
on
a
shelf,
and
it
was
shrink-wrapped
I
know
nobody
does
that
anymore,
but
that's
kind
of
what
I
mean
by
shrink-wrapped.
It
means
it's.
A
A
A
So
I'm
excited
about
that
all
right.
So
what
is
IBM
cloud
private,
so
IBM
cloud
private
is
an
on-prem
kubernetes
distribution.
It's
really
intended
to
be
used
as
a
private
cloud
and
enterprise
data
centers.
It's
it's
basically
IBM's
flavor!
Well,
I,
don't
say:
IBM
slaver
its
IBM's
distribution
of
kubernetes
for
on-prem
support
right.
A
It
has
enterprise
security
and
governance
capabilities
that
are
kind
of
built
in.
We
do
quarterly
releases.
So
every
quarter
we
do
lise.
So
we
can
try
to
keep
up
with
the
open-source
version
of
kubernetes.
It
was
really
hard
to
do,
but
we
try
to
keep
up
with
it.
We've
integrated
over
200
open-source
tools
and
products
within
two
kubernetes
to
have
these
enterprise
capabilities,
and
we
have
about
180
developers
located
in
about
four
different
countries,
including
a
lab
right
here
in
China.
A
So
how
did
we
get
here?
Well,
this
started
out
as
a
as
a
small
project
in
the
Ontario
lab
and
grew
from
there.
It
was
a
couple
dozen
developers
that
were
putting
together
their
own
kubernetes
I,
don't
it's
like
distribution,
but
they
were
cobbling
together,
kubernetes
and
writing
their
own
installer
to
make
it
easy
for
customers
to
stand
up
and
run
kubernetes.
A
We
did
this
because
we
kept
adding
more
functionality
to
the
distribution
and
then
we
also
added
support
for
PowerPC
and
s/390
X.
So,
as
390x
is
ibm's
z,
mainframe
right,
so
we
can
run
kubernetes
on
z,
on
power
and
on
x86
we
also
added
600
customers
in
two
years.
That's
a
massive
amount
of
growth.
I
don't
have
statistics
on
the
number
of
lines
of
code
that
we've
added.
Obviously
we
adopt
a
lot
of
the
open
source
and
kubernetes.
We
have
our
own
kind
of
management
layer.
A
A
So
DevOps
you
know,
was
our
kind
of
solution,
which
is
why
they
brought
me
in
to
basically
try
and
implement
a
DevOps
principle
or
mentality
or
for
our
process
right.
Our
process
was
largely
still
the
process
that
was
written
by
a
few
devs
that
were
in
Ontario
right.
That
was
mainly
focused
around
I
want
to
build
this
thing
from
my
laptop
right
and
that's
great
when
you're
a
small
dev
team
right,
a
very
talented
people.
A
What
happens
when
you
become
you
know
a
hundred
eighty
people
in
four
different
countries
and
five
different
time
zones
is
that
doesn't
work
anymore
right.
You
have
to
coordinate.
You
have
to
bring
all
the
pieces
together
that
are
getting
built
at
all
different
times,
and
you
have
to
make
sure
that
it
all
comes
together
and
works,
and
it's
tested
and
validated
before
it
ever
goes
out
the
door.
So
there
are
way
too
many
topics
to
cover
in
DevOps
since
nobody's
actually
ever
really
defined
it.
A
So
I'm
going
to
focus
on
on
these
three
automation,
shift,
left
and
culture
all
right,
I
think
everybody
knows
that
automation
is
shift,
left
I'll,
explain
a
little
bit
more,
but
that's
the
idea
of
taking
things
that
you
would
normally
do
towards
the
end
of
release
process
and
doing
them
earlier
on
right.
So
some
of
the
testing
that
you
would
do
at
the
very
end
of
a
product
before
release,
you
really
want
to
shift
that
to
the
left.
We
want
more
of
that
to
happen.
A
A
All
right,
so
you
know
when
I
joined
ICP
in
January
of
2018,
so
what
over
year
and
a
year
and
a
half
ago,
you
know
our
first.
My
first
focus
was
automation.
Right
we've
got
to
automate
as
much
as
possible,
like
that's
a
core
principle
of
any
DevOps
or
C
ICT
processes
to
automate.
So
we
had
a
few
guiding
principles
in
our
automation
right.
We
wanted
it
to
be
push-button.
A
We
wanted
it
to
be
full
stack
right,
our
product,
bundles
and
an
installation
via
ansible
scripts,
into
a
docker
image
to
make
it
easy
for
you
to
install
the
software,
but
it
does
nothing
about
provisioning
infrastructure
right,
so
we
needed
to
be.
We
needed
the
automation
that
be
full
stack
provision,
the
infrastructure
from
start
to
finish,
and
then
we
also
needed
to
need
it
to
be
easily
shared
easily.
It's
easy
to
use.
A
A
So
what
we
ended
up
coming
up
with
is
we
actually
forked
a
project
that
was
out
on
github
by
a
group
of
people
called
cloud
posse,
it's
called
build
harness
and
what
this
does
is
it
utilizes
make?
So
all
of
our
developers
were
already
using
make
to
build
all
their
components
right
and
makes
been
around
for
years.
It's
kind
of
one
of
those.
A
If
you
understand
it,
everybody
thinks
you're
a
genius,
and
if
you
don't
understand
it,
then
you
know
just
like
everybody
else,
but
makes
a
great
tool
and
it
works
across
all
the
different
architectures
right
and
what
built
harness
gave
us.
It
was
the
ability
to
create
a
set
of
shareable,
make
targets
that
we
could
store
and
github.
A
All
right,
so
this
is
a
an
example
make
file
sitting.
In
an
example,
you
know
build
template,
alright,
the
build
project
that
we
set
up
on
on
our
github.
It
just
sets
up
some
some
generic
properties.
These
are
you
know,
configurable
for
everybody
else.
You
can
see
that
I
only
have
this.
One
include
statement.
Well,
I
have
other
include
statements,
but
those
don't
do
anything.
So
this
includes
statement
here
pulls
into
our
bill
furnace.
Our
fork
build
harness
and
with
that
included,
then
I
can
start
to
do
fun.
Things
like
make.
A
See
it
alright,
so
with
that
I
now
have
these
shared
make
targets
that
anybody
can
include
in
their
own
make
file
to
build
on
top
of
the
logic,
that's
already
there.
Alright,
this
is
a
great
project
because,
like
github,
where
people
go
and
they
build
their
code
and
they
put
it
out
there
for
others
to
share,
and
then
communities
can
come
and
develop
across
those
and
enhance
things
and
make
them
better.
That's
exactly
what
this
bill
harness
does
right.
A
lot
of
these
targets
are
shared
across.
A
A
They
can
use
any
of
these
make
targets
in
their
own
make
file
and
we've
seen
some
of
the
squads
contribute
some
targets
that
weren't
in
here
to
the
build
harness
and
now
all
of
a
sudden
they
start
getting
shared.
So
we
have
linting.
We
have
stuff
that
do
build
docker
images
we
have
stuff
to
build,
helm,
charts
and
and
all
sorts
of
fun
fun
targets
right
that
all
of
our
squads
can
use,
and
you
can
stay
up
to
date.
That
way,
we
also
implemented
all
these
deploy
targets.
A
A
Into
this
include
in
our
built,
harness
which
may
sit
basically
makes
all
the
automation
that
the
CI
CD
team
has
written
using
terraform
available
to
all
the
squads.
So
when
a
squad
is
developing
their
particular
service,
they
can
build
their
docker
images.
They
can
build
their
home
charts,
they
can
push
them
in
the
artifactory
or
factories,
what
we
use
as
our
artifact
repository
and
then
they
can
deploy
them
all
from
just
simply
including
this
build
harness
into
their
own
their
own
make
file.
A
So
this
gives
them
access
to
be
able
to
test
their
deployment
against
AWS
or
against
IBM
cloud
or
OpenStack
or
VMware,
and
even
test
against
the
different
architectures,
so
Z
PowerPC,
II
and
x86
I'm
all
accessible
to
them.
It's
all
done
exactly
the
same
across
all
the
spots.
Nobody
has
a
custom
deployment,
it's
all
maintained
at
the
CI
CD
level
and,
of
course
we
do
take
contributions
from
our
squads.
A
So,
like
I
said,
we
had
to
support
all
of
our
different
cloud
providers
all
right.
The
way
we
executed
on
this
was
terraform
terraform,
combined
with
the
ansible
that
was
already
in
our
installer,
made
it
very
simple
for
us
to
use
terraform
just
to
stand
up
the
infrastructure,
and
then
we
wrote
a
terraform
module
to
call
our
installer
to
actually
execute
the
installation
of
the
infrastructure
that
we
stood
up.
A
We
also
added
capabilities
within
the
terraform
to
be
able
to
hand
different
topology
deployments
within
these
cloud
providers,
so
a
chain
on
h.a,
including
cluster,
FS
or
cephalic,
and
we
continue
to
add
more
and
more
options
to
those
Terra
forms.
Some
of
those
Terra
forms
are
available
publicly
on
github.
Some
of
them
are
still
internal
working
on
open
sourcing
them,
so
we
utilize
artifactory
or
our
packaging,
docker
images
or
storing
of
docker
images
and
helm,
charts
I,
really
like
artifactory
for
a
couple
of
reasons
number
one.
A
Number
two
is
that
we
use
and
I'll
talk
about
this
in
a
second
we
use
a
promotion
pipeline
promotion
pipeline
allows
us
to
put
gates
quality
gates
within
our
process,
and
artifactory
just
makes
that
a
lot
easier.
If
any
of
you
have
ever
tried
to
do
a
promotion
pipeline
with
docker
images,
the
one
thing
you're
gonna
find
out
is
that,
as
you
start
to
automate
that-
and
you
run
docker
pull
darker
tag,
docker
push!
Well,
if
you
do
that
across
one
image.
That's
you
know
ten
megabytes
or
twenty
megabytes
or
30
megabytes
in
size.
A
It's
not
a
big
deal,
but
if
you
have
300
images
and
they're
averaging
between
you
know
250
to
a
gig
in
size,
200
team,
eggs
to
a
gig
in
size.
Well,
then,
a
level
sudden
takes
6
hours
to
do
that
and
what
artifactory
does
for
us
is.
It
allows
us
to
run
a
set
of
rest
calls
that
just
change
a
pointer
in
order
factory.
So
when
we
promote
from
one
repository
to
another,
all
we're
doing
is
telling
artifactory
to
change
the
pointer
right
and
it
runs
orders
of
magnitude
faster,
like
20
minutes
instead
of
hours.
A
We
also
most
of
our
services
run
as
helm,
sharks,
so
they're
deployed
as
home,
charts
and
we've
written
a
bash
script
that
can
identify
the
helm
chart
and
then
identify
each
docker
image
and
they're
tagged
within
the
home
chart,
and
so
what
that
means
is
that,
as
we
start
to
promote,
builds
through
our
pipeline,
we
can
run
that
script
against
the
home
charts
that
we've
deployed
successfully.
We
know
that
they
work
to
identify
all
the
docker
images
and
the
tags
that
need
to
go
along
with
the
home
chart.
A
So
shift
left
so
I
spoke
about
this.
A
little
bit
earlier
shift
left
is
the
idea
that
I
want
to
take
certain
tasks
like
canari,
deploys
integration
tests,
functional
tests,
owner
ability,
scans,
docker
image
scans
and
instead
of
rolling
them
right
before
the
release.
I
want
to
run
them
much
earlier
in
my
process.
A
So
this
really
works
well,
when
you
use
promotional
stages
and
I
know
that
you'll
probably
see
if
you
read
on
Google
or
you,
google,
it
that
using
promotional
stages
with
docker
images
is,
they
might
even
say
it's
an
anti-pattern,
and
that's
probably
for
the
aforementioned
of
the
before-mentioned
problem
of
having
to
download
the
docker
image
and
retag
them
and
upload
them.
It
just
takes
too
much
time,
artifactory
kind
of
eliminates
that
for
us
a
little
bit
by
using
the
stages
promotional
stages,
we
actually
get
to
put
a
process
on
our
release
right.
A
So
it's
not
just
everybody's
building,
something
at
all
different
times,
and
you
know
we
never
know
exactly
what
state
the
product
is
in
at
any
given
time
in
the
release.
We
can
actually
tell
just
from
our
stages,
so
we
start
at
scratch,
which
is
where
all
of
our
PR
builds
go
integrations
where
our
master
builds
go,
and
these
are
updated
continuously.
We
have
like
I
said
earlier.
We
have
squads
that,
are
you
know
in
China
and
in
India
and
Canada
and
United
States,
so
all
different
time
zones
these.
A
These
particular
repos,
like
integration,
is
changing
all
the
time
continuously,
which
is
great,
except
for
when
you
need
to
test
something,
because
if
you
deploy
out
of
integration,
you
don't
necessarily
know
what
you're
deploying
right,
because
it
can
actually
change
underneath
you,
you
could
go
through
the
deployment
and
by
the
time
you
get
through
the
deployment
which
takes
about
an
hour
hour
and
a
half,
and
you
start
testing
which
takes
another
30
minutes.
Then
you
get
your
results
back.
A
You
may
find
that
there
was
an
issue,
but
you
go
back
and
look
at
the
image
and
it
may
not
be
the
image
that
was
actually
deployed
when
you
were
testing
it.
So
the
way
we
solve
this
problem
is
we
use
a
daily
repo
and
twice
a
day.
I
know
it
sounds
stupid
because
it's
a
daily
repo,
but
twice
a
day.
We
run
a
promotion
process
from
integration,
we're
basically
just
taking
a
snapshot
out
of
integration
right
and
moving
it
into
daily,
and
nobody
has
access
to
that.
A
Except
for
our
CI
CD
squad,
and
then
we
deploy
our
Canary
deploys
out
of
daily
to
validate
what
we've
seen
in
integration.
This
gives
us
the
feedback
of
where
we
are
exactly
in
the
process.
What's
working,
what's
not
working
and
we
can
go
back
to
our
squads
and
say
you
have
a
specific
problem
you
get
it
resolved.
Sometimes
they
tell
us,
we've
already
fixed
that
it
we
push
it
integration
great,
we'll,
promote
that
up
or
run
the
canaries
again.
A
So
the
stages
are
very
important
for
us
to
understand
where
we're
at
in
our
cycle
and
our
release
cycle
and
like
I,
said
earlier.
The
the
stages
also
give
us
give
us
the
ability
to
implement
quality
gates.
So
you
know,
whenever
you're
producing
a
product,
whether
it's
a
car
or
you
know
a
t-shirt
right,
there's
usually
somebody
that
inspects
it
for
quality,
right
and
and
everybody
should
know
that
you
should
be
running
some
degree
of
quality
testing
on
any
software
before
you
release
it
to
the
public.
A
What
we
do
is
we
utilize
our
staging
process,
our
promotion
process
to
enforce
quality
gates.
So,
like
I
said,
we
go
from
integration
that
daily
once
a
day,
that's
a
schedule
or
twice
a
day.
It's
a
scheduled
process.
We
run
Canaries
out
of
our
daily
out
of
our
daily
repo.
We
then
execute
regression
tests
against
that
appointment
and
if
it
all
passes
so
basically
every
cloud
provider.
We
support
we're
running
Canaries
on
and
then
we're
running
regression
on
every
cloud
provider.
A
If
it
all
passes,
then
we
will
promote
to
edge
an
edge,
is
the
first
stage
in
our
promotion
pipeline
that
everybody
within
IBM
has
access
to
so
because
we
use
artifactory
because
artifactory
is
offered
within
IBM
as
a
SAS
offering
it's
basically
a
software-as-a-service
everybody
within
IBM
has
access
to
the
repository
is
in
order
factory.
If
you
make
them
public
to
IBM,
so
edge,
stable
and
release
are
all
public
to
IBM
and
anybody
can
download
them
so
at
the
so
from
daily
to
edge.
We
are
running
Canaries
and
regressions.
A
We
run
promote
from
edge
into
stable
at
the
boundaries
of
our
Sprint's,
so
we
we
break
down
into
Sprint's.
At
the
end
of
a
sprint,
we
will
get
something
into
stable,
which
suggests
that
it's
a
slightly
more
stable,
API,
less
changes
going
on
between
Sprint's
and
then.
Finally,
we
have
release
right
and
at
each
of
these
stages
we
have
different
criteria
for
how
you
promote
from
one
repository
to
the
next,
and
if
it
doesn't
pass,
then
we
don't
proceed
right
so
by
exposing
these
repositories
to
IBM,
basically
IBM
cloud.
A
A
We
call
that
our
content
teams,
ok,
our
content
teams
want
quick
feedback
right,
so
they
want
to
be
able
to
deploy
their
content
onto
our
latest
edge,
builds
and
discover.
If
there's
a
problem,
if
we've
broken
some
of
their
API,
that
they
were
using
or
you
know,
we've
introduced
some
new
kubernetes
api
that
that
they
would
like
to
use.
A
So
they
can
pick
up.
These
builds
out
of
edge
on
literally
a
daily
basis.
Sometimes
we
don't
get
something
into
edge.
Every
day,
depending
on
how
the
canary
Canaries
deploy
and
how
we
do
on
that,
but
the
sooner
we
can
get
it
to
them,
then
the
sooner
they
can
give
us
feedback.
And
what
that
creates.
Is
this?
The
sense
of
we
might
have
a
good
idea
out
of
development.
Our
content
teams
might
pick
that
up
and
say
you
know
we
like
the
idea,
but
it
would
be
better
if
you
did
it
this
way.
A
Right
and
good
ideas
become
great
ideas
based
off
of
that
shorter
short
and
feedback
loop
right.
We
also
utilize
slack
internally.
All
of
our
squads
have
their
own
channels.
So
if
there's
a
particular
problem
with
a
particular
API,
a
particular
service,
anybody
can
go
to
that
channel
and
ask
questions
and
get
feedback.
See.
Icd
also
runs
their
own
channel
where
we
communicate
to
the
development
organization
as
a
whole,
whether
Canaries
are
passing
or
failing,
or
we
need
assistance
with
something
we
couldn't
figure
out.
A
So
we
all
utilize
slack
in
that
way,
and
then
we
also
utilize
github.
All
of
our
issues
are
opened
in
github
and
anybody
can
open
up
an
issue.
Anybody
within
IBM
can
open
up
an
issue
on
github
and
that's
how
we
track
improvements
to
the
product.
So
all
that
is
focusing
on
shortening
that
feedback.
Loop
and
shifting
everything
left.
A
A
Culture
really
has
to
do
with
how
we
change
what
developers
are
used
to
doing
right
and
it's
not
just
developers.
It's
everybody.
It's
it's.
You
know
offering
managers
its
development
managers,
it's
the
salespeople
I
mean
culture.
Is
it
goes
across
the
board,
it's
hard
to
change
all
right,
it's
hard
to
change
a
culture
once
something
gets
kind
of
set
in
stone.
Then
it's
almost
like
fighting.
You
know
an
endless
battle
developers
don't
like
to
give
up
control,
that's
a
given.
They
don't
like
to
change
their
processes.
Even
if
their
process
doesn't
make
sense.
A
A
So,
like
I
said
earlier,
we
we
forced
all
of
the
squads
that
give
up
control
of
their
not
their
particular
builds
but
of
the
build
of
the
product.
So
every
squad
still
owns
their
individual
build
process
for
their
their
particular
service,
so
they're
still
building
their
docker
images,
they're
still
building
their
home
charts.
A
The
CI
CD
team
is
controlling
the
process
once
they
contribute
it
to
integration,
so
the
assembly
of
the
entire
end
product
is
being
done
by
the
CI
CD
team,
and
that
was
a
bit
of
a
that
was
a
bit
of
a
challenge
to
get
developers
to
give
up.
Control
of
that
and
I
would
say
that
in
small
teams
that
process,
that
kind
of
you
know
dev
owned,
I'm
gonna
build
this
thing
on
my
laptop.
A
So
CIC
D
owns
the
combined
process
and
you
know
I
put
a
little
picture
of
a
train
on
here,
because
we
really
treat
it
like
a
release
train
right.
So,
with
this,
the
promotional
stages
and
the
quality
gates
and
all
the
testing
and
automation
that
we're
doing
it
is
a
release
train
you
get
on
the
train.
If
you
want
to
get
released
in,
if
you're,
not
on
the
train,
you're,
not
getting
release
right.
A
So
the
hardest
part
about
changing
culture
is
probably
going
to
be
spending
time
right.
You
got
to
spend
time
to
educate.
Why
you're
doing
things?
Why
are
you
changing
the
process?
Because
the
developer
is
gonna
fight
against
any
sort
of
process,
change
and
managers
and
release
managers
will
also
fight
against
the
process
change,
so
you've
got
to
spend
the
time
to
educate
people
to
let
them
know
why
you're
changing
it.
A
What
the
process
is
how
it
works,
how
it
will
flow
from
end
to
end,
not
just
how
but
it,
but
also
why
what
are
the
improvements
right,
so
the
promotion
pipeline
that
we
put
in
place,
you
know
I
just
spent
a
lot
of
time
explaining
to
people
why
it
was
necessary.
What
exactly
it
did
you
know
what
were
the
quality
gates,
what
were
the
criteria
for
moving
from
one
stage
and
that
promotion
pipeline
to
the
other
right?
A
So
that
was
a
lot
of
time
and
it
takes
a
long
time
when
your
team
spread
across
you
know
five
different
time
zones.
It
takes
you've,
got
a
spin
stay
up
really
late
at
night
to
have
meetings
so
people
in
China.
You
know
when
you're
explaining
why
why
this
is
working
the
way
you
wanted
to
work,
so
you
need
to
make
sure
everybody
understands
the
process.
A
This
is
very
important
right,
making
sure
everybody
understands
the
process,
because
just
knowing
the
how
without
the,
why
is
gonna
cause
you
a
lot
of
problems
later
on,
especially
when
you
get
close
to
a
release
and
that
process
tends
to
speed
up
as
we
get
closer
to
release,
and
then
you
start
getting
tons
of
questions
about.
Why
so
make
sure
everybody
knows
why
trust
the
process?
It's
gonna
be
tough.
You
know,
if
you
implement
something
like
this
and
you're
trying
to
change
a
culture,
it
will
be
tough.
It's
gonna
challenge
you.
A
So
I
just
want
to
take
one
second
to
call
out
Andrew
McNamara,
who
is
leading
the
the
CIC
squad,
he's
the
he's
the
squad
lead
for
CI
CD.
He
was
originally
supposed
to
be
here
to
give
this
talk,
and
his
wife
is
pregnant,
with
twins
due
any
day
now,
so
he
unfortunately
couldn't
come,
but
I
wanted
to
call
him
out
and
give
him
credit
for
you
know
the
great
work
that
he's
been
doing
with
the
squad
and
helping
me
out
with
this.
So
with
that
any
particular
questions.
Oh.