►
From YouTube: Spinnaker @ Cisco - Scaling To Support Large Diverse A... Anil Anaberumutt & Balaji Sivasubramanian
Description
Spinnaker @ Cisco - Scaling To Support Large Diverse Applications - Anil Anaberumutt, Cisco & Balaji Sivasubramanian, OpsMX
Cisco adopted Spinnaker more than 2 years ago and has onboarded more than 1000+ applications. Cisco IT started off using Spinnaker for deploying primarily container-based applications to on-prem cloud platforms (OpenShift). As Cisco found success in Spinnaker, it is being rolled out to larger engineering teams (10,000+ developers). The Spinnaker team at Cisco has to evolve Spinnaker to support varied development requirements around multiple clouds (AWS, Azure, GCP), infrastructure-as-code (Terraform), helm based deployments, custom scripts as well as deploying to Snowflake (SaaS data warehouse), Teradata, and Informatica (Data Analytics platforms).
In this session, you will learn how Cisco evolved Spinnaker for these various applications at scale as well as best practices and lessons learned.
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
A
My
name
is
balaji
shiva,
I'm
vp
of
product
at
appsomax
appsomex
for
people
who
don't
know
we.
We
provide
intelligent,
continuous
delivery
solution
based
on
spinnaker
cd,
besides
enhancing
spinach,
to
support
various
use
cases.
We
also
provide
intelligence
in
terms
of
approval,
include
intelligent
approval
as
well
as
mission
learning,
based
verification
policy
and
audit
and
other
features.
A
I'm
excited
to
be
part
of
this
cdcon
I've
been
part
of
the
spinnaker
community
for
over
four
years,
and
I've
seen
it
grow
from
from
the
very
beginning.
I'm
excited
to
be
here,
and
today
we
have
with
us
anil
annabermat,
he's
a
technical
system
architect
at
cisco
who's,
a
big
user
of
spinnaker
and
he's
going
to
share
with
us
their
learnings
and
lessons
learned,
vanilla.
Welcome
to
the
talk.
B
Thanks
baloji
and
thanks
for
the
invitation
on
the
cd
con,
I'm
pretty
excited
to
have
this
discussion
with
you,
I'm
working
as
a
technical
systems,
architect
in
cisco,
I'm
part
of
the
dell
cyclops
team.
It's
a
centralized
team
which
provides
continuous
delivery
services
and
the
tooling
services
in
the
cyclops
area
for
across
cisco.
B
A
Absolutely
today
we're
gonna.
We
have
by
the
way
we
have
done
this
talk
with
anil,
myself
or
minnie
spinnaker
summit.
Today
we
thought
we
will
combine
the
the
q
a
with
a
slide,
so
it
helps
the
conversation
so
we're
going
to
talk
initially
about
the
current
state
of
spinnaker
at
cisco
and
then
we'll
talk
about
some
of
the
current
deployment
patterns
that
is
used
and
as
well
as
future
deployment
patterns.
That's
planned
at
cisco
and
finally,
our
section
will
talk
about
some
of
the
requirements
that
anil
has
seen
as
well.
A
A
And
I
think
so,
how
do
you
I
mean
this
is
a
pretty
a
significant
deployment
right
of
spinnaker,
one
of
maybe
the
larger
size
deployments
that
I'm
aware
that
I'm
aware
of
what
is
your?
How
do
you
deploy?
Do
you
deploy
one
instance
of
spinnaker
or
multiple
instance
spinnaker?
How
do
you
deploy
spinnaker.
B
We
have
multiple
spinnaker
instances.
We
had
17
instances
last
year,
which
was
catering
to
multiple
business
areas
like
the
ide
and
the
customer
experience.
But
as
we
are
venturing
spinnaker
into
engineering
stuff,
we
have
couple
of
other
instances,
so
we
are
growing
in
terms
of
managing
spinaker
and
we
have
multiple.
The
spinnaker
installed
multiple
data
centers
to
ensure
we
have
high
availability
as
well
as
making
sure
that
we
have
disaster
recovery
and
resilience
covered.
So
that's
how
we
are
managing
our
spinnaker
instances
in
cisco,
but
we
are
growing
as
we
speak.
A
B
But,
as
you
rightly
said,
because
of
the
rise
of
sas
and
the
cloud
base,
we
are
also
looking
at
how
we
can
take
spinnaker
to
the
cloud
so
that
a
lot
of
a
lot
of
our
clients
who
are
already
on
the
client
on
the
cloud
can
leverage
spinnaker.
Instead
of
rather
coming
to
on-prem
and
going
to
cloud.
We
wanted
to
leverage
the
multi-tenancy
and
the
availability
of
spinnaker
in
the
specific
cloud
regions
and
that's
the
reason
why
we
are
trending
towards
aws
and
other
clouds
as
well.
A
A
So
let's
talk
about
the
the
cloud
deployment
patterns
that
you
that
you're
seeing
you
know
to
walk
us
through
sort
of
the
sort
of
the
main
big
bucket
of
deployment
patterns,
I'm
sure
a
large
company
like
yours,
you
have
a
lot
of
variations
of
patterns,
but
maybe
the
audience
will
be
more
interested
in
sort
of
like
what
are
the
big
things.
Big
areas
of
patterns
you're,
seeing
in
your
at
cisco.
B
Yeah,
I
think
there
are
a
couple
of
a
couple
of
areas,
primary
area
being
the
internal
cloud.
That
is
what
most
of
the
companies
tend
towards
a
lot
of
deployments
happen
on
the
internal
path,
and
we
call
that
a
cae
platform,
which
is
nothing
but
an
open
shift
platform,
and
most
of
our
deployments
are
catering
towards
the
internal
cloud
and
they
all
use
spinnaker
as
a
as
the
deployment
orchestration.
B
But
we
are
not,
but
also
there
are
a
lot
of
sas
products
right.
I
think
we
spoke
about
a
snowflake
last
time
when
we
had
our
discussion,
so
not
only
snowflake,
there's
sales
flows,
there's
or
also
there
are
other
sas
products
that
we
enable
deployments
to.
I
think
that
is
very
related
to
data
data
data
as
a
platform,
but
we
are
also
also
looking
at
clients
who
are
also
doing
application
deployments
to
multi-cloud.
It
could
be
aws,
it
could
be
gcp.
A
B
A
B
A
And,
and
what
are
the
some
of
the
I
mean,
you
know
we'll
keep
it
short,
obviously,
but
any
any
quick
learning
energy
scale
to
support
this
is
the
by
the
way.
This
is
the
the
ones
you
described
the
sas,
not
the
sas
part,
the
the
the
on-prem
kubernetes
clusters
like
openshift
and
the
public
cloud
is
sort
of
the
typical.
A
B
Of
our
challenges
in
terms
of
enabling
the
on-prem
loads
right,
spinnaker
didn't
have
any
of
those
out
of
block
out
of
box
plugins.
So
we
had
to
look
out
for
some
of
the
custom
development
that
we
worked
with
your
team
to
enable
that
right,
so
it
was
not
available
in
the
markets.
You
had
to
develop
some
of
those
custom
stages
to
enable
these
internal
cloud
deployments.
B
I
think
that
was
one
of
the
one
of
the
learning
that
we
had
that
there's
a
myth
in
spinnaker
that
is
not
flexible,
but
it's
not
true.
We
saw
that
our
experience
has
been
that
it's
quite
customizable
yeah.
You
can
have
multiple
workloads
and
you
can
also
tweak
around
in
terms
of
what
exactly
is
your
requirement
and
work
towards
that?
That's
been
the
biggest
learning
for
us.
A
Got
it
I
mean
yeah?
Definitely
I
think
spinnaker
can
be
extended,
but
that
definitely
you
need
the
expertise
right
and
so
I
think,
adopts
a
max.
We
work
with
you
for
towards
your
journey
and
and
but
you
know
it's
great
to
see
that
level
of
success
so
far,
all
right,
cool,
let's,
I
think,
I'm
more
excited
about
what's
sort
of
the
things
you're
looking
in
the
future.
Let's,
let's
talk
about
that
in
the
next
section
there.
A
So
in
terms
of
future
deployment
patterns,
I
think
from
what
you're
telling
me
is
obviously
for
what
I
know
that
this
is
primarily
driven
by
some
of
the
engineering
teams
or
you
are
now
starting
to
support
right.
You
primarily
started
the
cisco
it
usage
now
you're,
starting
to
support
more
diverse
groups
within
cisco
engineering
groups
that
want
to
use
spinnaker
for
their
for
their
use
cases.
So,
let's
talk
about
some
of
the
the
deployment
patterns
that
you're
looking
to
do
in
the
future
or
currently
in
the
process
of
an.
B
Yeah,
as
you
as
I
was
talking
about,
there's
been
a
huge
trend
of
cloud
adoption
right,
it's
just
not
cisco
or
in
specific.
I
think
the
market
trend
is
heading
towards
the
public
cloud
and,
as
we
are
engaging
with
our
engineering
teams
and
the
customer
experience
team,
we
see
that
we
see
that
there's
a
lot
a
lot
of
trend
towards
infrastructure
as
a
code
where
teams
are
leveraging
terraform
to
manage
their
infrastructure
along
with
their
code
till
now.
B
That
was
not
the
case,
but
they
want
to
make
sure
that
the
even
the
infra
is
managed
as
a
code.
I
think
that
is
the
one
of
the
biggest
primary
trends
and
that
we
are
working
towards
and
just
want
to
spend
just
not
the
terror
for
enabling
terror
form,
but
within
the
term
form
itself.
There
are
multiple
use
cases
in
terms
of
how
you
manage
your
kubernetes
clusters,
or
how
do
you
manage
your
ec2
instance
or
any
of
those
things
right?
A
I
think
yeah,
I
think
our
our
terraform
module
that
we
provide
as
part
of
our
spinnaker
is
very
popular.
Some
of
the
other
uses
I've
seen
is
like
people
wanna.
I
think
it's
more
about.
I
think
it's
more
about
towards
self-service
nowadays
right
people,
people
want
to
be
able
to
also
on
demand
infrastructure.
A
One
of
the
use
cases
where
you're
seeing
is
that
people
are,
like
you
know,
building
a
qa
cluster
on
demand
and
then
destroying
the
cluster
after
it's
done
right
in
a
way,
and
it's
also
part
of
the
pipeline,
so
you
have
full
access,
full
auditability
and
full
control
over
who
can
do
what
you
know
because
you're
now
part
of
that
process?
So
that's
definitely
use
case.
I've
seen
you
know
across
other
customers
as
well.
Okay,
what's
next
yeah.
B
Yeah,
I
think
just
adding
on
to
that
is
your
spot
on
in
terms
of
why
we
need
terraform.
Is
there
a
lot
of
engineering
teams?
They
do
want
to
test
their
patch
or
any
of
the
change
requests
that
they
are
doing
and
they
spin
up
their
clusters
and
then
destroy
test
and
destroy
that
it
doesn't
mean
the
primary
motive
in
terms
of
enabling
this
pipeline.
B
The
second
one
has
been
the
helm.
I
think,
once
the
clusters
are
clustered
are
spun
up.
There
are
a
lot
of
teams
that
they
want
to
leverage
how
come
to
home
to
for
their
deployments
into
not
only
production,
but
they
want
to
manage
their
templates
and
other
things
based
on
test
environment
or
the
staging
environment,
or
a
final
practice
run
or
specific,
multiple
production
environments
right.
So
we
see
helm
coming
into
picture
as
one
of
the
biggest
deployment
factions.
A
B
Yeah,
I
think
this
is
a
common
trend
across
the
market
and
I've
seen
that
people
leveraging
that,
because,
even
though
they'll
be
a
lot
of
applications
are
public
cloud.
But
there's
a
lot
of
things
that
we
at
an
enterprise
level.
You
still
see
a
lot
of
applications
which
are
unique
towards
each
unusual
companies
to
be
hosted
on
their
own
custom,
vm
or
vmwares
right.
There
are
a
lot
of
testing
that
happens
on
the
vms
before
the
those
images
or
the
patches
are
being
distributed.
B
So
we
we
kind
of
enable
that
in
the
spinnaker
pipeline,
so
that
there's
the
same
development
life
cycle
for
the
development
team,
irrespective
of
whether
it's
a
cloud
or
a
or
a
on-prem
or
a
sas.
So
we
want
to
make
sure
that
the
experience
remains
the
same
for
the
developer,
even
for
his
own
custom,
vm
and
testing
that
they
do.
B
That's
correct,
that's
correct!
We
have
developed
a
custom
stage
called
bring
your
own
script.
It's
just
not
bringing
your
own
script.
You
can
also
bring
your
own
container
image,
leveraging
that
you
can
execute
kind
of
a
serverless
architecture
where
you
spin
up
your
vms.
You
spin
up
update
your
rpm
files
on
the
vms
or
the
ova
files,
and
then
you
can
host
your
application
onto
that
run
your
tests
and
then
validate
whether
those
specific
images
are
good
to
go
right.
A
Yeah,
absolutely
I
think
now
I
was
thinking
when
you're
talking,
I
mean
we
should
probably
do
a
separate
session
on
this,
because
this
seems
like
a
very
interesting
topic
for
a
lot
of
people.
So
maybe
we
should
we
should
plan
for
another
day
on
that
one.
How
do
we
do
that?
Maybe
share
more
details
around
this
topic.
A
I
think
getting
your
thoughts
around
some
of
the
requirements
you
have
seen
for
your
platform,
as
you
start
scaling
right,
you're
already
at
a
good
scale
right
now
and
and
what
are
some
of
the
challenges
or
issues
or
things
that
you've
seen
and
and
how
you
maybe
have
solved
yourself,
but
also.
I
want
to
really
understand
your
sort
of
industry
perspective
as
well
right.
So,
let's
get
into
it.
I
think,
based
on
number
of
deployments
and
number
of
applications
I
saw
in
the
in
the
first
slide.
A
You
have
a
lot
of
teams
a
lot
of
developers
and
they
want
to
be
able
to
automatically
create
a
pipeline
and
be
able
to
deploy,
define
their
targets
and
be
able
to
I'm
guessing.
They
want
to
be
able
to
deploy
end-to-end
without
any
sort
of
a
ticketing
system
that
takes
10
days
to
provision
something
right.
You
know
so
so
tell
me
about
your
perspective,
of
how
you
think
self-service
requirements
are
in
the
industry
or
the
city
and
sort
of
how
cisco
may
have
implemented.
That.
B
Yeah,
I
think
this
cell
service,
when
you
talk
about
right
at
a
enterprise
level,
I
think
it's
one
of
the
key
stuff
when
your
volume
and
your
number
of
users
are
huge.
You
want
to
bring
that
self-service
more.
Irrespective
of
what
toolsets
you
have.
I
think
one
of
the
things
that
we
have
tracked
is
how
do
we
enable
a
developers
to
onto
the
spinnaker
not
only
spinnaker
the
whole
end-to-end
cicd
pipeline?
B
So
we
have
we
have
we
learned
this
over
a
period
of
time
where
we
used
to
have
our
own
ticketing
system.
As
you
rightly
said,
it
is
to
take
three
days
four
days
to
raise
a
ticket
and
get
them,
but
we
have
built
a
developer
experience
platform
where
a
user
can
go
and
request
for
a
pipeline
right.
The
pi
is
just
not
the
pipeline.
He
will
get
access
to
the
console.
He
can
get
access
to
all
the
service
details
that
he's
going
to
do
not
only
service
detail.
B
He'll
also
get
a
command
cli
where
he
wants
to
automate
his
own
services
right.
So
he
has
that
ability
to
do
that
and
the
last,
but
not
the
least.
He
will
also
have
the
ability
to
see
the
analytics,
as
well
as
a
part
of
his
stack
request.
So
this
is
one
of
the
common
trends.
I
think
people
in
in
the
in
the
market.
I've
seen
people
struggle
a
bit
they
try
to
manage
manually,
but
when
you
have
a
huge
volume,
you
need
to
have
a
self-service.
B
A
A
You
know
the
the
way
I
see
it
is
if
you
really
want
to
achieve
like
a
cd
at
a
scale.
You
need
to
do
this
faster
because
you
don't
you
can't
take
hours
to
you,
know
analyze,
something
and
also
triage
any
issues.
You
know
you
can't
take
hours
that
essentially
you
stop
the
pipeline
so
to
say
and
stop
the
deployment,
but
you
know
I
think
the
ability
to
do
that
at
scale
is
a
big
challenge.
B
I
think
the
automated
safe
deployment
verification-
I
think
it's
this
is
the
trend,
that's
in
the
market
right
and
everybody
wants
to
be
there.
We
are
in
within
cisco.
There
are
pockets
of
teams
who
are
matured
in
the
in
the
cloud
or
the
cloud
native
world
of
deployments.
They
understand
the
canary,
they
understand
the
building
deployments,
there
are
pockets
of
teams
having
leveraging
these
techniques,
but
it's
still
in
the
desired
state.
From
a
cisco
perspective,
we
are
trying
to
explore
to
see
how
exactly
we
can
scale
into
the
whole
enterprise
versions.
B
A
Got
it
yeah,
I
remember
another
customer
has
about
thousand
services
and
they
have
enabled
it
for
10,
which
is
100
services.
It's
not
bad,
but
still
definitely
you
know
from
an
adoption.
It's
still,
I
think
it's
it's
an
aspirational
goal
in
many
cases
I
think,
but
it
is
something
I
think
that
if
you
really
want
to
scale
your
cd
deployments
and
how
netflix
is
doing,
you
know
thousands
of
deployments
a
day,
they're
able
to
do
it
with
a
with
a
verification
on
automated
right
and
that's
the
only
way
to
do
it.
A
So
it's
great-
and
this
is
another
topic-
that's
around
the
similar
topic
around.
How
do
you
do
it
at
scale,
which
is
policy,
automation
right?
You
have
all
these
developers
creating
all
these
pipelines,
and
you
want
to
make
sure
that
there
are
multiple
policies,
enterprise
policies,
compliance
policy,
a
security
policy.
All
of
them
need
to
be
enforced.
So
maybe
you
could
share
your
thoughts
around
what
you
think
and
also
how
cisco
is
looking
into
it.
B
B
So
as
a
as
as
I
showed
you
this,
we
as
a
part
of
the
self-service
environment,
we
ensure
that
the
applications
are
associated
with
socks
or
pcids's
make
sure
that
those
specific
policies
are
met
from
the
get
go
of
our
back
room
based
access
control.
What
control
should
a
developer
have?
What
control
should
a
qa
should
have
a
manager
should
have
or
a
tester
should
have
right.
B
So
we
ensure
that
those
policies
are
met
from
the
get
go
of
a
pipeline,
but
we
also
ensure
that
there
are
a
lot
of
socks,
validation
specifically
on
the
finance
and
hr
applications
which
are
going
through.
The
production
changes
make
sure
that
those
are
met
within
the
final
practice
run
or
the
long-term
environments
before
even
they
go
to
the
production.
So
we
ensure
this
all.
Everything
is
automatic,
there's
no
manual
intervention
in
terms
of
policy
policy,
audit
or
policy
validation.
Everything
is
automated.
B
B
A
A
Yeah
the
way
the
wave
yeah
the
way
we've
built
it
is
through
the
oppa
policy
framework
right.
So
that
means
that
you
can
write
regular
policies
right,
regular
language
policies,
so
you
can
customize
it,
and
that's
really,
I
think,
what
you've
seen
interaction
with
developers.
You
know
want
to
write
and
and
change
their
you
know
maintain.
Obviously
you
want
to
you
know,
version
control.
You
know
you
want
to
approve
process
for
those
policies.
You
know
you're
not
going
to.
A
A
Yeah
exactly
no,
it
is
it
is.
It
is
actually
it's
good
and
actually
we
maintain
and
get-
and
you
know
you
version
control
it
right,
but
but
definitely
very,
very,
very
flexible
in
that
sense
right,
it's
not!
You
don't
have
to
go
through.
You
know
six
months
of
requirement
or
change
change
requirements.
It's
more
like
okay,
you
change
it.
Somebody
can
improve
where
you're
good,
to
go
on
that
policy
all
right.
So,
let's
maybe
before
we
wrap
it
up,
talk
about
a
couple
of
topics
which
is,
I
think,
very
interesting.
A
One
is
on
analytics,
which
is
basically
is
knowing
what's
happening
right
around
the
code
quality
or
release
what's
happening,
how
many
releases
are
happening?
Obviously
different
stakeholders
are
there,
so
you
you,
you
have
to
have
visibility
and
and
and
trackability
of
these
things
and
also
security
right,
and
so
let's
talk
about
like
what
is
your
thoughts
on
that
and
and
what
cisco
is
doing
specifically
in
this.
In
this
case,.
B
Yeah
analytics
has
been
the
talk
of
the
town
in
the
devops
space
right,
but
I
think
we
are
high
on
analytics
specifically
on
the
quality
when
I
say
quality
it's
at
all
levels
at
the
core
level.
It's
like
the
service
level,
it's
at
the
application
level,
even
at
the
organizational
level
right.
So
we
want
to
make
sure
that
we
control
some
of
these
qualities
and
attract
them
right
at
each
levels,
and
we
have
our
own
analytics
system
which
ensures
that
we
have
quality
analytics
track
at
the
core
level.
B
We
also
track
the
analytics
at
the
security
level.
We
want
to
make
sure
the
dash
security
is
met.
The
sas
made
security
is
nothing
but
the
source
code,
and
that's
this
is
nothing
but
the
dynamic
application
security
right.
So
we
want
to
make
sure
that
there
are
no
vulnerabilities
that
is
passed
on
to
any
of
the
production
environments,
because
it's
an
enterprise
portion.
We
are
not
only
that
we
also
look
at
the
release
analytics.
How
is
the
release
trending?
What
is
the
prediction
of
the
releases?
How
many
changes
are
going?
B
A
Got
it
and-
and
I
think
this
is
this-
is
absolutely
you
know
it's
surprisingly-
it's
not
surprising
really,
but
the
the
value
of
analytics
is
pretty
high
because
a
lot
of
people,
you
know
you
know
once
they've
sort
of
gotten
past
their
sort
of
day-to-day
like
okay.
How
do
I
automate
a
pipeline,
and
this
seems
to
be
the
most
interesting
for
a
lot
of
users,
because
they
then
get
a
higher
level
view
of
what's
happening
right,
and
so
we
have
at
lobster
max.
A
B
It's
not
just
the
application
deployment
audit
itself,
but
we
ourselves
as
a
platform
are
also
tend
to
be
audited
because
we
are,
since
we
are
handling
the
deployments
to
the
financial
hr,
and
the
data
flowing
through
spinnaker
pipelines
are
quite
intensive
in
terms
of
this
specific
right,
so
we
ours,
our
application
itself,
gets
audited
through
a
socks
or
any
of
those
six
sigma
and
other
things
right.
So
we
have
quarterly
six
months
yearly
audit
that
happens.
There
are
a
lot
of
audits
which
take
samples
of
deployments
applications.
B
B
We
are
currently
working
towards
that
we
are
still
not
at
the
desired
state.
We
tend
to
trace
from
a
release
back
to
the
source
code
and
that's
that's
our
goal.
How
do
we
say
that
a
change
that
happened
in
the
production
if
it
fails?
How
do
we
trace
back
to
the
code
or
how
do
we
trace
back
to
the
service
or
how
do
you
trace
back
to
the
specific
application
right?
A
Yeah,
absolutely,
I
think
at
ops
we
are,
we
are
absolutely
focused
on
providing
100
audit
and-
and
we
just
released
a
traceability
feature
in
the
latest
releases-
just
came
out
last
week,
because,
basically
the
steps
are
cops
right.
They
really
want
to
know,
what's
happened
right
and
they
want
to
know
how
to
trace
back
vulnerabilities
like
solar
fin,
solar,
solar,
winds,
kind
of
a
bugs,
etc.
So
that's
a
very,
very,
very
interesting
use
cases
now.
B
A
B
Make
sure
that
metrics
is
captured
at
the
code
level
at
the
bill
level
at
the
binary
level
at
the
deployment
orchestration
level
at
the
release
level,
and
not
only
we
don't
stop
there.
We
also
try
to
paint
to
pull
all
the
platform
unlocks
using
punk
and
other
stuff,
but
we
are
not
able
to
still
hook
it
up,
but
we
are
trending
towards
that
to
make
sure
that
any
production
issue
that
happens,
you
should
be
able
to
trace
back
using
all
this
metal
information.
A
Excellent,
so
thanks
so
much
anil.
I
think
this
was
really
useful.
Hopefully
the
audience
found
it
useful
in
terms
of
knowledge
sharing,
and
I
think
you
know
we
had
a
short
time
to
cover
a
lot
of
material
right
and
if
you
guys
have
any
questions
whatsoever
feel
free
to
reach
out
to
us.
You
have
our
email
addresses
right
there
or
twitter
anyways
reach
out
to
us
and
we'll
be
happy
to
share
more
information
and
walk
us
through
our
lessons
learned
again.
Anil.