►
Description
Kubernetes Community Days Bengaluru'21
In order to run Kubernetes and run applications in a cloud provider, operators must manage a fleet of cloud resources in addition to the cluster. These cloud resources include VPC, S3 Buckets, Security Groups, Private Links, NACL's, and VPC Peering and must be managed seamlessly to maximize the teams' productivity and provide visibility for troubleshooting, auditability, and security. This talk will walk through successfully implementing a GitOps model to manage the lifecycle of cloud resources using ArgoCD. You will learn how to integrate a custom configurator other than kustomize/Helm into ArgoCD and use custom Kubernetes controllers to manage cloud resources. We will also describe how using a GitOps model improved productivity, reducing the time taken to create cluster from one cluster per week to 50 clusters per hour while also providing visibility into resource creation, audibility, and security.
A
A
If
you
signed
up
for
this
session,
you
may
already
be
familiar
with
what
git
ops
is,
but
for
those
of
you
who
aren't
git
ops
is
a
process
that
uses
a
git
repository
to
store
the
desired
state
of
an
operational
environment
and
a
git
ops
operator
to
perform
the
operations
needed
to
make
the
actual
operational
state
match
the
desired
state.
It's
really
a
mechanism
for
doing
devops.
A
Instead
of
making
changes
directly
to
the
system
using
a
ui
or
cli,
changes
are
made
to
those
configuration
files
representing
the
desired
state
in
git.
The
gitops
process
would
start
with
someone
on
the
team
creating
a
branch
and
submitting
a
pull
request
containing
some
changes
to
the
desired
state
of
the
infrastructure.
A
Those
changes
can
then
be
reviewed
and
approved
using
standard
revision
control
processes.
Once
changes
have
been
approved
and
merged
to
the
main
branch.
A
git
ops
operator,
such
as
argo
cd,
is
responsible
for
changing
the
system's
current
state
into
the
desired
state
based
on
the
configuration
stored
in
git
in
an
ideal
implementation
of
git
ops
manual,
changes
to
the
system
are
not
permitted
and
all
changes
to
the
configuration
must
be
made
through
files
stored
in
git.
A
This
enables
repeatable
deployments
and
prevents
runtime
issues
caused
by
configuration,
drift
or
missing
dependencies
with
gitops
development
teams
are
empowered
with
self-service
deployments,
instead
of
being
dependent
on
an
external
operations
team
and
once
the
devops
engineer
can
treat
infrastructure
as
code.
The
logical
next
step
is
to
perform
code
reviews
on
the
infrastructure
changes
before
deployment
to
make
sure
they
are
correct.
A
A
A
Observability
is
the
ability
to
examine
and
describe
a
system's
current
running
state
and
be
alerted
when
unexpected
conditions
occur.
With
gitops,
you
can
easily
compare
the
desired
state
of
the
system
stored
in
git
with
the
actual
running
state
since
with
gitops
all
changes
are
revision,
controlled,
audibility
compliance,
disaster,
recovery
and
rollback
are
all
made
much
easier.
A
A
A
So
this-
and
this
is
the
way-
a
typical
kubernetes
controller
works.
Now,
however,
there's
nothing
to
stop
a
controller
from
making
external
api
calls
outside
the
cluster.
So,
for
example,
we
could
have
a
controller
that
makes
sdk
calls
to
aws
or
gcp.
So
this
is
really
where
we're
talking
about
managing
external
cloud
resources.
This
is
the
mechanism
to
do
that
within
kubernetes.
A
A
So
here's
what
an
example
cr
would
look
like
for
a
cloud
resource
deployment
in
this
case
we're
creating
an
s3
bucket,
the
this
first
section
in
the
spec,
the
assume
role
provider.
This
is
used
by
the
controller
to
manage
the
connection
to
the
target,
account
and
assume
the
necessary
role
to
actually
create
the
cloud
formation
stack.
A
We'll
talk
a
little
bit
more
about
that
in
the
next
slide,
then
this
next
part,
these
are
the
parameters.
These
are
the
parameters,
the
region,
the
stack
name,
the
template.
These
are
all
the
normal
things
that
you
would
need
when
calling
the
confirmation
sdk.
So
these
are
things
that
are
actually
passed
into
the
confirmation,
sdk
and
then
you
can
see
the
actual
template
itself
and
this
template
is
fairly
simple,
but
you
can
imagine
you
can
also
have
fairly
complex
templates
as
well.
A
However,
there
is
some
usability
issues
when
you
start
getting
very
big
big
templates
and
also
at
intuit,
we
very
rarely
have
a
custom
template.
That's
used
only
once
we
typically
make
patterns
of
templates
patterns
of
cloud
formation
templates
so
that
we
can
reuse
them
and
version
control
them
and
and
kind
of
keep
them
in
a
central
place.
A
A
A
You
set
up
an
application
in
argo
cd,
that's
watching
that
particular
repository
and
a
particular
subdirectory
in
that
repository,
and
when
it
this,
when
it
just
detects
any
changes,
it
will
call
call
some
tool,
usually
it's
customize
or
helm,
but
in
this
case
we're
calling
it
a
custom
tool
called
manny
and
ravi
will
mention
a
little
bit
more
about
benny
later.
But
this
is
really
our
custom
tool
that
we've
installed
into
argo
cd.
A
That
lets
us
represent
a
custom
project
hierarchy
in
the
deployment
repo
and
it
kind
of
takes
the
place
of
customizer
helm
in
this
kind
of
workflow,
but
whether
it's
a
custom
tool
or
customize
or
helm,
the
you
know,
regardless
of
which
tool
you're
using
the
end
result,
is
a
series
of
yaml
files.
A
series
of
kubernetes
manifests
that
are
then
can
be
applied
to
the
cluster.
A
It
may
contain
information
about
what
cross-account
role
to
use
to
deploy
this
resource,
in
which
case
it
will
use
that
cross-account
role
to
to
create
the
stack
in
the
target
account
the
when
the
stack
is
created.
It's
also
passed
a
service
role
so
that
the
cloudformation
service
would
use
a
specified
service
role.
That
would
then
create
the
actual
objects
in
the
aws
account
and
that
service
role
can
be
targeted
to
the
types
of
of
resources
that
you
want
to.
You
want
to
manage
using
get
ops.
A
A
A
B
B
It's
a
kubernetes
controller
built
with
your
building
the
custom
resource
for
this
controller,
essentially
embeds
the
formation
stack
into
it.
So
whenever
the
custom
discourse
is
done,
it
will
look
for
the
corresponding
state
of
the
stack
in
the
aws
and
then
the
controller
would
determine
based
on
that
state,
whether
to
stop
the
reconciliation
or
three
key
for
the
consolation
of
the
custom
resource.
B
So
we
identified
bunch
of
states
for
conformation
stacks
and
we
group
them
into
two
buckets.
One
is
intermittent
states
and
another
one
is
terminal
state
whenever
the
stack
lands
into
the
intermittent
state.
The
controller
would
update
the
status
as
in
progress
for
the
cr
and
it
would
require
reconcile
if
the
stack
lands
in
one
of
the
terminal
states,
it
could
update
the
status
accordingly
if
it's
a
successor,
failure
and
stop
the
reconciliation.
B
Let's
look
at
a
live
demonstration
for
this,
so
the
custom
resource
for
this
would
look
like
this,
where
the
client
is
cloud
resource
deployment-
and
this
is
the
api
version,
you
can
update
the
fields,
it
would
embed
all
the
confirmation
information
in
this
spec
and
the
template
for
the
aws
information
is
given
here.
It
will
pick
up
the
parameters
and
other
things
that
are
passed
in
the
parameter
section
of
the
c
aspect.
B
B
B
So
what
we
have
done
is
we
put
all
the
common
configurations,
like
account
number
environment
region
and
all
these
things
into
the
config
java
and
the
core
content
for
the
custom
resources.
Like
the
template,
the
cloud
formation
template,
we
have
put
it
into
s3
and
we
have
declared
that
it's
a
type
of
s3
and
and
then
we
have
passed
in
the
parameters
that
this
template
videos.
B
With
this
it
looks
like
receptor
kind
of
configuration
management
right,
but
scepter
would
ideally
create
the
resources
as
well.
What
we
wanted
to
have
is
based
on
this
information.
It
should
merge
all
this
data
and
generate
the
custom
resources
that
the
controller
can
accept.
So
we
created
a
custom
cli
and
we
have
configured
this
custom
cli
in
rpc.
B
The
way
we
can
use
this
custom
client,
argo
cds,
I
mean
this
content
is
same.
All
that
we
need
to
modify
is
the
manifest
section
where
we
update
the
name
of
the
plugin
that
we
want
to
use
to
be
many,
which
is
a
cli
that
we
have
so
we
update
this
plugin
name
to
be
manual
that
way
whenever
the
argo,
cd
repo
server,
which
actually
reads
the
data
from
the
github
repo.
B
We
execute
this
cli
command
and
translate
all
the
configuration
files
that
we
have
here
into
into
custom
resources
that
the
controllers
can
understand.
Now,
if
you
look
into
this
with
that
manifest,
this
is
similar
to
what
we
have
seen
for
rescue
configuration
in
this
earlier
app
right.
So
essentially,
here
we
haven't
used
any
cli
configurator.
B
B
Yeah,
so
they
are
in
progress
now,
as
you
see,
the
there
is
a
progress
bar,
that's
spinning
and
the
cloud
resource
is
in
progress
and
we
can
also
look
at
the
events,
so
it
started
creating
the
chain
sets,
so
we
should
see
the
events
the
stacks
getting
created.
So
these
are
the
stacks
that
we
are
trading
s3
bucket,
michael
in
the
security
group.
So
it's
like
the
security
group
is
already
completed.
B
B
We
can
look
at
all
the
events
that
this
cloud
formation
generates
in
the
argo
cd
events
itself.
We
do
not
have
to
log
into
the
account
and
watch
it
always.
I
was
just
showing
this
so
that
we
see
that
the
resource
is
actually
generated
and
that
we
could
see
the
security
group
actually
getting
created.
So
like
this,
we
can
create
all
the
cloud
resources
that
we
are
interested
in.
B
B
B
So
this
is
where
you
know
we
need
to
make
the
changes.
We
need
to
look
at
the
argo
cd
repo
server
deployment
object,
so
we
download
the
content
of
our
cd
deployment
from
the
open
source
and
we
edit
this
section
where
there
is
this
targo
cd,
repo
server.
What
we
need
to
do
is
we
need
to
introduce
an
init
container
and
this
electric
container
should
download
the
manu
binary
and
then
we
volume
on
this
binary,
industrialistic
location.
We
need
to
use
this
volume
mod
in
the
main
container
of
the
argosy
repo
server.
B
Organization,
this
is
one
part
of
it.
The
other
part
that
we
need
to
configure
is
what
action
it
should
do
whenever
it
treats
the
data
from
the
github
repository.
So
we
need
to
configure
that
in
the
config
map
of
the
author
cd
in
the
config
management
plugin
section,
so
we
need
to
declare
that
whenever
we,
whenever
the
argo
cd
repo
server
reads
the
files
in
the
given
path,
it
has
to
run
this
command
many
build.command
on
all
the
files
that
are
present
there
in
that.
B
In
that
path
of
that
repo,
like
by
default,
argo
city
supports
customize,
so
it
runs
the
customizable
dot
command.
Then
it
merges
all
the
files
that
we
have
declared
in
the
customization
email
and
then
it
generates
a
final
output.
So,
similarly
in
here
it
looks
up
all
the
files
that
we
have
configured
there
and
it
generates
the
custom
resources,
as
we
have
seen
earlier
in
our
classical
uiview
now.
B
B
So
these
two
are
the
changes
that
we
need
to
do
in
order
for
us
one
to
install
a
custom
cli
tool
in
the
argo
cd
to
in
order
for
our
custom
controller.
If
we
have
some
specific
state,
this
is
the
targo
cd
should
understand
and
throw
it
in
the
ui.
We
need
to
configure
using
the
resource.customizations.
B
Now,
let's
look
at
how
this
controller
is
working
underneath
in
the
aws
accounts
right,
so
first,
we
will
have
to
create
a
previous
source
account
and
we
have
to
create
a
humanities
cluster.
This
can
be
grab
space
for
salary
case,
for
whichever
we
choose
to
create,
and
we
need
to
deploy
this
controller
into
that
and
we
need
to
create
an
iem
role.
This
iam
role
will
have
to
have
permissions
to
assume
a
cloudformation
master
role
in
a
target
account.
B
There
can
be
multiple
target
icons,
so
we
need
to
create
the
same,
create
architect
on
where
we
want
to
create
the
database
cloud
resources,
and
this
will
have
permissions
to
create
cloud
formation
stacks
and
pass
a
cloudformation
service
role
into
the
stack,
and
this
service
role
will
have
a
power
user
access
in
the
account
so
that
it
can
create
the
change,
sets
and
execute
the
changes
to
create
the
resources
in
the
in
the
awx
account.
So
this
is
how
we
create
all
the
available
cloud
resources.
B
Now,
after
we
have
this
in
place,
we
have
seen
tremendous
results.
However,
I'll
walk
you
through
the
state
where
we
were
in
before
this
right.
So
when
we
initially
started
creating
clusters
with
the
chaos
based
approach,
we
were
creating
the
clusters.
Almost
a
single
glossary
should
take
two
weeks,
although
the
key
of
space
cluster
creation
as
such
for
the
core
kubernetes
association,
it
was
not
taking
much
time
but
creating
all
the
aws
resources
was
taking
time
and
managing
this
aws
resources
like
whenever
we
want
to
make
changes
to
these
resources
across
multiple
classes.
B
B
We
don't
know
whether
that
is
lining
up
in
a
successful
state
or
not,
so
we
didn't
have
an
easy
way
to
audit
who
is
making
the
change
to
them
and
control
it
and
stuff
like
that.
So
whenever
we
try
to
upgrade
clusters,
we
used
to
see
the
customer
upgrades
to
fail
because
some
of
the
resources
were
in
that
state
or
they
were
deleted
by
mistake
and
stuff
like
that.
So
this
this
was
all
a
big
problem
for
us
in
in
the
earliest
list.
B
We
we
introduced
this
github
space
management
of
aws
resources
using
this
project
called
the
biscuit
ops
with
that
we
were
able
to
create
as
many
as
50
clusters
per
day.
We
are
only
limited
by
ip
availability
so
that
we
don't
consume
all
the
capacity
given
to
us
and
all
the
eks
clusters
that
we
plan
to
create.
We
were
able
to
create
all
of
them
in
two
weeks
and
we
saved
about
15
000
manual
bars
with
this
bulk
transformation.