►
Description
Bridge the Gap Between Jenkins and Kubernetes - Bartek Antoniak, VirtusLab
Speakers: Bartek Antoniak
This session focuses on important aspects of running Jenkins in a cloud-native environment like Kubernetes with the full advantage of public cloud capabilities like backups, observability and security. Bartek will show how to free up engineering teams from typical infrastructure activities and lets them focus on the business value by using Jenkins Operator project. With declarative configuration and full lifecycle management based on Operator Framework, this can become the de facto standard for running Jenkins on top of Kubernetes.
For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/
A
Welcome
everyone
thanks
for
joining
today,
we'll
be
talking
about
bridging
the
gap
between
jenkins
and
kubernetes,
we'll
try
to
understand
what
makes
jenkins
suitable
in
a
cloud-native
environment
like
kubernetes.
I
would
like
to
tell
you
more
what
kind
of
additional
plugins
components
configuration
is
required
to
run
jenkins
effectively
in
kubernetes
environment.
A
A
This
presentation
is
mostly
based
on
our
own
experiences
when
building
janky's
kubernetes
operator
for
over
two
years
now,
and
that's
pretty
much
it.
Let's
start
a
few
words
about
me:
I'm
bartek
antoniak.
I
work
at
virtuslab
as
cloud
engineering
manager,
I'm
mainly
responsible
for
internal
product
initiatives.
A
Partnerships
open
source
projects
as
well
as
I
help
customers
with
cloud
transformation
or
digital
transformation
processes,
feel
free
to
find
me
on
twitter
and
again,
please
engage
into
conversation
cool,
let's
bridge
the
gap
as
mentioned
today.
I
would
like
to
minimize
the
gap
between
tankies
and
kubernetes.
A
A
But
before
we
get
started,
it's
really
important
to
have
a
common
understanding
of
the
definition
of
cloud
native
environment
depending
on
number
of
people.
You're
gonna
ask
about
definition
of
cloud
native,
I'm
pretty
sure
you
will
have
slightly
different
answers.
That's
why?
Let's
have
a
look
at
cloud
native
computing
foundation,
definition
and
use
this
as
a
good
starting
point
for
the
rest
of
presentation,
cloud
native
technologies,
empower
organizations
to
build
and
run
scalable
applications,
and
these
applications
are
running
in
modern,
dynamic
environments
in
public
cloud.
A
However,
for
me
personally,
the
key
aspects
of
cloud
native
ecosystem
are
a
containers
service
mesh
which
helps
with
inter-service
communication,
especially
with
a
large
number
of
loosely
coupled
microservices
and
immutable
infrastructure,
together
with
declarative,
api
or
declarative
configuration
as
code,
which
helps
a
lot
when
it
comes
to
automation
of
some
processes
and
these
techniques
enable
loosely
coupled
services
to
be.
Basically,
it
enables
the
dev
teams
to
build
and
deploy
resilient
software,
which
is
easy
to
manage
and
and
fully
observable,
and,
of
course,
the
process
is
end-to-end
automated.
A
We
can
deploy
changes
frequently
using
predictable
workflows
with
minimal
operational
toil,
so
that's
the
definition
of
cloud
native
environment
and
now,
if
we
look
at
the
old
model
of
running
jenkins
versus
the
new
cloud
native
model,
according
to
the
previous
definition,
we
have
some
key
elements
here.
Like
empower.
We
want
to
free
up
our
engineering
teams
from
operational
work
and
let
them
focus
on
something
more
relevant
than
constant
troubleshooting,
deploying
new
jenkins
instances
every
day,
then
scalable
and
dynamic.
A
A
Speaking
of
resiliency,
eventually,
we
need
to
change
our
like
ways
of
working
from
manual
troubleshooting
to
kubernetes
self-healing
mechanism.
So
we
want
to
write
software.
We
want
to
write
automation
which
is
faulted
all
around
and
take
an
advantage
of
self-healing
capabilities
provided
by
the
underlying
platform,
which
is
kubernetes
in
our
case
manageable,
from
manual
deployment
to
configuration
as
code.
We
want
to
have
one
single
source
of
true
when
it
comes
to
configuration.
A
All
changes
needs
to
be
driven
by
code.
A
version
in
git.
This
basically
helps
to
maintain
predictable
workflows,
have
a
good
discipline
and
scale
our
teams
to
basically
managed
more
jenkins
instances
or
any
other
services.
As
part
of
our
infrastructure
observable
from
logs
to
events
and
traces,
we
want
to
understand
how
the
system
behaves.
We
want
to
know,
what's
happening,
what
went
wrong?
What
was
the
problem
before?
A
A
Then
it's
also
really
important
subject
to
understand:
what's
the
definition
of
cloud
native
readiness
in
terms
of
jenkins,
so
what
makes
rankings
cloud
native
ready
in
our
opinion,
so
declarative
configuration
is
called.
That's
the
probably
one
of
the
most
important
elements
here,
because
we
want
to
automate
as
much
as
possible.
We
want
to
scale
our
teams,
we
want
to
handle
and
manage
more
services
in
production
without
good
level
of
automation.
That's
almost
impossible
security.
By
default.
A
A
We
want
to
automate
process
of
deployment
upgrades
disaster
recovery-
that's
something
we
usually
do
manually
these
days
with
the
technology
on
the
market
can
be
quite
easily
automated,
which
I'll
be
talking
about
on
the
next
slides,
authentication
and
authorization.
That's
a
really
important
element
of
as
part
of
security.
Also,
we
need
to
have
a
fine-grained
access
control
mode
within
our
organization
and
also
we
want
to
have
the
same
mode,
authentication
and
authorization
model
for
all
services
as
part
of
the
infrastructure.
A
A
Before
we
jump
into
all
these
elements,
I
would
like
to
discuss
the
core
mechanism
of
extending
jenkins
or
configuring
junkies.
As
you
already
know,
jenkins
is
really
modular
and
entire
configuration
and
extension
is
based
on
plug-in
ecosystem,
which
is
really
rich.
You
can
find
more
than
1500s
plug-in
on
the
market.
A
A
With
the
actual
programming
language,
we
can
configure
low-level
config
low-level
properties
of
10
keys
and
directly
execute
groovy
code,
or
also
we
can
invoke
some
some
methods
remotely
if
you
really
need
to.
But
this
is
not
recommended,
and
there
is
a
really
interesting
mechanism
called
init
scripts.
So
all
groovy
scripts,
which
are
located
in
jenkins
home
in
it
grovid,
are
executed
automatically
during
the
jenkins
startup
process
in
alpha
numerical
order.
So
that's
the
core
mechanism
of
initial
configuration
of
jp's.
A
On
top
of
that
we
have
configuration,
has
called
plugin,
which
brings
additional
abstraction
layer
on
top
of
groovy
scripts.
That's
a
yaml
based
configuration,
so
it's
human
readable,
slightly
easier,
easier
compared
to
groovy
scripts.
You
have
a
lot
of
predefined
tasks,
so
go
to
solution
should
be
configuration
as
code
with
less
overhead.
A
So
if
we
look
deeper
at
this
diagram,
it
shows
an
overview
of
entire
pipeline
as
called
workflow.
On
the
left
hand
side
we
have
the
definition
of
of
pipeline
with
the
actual
pipeline
below,
with
some
automation
logic.
Everything
is
versioned.
In
git
repository,
it's
often
called
seed
jobs
repository
acting
as
a
single
source
of
true
when
it
comes
to
jobs
when
it
comes
to
configuration
files,
so
we
have
everything
in
a
single
place
and
jenkins
using
dsl
job
plugins,
automatically
picks
up
these
files
from
git
repository
and
configure
jobs.
A
A
Moving
on
to
full
lifecycle
management
in
terms
of
jenkins,
it
usually
involves
initial
configuration
installing
upgrading
rankings
installing
upgrading
plugins,
which
is
quite
problematic,
sometimes
especially
in
terms
of
dependency,
how
managing
different
versions.
So
it's
always
good
to
have
a
kind
of
automation
here,
of
course,
monitoring-
and
this
leads
us
to
constant
reconciliation
process.
A
What
does
it
mean
thanks
to
operator
framework
which
provides
such
a
capability
to
automatically
reconcile
the
state,
the
actual
state
of
gen
keys,
which
is
the
snapshot
of
of
the
service,
including
configuration
running
jobs,
everything
which
has
been
configured
and,
on
the
other
hand,
we
have
desired
state
desired
state,
that's
a
receipt.
Basically,
we
provided
and
operator
will
make
sure
the
actual
state
reflects
desired
state
constantly.
A
A
A
I
would
say
it's
good
enough
to
to
to
to
provide
a
really
really
good
value
for
the
team
and
reduce
number
of
operational
activities,
and
it's
called
full
life
cycle,
including
application,
lifecycle,
storage,
life
cycle
backups
and
some
disaster
recovery
mechanism
as
well.
Of
course,
if
you
want
to
go
beyond
that,
that's
also
possible.
It
adds
a
lot
of
complexity
and
but
I
I
think
it's
worth
the
effort,
especially
if
you're
developing
an
operator.
A
You
should
aim
for
autopilot
at
the
end,
but
let's
go
back
to
the
level
number
four,
which
is
deep
insights,
so
this
usually
involves
metrics
and
other
alerting
mechanisms
together
with
some
analytical
flow
and
the
end
goal
is
autopilot,
with
horizontal
and
vertical
auto
scaling
some
automatic
tuning
or
scheduling,
depending
on
capacity
of
your
underlying
platform
and
below.
We
have
three
different
technologies
to
implement
operator
helm.
Super
simple,
I
would
say
it's
good
enough
for
a
basic,
install,
maybe
seamless
upgrades,
but
definitely
not
for
life
cycle
management.
A
Unsibo
is
more
like
a
configuration
management
tool
and
then
finally,
golang
is
actual
programming
language.
So
you
have
a
really
good
flexibility
here.
You
can
implement
any
business
flow
you
want.
You
can
use
some
external
libraries,
sdks
clients,
you
can
implement
end-to-end
tests,
so
countless
number
of
possibilities
out
there
and
that
should
be
go
to
position.
A
As
part
of
your
infrastructure,
I
would
recommend
looking
at
github
authentication
plugin,
which
is
based
on
off.
We
can
maintain
everything
in
a
single
place,
including
configuration
as
code
including
pipeline
source
code,
as
well
as
authentication
configuration
and
also
some
other.
Some
2.0
identity
providers
are
also
supported.
A
Security
is
one
of
the
most
important
aspects
in
jenkins,
especially
jenkins.
Has
its
own
history
mainly
related
to
plug-in
ecosystem,
a
lot
of
vulnerable
abilities,
a
lot
of
deprecated
protocols?
So
it's
really
important
to
take
a
look
at
jenkins
recommendation,
which
is
quite
detailed
and
follow
the
steps
and
make
sure
we
address
all
these
problems.
One
by
one
additionally
configure
kubernetes
airbag
model,
make
sure
you
have
good
namespace
level
isolation.
A
You
use
specific
set
of
kubernetes
apis
instead
of
giving
one
big
cluster.
All
it's
especially
important
if
your
kubernetes
is
a
multi-tenant
environment,
if
you
have
more
than
one
teams
maintaining
their
own
workloads,
their
own
ci
cd
services,
you
need
to
provide
a
kind
of
isolation
mechanism
and
and
secure
jenkins
by
default.
A
In
terms
of
backup
and
external
storage,
jenkins
state
should
be,
and
for
all,
basically,
what
does
it
mean
that
every
time
we
change
something
manually
using
web
ui
using
crest
api,
it
will
be
overwritten
by
by
operator
by
automation
during
the
restart
process.
So
we
should
have
a
good
discipline
of
configuring,
everything
from
code
versioning
in
git
as
a
single
source
of
true
and
not
doing
any
manual
changes.
A
Otherwise,
we'll
end
up
with
a
big
snowflake
instance
of
jenkins,
which
is
really
hard
to
recover,
and
we
can
barely
touch
it
in
terms
of
backups.
We
want
to
backup
jobs
and
build
artifacts
only.
We
don't
want
to
backup
these
heavy
xml
files
because
they
are
already
part
of
configuration
as
cold
and
why
jobs
and
artifacts,
because
we
want
to
have
visibility,
what
happened?
A
A
In
terms
of
restore,
we
highly
recommend
to
run
restore
on
the
startup
process,
which
is
one
of
the
very
first
activities
before
jenkins
is
fully
operational.
It
can
be
done
either
using
init
scripts
are
written
in
groovy
or
using
kubernetes
sidecar
pattern.
I
think
sidecurve
pattern
is
better
option
because
you
can
separate
the
life
cycle
of
jankies
itself
with
with
the
backup
restore
flow.
So
we
have
these
two
processes
running
in
parallel
in
terms
of
logs
events
and
observability
in
general.
A
A
A
A
This
is
something
which
we
were
working
on
for
over
two
years
now
and
then
we
want
to
solve
a
problem
of
running
junkies
in
kubernetes
in
a
really
nice
and
automated
fashion.
So
janki's
operator
takes
care
of
life
cycle
management
of
jenkins.
It
also
provides
some
additional
configuration.
Options,
like
pipelines,
are
called
like
everything
what
we
discussed
so
far.
It's
part
of
the
junkies
operator
itself
so
definitely
worth
to
take
a
look
and
give
a
try
and
that's
the
final
snapshot
of
everything.
A
A
A
A
So
jenkins
is
running
back
upside
car
container,
which
uses
external
storage
for
storing
backups,
restoring
them
when
drinking
starts,
and
also
there
is
a
persistent
volume
for
some
resistance
state
if
needed,
service
on
top
of
that
with
type
internal.
This
is
intentional
because
you
shouldn't
expose
jenkins
over
the
public
internet.
A
We've
heard
the
stories
about
script
console
exposed
with
admin
rights.
That's
a
huge
security
thread
for
your
entire
infrastructure,
so
we
use
dmz,
either
bastion
or
vpn
or
any
other
tunneling
technology,
depending
of
your
network
architecture
in
terms
of
configuration
both
for
authentication
authorization
and
pipelines,
use
git,
that's
a
the
very
nice
platform
and
and
kubernetes
plug-in
takes
care
of
ad-hoc
provisioning
of
jenkins
jobs.
It's
worth
to
take
a
look
at
the
agent
on
top,
which
is
a
kind
of
specific
agent
called
seed
job
agent.
A
This
agent
is
responsible
for
running
any
system
processes
like
executing
groovy
scripts
configurations
called
some
other
configure
installing
plugins.
We
want
to
separate
this
too
and
make
sure
we
have
a
dedicated
sandbox
environment
to
run
this
sensitive
operations,
and
we
have
more
user
defined
agents
depending
on
your
configuration
in
git.
A
A
A
I
want
to
leave
you
with
the
following
conclusion
that,
at
the
end
of
the
day,
we
should
free
up
our
engineering
teams
whatever.
That
means
we
should
solve
business
problems
and
do
it
with
more
efficient
engineering.
So
if
we
can't
so
if
we
can
develop
a
software
which
will
automate
some
processes,
especially
within
a
dev
tooling
space,
that
can
bring
a
significant
improvement
and
and
significant
reduction
reduction
of
operational
costs
for
your
organization
and
that's
pretty
much
all
please
connect
on
linkedin
twitter
engage
into
conversation
provide
feedback.