►
From YouTube: GitOps Based Cloud Native 5GC Lifecycle Management – Case Study Base... Michal Sewera & Samy Nitsche
Description
GitOps Based Cloud Native 5GC Lifecycle Management – Case Study Based on Mavenir 5GC Implementation in Deutsche Telekom Technik with Das Schiff (T-CaaS @ TDG) Platform - Michal Sewera & Samy Nitsche, Deutsche Telekom
Running a Kubernetes-based application does not mean it's cloud native. We get the real advantages of the cloud native ecosystem only when we implement a fully automated platform life cycle management system based on GitOps, automating implementation of so called “desired state of the network”. This talk will present lessons learned and experiences from the transformation to the GitOps NF lifecycle management model based on a case study of Mavenir 5G SA platform implementation at Deutsche Telekom Technik, focusing on:
- CICD infrastructure and implemented practices
- GitOps based configuration management for 5G applications
- Requirements to support infrastructure rolling upgrades
- Challenges and ideas to support ISSU without a need for a traffic migration
A
Good
morning,
everybody
we
are
a
little
bit
delayed,
but
we
promise
that
we
will
at
least
try
to
compress
our
presentation
as
much
as
possible.
So
it's
really
a
pleasure
to
be
here,
especially
after
all
these
crazy
coveted
times
seeing
the
people
together.
That's
really
wonderful.
It's
really
a
pleasure
to
be
here,
so
my
name
is
michael
severa
and
today,
together
with
semi
nietzsche,
we
will
present
the
stuff
related
with
addressing
the
githubs.
A
So
the
title
is
github's
based
cloud
native
5gc
lifecycle
management.
I
think
that
we
will
have
a
lot
of
insights
coming
from
our
experiences
during
the
deployment
but
before
we
start
just
a
short,
intro
and
and
presentation.
So
once
again
my
name
is
michael
severa.
Actually
I
not
talking
about
the
technology
stuff.
I
play
guitar,
I'm
a
physicist
enthusiast
and
I
also
I'm
also
a
member
of
open
networking
foundation.
A
C
A
Perfect,
very
good,
so,
let's
get
started
actually
the
agenda
for
today
is
a
sort
of
journey,
and
we
would
like
to
take
you
with
this
journey,
focusing
on
how
we
started
and
where
we
are
today
so
we'll
go
quickly
through
an
evolution
of
mobile
core
networks,
then
getting
through
the
5gc
architecture,
which
we
have
and
then
finally
focusing
on
the
transition
and
the
technical
platform
deployment.
So
that's
basically
the
plan.
Of
course
we
will
finish
with
the
q
a
session.
So
whatever
questions
you
have,
it
will
be
a
pleasure
to
discuss
things
with
you.
A
So,
let's
start
with
the
evolution
of
mobile
core.
Actually
I
think
that
we
hear
the
keyword
transformation
every
five
to
ten
years
when
I
started
my
work
at
telecom
industry
like
20
years
ago,
that
was
a
world
of
boxes.
So
basically
everything
was
a
box
and
still
plenty
of
people.
From
that
times
they
are
telling
like
yeah,
that's
the
highest
quality.
That
was
the
very
good
sort
of
telecommunication
right,
so
that
was
a
close
shot
box
standard,
tca
or
atca,
and
basically
that
was
a
completely
built
ecosystem,
including
the
hardware.
A
Now
I
think
that
a
decade
ago,
we
as
an
industry,
started
this
sort
of
transformation
trying
to
get
the
benefits
of
the
virtualization,
so
that
was
the
famous
moment
of
doing
the
virtualization.
Actually,
even
in
our
company
in
deutsche
telekom,
we
started
a
program
called
pun
net,
with
the
ambition
actually
to
to
move
the
telco
platforms
to
this
frame.
But,
honestly
speaking
from
my
perspective,
that
was
really
not
changing
significantly,
because
that
was
any
gap.
A
Still
in
the
world
of
boxes,
we
had
a
box
which
was
pretty
much
still
owned
by
the
supplier.
Okay,
the
technology
was
different,
but
the
processes
behind
it.
That
was
still
pretty
much
the
same.
If
you
will
go
to
the
industry,
checking
the
things
you
will
easily
discover
that
a
lot
of
operators
across
the
world.
They
have
a
box-based
virtual
virtualized
solution,
which
means
from
contractual
perspective,
that's
and
process
perspective.
That's
not
significant
difference
compared
to
to
a
world
of
previous
generation
of
boxes.
A
Now,
right
now
we
are
in
the
front
of
the
huge
opportunity,
and
actually
that
was
a
fantastic
presentation
from
cisco
showing
that
5g
is
actually
enabler
and
we
consider
exactly
the
same.
So
we
believe
that
5g
is
the
first
platform
which
really
gives
the
opportunity
to
take
to
take
the
to
to
do
things
differently
and
that's
the
sort
of
cloud
native
approach
right
now.
A
In
order
to
understand
what
does
it
mean
cloud
native,
I
would
like
to
first
of
all
state
a
following
statement,
which
I
always
try
to
repeat,
is
that
running
application
as
a
container
does
not
equal
does
not
mean
that
the
application
is
cloud
native,
and
I
think
that
is
a
fundamental
statement.
There
is
a
lot
of
claims
in
the
market
that
hey,
we
are
super
cloud
native
and
we
have
a
platform
which
is
cloud
native.
That's
really
not
true.
A
Cloud
native
means
it's
no
longer
a
box
based
solution
and
when
I
say
it's
no
longer
a
box-based
solution,
it
means
that
it's
completely
decomposed
not
only
from
the
technology
perspective,
but
also
from
the
processes
perspective.
So
if
you
will
see
here
on
this
picture,
you
have
a
lot
of
moving
parts
and
those
moving
parts
has
different
have
different
ownerships.
A
So
actually
you
have
images,
hand,
charts
pipelines,
you
have
things
like
kubernetes
network
policies,
you
have
the
things
like
airbag
resources
right
and,
of
course,
if
you
will
go
to
the
bottom,
you
will
see
that
okay,
there
are
also
platform
as
a
service
components
which
are
not
coming
from
a
supplier.
They
are
generic
right,
so
definition
of
system
is
completely
different.
A
There
is
no
longer
a
a
world
of
box,
we
are
talking
about
heavily
distributed
system
and
each
of
those
components
has
their
own
life
cycle,
which
means-
and
that's
super
important
statement-
is
that
there
is
no
longer
a
concept
like
yeah.
Let's
freeze
everything,
let's
keep
it
do
not
touch
it
for
next
half
a
year.
Let's
do
big
sort
of
testing
and
then,
after
this
big
sort
of
testing,
let's
do
once
upgrade
and
we
are
done
for
one
year.
No,
actually,
we
need
to
enable
the
process
to
allow
individual
changes
of
individual
platforms.
A
So
if
you
will
travel
with
time,
I
love
the
the
the
movie
back
to
the
future.
If
you
will
do
this
travel
in
time
and
speak
with
the
guys
20
years
ago,
that
will
be
a
total
disaster
for
him,
because
it
means
that
potentially
every
week
every
month,
one
of
those
components
will
be
changed.
So,
for
example,
you
have
a
security
vulnerability
and
you
need
to
address
it
on
kubernetes
layer.
What
does
it
mean?
A
So
I
was
mentioning
this
movie
back
to
the
future
and
I
really
like
it
because
I
think
that
essential
part
of
the
cloud
native
ecosystem
is
to
have
a
sort
of
time.
Machine
and
sami
will
explain
the
technical
details
of
this
later,
but
I
think
this
analogy
is
really
very
well,
so
you
should
have
the
ability
to
put
your
time
in
back
to
time
whenever
you
have
an
issue,
so,
for
example,
you
are
doing
an
upgrade
or
one
comp,
one
of
the
components
is
being
upgraded
and
then
you're,
saying.
Okay,
we
have
a
problem
right.
A
You
need
to
be
able
to
automatically
revert
all
the
changes,
not
only
in
terms
of
configuration
in
terms
of
everything
images
deployments
things
like
this,
so
I
think
that
this
one
is
showing
this
sort
of
challenge
now
to
go
to
the
5gc
architecture
itself.
I
was
just
showing
the
framework
now,
if
you,
if
we
think
about
what
does
it
mean
in
himself
in
terms
of
complexity,
we
will
need
to
multiply
this
challenge
by
each
of
these
platforms.
A
So
I
think
that
the
colleagues
from
swisscom
they
were
they
were
mentioning
the
the
combo
core
base.
Upload
so
converge
core
means
5g
and
4g,
and
that's
of
course,
natural
way
how
you
you
you
need
to
produce
the
service.
You
cannot
just
say
hey,
I
will
just
do
the
5g
part
and
customers
will
be
happy.
Of
course,
it's
not
impossible,
not
not
possible
right.
A
We
need
to
have
a
converged
course,
so
we
need
to
take
the
previously
the
legacy
which
we
really
hate
because
those
legacy
like
lte
that
was
completely
not
prepared
for
a
cloud
native
way
of
producing
the
service,
so
that
increased
the
complexity
even
more
right.
We
have
the
components
like
amf,
which
are
pretty
much
prepared
for
cloud
native,
but
we
need
to
support
the
components
like
mme
and
if
we
go
even
more
actually
we
have
a
lot
of
protocols
which
we
really
don't
like
on
the
right
side.
A
On
the
slide,
you
will
have
an
fantastic
sbi
stack,
which
we
really
love,
because
that's
purely
designed
for
a
cloud
native.
If
we
have
an
http
based
rest
interfaces,
that's
really
fantastic.
You
can
use
kubernetes
ingress
services.
Kubernetes
is
basically
designed
for
it
right,
but
if
you
want
to
produce
the
service
containing
both
5g
and
4g
still,
you
have
interfaces
like,
for
example,
m3,
and
when
we
are
saying
about
the
interface.
A
If
we
are
discussing
the
details
about
the
interface
like
n3,
they
are
really
far
away
for
being
a
kubernetes
sort
of
compliance
right.
The
things
like
gtp
and-
and
things
like
this-
that's
immediately
enabling
the
light
in
our
head,
like
okay,
which
means
multus,
which
probably
means
a
couple
of
other
things.
Maybe
in
the
future,
you
know
doing
things
in
smart
knicks,
but
not
not
today,
right,
so
this
complexity
actually
is
because
we
have
a
lot
of
vegas
legacy
in
our
backpack.
A
A
I
think
that
trying
to
implement
the
cloud
native
with
just
keeping
the
same
sort
of
box
way
approach
would
be
a
disaster.
One
of
the
examples
which
I
can
make
is
that
I
had
a
couple
of
colleagues
and
they
were
always
asking
me,
but
where
is
the
command
line
where
I
can
log
into
a
box?
And
I'm
saying
there
is
no
longer
a
box
until
we
will
not
get
out
from
the
side
sort
of
thinking
like
okay,
I
need
to
upgrade
site
a
site
b
and
I
need
to
move
the
traffic.
A
That's
really
not
cloud
native.
Of
course
there
is
a
journey.
We
will
need
to
go
to
this,
but
I
think
that
the
holy
grail
of
cloud
native
stuff
is
thinking
beyond
the
sides.
It's
thinking
beyond
the
boxes
and
that's
the
essence.
So,
from
our
perspective,
the
idea
to
to
make
it
happen
was
to
merge
two
types
of
personalities
on
left
side.
A
You
have
the
telco
experts.
They
know
details
about
3gpp.
They
know
everything
about
architectures
complex
interfaces,
those
kind
of
things
on
the
right
side.
You
have
typically
young
guys-
and
I
see
it
also
in
this
sort
of
community-
that
we
are
sort
of
mixed
and
giving
a
fantastic
experience
about
the
systems
related
with
cloud
native.
The
question
is
how
to
mix
those
two
worlds
together,
because
in
in
our
view,
doing
it
separately
means
like
let
the
application
guys
lets
the
telco
experts
work
completely
separately.
Out
of
the
guys
from
from
cloud
will
not
work.
A
So
we
took
the
journey
in
a
way
like
creating
a
team
trying
to
merge
the
the
skills
trying
to
put
the
experts
from
both
domains
and
we're
just
really
fundamental.
Let's
do
the
the
clean
approach.
So
in
our
case,
when
we
try
to
build
this
new
solution,
we
had
an
idea
actually
to
to
question
every
single
thing
from
a
legacy.
One
of
my
favorite
examples
is
that
ipv6
we
always
had
this
sort
of
debate
like
okay
is
ipv6
future
or
not.
A
Yes,
of
course,
it
is
so
why
we
are
using
v4
right
in
our
deployment.
We
erased
before
completely
right,
and
if
we
will
go
to
an
examples,
actually,
we
were
really
successful
to
question
the
legacy
things.
There
are
a
bunch
of
legacy
things
which
needs
to
be
still
questionable.
One
of
my
favorite
one
is
multipathing,
which
was
designed
conceptually
like
20
years
ago,
where
the
world
of
boxes
was
now
in
the
cloud
native.
That's
completely
irrelevant.
A
It's
like
trying
to
port
horse
into
a
front
of
a
car
right
because
yeah
we
had
horses
some
years
ago,
and
I
think
that
the
concept
of
questioning
everything
doing
a
clean
design
and
mixing
the
things
is
a
sort
of
was
a
sort
of
fundament
in
in
case
of
our
approach.
So
if
we
will
go
further
actually
comparing
those
two
worlds,
we
believe
that
the
fundamental
thing
is
to
to
be
sure
and
to
be
aware
about
the
differences.
A
So
if
we
think
about
the
5g
cloud
native
5g
ecosystem,
we
believe
that
the
change
is
really
an
asset.
So
in
a
traditional
world-
and
I
know
it
from
from
years
of
experience
before
that-
was
always
the
sort
of
big
day
you
are
preparing
for
an
upgrade
half
a
year.
You
are
doing
a
thousand
regression
tests
with
spirent,
and
then
there
is
a
sort
of
big
day
like
yeah.
Today
we
are
doing
a
pilot
and
we
are
putting
a
new
new
software
whatsoever
right
here.
It's
completely
opposite,
I
mean
we
are
doing
massive
changes.
A
We
are
doing
changes,
sometimes
couple
of
changes
even
per
week,
but
there
are
really
small
ones,
and
I
think
that
the
essence
is
to
have
the
framework
which
really
supports
it.
So,
from
my
perspective,
once
you
have
a
framework
which
you
can
really
trust,
and
it's
not
using
the
sort
of
brain
interface,
so
protein
base
interface,
like
human
beings,
for
every
single
single
work
order
or
change,
you
can
really
trust
it
and
then
once
this
will
be,
this
is
set
up.
A
You
have
a
lot
of
other
benefits
like
you
can
do
small
improvements,
dynamic,
scaling,
recovery
of
system.
What
I
love
in
actually
cloud
native
approach
is
that,
with
this
travel
and
time
you
can
say
like
yeah,
I
want
to
be
again
on
monday
9
a.m,
with
everything
right
and
with
git
ups,
actually
that's
possible
and
imagine
that
you
have
a
system
which
contains
10
sides.
You
just
did
something
and
you
would
like
to
revert
everything
in
30
minutes.
It's
absolutely
possible
in
a
world
of
legacy.
A
Typical
behavior
is
that
okay,
we
need
to
do
redeployment,
which
means
sending
professional
services
on
site
or
doing
things
remotely,
but
the
long
time
right,
and
I
think
that
the
fundamental
difference
is
exactly
that.
You
have
built
in
mechanisms
to
support
this
frequent
changes
and
in
order
to
understand
how
we
did
it,
it
would
be
a
pleasure
to
to
hand
over
to
sami
sami
was
a
significant
contributor
in
the
project.
Actually,
he
forced
couple
of
solutions
which
were
very
proud
of
and
sammy.
The
floor
is
yours.
Thanks
victor.
C
Telecommunication
hardware
and
software
specifically
designed
to
support
very
strict
requirements
like
5
9
availability.
On
the
other
hand,
it
hardware
components
are
not
specifically
designed
for
taco
application.
Even
that
of
high
quality.
Dt's
target
is
to
achieve
at
least
as
good
capacity
performance
and
availability
on
the
cloud
as
it
is
provided
nowadays
on.
C
C
C
So
I
would
like
to
take
the
chance
to
advertise
also
the
cube
contract
from
our
colleagues
on
friday
afternoon.
So
if
you
like
would
like
to
know
more,
especially
on
the
networking
part,
then
please
make
sure
to
attend
this
talk,
and
this
picture
is
actually
from
them
describing
how
they
feel
about
telco.
C
C
On
top
of
that,
the
host
os
runs
with
container
runtime,
followed
by
the
cast
layer
which
includes
kubernetes
as
orchestration
api.
The
path
layer
is
mixed
responsibility
and
we
already
got
challenged
several
times
in
troubleshootings.
Identifying
issues
on
the
border
between
application
and
cars
and
components
like
observability
and
monitoring
are
managed
sas
components
by
the
shift.
On
the
other
hand,
components
like
ingress
or
specific
cni's,
like
already
mentioned
multus,
are
brought
and
managed
by
the
5gc
devops
team
themselves,
and
on
top
of
that,
we
are
running
all
our
5g
cnfs.
C
The
git
repository
is
defined
as
a
desired
state,
whereas
the
kubernetes
cluster
itself
represents
the
actual
or
the
current
state.
The
task
of
the
git
ops
operator
is
to
always
make
sure
that
the
current
state
is
equal
to
the
desired
state.
This
process
is
called
reconciliation
and
it's
a
fundamental
concept:
how
kubernetes
works
internally
as
developers?
C
So
what
do
we
gain
from
this?
With
git?
At
the
center
of
operations?
We
increase
the
deployment
frequency
by,
on
the
same
time
decreasing
the
change
failure
rate
we
reduce
elite
times,
and
we
have
an
increased
productivity
and
maintainability
through
the
usage
of
best
practices
like
infrastructure.
As
code
or,
don't
repeat
yourself,
we
decrease
the
potential
of
human
error
as
we
have
an
improved
data,
quality
accuracy
and
transparency
through
the
removal
of
data
silos.
C
A
A
Honestly,
I
think
that
reconciliation
is
the
heart
of
this
concept.
If
you
have
reconciliation
for
all
the
layers
for
configuration
for
images
for
everything,
then
you
are
very
close
to
a
target
concept
of
cloud
because
it
basically
means
that
sooner
or
later
you
could
start
to
think
I
don't
care
how
many
sites
I
have
5
10.
A
I
have
a
full
automation.
So
if
I
have
a
problem
actually,
I
can
revert
to
a
day
before
with
everything
and
that's
exactly
reconciliation,
so
it
means
there
is
no
longer
a
sort
of
human
being
saying
like
oh,
I
need
to
do
some
manual
work
or
where
is
the
documentation
for
our
lost
site?
Okay,
maybe
and
he's
on
holiday.
Sorry,
he
cannot
do
it
today,
right
with
github's.
Actually,
everything
is
documented.
A
You
can
always
revert
to
a
state
n
minus
one,
and
you
have
you
need
to
you-
need
to
fully
trust
to
it.
So
I
think
that
reconciliation
is
a
key
word
now
in
order
to
think
about
the
sort
of
challenges,
because
you
would
say
like
yeah,
fantastic,
wonderful,
actually,
just
go
for
it
and
yeah.
Everything
is
simple.
A
So,
first
of
all,
issu,
so
in
software
system
upgrade
is
the
sort
of
holy
grail.
What
does
it
mean?
It
means
that
actually
colleagues
from
infra
or
we
can
do
the
rolling
upgrade
of
the
infra
or
the
upgrade
of
the
application,
without
even
thinking
that
hey,
I
need
to
move
the
traffic
from
one
side
to
the
other.
A
That's
the
holy
grail
well,
unfortunately,
not
so
simple,
because
we
still
have
a
lot
of
protocols,
as
you
remember,
from
the
slides
before
which
are
not
cloud
native,
and
it
basically
means
you
still
will
have
a
certain
interruption
of
the
of
the
traffic.
So,
for
example,
if
you
have
sctp
session
on
the
ngap
level,
so
ngap,
which
is
application
level
interface
for
amf,
it
will
survive,
but
sctp
will
have
will
have
a
glitch
right
right.
The
second
part
is
how
to
do
it.
A
Stateless,
so
imagine
that
you
have
millions
of
subscribers
connected
to
your
network
and
while
they
are
using
a
service,
you
are
doing
an
upgrade.
So
you
are
taking
note
by
node
by
node,
which
means
that
whenever
microservices
is
being
drained,
the
new
one
needs
to
fetch
the
last
state
from
the
subscriber,
and
it
needs
to
be
smooth
for
the
customer.
A
Of
course,
it's
a
sort
of
holy
grail
that
would
be
fantastic.
I
could
go
sleep
and
say:
hey,
okay,
rolling
upgrade
is
happening
itself.
I
don't
need
to
care,
but
unfortunately
we
are
not
there.
Yet.
I
think
that
for
industry
this
is
one
of
the
holy
grails
and
once
we
will
do
it,
we
will
be
really
having
a
significant
benefits
of
running.
The
cloud.
A
Second
part
is
configuration
management.
Configuration
management
needs
to
really
follow
these
12
factor
principles
in
the
beginning
of
session.
I
think
that
there
was
this
q
a
and
someone
was
posting
12
factors
right.
I
believe
that
12
actors
is
the
heart
of
this.
If
the
application
is
not
fulfilling
12
factors,
it's
really
not
cloud
native
right.
It
needs
to
really
be
completely
separated
from
health
charts,
it
needs
to
be
separated
from
from
code,
and
it
needs
to
be
stateless
and,
and
that's
absolutely
essential-
the
next
one,
our
famous
example
upf.
A
You
know
in
a
world
of
boxes,
there
was
a
statement
like
have
we
completed
all
the
regression
tests.
Yeah
project
manager
is
calling
saying
how
much
percentage
of
the
tests
you
you've
completed,
or
a
famous
management
question.
Well
guys
tell
me
how
many
of
test
cases
you
have
automated
well
in
cloud
native?
This
is
no
longer
the
question
in
cloud
native
word.
The
question
is
well
how
many
test
cases
you
have
automated
for
a
root
cause
analysis,
because
you
need
to
run
the
test.
A
The
test,
automation
always
continuously,
and
the
fact
that
you
will
be
having
thousands
of
pickups
from
the
results
does
not
mean
that
you
know
what's
wrong.
So
I
will
park
with
this
one.
I
would
love
to
have
this
sort
of
q
a
right
now
yeah,
and
that
was
basically
it
what
we
prepared.
Thank
you
very
much.
Thank.