►
From YouTube: How Kubernetes helps saving the German State 10 Billion Euro - A.Grzesik &M.Breidenbach, GK Software
Description
If you go in Germany to your supermarket or in your favorite restaurant you will see some cryptic string on the receipt. What does this mean and what has that to do with Kubernetes? And how does it save the state 10 billion Euro?
We will talk about our Cloud Native Journey and how we built up our Fiskal Cloud on Kubernetes to prevent tax fraud in Germany. Starting from zero, we had a lot of learnings during this journey and did a good number of mistakes. We will talk how our API First approach helped taking customers with us from day one. How we implemented GitOps when ArgoCD and Flux were not as prominent as today and what we learned. We will explain why we violated the Microservices pattern to improve stability. We will tell you how we shaped our first true Cloud native engineering unit and adopted the SRE concepts to sign billions of transactions in Germany 24/7 with 100% uptime.
Website: https://www.gk-software.com/de/
Organized by @Microsoft @kubermatic7173 @SysEleven
Thanks to our sponsors @CapgeminiGlobal, @gardenio, @sysdig, @SUSE, @anynines, @redhat, nginx, serve-u
A
Just
to
give
you
some
few
background,
information,
Who,
We
Are,
so
GK
was
founded
in
1990
and
we
call
it
Saxony
Valley,
similar
to
Silicon
Valley.
So
it's
a
typical
story
of
a
startup
company
which
was
founded
in
a
small
flat
by
two
people,
so
Mr,
glass
and
Mr
formula,
and
this
company
is
growing
over
the
years
up
to
now
a
mid-sized
Enterprise
company,
with
more
than
1
000
employees
located
in
several
countries
and
I.
A
A
A
couple
of
companies
are
listed
here
in
the
slide,
which
are
our
customers,
and
today
we
want
to
talk
about
the
subsidiary,
which
is
the
deuts
of
the
sky,
100
subsidiary
of
GK,
and
this
company
was
founded
in
2019
to
yeah,
to
create
a
solution
to
give
our
customers
and
other
customers
in
the
retail
Market
the
possibility
to
fulfill
legal
requirements.
Regarding
fiscalization,
so
the
physicalization
started
or
becomes
into
the
game
in
2020
first
of
January,
and
what
is
it?
A
Fiscalization
is
to
yeah
to
record
or
transactions
and
to
put
a
digital
signature
on
each
transaction
to
make
it
provable
for
text
Auditors
at
a
later
stage.
So
it's
something
to
yeah
to
protect
the
data
against
manipulation
and
so
to
save
tax
income
for
the
journal.
State
I
called
it
German
gruntly
kite,
and
this
was
one
of
the
biggest
challenges
for
us
to
combine
the
native
Cloud
approach.
So
we
wanted
to
provide
a
native
Cloud
solution
for
our
customers,
which
is
easy
to
maintain,
because
we
have
to
keep
in
mind.
A
You
are
not
able
to
earn
any
single
Bureau
by
using
this
solution
as
a
company
as
a
retailer.
So
you
just
have
to
spend
the
money
to
fulfill
German
more
requirements,
so
it
has
to
be
easy
to
use.
It
has
to
be
very
stable.
It
has
to
be
reliable
and
therefore
we
started
this
journey
and
we
had
to
keep
in
mind
to
fulfill
all
these
dedicated
requirements
which
are
listed
in
several
laws
and
and
things
like
in
Jordan's,
foreign.
B
B
When,
when
the,
when
the
government
said,
we
need
to
use
it,
we
had
a
challenging
timeline
must
be
stable,
24
7
operations,
because
we
tell
us
we
have
also
a
gas
station
and
talking
about
that
open,
Around,
the
Clock
and
finally,
a
high
traffic
and
make
it
also
complicated.
The
law
requires
that
you
have
something
installed
on
the
actions
and
the
devices
inside
the
stores.
They
must
continue,
so
they
have
Bots
to
maintain
and
monitor
Edge
components.
B
So
basically,
it's
also
some
kind
of
iot
project
became
some
kind
of
iot
project.
So
to
that
yeah,
Cloud
native,
we
set
the
page,
then
we
must
start
for
Native.
It's
the
only
way
to
go.
We
were
convincible,
but
we
had
no
experience.
So
we
basically
started
from
zero.
We
didn't
play
The
100
kubernetes
with
the
docker
experience,
but
nothing
else.
So
we
started
really
from
scratch.
B
B
That
tells
all
the
monitoring
management
billing
of
the
solution,
and
we
had
our
partners
running
a
graphic
part
of
the
solution
that
makes
a
signature
because
that
required
a
specific
data
center.
Only
two
data
center
in
Germany
are
qualified
for
running
that
Solution
on
specific
data,
proof
Hardware,
so
that
was
that
was
integrated
and
seamlessly
over
our
cloud.
B
The
reality
looks
a
bit
more
complex,
so
we
had
a
lot
of
components
built
up.
We
started
small
with
service
upper
parts
to
create
this
red
sandbox
environment,
to
onboard
customers,
starting
with
some
on
Azure
with
AKs
clusters
and
later
moved
on
to
some
production,
ready,
State,
one
of
the
first
adopters
of
the
new
term
data
centers
of
Microsoft.
A
B
B
B
B
So
we
had
some
not
so
much
experience
how
to
slice
services
to
create
the
services
that
required
very
the
same
data
and
to
to
that
course
that
we
had
always
on
tubes,
and
so
we
decided
okay,
civil
access.
We
share
database
between
services,
it
works
somehow,
but
I
would
not
do
this
again.
Let's
say
I
would
really
say
the
idea
of
microservices
being
isolated.
Is
it's
really
good
because
they
have
a
great
dependencies
that
also
make
a
little
difficult
to
apply
changes?
B
We
also
build
our
own
API
Gateway.
Instead
of
using
something
yeah.
Of
course
it's
complex
you
can't
handle,
don't
need
so
much
build
something
based
on
explaining
Cloud
Gateway,
but
also
that
means
we
are
now
running
one
API
Network
Three,
because
we
have
different
policies,
if
you
otherwise
couldn't
really
handle,
and
nowadays
I
would
maybe
go
with
something
like
a
problem
or
gravity
or
something
else
that
is
existing
to
make
it
really
easier
because
finally
needs
functionalities.
B
B
The
cost
us
also
some
troubles
regarding
performance
latency,
because,
finally,
we
can
run
them
into
the
clouds.
They
are
managed
in
the
cloud,
but
they're
not
really
suited
for
the
for
the
scalable
performance
and
last
but
not
least,
of
our
bad
ideas.
Let's
say
was
so
we
said:
okay,
we
can
use
Cosmos
DB
as
replacement
for
document
database
mongodb
because
it
provides
same
API.
B
Finally,
that
doesn't
work
out,
because
Cosmos
DB
works
well
for
a
specific
use
case
for
about
four
hours,
so
this
is
different
load
levels
and
and
using
videos.
Api
is
not
it's
not
100
compatible.
So
we
had
some
troubles
and
decided
two
months
before
going
live
to
replace
it
completely
with
longodb
at
last
managed
and
highly
due
to
Microsoft's
approach.
We
were
able
to
use
this.
B
Other
learnings
we
had
so
project
set
up.
If
you
have
a
lot
of
services
is
difficult,
so
you
should
think
about
scaffolding
process,
some
templating
or
using
for
Maven.
We
are
using
some
archetypes
for
creating
really
new
Services
out
of
the
box
that
work
and
then
the
team
can
work
on
it
and
as
an
Enterprise
environment,
you
also
said
cannot
choose
any
technology
but
plastic
limit,
because
otherwise
it's
helpful
to
share
stuff
and
knowledge.
B
The
development
student
is
really
really
possibly
been
great.
So
this
isolation
of
microservices
and
the
Really
enforcement
of
these
component
components.
We
are
much
much
faster
in
implementing
features
and
very
predictable,
but
to
this
process
really
became
a
little
bit
problematic
with
a
lot
of
services
maintained
and
especially
in
the
high
frequent
development
phases,
we
had
to
somehow
release
multiple
Services
together,
because
we
had
API
changes,
set
of
stable
and
and
got
their.
B
That
would
tell
you,
so
this
is
something
you
need
to
automate,
so
we
have
to
implement
some
automation-rounded
pipelines,
automations
and
such
kind
of
stuff.
The
service
lighting
is
another
challenge.
I
think
for
the
microservices
world
really
finds
the
right
slicing
and
there
really
is
a
slicing
by
domains
this
vertical
size.
B
Documentation,
something
like
backstage
our
API
first
approach
was
another
thing
that
we
give
us
some
great
benefit,
so
we'd
find
the
apis
and
provide
to
the
customers
before
we
even
have
them
implement.
It
allows
them
to
integrate
early
on
their
stuff,
with
some
box
around
it
that
that
was
really
something
that
that
solved.
B
This
is
this
problem
that
customers
need
something
and-
and
we
are
not
ready-
because
we
simply
had
so
many
many
stuff
so
using
open,
API
different
tools
like
IntelliJ
Visual,
Studio
code
or
stoplight
Studio,
have
the
API
within
separate
Gateway
course
as
their
own
artifact
and
from
there
publish
them
into
our
developer
portal,
but
make
it
generate.
Server
and
client
stops
with
open
API
generator
and
also
the
QA
can
already
create
the
tests
in
the
implementations.
That
is
running
because
the
API
is
already
available
and
everything
is
awesome
for
playing.
B
B
The
London
guidelines
is
very
useful
that
they
are
very
complete
and
a
good
starting
point
of
view
to
find
your
own
guidelines
so
other
learnings,
we
had
to
understand
the
difference
between
container
got
a
little
bit
problematic
for
someone
who's
not
used
to
it,
causes
some
troubles
about
memory
limits
and
and
settings
what's
running.
Jvm
on
kubernetes
is
a
little
bit
tricky
because
standard
jvm
settings
are
quite
conservative.
They
waste
a
lot
of
memory.
B
On
the
other
hand,
yeah
if
you
tune
it
too
much
and
don't
didn't
pay
into
account
that
the
Heap
is
only
one
part
of
the
jvm
memory.
Two
night
star
starts
killing
your
pots
because
they
use
more
resources
because
there
are,
there
are
other
other
memory
consumptions
we
had
two
lot.
Of
course
we
had
a
lot
of
out
of
memory
fields
from
kubernetes.
B
Also
our
popes
we
have
to
get
right
of.
There
were
issues
that
that
life
is
very
popes.
We
put
too
much
in
life
and
Scopes
that
took
too
long
to
to
check
and
two
minutes
start
cleaning
our
pox,
even
if
they
are,
would
be
just
not
answering
because
they
had
too
much
load
because
lightness,
but
that's
too
much
things.
So
it
kind
of
answer
in
time.
B
Debugging
is
the
problem,
especially
during
job
applications
in
kubernetes.
So
we
solved
this
mainly
by
adding
additional
metrics
custom
spans
or
what
facing
and
adding
application
metrics
that
allows
us
to
see
really
what
is
happening
inside
so
there's
a
pots
and,
last
but
not
least,
yeah
separating
these
resources
consumed.
A
B
B
And
creating
some
kind
of
trust
for
the
services
behind
also
allows
us
to
really
put
some
area
that
you
can
also
switch
off
some
some
stuff
coming
in,
or
you
have
some
talks
here
today
about
that
and
Finance
Cloud
security
side
using
the
cloud
provider,
security
provided
by
Azure
defined
by
access
policies
to
web
application,
firewalls
and
executing
regular
dentists.
So
all.
A
B
B
This
configuration
and
all
the
helm,
charts
related
to
the
environment,
so
you
copy
over
the
home
charts
and
then
the
last
card
contains
everything
that
is
needed
for
the
deployment
and
from
there.
Then,
when
there's
a
change,
the
changes
then
pushed
into
the
kubernetes
cluster
and
with
tagging
the
Veloster
we
could
push
everything
that
is
in
stage
environment
that
was
released
and
tested
as
one
part
as
atomic
operation
to
stable
environment
which
helped
us
to
to.
We
also,
we
always
know
what
is
what
the
stable
statement.
B
So
we
don't
publish
single
services,
but
only
only
sets
of
services
that
we
that
run
all
the
tests
together,
but
this
makes
a
little
bit
difficult
because
sometimes
big
deployments
with
timeouts
that
you
change
it
and
also
using
hot
fixes
as
a
manual
work
settings.
It's
not
so
easy
last
part
of
leaves
possibility.
So
we
utilize
a
bunch
of
monitoring
tools.
B
That's
probably
everybody
knows
so
using
the
finance
resource
for
metrics
and
bloopy
for
the
logging
for
Edge
monitoring
very
collect
events,
which
is
really
something
that
helped
us
a
lot
through
this
airplane
from
The
Edge.
We
send
events
and-
and
we
could
see
what
happens
so
we
can
see-
we
can
see
problems
at
the
customer
side,
because
customers
even
recognize
that
problem
that
helps
us
really
a
lot
and
use
yoga
for
tracing
and
spring
will
admin
for
some
internal
development
access,
because
it
gives
us
the
insight
into
the
Springwood,
the
containers.
B
B
Now
we
have
something
some
some
anecdote
watches
the
Watchers,
so
this
is
our
first
day
in
in
full
production,
first
of
April
of
last
year,
which
was
also
the
Thursday
before
Easter,
which
is
the
highest
volume
day
in
in
retail.
Ever
in
the
year
and
suddenly
afternoon
we
got
pause
Modified
by
our
management,
which
are
looking
at
that,
because
it
was
our
first
really
full
production.
Every
customer
must
have
used
it.
B
We
have
no
data.
What's
what's
happening,
everything
is
broken,
no
everything
else.
She
seems
to
work
and
what
you
find
out.
If
you
guys,
okay,
yes,
we
have
some
some
stuck
in
the
in
the
in
the
channel,
so
the
the
resources
needed
Force
monitoring
profit,
for
instance,
is
configure
so
and
it
gives
them
too
much
resources,
because
too
much
traffic
of
the
monitoring
instruments
kills
them.
B
So
we
don't
didn't
get
the
data,
so
the
system
worked
completely
fine
with
with
I
think
80
million
signatures
this
day,
but
we
simply
had
had
to
break
down
on
the
on
the
logging
breakdown.
So
we
increased
the
number
of
parts
and
wrote
again
and
then
everybody
was
and
similar
also
the
Azure
metrics
that
Azure
provides.
They
want
to
able
to
provide
it
in
time.
So
we
had
some
gaps,
some
something
big
database
up
and
took
some
minutes
to
pick
up.
A
Okay,
let's
continue
so
what
about
the
operations
itself
of
this
solution?
Now,
I
think
you
are
all
aware
if
you
work
in
devops
about
the
regular
Works
tasks
and
the
challenges,
so
automation
is
key
and
the
team
or
the
main
task
of
the
team
is
to
automate
as
much
as
possible
why
they
have
to
do
it
because
they
need
time
to
identify
and
to
investigate
the
tool
landscape.
A
And
if
you
look
into
the
native
Cloud
Foundation
picture,
the
big
one,
you
have
nearly
a
thousand
tools
and
an
application
which
can
be
used,
but
you
have
to
decide
which
tool
fits
best
for
your
solution
for
your
requirements
and
therefore
the
devops
team
need
enough
time
to
spend
into
investigation
and
to
find
out
what's
the
Right
View.
So
if
you
are
able
to
automate
as
much
as
possible,
you
can
save
time,
and
then
you
can
spend
this
time
into
other
important
things.
A
A
Next
thing
is
to
use
it
on
a
Gear
redundancy
mode,
so
using
different
data
centers
within
Azure,
you
have
always
data
center
pairs,
which
means
there's
always
if
you
use
Germany
was
Central.
The
other
one
is
Germany
North
and
then
to
run
this
solution
over
different
data.
Centers
is
the
next
thing
to
increase
the
reliability.
A
In
addition-
and
this
is
something
special-
this
devops
team
I
built
up
is
also
responsible
for
second
level
support,
so
we
are
serving
our
customers,
and
this
is
another
level
of
the
devops
methodology
or
the
devops
approach
so
eat
your
own
dog
food,
take
care
about
your
own
bugs
and
make
customers
happy
as
a
whole
team,
and
therefore
we
are
responsible
and
thanks
to
the
team,
some
of
them
are
here.
They
are
doing
a
great
job.
A
So
how
we
work
scrum
is
more
or
less
commodity.
Everybody
knows
these
words
these
buzzwords,
but
what
does
it
mean?
Really
so
in
the
development
team?
It
makes
sense
to
to
work
with
a
two-week
sprints,
and
after
these
two
weeks
we
we
are
doing
a
review
of
two
hours
and
every
developer
can
show
and
present
to
the
whole
team
what
he
has
developed
in
these
last
two
weeks
and
can
present
the
results.
Fine
in
the
devops
team.
A
It
doesn't
work
because
there
are
too
much
dependencies
from
external
circumstances
which
are
not
allowing
us
to
map
to
set
everything
within
this
two
weeks.
Sprints
and
therefore
we
decided
to
do
it
in
a
mixed,
like
strum
and
come
down.
So
we
call
it
Brahman
and
it's
not
a
big
issue.
If
one
or
two
tasks,
some
tasks
are
handed
over,
our
postpones
to
the
next
term
expensive.
A
A
It
doesn't
help
at
all.
If
you
say
you
made
a
mistake,
so
yes,
there
was
a
mistake
and
we
have
to
learn
from
it,
and
this
spirit
is,
is
the
key
and
and
the
the
central
mindset
of
the
whole
team
as
well
for
the
developers
and
we're
acting
as
one
team
with
the
devops
in
the
development
team,
and
we
made
a
very
good
experience
about
that.
A
So,
all
in
all,
we
are
serving
roughly
600
customers
with
roughly
200
000
cash
registers.
We
have
installed
as
an
edge
component,
roughly
120
000
applications
running
in
the
stores
across
Germany,
and
we
are
executing
between
30
up
to
40
million
transactions
per
day
and
you
have
to
have
two
signatures
per
transactions.
So
this
means
we
have
to
produce
up
to
80
million
signatures
for
these
transactions
per
day
and
the
overall
error
failure
rate
for
these
signatures
is
0.050
as
well.
The
availability
of
the
our
Central
Cloud
solution
is
about
100,
sensible
life.
A
So
it's
a
nice
challenge
to
keep
this
because
increasing
is
not
possible
but
to
keep
it
is
challenging
enough
and
the
whole
solution
is
operated
by
six.
Six
colleagues
and
customer
support
rating
was
nice
and
yeah.
If
you
are
interested
to
learn
more
about
that
or
we'll
talk
about
it,
let's
do
it
afterwards,
while
having
a
beer.
B
Okay,
then
30
seconds
for
the
final
thoughts,
but
just
to
summarize
what
you
learned
that
we,
the
cloud
native
approach,
really
worked.
It
really
worked
for
us
starting,
but
it's
also
an
organizational
challenge.
So
we
had.
The
mindset
must
be
there.
We
had
the
executive
support
for
that,
so
we
were
able
to
build
also
and
and
work
in
that
way.
Empowering
the
people
and
and
3D
working
together,
microservice
really
worked
giving
us
a
big
advantage
of
producing
fast
new
features
in
a
stable
way
and
yeah
automation.
B
B
Customers
are
treated
as
partners
and
due
to
the
approach
of
getting
them
early,
integrated,
getting
fast
feedbacks
talking
together
and
having
the
operations.
Development
team
helps
us
really
to
improve
that
and
makes
customers
happy.
So
yeah
cloud
data
works
and
looking
forward
to
making
yeah
transforming
our
whole
company.
That
approach.