►
From YouTube: Running Kubernetes in a Manufacturing Line – What Could Possibly... Mario Fahlandt & Tobias Schneck
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Running Kubernetes in a Manufacturing Line – What Could Possibly Go Wrong? - Mario Fahlandt & Tobias Schneck, Kubermatic GmbH
Imagine your manufacturing line is controlled by services running in your datacenters’ Kubernetes clusters. You have facilities in locations all over the world. You provide a managed service with uptime SLA. Now, there is an issue with the internet connection. Or security is shutting down all connections to defend against a cyberattack. And your production line must keep working because every downtime is money. This was the challenge to solve, and we did! Did you ever think about - How to run basic Infrastructure Services like DHCP and DNS in a cloud native way for manufacturing services? How to autoscale cluster on-prem? Follow us in the rabbit hole using all kinds of CNCF projects to build a setup that scales, is able to shift and redeploy workloads, and continues to function without relying on cloud vendors or external services. We will show you the obvious and non-obvious challenges of cloud native adopters in the industry 4.0 sector, including some true edge computing cases.
A
B
Exactly
so,
what
is
this
about?
We
luckily
won
a
project
to
implement
kubernetes
platform
at
a
chemical
producing
company.
So
that
was
the
challenge
itself,
because,
if
you
think
about,
if
you
produce
chemical
goods,
something
went
wrong,
then
potentially
it
is
not
so
good
for
everyone,
so
yeah.
But
before
we
go
in
detail
a
few
things
about
us,
so
my
name
is
tobias.
You
can
say
toby,
I'm
working
for
cuba
medic
since
four
years
now
as
a
principal
architect,
helped
to
adapt
the
customer
side
kubernetes
in
a
scalable
way
and.
A
Yeah,
I'm
mario,
I'm
also
working
for
clippermatic,
I'm
a
professional
service,
kubernetes
consultant.
We
basically
help
our
customers
to
build
up
their
chromatic
solution
inside
of
their
data
centers
or
in
the
infrastructure,
and,
as
you
can
see,
we're
both
from
bavaria.
That's
why
we
were
litterhosen,
of
course,
and
now
to
go
deeper
into
everything.
A
We
need
to
go
back
in
history.
So,
yes,
sorry,
it's
friday
and
history.
Lessons
are
always
bad
and
no
one
likes
history
lessons.
But
when
we
go
back
into
the
end
of
the
18th
century,
we
had
the
first
weaveries
that
started
mass
producing
goods.
This
was
called
industry
1.0.
Now
I
mean
it's
new,
but
we
go
with
it
and
then
we
have
a
involvement
in
the
industry
line.
So
we
had
mass
production
in
this
at
the
start
of
the
20th
century
and
we
had
computer
automation
in
this
70s.
A
So
when
we
look
at
industry
4.0,
it's
like
every
manufacturing
plant
that
we
have
need
to
be
self-sufficient,
but
also
interconnected
to
different
other
plans,
because
we,
the
the
whole
production
line,
maybe
spends
over
a
lot
of
factory
plans,
and
we
are
currently
into
the
process
that
we
have
like
data
analytics
in
every
single
step
and
also
security
and
observability.
That
might,
as
our
systems
are
still
running
and
also,
we
want
to
improve
every
single
step
of
our
manufacturing
process.
A
So
the
problem
here
is:
we
now
need
a
lot
of
compute
power
and
we
also
need
a
lot
of
a
lot
of
servers,
but
we
cannot
put
them
in
the
cloud
because
the
amount
of
data
that
we
would
move
to
the
cloud
and
get
back
from
the
cloud
is
just
too
big.
So
everything
is
placed
next
to
the
manufacturing
manufacturing
line.
So
all
of
those
machines
need
to
work,
but
there
are
some
locations
that
are
very
small.
A
B
Yeah
and
then
we
said:
okay,
kubernetes,
cool
chemical
manufacturing,
oh
scary,
okay,
so
let's
get
this
started
and
we
said
okay
come
on,
we
are
experiencing
kubernetes
what
should
go
wrong
and
then
we
thought:
okay,
what's
mainly
the
targets
for
halls
this
project,
so
we
thought
okay,
let's
start
with
some
workshops.
We
discussed
it
and
then
we
came
down
to
a
few
key
project
targets.
So
basically
is
one
is
hosting
manufacturing
software.
B
Why
it's
different
manufacturing
software
is
mostly
not
as
like
cloud
native
software
are.
You
are
need
to
connect
to
erp
systems
like
shp.
You
need
to
connect
to
machines,
you
need
to
connect
to
devices.
We
have
a
requirement
that
some
windows
templates
are
needing
some
of
our
services
and
it's
like
printers
did
you
ever
get
asked
in
a
project
about
printers
with
kubernetes.
B
We
are,
and
that
was
the
thing
so
and
then
we
realized.
Okay,
that's
also
different
kind
of
network.
We
are
used
to
like
it
network,
but
that's
a
manufacturing
network.
We
have
at
least
three
firewall
as
far
we
know
across
to
reach
this
network,
so
we
have
a
firewire
from
our
managed
service
to
the
customer
from
the
customer
to
the
cloud
from
the
cloud
to
the
on-premise
from
on-premise
to
the
user
workload.
So
you
see
there
are
a
lot
of
components.
B
What
are
really
like
interconnected
and
we
need
to
first
understand
what
is
the
topic
of
manufacturing
and
that's.
What
they
want
is
a
highest
ability
in
a
high
grade
automation.
Why?
If
it
could
be
that
the
data
center,
our
data
center
and
manufacturing,
get
destroyed
by
an
accident
or
something,
and
they
need
to
be
capable
to
rebuild
this
reputable
repute,
reproduce
this
quickly
and
yeah.
Then
we
need
independency,
so
our
operation
of
kubernetes
does
not
in
should
not
affect
any
other
micro
service
deployment
or
the
manufacturing
software
deployment.
B
We
need
higher
availability
and
that's
across
the
disputed
locations.
So
this
will
do
the
location
means
we
have
currently
running
on
three
data
centers,
but
the
plan
is
to
run
it
on
30
data,
centers
so
and
then
yeah.
We
need
e
to
e
operations,
so
we
need
a
team
who
can
operate
across
the
globe's
data
centers,
so
the
next
data
center
is
u.s,
so
it
will
be
completely
different,
as
we
currently
are
working
in
europe.
B
Okay,
then
independency
what
means
in
this
in
detail,
so
it
means
that
we
want
to
decouple
things
to
be
thinking
in
microservices,
but
it
also
means
that
an
update
of
component
a
should
not
block
or
roll
out
of
component
b,
and
that
goes
back
to
the
infrastructure.
So
if
we
blocking
people
to
deploy
their
changes,
we
are
like
kind
of
the
bottleneck,
and
that
is
not
possible
in
this
case.
We
wanted
to
be
fast
because
fast
means
for
them
success.
B
B
Everything
is
really
complex.
I
think
kubernetes.
Now
is
a
point
where
we
say:
okay.
This
is
really
really
complex.
It's
maybe
too
complex
already
and
that's
something
what
you
need
to
keep
in
mind
that
you
make
things
simple,
more
consumable
and
more
replaceable
that
you
say:
okay,
we
still
can
handle
it
in
a
two
pizza
team,
okay,
so
high
availability,
everyone
has
a
different
integration
of
it.
In
our
case,
it
was
combined
h,
a
across
multiple
data
centers,
so
not
only
in
one
data
center.
B
B
So
this
means
monthly
monthly.
We
have
70
minutes
downtime.
What
is
not
much
when
upgrade
fails
or
something.
So
this
is
really
a
small
number
where
we
can
fail
and
that's
something
what
you
think
in
a
year,
it's
only
three
hours
a
year
is
really
long
so
three
hours,
if
something
fades,
could
be
really
a
tough
time
for
him
to
fix
it.
But
currently
we
are
fine,
we
managed
to
do
it
and
we
now
go
in
detail
how
we
did
it.
Yes,
so.
A
We
have
another
component
that
you
usually
not
think
of
is
we
are
in
a
corporate
world.
A
corporate
world
has
a
lot
of
departments
and
they
don't
necessarily
want
to
speak
to
each
other,
and
they
are
really
really
big
and
you
need
to
communicate
with
everyone
and
for
this
we
needed
to
basically
go
to
our
to
our
customer
and
say
we
cannot
do
everything
by
ourselves
so
and
we
need
we
need
a
counterpart.
A
So
for
this
we
have
a
stakeholder
team
at
our
customer
to
with
whom
we
talk,
with
whom
we
we
design
the
whole
architecture
and
with
whom
we
yeah
talk
on
a
daily
base
and
create
the
complete
architecture,
but
also
we
need
to
talk
to
the
consumer
departments
because
they
are
creating
the
requirements
for
what
they
want
to
run
on
the
cluster,
what
they
need.
Well,
for
example,
we
now
need
a
200
terabyte
of
data
storage
somewhere
to
be
accessible
in
every
data
center.
A
This
is,
this
is
new
requirements,
so
you
you're
constantly
talking
to
all
of
the
teams,
and
also
we
have
the
team
that
everyone
likes
and
everyone
loves
the
security
who,
basically
later,
maybe
they
know
a
little
bit.
But
the
first
thing
is
they
ask?
Are
you
do
you
really
want
to
do
this?
How
you
want
to
do
this?
No,
you
can't
do
this
and
it's
a
constant
yeah
communication
with
them
to
bring
them
up
to
speed
and
also
to
teach
them
so
that
you
work
together
as
a
team.
So
we
came
up
with
the
idea.
A
Why
not
splitting
the
competences
into
small
departments
so
that
everyone
knows
exactly?
This
is
my
part.
This
is
where
I
can
focus
on,
and
this
is
where,
where
my
core
competencies
are,
so
we
started
with
one
team
which
is
part
of
the
part
of
the
customer,
which
is
the
core
infrastructure
team.
These
teams
are
there
for
providing
the
basic
level
like
I
need
my
data
center.
A
I
need
my
network,
I
need
my
firewall
appliances,
I
need
my
azure
accounts
and
so
with
these
teams
we
basically
first
designed
the
base
layer
of
of
everything,
and
the
next
theme
is
the
application
platform
team.
So
this
is
the
team
that,
with
whom
we
work,
so
you
can,
you
can
say
it:
it's
basically
a
vendor
inside
of
the
corporate
and
they
and
their
customers
are
the
different
departments,
and
they
ask
you
basically
hey.
A
We
need
a
kafka,
but
we
don't
need
a
kafka
for
us
alone.
Can
we
just
jump
on
a
kafka,
so
this
team
is
providing
very
large
services
on
the
bay
on
the
global
level
and
you
can
basically
just
go
in
and
buy
stuff
from
them,
and
the
last
team
from
the
customer
side
is
not
one
team.
It's
we
don't
know
how
many,
because
we
never
talk
to
them,
but
they
are
currently
always
talking
to
to
the
to
the
department
above
them.
A
These
are
the
teams
that
are
actually
running
their
workload
on
those
kubernetes
clusters
that
we
that
we
provide,
together
with
the
with
the
customer
team,
and
they
there
come
all
of
the
requests
for
we
need
this.
We
need
that
we
need,
we
need
all
of,
we
need
more
storage
and
we
need
to
con.
Take
all
of
this
in
account,
and
this
is
our
job.
A
We
are
basically
the
cloud
native
infrastructure
team,
so
we
provide
large
services
in
all
of
the
data
centers
and
we
are
doing
the
consumption
reporting,
we're
doing
the
metering
and
we
are
creating
the
the
clusters
and
we
are
creating
the
all
of
the
infrastructure
and
also
we
are
taking
care
of
the
infrastructure.
So
we
are
the
people
where
you
can
say:
hey
everything
is
down,
do
something
now,
and
then
we
come
to
the
point
where
we
say
running
kubernetes
in
manufacturing.
B
We
said
talent
accepted
what
can
go
wrong
now.
We
are
more
on
a
technical
side,
so
we
solve
kind
of
the
setup.
But
then
our
core
idea
was
okay
to
handle
this.
We
need
repeatable
function
units,
and
this
does
not
mean
that
it's
technical,
a
unit.
It
must
be
a
component.
What
we
can
use.
This
is
not
just
kubernetes.
This
is
infrastructure.
We
have
lots
of
things
to
care
about
vsphere
permission
models
and
everything
should
be
managed
by
api
and
get
ops.
B
B
B
It's
really
hard
it's
hard
and
we
called
it
wonderball
because
you
never
know
if
there's
one
via
more
or
more
and
that's
what
we
did,
then
we
built
kind
of
object
groups.
We
said:
okay,
we
structurize
it.
We
structuralize
the
firewall
rules
in
objects
and
said:
okay
component.
A
of
data
center
need
to
talk
with
component
b
of
data
center,
but
the
data
centers
are
a
generic
object
where
we're
putting
different
data
centers
into
it
and
make
it
then,
with
this
rule,
set
scalable.
B
Okay,
what
else
I
think
everybody
heard
it
infrastructure
is
code
is
great
and
automation.
Automation
is
our
key
success
factor
and
this
starting
with
the
setup
of
how
our
team
can
set
up.
So
we
need
to
really
clear
how
someone
can
bootstrap,
because
it
can
be
possible
that
we
call
that
they
get
a
call.
B
Some
of
my
colleagues
here
also
sitting
here,
hi
get
called
in
the
night,
need
to
check
out
the
repository
and
somehow
need
to
find
something,
and
that's
mostly
the
time
what
you
lose
and
you
remember
if
everything
is
down,
we
have
70
minutes
in
the
months
and
that's
why
we
said.
Okay,
documentation
must
be
also
in
the
code
repository
that
you
have
a
single
point
to
come
into
it
and
we
need
a
centerized
scaling
architecture.
So
we
have
one
core
concept
or
a
few
core
concepts.
What
we
can
then
duplicate
similar.
B
We
think
data,
centers
kind
of
are
ports
and
we
say
hey.
We
want
to
scale
up
the
data,
centers
and
also
users
want
to
consume
it
like
that.
So
the
difference
between
a
cluster
provisioning
between
clouds
are
only
a
full,
a
few
parameters
that
brought
us
to
the
idea.
Okay,
we
standardize
the
data
center
standardized
data
center
means
for
it.
It
must
be
independent
and
universal,
but
we
have
standard
interfaces.
B
How
we
connect
it,
how
we
manage
services
there
and
we
don't
use
any
cloud
specifications
and
that's
where,
basically,
we
say:
okay
data
center,
we
need
to
quickly
put
everything
in
kubernetes,
that's
our
interface,
where
we
can
implement
it
and
make
it
for
our
end.
Customers
easy
to
consume
and
this
setup
is
really
important.
This
is
repeatable
in
short
time,
because
the
disaster
recovery
case
is
one
of
the
main
factors
of
this
company.
B
A
Yeah
so
we
started
when
we
go
back
like
back.
In
the
time
we
had
like
we
introduced.
Containers
containers
are
now
like
cattle,
so
we
don't
care
about
them,
we
throw
them
away.
Then
we
did
the
next
step.
We
say
my
my
notes.
Are
I
don't
care
about
them?
We
can
throw
them
away.
Now
we
go
to
the
point
where
we
say
kubernetes
clusters:
I
don't
care
about
kubernetes
clusters.
Kubernetes
classes
are
also
catalog.
A
But
this
doesn't
this
doesn't
matter
and
we
are
using
our
own
software,
which
is
an
open
source.
Software
called
kibomatic
like
our
company,
so
we
basically
say
we
run
kubernetes
inside
of
kubernetes,
so
we
have
on
our
master,
we
have
or
on
a
seat,
so
we
have
a
seat
in
each
of
the
locations
and
you
run
there.
Your
cluster,
with
the
control
plane
and
inside
of
the
cont
inside
of
the
worker
nodes
of
your
seat
or
of
your
master,
you
have
the
control
planes
of
the
user
clusters
instead
of
containers.
A
So
this
makes
the
overhead
really
really
small,
because
user
cluster
only
needs
0.3
cpus
to
run
in
an
hca
setup,
and
you
only
the
worker
nodes
are
really
then
there
for
for
your
workload,
and
you
can
quickly
spin
those
clusters
up
in
each
of
the
environments
that
you
potentially
create,
because
you
don't
want
to
run
only
production.
You
also
want
to
have
a
dev
and
an
integration
and
probably
a
sandbox
where
you
want
to
test
things.
But
the
problem
is:
when
you
template
a
data
center,
you
don't
need
just
kubernetes
clusters.
A
There
are
a
lot
of
more
things
that
you
need
in
every
state
single
data
center,
and
this
brought
us
to
the
idea:
hey.
Why
not
click
create
a
service
cluster
right
next
to
our
seat,
that
is
in
every
single
data
center
and
with
this
service
cluster,
we
provide
all
of
the
basic
services
that
you
need
to
run
the
data
center.
So
we
started-
and
this
is
now
bringing
us
to
all
of
the
cncf
world-
we
started
to
create
this
the
service
clusters.
A
A
And
printers
yes
and
tablets
and
printers-
and
you
and
there's
nothing-
I
mean
there
is
something
we
figured
out.
I
see
dhcp,
but
I
see
dhcp
does
not
run
in
a
container,
so
we
created
isc
dhcp
inside
of
a
contrainer
and
now
can
basically
give
ip
addresses
everywhere
outside
of
our
kubernetes
cluster,
while
running
the
dhcp
server
inside
of
the
kubernetes
cluster,
and
you
need
storage
to
store
like
pictures,
every
single
manufacturing
process
is
taking
a
photo
so
that
you
basically
see
is
everything
all
right.
Can
we
improve
on
this?
A
So
there's
a
lot
of
pictures
being
taken
and
also
you
need
a
repository
but
wait
a
minute.
They
are
not
all.
They
are
not
all
errors
connected
to
the
master
service,
cluster
yeah
for
the
s3.
We
said:
okay,
we
don't
want,
we
don't
want
to
have
the
data
replicated
to
the
master,
because
it's
just
too
much
data
and
it's
not
important.
A
In
the
same
way,
so
you
need
to
greatly
figure
out
what
is
important
for
your
use
case
and
what
is
not
important
for
your
use
case
and,
as
you
can
see,
we
set
up
the
idea
that
everything
has
its
local
address,
which
means
that
every
single
service
in
all
of
the
data
data
centers,
has
always
the
same
address.
So
when
I'm
in
data,
when
I'm
in
a
manufacturing
plant
a-
and
I
call
local
dot
document,
I
get
the
nearest
I
get.
A
The
nearest
coordinates
if
I
do
the
same
thing
in
data
center
b,
it's
the
same
address,
but
it's
also
the
local
one,
and
this
brings
us
to
our
flexibility
that
we
have
when
we
hit
a
disaster
case
because,
as
we
said,
half
an
hour
outage,
one
million
loss
so
basically
the
data
center.
If,
if
something
happens
with
the
internet
connection,
we
still
need
to
our
plan
to
be
operational,
so
we
made
our
failover
case
the
default.
A
So
our
failover
case
is
always.
This
is
how
it
the
the
the
request
is
typically
looks.
So
we
always
call
the
local
service
and
we
don't
care
if
we
have
the
master
available
or
we
don't
care
if
the
other
data
centers
are
available.
This
also
is
a
speed
improvement,
but
what
if
my
local
installation
is
failing,
that's
also
easy.
We
can
just
fail
over
to
the
master
or
fail
over
to
the
to
the
second
data
center
location
and
how
we
managed
to
do.
B
Yeah,
basically,
you
can
think
that
it's
not
everything
work
from
scratch
so
and
it
was
more
like
a
trail
map.
So
it's
really
something
what
you
consider.
Each
project
has
its
own
characteristics.
Each
data
center
has
its
own
correct
characteristics
and
there
are
people
who
also
are
involving
there.
So
you
cannot
just
say
hey.
We
do
it
that
way
and
it's
the
only
way
we
do
because
security
says,
I
don't
care.
What
do
you
want?
You
need
to
discuss
this
with
us
and
we
need
to
approve.
B
So
we
are
here
the
final
one
who
approves
it
so
that's
yeah
and
that
sprung
brought
up
to
that
here.
So
it's
mostly
like
a
trail
map.
What
you
repeat
and
repeat
and
try
to
improve,
so
our
core
success
was,
we
started
small,
so
we
said
okay,
we
want
one
time.
The
first
end
to
end
thing
solved
and
then
we
improving
by
time
and
the
first
thing.
What
did
we
take?
A
look
was
the
local
infrastructure,
so
we
needed
to
get
into
the
data
center
setup.
We
needed
to
get
explained
it
to
us.
B
I
mean
to
understand:
what
are
the
policies
how
we
can
connect
to
this
even
like
to
have
a
jump
server?
There
was
really
a
hard
task
because
first
you
need
to
convince
them.
Okay,
we
have
to
save
implementation
for
it
and
we
our
we
call
it
operation
center,
it's
really
safe
and
it's
safe
that
we
can
access
your
critical
production
network,
also
backup
what
means
backup
I
mean
yeah.
B
We
do
automatic
backup
every
20
minutes
from
our
clusters,
no
problem,
but
that's
not
the
end
of
story,
because
you
need
to
think
about
where
to
store
it
in
the
disaster
case,
because
it
doesn't
help
you
if
you
have
it
in
your
local
data
center
and
it's
burnt
down
so
there
you
need
to
talk
with
the
customer.
Okay,
what
are
your
failover
strategy
where
we
can
store
it?
Do
we
have
s3
service
to
store
it
and
so
on
and
yeah,
then
core
infrastructure?
B
Basically,
we
thought
that's
the
easy
part,
but
we
figured
out
that
yeah
vsphere
kind
of
is
different
as
we
expected
it
as
we
saw
it
at
other
customers.
Somehow
we
have
now
we
send.
We
had
dedicated
zones
where
we
wasn't
aware
about,
and
we
need
to
figure
out,
okay,
how
we
can
use
this
to
make
it
really
reliable
and
network.
I
mean
yeah,
you
know
when,
if
you
have
a
name
for
something
like
wonderball,
the
network
was
really
a
challenge,
because
sometimes
we
didn't
even
know
there's
a
firewall
in
between
security
firewall.
I
said
it.
B
Segmentation
and
authentication
authentication
is
important.
We
need
to
have
every
person
identified,
then
how
you
onboard
a
managed
whole
managed
service
team
with
customer
accounts.
This
was
really
really
an
organizational
challenge,
good.
What
else
so
services
we
said
everything
we
want
to
consume
needs
to
be
a
service.
Mario
told
you
there's
three
service
for,
for
example,
and
this
must
be
local.
The
nest
service
be
local
caches.
This
was
this
implementation.
B
Detail
that
you
can
reconcile
was
key,
so
same
are
independent,
but
they
can
reconcile,
and
that
is
the
point
here
on
the
independent
sing.
So
we
said
central
master.
This
is
the
holy
grail.
If
we
change
our
configuration,
that's
there
and
on
the
locals,
we
just
have
replicas
similar
as
spots.
They
can
go
away,
they
can
spin
up,
they
can
scale
up.
B
I
think
automation
is
for
everyone
clear,
but
if
you
have
things
what
is
not
automated,
and
maybe
you
have
new
tools
what
you
don't
know,
then
you
think
about
declarative
management
too.
So
you
we
created
wrapper
around
things.
What
maybe
are
not
in
our
responsibility,
but
we
needed
to
configure
them
and
need
we
needed
to
bring
that
to
other
departments
like
the
firewall
ruse
cluster
management.
That's
mostly
our
core
topic
where
we
came
from,
but
this
was
only
one
path
and
that's
what
we
find
out.
So
one
path
is
there.
B
We
have
a
global
service
per
api
to
provision
across
the
clouds.
That
means
I
can
talk
to
the
same
endpoint
to
provision
cluster
in
azure
or
on
the
local
data
center
in
europe
or
in
the
local
data
center
in
u.s,
and
that's
helped
us
that
we
have
the
ops,
tooling,
west
dashboards
and
also
there.
We
need
to
be
aware:
okay,
how
can
we
come
operational
when
a
data
center
is
disconnected,
so
this
was
also
an
use
case
to
think
about.
B
B
Okay,
we
solved
this
one
and
then
we
came
to
like
the
applications.
Okay,
we
find
that
the
application
has
some
other
dependencies
as
results
are
using
kafka.
They
had
some
really
classical
workload
as
well
and
they
have
super
nice
microservices
and
then
we've
like
consumed
a
lot
of
storage,
and
we
run
everything
fine
and
vsphere,
and
then
we
moved
to
cloud
and
looked
also
fine,
but
after
a
time
we're
getting
okay,
we
only
have
10
utilization
in
a
humidity's
cluster.
Why
we
had
our
cluster
auto
scaler
as
well
as
on
programmers.
B
We
have
cluster,
auto
scaling,
but
on
premise,
vms
can
mount
a
lot
of
more
volumes
as
the
machine
type.
What
we
choose
in
azure.
So
I
don't
know
if
everyone
knows
that,
but
in
azure
you
have
a
limitation
of
disk
what
you
can
mount
to
the
machine
and
that
brought
us.
We
spent
a
lot
of
money
for
nothing
because
the
cpu
memory
was
idle,
so
we
redesigned
it,
learned
it
and
said:
okay,
we
need
different
groups
of
node
pulse
and
then
the
autoscaler
was
also
like
modified
it's.
B
It
can
scale
individually
based
on
the
volume
workload
and
based
on
normal
workload
and
yeah
that's
scaled
and
up,
and
then
we
said.
Okay,
now
we
understand
it.
We
said:
okay,
let's
continuous
watch
and
that's
the
key
point
that
you
start
the
whole
thing
again.
You
cannot
implement
this
in
one
step.
If
you
want
to
do
it,
it
will
be
slow,
you
never
have
a
result
and
the
people
will
get
asking
hey.
Is
there
something
it's
like
a
black
box
and
frostwater
key
success?
Factor
hey.