►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Kubernetes On the Edge With K3s For a Smart Metering Use Case - Harry Lee, Melio AI
Speakers: Harry Lee
Running Kubernetes in the cloud is a common use case. You can make use of all the features that hyperscalers have to offer: resiliency, scalability, load-balancing etc. But what if you only have a single machine sitting on-premise that lacks consistent internet access? Should you still use Kubernetes? Our client has embarked on a journey to build a smart energy metering solution for remote industrial plants. The solution aggregates data produced by IoT measurement devices to a central point on site before sending the data to the cloud. The challenge for us, of course, is the design of this central aggregation point on site. This talk describes how we analysed the requirements to decide on using Kubernetes (K3s) as a central aggregation point on site. We will go through the why of the design and how we have also incorporated other CNCF projects (Prometheus, Helm, Cert-manager and Longhorn) into the final solution.
A
Yeah
morning,
everyone,
my
name,
is
Harry
Lee
and
today,
I'll
be
talking
about
kubernetes
on
the
edge
with
k3s
for
a
smart
metering
use
case.
So
just
a
quick
introduction
about
myself,
I'm
the
co-founder
and
CTO
at
media
AI
I've
been
a
devops
Enthusiast,
since
I
can
remember
nowadays,
I
help
organizations
adopt
devops
practices
and
Implement
Cloud
native
Technologies
by
helping
them
navigate.
The
cloud
native
landscape
I've
been
working
with
kubernetes
for
more
than
five
years
now,
I'm
also
a
certified
kubernetes
administrator
as
well
as
certified.
A
Kubernetes
security,
specialist
I've
been
on
the
cncf
Ambassador,
based
in
South
Africa
for
three
years
now.
I've
also
founded
the
cloud
native
Computing
Meetup
in
Johannesburg
South
Africa.
Since
four
years
ago,
I
currently
we're
980
members
strong
still
growing.
You
can
find
us
on
the
cncf
community
group,
I
love
connecting
with
Cloud
native
enthusiasts,
so
yeah.
If
you
would
like
to
connect
with
me,
yeah
follow
me
on
Twitter,
as
well
as
LinkedIn,
so
quick,
two
words
on
Miele
Ai
and
what
we
do.
We
are
an
mlops
consultancy.
A
We
specialize
in
ml
and
devops
practices,
helping
data
science
teams,
build
and
deploy
machine
learning.
Solutions
reliably
and
efficiently
effectively
helping
them
deliver
business
value
as
fast
as
possible.
And,
of
course,
we
do
everything
in
a
cloud
native
way,
if
you're
interested
in
finding
out
more
about
us
check
us
out
at
milner.ai,
and
we
do
regular
posts
on
our
LinkedIn
page.
So
if
you
want
to
know
more
about
email
Ops,
please
follow
us
on
LinkedIn
cool.
A
A
The
design
aspect
of
the
talk
will
be
continuously
updated
as
I
go
through
the
challenges
and
then
I'll
end
off
by
showing
you
the
final
design
that
we
have
implemented.
So
why
are
we
building
this?
In
South
Africa?
We
have
been
facing
an
energy
crisis
for
the
past
15
years,
where
our
demand
for
electricity
is
far
greater
than
the
supply,
so
our
national
power
generator
yescom
to
tackle
this
problem,
Implement,
something
that
we
endearingly
called
load
shedding.
A
So
basically,
it
is
just
scheduled
rolling
blackouts
to
the
National
Grid
when
the
load
increases.
So
we
actually
sometimes
only
get
like
a
few
days
worth
of
notice
before
they
routinely
switch
off
your
power
to
your
house,
and
recently
we
actually
only
get
a
few
hours
of
notice
before
they
switch
it
out.
So
you
can
imagine
the
economic
impact
this
has
on
businesses
within
South
Africa.
A
So
the
aim
is
to
Target
big
energy
consumers
such
as
your
office
blocks,
as
well
as
industrial
factories,
as
I
mentioned
mining
industry
specifically,
and
the
aim
is
to
use
iot
devices
to
measure
energy
usage
from
that
do
some
calculation
to
estimate
the
costs
and
then
from
that
optimize.
Electricity
usage
with
some
form
of
automation,
and
a
very
good
use
case
of
this
is
to
switch
off
appliances
at
night
when
no
one
is
using
them
or
recommend
the
best
time,
such
as
off
peak
to
run
various
heavy
machinery.
A
So
why
does
our
client
decide
to
build
this
and
not
just
buy
something
off
the
shelf,
because
iot
is
really
something
that's
been
trying
to
tested.
So
one
thing
to
note
is
that
our
clients
is
actually
in
the
hardware
manufacturing
business.
Their
expertise
is
in
building
low-cost
Energy
Measurement
devices
so
therefore
see
this
as
an
additional
service
offering
that
they
can
provide
to
their
existing
customers
to
reduce
their
energy
spending
and
from
doing
this,
they
envisioned
that
they
can
capture
the
inside
value
chain.
A
So
how
does
it
actually
work?
Let's
say
our
client's
customer
could
be
an
office
block
with
multiple
offices.
There's
appliances
in
each
office,
such
as
the
air,
conditioner
or
water
heaters,
maybe
there's
a
lab
full
of
murders.
The
idea
here
is
to
attach
iot
devices
the
designed
and
built
by
our
clients
to
be
installed
on
site
and
on
the
power
supplies
of
these
appliances.
So
the
energy
consumption
can
be
measured.
A
The
data
from
these
RT
devices
can
then
be
sent
to
an
on-premise
Gateway,
which
is
also
designed
to
build
our
clients.
The
Gateway
provides
a
secure,
six
low-pad
Network
so
for
secure
communication
between
iot
devices.
Then
the
data
is
then
transmitted
from
the
iot
devices
through
the
gateway
to
a
central
data
aggregation
platform
via
the
mqtt
protocol.
A
The
customer
is
then
able
to
monitor
the
iot
devices
on
site
and
perform
some
operational
tasks
such
as
switching
on
and
off
the
iot
devices,
as
well
as
generate
detailed
usage
reports
as
part
of
the
service
offering
the
central
data
aggregation
platform
is
also
able
to
be
connected
to
the
cloud
to
provide
Advanced
data
analytics
such
as
forecasting
the
energy
costs
for
the
current
month
and
long-term
data
storage.
But
this
is
not
a
requirement
for
the
solution
to
work,
since
some
customers
may
not
want
their
data
to
be
shared
with
the
cloud
as
they
Envision.
A
This
could
be
some
sort
of
a
security
and
government
concern
recover
this
concern.
So,
as
you
can
see,
there
are
quite
a
few
components
that
require
different
expertise
and
because
of
this,
multiple
teams
were
brought
into
this
project.
There's
a
team
focusing
on
building
the
iot
devices
team,
that's
building
the
Gateway
team,
that's
building
the
iot
platform
and
a
team.
That's
doing
their
Advanced
Data
analysis
in
data
analytics
in
the
cloud
so
for
this
talk,
I
will
be
focusing
on
the
design
and
implementation
details
of
the
central
data
aggregation
platform.
A
So
what
are
some
considerations
since
we're
building
a
solution
that
is
primarily
for
companies
and
Industrial
plants?
It's
common
for
these
industrial
plants
to
be
located
in
rural
areas.
So
there
are
several
considerations:
around
infrastructure
limitations,
first
of
all,
instruments
and
network
connectivity
between
the
sites
and
the
public
internet
is
to
be
expected.
Since
these
plants
and
factories
are
out
in
the
rural
areas,
they
may
rely
on
something
like
GSM,
3G,
4G
mobile,
cellular
towers
for
internet
connectivity,
and
we
know
that
it's
not
the
most
reliable.
A
So
this
means
our
solution
still
needs
to
work.
Even
when
we're
offline
since
we're
dependent
on
mobile
networks,
the
data
transfer
costs
must
be
considered
so
you'll
become
quite
expensive.
If
you
don't
keep
it
under
check,
this
means
they.
Wouldn't
this
means
that
we
wouldn't
be
able
to
continuously
stream
data
into
the
cloud.
A
So
our
client
also
wants
the
flexibility
of
moving
to
another
cloud
provider
in
the
future
with
minimal
friction,
so
they
don't
want
the
solution
to
be
too
coupled
with
the
cloud
providers
managed
services.
On
top
of
this,
they
also
want
us
to
use
as
many
open
source
Technologies
as
possible,
and
this
is
music
to
our
ears,
because
the
screams
Cloud
native
yay,
so
we
perceive
that
there
may
be
customers
who
don't
really
want
to
be
connected
to
the
cloud,
so
they
only
want
operational
components
of
the
solution.
A
A
So
our
solution
needs
to
ensure
that
the
RT
devices
and
applications
are
able
to
buffer
messages
when
connections
is
lost
so
that
they
can
continue
to
transmit.
When
connection
is
installed
and
to
reduce
setup
costs,
we
would
leverage
the
customer's
existing
Network
infrastructure,
which
means
we
wouldn't
have
full
control
over
it
and
during
power
outages
due
to
load
shedding.
A
Our
solution
needs
to
be
resilient
and
continue
to
work
when
the
power
does
come
back,
and
since
this
is
an
on-premise
solution
and
the
customer
may
not
have
a
server
room,
let's
say
the
servers
are
prone
to
things
like
Hardware
failures
due
to
environmental
factors,
and
this
could
be
dust
damage.
Water
damage,
Etc,
human
errors
is
also
quite
common,
such
as
someone
just
accidentally
unplugging
the
power.
A
So,
by
adding
multiple
servers
scattered
around
the
premises,
we
reduce
the
risk
of
a
single
point
of
failure,
and
this
ensures
that
the
iot
devices
can
continuously
send
data
to
the
central
data
aggregation
platform,
of
course,
for
customers
that
have
a
high
risk
appetite
and
don't
mind
the
downtime
in
when
to
save
costs,
we
should
still
provide
them
with
the
ability
just
to
install
a
single
server
cool.
So
why
kubernetes
once
we
have
some
clarity
about
the
requirements?
The
first
thing
that
comes
to
mind
is:
let's
use
kubernetes
for
this.
A
Of
course,
this
is
not
a
hammer
and
everything
is
a
nail
type
of
situation.
Kubernetes
does
bring
a
lot
of
benefits
to
what
we're
trying
to
build,
and
here
are
a
few
main
reasons
why
we've
decided
to
go
with
kubernetes
when
the
power
does
come
back
from
these
power
outages,
kubernetes
pods
will
continue
to
run
from
the
previous
state,
since
kubernetes
is
built
for
resilience,
High
availability,
and
this
needs
no
introduction.
A
A
A
So
now,
we've
decided
that
we're
going
to
use
kubernetes
with
the
vast
amount
of
kubernetes
distributions
out
there,
which
one
should
we
choose.
So
let's
take
a
step
back
and
look
at
what
we
have
to
work
with
here.
So
the
server
that
our
clients
has
procured
is
actually
Intel,
Nook
Mini
PC,
which
has
four
CPU
cores
and
eight
gigabytes
of
RAM
and
500
gigabytes
of
SD
Storage.
A
A
A
The
most
compelling
reason
for
actually
shortlisting
Cube
Edge
was
that
the
architecture
diagram
that
was
on
the
front
page
of
the
GitHub
repository
was
nearly
identical
to
the
architectural
diagram
that
we're
trying
to
build.
It's
got
both
the
cloud
and
Edge
components.
It's
got
a
built-in
mqtt
broker,
which
uses
Eclipse
mosquito,
and
it's
got
out
of
the
box
ability
to
sync
iot
device
statuses
to
the
cloud.
So
why
don't
we
choose
Cube
age?
A
So
if
you
recall
that
one
of
the
requirements
is
to
actually
allow
the
customers
to
have
an
on-premise,
only
installation
so
having
a
cloud
counterpart
doesn't
actually
work,
so
Cuba
has
a
way
to
actually
do
away
with
the
cloud
console,
but
it
proved
to
be
a
lot
difficult
than
what
it's
worth
and
since
we're
not
the
only
team.
That's
working
on
this
project.
We
have
to
consider
other
teams
that
are
also
involved
and
because
of
this,
we're
not
in
a
position
to
to
dictate
how
other
teams
build
their
solution.
A
So
the
built-in
mqtt
which
uses
Eclipse
mosquito,
is
actually
not
something
that
our
core
messaging
team
wants.
They've
already
decided
that
they're
going
to
use
rabbit
in
Q
for
this
and
the
ability
to
sync
iot
device
statuses
to
the
cloud.
Our
iot
platform
team
have
already
decided
that
they're
going
to
build
themselves
in
the
Java
application,
so
this
opinionated
architecture
on
how
to
actually
set
up
an
IRC
platform
isn't
really
something
that
we're
looking
for.
A
Since
we
have
so
many
custom
layers,
so
we
needed
a
kubernetes
distribution
that
is
flexible
enough
so
that
we
can
customize
it
to
our
needs.
So
we
ended
up
choosing
k3s
because
it
doesn't
have
an
opinionated
architecture
for
an
iot
use
case.
What
we
also
like
about
k3s
is
that
it
supports
something
like
a
static
pod
but
with
Helm
charts.
So
it's
got
a
home
chart
operator
that
automatically
install
Helm
transfer.
A
You
and
Helm
chart
is
the
the
factor
of
standard
that
we're
going
to
use
to
install
applications,
Within
kubernetes,
and
it
also
supports
a
single
node
out
of
the
box.
So
it
gives
you
the
ability
to
install
the
control
plane,
as
well
as
the
worker
node
together,
which
is
great
for
the
use
case
where
our
customer
only
wants
a
single
server
great.
So
now
we
have
decided
that
we
want
to
use
k3s.
A
What's
next,
we
have
to
install
it
on
the
server.
So
how
do
we
actually
install
kubernetes
on
bare
metal?
This
comes
to
our
first
Challenge
provisioning
kubernetes,
and
you
would
think
that
it's
actually
not
that
difficult
right,
I
mean
from
k3's
homepage.
You
just
run
a
single
line
and
the
k3s
will
be
installed.
The
real
challenge
here
is
actually
to
get
it
installed
on
the
fleet
of
these
Baymeadow
servers.
A
So,
first
we
need
an
operating
system,
we've
chosen,
Ubuntu
20.04,
and
it's
just
because
our
clients
Engineers,
are
more
comfortable
with
Ubuntu
as
a
distribution.
That's
any
distribution
that
supports
k3s
will
work
here
and
now
it's
time
to
get
this
operating
system
installed
on
the
feet
of
bare
metal
servers,
and
we
didn't
really
reinvent
the
wheel
here.
We've
decided
to
just
go
with
the
normal
pixie
near
booting
approach,
so
this
is
a
way
to
just
automatically
install
operating
systems
on
servers
over
a
private
Network.
A
Next,
we
need
to
do
some
configuration
management
right
because
now
we
have
to
install
k3s
for
this
we've
chosen
ansible
as
our
configuration
management
tool
it's
just
because
our
team
is
more
familiar
with
it,
but
any
tool
such
as
Chef
or
puppet,
will
work.
So
we've
installed
ansible
onto
the
provisional
server,
which
already
is
configured
and
attached
to
the
bare
mental
service
and
yeah.
A
We
just
ran
the
table
from
there,
so
the
tasks
that
we
ran
listed
here,
updating
and
installing
operating
system
packages
such
as
enabling
app
armor
doing
some
security
settings
installing
extra
packages
such
as
CA
certificates,
of
course,
installing
k3s
itself.
For
this
we're
actually
leveraging
very
popular
open
source
projects
by
pirate
Labs.
It's
fantastic.
It's
got
sensible
out
of
the
box
defaults
and
you
can
actually
get
a
k3s
up
and
running
with
minimal
configuration.
A
So
once
we
have
installed
k3s,
we
also
installed
some
platform
services,
and
this
includes
services
that
will
support
logging
and
monitoring
such
as
Prometheus
grafana,
as
well
as
fluent
bits
and
then
last
but
not
least,
we
needed
to
generate
clients
as
well
as
customer
specific
configurations
such
as
the
you
know.
The
device
list
for
the
iot
devices,
as
well
as
some
IP
addresses
that
we
need
to
pre-configure,
and
then
we
copy
them
onto
these
bare
metal
servers
great.
A
So,
let's
look
at
what
is
at
the
end
of
the
stage,
so
we've
got
a
kubernetes
cluster.
We've
installed
some
Services,
we
broadly
categorize
all
of
the
application
and
services
into
two
categories,
so
the
workload
services
and
the
platform
services.
The
workload
Services
consist
of
custom
applications
that
our
team
has
built
that
serves
specific
functionalities.
So
over
there
we've
got
the
iot
platform
and
the
RT
platform
is
the
brain
of
the
entire
operation.
It
allows
end
users
to
interact
with
iot
devices
view
the
current
energy
usage
Etc.
A
The
message
service
is
responsible
for
data
scrubbing
compression
aggregation
and
routing.
The
automation
service
is
actually
just
a
group
of
cron
jobs
and
jobs
that
is
triggered
by
the
other
services,
and
last
we've
got
the
manager
service,
which
is
used
to
bootstrap
the
entire
cluster,
as
well
as
to
ensure
that
the
server
is
actually
in
healthy
States.
A
So
the
platform
Services
includes
peripheral
services
such
as
Prometheus
and
some
of
the
operators
that
we
have
built.
The
operator's
job
is
just
to
make
sure
that
the
services
inside
the
workload
category
is
running
in
healthy
States,
they're,
also
optional
platform
services
such
as
the
efk
stack,
which
is
a
elasticsearch
fluent
bid
and
Cabana,
as
well
as
other
dashboards
such
as
grafana
and
the
kubernetes
dashboard.
The
reason
these
are
optional
is
because
of
the
limited
Computing
resource
we
have
on
our
hands
so
for
customers
that
are
willing
to
purchase
a
much
bigger
server.
A
So
if
you
recall,
we
are
using
eight
gigabytes
of
RAM,
but
if
they
offer
the
16
gigabytes
of
RAM,
we
would
then
optionally
install
these
for
them
great.
Now
we
have
all
of
the
components,
let's
see
how
we
should
deal
with
the
Ingress
traffic
coming
into
our
cluster,
so
this
comes
to
our
Second
Challenge
networking.
A
So,
let's
revisit
how
we
intend
the
data
to
flow
from
the
iot
devices
into
our
kubernetes
cluster.
The
iot
devices
send
data
to
the
Gateway
by
the
six
low-pad
Network,
the
Gateway,
which
is
connected
to
the
internal
Network,
then
Aggregates,
and
sends
the
data
to
the
rapidmq
message
broker
within
our
cluster.
A
So
let's
talk
about
how
we
would
do
service
Discovery.
The
straightforward
answer
here
is:
let's
just
assign
a
static
IP
to
this
Bay
middle
server
right
so
we'll
go
to
the
router,
we'll
find
an
IP
address,
that's
not
part
of
the
DHCP
server
and
then
we'll
reserve
it
by
its
Mac
address.
So
that's
great
and
all.
But
what
if
we
have
a
cluster
that
has
multiple
nodes
now,
do
we
do
the
same
thing
on
the
router?
Do
we
go
into
router
and
then
configure
a
static
IP
address
for
each
node?
A
So
it's
fine
for
a
three
node
cluster,
but
what
if
we
have
a
five
minute
cluster
or
a
seven
node
cluster,
so
that
becomes
quite
hectic
as
now,
you
have
to
send
this
list
back
to
the
Gateway
or
the
end
user
to
make
sure
that
they
can
do
the
service
discovery,
and
this
also
begs
another
question:
how
would
failover
work
in
this
case
all
right?
What
if
we
add
a
load
balancer
in
front
of
the
cluster,
because
this
is
what
we're
doing
Cloud
right?
A
You
add
a
load
balancer
that
the
cloud
provider
provides
open
up
some
node
ports
point
the
load,
balance
to
those
node
ports
and
off
you
go,
but
because
we're
working
on
on-premise
edge.
This
actually
doesn't
really
work
that
well,
because
now
we
have
to
add
an
extra
server
which
we
can't
guarantee
High
availability,
and
this
is
actually
a
single
point
of
failure,
and
now
we
also
have
to
implement
custom
Logic
for
service
discovery
on
this
load.
Balancer.
It's
just
too
much
work,
so
we
came
across
something
called
the
metal
lb.
A
So,
in
order
for
me
to
lb
to
work,
we
basically
bringing
the
external
load
balances
logic
into
our
cluster.
So,
first
of
all,
we
need
to
make
sure
that
meter
RP
has
an
IP
address
pool,
so
we
need
to
find
an
IP
address
range
that
is
not
part
of
the
DHCP.
Server
can
be
handed
out
to
the
services
that
require
a
load.
A
Balancer
type
service
and
metal
RB
actually
allows
you
to
operate
in
two
modes,
the
bgp
mode,
as
well
as
layer
2
mode,
which
is
the
ARP,
which
is
the
ARP
mode,
and
we've
opted
for
the
layer
2
mode,
because
ARP
is
kind
of
like
a
the
factor
standard
for
any
ethernet
Network
and
remember.
One
of
the
requirements
was
that
we
couldn't
make
any
assumptions
on
the
customers
on-premise
infrastructure.
A
So
this
may
seem
like
a
lot
of
jargon
right.
Let's
see
how
it
actually
works.
So
imagine
we
have
a
three
node
k3s
cluster,
all
right,
node,
one
node
to
node
three
and
we've
got
a
router,
that's
running
a
DHCP
server
within
the
range
of
102.168.1.10,
all
the
way
to
Dot
200.
great,
and
we
know
this
DHCP
range
beforehand
and
we
decided
to
find
an
IP
address.
That's
outside
of
this
range.
Let's
say
we
use
0.201,
so
we
now
configure
meta
RB
by
ansible
to
use.201
as
an
IP
address
it's
going
to
grab.
A
So
when
the
nodes
are
connected,
each
of
the
nodes
gets
automatically
assigned
an
IP
address
as
part
of
the
CCP
range.
Let's
say:
Node
1
gets.11,
no
2.12,
node
3
gets.13
upon
starting
up
metal
RB.
Since
we've
configured
it
to
grab
201.
It's
going
to
do
so.
It's
going
to
grab.201
to
be
the
IP
address
and
assign
it
to
node
1..
So
now
we
can
let
all
of
the
other
external
Services
know
that
you
can
use
192.168.1.201
to
access
our
cluster
cool.
A
So
when
excellent
request
comes
into
the
router,
the
router
sees
it
sends
it
to
no
one.
So
what
happens
when
no
one
fails
right,
because
metal
lb
is
actually
installed
as
a
demon
sets
on
all
of
the
nodes,
it
see
that
no
one
has
failed
and
then
it
will
actually
move
that
IP
address
to
node
two.
So
now
we're
still
going
to
stick
with
that
201.
A
We
don't
have
to
worry
about
telling
the
external
Services
about
this
changing
IP
address
great
now,
we've
decided
to
use
meter
lb
as
something
to
control
all
of
our
Ingress
traffic
k3s
by
default
actually
comes
with
its
own
service
load
balancer,
which
is
called
the
cclb.
That's
something
that
we
have
to
disable,
as
well
as
the
Ingress
controller
that
comes
defaults
with
it,
which
is
the
traffic
Ingress
controller.
We
also
have
the
disable
it
in
order
for
meta
RV
to
work
cool.
So
now
we've
addressed
installing
k3s
onto
this
Bay
Mental
server.
A
We've
also
addressed
how
we
manage
traffic
coming
into
our
server,
so
this
actually
works.
If
we
have
stateless
applications
right
because
when
no
one
fails,
we
move
the
Pod
to
node.
Two
life
goes
on.
Everything
is
great,
but
with
this
work,
if
we're
failing
over
stateful
applications
so
for
Safer
applications
with
k3s
by
default,
it
comes
with
raunch's
own
local
path
provisioner,
but
this
only
gives
you
the
ability
to
create
persistent
volumes
on
the
current
node
that
the
part
is
provisioned.
A
So
what
happens?
If
that
node
dies
right,
the
Pod
can
move,
but
the
persistent
volume
wouldn't
because
personal
volume
is
actually
allocated
onto
the
first
node.
So
we
need
a
way
to
also
replicate
or
move
that
persistent
volume
to
the
second
node,
and
this
is
something
that
we
need
distributed.
Persistent
storage
solution
for,
so
we
started
looking
for
a
distributed,
persistent
storage
solution.
A
The
first
thing
that
we
came
across
was
Rook
just
because
we've
used
Rook
a
few
years
back,
it's
great
it's
based
on
SEF,
which
is
a
highly
scalable,
distributed,
storage,
solution
for
Block
storage,
object,
storage
and
a
shared
file
system.
It's
feature
Rich
and
built
for
resilience
and
scale,
but
it's
actually
a
little
bit
too
complex
for
our
current
use
case,
because,
especially
for
a
single
node
setup,
you
actually
have
a
lot
of
overhead
just
to
run
Rook.
A
So
we
set
out
so
also
look
for
an
alternative,
persistent
storage
solution,
Optus
more
Googling
research
and
testing.
We
shortlisted
yet
another
true
CNC
of
project
Longhorn
and
open
EBS.
Again,
both
are
great
projects
with
grades,
community
support
and
a
very
good
fit
for
our
use
case,
because
both
are
easy
to
install
both
supports
PB
replication,
which
is
what
we
want.
We
wanted
to
make
sure
that
the
PV
actually
is
replicated
to
another
node
when
the
failover
happens.
A
Both
supports
incremental
snapshots,
great
for
backup
and
restoration,
but
in
the
end,
through
our
comparison,
we
still
select
a
long
horn.
Why?
It's
just
because
the
only
most
compelling
reason
for
that
is
because
it's
under
the
Rancher
umbrella,
so
K3
is
also
a
Rancher,
backed
project
officially
supported.
So
we
perceive
that
future
upgrades
and
compatibility
will
be
a
lot
smoother
if
we
go
with
Longhorn
great.
So
now
we
have
a
way
to
solve
our
state
for
application
available.
So
now,
we've
added
Longhorn
into
our
platform
services.
A
A
And
this
we
use
something
called
search
manager,
but
before
we
get
into
that,
let's
talk
about
how
the
traffic
flows
through
our
cluster
again.
So
we
know
that
our
cluster
currently
hosts
a
few
web
services
such
as
dashboards
and
the
iot
platform
itself,
as
well
as
the
message
broker,
which
uses
rabbit
in
queue.
So
the
RC
devices,
when
communicating
with
the
Gateway,
uses
the
six
load
pair
Network,
that's
fine.
The
Gateway
and
rot
team
they've
got
their
own
solution
to
deal
with
secure
communication,
so
that's
outside
of
our
jurisdiction.
A
A
So
the
reason
why
we
decided
to
go
with
cert
manager,
it's
actually
there's
no
other
tools
out
there,
that's
comparable
with
cert
manager
inside
the
cncf
landscape,
so
it's
kind
of
de
facto
tool
for
any
sort
of
certificate
management
within
kubernetes,
and
we
have
great
successes
with
it
in
the
past.
So
we
didn't
look
too
deep
into
other
alternatives.
A
So
how
do
you
manage
the
certificate
management?
Sorry,
how
do
we
manage
this
issuing
of
certificates?
First
of
all,
we
needed
to
self-sign
a
root
CA,
so
we
created,
and
then
we
make
sure
that
we
store
it
in
a
very
secure
and
secret
location.
Then,
from
that
we
create
an
intermediate
CA
per
customer
from
that
we
created
another
intermediate
CA,
but
on
a
per
site
or
per
Factory
level,
and
then
we
move
that
into
media
CA
into
our
cluster
for
sort
manager
to
use
as
a
cluster
issue
or
custom
resource.
A
Great
one
thing
to
note
is
that
the
creation
of
the
root
CA,
as
well
as
the
issuing
of
the
intermediate
Cas,
are
handled
outside
of
the
cluster
handle
on
organization
level
and
the
tool
we
use
for
that
is
hashico
bolts,
but
we
don't
have
to
get
into
that.
The
most
important
thing
is
the
final
intermediate
CA
gets
into
our
cluster
and
being
used
by
the
certificate
manager,
so
the
certificate
manager
upon
receiving
the
intermediate
CA
will
do
its
thing
and
start
issuing
leaflet
certificates
so
for
https.
A
The
ones
that
we
need
is,
for
example,
the
rabbit
mq
management
portal.
There
needs
to
be
on
https
because
that's
how
the
portal?
That's,
how
the
end
user
will
access
rambling,
Cube,
kubernetes,
dashboard,
Prometheus,
dashboards,
grafona
dashboards
Etc,
will
all
use
leaflet
certificates,
that's
issued
by
search
manager.
Of
course,
our
custom
workloads
also
needs
to
be
protected
by
https
on
the
mqtts
friends,
we
use
something
called
Mutual
TLS.
So
all
of
the
clients-
that's
actually
communicating
with
rabbitmq,
including
the
gateways,
needs
to
have
a
client
certificate
and
soap
manager
actually
allows
you
to
do
that.
A
A
So
in
conclusion,
we've
just
talked
about
the
product.
What
we
actually
try
to
build
and
the
considerations
for
an
on-premise
and
Edge
use
case
is
what
we've
highlighted.
We've
also
talked
about
how
we've
implemented
the
central
data
aggregation
platform
using
kubernetes
and
a
few
cncf
projects,
k3s
Longhorn
and
serve
manager.
Metal
RB
is
not
a
cncf
project.
Yet
we've
also
outlined
some
of
the
design
challenges
along
the
way,
which
is
the
four
main
challenges
on
provisioning:
networking,
High
availability
and
certificate
Management
in
a
on-premise
edge
use
case
so
yeah.
A
B
A
Yeah
so
because
some
customers
will
have
a
gapped
insulation
so
over
the
updates
is
kind
of
like
not
possible.
So
what
we
do
is
we
actually
dispatch
a
technician
who
brings
the
ansible
script
to
the
site
and
they
run
ansible
script
to
updates
or
provisioned
as
iot
devices,
and
this
includes
updates,
as
well
as
adding
and
removing
devices
yeah
cool.
Thank
you.
A
Yeah,
so
we
didn't
so
basically
all
the
services
just
have
a
load
balancer
type
and
then
that
just
automatically
messed
up
you
will
see
it
and
automatically
assigns
the
same
IP
address
to
all
the
load
balances.
So
the
thing
is
we
just
have
to
make
sure
that
the
ports
don't
conflict
yeah,
because
we're
operating
on
a
sort
of
like
a
lower
layer,
not
non-layer,
7
or
anything.
So
we
have
to
just
make
sure
that
the
ports
are
in
Conflict
yeah.
C
Yeah,
just
sort
of
curiosity
really
when
you
sort
of
mentioned
cncf
projects
or
other
cncf
projects
and
was
a
reason
why
you
sort
of
went
down
the
kubernetes
road
and
not
something
like
Nats,
which
is
a
sort
of
high
performance
sort
of
messaging
system.
That's
sort
of
quite
suited
for
something
like
this.
So
just
just
curious.
A
So
the
reason
why
we're
going
for
with
kubernetes
is
actually
because
we
have
a
lot
of
teams,
that's
working
on
those
projects
right
and
then
one
thing
that
we
said
upfronts
is
that
everyone
should
just
use
containers
just
because
it's
a
lot
more
portable,
the
people,
who's,
actually
provisioning
and
installing
it
don't
have
to
understand
like
dependencies,
because
you
just
wrap
it
up
in
a
Docker
image.
You
can
deploy
it.
A
So
we
just
wanted
to
say
something
such
as
containers
and
we
need
something
to
orchestrate
it,
and
that
was
the
only
reason
why
we
use
kubernetes.
That's
the
most
compelling
reason
and
then,
of
course,
there's
also
other
bells
and
whistles
that
comes
with
it,
such
as
high
availability,
resiliency,
Etc
yeah.
But
the
main
reason
was
the
standardization
that
we
needed.
Yeah.
E
A
Yes,
yes,
okay,
so
there's
actually
only
a
single
kubernetes
cluster
in
the
cloud
that
all
of
the
customers,
as
well
as
the
sites
connect
to
so
it's
kind
of
like
a
hub
and
spoke
sort
of
setup.
So
just
one
big
cluster
in
the
cloud
and
a
multiple
small
cluster
all
over
the
place
on
edge
and
the
reason
just
for
doing
that
is
because
we
just
we
didn't
want
to
have
a
maintenance
overhead
of
maintaining
a
lot
of
kubernetes
clusters
in
the
cloud.
So
we
just
have
one
team:
that's
managing
that
and
that's
a
setup.