►
From YouTube: Feeding Realworld Data into Prometheus Microservice Apps on Kubernetes - Burkhard Noltensmeier
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Feeding Realworld Data into Prometheus Microservice Applications on Kubernetes - Burkhard Noltensmeier, teuto.net (Intermediate Skill Level)
Burkhard Noltensmeier shares his experience developing a Microservice data aquisition platform on Kubernetes. The Platform consists of Edge Devices with to import Machine Data from factory floor or building control. The Author lays out the Microservice components (mqtt, prometheus, grafana, cassanda, tensorflow) and the how and why there are stitched together. Following a prototype for feeding Prometheus Data into recurrent neural networks for alarming and load prediction.
About Burkhard
1992 Fernuniversität Hagen Computer Sience 1994 founding teuto.net 2012 start Openstack within teuto.net 2015 Talk: Oenstack SDN principles at Openstack Dach Tag Berlin 2016 Talk: Openstack und Kubernetes at Deutsche Openstack Tage Cologne 2016 start Building Data aquisition Platform n Kubernetes 2017 Talk: Understanding Kubernetes at Cloud Export Frankfurt
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Well,
my
name
is
Boca
norton's
Maya
I'm,
the
founder
and
CEO
of
thought
in
it.
So
this
is
my
first
talk
in
English
and
a
little
bit
nervous,
but
yeah.
Let's
begin
with
so
we've
our
hosting
company
from
Germany
we
do
hosting
since
20
years
ago,
and
since
five
years
we
are
operating
hosting
OpenStack
and
since
two
years
we
were
working
with
kubernetes
and
yeah.
I
am
I'm
from
the
region,
stressful
and
Lipper,
in
Germany,
in
Bielefeld
and
in
our
region.
There's
there
are
very
strong
companies
that
are
in
the
industrial
control.
A
A
Cloud
native
into
the
industrial
domain
and
now
region
is
also
university,
Fraunhofer
Institute
and
they
run
a
smart
factory
for
for
integrating
new
concepts
and
we
reached
out
to
them
and
applied
for
research
project
and
his
view
on
the
factory
floor.
So
they
integrate
all
kinds
of
machines
and
technologies
and
yeah.
We
set
up
a
research
project
with
them
to
monitor
data
from
the
industrial
domain,
and
this
leads
to
our
mission
statement
getting
a
sense.
What
is
going
on
and
aggregate
information
with
context
in
a
DC
boot
world,
so
yeah.
A
This
is
a
broad
statement
and
it's
difficult
to
start
with
and
yeah
we
choose
the
the
bottom-up
approach
and
yeah
the
world
is
is
big
and
they're
a
whole
lot
of
protocols,
and
this
applies
also
to
the
industrial
domain.
There
are
scattered
locations,
multiple
factories,
for
instance.
There
are
multitude
of
protocols
from
several
vendors
and
over
the
years.
They
are
also
changing.
A
There's
missing
context:
you
might
have
only
special
data,
but
without
an
annotation
which
machine
it
is
and
everything
together
it
is
an
overwhelming
information
to
get
along
with
and
yeah.
For
instance,
I
would
characterize
maybe
to
product
to
two
kinds
of
protocol.
One
is
some
the
the
binary
protocol,
which
has
only
binary
data,
but
no
formal
meaning.
So
you
have
to
add
this
meaning.
You
have
to
write
custom
exporters
for
them
to
use
Prometheus
speech
and
yeah
you
have
to
describe
okay
rig
is
the
one
is
the
temperature
rig
is
the
two?
A
Is
the
motor
speed
or
something-
and
this
applies
to
mod
pose,
which
is
a
very
old
protocol.
Prophy
net
is
new
protocol.
They
have
got
a
formal
description
for
the
data
format,
but
unfortunately
most
vendors
when
they
give
away
a
machine.
They
don't
to
give
away
this
form
of
specification,
and
you
end
up
doing
reverse
engineering
these
protocols,
and
so
this
is
one
thing
to
get
these
data
at
the
proximity
of
the
machine.
So
we've
got
one
example.
A
This
is
an
electricity
meter
that
we
integrate
it
and
is
it
has
got
a
serial
for
the
call
is
that
at
least
some
context
you
can
get
the
serial
number
out
of
these
machine
to
to
identify
it.
So
this
is
very
good
and
and
on,
and
also
the
the
data
about
what
it's,
what
you
need
and
on
the
other
end
there
are
protocols
like
OPC,
UA
UA
stands
for
universal
architecture.
A
A
So
one
example
is
like
a
control
and
industrial
controller
like
this,
for
instance,
that
we
use
in
our
climatization
in
our
server
room
so
and
yeah.
We,
we
needed
edge
devices
for
protocol
conversion,
so
these
busses,
like
these
binary
buses
like
Matos
and
are
mostly
unreachable
in
the
proximity
of
the
machine.
A
So
we
have
to
extend
our
reach
into
the
factory
floor
and
this
is
what
we
choose
for
so
there's
these
Raspberry
Pi
in
desert
grade
and
as
you
see
here
from
the
inside,
there
is
a
dim
module
with
the
Raspberry
Pi
in
it,
and
you
can
apply
this
directly
at
the
machine
and
we
need
this
as
a
foundation
for
our
integration.
So
we
decided
to
yeah.
A
So
I
think-
and
this
will
also
be
ever
more
important
in
the
kubernetes
space,
because
servers
are
on
the
rise
also,
and
so
this
will
be
more
important
in
the
future
to
have
this
kind
of
architecture
integration,
so
we're
still
struggling
with
remote
management.
We've
got
a
whole
lot
of
arm
machines
out
there
and
we
have
to
manage
them
and
we
are
not
decided
which
approach
to
to
go
so.
There's
some
open
source
software
like
al
IOT,
which
runs
which
you
can
give
containers
to
run
remotely.
Maybe
we
will
go
to
the
communities.
A
Federation
wrote
and
put
up
a
small
communities
class
on
each
Raspberry
Pi
to
leverage
the
kubernetes
Federation
API
in
the
future,
and
then
we
would
be
able
to
continuous
integrate
in
continuous,
deploy
up
to
the
factory
floor
so
yeah.
Now
we
have
got
these
laid
out
this
foundation
and
then
how
to
go
further
from
that.
So
we
took
Club
native
for
the
for
the
rescue
and
taking
Prometheus
as
a
building
block
and
build
micro
services
on
top
with
Prometheus
and
also
adding
a
control
plane.
So
I
will
further
dive
into
the
contact
concept.
Yeah.
A
Okay,
a
few
words
for
Prometheus
by
design,
I
would
say:
we've
got
a
very
powerful
language.
It
has
got
labels
for
adding
context.
It
has
got
a
very
efficient
design.
It
has
got
conservation
rules
and
alarm
rules
that
can
be
executed
along
in
in
real-time,
but
also
by
design.
To
me,
this
has
no
event:
processing,
no
sub
second
resolution,
no
multi-tenancy
no
receiving
of
data,
so
everything
is
scraped,
but
you
cannot
send
it
to
Prometheus
so
the
first
two
event,
processing
and
sub
second
resolution.
A
We
can
give
away
that,
but
we
need
it
multi-tenancy
in
our
platform
and
we
also
needed
the
the
receiving
site
to
overcome,
not
gateway
problems
so
and
for
the
transport
we
choose
to
go
with
with
a
Prometheus
format
with
open
metrics.
You
may
know
this
from
our
already,
so
we've
got
the
metrics
on
the
left
side
and
then
the
labels
and
then
the
actual
amount
like
1027
and
also
timestamp
with,
is
seldomly
used.
A
So-
and
this
is
how
it
looks
now-
we
have
got
our
revolution
pie
with
open
metrics
with
has
got
a
small
container
on
it
with
a
titin
conversion
program
which
manually
has
to
be
adjusted
for
for
each
machine
and
then
we
will
send
it
to
MQTT
broker.
These
amputee
broker
already
runs
in
our
community's
cloud,
and
then
we
developed
a
little
bit
of
loop
load
like
amputee,
to
chrome,
to
get
the
amputee
theater
to
subscribe
on
this
channel
and
pretend
a
scrape
endpoint
for
Prometheus
to
get
the
data
into
Prometheus.
A
So
this
might
not
be
in
the
most
elegant
way,
but
this
is
a
solution
that
we
come
up
with
and,
as
you
can
see,
the
Prometheus
pulse
the
data
out
of
the
amputee
prom
gateway,
and
we
also
set
up
a
second
channel
to
put
these
raw
data
into
Cassandra.
So
if
we
need
some
special
service
or
as
a
special
report,
we
can
run
this
in
Cassandra.
A
So
this
our
data,
plane,
microservices
and
then
you've
got
a
control
plane
to
to
stall
the
configuration
data,
so
the
the
MQTT
channels
and
also
the
the
users
for
the
the
broker
and
this
kind
of
stuff
and
yeah.
The
main
thing
that
we
developed
is
a
bunch
of
rest,
api
s--
that
is
running
these
control
planes
in
synchronizing
the
configuration
to
the
to
each
parts
and
yeah.
A
On
the
left
side,
we
have
got
these
amputee
broker,
which
do
kind
of
basic
authentication
about
HDDs
and
on
the
consuming
side
we
have
got
yeah
you
beta
and
graph
on
a--,
and
so
there's
no
multi-tenancy
in
Prometheus
and
also
in
graph
on
ax
and
Jupiter.
So
we
run
a
bundle
of
Prometheus
graph,
Anna
and
Jupiter
for
each
tenant
and
then
we
to
have
single
sign-on.
We
implemented
up
my
D
in
front
of
these
with
a
reverse
proxy
server,
so
each
tenant
can
use
its.
A
There
are
services
with
the
same
password
and
yeah
in
the
front.
We've
got
a
communities,
communities,
ingress
server
and
each
tenant
configuration
got
has
got
its
own
host
name
under
which
directories.
Then
these
services
will
be
located
and
yeah
the
service
can
be
consumed
either
with
a
browser,
and
we
also
developed
a
command-line
client
with
golang.
A
You
can
see
the
the
syndics
of
the
command-line
client,
so
we
mainly
develop
these
command
line
kind
client
to
be
flexible
and
to
also
be
on
the
windows
side.
So
with
the
golang,
we
can
easily
compile
a
Linux
and
windows
binary
and
we
don't
have
to
hesitate-
and
we
can
give
this
to
customers
for
self
administration,
so
they
can
administer,
they
are
accounts
and
they
are
certain
machine
configuration.
A
So
this
leads
to
the
workflow,
then,
on
the
factory
floor,
so
first,
the
the
the
tenant
has
to
use
an
API
user
with
a
sensor
path
of
the
Kuban.
It
is
to
sorry
with
mosquito
path
to
deliver
the
data.
Then
he
has
to
configure
his
edge
device
with
a
certain
configuration
for
importing
the
machines
like
register
mapping
connected
with
the
internet
and
then
is
able
to
configure
the
graph
on
a
dashboard.
So
this
is
a
little
bit
of
a
manual
configuration
and
I
think
yeah.
A
What
we
heard
in
the
last
talk
to
configure
the
the
kevanna
dashboards
and
the
police's
automatically
will
also
wrote.
I
think
that
we
would
like
to
go
on
this
place
to
automate
this
process
further
and
to
have
real,
continuous
integration
on
that.
So
this
is
an
example
of
our
server
room.
This
is
a
graph
on
our
dashboard
screen
chief
ofagra
on
a
dashboard,
and
we
use
the
mayor
made
for
delivering
this
diagram
in
the
upper
part.
So
this
diagram
is
generated
programmatically
with
with
Mamet,
so
you
don't
have
to
draw
it
somehow
and
yeah.
A
You
see,
there's
on
the
left
side.
There's
the
temperature
from
the
outside,
so
we've
got
free
cooling
and
then
the
temperature
goes
inside
and
then
we
mix
the
makes
the
air
yeah
prevent
too
cold
air
from
going
inside,
and
then
we've
got
an
air-conditioned
climber
and
logger,
and
this
air-conditioned
can
also
be
monitored
and
yeah.
As
a
result,
we
can
see
the
the
temperature
and
yeah
we
also
import
open
weather
data
to
have
a
forecast
to
see
what
we
might
might
need
and
yeah
you
may
know
it
with
Gravano.
A
A
So
now
we
did
a
prototype
with
tensorflow.
This
time
we
don't
use
a
real
machine.
We
use
vertel
machines
over
a
machine
out
of
our
cluster,
and
these
other
machines
also
have
certain
characteristics
and
yeah.
We
took
four
dimensions
from
a
vertol
machine
like
CPU
time
like
written
bytes
on
the
disk
like
Network,
bytes,
receiving
and
yeah.
We
took
these
data
and
splitting
it
into
a
training
set
and
a
validation
set,
and
then
we
had
a
student
doing
his
master
thesis.
He
set
up
an
LT,
SM,
Network
and
trained
it
and
yeah.
A
So-
and
this
is
what
it
looks
like
so
the
gray
curve
is
the
the
actual
curve
and
the
red
curve
is
the
predicted
curve
you
might
notice.
There
are
some
gaps
in
the
red
curve.
That's
because
we
did
two
training
sets
for
five
minutes
and
each
five-minute
period
is
yeah
trained
independently.
So
unfortunate
doesn't
mix
very
good
yeah.
A
A
So
yeah
we
thought
other
training
is
get
valid
predictions
it's
very
very
hard,
but
the
other
things
that
we
accomplished
long.
So
this
cloud
native,
Johnny
and
research
project
took
us
a
cause
over
one
year
and
but
we
gained
confidence
in
the
team
with
that.
The
micro
servers
are
doable
and
we
discovered
that
the
CI
testing
of
Microsoft's
is
very
hard.
A
So
that's
what
we've
learned
so
you've
got
dependencies
between
the
micro
services
and
when
you
want
to
test
the
one
micro
service,
you
have
to
mock
up
the
three
other
micro
service
to
be
able
to
test
this
one
micro
service.
So
this
is
hard
and
yeah
see.
I,
built
of
x86
and
arm
contain,
and
one
run
is
very
good.
In
my
opinion,
and
yes,
I
thought
in
terms
of
flawed
prototype,
doing
a
prototype
is
hard,
but
it's
doable,
but
getting
valid
prediction
is
very
very
hard.
So
we
decided
to
not
go
this.
A
This
road
for
the
short
run.
Yeah
now
I
come
to
the
end.
So
I've
got
some
references
of
what
we're
using
so
there's
an
exporter
for
the
exam
meter
available
that
we
use
also
as
a
foundation.
There's
this
glue
open,
ID
connect
server
that
we
use
for
our
implementation.
So
we
try
to
decks
from
cause.
We
won't
weren't
able
to
get
decks
running.
We
also
used
key
cloak
or
tested
key
clock
from
reddit.