►
From YouTube: Observability for Air Conditioners: An IoT Story with Talos - William Gathright, Talos IoT
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Observability for Air Conditioners: An IoT Story with Talos - William Gathright, Talos IoT
The internet of things (IoT) is all about connecting real-world objects to get timeseries data out of them. And what better thing to do with timeseries data than feed it to Prometheus? Hear the story of how start-up company Talos IoT is applying Prometheus and the rest of the observability stack to critical equipment like air conditioners. The talk covers why Prometheus is a natural fit for IoT data before going into some of the challenges that happen when we stop monitoring Kubernetes Pods and start getting real!
A
A
A
So
that
being
the
sort
of
state
of
the
world,
that's
the
state
of
the
market
right,
there's,
actually
a
lot
of
value
in
just
making
that
data
available
and
then
performing
some
basic
services
on
it
right
I'll
give
you
maybe
an
example
would
help
here
so
I'll.
Give
you
an
example
from
my
own
work,
so
telos
uses
iot
kind
of
technologies
for
automatic
fault,
detection
and
diagnostics
of
mission,
critical
equipment
and
we're
starting
with
hvacs
right
heating
and
air
conditioning
units.
A
What
we
do
is
we
start
off
with
a
small
sensor
board,
so
one
of
these
gets
installed
into
every
hvac
cabinet,
and
this
lets
us
send
data
from
a
few
different
sensors.
So
a
few
different
electrical
sensors
and
temperature
sensors
up
to
the
cloud,
and
you
can
tell
a
surprising
amount
about
the
health
of
an
hvac
unit
just
from
a
few
temperature
and
and
electrical
current
readings
right.
So
once
this
data
is
in
the
cloud
we
can
analyze
it
against
a
few
different
sets
of
rules.
A
One
set
of
rules
was
handcrafted
by
hvac
experts
and
then
another
one
was
learned
based
on
the
performance.
The
past
performance
of
this.
You
know
that
particular
unit
and
then
similar
units,
so
your
hvac
in
a
commercial
or
industrial
building.
It's
really
a
critical
piece
of
mechanical
machinery
right
climate
control
is
just
absolutely
crucial
to
many
industrial
processes
and
a
good.
You
know
many
buildings,
especially
here
in
you
know
here
in
the
west.
A
Here
in
america
they
become
completely
uninhabitable
when
the
hvac
goes
down
right,
I
mean
one
of
the
one
of
the
hallmarks
of
so-called
class
a
office
space.
Is
that
you
can't
open
the
windows.
You
can't
actually
get
ventilation
into
the
building
by
design.
This
is
a
selling
feature,
so
you
you
gotta,
leave
if
the
hvac
dies.
A
At
the
same
time,
your
hvac
is
really
hard
to
monitor.
Right,
I
mean,
if
you
think,
about
an
hvac,
it's
typically
up
on
the
roof
of
the
building,
or
it's
some
place
very
difficult
to
get
to
maybe
tucked
behind
the
building,
and
it
can
take
a
trained
technician
to
really
be
able
to
see
if
the
unit
is
healthy
or
not
right.
Your
sort
of
typical
person
doesn't
doesn't
quite
know
enough
about
the
refrigeration
cycle
to
diagnose
one
of
these
things.
A
A
Because
of
this,
because
of
you
know
this
sort
of
this
this
this
trade-off.
There's
many
companies
that
just
choose
to
run
their
equipment
to
failure
and
yeah.
You
know
they
have
to
deal
with
the
occasional
emergency.
Sometimes
it
dies
when
you
really
wish
it
hadn't,
but
that's
expensive
right,
so
the
current
sort
of
state
of
the
art,
the
current
best
practice
in
the
industry,
is
to
perform
quarterly
preventive
maintenance.
So
this
is
where
a
technician
goes
to
a
unit
and
lays
on
hands.
A
You
know
really
whether
the
unit
needs
it
or
not,
and
that's
that's
the
cycle
that
we're
trying
to
break
here
right
so
by
using
you
know
some
of
these
iot
devices.
We
can
perform
preventive
maintenance,
essentially
once
a
minute,
instead
of
once
a
quarter
and
that's
going
to
make
this
equipment
a
lot
more
reliable
and
it's
also
going
to
make
technicians
a
lot
more
productive
right,
because
then
they
can
only
go
have
to
service
the
units
that
actually
need
their
help.
A
But
you
know
maybe
of
interest
to
this
crowd.
The
most
is.
We
can
actually
do
one
step
better
than
that.
Can't
we
we
can.
We
can
take
this
new,
very
rich
data
set
and
track
the
time
series
data
that
comes
off
of
it
now
time.
Series
data
is
so
much
more
useful
than
maybe
static
data
for
assessing
the
health
of
any
machine,
because
you
can
pick
out.
You
can
pick
out
trends
right.
A
So
an
hvac
has
two
very
important.
You
know
we'll
call
them
pipes
or
lines
that
carry
the
refrigerant.
These
are
called
the
suction
liquid
line
and
if
you
come
up
to
me
or
any
technician-
and
you
tell
me
that
the
difference
in
temperature
between
the
suction
line
and
liquid
line
is
25
degrees,
fahrenheit,
then
I
mean
I
don't
know.
I
have
to
tell
you
that
the
unit's,
probably
fine,
whatever
that's
a
lot,
but
if
you
tell
me
that
the
difference
is
25
degrees
fahrenheit,
but
it's
falling
at
five
degrees
per
hour.
A
I
can
tell
you
that
you're
leaking
refrigerant
in
a
in
a
pretty
severe
way.
Actually,
like
you
need
to
get
that
thing
patched
up
right
now,
it's
gonna,
it's
gonna
die,
it's
gonna
fail.
So
a
quick
summary.
What's
the
tldr
industrial
iot
is
about
exposing
and
analyzing
time
series
data
that
used
to
be
difficult
or
impossible
to
gather
from
real
world
equipment.
A
So
some
version
of
that
statement
was
what
my
co-founder
came
to
me
with
when
we
started
telus-
and
you
know
that
was
my
that
was
essentially
my
job-
that
was
my
charge
was
to
was
to
figure
out
how
to
serve
this
data
and
track
it
and
deal
with
it.
And
after
setting
up
a
few
prototypes
based
on
different
types
of
text
stacks
the
you
know,
we
really
had
sort
of
an
aha
moment
at
the
most
abstract.
A
What
the
goal
is
is
to
track
metrics
about
a
metal
box
that
is
connected
to
the
internet.
Right.
That's
what
this
does
is
it
connects
that
metal
box
to
the
internet,
so
the
aha
moment
is
that
servers
right
computer
servers.
Internet
servers
also
happen
to
be
metal
boxes
that
are
connected
to
the
internet,
they're
out
there
in
the
world,
they're
invisible
to
the
people
using
them
for
the
most
part
right
and
the
world
of
we'll
call
it
site.
A
A
Now
it
all
starts
with
something
that
most
people
are
pretty
used
to
looking
at
right.
This
is
sort
of
an
interactive
map.
We
use
google
maps
and
instead
of
driving
directions,
we
can
now
give
directions
to
broken
hvacs.
So
these
are
hvac
units
just
like
any
others
out
there,
except
that
we
have
these
talos.
These.
You
know
these
guardian
sensor
boards
on
them.
A
A
A
A
Recently,
we
can
again
confirm
the
location
of
it
and
I
have
the
key
metrics
that
I
want
coming
off
of
here
right,
so
I
can
see
how
much
electrical
current
the
unit's
using
for
you,
hvac
geeks
out
there
we've
separated
the
compressor
right.
That's
the
sort
of
the
the
the
beating
heart
of
the
hvac.
If
you
will
from
the
indoor
fan
motor
and
outdoor
fan
motor
and
here's
the
you
know
the
sort
of
formation
liquid
and
suction
lines,
along
with
the
outdoor
air
temperature
that
we're
picking
up
from
from
a
local
weather
station.
A
So
we
can
run
some
basic
analytics
on
this.
We
can
see
some
faults,
so
let's
see
here's
the
first
fault.
The
first
fault
is
that
this
compressor
is
on
a
lot
of
the
time,
even
though
it's
only
70
degrees
fahrenheit,
that's
bad,
because
if
the
next
day
were
to
be
say,
90
degrees
fahrenheit,
which
is
uncomfortably
warm,
the
the
people
in
this
building
are
going
to
be
hot
right.
So
this
is
a
classic
use
case.
A
The
people
in
this
building
are
still
comfortable
on
this
on
this
day
that
I've
selected
this
data
from
so
nobody
knows.
There's
a
problem,
but
we
do.
I
can
predict
a
failure,
because
if
this
is
you
know,
if
the
day
gets
much
hotter,
then
the
the
hvac
won't
be
able
to
keep
up
all
right.
It's
very
bad
side.
A
A
A
You
really
want
that
at
about
45
degrees,
because
it's
impossible
for
an
air
conditioner
to
make
55
degree
fahrenheit
air,
which
is
what
it's
designed
to
do
if
the
suction
line
is
hotter
than
that,
we'll
keep
going
just
see
one
other
one.
Other
interesting
thing
that
we
were
able
to
pick
up
on
this
unit
was
this
example
here.
So
here
we
see
that
the
indoor
fan
motor
is
actually
pegging
out.
My
sensor,
my
sensor
can
only
detect
50
amps,
it's
maxed
it
out.
A
This
is
a
condition
called
locked,
rotor
amps,
without
getting
into
too
many
of
the
details.
You
just
have
to
know
that
this
is
a
bad
thing,
but
it's
a
very
fleeting
state.
This
lasts
only
for
a
very
short
period
of
time
and
it's
hard
to
detect
unless
you
happen
to
be
standing
right
there
listening
for
it
and
we
were
able
to
detect
this
thing.
You
know
pulling
locked
ramps
twice
in
a
row.
It
means
the
fan
motor
is
going
to
die.
You
got
to
go,
replace
it
fan
motor
dies.
A
A
So,
with
a
little
bit
of
planning,
you
can
get
extremely
powerful
behavior
for
monitoring
a
fleet
of
iot
devices
like
this.
So
again,
in
my
experience,
I
don't
pretend
to
be
an
expert
at
this
stuff,
but
there
were
two
things
that
attracted
me
to
prometheus,
so
the
first
one
was
really
how
easy
it
is
to
administer
the
alerts.
A
All
you
have
to
do,
pull
this
back
up
here.
All
you
have
to
do
is
just
you
know,
edit
your
yaml
file
hit
reload
you
get
tons
of
this
rich
behavior
for
free.
You
get
reliable
evaluation.
You
get
these
alert
delays.
You
get
silences
deduplication
the
whole
deal
it
all
sort
of
works
out
of
the
box
right,
which
is
which
is
which
is
really
nice.
It's
really
powerful.
A
The
second
was
the
query
language
itself.
Right
promql
is
really
a
remarkable
tool,
so
it
takes
a
little
bit
of
a
learning
curve,
but
once
you
get
your
head
wrapped
around
it,
you
know,
promql
is
really
a
rich
and
expressive
and
powerful
language
right.
So,
let's,
let's
kind
of
you
know,
maybe
for
the
second
part
of
this
talk.
I
wanted
to
go
over
go
over
an
example
for
you.
A
So,
let's
start
to
craft
an
alert
yeah,
so
you
know
in
the
you
know
in
the
hvac
world
here
right
when
your
suction
line
gets,
is
what
gets
cold
right.
That's
what
I
mentioned
earlier.
The
suction
line
is
ideally
like
45
degrees
once
it
hits
32
degrees,
bad
things
start
to
happen
because
at
32
fahrenheit
things
you
know
water
starts
to
freeze,
it
will
freeze
out
of
the
air
it's
going
to
turn
your
hvac
into
a
block
of
ice.
A
If
the
when
the
hvac
turns
off
that
block
of
ice
is
going
to
melt
now,
you
have
a
bunch
of
water,
it
can
cause
water
damage,
it
can
cause
mold.
You
really
want
to
avoid
this,
but
these
sort
of
freeze
thaw
cycles,
I've
seen
them
take
place.
You
know
out
in
the
wild
over
weeks
and
months,
just
because
nobody
knows
they're
happening
the
unit's,
still
pumping
out
cold
air,
so
the
people
in
the
building
don't
notice.
It's
just
really
struggling,
and
now
the
building
owner
has
a
bunch
of
mold
damage
sucks.
A
A
What's
our
metric
going
to
look
like
well,
the
design
decision
that
we
made
was
to
define
a
single
metric
per
equipment
type,
so
the
type
of
equipment
that
we
were
dealing
with
here
right
this
this
this
equipment
right
here
that
use
oops
that
you
see
on
the
screen
that
is
called
a
package
unit.
It's
a
rooftop
package
unit
right!
It's
just
it's
one
of
these
big
metal
boxes
that
you
see
on
commercial
rooftops,
and
so
let's
define
a
gauge
metric
for
the
equipment
of
a
package
unit.
So
we'll
call
it.
You
know.
A
Okay,
now
each
of
these
sensors,
so
all
you
know
all
data
from
all
packagings
that
I'm
monitoring
will
come
to
this
metric,
this
gauge
and
each
of
these
sensor
boards.
It
has
an
individual
id
number.
I
call
we
call
this
the
device,
eid,
the
device
entity
id
and
so
in
order
to
distinguish
between
different
data
streams.
A
Okay,
so
quick
aside
here,
I'm
sure
that
the
observability
nerds
in
the
audience
are
going
to
be
really
quick
to
point
out
that
this
is
technically
unbounded.
Cardinality,
that's
a
very
good
catch
right,
very
smart,
pat
yourself
on
the
back.
You
are
technically
correct,
which
is
of
course,
the
best
kind
of
correct.
A
We
have
made
the
business
decision
that
if
this
startup
is
so
successful
that
we
have
so
many
units
out
there
that
this
becomes
a
problem,
then
we'll
have
plenty
of
money
to
pay
all
you.
You
know
one
of
you
all
to
come
in
and-
and
you
know
patch
this
up
for
us,
so
you
know
quick
aside,
but
yes,
that
is
a
you
know.
That
is
a
known
danger
of
of
doing
exactly
this.
A
Okay,
there's
also
so
the
other
thing
we
have
to
keep
in
mind
back
to
our
back
to
our
metric
design
is
that
there's
many
data
points
that
come
off
of
a
single
unit
right,
we're
doing
the
temperature
of
the
suction
line
liquid
line,
we're
doing
a
few
different.
You
know,
seat
amp
readings
ct
readings,
so
we
have
to
add
another
label,
so
the
label
that
we
want
here
is
going
to
be.
A
A
A
little
four
clause
and
a
bunch
of
labels,
some
annotations
and
we're
done
right,
ship
it
go
live
so
maybe
not
quite
okay.
First
of
all,
let's
generalize
this,
so
this
is
an
alert
for
one
particular
device,
but
all
I
have
to
do
is
blow
away
this
piece
of
logic,
and
now
this
is
valid
for
every
package
unit
in
the
fleet
very
powerful,
but
unfortunately
the
logic
is
still
not
quite
right.
A
So
what
if
this
is,
let's
say
a
cold
winter
day
now
no
cooling
is
going
to
go
on
on
a
cold
winter
day,
but
the
suction
line
is
just
a
piece
of
copper
exposed
to
the
elements
it's
going
to
get
at
least
where
we
live
in
richmond
virginia,
it's
going
to
get
below
freezing
so
this
will
you
know
this
condition.
Will
fire
at
that
point?
It's
an
alert,
but
you
don't
want
to.
That
would
be,
in
this
case
a
false
alarm
right,
really
bad.
A
So,
let's
put
in
a
little
bit
more
logic
to
to
prevent
this
right.
That's
not
an
alert
condition
if
the
day
happens
to
be
cold.
In
fact,
it's
only
an
alert
condition
if
the
hvac
unit
is
running
and
then
the
suction
line
freezes.
So
no
problem
right,
let's
add
a
little
bit
more
logic.
So
let's
say
that
we
want
this
to
be.
You
know
what
we
want
and
we
want
to
quit
package
in
this
case.
A
You
would
realize
that
we
have
to
do
this
on
device
eid
all
right
so
now
for
any
device.
Eid
right.
It
only
compares
those
devices-
the
particular
device,
the
id
here
particular
device.
I
idea
here
now
when
the
suction
line
is
cold
and
the
unit
is
using
electrical
power
right,
amps
are
greater
than
one
one's,
a
very
small
number
of
amps
for
a
compressor
to
be
using.
So
we
know
it's
on
then
we're
gonna
we're
gonna,
get
it
right.
We're
gonna
get
an
alert,
fantastic
right
and
this
rule
actually
looks
pretty
good.
A
In
fact,
I
put
a
a
version
of
this
rule
into
production
once
upon
a
time,
but
when
we
did,
fortunately
we
caught
it
the
problem
before
it
became
an
issue.
We
actually
ran
into
a
little
bit
of
a
gotcha
and
this
one's
a
little
bit
more
subtle
data
coming
off
of
an
iot
sensor,
especially
one
with
a
wireless
connection,
is
not
100
reliable.
A
A
It
could
mean
that
the
sensor
is
failing
to
send
data
for
whatever
reason
right,
you
can
end
up
with
stale
data
and
a
lot's
been
written
about
stale
data
in
prometheus,
I'm
not
going
to
retread
all
of
that
here.
So
my
greater
point
is
that
this
is
actually
a
pretty
big.
You
know
issue
in
the
iot
world.
Is
this
notion
of
stale
data
to
work
around
this?
I
actually
separately
defined
another
metric
that
is
going
to
keep
track
of
the
timestamp
at
which
I
received
a
data
point.
A
I
do
this
in
seconds
since
the
epoch,
so
we're
going
to
call
this
metric
we're
going
to
call
this
metric
feeds
last
scene
right.
The
last
seen
time
that
I
have
you
know
gotten
new
data
from
a
certain
from
a
certain
data
feed
and
in
order
to
make
this
data
again,
it's
it's
tracked
in
seconds
since
the
epoch.
So
in
order
to
make
this
useful
again,
I
need
to
do
this
on
device
eid,
and
I
need
to
take
the
rate
of
increase
of
this
time
stamp
over
some
period
of
time.
A
A
If
I'm
just
reading
the
same
data
over
and
over
again
the
same
exposed
metric
over
and
over
and
over
again,
then
this
won't
increase
and
the
rate
of
increase
will
be
zero,
and
then
this
alert
won't
fire.
I
could.
I
could
choose
to
set
up
a
separate
alert
showing
that
I'm
getting
stale
data.
If
I
want
to
it's
fantastic,
so
we
want
to
fire
this
alert
only
if
new
data
comes
in
again
to
reaffirm
that
the
frozen
suction
line
condition
is
actually
still
frozen.
A
Now
we
have
a
actually
pretty
nice,
reliable,
alerting
rule
here
and
what
happens
if
we
add
another-
let's
say
100
boards
onto
package
units
all
over
the
place
right.
So
the
beauty
here
is
that
I
don't
have
to
do
anything.
We
don't
have
to
update
this
rule
at
all.
It's
really
cool.
I
don't
have
to
touch
my
config
file.
I
don't
even
have
to
reload
prometheus.
We
get
this
all
for
free,
that's
very
powerful
and
it's
very
scalable.
A
So
this
is
a
like.
This
is
a
ton
of
functionality,
all
in
a
very
small
lightweight
package.
It's
pretty
easy
to
administer.
You
know
the
company's
grown
quite
a
bit
since
I
first
set
all
this
stuff
up,
but
telos
started
life
as
a
two-person
startup.
I
was
the
only
engineer
and
the
fact
that
a
single
person
could
stand
up.
You
know
all
this
sort
of
scalable
infrastructure.
A
You
know
really
without
any,
without
any
prior
expertise
and
this
kind
of
technology.
I
think
that
shows
how
you
know
how
lightweight
it
is,
while
still
being
very,
very
scaled.
So
I
hope
this
has
given
you.
I
really
hope
this
has
given
you
all
a
taste
of
you
know
a
how
we
run
prometheus
for
iot
and
again
I
want
the
point
of
this
talk
not
to
be
like
how
you
do
this.
You
know
perfect
how
a
world's
expert
might
do
this
perfectly.
A
The
point
of
this
talk
is
really
to
show
our
experience
in
standing
up
this
system
again
with
no
prior
expertise
in
the
technology.
So
I
have
no
doubt
that
you
know
kind
of
the
prom
ql
wizards
in
the
audience.
You
all
have
probably
come
up
with.
You
know,
10
different
and
better
ways
that
we
could
have
designed
this
data.
A
You
might
be
getting
real
agitated
by
now,
and
if
this
is
you
please,
you
know,
I
welcome
your
your
suggestions
in
in
this
matter
and
even
if
that
doesn't
describe
you
I'll
invite
any
questions
that
you
have
as
well
here.
So
we
are
just
you
know
if
you
want
to
get
in
touch
with,
if
you
want
to
get
in
touch
with
me,
this
is
where
I'm
I'm,
my
name
is:
will
gatherette
and
with
telos
iot.