►
Description
Challenges of Logging on the Edge - Mickey Pashov, Medtronic Digital Surgery
In this talk we will describe our journey from arbitrary log collection from edge devices to building an observability pipeline with FluentBit. We will discuss the reasons why we chose it over other agents - highly performant with a small memory footprint, open-source, vendor-agnostic, great community support, extensibility via Lua filters, and a plethora of core input and output plugins. Some of the challenges of deploying on-prem such as firewall and web filter rules; authN and authZ to the cloud; offline-mode and packaging and deployment using an unfamiliar toolchain. We will also discuss the the challenges of deploying an observability pipeline on Kubernetes with FluentD. From setting up the Helm charts and livenessProbes to TLS termination and autoscaling.
A
A
Let's
start
with
a
little
bit
about
me,
I'm
based
in
london,
first
and
foremost,
I'm
a
back-end
python
engineer,
but
my
curiosity
has
led
me
into
the
world
of
devops
and
cloud-native
technologies.
I've
been
working
at
medtronic
digital
surgery
for
the
last
18
months
to
begin
I'll,
cover
medtronic
and
what
it
does
I'll
get
into.
A
A
A
I
want
to
point
out
that
the
edge
isn't
quite
the
same
as
on-prem
or
the
cloud.
It's
a
distributed,
computation
model
where
computation
is
closer
to
the
source
of
data.
For
us,
this
means
lower
latency
for
insights,
reduced
cost,
since
we
can
run
ai
algorithms
on
the
device
and
it
can
run
offline,
which
means
surgeons
can
benefit
without
the
need
for
internet
access
and,
most
importantly,
privacy
and
security.
A
A
A
We
collected
arbitrary
logs
for
our
running
system
d
services
as
well
as
os
logs
in
terms
of
forwarding.
We
were
doing
the
following.
We
were,
we
had
a
timer
where
every
24
hours,
we
would
upload
archived
journal
d
logs
to
s3
and
they
would
have
a
timestamp
as
their
file
name.
Of
course,
that's
assuming
that
there
was
connection
available.
A
A
A
We
needed
an
approach
to
log
management.
The
basic
use
case
was
to
have
the
ability
to
stream
logs
to
a
back
end
such
as
cloudwatch
elasticsearch
datadog,
for
that
we
needed
an
agent
to
forward
these
requests.
My
first
thought
was
to
use
fluent
bit
with
our
existing
logging
back
end.
After
all,
that's
what
I
had
done
previously
for
a
kubernetes
cluster,
however,
getting
fluent
but
deployed
on
an
edge
device,
isn't
quite
as
simple
as
helm
install.
I
was
faced
with
a
completely
new
tool
chain
things
like
the
octo
project,
open
embedded
in
bitbake.
A
A
It
didn't
have
cloud
watch
support
or
an
http
plugin
out
of
the
box,
so
I
changed
tact.
I
decided
to
use
a
vendor
product
since
we
already
had
some
upfront
cost
the
data
dock
solution
attracted
me.
We
were
on
the
market
for
a
better
user
experience
on
the
platform
side
and
datadock
had
the
promise
of
an
agent
metrics
and
logs
with
archival
storage
to
s3
all
with
enterprise
support.
A
A
That's
when
the
fun
began.
It
worked
well
out
of
the
box
tailing
files
and
journal
d-logs.
The
documentation
felt
a
bit
sparse,
however,
and
it
felt
like
it
had
a
focus
on
metrics,
which
are
very
useful.
Of
course,
the
ability
to
root
and
filter
happened
cloud
side,
and
I
felt
like
that
would
be
locking
us
in.
A
We
came
up
with
these
takeaways
first
offline
mode.
Our
devices
were
deployed
in
a
sub-optimal
network
environment
and
often
there
was
no
ethernet
cables
in
the
operating
room,
which
meant
that
logs
would
be
far
older
than
24
hours.
Cross-Compilation
of
the
agent
just
needed
to
be
simple
as
possible,
we're
a
small
team
and
we
didn't
have
time
to
rework
our
always
build
pipeline.
A
A
In
hindsight,
I
think
that
key
could
be
used
for
a
whole
fleet
as
long
as
you
have
traffic
encrypted.
Finally,
data
export.
We
wanted
to
have
the
data
for
long-term
auditing
purposes
and
for
our
data
science
team
to
be
able
to
access
and
datadock
provided
s3
archival,
but
you
have
to
conform
to
their
folder
and
file
structure,
and
so
it
felt
like
it
was
time
for
fluent
bit
again.
A
A
A
A
Let's
talk
about
offline
mode,
it's
worth
making
the
distinction
between
a
lossy
or
noisy
network
versus
no
network
in
hospitals.
You
certainly
get
both,
but
it
was
the
latter
that
we
were
more
interested
in.
We
would
have
systems
offline
for
weeks
at
a
time
and
to
solve
this,
I
think
we
had
several
options
available.
A
A
You
can
use
it
to
set
the
service
in
the
service
section,
and
it
sets
the
wait
time
before
you,
your
process,
exits.
You
specify
this
in
seconds.
Of
course,
the
current
behavior
makes
it
so
the
task
retries
are
not
handled
once
the
engine
is
shutting
down,
and
as
of
this
talk,
there's
an
open
pr
to
address
this
issue.
A
A
A
A
A
A
A
A
We
wanted
to
keep
things
simple.
Initially,
we
started
with
an
agent
deployment
as
our
architecture.
Basically,
each
node
has
an
agent
running
that
fires
into
the
back
end.
This
has
to
be
the
most
common
way
people
deploy
flowing
bit
in
the
cloud
today,
however,
there
are
some
disadvantages
and
we
run
into
them.
A
You
can
allow
processing
at
scale
because
you
can
independently
scale
your
aggregator
tier
and
you
can
easily
add
back
ends
and
configuration
changes
because
you're
changing
it
in
one
place.
The
disadvantages,
of
course,
are
that
you
have
to
have
dedicated
resources
to
to
run
this
aggregator
instance.
A
It
simply
refers
to
the
fact
that
there
are
some
streaming
technologies
in
the
middle
and
you
can
do
your
own
routing.
However,
I
wouldn't
recommend
it
as
your
first
approach
unless
you
have
the
expertise
or
you
have
the
scale
of
say,
10
000
messages
per
second,
although
we
had
kafka,
we
certainly
weren't
experts.
A
So
we
settled
on
the
common
aggregator
approach,
but
we
needed
to
test
and
harden
it.
One
small
tip
I
have
here
is
to
set
up
synthetic
transactions
or
height
beats
you
initially.
Your
pipeline
won't
have
much
traffic,
so
you
need
some
way
of
of
passing
events
through
it,
and
this
can
be
done
by
setting
up
a
small
fluent
bit
agent,
for
example,
that
has
dummy
messages
being
sent
to
the
aggregator.
A
A
A
A
We're
still
building
our
kafka
integration
once
that's
up
and
running,
and
once
we
have
our
back,
I
think
we
can
reevaluate
the
observability
pipeline,
but
again
we
probably
need
the
scale
first.
A
fluent
bit
now
has
support.
It
has
support
for
the
kafka
protocol
as
well
as
well
as
the
kafka
rest
proxy,
as
output
plugins
and
since
version
1.9
there's
a
kafka
input
plugin,
but
it's
still
a
proof
of
concept,
but
it's
still
awesome
at
the
very
start.
A
I
mentioned
that
we
use
mqtt,
which
is
a
lightweight
pub
sub
protocol
that
we
already
use
for
events
and
metrics.
One
experiment
that
would
have
been
nice
to
have
is
to
ship
our
logs
using
it.
However,
this
time
it's
only
a
future
request,
maybe
it's
about
time.
I
learned
c
thank
you
for
listening,
I'd
like
to
connect
and
learn
from
your
experience,
so
please
feel
free
to
ask
questions
on
slack
today.