►
From YouTube: Rusty Days 2020 - Luca Palmieri: Are we observable yet?
Description
Agenda ► https://rusty-days.org/agenda
Slides ►https://rusty-days.org/assets/slides/07-are-we-observable-yet.pdf
Playlist with all talks ► https://www.youtube.com/playlist?list=PLf3u8NhoEikhTC5radGrmmqdkOK-xMDoZ
Follow ►
Facebook: https://rusty-days.org/facebook
Twitch: https://rusty-days.org/twitch
Twitter: https://rusty-days.org/twitter
This video ►
Is Rust ready for mainstream usage in backend development?
There is a lot of buzz around web frameworks while many other (critical!) Day 2 concerns do not get nearly as much attention.
We will discuss observability: do the tools currently available in the Rust ecosystem cover most of your telemetry needs?
I will walk you through our journey here at TrueLayer when we built our first production backend system in Rust, Donate Direct.
We will be touching on the state of Rust tooling for logging, metrics and distributed tracing.
A
Amazing
good
evening,
everybody
thanks
for
joining
for
the
last
day
of
rusty
days,
we're
gonna
chat
for
the
next
30
minutes
or
so
about
observability,
in
particular.
We're
going
to
discuss
if
the
rust
ecosystem
at
this
point
in
time
provides
enough
tooling
to
provide
observable
apis
and
we're
going
to
go
through
the
journey
of
writing
one
and
see
how
that
came
along.
A
My
name
is
luca
palmieri.
I
work
as
a
leave
engineer
through
layer
we're
going
to
spend
some
words
about
that
in
a
second
in
the
rustic
system.
I
contribute
to
the
rust
longer
user
group,
where
I
curate
the
code
dojo
I've
been
contributor
and
maintainer
of
variety
of
crates
in
the
open
source,
ecosystem,
lympha
wiremock
and
some
others,
and
I'm
currently
writing
zero
to
production,
which
is
a
book
on
brass,
became
development,
which
I
publish
chapter
by
chapter
on
my
blog,
which
you
can
see
linked
down
there.
A
This
is
a
little
bit
our
agenda,
so
we're
gonna
see
what
donate
direct
is
trying
to
write
is
an
application
and
it's
going
to
drive
a
role
journey.
We're
going
to
see
what
it
entailed
to
bring
that
application
to
production
and
then
we're
going
to
zoom
in
on
three
types
of
telemetry
data,
which
are
often
collected
to
observe
the
behavior
of
applications
in
production
environments,
metrics
logging
and
distributed
traces.
A
If
you
don't
know
what
they
are
or
you
have
an
experience
working
with
them
before.
That's
not
a
problem.
We're
gonna
give
all
the
details
and
I'll
walk
you
through
why
they're
useful
and
how
we
collect
them.
So,
let's
start
from
the
very
basics.
What
is
donate
direct
well
before
that?
Let's
get
two
words
on
what
layer
does
which
is
gonna
frame.
The
conversation
now
truly
is
a
company
which
are
praised
in
the
financial
technology
space.
In
particular,
we
provide
apis
for
people
to
consume.
A
We
mainly
provide
two
types
of
apis,
one
for
accessing
banking
data
on
behalf
of
a
user,
and
then
we
want
to
initiate
a
banking
transfer
once
again
on
the
alphabet
user.
So
you
pay
using
your
own
bank,
account
without
credit
cards
without
intermediaries
of
other
type
during
the
kovit
pandemic.
As
many
of
the
people,
we
kind
of
tried
to
think
what
we
could
do
in
any
way
to
help
relieve
pressure
or
contribute
to
what
was
happening.
A
So
myself,
with
a
group
of
other
colleagues
put
together,
an
application
called
donate
direct
which
lets
you
use
our
payment
initiation
technology
to
donate
money
to
charities.
So,
as
you
can
see
in
the
gif
on
the
left,
the
flow
is
very
simple,
so
you
select
a
charity
from
a
list.
You
specify
how
much
amount
you
want
to
donate.
A
A
A
Now
it's
not
our
first
round
with
thrust,
but
it
was
our
first
rust
api
in
production
here
through
layer.
It
was
the
first
time
we
were
actually
shipping
code
that
was
responding
interactively
to
users
coming
from
the
wild
web.
So
I
want
to
see
lots
of
emojis
when
I
re-watch
the
stream
at
this
specific
slide.
A
Now,
as
I
said,
we
experimented
with
it
before
so
we
were
doing
build
tooling.
We
were
doing
clies,
we
were
doing
some.
We
had
kevin
these
controllers
for
non-critical,
stuff
and
so
on
and
so
forth,
but
once
you
actually
put
an
api
in
front
of
a
user,
then
the
bar
of
our
that
api
needs
to
be
raised
significantly,
which
brings
us
to
our
journey
to
production.
A
A
A
So
what
happens
if
one
of
those
blue
dots,
for
example,
in
the
monster
cluster,
goes
down?
What
happens
in
one
of
those
blue
dots
start
responding
more
slowly
than
it
generally
does
or
is
supposed
to
do,
or
it
doesn't
elastically
react
to
certain
traffic,
all
these
kind
of
behaviors
in
a
very
connected
graph
like
that,
can
cause
cascading
figures
and
it
becomes
very,
very
difficult
once
something
like
that
is
happening
to
troubleshoot
y
and
fix
it
if
possible.
A
Now
what
does
each
of
those
blue
dots
actually
is
now
into
layer's
case?
We
run
a
kubernetes
cluster,
so
all
our
production
applications
are
deployed
on
top
of
kubernetes,
which
means
those
blue
dots
are
kubernetes
deployment.
Cubenase
deployment
is
just
a
service
definition
which
is
going
to
orchestrate
a
bunch
of
copies
of
the
application.
A
A
A
The
pre-production
checklist
can
be
a
significant
and
hard
goal,
because
a
lot
of
things
that
you
need
to
do
in
order
to
actually
see
your
application
out
there
and
so
keeping
along
with
the
lord
of
the
rings
metaphor.
They
might
look
a
little
bit
like
the
ghost
of
the
rings
and
they
look
quite
scary
at
this
point
in
time.
This
is
like
the
first
movie.
A
So
what's
the
dilemma
on
the
left,
application
developer
or
in
general,
like
you're,
building,
something
that
looks
really
cool,
you
want
to
ship
it
and
when
you
are
at
the
beginning
of
your
startup
journey.
So
when
you
are
askanti
group
doing
a
skanky
up
that
is
only
built
by
a
bunch
of
people.
We
increasingly
know
that
what
you're
doing
is
particularly
risky
because
a
new
product
is
a
new
company.
You
just
did
the
rating
fast,
it's
fine
to
just
ship
it,
so
you
put
your
kaboy
hat
on
and
you
just
deploy
to
production.
A
A
So
software
has
to
be
treated
as
we
should
critical
as
much
as
possible
and
to
be
reliable.
There's
a
lot
of
bets
and
wizards
that
you
need
to
touch
to
your
application,
so
metrics
tracing
logs,
horizontal
protocol
scaling
alerts
to
know
when
something
goes
wrong:
network
policies
to
prevent
escalations,
liveness
and
radium
probes
to
drive
cuban
needs
when
to
restart
something
and
so
on,
and
so
forth
at
the
least
can
get
very
long,
and
that
is
troublesome
because
in
the
end,
that's
my
personal
personal
model,
I
would
say,
is:
convenience
beats
correctness.
A
A
A
Then
it
fails
and
then
your
problems,
so
you
won't
be
able
to
fall
into
the
so-called
peter
success
that
you
naturally
converge
to
doing
the
right
thing
because
doing
the
right
thing
is
the
easiest
thing
to
do
now:
I'm
not
going
to
cover
all
the
possible
things
that
we
require
application
to
do,
because
that
would
be
long
and
potentially
quite
boring,
we're
still
going
to
focus
on
telemetry
data.
So
I
think
in
topic
with
the
talk,
are
we
observable
yet
what
kind
of
telemetry
data?
A
A
Why
do
you
want
to
collect
metrics?
Well,
generally
speaking,
you
want
to
collect
metrics,
because
you
want
to
be
able
to
produce
plots
that
look
exactly
like
this,
so
you
want
to
be
able
to
see
well,
what's
the
latency
of
this
application
in
the
last
30
minutes
and
potentially
break
it
down
by
percentile,
so
the
50th
percentile,
the
70th
percentile,
the
90th
and
the
99th,
depending
on
the
type
of
application,
your
performance
profile
or
you
might
want
to
know,
what's
the
response
breakdown
so
how
many
200s
only
500s,
400s
and
so
on
and
so
forth.
A
A
A
What
do
metrics
look
like,
so
how
do
you
actually
get?
Those
plots
that
we
just
saw
metrics
are
generally
looks
somewhat
like
this,
so
you
have
a
metric
name
which
in
this
case
is
http,
requests,
duration
seconds
bucket,
it's
a
bit
more
full,
but
it's
very
precise.
So
we're
talking
about
the
duration
of
http
requests
and
we're
looking
at
the
histogram,
so
we're
looking
at
buckets
of
requests
at
different
type
at
different
thresholds
of
latencies.
A
On
this
metric
we
have
a
set
of
labels
that
we
can
use
to
slice
the
matrix
value.
So
we
have
endpoint
so
bottom
point
http
method
get
post
boot
match
whatever
the
status
code
we
returned
so
in
this
case
404,
and
then
you
have
the
bucket
that
we're
looking
at
so
5
milliseconds,
10,
milliseconds,
25
and
so
on
and
so
forth,
and
then
the
number
of
requests
falling
inside
that
bucket
now
this
was
a
super
fast
photo
four,
so
all
thousand
six
hundred
or
one
fell.
A
We
know
beneath
the
five
milliseconds
that
generally
it's
gonna
be
a
little
bit
more
varied.
This
is
basically
a
time
series,
a
time
series
with
a
variety
of
values.
You
can
slice
and
dice
from
these
time
series
are
produced
by
the
application
and
then
are
aggregated
by
prometheus,
at
least
in
our
specific
setup.
So
chromidius
hits
the
slash,
metrics
and
point
on
all
the
copies
of
an
application.
It
could
be
another
endpoint,
but
that's
generally
the
default
aggregates
all
these
metrics
indexes
them
and
then
allows
you
to
perform
queries
against
them.
A
One
way
to
perform
queries
is
to
do
alerts
so
alert
manager.
You
define
a
variety
of
queries
which
I'll
wait
for
boolean.
So,
as
we
said
before
is
the
error
rate,
so
the
number
of
500s
above
or
below
10
for
15
minutes.
If,
yes,
then,
through
pagerduty
get
an
uncooler,
so
I'm
going
to
call
engineer
to
look
at
the
system
because
something
is
wrong.
A
Otherwise
you
can
use
grafana
if
you
just
want
to
do
some
pd
visualization.
So
if
we
go
back
to
the
slide
we
saw
before,
which
is
this
one,
this
is
grafana,
so
we're
looking
just
that
premiersquare
is
visualized.
This
is
very,
very
useful
for
an
uncool
team
or
operation
team
to
actually
understand.
What's
going
on
now,
how
do
you
actually
get
metrics?
So
how
do
you
get
your
api
to
produce?
A
Metrics
redirect
was
developed
using
optics
web
for
a
variety
of
reasons
brought
a
piece
about
that
a
couple
of
weeks
ago,
feeble
curtis,
it's
very,
very
easy.
So
there's
a
package
on
craigslist
called
arctic's
web
prom,
so
active
swept
prometheus
and
you
just
plug
the
middleware
inside
your
application
is
that
dot
drop
from
dot
clone
line.
The
middleware
takes
some
very,
very
basic
configuration
parameters,
so
a
prefix
for
the
metrics
and
the
point
you
want
to
use
and
then
you're
set
up
now
you're
just
going
to
experience,
slash
metrics.
A
Now
you
might
want
to
customize
it
for
your
specific
application,
because
you
might
need
to
collect
metrics,
which
are
known
standards
you
might
have
specific
naming
conventions
and
so
on
and
so
forth.
Lexus
web
prompt
is
like
a
single
file,
type
of
crate,
so
you
can
go
there,
use
it
as
some
kind
of
a
blueprint
and
adopt
it
to
do
whatever
you
need
to
do
so.
Metrics
useful,
very
easy
to
collect
just
plug
and
play
if
you're
using
optics.
A
A
As
we
saw,
metrics
are
about
what
is
happening
in
the
system
in
the
aggregate
at
this
very
very
moment
so
low
latency,
fairly
aggregated
type
of
data
logs
are
instead
useful
to
answer
the
question.
What
is
happening
to
this
specific
request,
such
as
what
happened
to
users
who
tried
to
do
a
payment
from
let's
say,
hsbc,
to
barclays
in
the
uk,
between
5
pm
and
6
pm
on
the
27th
of
july?
A
A
A
This
is
an
example
taken
straight
from
the
log
crate,
documentation
or
a
place.
I
think
it
was
you
enter
into
the
shape.
The
yak
function
takes
a
yak
in
a
mutable
reference
to
yak.
You
meet
a
trace
level
statement,
so
you
announce
to
the
world.
We
are
commencing.
The
shaving
is
trace
level,
so
it's
at
a
very,
very
bose
logging
level.
In
most
cases
it's
going
to
be
filtered
out,
then
you
loop
and
try
to
require
a
razor.
A
If
you
get
a
razor
info
level,
lock
statement
presser
located
display
implementation
of
the
razor
you
shape
the
yak
and
you
break
from
the
loop
and
then
you
exit
the
function.
If,
instead,
you
fail
to
find
the
razor,
then
you
need
a
warning
saying
I
was
unable
to
look
at
the
razor
you're
going
to
retry.
A
Now
facade
means
that
you
have
no
idea
what
is
actually
going
to
consume.
These
log
statements
just
instrument
your
code
and
then
generally
at
the
entry
point
of
your
binary,
you're
gonna
introduce
a
local
implementation,
so
an
actual
implementation
that
takes
this
log
data
and
then
does
something
with
them
where
something
is
generally
shipping
them
some
places.
A
If
you
use
the
simplest
possible
logger,
which
is
generally
a
blocker,
you're
gonna,
see
something
like
this,
so
you
log
into
the
console
standard
out
in
this
specific
execution,
which
I
made
you
get
unable
to
locate
the
razer
for
three
times.
So
we
loop
in
three
times
and
then
you
actually
locate
the
razor.
So
you
have
the
log
message,
the
name
of
the
function,
and
then
you
have
a
timestamp
and
the
log
level.
A
A
These
applications
are
generally
some
kind
of
server
either
a
web
server
or
a
new
consumer,
or
something
like
that
and
they're
executing
many
many
requests
concurrently
and
you
want
to
be
able,
at
a
certain
point
generally
later
so
you're,
not
really
there
tailing
the
logs
to
say
what
happened
to
request
xyz,
which
was
about
this
type
of
users,
as
we
discussed
before,
and
the
only
way
you
can
do
that
in
plain
logging
is
using
text
search,
but
text
search
is
not
easy
to
search.
First
of
all
is
expensive.
A
It
cannot
be
indexed
and
requires
also
a
lot
of
knowledge
about
how
the
logs
are
structured.
So
you
end
up,
if
you
want
to
do
anything
that
is
non-trivial,
so
anything
which
is
not
tell
me
if
this
substring
is
in
the
log,
you
end
up,
writing
pregnancies
and
running
reg.
Exes
means
that
you
are
coupled
to
the
implementation
of
the
login
inside
the
application,
which
makes
it
very
very
complicated
for
operators
and
support
people
to
actually
go
and
use
these
logs.
A
So
let's
have
a
look
and
how
we
could
do
structure
logging
so
similar
example
not
fully
identical
this
time.
The
debug
macro
is
coming
from
this
log
crate.
It's
log
standing
for
structure
logging
slot,
create
this
one
as
well,
but
established
been
there
for
quite
some
time.
It
allows
you
to
specify
the
log
message
so
very
similarly
to
what
we
were
doing
before
and
that
allows
you
to
specify
using
the
old
macro
sum
key
value
pairs
to
be
attached
to
your
locks.
A
Now
slog
has
been
for
a
very
long
time.
The
only
way
to
do
structure
logging
in
rust
recently,
if
I'm
not
mistaken,
the
lock
crate,
has
update
a
feature
to
add
key
value,
key
value
pairs
to
log
statements,
but
once
again,
as
far
as
I've
seen
at
least
after
a
month
ago,
almost
none
of
the
logger
implementations
actually
supported
key
value
pair
logging,
so
you
once
again
down
to
zlog
for
doing
structure
logging.
A
So
what
are
we
trying
to
do
here?
We're
trying
to
do
here
is
what
we
generally
want
to
do
in
distributed
applications.
So
I
want
to
know
when
something
is
beginning.
I
want
to
do
some
stuff
which
might
be
composed
of
some
subroutines,
so
this
sub,
you
need
the
work
function
that
might
have
made
their
own
logs.
A
So
this
event
log,
so
cool,
and
then
you
want
to
know
when
that
thing
has
ended
and
then,
given
that
we're
shaving
the
yacht
on
behalf
of
somebody
else,
so
we're
taking
this
user
id,
I
want
the
user
id
to
be
associated
to
every
lock
line,
and
I
also
want
to
capture
along
the
whole
application
took.
So
I
want
to
capture
that
elapsed.
Milliseconds
at
the
bottom,
if
once
again
we
plug
into
it
the
most
basic
type
of
a
matter.
So
in
this
case
it's
a
boolean,
formatter
login
to
standard
out.
A
Now,
let's
go
back
to
the
code,
you
may
agree
with
me
that
this
is
very
verbose.
It's
very,
very
noisy,
like
you
have
a
lot
of
lock
statements
which
are
interleaved
with
the
application
code.
You
don't
even
see
the
application
code
here,
but
this
function
is
really
looking
a
little
bit
hairy,
and
this
is
because,
generally
speaking
for
most
use
cases
at
least
the
ones
the
one
I
encountered
in
the
world
having
or
fun
log
events
is
generally
the
wrong
abstraction.
You
reason
about
tasks
and
tasks
have
a
start
time.
A
A
We
are
moving
away
from
slog
so
we'll
leave
this
log
behind
for
the
time
being
and
we're
moving
on
to
the
tracing
crate.
Tracing
crate
is
part
of
the
tokyo
project
and
I
think
it's
not
an
overstatement.
It's
one
of
the
most
impactful
crate,
at
least
on
what
they
do
on
a
daily
basis,
which
has
been
released
in
the
past
year
or
so
provides
extremely
quality
implementation
and
we're
gonna
see
it
suits
our
needs
perfectly.
A
So
aspar
we
enter
into
the
function
and
we
create
a
spawn
the
bug
level.
So
we
set
the
level
as
if
we
were
doing
logging.
We
tell
what's
the
name
of
the
spawn
yacht
shave
and
we
associate
with
the
spawn
the
user
id
now.
The
threshing
crate
uses
a
guard
button.
So
when
you
press,
when
you
press,
when
you
call
the
dot
enter
function,
then
you're
gonna
enter
inside
the
spawn.
A
Everything
that
happens
between
the
enter
function,
method,
invocation
and
the
dropping
point
of
the
underscore
enter
guard
is
going
to
happen
in
the
context
of
the
same
spawn,
which
means
there's
no
need
for
us
to
add
once
again,
the
user
id
to
debug
there's
also
no
need
for
us
to
do
anything
weird
about
sub-unity
work.
So
being
the
work
can
ignore
the
fact
that
it's
part
of
the
yak-shay
function.
A
A
It's
as
we
were
saying
before
much
more
convenient,
it's
much
easier
for
developers
to
slap,
slash,
hush,
tracing
instrument
on
top
of
a
function
and
so
allow
them
to
build
very,
very
domain,
oriented,
trace
bonds
and
do
that
consistently.
If
that
does
not
involve
finding
a
load
of
code,
this
does
not
involve
polluting
the
function
code
and
it's
generally
transparent
to
the
application.
A
Now
tracing,
just
like
log
and
just
like
slog,
is
a
facade
pattern.
So
what
you
do
you
instrument
your
application
using
those
macros,
and
then
you
have
subscribers.
Subscribers
are
the
ones
that
actually
receive
these
tracing
data
and
can
do
something
with
it.
So
tracing
can
be
used
for
structure
logging.
I
think
at
this
point
in
time,
is
the
best
crate.
If
you
really
want
to
do
structure
logic,
so
you
can
log
all
those
funds
to
standard
out
or
to
a
file
or
whatever
you
think
is
useful
to
you
at
the
same
time.
A
Using
spawns
and
spawns
are
exactly
the
concept
used
by
distributed
tracing
as
we'll
see
in
a
second,
so
one
type
of
instrumentation
tracing
and
you're
able
to
get.
At
the
same
time.
Writing
no
extra
code,
both
structure,
locking
and
dcbd
tracing-
and
this
is
extremely
powerful
and
also
extremely
consistent,
because
you're
going
to
get
the
same
spans
across
the
two
type
of
telemetry
data,
so
timely
data.
How
do
we
actually
process
logs
and
we
actually
process
traces
so
logs
we
take
tracing?
Then
we
have
a
subscriber
that
prints
logs
to
standard
out
in
boone
format.
A
Is
the
tracing
bunion
log
for
matter
which
I
wrote
for
donate
direct
and
some
crates
of
io.
If
you
want
to
use
it,
then
standard
out
is
tailed
by
vector
vector.
Is
another
rust,
lock
corrector
that
we
use
to
get
logs
from
standard
out
to
elaborate
kinesis,
which
is
then
going
into
elastic
search,
which
we
then
search
using
kibana?
So
there's
a
bunch
of
hops
bring
the
engine
up
in
hibana
and
kibana
is
fairly
good
to
search
logs
and
you
don't
need
to
be
a
developer
to
search
logs
in
kibana.
So
you
go
there.
A
A
So
when
you
talk
about
logs,
it's
generally
about
a
single
application,
so
you
have
this
application.
That
is
there
and
it's
doing
stuff,
and
it's
emitting
blocks
now
in
a
micro
service
architecture
as
the
one
we
have
here
later
and
in
many
places
at
this
point
to
serve
a
single
request
which
is
hitting
the
edge
of
your
cluster.
A
Now,
when
a
customer
comes
to
you
saying
I
tried
to
do
x
and
it
didn't
work,
you
need
to
understand
where
exactly
the
problem
is,
so
you
need
to
be
able
to
trace
that
request
across
the
different
microservices
and
it
should
be
easy
to
do
so.
The
way
you
do
this,
or
one
of
the
possible
ways,
is
by
adhering
to
the
jaeger
tracing
format
which
is
now
being
evolved
by
the
open
tracing
tracing
format,
which
is
now
being
merged
into
the
open,
telemetry
format.
A
So,
on
the
tracing
crate,
you
have
a
tracing
open,
telemetry
subscriber,
which
is
maintained
in
the
same
repository.
You
can
use
that
and
we
do
to
ship
logs
into
jager
to
ship
traces
into
jager.
Jager
is
once
again
backed
by
elasticsearch,
so
it's
more
or
less
the
same
infrastructure
and
allows
you
to
have
this
kind
of
view.
A
So
each
of
the
units
of
work
appear
as
a
bar.
You
track
along
each
of
those
stage
and
you
can
see
all
the
different
services
that
a
single
request
coming
from
the
outside
flows
through
that
is
very,
very
powerful
to
understand
when
something
went
wrong,
so
you're
able
to
correlate
a
request
across
everything
that
is
happening
inside
the
cluster.
A
As
we
said,
production
environments
are
extremely
complex
and
if
you
don't
have
any
way
to
observe
what
is
happening,
and
that
generally
means
having
some
kind
of
telemetry
data,
then
your
production
environment
is
a
ticking
pump.
It
might
be
in
life
today,
but
it's
going
to
go
off
at
a
certain
point
in
the
future
and
you're
not
gonna
like
it.
A
A
Now
the
netdirect
was
an
experiment
in
using
rust
in
a
live
production
application
and
we
liked
it
so
in
one
way
or
another,
probably,
the
cto
was
not
fully
sovereign
when
he
said
that,
but
we
chose
to
bet
on
rust
to
do
some
new
core
projects
in
particular
providing
a
core
banking
application
which,
in
a
nutshell,
means
cleaning
accounts,
programmatically
moving
money
in
and
out
programmatically
once
again
we're
assembling
a
team.
We
hide
writing
bunch.
We're
still
looking
for
one
raspberry
can
engineer.
A
A
Well,
it
depends
on
how
we,
what
do
we
mean
by
integrating?
Well,
in
our
specific
use
case,
we
do
have
some
structures
that
we
expect
application
to
follow
in
the
type
of
telemetry
data
that
they
produce
so
for
expect.
For
example,
we
expect
metrics
exposed
by
apis,
but
a
certain
format
on
a
certain
naming
convention.
A
So,
generally
speaking,
there
needs
to
be
a
little
bit
of
coordination
because
of
course,
if
somebody
goes
with
the
net
core
defaults
format-
and
I
go
with
the
ras-
the
full
format
and
you
go
with
the
python
default
format-
it's
very
unlikely
that
they're
going
to
really
match
up
really
nicely,
but
you
can
use
architectural
decision
records
to
just
say
these:
are
we
do
logs
and
then
everybody
implements
in
such
a
way
that
they
can
interoperate?
So
it
needs
a
little
bit
of.
A
A
Okay,
so
another
one
from
twitch
that
chris
is
asking
the
trace,
crate
looks
very
powerful.
Are
there
any
features
that
you
wish
it
had?
They
don't
have
to
be
easy
features.
I
just
like
to
hear
your
thoughts
on
the
design
space
more
well
yeah.
The
trading
credit
is
extremely
powerful.
I
did
raise
some
issues
for
some
of
the
things
that
kind
of
surprised
me,
so
those
some
of
those
have
made
their
way
into
the
tracing
crate
itself.
Also
bug
fix
on
a
core
dump.
A
A
For
example,
one
thing
is
traces,
so
the
the
metadata
you
collect
about
the
spawn
is
statically
determined
at
the
moment
of
spawn
creation,
and
that
is
great
because
then
everything
is
much
faster.
It
consumes
less
memory,
but
sometimes
for
the
way
certain
applications
are
architected.
You
would
like
to
be
able
to
add
additional
metadata
dynamically,
even
if
that
means
allocating
or
doing
stuff.
That
might
not
be
what
you
want
to
do
in
an
off
loop,
maybe
for
that
application
and
its
performance
profile
works
fairly
well.
A
Another
thing
that
we
found
was
a
little
bit
of
a
slippery
slope
was
the
instrument
macro,
which
is
very,
very
convenient
because
it
captures
the
name
of
the
function
body
captures
by
default,
all
the
arguments
of
the
function,
and
that
can
somewhat
be
tricky
if
you're
managing
secrets.
So
if
you're
managing
things
that
you
don't
want
to
log,
and
so
it's
very
easy
to
write
a
function
today,
put
the
instrument
macro
up
there.
A
Okay,
there's
another
question
coming
from
youtube,
so
jeff
barzeski.
I
hope
I
pronounced
that
even
remotely
correctly,
how
do
you
configure
tracing
to
send
its
data
to
the
various
back
ends?
Are
there
docs?
That
is
also
support
cloud
distributed
tracing
backends
like
aws
x-ray,
so
there
are
docs
absolutely
so.
If
you
go
on
the
tracing
subscriber
crate,
there
are
very
little
docks
on
how
to
add
different
subscribers,
the
tracing
pipeline.
A
I
think
the
work
that
has
been
done
in
open
telemetry
for
us
means
that
you
probably
have
an
implementation
of
the
standard
and
you
might
have
to
drive
your
own
subscriber,
probably
using
resolver.
That
should
be
not
too
complicated
to
actually
ship
it
to
x-ray,
but
I
haven't
used
it
personally.
So
I
don't
know
if
it's
out
there.
A
A
A
Okay,
once
again
from
solace
waffle,
do
you
have
an
approach
to
avoid
handling
fields
that
contain
personal
identifiable
information
in
telemetry
data?
Well,
the
approach
at
this
point
is
trying
not
to
put
up
that
which,
as
I
said
before
so
responding
to
the
other
question,
can
sometimes
be
tricky
because
of
the
way
instrument
works.
A
A
We
use
a
combination
of
this
error
and
anyhow,
so
we
use
this
error
for
all
the
places,
but
we
need
to
end
the
letters,
so
it's
very
nice
to
get
structured
names
and
you
can
match
on
and
then
do
different
things
depending
on
the
variant
and
then
when
we
just
want
to
report
errors.
So
we
just
want
to
have
something
that
we
log
on
with
the
time
to
people
as
a
response.
Then
we
use
any
out
and
we
generally
use
them
in
conjunction.
A
So
you
might
have
an
aerodynami
which
is
using
this
error
to
get
the
error
implementation
and
then
the
different
variants
are
actually
dropping
in
any
hour
and
what
we're
starting
to
do
recently
once
again
leveraging
the
tracing
crate
is
using
tracing
error.
So
capturing
spun
traces
in
our
errors,
so
that
when
we
get
logs,
they're,
actually
very
detailed
about
what
that
pinch
and
that
allows
us
to
debug.
A
A
Okay,
I
guess
that
means
it's
all
for
today.
In
terms
of
questions,
I've
been
asked
by
the
friends
of
rusty
bay's
tweeker
winner
for
a
manning
book,
promo
code.
I
assume,
in
terms
of
best
questions,
that's
going
to
be
solace
waffle
from
twitch,
so
I
think
you
need
to
stay
online
for
them
to
reach
out
to
you.
A
It
seems
there's
one
more
question,
though,
once
again
by
you,
so
that
doesn't
change
the
winner
anyway,
what
was
your
thought
process
for
deciding
to
build
this
project
in
brust?
Were
there
any
attributes
that
made
this
project
a
good
fit
for
the
first
production
grass
application
at
your
company
in
terms
of
the
application
itself?
A
Nothing
specifically
like
we're
talking
of
basically
a
client
of
an
api
that
we
suppose
publicly
is
going
to
power
a
ui,
so
it's
not
necessarily
doesn't
need
necessarily
to
be
the
fastest,
doesn't
mean
necessarily
all
the
guarantees
that
gives
you.
We
could
have
done
that
in
any
language,
but
we
were
looking
to
use
rust
for
other
types
of
projects
so
for
mission,
critical
projects
in
particular
to
leverage
rus
very
strong
type
system,
as
combined
with
this
very
predictable
performance
profile.
A
But
it's
somewhat
of
a
big
leap
to
adopt
a
new
language
when
driving
a
new
mission,
critical
project
only
to
find
out
when
you
actually
release
it
that
you
might
have
with
it
all
the
time.
So
this
was
a
very
nice
incremental
step
to
de-risk
the
technology.
So,
for
example,
look
at
all
the
observability
situations
say:
is
this
actually
ready
for
what
we
need
to
do
and
look
at
all
the
things
that
we
need
in
an
api?
And
can
we
actually
provide
apis
with
this
and
so
on
and
so
forth?
A
So
it
was
very
much
the
risk
operation
and,
as
with
the
risk
all
of
these
aspects,
then
it
became
possible
for
us
to
say
okay
now
we
can
confidently
bet
on
it
for
this
other
new
product
that
we
want
to
do
and
there's
a
huge
project,
and
that
fits
ras
profile
for
a
variety
of
reasons.
And
now
we
know
we're
not
risking
too
much
we're
still
taking
some
risk,
but
it's
not
as
big
of
a
risk
of
passing
from
small
cli
to
mission
critical
product
as
they.