►
Description
"Speaker: Lynn Root
Everyone’s talking about it. Everyone’s using it. But most likely, they’re doing it wrong, just like we did.
By building a simplified chaos monkey service, we will walk through how to create a good foundation for an asyncio-based service, including graceful shutdowns, proper exception handling, and testing asynchronous code. We’ll get into the hairier topics as well, covering topics like working with synchronous code, debugging and profiling, and working with threaded code. We’ll learn how to approach asynchronous and concurrent programming with Python’s `asyncio` library, take away some best practices, and learn what pitfalls to avoid.
Slides can be found at: https://speakerdeck.com/pycon2019 and https://github.com/PyCon/2019-slides"
A
B
B
And
so
I'm
a
staff
engineer
at
Spotify
for
the
past
few
months,
I've
been
building
infrastructure
to
help
support
those
that
do
some
like
machine
learning
models
to
do
digital
signal
processing.
So
that's
quite
interesting,
quite
fun,
I'm
also
fossa
Vangelis
at
Spotify
I
help
folks
release
their
code
under
the
Spotify
github
organization,
and
then
you
might
know
me
from
PI,
ladies
as
well,
and
if
you
don't
know
about
PI.
B
So
this
is
the
agenda
for
today
it
doesn't
look
like
it,
but
it's
pretty
jam-packed
we're
gonna
be
covering
some
graceful
shutdowns
exception,
handling
and
threading,
along
with
testing
debugging
and
profiling
and
I.
Think
I'll
probably
use
all
of
my
time,
but
it
will
save
the
questions
for
the
hallway
either
way.
This
presentation
is
pretty
code.
Heavy
I
will
show
this
link
at
the
end
as
well.
So
don't
worry,
but
the
slides
are
there
all
right?
So,
let's
get
to
it
a
sync
I/o,
the
concurrent
Python
programmers
dream
the
answer
to
everyone's
prayers
right.
B
B
So
we're
led
to
believe
that
we
were
able
to
do
a
lot
with
the
structure
at
a
sink
and
a
weight
API
layer,
some
tutorials
well
great
for
the
developer
to
get
their
toes
wet,
try
to
illustrate
real-world
examples,
but
they're
just
beefed
up
hello
world
examples.
Some
even
misuse
the
async
I/o
interface,
allowing
one
to
easily
fall
into
the
callback
through
the
callback
hell
that
we're
familiar
with,
and
then
some
I
get
you
easily
up
and
running
with
async
I/o.
B
So
for
me
my
problem
got
difficult
quickly
and
I'm,
so
allow
me
to
provide
you
with
a
real
bowl
example
that
actually
comes
from
the
real
world
and
I
recently
had
Spotify
actually
a
few
years
ago
at
Spotify,
and
we
built
a
service
that
does
periodic
hard
restarts
of
our
entire
fleet,
of
instance,
and
that's
what
we're
gonna
do
here.
Has
anyone
heard
of
that
chaos?
Monkey
yeah,
alright,.
A
B
We're
gonna
build
a
service
called
mayhem,
mandrel,
and
it
will
listen
for
a
pub/sub
message
and
then
restart
a
host
based
off
of
that
message
and
as
we
build
this
service
I'll
point
out
some
best
practices
that
I
have
learned
along
the
way,
slash
traps
that
I've
fallen
into,
and
this
will
essentially
become
the
type
of
resource
that
pass.
Lin
would
have
wanted
about
like
three
years
ago
and
so
we're
gonna
start
off
with
some
foundational
code.
We're
gonna
write
a
simple
publisher
and
here's
we're
gonna
start.
B
We
have
a
simple
while
true
loop
I,
you
a
unique
ID
for
each
message
to
publish
to
our
queue.
I
want
to
highlight
that
we're
not
using
the
wait
on
the
acute
output
of
a
message
we're
using
a
single
create
task,
because
it
will
actually
schedule
the
care
routine
on
the
loop
without
blocking
the
rest
of
the
for
loop.
The
create
task
method
returns
a
task,
but
we
essentially
are
using
this
as
like
a
fire-and-forget
mechanism.
B
B
So
now
that
we
have
a
publisher,
co-routine
function,
we
now
need
a
consumer.
So
this
is
our
consumer
for
the
published
message
messages.
It's
still
it's
kind
of
similar
to
the
publisher
itself.
We
do
have
a
while
true
loop,
but
then
we
await
on
the
q4
message.
We
don't
want
to
create
a
task
out
of
queue
duck
yet
because
it
doesn't
really
make
sense
to
block
it
makes
sense
to
block
the
care
routine
too,
because
you
can't
really
do
much
without
a
message
itself.
B
We
may
also
want
to
do
more
than
one
thing
per
message.
For
example,
in
addition
to
restarting
a
host,
maybe
we'd
like
to
store
that
message
in
a
database
for
potentially
replaying
later
on,
so
we'll
make
use
of
a
single,
create
task
again
for
saving
the
for
the
save
cover
team
to
be
scheduled
on
the
loop
and
basically
like
checking
it
over
to
the
loop
to
execute
for
when
it
can.
B
So.
In
this
example,
the
two
tasks
of
restarting
and
saving
don't
necessarily
depend
on
one
another
and
I'm
completely
sidestepping
the
concern
that,
if
we
should,
we
start
a
host.
If
we
can't
save
a
message.
But
maybe
you
do
want
your
work
to
happen.
Serially
maybe
do
not
want
to
have
concurrency
for
some
asynchronous
tasks.
So
for
this
instance,
maybe
you
want
to
restart
hosts
that
only
have
an
uptime
of
more
than
seven
days.
B
Just
like
you
should
check
the
balance
of
your
checking
account
before
you
actually
debit
it
so
needing
code
to
be
serial
to
have
steps
or
dependencies.
It
doesn't
mean
that
you
can't
be
asynchronous.
The
wait
last
restart
date
will
yield
to
the
loop,
but
it
doesn't
mean
that
restart
host
will
be
the
next
thing
that
the
loop
executes.
It
just
allows
other
thing,
other
things
outside
of
that
care
routine
to
run.
B
B
Saving
a
message
didn't
block
restart
host
of
needed,
so
we're
going
to
make
that
a
task
again
and
we're
just
going
to
remove
the
uptime
check
and
a
restart
host
indiscriminately,
because
Yolo,
so
we've
pulled
a
message
from
a
queue
and
found
out
work
based
off
of
that
message.
But
now
we
need
to
perform
like
any
finalization
work
from
that
message,
so
often
with
pub/sub
technologies.
If
you
don't
acknowledge
a
message
with
a
predefined
within
a
predefined
time
frame,
it
will
get
redeliver
due
for
a
finalization
task.
B
What
we
therefore
want
is
somehow
to
wrap
both
tasks,
since
we
have
to
wait
on
both
and
then
do
the
cleanup,
and
with
that
we
can
make
use
of
async
it
dot
gather,
which
returns
a
feature
like
object
and
with
that
future
like
object,
we
can
then
attach
a
callback
to
it,
and
then
we
can
just
a
wait
on
the
future
itself.
And
so,
when
we
run
this,
we
can
see
that
both
the
save
care
routine
and
the
restart
care
routine
are
complete.
And
then
the
cleanup
is
called
and
then
we
still
maintain
concurrency.
B
B
B
For
instance,
I
might
have
to
call
a
customer
service
of
my
bank
and
I'm
on
hold
and
I
can
put
that
down
and
put
it
on
speakerphone
and
then
play
with
my
super
needy
cat
as
I
wait.
So
I
might
be
a
single-threaded,
but
I
can
still
sort
of
multitask
like
CPUs,
so
often
you'll
want
to.
You
want
your
service
to
shut
down
gracefully
if
it
receives
a
signal
of
some
sort.
B
So
if
we
happen
to
restart
our
own
instance
of
our
mayhem
mandrel
service,
we
should
essentially
clean
up
our
own
mess.
So
here's
a
typical
boilerplate
code
of
getting
a
service
running.
We
have
a
queue
instance
and
setting
up
the
loop
scheduling,
the
publish
and
consume
tasks
and
then
start
and
then
close
the
event
loop,
maybe
even
catch
the
keyboard
interrupt
exception.
B
So
if
we've
run
this
and
as
is-
and
we
give
it
the
SIGINT
signal,
we
do
see
that
we
get
into
the
accept
and
finally
block
of
those
two
log
lines
that
we
had.
But
if
we
were
to
send
another
signal
to
our
program
like
sig
term,
we
can
see
that
we
don't
reach
that
finally
clause
that
we
have
defined
here.
B
So,
instead
of
catching
keyboard,
interrupt,
let's
attach
a
signal
handler
to
the
loop,
so
we'll
define
we'll
define
a
shutdown
care
routine
that
is
responsible
for
doing
all
of
our
necessary
shutdown
tasks.
And
here
what
I'm
doing
is
simulating
like
closing
the
database
connections.
Returning
the
pub/sub
messages
is
not
acknowledged
so
that
they
can
be
redeliver
and
not
dropped
and
then
collecting
all
the
outstanding
tasks
its
except
for
the
shutdown
task
itself
and
then
canceling
them
and.
B
B
So
let's
then
add
this
shutdown
care
routine
to
our
event,
loop.
The
first
thing
we
should
do
is
to
set
up
our
loop
and
then
add
the
signal
handlers
that
we
want
to
respond
to,
and
then
we
can
remove
that
keyboard
interrupt
catch
so
running
this
again.
We
see
that
we
do
get
to
that
finally
clause,
as
well
as,
like
the
whole
shutdown
log
lines
that
we
see,
so
you
might
be
wondering
which
signals
to
react
to
now,
and
apparently
there
is
no
standard.
B
B
Misleading
API
or
a
confusing
API,
at
least
for
me,
a
sink
I/o
debt
shield.
Now
the
docs
say
that
it
means
it's
a
means
to
shutdown
or
to
shield
a
future
from
being
like
cancellation.
But
if,
if
you
have
a
curtain
that
must
not
be
cancelled
during
shutdown,
async
up
that
shield
will
not
help
you
at
all,
and
this
is
because
the
tasks
that
that
async
I/o
debt
shield
creates,
gets
included
in
async,
I/o,
dot
all
tasks
and
therefore
receives
the
cancellation
signal.
Just
like
the
rest
of
tasks.
B
A
B
A
CGI
o
dot
shield
and
so
running
this
and
canceling
it
after
a
second.
We
see
that
we
don't
actually
get
to
that
done
line,
but
it's
immediately
canceled
and
to
be
honest,
I
actually
could
not
get
shield
to
work
under
any
sort.
Sansa's,
so
I've
taken
out
the
signal
handler
and
I
feel
like
this-
should
work
and
running
this
in
cancelling
and
after
a
second
I
still
don't
get
to
that
done
line
and
try
it
again.
B
I
thought
I
might
have
misinterpreted
the
docks,
so
I've
tried
canceling
the
carotene
from
another
care
routine,
but
no
I,
just
I
still
couldn't
get
it
to
work.
I've
tried
shielding
a
task
from
another
crew
team
that
raises
it
a
canceled
error
itself,
but
I
couldn't
get
that
to
work.
I
ran
into
URI,
one
of
the
core
developers
of
async
ago
and
I
was
complaining.
I
was
like
I
must
be
stupid,
but
he
told
me
that
no
async
I/o
is
stupid.
B
It
makes
me
feel
a
little
bit
better,
but
I
still
feel
like
I'm,
an
idiot
I'm,
probably
missing
something.
So
if
someone
in
the
crowd
come
to
me
afterwards,
tell
me
what
I'm
doing
wrong
that'd
be
super
helpful
anyways.
So
we
don't
necessarily
have
like
nurseries
to
help
help
us
clean
up
after
ourselves.
We
need
to
be
responsible
and
close
up
connections.
B
The
files
that
we've
opened
respond
to
our
outstanding
requests,
basically
leaving
things
how
we
found
them
and
doing
a
cleanup
in
a
finally
Clause
isn't
enough,
though,
since
a
signal
could
be
sent
outside
of
that
try
except
clause.
So
we
should
do
it
when
we
construct
the
loop
and
we
should
tell
how
the
loop
should
be
deconstructed
as
soon
as
the
program
gets
interrupted.
Some
way
this
ensures
that
all
of
our
bases
are
covered,
and,
finally,
once
we
once,
we
are
aware
of
how
our
program
should
be
shut
down.
B
B
So,
let's
revisit
our
restart
host
cur
routine
and
we're
gonna
add
a
super-realistic
exception,
and
so,
when
we
run
this,
we
do
see
that
that
super-serious
exception
was
raised,
but
that
we
also
get
a
task
except
exception
was
never
retrieved
because
we
don't
properly
handle
the
result
of
a
task
when
it
raises
what
we
can
do
is
define
an
exception
handler,
super-complicated
I
know,
and
then
we
can
attach
it
to
our
loop.
Much
like
the
signal
handling.
B
And
so
when
we
rerun
this,
we
see
that
our
logging
of
exception
does
happen
oops
and
we
don't
get
that.
Try
or
the
task
cannot
be
retrieved,
and
so
we've
set
exception.
Handling
on
the
global
level
is
like
a
global
default,
but
perhaps
you
want
to
treat
some
exceptions
differently
for
certain
tasks,
so
we're
going
to
revisit
our
handle
message,
care
routine
here
so
say,
for
instance,
you're
fine
with
a
logging
when
the
save
message
fails.
B
So
I
guess
be
sure
to
set
some
sort
of
exception,
handling
either
globally
or
individually,
or
you
probably
want
a
mix.
Otherwise,
exceptions
may
go
unnoticed
or
cause
some
weird
behavior
I
personally,
like
async
IO
dot
gather
because
of
the
order
of
return
results.
It
is
deterministic,
but
it's
easy
to
sort
of
get
tripped
up
with
it
by
default.
It
will
swallow
sort
of
exceptions
and
then
happily
continue
working
on
the
other
tasks
that
it
was
given.
So
if
an
exception
is
never
returned,
then
weird
behavior
can
happen
all
right.
B
Sometimes
you
need
to
work
with
threads
and
I'm.
Sorry,
if
you
do
so,
maybe
you
have
like
a
threaded,
pub/sub
client
and
you
to
consume
a
message
on
one
thread
and
then
handle
the
message
on
the
care
routine
with
a
curtain
on
your
main
event
loop.
So,
let's
first
attempt
to
use
the
async
I/o
API
that
we're
familiar
with
and
update
our
synchronous
callback
function
with
creating
a
task
via
a
single,
create
task
and
create
a
task
out
of
handle
message.
B
So
if
we
take
what
we
have
right
now
and
update
our
function
to
use
the
main
event
loop,
if
we
try
running
this,
it
looks
like
it
works,
but
it's
actually
deceptive
we're
not
being
thread
safe.
So
we're
gonna
try
this
again.
So,
instead
of
a
loop
dot,
create
task,
we're
gonna
make
use
of
the
thread
safe,
API.
That
should
give
us
a
clue
right.
B
So
I
guess,
in
my
opinion,
it's
not
like
too
difficult
to
work
with
threaded
code
and
async
I/o,
it's
similar
to
how
we
work
with
non
async
code
in
the
async
world,
we'll
make
use
of
like
the
thread
pool
executor,
which
essentially
creates
an
avoidable
for
us.
However,
it
can
be
difficult
in
the
sense
of
when
you
need
to
share
state
between
both
of
them
and
when
you
want
the
thread
to
schedule
something
on
the
main
event
loop.
B
B
Since
the
save
care
team
we
need
to
is
a
curtain,
we
need
to
run
it
in
our
event
loop,
like
so,
and
which
Python
3:7.
That
makes
it
easy
for
us
with
the
async,
I/o
dot
run
function,
older
python
versions,
python,
3
versions,
we'll
have
to
construct
and
deconstruct
the
loop
yourself,
but
some
of
you
might
be
saying:
there's
a
better
way,
and
there
is.
There
is
a
PI
test.
Plugin
called
PI
test,
async
I/o.
That
will
essentially
do
that.
The
hard
work
for
you.
B
You
just
need
to
mark
the
particular
tests
that
they
are
async
code.
That
needs
to
be
run
with
this
decorator
and
you
need
to
make
the
test
function
itself
a
care
team.
Now,
when
running
the
tests,
the
plug-in
will
essentially
do
the
work
for
you
of
constructing
and
deconstructing
the
event.
The
event
loop,
so
the
PI
test,
async
I/o
plug-in,
can
get
you
pretty
far,
but
it
doesn't
help
you
when
you
need
to
mock
out
co-routines.
B
Now
you
don't
actually
want
to
wait
for
async
IO
dot
sleep
to
complete
while
you're
running
the
tests,
or
do
you
actually
want
to
call
a
database
while
you're
running
your
tests,
and
so
both
the
unit
tests,
mock
library
and
the
PI
test
mock
package?
Do
not
support
asynchronous
mocks,
so
we'll
sort
of
have
to
work
around
this
and
we
are
going
to
make
use
of
the
PI
test
mock
lot
library
and
create
a
fixture.
Now,
that's
essentially
returning
a
function.
B
So
then
we're
going
to
create
another
PI
test
fixture
that
will
use
this,
create
co-op
fixture
that
we
just
defined
to
mock
and
patch
the
async
I/o
dot
sleep,
and
we
don't
need
the
stub
care
routine.
That
is
returned,
so
we
can
just
sort
of
throw
that
away
and
then
we
can
use
the
mock
sleep
fixture
in
our
test
save
function.
B
So
what
we've
done
is
basically
patched
a
sync
I/o
dot
sleep
in
our
mayhem,
module
with
the
a
stub
co-routine
function,
and
then
we
can
assert
the
mock
mock
to
Isengard
that
sleep
object
was
called
once
the
mayhem
dot,
save
care
routine
was
called
because
we
now
have
a
mock
object
instead,
instead
of
the
actual
care
routine,
we
can
now
do
anything.
That's
supported
with
mocks.
We
can
do
assert
called
once
with
and
play
with
the
return,
values
and
side
effects
as
we
need
etc.
B
Remind
you
when
I'm
testing
I
do
tests
with
to
assert
that
parameters
were
called
as
accordingly
and
not
just
the
simple
count
thing.
So
that's
simple
enough,
but
sometimes
you
need
to
test
code.
That
is
using
the
create
task
method
and
we
can't
simply
use
the
create
curve
or
mock
fixture
that
we
defined
a
second
ago.
B
So
first
need
a
couple
of
fixtures
for
that
Q
we
will
mock
and
patch
the
async
a
OQ
class
within
our
module
and
then
we'll
use
the
mock
Q
fixture
in
another
one
in
the
mock
get
fixture.
And
so,
unlike
our
mock
sleep
fixture,
we
will
use
that
stub
care
routine,
that
create
mock
Co
returns
and
set
it
to
the
mock
queues.
B
Get
so
here
is
our
test
consume
function
where
we
are
giving
the
newly
created
fixtures,
and
so
we
will
try
to
use
the
create
coder
mock
to
mock
and
patch
the
call
to
a
handle
message.
Carotene,
a
via
that
create
task
and
I
want
to
highlight
that
we're
setting
the
mock,
get
side-effect
to
one
real
value
and
one
exception
to
make
sure
that
we're
not
permanently
stuck
in
that,
while
true
loop
that
the
consume
has.
B
And
finally
we
want
to
assert
that
our
mock
for
handle
message
has
been
called
after
consume
has
been
running
so
when
running
this,
we
see
that
mock
handle
message
does
not
actually
get
called
like
we
were
expecting,
and
this
is
because
these
scheduled
tasks
are
only
scheduled
and
pending.
At
this
point,
we
sort
of
have
to
nudge
them
along
and
so
to
do
this
and
we
kind
of
collect
all
the
running
tasks,
except
for
the
test
itself,
and
then
it's
run.
This
explicitly
I
know.
A
B
Is
a
bit
clunky?
Perhaps
this
isn't
an
opportunity
to
contribute
to
PI
test
during
a
sprints,
but
if
you
use
the
yes,
the
standard,
the
unit
test,
a
library
from
the
standard
library
there's
a
package
called
async
test
that
handles
this
and
it's
a
bit
better
and
it
exhausts
the
scheduling
of
tasks
for
you.
B
B
We
can't
exactly
use
the
the
event
loop
fixture
that
the
PI
test,
async
I/o
library,
gives
us.
We
need
to
manipulate
that
fixture
that
gets
ends
up
getting
injected
into
our
code.
I
mean
do
this
by
updating
the
event
loop,
so
we
can
override
the
clothes
behavior.
If
we
close
the
loop
during
the
test,
we
actually
lose
access
to
the
exception
and
signal
handlers
that
we
set
up
within
the
main
function.
B
B
B
So
we're
pretty
decent
programmers
and
we
have
good
code
coverage,
but
sometimes
stuff
breaks.
We
need
to
figure
out
what's
going
on,
so
we
can
use
everyone's
favorite
debugger,
printing
sort
of.
So,
if
you
have
just
one
small
thing
to
debug,
it
might
help
to
use
the
print
stack
method
on
task
instance.
And
so,
when
you
run
this,
you
can
see
that
the
print
attack
will
like
you'll,
see
the
stack
for
every
running
task
and
you
can
increase
the
number
of
frames
that
are
printed
as
well.
B
B
We
can
see
that
the
TAS
was
never
achieved,
but
we
also
get
information
about
the
task
that
it
was
affected
and,
what's
called
a
source,
trace
back
to
give
us
more
contacts
in
addition
to
the
trace
back
so
without
debug
mode,
we
get
told
that
there
is
an
exception:
that's
not
properly
handled
and
with
debug
mode.
It
gives
us
more
additional
clues.
What
might
be
happening?
B
Another
very
handy
thing
wish
I
knew
a
few
years
ago.
Is
it's
able
to
tell
you
if
your
thread
safe?
So
if
you
got
the
thread,
if
you
got
threads
an
event
and
the
event
loop,
interacting
with
each
other,
the
bug
mode
will
surface
any
non
thread.
Safe
operations
for
you
as
a
runtime
error,
and
then
one
really
nice
feature
about
debug
mode
in
async.
Io
is
how
it
sort
of
acts
like
a
tiny
little
profiler
that
will
log
asynchronous
calls
that
are
slower
than
100
milliseconds.
B
B
B
So
much
like
some
people's
testing
philosophies.
Sometimes
you
need
to
debug
in
production,
but
usually
don't
want
full-on
debug
mode
while
in
production.
So
there's
a
lightweight
package
called
a
IO
debug
that
will
log
slow
callbacks
for
you,
and
it
also
comes
with
the
ability
to
report
delayed
calls
to
stats
D.
If
you
use
that
now,
that's
the
only
thing
that
the
mod
or
that
the
package
will
do
so.
It's
super
lightweight
so
to
wrap
up
debugging.
B
You
can
easily
print
the
stack
of
the
tasks
if
needed,
but
you
can
also
get
a
lot
with
async
IOT's
debug
mode.
It
gives
more
information
around
unhandled
exceptions
when
you're
not
being
thread
safe
and
when
they're
slow
to
complete
tasks,
and
if
you
want
to
understand
so
to
complete
tasks
in
production,
a
o
debug
is
a
lightweight
library
that
essentially
does
only
that
all
right.
B
So,
as
we
saw
with
ASIC
IOT's
debug
mode,
the
event
loop
can
already
track
her
routines
that
take
up
too
much
CPU
time
to
execute,
but
it
might
be
hard
to
tell
what
is
anomaly
and
what
is
a
pattern.
So
folks,
a
lot
of
folks
might
first
reach
for
a
seat
profile
to
try
and
understand
performance,
and
we
can
try
that
out
here
too,
but
there's
not
much
to
actually
glean
from
it
I
kind
of
snip
it
for
the
slide
a
little
bit.
But
the
top
item
here
is
essentially
the
event
loop
itself.
B
B
Now,
if
you
limit
the
view
to
our
own
script
and
clicking
around
the
curtain
functions,
we
can
kind
of
get
an
idea
of
where
time
is
spent.
This
visualization
kind
of
groups
modules
together
by
color,
and
so
when
I
was
first
actually
profiling.
This
I
saw
a
lot
of
blue
and
you
can
kind
of
click
through
and
get
a
little
more
information,
and
you
realize
or
I
realize
that
all
that
blue,
that
we
see
is
related
to
logging.
Now
we're
gonna
hold
on
to
that
thought.
B
So
Kay
cash
grinding
will
allow
us
to
get
sort
of
a
broad
picture
of
what's
going
on
and
give
us
some
visual
clues
of
where
to
look
for
potential
areas
of
unnecessary
time
spent
and
then
there's
a
line,
profiler
package,
and
with
this
we
can
kind
of
hone
in
on
areas
of
our
code
that
we
were
suspicious
of.
So
after
installing
the
line
profiler,
you
can
add
the
profile
decorator
where
you
want
to
profile
them,
and
here
I'm
only
a
decorating
with
the
save
Co
routine.
B
So
the
line
profiler
library
comes
with
a
CLI
tool
called
Kern
prof
that
we
will
invoke
our
script
with,
and
then
we
render
the
output
with
the
align
profile
module
itself,
which
gives
us
a
line-by-line
assessment
of
our
decorated
code.
So
the
total
time
spent
here
in
this
function
is
just
over
two
milliseconds
and
then
majority
of
that
time
was
spent
in
logging.
Now,
if
only
there
was
something
we
could
do
about
that.
B
So
if
we
switch
out
our
default
logger
with
a
io
logger
and
rerun,
the
profiler
I
can
see
that
our
total
time
spent
on
the
function
is
halved,
as
well
as
a
time
spent
logging
itself,
and
so
certainly
these
are
like
minuscule
improvements
that
we're
doing
here.
But
you
can
extrapolate
on
a
much
larger
scale
and
also,
as
I
see
it
if
we
have
an
event
loop,
let's
try
and
make
take
full
advantage
of
it.
B
So
we've
profiled
a
C
profile
with
C
profile
in
with
a
line
profiler,
but
we've
had
to
stop
the
service
in
order
to
look
at
results.
So
perhaps
you
would
like
a
live
profiler
to
go
along
with
your
production,
debugging
there's
this
package
called
a
profiling
that
provides
an
interactive
UI
and
supports
async
I/o,
as
well
as
threads
and
green
LEDs.
Granted
you
can't
attach
to
a
running
process
with
this
particular
tool.
B
So
to
get
initial
picture
of
your
services
performance
using
C
profile
with
K
cache
grind
can
help
surface
areas
to
investigate.
Without
that
visualization
it
can
be
difficult
to
identify
hotspots.
But
then,
once
we
do
have
some
sort
of
idea
of
where
hotspots
are,
you
can
use
a
line
profiler
to
get
a
line-by-line
performance
data
and
then,
finally,
if
you
want
to
profile
with
production,
data,
I
would
take.
I
would
suggest
taking
a
look
at
the
profile
package.
Profiling
package.
B
Alright,
that
was
a
lot,
but
in
essence,
this
talk
is
something
that
I
would
have
liked
to
have
like
three
years
ago,
so
I'm
kind
of
talking
to
pass
Lin
here,
but
I'm,
hoping
that
someone
else
or
you
all
have
some
benefit
from
this
use
case.
That
is
not
a
web
crawler,
so
I
know
it's
lunchtime,
but
I
will
be
around
here
through
all
the
conference
as
well
as
the
sprints.
So
if
you
have
questions
you
can
come
find
me,
but
thank
you.