►
From YouTube: Getting Started with Logging in Kubernetes - Eduardo Silva, Treasure Data (Any Skill Level)
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Getting Started with Logging in Kubernetes - Eduardo Silva, Treasure Data (Any Skill Level)
A good practice when deploying applications in Kubernetes is to set proper instrumentation to gather insights and solve general monitoring needs. Logging is a fundamental piece of the instrumentation cycle and is continually evolving to solve pains associated with unstructured formats, performance and monitoring. In this presentation you will learn the concepts involved in log processing for containerized applications. You will also be introduced to these hot new features in Logging: metering the logging pipeline with Prometheus, performance improvements, scalability and the ability to customize the log processor behavior through declarative resource annotations.
About Eduardo
Eduardo is an Open Source Software Engineer at Treasure Data. He currently leads the efforts to make logging more scalable in Containerized and Orchestrated systems such as Kubernetes.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Good
I
know
that
it's
a
bit
late
after
lunch
is
not
always
a
good
time
to
attend
a
talk.
So
thanks
for
coming,
please
take
a
seat
yourself,
eh.
My
name
is
Eduardo
Silva
I'm,
a
software
engineer
at
this
company,
which
is
called
a
treasured
data
and,
as
the
company
name
said,
we
care
a
lot
about
data.
We
have
a
assess
platform,
so
you
can
ingest
your
data,
your
marketing
data
or
whatever,
and
you
can
get
some
insights
from
your
data.
A
But
when
you
have
some
kind
of
platform
to
manage
data,
there
is
one
missing
step
you
can.
You
can
have
a
platform,
but
what
really
gives
you
the
value
is
the
data
and
that's
why
we
got
into
this
world
of
logging
like
four
years
ago,
creating
some
data
tools
in
an
open
source
way
to
collect
data
from
your
old
services
or
your
own
applications.
A
So,
and
this
talk
is
about
how
to
get
started
with
login
in
coronaries.
Our
company
is
one
of
the
sponsors
also
of
a
project
flew
in
D
and
flew
in
bed.
Would
you
please
raise
your
hand
whose
use
you
flew
in
D
right
now?
Ok,
this
is
micro
thanks,
so
I
said:
I
am
a
mobile
software
developer
at
Trisha.
Dara
I
am
part
of
a
fluent
D
open
source
team,
but
my
primary
role
is
also
maintain
and
develop
it.
A
This
little
project,
which
is
called
flow
in
bed,
that
maybe
you
are
hearing
a
lot
about
it
in
the
last
year.
So
let's
talk
about
applications
and
login.
When
you
have
a
some
kind
of
application,
they
all
the
way
to
perform.
Login
is
just
to
write
some
message
to
some
special
place,
and
sometimes
this
is
the
hardest
or
a
log
file,
and
that
for
that,
for
us
is
called
logging.
A
If
you
are
over
30
years
old,
you
realize
that
you
play
a
lot
with
syslog
file
system
log
files,
if
you
are
in
your
20s,
maybe
you're
looking
on
what
is
that
you're
using
Journal
d
or
something
more
fancy
right
now:
right:
okay,
but
login
a
happens
in
different
ways,
not
just
to
the
file
system.
When
you
have
a
unique
process
running
in
your
environment.
Actually
you
have
like
three
main
channels
of
communications.
A
The
first
one
is
a
standard
input
and
which
can
be
used
to
send
messages
to
the
application
when
you're
using
your
terminal,
and/or
stand
Mac,
OS
or
Linux,
and
you
type
some
common
and
put
a
pipe.
What
you're
doing
it
just
redirecting
the
output
of
one
program
to
the
input
of
the
other
and
piping
happens
because
of
this
streaming
interface.
But
when
we
talk
about
login
in
general
and
specifically
in
the
container
world,
we
are
talking
about
this
hey.
We
have
some
lag
here:
Oh
battery.
A
There
you
go
I'm
going
to
stay
on
this
side
so
and
the
login
happens
specifically
on
the
standard
output
and
understand
that
error
interface.
And
if
you
are
dealing
with
containers,
do
you
realize
that
this
sounds
a
little
bit
familiar
so
log
in
and
in
talker
containers?
I
assume
that
most
of
you
know
most
of
you
but
help
you
are
using
fluently,
but
who's
using
docker
at
the
moment
in
sublevel,
okay,
80
percent
cool,
so
undocking
has
also
its
own
strategy
to
perform
logins.
A
Now,
first
of
all,
I
understand
that
you
know
what
a
container
is
right.
A
container
is
not
docker.
Token
is
just
a
wrapper,
a
tool
that
allows
to
create
this
concept
of
container
a
container
is
a
process
random
in
a
container
I
set
space
which
is
restricted
by
the
Linux
kernel,
meaning
that
you
have
some
namespace
and
cgroups,
which
allows
to
say
to
this
application.
A
You
only
have
access
to
the
filesystem
or
maybe
not
you
have
access
to
a
network,
or
maybe
this
amount
of
CPU
time,
but
dealing
with
those
kind
of
system
calls
or
using
the
common
light
with
Alexi
takes
some
time
and
that's
why
docker
allows
you
to
provide
a
full
interface
on
top
of
that
and
manage
containers,
and
also
it
cares
about
logins.
So
if
one
application
sent
a
message
like
hey,
you
come
actually
what
will
happens
is
that
docker
will
say.
A
And
if
you
look
carefully
I,
don't
know
how
you
look
at
that
the
log
files
of
docker,
ok,
yeah,
nobody
do
it,
you
don't
want
to
do
it
right,
because
it's
like
a
path,
a
lot
of
hash
and
things,
and
not
very
clear
about
that.
You
don't
need
to
it's
a
waste
of
time
unless
you
are
trying
to
create
some
login
solutions.
A
Also,
I,
don't
know
if
you
look
carefully
in
the
previous
slide,
but
the
message
is
stored
in
one
key,
which
is
called
lock
right.
The
low
key
is
important
one,
because
that
is
the
one
they
contain
the
message
and
of
course
you
want
to
obtain
some
kind
of
metadata.
So
let's
try
to
look
very
quickly
about
how
this
works
locally.
A
So
if
I'm
not
wrong
I'm
going
to
use
my
comments
here,
so
we
don't
waste
time
with
typos
I
learned
that
in
the
past
conference,
so
basically
we
are
going
to
run
a
docker
container
with
busybox
and
just
print
out
the
message
but
I'm
running
with
a
demon
flag,
which
means
that
runs
behind
the
scenes,
but
also
will
give
me
the
container
ID.
That
is
the
one
that
I
want
to
use
later
to
gather
the
logs
in
the
filesystem.
A
Of
course,
you
can
use
the
token
locks
common,
but
here
we
are
doing
the
manual
stuff.
Okay,
this
is
the
container
ID
in
the
container
hash.
Okay,
of
course,
you're
not
going
to
remember
that.
But
what
I
want
to
do
is
to
dig
into
my
file
system
and
see
where
this
exists:
I'm
going
to
become
sudo,
Bar
leave
docker
containers.
Oh,
we
have
a
bunch
of
containers,
so
is
that
fine
or
too
small?
Okay,
thanks
you're
still
young
I'm?
A
A
If
I
cut
the
log
file,
I
will
find
my
message
here.
Is
it's
a
JSON
message
in
the
filesystem?
Now
we
can
do
this
like
with
JQ,
it's
more
beautiful,
okay!
That's
it
your
application
to
get
a
message
you
got
outside
through
some
streaming
interface
and
then
it
wraps
by
the
docker
engine.
Dr.
engine
creates
a
new
log
file
and
this
log
file
can
be
seen
by
everybody
or
the
docker
tools.
That
does
exactly
the
same
that
I'm
doing
right
now.
A
Okay,
I
wanted
to
show
you
that,
because
you
need
to
understand
that
before
to
move
forward
so
present
so
applications
in
kubernetes
a
quick
overview,
who's
running
kubernetes
right
now.
Ok,
so
I
got
an
idea.
Ok,
pretty
quickly!
You
are
going
to
see
the
concepts
you
have
one
application
in
docker,
you
have
one
application
runs
in
a
container
and
that's
fine,
but
here
we
have
a
few
additions.
For
example,
an
application
runs
in
a
container,
but
for
kubernetes
a
container
is
not
like
an
object
or
a
concept.
What
kubernetes
is
about
knows
about.
A
It
is
a
pot
and
the
pot
is
a
concept
which
allows
you
to
group
different
containers.
So,
for
example,
if
you
have
a
web
server
and
you
have
some
database,
you
can
run
both
in
the
same
pod
because
it
belongs
to
the
same
context.
That's
an
example.
You
can
run
it
on
that
way
or
not
ok,
but
you
can
have
many
a
containers
running
in
the
same
pot.
Usually
you
have
one,
but
you
can
have
many,
but
a
node,
a
node
I
mean
a
host.
A
physical
machine
or
a
beautiful
machine
can
have
many
pots.
A
That
means
multiple
applications,
and
you
know
that
kubernetes
cluster
has
many
nodes,
so
they
think
is
how
do
you
deal
with
blogging
when
you
have
a
cluster
when,
for
example,
you
deploy
your
application,
your
application
got
some
replica
replicas
in
different
nodes.
How
do
you
gather
the
login,
how
you
centralize
the
logs,
because
kubernetes
is
really
good
for
this?
A
It
has
a
scheduling
you
can
create
your
own
up,
kion's,
self-healing
and
all
the
things,
but
for
logging
we
have
new
challenges:
it's
not
enough
to
go
to
the
file
system,
as
I
did
right
now
and
query
each
log
file,
because
ten
years
ago,
if
you
have
some
issue
in
some
application,
you
used
to
SSH
into
each
server
or
use
some
kind
of
remote
syslog
to
do
some
troubleshooting.
But
here
the
things
change
is
a
game
changing
and
I
would
say
that
is
for
good
for
who
manage
login.
A
Of
course,
we
need
to
do
more
work
actually
I'm
sure
that
nobody
here
finds
that
login
is
fun
because
not
login
is
not
fun,
but
if
you
are
here
is
because
it
had
to
deal
with
it
right
and
we
had
to
fix
it.
So
a
login
context
in
kubernetes
in
Indo
Kern,
we
used
to
have
like
the
pipeline
standard
output,
maybe
in
the
log
message,
and
we
have
in
the
time
stamp.
But
if
you
are
running
in
kubernetes
there
is
some
extra
information,
for
example
the
pot
name
which
pot.
A
This
belongs
to
the
poor
lady,
because
maybe
you
can
have
the
same
pot
name
but
restarted
like
ten
times,
and
the
Paradis
will
be
different,
the
namespace
that
it
belongs
to
the
node,
the
labels
and
annotation.
Do
you
know
what
are
the
labels
No?
Okay,
imagine
that
you
have
several
applications
and
you
have
different
environments.
For
example,
you
have
production,
you
have
testing
and
development.
A
How
do
you
make
some
differentiation
between
them?
Okay,
I,
create
namespaces
cool.
You
can
create
namespace
and
group
the
port's
associated
with
development
on
that
namespace
or
in
production
or
testing
whatever.
But
if
you
want
to
query
the
API
server
in
kubernetes
to
say,
please
show
me
all
databases
from
the
three
environments.
You
use
levels
means
that
when
you
create
the
port,
you
attach
some
kind
of
label
to
the
pot.
You
say,
and
this
is
a
new
application
version.
Four.
A
So
if
we
think
about
the
log
processor,
we
need
we
need
to
think
about.
We
have
the
logs.
That
is
a
message,
but
we
need
to
understand
what
is
the
context
of
that
message,
because
it
belongs
to
a
container
name
to
a
pod,
namespace
hot
labels
and
annotation.
If
we
cannot
get
the
context,
the
information
is
irrelevant
so
from
a
log
processor
perspective
I'm
talking
about
in
a
general
way,
it
doesn't
matter
the
log
processor
that
you
are
running
it
had
to
deal
with
this.
A
When
you
have
a
kubernetes
cluster,
you
have
an
API
server
or
which
is
called
a
master,
and
the
master
knows
about
well
through
it.
Etcd
knows
all
the
states
of
imports
or
all
the
nodes.
So
if
I'm
running
a
port
in
node
B,
the
API
server
knows
what
are
the
labels
and
annotations
associated
with
this
pot
running
on
that
node.
So
at
some
point
you
need
to
take
the
whole
information
and
merge
it.
A
A
So
it's
like
this.
If
you
are
doing
logging
it
because
you
want
to
do
some
data
analysis,
but
in
order
to
perform
data
analysis,
do
you
want
to
centralize
your
information?
Otherwise
you
cannot
do
it.
Okay
and
the
whole
deal
of
LOC
processing
and
login
in
kubernetes
in
general,
is
that
you
need
to
check
the
information
from
multiple
sources
unify
this
information
in
a
central
like
a
database.
A
This
is
how
it
works.
Basically,
do
you
know
what
is
a
demon
set?
Okay,
everybody
knows
what
is
a
pod
yeah
I
just
say
that
right
now,
pod
is
a
container
which
is
running
in
a
node.
A
demon
set
is
a
special
port
which
run
on
every
node
of
your
cluster.
If
you're
going
to
deploy
your
application-
and
you
said
okay,
this
is
a
pod.
No,
it's
not
a
pod
is
going
to
be.
A
So
if
we
want
to
solve
log
in
in
kubernetes,
we
want
that
our
log
processor
runs
on
every
node
of
your
cluster,
because,
if
you
imagine
that
you
have
just
won
a
okay
one,
node
one
built
on
machine
and
the
spiritual
machine
has
many
ports.
All
the
logs
from
those
ports
are
in
the
file
system
own
in
journal.
D.
On
that
specific
environment,
the
API
server
doesn't
know
anything
about
logging,
okay.
So
what
you
need
to
have
is
to
have
the
log
processor
running
locally
in
the
node.
A
A
So
what
we
do
is
basically
deploy
the
low
processor
as
on
set,
and
then
we
start
reading
the
logs,
because
we're
going
to
read
each
container
log
file
and
that
is
expensive
by
Network.
It
doesn't
matter
if
it
is
file
system
or
Journal
D,
it
will
be
expensive,
anyways
expensive
in
terms
of
computing,
and
then
we
need
to
go
with
the
second
steps,
which
is
gather
the
metadata
labels
annotations
from
the
API
server,
because
we
want
to
merge
this
information.
A
Let's
make
sure
that
we
have
okay,
everything
is
fine
right.
This
is
my
own
single
stuff.
What
I'm
going
to
do
now
is
I'm
going
to
deploy
a
pot,
an
application,
and
this
application
is
just
a
fake
simulator
of
Apache
web
server.
So
what
it
does
it
just
to
write
a
simple
log
file
as
a
simulate
Apache
log
files
to
the
standard
output
randomly
okay,
everyone
every
second,
okay.
This
is
the
base
image.
If
you
have
your
computer,
you
can
run
it.
A
A
Okay,
so
keep
CTL
create,
but
patchy
that
demo
so
keeps
it
here,
get
pods
watch.
Oh
it's
a
random.
So
let's
use
now
the
the
kubernetes
a
login
tool
to
see
the
logs
from
that
specific
pot.
So
it
will
be
cube,
CTO,
logs,
Apache
logs
and
follow
the
locks
there
you
go
so
the
pod
is
running,
but
right
now
those
who
logs
are
just
in
the
filesystem
of
mini
cube,
and
we
can
dig
into
that.
If
you
do
mini
cube,
SSH
just
for
demo
purposes,
bar
leave,
containers.
A
A
You
get
the
JSON
message,
for
example
this
one
okay,
that's
fine!
Until
now,
there's
nothing
fancy
okay,
I'm
going
to
exit
from
mini
cube,
okay,
cube,
City
I'll
get
pods
Apache
logs
is
running,
but
what
I
wanted
to
do
now
is
to
install
my
log
in
my
log
processor,
as
a
daemon
set,
run
locally,
take
out
the
logs
process,
the
loss
and
injustice
logs
into
a
database
for
demo
purposes
and
to
use
elasticsearch.
A
We
will
see
if
that
is
working,
because
I
reduce
the
memory
usage
right
now.
So,
okay,
it's
still
alive
cool
and
just
for
sanity
purpose.
We
are
going
to
wipe
the
database.
Don't
do
this
at
home
or
in
production.
This
is
a
magic
key.
Oh
there
you
go
so
if
we
do,
for
example,
we're
using
cool
like
the
HTTP
client
to
query
the
database
to
see
if
we
have
some
records
cut
in
this
is
pretty
oh
there's
nothing.
We
have
nothing
cool
it's
empty,
so
the
next
step
is
to
deploy
the
demon
set.
A
Of
course,
if
you
want
to
install
a
log
processor,
you
just
need
to
get
the
right
documentation.
The
right
comments,
for
example,
I'm,
going
to
show
the
namespaces
and
have
created
before
a
namespace
which
is
called
login,
so
I
deploy
my
demo
set
a
namespace
which
is
called
logging
just
for
sons
having
something
looking
good.
Okay,
there's
no
relevant
thing
here
and
I'm
going
to
deploy
a
fluent
bit
fluent
bit,
which
is
a
log
in
log
processor,
I'm,
going
to
talk
more
about
it
later.
So
we're
going
to
deploy
is
a
config
map.
A
Yes,
ok,
when
you
have
a
container
running
and
you
want
to
ingest
configuration,
you
have
two
ways
or
you
create
your
own
container
and
when
you
create
a
container
put
the
log
file
inside
the
container.
But
if
you
want
to
change
the
configuration
you
have
to
rebuild
it
or
you
can
pass
the
configuration
files
as
mounted
volumes.
So
when
the
container
start
it
can
see
the
files
as
a
volume
that
was
mounted
ok
and
a
config
map
is
an
object
in
kubernetes
that
allows
you
to
ingest
configuration
inside
the
pot.
A
This
is
the
input
from
where
I'm
going
to
collect
the
logs
right
now,
I'm
going
to
process
only
the
Apache
logs,
I'm,
going
to
say
parser
for
docker
and
going
to
filter
the
data.
What
a
filter
does
on
this
case
go
to
the
API
server,
go
to
the
labels,
metadata
annotations
and
merge
that
and
then
I'm
going
to
ship
this
information,
because
this
is
the
output
to
elasticsearch.
A
A
From
paid
config
map,
okay,
ready
and
I'm
going
to
deploy
flow
n
bit
the
demon
set
for
mini
cube.
This
is
the
docker
image
for
fluid
camp.
Then
I'm
pushing
right
now.
That
version
was
released
at
this
Monday.
So
it's
like
the
new
version,
so
cube
CTL
create
des
mini
cube.
There
you
go
so
kissing,
get
ports.
A
A
Discover
locks
and
we
are
going
to
create
the
timestamp
thing
choose
that
we
need
to
choose
the
right
things.
Otherwise
we
don't
get
the
locks-
and
here
here
the
locks,
for
example,
I'm,
going
to
choose
this
one,
this
entry
and
then
going
to
the
JSON
that
was
ingested
into
the
database.
If
you
look
carefully
here,
we
have
the
stream,
the
timestamp,
and
we
have
this
new
addition
of
information
in
that
information
comes
from
the
API
server.
So
when
you
have
centralized
all
your
logs
in
your
database
now
you
can
query
the
information
you
can
say.
A
Please
show
me
all
the
logs
that
a
pot
name
belongs
to
a
patch
and
you're
going
to
get
everything
about
the
patch
okay.
So
that
was
a
quick
interview,
so
I
review
about
that.
So
I
know
that
most
of
you
are
using.
They
are
using
flu
and
D,
and
flu
in
D
is
more
than
just
a
strong
logging
solution.
Actually,
flu
and
D
is
a
it's
a
fluent
ecosystem
and
flu
in
D
I,
don't
know
if
you're
worried
about,
but
flu
indeed
join
at
the
CN
CF.
It's
an
inception.
A
Project
incubation
break
sorry,
and
we
are
in
the
process
to
a
full
graduation,
maybe
in
a
couple
of
months
or
before,
the
next
cube
calm.
But
flu
India
also
has
like
a
huge
community.
We
have
like
700
plugins,
available
for
storage
in
Casca
elasticsearch,
my
sequel,
Reddy's
and
so
on
on
different
filters,
and
when
we
donated
this
also
a
there's
a
strong
commitment
from
the
company
to
continue
growing
fluency
in
all
aspects,
even
the
community,
for
example.
A
A
In
some
scenarios
we
set
a
creating
a
new
solution
and
a
tentative
solution,
which
is
called
flow
in
bed
and
flow
in
bed,
is
like
a
C
language
version
of
windy,
but
F
with
a
strong
focus
on
kubernetes
and
docker
containers
and
Fulham
beat
sealed
at
13
was
released
just
this
week
and
one
of
the
future
is
annotated.
Pots.
I,
don't
know
if
you
remember
when
you
look
at
the
elasticsearch
stuff,
for
example,
we
don't
have
too
much
time,
but
we'll
try
to
do
it
very
quickly,
for
example
the
Apache
log.
Here,
where
is
look?
A
You
know
that
is
a
row
string,
but
in
your
mind
you
know
that
the
Apache
Lo
has
an
IP
at
Einstein
and
HTTP
method
and
status.
So
one
of
the
new
futures
of
flue
embed
for
these
cases
is
that
we
implemented
annotated
ports.
So
when
you
deploy
your
pot,
you
can
say
okay,
this
is
my
pot,
but
also
please
fluent
bit
processor
used
Apache
parsers.
You
can
suggest
that
from
an
operational
perspective,
so
let's
wait
pretty
quickly.
A
A
A
That
was
not
there
before,
because
Apache
logs
doesn't
have
a
structure
but
suggest
in
a
parser
to
the
log
processor.
You
can
get
that
right
away
inside
your
database
before
this
you
had
to
do
a
lot
of
magic
in
order
to
make
it
happens,
and
now
a
fluent
bit
can
do
it
for
you,
that's
one
of
the
biggest
thing
and
another
future
I'm
not
going
to
demo,
because
if
they
pretty
much
the
same
that
you
can
do.
For
example,
a
you
can
say:
please
exclude
my
logs,
you
a
support.
A
You
say
an
annotation
flew,
impede
that
IO
is
latch
exclude.
True,
please
exclude
my
dogs
because
there
are
some
times
that
the
posts
are
too
noisy
are
running
in
debug
mode,
and
you
don't
need
that
we
added
matrix
for
JSON
and
permit
use.
So
that
means
that
you
can
query
also
how
your
login
solution
is
working
internally.
I
think
that
this
is
one
of
the
most
expected
futures
and
since
we
have
a
few
minutes,
I'm
going
to
run
it
locally
in
my
computer
I'm
not
going
to
run
parameters
right
now,.
A
Flow
embed,
here's
so
I'm
going
to
run
through
embed
locally
in
my
computer
I'm,
going
to
use
the
input.
A
fluid
can
read
many
things:
log
files,
TCP
messages,
but
also
CPU,
metrics
CPU
metrics
and
send
the
metrics
to
the
standard
output
every
second
but
I
didn't
start
the
web
server.
So
I
cannot
get
the
metrics
Lego.
A
A
A
So
in
the
input
you
have
one
input
scipio,
we
are
processing
37
records,
which
belongs
to
eleven
thousand
bytes
and
so
on.
So
if
you
refresh
this
information,
you
will
see
how
it
grows
and
what
you
do
is
specifically
is
that
let
Prometheus
consume
this
endpoint,
but
you
know
oh,
this
is
Jason,
but
perimeter
has
a
different
format:
slash
Prometheus
there
you
go
okay!
A
Looks
like
you,
you
love
parameters,
right,
yeah.
No,
it's
really
cool
so
yeah.
So
this
is
one
of
our
additions
on
also
people
say
from
the
enterprise.
We
need
more
Enterprise
connectors.
We
solve
all
the
data
collection
for
kubernetes,
but
now
we
can
in
want
to
ingest
data
in
all
places.
So
we
added
support
for
Asher
for
Kafka
and
also
Splunk.
Maybe
you
love
it?
Maybe
you
hate
it,
but
we
have
the
support
for
it
because
people
want
it
ok
and
annotated
ports.
A
You
can
use
the
parser
to
suggest
a
person
and
in
saying
suggest,
because
in
the
fluent
bit
configuration
you
can
reject
that
rules.
Maybe
some
guy
want
to
deploy
a
pod
which
one
to
screw
their
locks,
but
if
that
flag
is
not
a
in
Abel
or
allow
it
in
fluent
bit,
it
will
not
happens
it
with
log
it
area,
anyways,
I,
know
teleports.
We
just
see
that
how
we
can
put
an
annotation
well
flowing
bit
with
images
with
primitives
and
that's
whether
we
show
you
a
now
well
I.
A
Don't
think
that
we
have
time
for
the
last
a
demo,
but
we're
going
to
run
pretty
quickly
in
one
minute
the
start
of
a
project
we
have
so
far.
In
less
than
three
years
we
had
like
74
releases
more
than
32
contributors.
This
is
low,
but
we
can
grow
with
you
guys.
Every
day
we
have
more
than
$50,000
every
day
since
the
last
month,
and
that
means
that
we
are
getting
more
than
three
millions
of
do--let's
in
the
in
the
last
year
and
what
means
$50,000,
fifty-thousand
notes
that
are
created
everyday
somewhere.
A
Somebody
here
was
testing
it,
but
he
didn't
like
it
having
that
in
the
last
version,
0.13,
we
were
doing
all
these
tests
about
HTTP
matrix
and
we
did
are
like
20
versions
since
January
once
a
week
and
people
was
tested,
testing
and
people
were
so
excited
that
some
of
them
said:
hey
I,
deployed
this
in
production
and
what
happens
at
the
end.
Sometimes
we
have
some
crashes
from
bugs
and
they
say
oh,
but
it
was
running
this
in
production.
Don't
do
it
is
a
test
version,
but
now
we
are
really
good.
It's
pretty
stable.
A
We
don't
release
something,
but
you
have
unit
tests.
We
are
going
forward
now
with
the
CI
CD
and
all
the
things
so
a
we
just
started
the
next
version.
Co-Taught
14,
the
development
we
started
just
after
coop
con
right
now,
so
our
goals
is
to
add
load
balancing
for
multiple
outputs.
Maybe
if
you
are
sending
to
one
elasticsearch,
you
can
have
many
at
failure
overrun
robbing
a
support
nest.
A
multi-line
in
Dhaka
JSON
logs,
for
example,
Java
stack
traces
and
we
need
help
with
documentation
in
testing
with
everything
that
relates
to
an
open
source
project.