►
From YouTube: LinkedIn + Redpanda Fireside Chat
Description
Join Redpanda's CEO Alex Gallego as he does a deep-dive talk with a team of storage experts at LinkedIn that built Kafka and Samza.
0:00 Intro
1:14 Timeline of streaming data
2:55 Performance of CPU by the numbers
5:06 RPC + Microservices
6:40 Weakness of storage engines
11:10 Low level architecture
15:14 Seastar + futures
19:10 Thread per core architecture
24:15 DMA + Sparse Filesystem Support
26:06 High level architecture
30:25 Produce request
35:00 Q&A
-----------------------------------------
To learn more about the fast, simple, reliable Redpanda streaming data engine please visit https://redpanda.com:
◦ The Kafka API is great; now let's make it fast! - https://rpnda.co/fast-safe
◦ How we built Redpanda’s tiered storage capabilities - https://rpnda.co/tiered-storage
◦ Kafka-compatible fast distributed transactions - https://rpnda.co/fast-transactions
◦ Building our data transformation engine with Wasm - https://rpnda.co/wasm-transforms
Join our Slack community! https://rpnda.co/slack
Contribute to our GitHub repository: https://rpnda.co/github
Redpanda is the modern streaming data platform for developers. API-compatible with Apache Kafka®, Redpanda enables a simple, fast, reliable, and unified engine of record for both real-time and historical enterprise data. 10x faster. Safe by default. No Zookeeper®, no JVM, and no code changes required. Built with C++ and Go, using technologies like Seastar.
Follow us on social media:
Twitter - https://rpnda.co/twitter
LinkedIn - https://rpnda.co/linkedin
A
Well,
so
so
what
are
we
building
here?
The
context
is
obviously
kafka.
We
red
panda
is
the
name
of
the
product.
It's
also
a
really
cute
pet
and
mascot
is
we're
building
a
real-time
engine
for
for
modern
applications
where
kafka
api
compatibility
is
only
part
of
the
story.
Fundamentally,
we
wanted
to
eliminate
zookeeper
and
you
know
make
it
easy
to
run
so.
A
Abstractly
speaking,
we
were
thinking
about
just
having
a
single
binary
that
you
can
just
ship
around
and
scp
to
a
bunch
of
computers,
we'll
talk
at
length
about
specifically
zero
data
loss
using
raft.
So,
instead
of
having
like
your
your
log
flush
rate
to
to
disk,
be
like
one
on
the
kafka
broker
setting,
you
can
choose
it
dynamically
with
like
x,
negative
one.
It
actually
means
that
there
was
a
full
core
on
right
and
data
was
flush
to
disk.
A
We
rewrote
it
from
the
ground
up
for
for
modern
hardware,
you'll
see
what
this
means
and
like
what
what
is
actually
10x
faster
yeah,
it's
basically
p99
tail
latency
is
what
we
claim
so
talk
about
foundations
and
set
the
context.
I
speak
relatively
quickly.
If
people
have
questions,
let
me
know
I
can
also
slow
down,
and
I've
had
two
cups
of
coffee,
so
I'm
seeing
the
future
fundamentally,
this
graph
is
really
just
a
graph
of
of
of
what
has
changed
in
in
bottlenecks.
A
You
guys
know
this
for
context
of
of
others
in
the
room.
This
is
my
view
of
streaming
innovations
as
I've
been
part
of
this
for
the
last
decade,
or
so.
Rabbiting
queue
really
big,
big
innovation
and
I'll
give
more
context,
because
I've
been
talking
to
a
lot
of
business
people
about
this
really
like
shattered
massive
massive
products
in
the
enterprise.
It
was
really
permissively
licensed
and
people
loved
the
api.
A
I
think
it
was
ampq
and
it's
just
like
sort
of
the
abstraction
it
didn't
necessarily
scale
for
well
full
stop,
and
I
think
in
kafka's
you
know
was
my
view
on
kafka's.
Innovation
was
that
we
was
trying
to
take
advantage
of
basically
cheap
spinning
discs
for
you
guys,
but
I
don't
preach
into
the
choir
pulsar
innovation
later
on
was
really
to
disaggregate.
The
computer
and
storage,
and
the
context
for
pulsar
is
probably
some
positive
devs
in
the
room,
too
was.
Was
yahoo
right
and
so
yahoo?
A
If
we,
if
we
decouple
them,
we
can
basically
lower
the
cost
of
doing
streaming
for
them
kind
of
what
happened
at
the
same
time
is:
s3
became
the
data
lake
and
it
became
the
disaggregation
of
compute
and
store,
hence
the
snowflake
launching
the
most
successful
public
offering
in
the
world,
but
really
kind
of
what
matters
here
that
I,
the
point
that
I
wanted
to
to
to
drive
at
is
performance
of
of
cpu
by
the
numbers.
So
let's
look
at
them.
A
Cpu
frequency
is
now
three
times
faster
than
it
was
a
decade
ago.
So,
actually
not
not
any
real
gains.
We
computers
have
gotten
about
20
x,
more
cores,
so
this
is
actually
where
the
substantial
improvements
have
come
from.
This
are
are
100,
but
it's
actually
a
thousand
times
like
a
modern
nvme
is
about
a
thousand
times
faster
than
it's
been
in
this
box
over
a
decade
ago
and
they're
they're
cheaper.
A
So
if
you
go
here
on
like
the
2008,
you
can
go
to
a
black
black
blaze,
which
is
a
company
that
hosted
provided
that
that
storage.
They
have
all
of
these
numbers
over
the
last
decade
and
it
shows
like
the
cost
of
terabyte
per
ssd,
was
roughly
2
500
a
decade
ago,
and
it's
now
about
two
hundred
dollars
per
terabyte
that
you
can
rent
this
on
amazon.
A
A
You
know
a
decade
ago
when
I
was
when
I
was
building
an
attack
in
new
york
like
a
gigabit
for
a
second
was
actually
optimistic,
so
more
more
stuff
about.
I
have
about
30
slides,
so
I'm
gonna
try
myself!
I'm
gonna
go
through
a
lot
of
the
content
relatively
quickly,
so
to
get
to
the
interesting
thing,
which
is
the
the
new
fundamental
architecture.
A
So
why
does
computer
actually
feel
faster
it
fundamentally,
this
came
from
from
like
ip
instruction
level
parallelism
so
and
you
can
think
of
it
like
rpcs
right.
So
a
cpu
is
actually
a
network
of
ports
and
you
can
you
can
make
an
rpc
from
the
front
end
to
to
the
executing
machine
for
like
floating
point
arithmetic,
there's,
a
couples
for
for
additions
and
so
on,
and
the
next
big
improvement
here
is,
of
course,
instruction.
A
A
So
in
in
storage,
engines
really,
and
I
hope
to
focus
the
rest
of
the
conversation
on
on
on
ads
in
the
search
engine
and
actually
try
to
get
a
slow
level.
I'm
just
trying
to
to
get
to
the
place
where
we
can
have
a
discussion
is
compile.
Time
really
helps
us
a
lot
in
thinking
in
terms
of
profiling,
just
basically
telling
the
linker
to
eliminate
a
bunch
of
that
code,
helping
the
branch
predictor.
A
You
know
pessimize,
actually
different
code
paths,
so
this
is
super
useful
in
let's
see,
check
zoning
compression
so
where
you
have
like
your
your
lc4
64
kilobyte,
punk
encoder.
For
for,
like
the
kafka
chunk
encoder,
you
can.
You
can
just
always
customize
everything
else,
because
for
us
this
is.
We
are
storage,
we're
a
new
storage
engine
and
the
only
interface
is
the
kafka
batch
model.
A
So
we
it's
in,
like
that's
literally
the
disk
interface
and
everything
has
been
optimized
around
it,
including
like
rewriting
parts
of
the
of
the
of
the
compressors.
Today,
like
you
know,
for
lg
forces
for
for
the
z
standard,
it
is
always
very
likely
that
you're
going
to
have
multiple
chunks,
well,
gdp
sucks
as
a
they're,
actually
very
cpu
intensive,
it's
good
for
compression,
but
very,
very
deeply
intensive.
It's
like
an
order
of
magnitude
larger
than
the
others
and
and
and
snappy.
A
So
that's
one
a
second
one
that
was
really
helpful
for
us
in
terms
of
trying
to
improve
the
achilles
heel
was
lpl,
so
we
tried
also
pgo,
which
is
profile,
guided
optimization
where
you
you
take
a
binary.
You
tell
you,
tell
it
to
record
a
trace
to
disk
and
you'd
say
like
this.
Is
the
typical
workload
record
a
trace
to
disk
and
then
rerun
the
linker
again
and
say
like
optimize
this?
A
A
So
let's
say
that
yeah
you
have
a
classical
inheritance
right
like
you,
you
have,
for
example,
your
allocator.
You
have
like
an
abstract
allocation
interface,
and
then
you
want
to
hide
the
details
on
that.
It's
really
slow
for
us
to
link.
It
really
takes
a
long
time
like
multiple
multiple
seconds
to
to
do
this,
and
but
it
gives
us
about
five
to
seven
performance
improvement.
A
Procore
architecture
we
will
talk
about
in
just
a
second
other
things
that
have
have
significant
impact
for
us
is
because
we
have
exactly
one
case
which
we're
trying
to
optimize,
which
is
we're
trying
to
to
to
shove
bytes
from
from
the
nic.
You
know
we
check
dominance
on
to
to
disk
as
fast
as
possible.
A
We
understand
which
functions
must
be
in
line
to
give
us
like
the
right
throughput
that
we're
expecting
so
being
able
to
tell
the
compiler
and
force
the
compiler
to
inline
particular
one-liners
are
super
useful
and
then
the
other
one
from
an
instruction
cache
level
is
using
sophisticated
data
structures.
So
I
think
they've
been
participating
in
in
the
in
the
absolute
flat
hash
map.
A
A
All
of
this
have
helped
tremendously,
and
this
also
just
adds
more
color
as
like,
we
were,
and,
of
course,
the
only
ones
to
understand
that
cpu
is
the
new
bottleneck
and
the
nvme
ssd
devices
are
now
super
fast
and
so,
like
things
have
simply
moved
around,
you
can't
eliminate
things
you're,
just
sort
of
moving
moving
things
around
great.
So
let's
talk
about
low
level
architecture.
First
of
all,
questions
there
on
cpu's,
the
new
bottleneck,
I'm
just
thinking
of
the
context,
probably
nothing,
nothing
that
you
know.
Most
people
here
probably
agree
with
that.
A
Yet
we
so
what
we
do
is
we
eliminate
on
the
at
compile
time
we
eliminate?
Oh,
we
do
one
thing
only,
which
is
so.
We
eliminate
all
of
the
old
code.
We
say
like
support.
Tls
1.3
like
eliminate
all
of
the
all
of
the
older
code,
and
then
we
compile
sadly
for
westmere
cpu
architectures
and
that's
just
like
what
we
have
to
compile
to
so
people
can
run
in
on
on
amazon.
A
But
if
this
was,
you
know
when
we
run
our
cloud,
which
will
come
probably
in
like
february
or
so
that's
as
ga,
we
will
actually
probably
ship
compiling
infrastructure
to
to
these
edge
machines
to
to
optimize
for
that
particular
cpu
architecture
like
localhost,
cpu
architecture,
really
on
on
things
like
bit
vectors
or
compaction.
We
use
roaring
bitmaps
for
for
compaction
and
being
able
to
sort
maps
of
of
like
bit
vectors
using
avx2
instructions,
as
opposed
to
just
st3.
A
Instructions
really
have
a
big
performance
impact
for
us,
but
sadly
the
only
thing
that
we're
doing
for
tls
as
the
security
protocol
is
just
targeting
the
westmere
architecture
and
say,
like
you
know,
and
that's
that
just
emit
instruction
set
for
this.
It
helps
a
little
bit,
but
quite
honestly,
probably
not
not
significant
in
any
way
sean
did.
You
have
any
other
thoughts
around
that
like
what
what
what
would
you
suggest
would
be
like
additional
improvements.
A
Let's
see,
I'm
trying
to
see
what
I
can
share
from
another
company
I
worked
on
what
other
companies
do
is
that
they
offload
a
lot
of
these
to
fpgas,
and
that
was
that's
fairly
successful.
I
think
there
are
some
expensive
nicks
that
that
you
can
buy.
That
will
just
broke
will
just
like
give
you
an
unencrypted
socket
from
the
application
level
and
the
fpga
will
do
the
decryption
close
to
line
rate,
but
it
is
more
expensive
and
the
knicks
are
expensive
and
you're.
A
A
There
wasn't
that
massive
improvement,
and
it
happened
to
be
this
particular
architecture
that
I'll
talk
about
in
a
second,
but
that
was
the
gis,
is
like
treat
everything
like
like
an
asynchronous
like
you
treat
a
network
call
which
means
like
you
know
they
may
fail.
You
need
to
retry,
you
need
to
have
quotas
and
and
back
pressure
and
all
of
this
other
infrastructure
that
comes
from
making
a
network
call.
So
we
build
on
a
framework
called
sistar
that
I
own.
A
I
think
the
two
things
that
that
are
useful
here
is
that
futures
are
like
actors,
so
whoever
has
written
code
in
in
akka
knows
this.
The
minute
you
introduce
aka
into
a
code
base.
The
whole
thing
becomes
an
actor.
It's
like
you
just
have
to
wrap
things
into
actors,
because
it's
just
it's
easy
to
do
and
it
kind
of
like
lends
itself
nicely.
I
have
never
written
orleans
only
on
the
akka,
and
so
so
these
star
futures
are
very
much
the
same.
They
are
a
viral
primitive.
A
What's
nice
is
similar
to
actors
right,
you
can
like
make
them
reduce
them,
filter
them
change
them,
complete
them
generate
and
fulfill
them
slip
them
require
them,
expire
them,
etc.
There's
all
of
these
useful,
primitives
and
and
on
what
you
do
when
you,
when
you
have
this
viral
and
very
opinionated
by
the
way,
there's
no
other
way
to
do
to
write
code
with
csr.
A
Like
you,
you
simply
have
one
one
primitive
and
those
of
us
that
have
written
code
for
let's
say
like
facebook,
wengel
and
and
the
other
c
plus
plus
frameworks
out
there,
because
we
built
concord
on
on
top
of
these
other
frameworks,
where
you
have
multiple
concurrency
paradigms
inside
a
single
framework,
it's
really
difficult
to
understand
as
a
programmer
when
you're
writing
code.
But
what
is
the
actual
useful
thing
like?
What
is
the
right
part
that
versus
when
you
have
one
and
it's
opinionated?
A
A
So
this
is
how
it
looks
before
I
dive
into
this.
Sometimes
it's
not
pretty.
This
is
how
it
looks
in
c
plus.
This
is
a
wrapped
implementation,
and
so
a
couple
of
things
to
to
point
down
here-
and
I
want
to
talk
about
deeply
about
threat
per
core-
is
when
I
talked
about
treating
everything
like
a
a
network.
Call
I
mean
it
so
get
get.
Scheduling
group,
for
example,
is
the
same.
A
It
uses
the
similar
algorithm
as
linux
kernel,
completely
fair
scheduler,
there's,
basically
a
hierarchy
of
shares
and,
and
you
can
create
more
groups
in
this
hierarchy
of
shares,
but
it
says,
run
this.
This
lambda
with
this
particular
share
hierarchy
and
it's
just
an
integer,
so
it's
relatively
cheap,
then
integer
id
through
to
a
thread
thread
local
tls
in
the
third
local
sense,
third,
local
storage,
and
but
this
only
one
that
is
perhaps
more
interested.
Is
that
from
this
core
we're
making
a
message
pass
to
this
remote
core
called
shard.
A
But
then,
even
when
we
do
that,
we're
telling
it
by
the
way,
use
this
service
group.
So
this
is
where
back
pressure
starts
to
kick
in
right.
So
it's
fundamental
from
the
ground
up.
It's
not
just
a
back
pressure
at
the
tcp
level,
where
you're
just
deterioralizing
the
ttp
buffer-
and
you
know,
passing
the
kafka
batch
header,
etc.
You're
actually
saying
to
to
make
a
call
to
to
a
remote
node
and
it
could
be
a
remote
socket.
A
So
let's
take
the
case
of
enuma
number
two
numa
domains
where
you
have
two
cpu
sockets
and
two
separate
enema
domains
in
order
to
make
an
rpc
call
sorry,
an
itc
ipc
enterprises
call
to
a
remote
core
you're,
telling
the
scheduler
to
abstract
one
share
from
this
scheduling
group
and
this
service
group.
And
so
it's
really
from
from
the
grounds
of
this
approach
of
back
pressure.
A
The
way
this
works
is
it
a
system
uses
this
framework
called
sidstar.
It
uses
a
trapper
core
architecture
with
with
with
locked
memory
and
to
give
you
even
kind
of
a
deeper
sense.
The
there's
a
body
allocator
that
and
so
so,
when
the
and
we
have
to
work
around
that.
Sometimes
it's
really
painful
is
when
you
start
up
the
computer.
A
It
allocates
one
segment
right
so
classical
body
allocator
algorithm
here,
and
so
let's
say
that
you
have
for
simplicity,
10,
gigabytes
of
memory,
there's
10
porous
every
core
is
going
to
get
it's
going
to
allocate
the
first
body,
which
is
a
gigabyte,
and
that
memory
is
going
to
be
locked
and
so
typically,
what
happens
for
production
systems?
Is
you
reserve,
you
know
like
maybe
a
gigabyte
for
or
like
cron
or
other
things
that
are
going
to
use?
A
You
know
your
your
dns
and
whatever,
and
then
you
you
lock
the
memory,
and
so
what
that
does.
Is
that
if
you
make
so
there's
two
things,
one,
the
allocations
are
core
local
and
you
have
to
be
very
careful
about
how
you
share
memory
between
between
threats
right
because
he
might
be
accessed
from
a
totally
different
pneuma
domain.
So
does
and-
and
the
way
sister
does.
A
This
is
that
every
core
has
an
outgoing
mailbox
for
every
other
core
and
so
there's
a
loop
is
it's
called
the
reactor
loop,
and
this
is
those
of
us
that
have
written
financial
systems.
There's
there's
nothing
novel
here.
Hft
has
been
doing
this
for
like
over
20
years,
so
there's
a
loop
on
every
core
and
the
loop
will
pull
first
for
io
right,
and
so
you
get
500
microseconds
to
execute
a
loop.
A
And
if
you,
if
you
fall
below
that
above
that,
500
microseconds
sister
will
actually
print
a
printed
structures
and
says
this
function
took
longer
than
500
microseconds.
Why
is
that
useful?
The
reason
that's
useful
is
because
there's
a
single
loop
that
is
pulling
and
it's
doing
everything
right
and
and
the
point
here.
The
reason
why
that
works
is
that
you,
wrapped
every
single
computation
in
the
future
and
future
are
comparative
units
of
work.
A
Every
time
you
return
a
future
that
the
sisters,
our
scheduler,
can
yield
a
different
ordering
and
it
has
priority
where
io
ripping
from
the
nic
has
much
higher
priority
than
executing
cpu
intensive
applications.
The
reason
this
works
for
io
heavy
engines
like
a
storage
engine-
and
this
wouldn't
work
well
for
like
a
if
you're,
trying
to
do
like
gpu,
computations
or
something
is
that
we
are
explicitly
choosing
to
prioritize
io
over
a
lot
of
the
cpu
intensive
tasks
and
mostly-
and
so
why
is
that
is
mostly
because
io?
A
A
This
is
the
linux
kernel
setting
and
two
you
can
coalesce
the
interrupts,
there's
another
linux
kernel
and
so
and
and
then
you
spread
the
interrupts
around
right.
So
those
are
really
the
only
two
techniques
that
you
have
from
from
the
linux
kernel
kernel
level,
so
it
remains
stored.
Local,
I
mean
once
you
cross
the
barrier
into
like
direct
aio
when
you're
flashing
to
to
xfs
or
something
like
you,
don't
control
that
right.
This
is
only
for
like
within
within
the
applications,
and
you
can
tune
the
kernel
for
some
particular
settings.
A
The
other
setting
that
is
useful
is
to
tell
the
nyx
kernel
to
not
merge
your
your
block
item
all
right.
So
what
happens
is
when
you
submit
a
bunch
of
stuff
to
might
be
the
next
line?
Well,
this
is
the
impact
you
can
tell
me
that
all
right,
you
can
look
up
sister
that
I
have,
but
I
wanna
this
is
useful
in
the
discussion.
So
right,
so
you
can.
Oh,
the
other
one
is
disabling.
A
I
o
merges
and
what
it
means
is
that
in
this
picture,
let's
say
you
have
whatever
some
some
memory
align
buffers
and-
and
you
can
ask
the
file
system,
what
the
memory
alignment
is.
You
don't
want
the
linux
schedule
to
think
like.
Oh,
I
actually
know
better
than
you.
Let
me
let
me
group
this
into
a
bigger
thing
and
let
me
do
a
bunch
of
reallocations.
A
A
A
Let
me
go
back
to
this
one
one
case
you
guys
can
can
go
to
a
system
that
I
o,
so
I
didn't
want
to
add
a
bunch
of
our
benchmark,
also
because
I
only
had
30
minutes
to
put
together
this
presentation,
but
roughly
you
can
see
sort
of
another
magneto
improvement
in
in
general
or
or
this
is,
I
guess,
just
for
memory
and
cpu
bound
requests.
Also,
there
was
a
new
paper
on
nsgi,
21
or
20.
on
on
the
the
impact
of
purple
core
on
tail
latencies.
A
So
let's
talk
about
high
level
architecture,
because
this
is
maybe
more
more
generally
useful
to
the
rest
of
the
audience
dude.
I
only
have
18
minutes
left,
we'll
probably
get
through
half
of
this,
and-
and
maybe
we
do
a
comeback
here
so
on
on
ring
zero
is
so
we're
a
raft-based
system
and
and
which
means,
and
the
reason
we
wanted.
This
is,
of
course
we
didn't
want
to
have
a
dependency
on
zookeeper.
A
We
can
test
the
system
for
longer
so
but
but
at
an
architectural
level,
when
you
started
the
system,
you
have
three
three
like
red,
panda
binaries
and
you
said
the
configuration
that
is
your
degenerative
case,
all
right,
and
so
so
anything
else
or
the
bootstrap
from
this
ring
zero.
This
is
similar
to
the
capture
controller.
A
You
know
in
kafka,
you
have
a
controller,
so
there's
ring
zero
ring,
zero,
elixir
leader
just
classical
leader
election
to
raft
it
is
stable
leadership
which
is
nice
for
particularly
for
throughput,
and
actually,
I
think,
about
yeah
and
then
every
other
group
is
in
and
so
the
leader
of
ring
zero
becomes
the
cluster
controller
and
then
it
it
does.
The
the
physical
partition
allocation
of
like
which
physical
compute
is
going
to
own
that
and
every
partition
is
a
wrap
group
and
what
what
we
did
there
is
that
we
actually
then
optimize.
A
It
became
a
mechanical
translation.
We
didn't
want
it
to
invent
a
new
protocol.
What
we
wanted
to
do
is
like
we're
good
at
making
things
fast,
the
team-
and
so
let's
start
with
the
known
protocol
and
actually
focus
the
engineering
work
on
making
these
things
fast
and
scalable,
and
last
time
I
tested
was
about
360
something
thousand
groups
for
her
computer
and
we're
probably
going
to
remove
that
limit
at
some
point
given
enough
over
subscription
on
a
machine
or
partitions
you're
running
to
network
bottlenecks
and
but
anyways.
A
No,
we
implemented
our
own
from
sistar.
So
so,
if
you
go
back
here,
this
is
literally
how
it
looks
it
is.
It
is
asynchronous.
Let
me
see
with
the
annotations
in
particular,
it's
useful,
because
futures
are
a
viral
primitive
there's
no
way
so
threading
is
a
fundamental
building
block.
It's
almost
like
a
mental
decision
of
how
you're
going
to
build
your
framework,
and
there
doesn't
exist,
a
library
that
understood
a
threat,
local
approach
with
futures
as
a
yielding
mechanism
that
was
in
c
plus
that
we
could
use.
A
So
as
far
as
I
understand
there's
our
raft
implementations
which
we're
going
to
open
source
all
of
this
next
week.
So
we
want
you
guys
there.
If
there
is
interest
in
exploring
something
together,
I
would
love
to
figure
it
out,
and
I
know
cilla
is
also
trying
to
write
their
own.
I
also
know
that
ceph,
the
file
system
at
red
hat,
is
looking
at
their
own
consensus
protocol.
A
But
when,
when
I
started
this
in
2019,
there
was
nothing
so
we
had
to
rewrite
it
from
scratch
and
the
reason
was
we
wanted
to
propagate
tail
latencies
like
from
from
the
the
lowest
level
it
was
like.
It
was
a
conscious
decision
and
it
took
us
18
months,
so
maybe
from
a
from
a
company
building
perspective,
we
should
have
used
an
open
source
library,
but
we
wrote
it
from
scratch.
A
Thank
you
sure,
okay,
so
this
is
probably
the
most
familiar
picture
for
well,
maybe
not,
but
for
those
of
you
that
have
been
working
in,
I
want
to
leave
five
minutes
for
questions
just
just
so.
Everyone
knows,
let's
see
right
so
so
you
have
a
fetch
request,
a
produce
request.
Rather,
a
defense
request
is
much
more
complicated
because
of
consumer
groups
and
so
on.
A
So
so
I'm
going
to
talk
about
produce
requests,
because
this
is
maybe
a
simplest
one
through
the
layers
of
kind
of
how
we
we
went
about
re-architecting
this
thing
from
scratch,
where,
where
kafka-
and
I
talked
about
in
the
beginning-
about
optimizing
for
structural
caches
like
how
this
where's
all
of
these
pictures
come
together,
and
so,
if
you
look
at
this
architecture
like
what
does
this
really
mean
in
practice
from
when,
like
in
code
and
and
how
it
executes?
So
you
have
a
kafka
request
at
the
top.
A
Maybe
a
cloud
of
kafka
requests
they're
coming
through
an
rpc
handler
I'll,
tell
you
how
we
guarantee
compatibility,
but
then
fundamentally
there
is
no
global
metadata
cache
right
like
if
you
I'm
not
sure
of
the
top
of
my
head.
I
was
the
last
time
I
look
at
the
the
kapka
code,
but
when
you
have
to
look
up
one,
are
you
the
owner
for
this
partition
and,
like
the
machine
you
know,
did
I
get
the
right
machine
or
do
I
have
to
tell
the
client
like
I'm?
A
Not
the
owner
of
this
partition
or
this
foundation
was
deleted
or,
like
you
know,
all
of
that.
So
you
do
this
minute.
Data
cache,
instead
of
it
being
a
global
data
structure
that
people
share
around,
even
if
it's
read-only
and
you
can
guarantee
well
with
kafka,
is
eventual
consistency
of
the
metadata
gets
propagated
to
all
those
nodes.
So
even
with
relaxed
consistency
guarantees
it's
a
totally
different
approach,
because
what
you're
optimizing
here
is
for
latency,
so
you're
you
do
consume
additional
memory
on
every
core
to
keep
a
particular
view.
A
You
don't
keep
the
whole
thing
because,
like
there
is
a
there's,
a
computer,
global
allocator,
so
we
don't
have
to
get
into
that.
But,
but
fundamentally
this
is
that
maybe
the
most
interesting
part
where
every
core
holds
a
view
of
what
partitions
this
particular
machine
holds,
and
so
everything
else
is
not
materialized.
A
A
For
this
this
request,
I'm
gonna
send,
via
spsc
bi
network
of
spscqs
a
request
to
core
one
to
then
do
two
things:
one
dispatch
concurrently
to
the
followers,
but
well
concurrently
on
the
core
entirely
within
the
network,
execution
and
also
dispatch.
Another
network
request
to
storage,
which
is
again
if
you
treat
everything
like
a
network
with
backup,
retries
failure,
domain
guarantees
etc
and
back
pressure.
In
particular.
A
This
is
sort
of
a
really
useful
thing
and
then,
when
you
finish
the
acknowledgement,
you
don't
just
return
back
to
the
handler,
because
that
would
violate
basically
memory
ordering
and
guarantees.
So
you
you
push
back
a
response
to
the
network
of
sbs
cqs
and
those
are
small
messages.
So
let
me
give
you
orders
of
magnitude
so
that
you
can
understand
the
impact
every
core
could
probably
handle.
I
know
tens
of
millions
of
futures,
because
they're
they're,
relatively
small
structures
and
basically
bytes,
probably
maybe
a
little
smaller.
A
C
A
Yeah,
so
so
it
routes
metadata
to
the
correct
core,
but
it
doesn't
like
and
then
it
so
so
you
can
have
that's
a
great
question.
You
can
have
read-only
memory
from
other
cores,
but
you
can
never
mutate
memory
from
other
course.
There's
like
this
explicit
protocol
for
handing
off
memory
to
another
card.
C
A
A
Well,
no,
it
has
to
be
a
different,
a
different
I
want
we
want
to
offer
it
to
to.
We
want
to
rip
it
from
a
different
event.
Look
because
remember
that
in
order
to
get
good
locality-
and
this
is
this-
is
like
it's
not
every
threadback
architecture
has
this
problem,
we'll
we'll
solve
it?
The
basically!
A
I
you
you
don't
know
if
that
tcp
connection
is
actually
going
to
be
steadily
producing
data
through
core
zero
or
steadily
producing
data
to
core
one
or
it
happens,
to
produce
data
across
a
bunch
of
multiple
cores.
So
it's
actually
a
much
more
difficult
problem
than
just
moving
the
socket
on
a
stable
right
for
stable
rights.
What
we
do
on
our
proxy.
A
Yeah,
it's
it's
it's
true!
It's
it's
hard.
We
haven't
done
it,
and
so,
if
you
look
at
the
linux
scheduler
assignment
when
you
hash
the
five
tuple
on
the
on
the
tcp
connection,
right,
which
is
like
the
external
port,
the
internal
port
destination
address
and
so
on,
like
the
linux
scheduler
with
the
default
network
allocation
will
just
land
it
on
a
particular
core
and
every
core
reaps
rips
particular
events.
A
So
once
a
core
opens
the
file
handle,
it
will
it'll
own
it
private
right
like
once
the
core
that
particular
car
accepts
the
connection
it
will
own
it
permanently
so
to
get
around
some
of
this
internally
remember.
This
is
only
external
on
the
rpc
so
externally
to
beco
on
our
own
http
proxy.
We
actually
try
to
dial
multiple
times
to
the
machines
until
we
get
a
good,
a
good
port
assignment,
it's
it's
hacky.
A
A
So
let's
say:
every
computer
opens
up
like
three
tcp
connections
to
to
computers
that
happen
to
be
in
that
particular
quorum
and
that
will
that
give
us
some
form
of
locality
because
you
actually
have
the
same
problem
when
you're
sending
data-
let's
say
a
heartbeat,
but
you
only
have
three,
so
you
have
144
cores,
but
you
have
heartbeats
to
that
destination.
Node
coming
from
multiple
nodes,
and
so
we
we
we
lower
the
cost
of
the
the
spsc
network
by
opening
up
multiple
pcp
connections.
A
It
works
well
in
practice,
but
there's
definitely
improvements
there.
I
think,
for
smaller
clusters.
You
actually
can
have
an
n
square
number
of
connections
open
in
the
cluster
and
it
would
be
fine,
so
let's
say
10
nodes
or
something,
and
there
will
be
no,
no,
no
movement,
but
right
now
we
do,
I
think,
a
default
of
three.
I
have
five
minutes,
I'm
probably
going
to
stop
the
presentation,
and
then
we
didn't
get
a
chance
of
reviewing
draft
versus
the
isr
protocol
and
synchronicity
etc.
E
A
And
and
kind
of
the
I
really
skip
through
through
this,
but
because
we're
using
dma
we
and
we
force
people
to
use
xfs
because
activision
software
actually,
but
it
has
one
particular
feature
we
care
about,
which
is
sparse
file
system
support.
It
means
you
can
dispatch
concurrent
io
to
to
that.
That
I
think
exe4
also
supports
this,
but
I
know
octopus,
so
you
can.
A
So
if
you're
using
ax1,
it
means
like
the
client
will
not
send
their
next
request
until
and
deliver
gives
an
acknowledgement
from
the
previous
thing,
and
so
the
the
really
bottleneck
we
can.
We
can
observe
latency
bubbles
straight
from
the
nvme
device
and
then
propagate
that
correctly,
even
across
course
through
through
the
number.
A
A
D
A
Yeah
yeah
we-
and
this
is
by
design-
we
didn't
want
to
speak
inter-broker
communication,
like
I,
I
talked
about
this
on
the
data
engineering
podcast.
But
basically,
if
you
look
at
the
number
of
versions
per,
let's
take
the
produce
and
consume
requests,
so
fetch
and
produce,
there's
basically
like
12
versions,
with
the
new
flexible
protocol
and
and
multiple
like
half
the
client
versions
and
multiple
broker
versions.
A
So
in
totality
there
is
something
like
144
apis
just
for
client-side
requests,
and
so
it's
too
much,
and
so
we
said
we'll
speak
the
client
protocol,
because
you
have
enterprise
applications
that
want
to
leverage
this,
but
but
internally
we
want
it
to
be
totally
different
and
we
want
it
to
be.
That's
the
only
guarantee
we
wanted
to
give
so
currently,
our
our
on
this
format
is
actually
quite
similar,
but
we
hope
to
change.
A
lot
of
like
our
offset
index
is
different.
Our
time
index
is
different.
C
I
have
a
question
about
cluster
balance,
so
one
of
the
things
about
kafka
is
that
it
it
doesn't
shuffle
around
partitions
or
topics.
So
right
you
have
a
cluster
and
you
add
a
broker.
No
one
cares,
nothing
happens.
Yeah.
Does
this
behave
any
better,
because
you're
saying
that
you
still
have
you
still
have
segments
for
partitions,
yeah
and
protocol
actually
never
mentions
segments
they're,
not
a
part
of
the
kafka
api
ever
yeah.
Do
you
suffer
the
same
balance
problems
and
needs
that
kafka
does.
A
So
there's
there's
saturation
not
just
on
on
on
disk,
but
also
on
cpu
and
and
network.
Actually,
when
you,
when
you
start
to
to
drive
taller
and
taller
computers,
network
saturation
becomes
more
of
a
problem
for
for
a
lot
of
people
and,
like
you
know,
especially
when
you
start
to
oversubscribe
the
machines,
let's
say
you
assign
70
000
or
80
000
partitions
for
broker,
and-
and
so
so
that's
actually
much
more
difficult
and
there's
two
solutions
to
that.
A
A
I
didn't
talk
about
shadow
indexing,
which
is
the
ability
that
we
actually
will
read
transparently
from
s3,
that's
probably
partially
implemented,
and
so
what
we
have
to
transmit
to
the
new
machine
is
basically
the
segments
that
haven't
been
pushed
to
s3
stably.
Yet
so
you
can
push
an
index
to
do
a
read.
Replica.
Add
it
to
the
to
the
kafka
group:
do
a
joint
consensus
transition
to
add
the
new
broker
and
actually.
C
A
It's
it's
it's
exposed
through
through
a
corel
api,
but
not
yet
we
plan
to
we
just
don't
know,
what's
a
good
timeline,
yet
we
need.
We
need
more
experience
with
the
product.
To
be
quite
honest,
like
do
we
check
every
10
minutes?
Do
we
check
every
five
minutes?
Do
we
check
every
time?
The
note
adds
I
don't
know.
What
is
the
good
timeline
most
of
the
time
people
aren't
at
the
scale
that
you
guys
are
seeing
and
so
for
them
adding
computers
is
actually
about
partition
capacity
and
less
about
resource
saturation.
A
A
F
A
C
A
Yeah
we
do,
we
have
a
wasm
engine.
I
didn't
you
know,
I
think
we're
three
minutes,
I'm
happy
to
stay
over.
I
can't
have
another
20
minutes
left,
but
if
people
need
to
drop
feel
free,
so
we
we
didn't
talk
about
it
because
this
soccer
schedule
for
an
hour
is
we
actually
ship
a
wasm
engine
and
we
wrote
a
protocol
where
you
can.
You
can
subscribe
while
some
transformations,
so
the
first
one
is
going
to
be
just
javascript
as
the
sdk.
A
So
you
can
think
of
like
an
observable
in
javascript
like
subscribe
from
this
topic,
make
a
transformation,
I
think,
for
those
it's
not
for
high
throughput
or
low
latency
use
case,
but
actually
for
convenience
of
or
gdpr
compliance,
so
we're
talking
to
another
huge
food
delivery
company
and
they
use
the
they're
thinking
about
using
the
wasm
engine
to
guarantee
gdpr
compliance
for
the
machine
learning
workflows.
So
so
you
get
a
request
and
then
the
engine
will
just
you
know,
submit.
A
C
A
Yeah,
that's
fine
right
because
for
a
lot
of
people
like
I
mentioned,
I
think
that
most
workloads,
if
you,
if
you
ask
a
lot
of
kafka
users,
they're
they're,
like
nowhere
near
the
the
throughput
of
of
of
kafka-
that,
like
the
limit
of
kafka,
I
think
what
they're
looking
for
is
really
primitives
to
build
applications.
And
it's
not
I
mean
it's
obtained
right.
C
G
C
A
Maybe
I
think
you're
thinking
of
a
stray
representation
of
the
upstream
java
and
we
don't
necessarily
use
the
same
representation
internally
but
let's,
but
even
even
if
we
do
that.
So
that's
that's
fine.
Let's
say
50
megabytes
per
core
and
you
and
and
typically
very
tall
computers
will
have
like
768
gigabytes
of
memory
like
if
you
look
at
amazon.
C
A
A
We
could
we
keep
so
many
data
and
we
can
always
stretch
it
but
there's.
Basically,
the
idea
here
is
that
there's
there's
some
metadata.
That
is
always
materialized
per
core,
which
is
the
things
that
is
most
commonly
used
and
then
there's
no
metadata
that
can
be
rehydrated.
But
it's
all
local,
like
it's
okay,
to
make
a
remote
core
communication,
let's
say
to
core
zero
to
to
get
the
global
cluster
view
and
then
return
it
to
the
to
the
client.
A
Yeah
and
you
can
do
a
lot
of
bit
packing
in
simplest
buzz
is
relatively
straightforward,
but
I
mean
there's
like
there's
basically
essential
complexity
like
you,
you
can't
like,
if
you're
trying
to
replicate
data
like
you,
have
to
send
data
you.
If
you
need
to
answer
metadata
requests
on
on
every
node,
then
you
need
to
answer
metadata
requests.
It's
like
things
that
the
protocol
imposes
on
the
implementation.
A
Yeah,
so
if
you
have
fixing
you
have
16
16
physical
cores
or
virtual
cores,
every
chord
will
have
like
16
spl
cqs
and
the
reason
is
it'll
it.
It's
basically
cash
cash
line,
invalidation
and
atomic.
So
you
can
say,
like
I'm,
gonna
write
this
with
like
relax
memory
and
then
I'm
gonna
read
this
with
a
strong
memory
and
you
can
do
that
that
kind
of
trick
relatively
cheaply.
That's
what
makes
it
kind
of
a
network
of
spsc
cues.
The
alternative.
Is
you
have
a
burner
core,
so
bobby
bobby
blumo's.
Algorithm.
B
G
A
Until
it
says
it
has
been
electrocuted
it
basically
just
like
receiving
mailboxes,
and
so
it's
like
this.
Do
I
have
a
message
in
this
mailbox,
so
I
have
messages
and
so
on,
and
then
it
has
another
different
priority
for
locally
produced
mailboxes.
That's
like
that's
the
highest
priority.
Once
you
run
out
of
work
and
there's
like
there's
a
there's,
a
there's,
a
scheduling
algorithm.
Doesn't
it's
basically
a
schedule
on
on
how
it
it
does
this?
A
A
I
Thanks
hi
hi
kyle
here,
I'm
I'm
curious,
so
sharding
the
state
of
the
broker
amongst
numa
domains.
That
seems
really
intuitive
to
me
that
that
would
really
help
with
throughput
but
and
latency.
But
I'm
curious,
if
you
have
a
sense
of
like
that,
the
real
impact
it
would
make
in
the
system,
if
you
like,
didn't
shard
to
individual
cores.
A
So
this
this
optimization
is
really
for
tail
latencies,
not
necessarily,
I
think
that's
just
mostly
for
related,
and
it
does
improve
throughput.
But
you
know,
I
think,
pick
your
back.
We
had
to
pick
our
battle,
so
so
that's
the
intuition
that
we've
seen
in
real
in
real
life
is
that
when
the
system
is
at
full
pill,
then
then
then
you
see
actually
latency
passes
latency
spikes,
but
other
than
that.
A
If
the
system
isn't
like,
if
you're
running
some
low
level
of
utilization,
it's
probably
overkill
for
for
that,
the
goal
is,
I
feel
like
when
you're
building
a
system
you
you
have
like
some
primitives
and
some
opinions
of
how
you
see
this
thing
working
and
then
you
try
it
and
see
if
it
works,
because,
like
you
have
to
test
it
in
real
life,
some
things
work
work
nicely.
Some
some
don't
like
the
network
problem
that
radar
was
was
pointing
is
a
real
one.
That's
something
that
we
haven't
picked.
A
A
Oh
there's
all
this
waste,
it's
actually
not
that
bad
in
practice,
but
the
the
trade
of
here
for
having
this
network
of
queues
is
that
when
the
system
is
not
really
busy
with
user
events,
then
it
looks
superficially
busy
and
you're
like
I
don't
know
what
the
hell.
This
thing
is
doing
I'm
only
sending
in
100
requests
per
second.
A
C
C
C
A
F
J
J
C
C
So
this
is
a
little
tricky,
because
the
the
contents
of
underscore
underscore
consumer,
offsets
specifically
for
kafka
are
are
not
are,
are
never
a
client's
business
right.
This
is
a
kafka
internal
state
topic,
although
plenty
of
systems
know
what's
going
on
there
and
expect
it
to
exist.
So
the
the
question
is:
do
you
guys
recreate
it?
It
is
not
strictly
a
client.
J
Yeah,
so
we
don't,
we
don't
inspect
internally
those
payloads.
We
have
an
internal
topic
for
consumer
group
data,
just
like
kafka
does,
although
I
think
we
we
don't
use
a
topic.
We
use
a
raw
raft
group
for
that
the
yeah
we'd
have
to
get
into
like
specific
details.
I
haven't
looked
at
this
stuff
in
like
a
month,
so
it's
fully
outside
of
my
my
brain
at
the
moment,
but
yeah
I
mean
the
data
is
preserved,
the
same
way
the
clients
send
it.
J
A
H
A
That
are
connections,
work
transactions,
don't
work
yet
admin
api
should
work.
Fetch
on
produce
should
work.
So,
okay,
well,
actually,
here's
the
answer:
we're
gonna,
open
source.
Next
tuesday,
we
have,
we
checked
in
the
upstream
json
json
files
and
we
co-generate
the
apis.
We
actually
wrote
a
parser
combinator
library
for
c
plus
plus,
so
we
don't
manually
say
like
if
version
two,
so
you
could
just
check
whatever
is
code
generated
there.
It's
code
generated
yeah,
but.
J
J
G
A
A
A
A
Yeah,
oh
okay,
so
we
just
released
this
like
two
hours
ago.
Actually,
the
the
difference
in
availability
or
unavailability
of
the
different
protocols
and
like
so
so
definitely
send
us
a
tweet
or
an
email.
If
you
want,
if
you
guys
want
a
discussion,
we'll
probably
also
only
card
about,
I
don't
know,
40.