►
Description
0:00 - Introductions
4:41 - What is Memphis?
12:12 - Demo
31:44 - Motivation for building Memphis
34:35 - How to contribute
37:09 - Fun Chit Chat
Join @blackgirlbytes live for Open Source Fridays at https://www.meetup.com/github/.
Thanks!
Connect with us.
Facebook: http://fb.com/github
Twitter: http://twitter.com/github
LinkedIn: http://linkedin.com/company/github
About GitHub
GitHub is the best place to share code with friends, co-workers, classmates, and complete strangers. Millions of people use GitHub to build amazing things together. For more info, go to http://github.com
Chapters
A
This
is
open
source
Friday,
as
I
normally
chat
with
a
maintainer
every
Friday,
except
for
times
that
I'm
busy
and
I
chat
with
them
about
different
projects.
They've
created
why
they've
created
it
and
they
do
a
little
demo
for
us.
So
today
we
have
Yaniv
on
and
he
created
a
project
called
Memphis
and
he's
going
to
tell
us
all
about
that.
So
you
mean
hey.
How
are
you
doing
I'm.
A
Awesome,
yes,
it's
great
to
chat
with
you
too
I
love
chatting
with
you
on
the
Twitter
space.
I
learned
so
much
about
message,
Brokers
and
like
microservice
architecture,
so
really
excited
about
that.
Why
don't
you
introduce
yourself
and
then
for
anyone
in
the
chat?
Please
tell
us
where
you're
calling
in
from
or
where
you're
listening
in
from
I
mean
to
say.
B
Sure
so
my
name
is
Yaniv
and
I
live
in
Israel
born
and
raised
in
Israel
I
co-founded,
a
Memphis,
alongside
with
my
three
other
friends
from
from
college.
Basically
I've
been
a
developer
since,
like
all
my
life,
basically
and
that's,
my
passion
went
through
back
end
to
front
end
to
data
and
actually
mainly
for
the
past
six
years.
I've
been
definitely
focusing
on
on
the
data
world
and
specifically
on
data
engineering
data,
warehousing
building
data
pipelines,
data
Frameworks
and
for
startups
to
Enterprises
to
various
organizations,
and
me,
and
my
friends,
thought
it
would
be.
B
B
Wow
that
definitely
definitely
really
if,
if
someone
asks
me
what
would
be
my
biggest
advice
before
building
your
own
startup
and
your
own
company,
your
own
project,
even
do
it
with
people
that
you,
first
of
all
that
you
love
that
you
would
love
to
to
drink
beer
with
them
and
not
just
like
they're
writing
great
code
or
something
like
that.
They're
really
really
smart.
They
have
to
be
friends
and
do
it
with
Partners
I
mean
you
can't
do
it
yourself,
yeah.
B
B
A
B
Definitely
so
maybe
a
quick,
a
quick
context
of
what
a
message
poker
is
so
before
diving
into
Memphis
itself.
So
a
message
broker
for
the
people
that
watching
us
right
now
and
not
so
familiar
with
the
term,
because
sometimes
it
can
be
shown
as
messaging
queue
or
messaging
bus.
So
they
are
basically
from
the
same
family.
But
there
are
some
minor
differences
that
really
that
that
are
important
to
understand.
So
a
messagebook
is
a
messaging
bus
and
that
also
acts
as
a
temporal
Restore
for
events
and
moving
data.
B
Right
now,
from
Netflix
to
Uber
to
Lyft
all
the
way
to
small
startups,
which
require
to
a
collecting
and
processing
data
and
message
broker
is
the
core
that
enables
that
data
movement
between
the
different
applications
and
different
Services.
That's
like
the
the
general
idea
behind
a
message
book
or
a
queue
in
general.
B
And
so,
basically,
it's
about
reducing
a
lot
of
the
code
that
developers
build
around
the
message
Brokers
because
at
the
end
of
the
day,
like
Apache,
Kafka,
red
panda
and
and
other
cues
are
event
store
or
Temporaries
place
to
store.
Your
data,
while
in
transit
or
while
in
movement
between
microservices
and
Memphis,
is
all
about
doing
that,
but
also
give
you
a
full
ecosystem
around
it.
To
reduce
code
from
the
surrounding.
A
B
Exactly
so,
we
are
pushing
to
build
a
hub
of
connectors,
so
a
Marketplace
of
connectors,
very
similar
to
the
Simplicity
that
the
SDK
of
erbite
and
that
everyone
can
just
Implement
their
own
connector
in
into
airbites.
So
Memphis
is
pushing
to
build
the
same
ecosystem
of
connectors.
That
would
be
super
easy
for
everyone
and
to
implement
their
own
connector
for
any
third-party
vendors
or
consumed
and
other
connectors
that
build
by
the
community.
So
that's
the
first
one.
B
The
second
thing
is
processing
nowadays
and
you
would
need
to
stitch
Kafka
and
or
any
other
book
here
to
Flink,
for
example,
which
is
another
Apache
project.
So
we
took
both
of
the
worlds
and
put
in
and
put
it
under
the
same
Hood.
So
basically
we're
not
just
providing
the
ingestion
there
or
the
collection
of
the
data,
but
we
also
let
you
embed,
processing
an
application
and
functions
that
would
process
your
data
while
it's
in
inside
Memphis
and
the
third
one,
and
it's
all
about
schema
management.
B
So
when
you're
working
with
data,
actually
not
just
like
big
data
use
cases,
but
also
communication
via
apis
I,
think
that
many
many
developers
front-end
developers,
back-end
developers
will
will
will
have.
This
will
understand
that
struggle
when
a
from
the
developer
or
back-end
developers
needs
to
communicate
with
each
other
and
sharing
API,
schema
or
API
structures
and
change
this
key
or
change
that
value
or
change
this
type.
And
so
this
is
what
we
call
like
the
structure
of
the
data.
While
it
goes
between
microservices.
B
So
both
sides
would
be
aligned
on
the
data
structure
that
going
through
different
services,
and
so
so
those
three
pillars
are
like
the
developer.
Experience
features
that
we
put
it
put
in
Memphis
and
we're
constantly
building
and
I
think
that,
beside
that
beside
developer
experience
there,
there
are
three
more
pillars
that
Memphis
is
pushing
to
be
like
the
best
of
breed
in
the
industry,
which
are
performance,
we're
doing
a
lot
more.
B
A
lot
of
our
work
in
the
memory
and
cash
flow
and
we're
doing
our
best
to
be
the
most
resilient
message
broker
in
the
world.
So
basically
doing
everything
in
our
power
before
you
lose
a
message
or
your
cluster
gets
crashed,
which
I
think
helping
developers
sleep
better
at
night,
definitely
warts
the
worth
the
effort
and
the
third
one
would
be
the
developer
experience
as
as
I've
explained
before,.
A
B
Let's,
let's
do
it
yeah,
so
in
the
screen
sharing?
Basically,
we
see
Memphis
website.
We
can
also
get
the
installation
commands
through
our
GitHub
repo.
We
are
fully
open.
Source
always
will
be
open
source.
So
please
feel
free,
like
we
are
part
of
the
open
source.
Community
feel
free
to
contribute
and
to
to
visit
our
GitHub
people
to
maybe
share
your
thoughts
with
us
using
feature,
requests
and
issues
and
we're
always
welcoming
new
members,
and
so
even
we
can
see
like
both
of
the
runtimes.
B
That
Memphis
can
be
installed
on,
which
is
one
of
them.
Is
the
docker
swarm
for
more
local
environments,
laptop
environments?
If
you
just
want
to
spin
up
a
Memphis
cluster
and
try
out
its
features,
so
we
have
Docker
and
we
also
have
Elm
for
more
production
and
production
grade,
runtimes
like
kubernetes,
and
so
we
provide
Elm
installed
and
oh
Elm
based
install
package,
and
so
just
for
the
sake
of
the
demo,
and
we
would
choose
the
docker
version.
B
Hopefully,
everything
would
be
okay,
so
basically,
what's
going
on
right
now.
Is
that
we're
deploying
Memphis
cluster
with
Memphis
and
mongodb
version
and
also
Memphis
UI?
So
basically
the
is
here
just
for
application.
State,
we
store
what
happened
in
the
management
wrapper
and
in
the
UI
inside.
It's
not
where
and
data
store
itself.
The
data
I
store
is
stored
on
on
Memphis
cluster
itself
and
we
are
fully
ready
to
start.
B
B
B
You
thank
you.
Actually.
We
we
get
it
a
lot
and
people
love
the
brand
yeah.
The
idea,
basically
behind
the
brand,
would
be
to
bring
colors
and,
to
a
very
I,
would
say,
grace
grayish
topic
like
that
data
yeah,
and
that
also
what's
going
on.
What's
what's
the
origin
behind
the
the
name
Memphis,
and
to
create
a
vibrant
atmosphere
around
around
the
the
application
itself,
yeah.
A
B
And
so,
when
we
first
sign
up
to
Memphis
after
we
during
the
the
sign
up,
which
is
basically
our
first
user
to
the
system,
our
first
old
user,
we
get
the
next
steps
and
model
which
basically
guide
us
through
the
very
first
steps
of
creating
our
first
station.
Our
first
factory,
if
someone
is
familiar
with
Kafka
station,
is
basically
our
terminology
to
a
topic.
B
So
I'll
quickly
go
with
the
with
the
wizard
itself
and
we
will
see
the
magic
happen,
so
a
factory
name
would
be
a
demo,
and
station
name
would
be
channel
station
as
well.
I
might
need
to
change
some
things
in
my
code
later
on
so
a
retention
type
inside
the
topic
or
inside
the
station.
We
basically
can
say
it's
always
be
a
temporary
place
for
data,
so
we
have
to.
B
We
have
to
choose
some
retention
type,
so
it's
have
to
be
based
on
time,
size
or
messages
you
can
choose
based
on
your
use.
Cases
based
on
your
use
case,
so
just
for
for
the
sake
of
our
demo,
I
will
choose
a
retention
type
based
on
messages
and
I
will
just
send
messages.
So,
basically,
in
any
given
time,
our
station
will
store
10
messages
at
maximum,
so
we
cannot
push
more
messages,
I
mean
more
than
10
messages
and
it
will
create
an
automatic
cycle.
B
So
the
last
one
and
the
first
one
that
got
in
will
be
removed
and
sort
of
sort
of
a
fifo,
okay,
so
storage.
Let's
choose
memory,
we
know
how
to
work
in
the
memory
layer
and
not
just
persistency
on
file
and
username.
Let's
call
it
demo,
demo
user,
create
it
demo,
user
and
connection
token
Memphis
both
of
them.
We
will
need
it
for
our
SDK
connection.
Here
we
can
here
we
can
create
our
first
producer,
so
we
have
different
type
of
languages
that
we
support.
B
B
Consumer
can
also
produce
the
data
that
it
consumed
to
a
different
station,
and
that
brings
me
to
another
discussion
that
we
should
totally
have
someday
about
building
a
distributed
micro
Services
based
streaming
pipeline,
but
that's
basically
the
building
block
for
it,
so
node.js
and
I
will
quickly
go
to
it
hey.
What
does
this
directory
do
in
here?
Demo
result.
B
So
I
already
installed
Memphis
package
before,
but
we
can
definitely
do
it
again,
using
npm
package
manager
and
that's
it
and
here
I
can
just
copy
our
term.
Okay
and
here
I
can
just
copy
and
I
can
demo
code
just
for
you
to
to
be
able
to
quickly
onboard
Memphis
and
start
moving
data
and
do
it
and
from
it.
So
producer
data
and
I
already
created
like
such
a.
A
B
B
A
A
A
B
You
can
see
we
published
100
messages,
but
because
of
the
retention
type
based
on
messages,
and
we
we
configured
10
messages.
At
most
we
only
consumed
10
hello,
world
messages,
gotcha.
So
going
back
here
success
we
created
our
first
consumer,
that's
it.
Let's
go
to
the
station,
and
here
we
can
see
the
producer
that
is
currently
live
and
producing
data,
and
here
we
can
see
our
consumer,
which
is
live
and
also
consuming
data.
B
Here
we
can
see
the
messages
that
are
inside
the
station
at
the
moment,
usually
when
we
work
with
Kafka
and
other
Solutions
getting
the
internals
of
what's
going
on
inside
the
topic
or
inside
the
station
is
really
really
rough.
You
have
to
configure
several
stuff
to
get
that
observability.
So
again,
making
developers
live
Easier
and
happier.
We
put
it
right
in
the
middle
yeah
and
we
can
see
each
message
with
all
of
its
metadata
and
the
payload
itself,
and
now
I
wanted
to
demonstrate
a
quick
scenario
of
poison
messages.
So,
basically,
sometimes
not
sometimes.
B
Actually
it's
really.
It
happened
really
often
and
some
producers,
because
in
modern
organizations
we
have
hundreds
of
producers,
hundreds
of
consumers,
each
one
consuming
data
of
different
producers,
and
sometimes
something
happened.
Some
new
producer
joined
in
and
sending
message
that
the
other
consumer
for
some
reason
not
not
so
and
was
not
able
to
to
consume
it
and
not
acknowledging
the
message.
So,
basically,
we
have
a
function
called
acknowledge
and
which
basically
says
I
received
the
message
I
process.
It
don't
send
me
the
same
message
again.
A
B
When
the
service
on
the
other
side
gets
crashed,
didn't
process
the
message
correctly,
it
will
not
Arc
the
message,
so
we
don't
want
to
go
into
an
infinite
Loop
of
getting
the
same
message
again.
We
want
to
stop
everything
we
want
to
freeze
the
situation,
notify
the
developer.
Tell
him
hey
something
something
is
is,
is
went
wrong
and,
and
you
need
to
fix
it,
because
certain
consumer
cannot
consume
the
messages
and
imagine
the
scenario
that
that's
that
consumer
is
basically
responsible
for
payments
or
stripe
transactions.
So.
A
B
The
moment
your
organization
is
not
able
to
and
to
receive
payments
which
can
be
a
huge
damage.
So
to
emphasize
this
scenario,
I've
prepared,
like
a
small
and
actually
we
can
use
the
same
producer.
Nice
I
will
run
it
in
a
sec
and
just
to
show
the
code
for
it
for
for
a
second
and
consumerpoison.gs.
So
basically,
what
that
code
means
is
that
we're
creating
another
consumer
from
a
consumer
call
from
a
consumer
group
called
bicg,
which
says
the
max
message:
deliveries
that
I'm
able
to
receive
before
I'm
dropping
the
message
would
be
two.
B
So
if
I'm,
if
Memphis
tried
to
send
me
the
same
message
two
times
and
I,
didn't
acknowledge
it
stop
sending
it
to
me
in
in
other
Brokers.
It
basically
means
that
that
message
gone
but
Memphis
created
created
something
that
called
a
dead
letter
station
which
basically
gets
created
automatically
stored.
That
poison
message
that
unacknowledged
message
on
the
side.
Let
the
developers
know
that
there
is
something
wrong
fix
it
and
then
resend
it
to
the
problematic
consumer.
B
B
Fire
up
this
one,
so
it's
a
civil
world
and
it
knows
how
to
process
it.
So
everything
is
okay
and
it
gets
acknowledged-
and
here
we
can
see
the
bicg
and
my
consumer
and
everything
is
okay.
But
what
happens
if
we
start
to
produce
data
with
the
world
or
with
the
with
the
phrase
the
low
world
one
producer,
poison.
B
Exactly
and
our
consumers
saying
sorry
it's
a
message:
I
can
I
can't
handle
yeah.
What
happened
right
now
is
basically
that
here
we
can
see
10
poison
messages,
so
basically
it
received
messages
that
it
did
that
it
could
not
acknowledge
yeah
and
if
I
will
go
to
that
letter,
Q
I
would
see
all
of
those
messages.
So
we
didn't
drop
any
messages.
We
didn't
lost
any
message
and
what
happened
right
now
is
basically
I
can
do
two
things.
I
can
say:
maybe
it's
a
resource
issue.
B
A
B
A
B
B
We
can
see
two
things
now:
I
don't
receive
the
messages
immediately
because
Memphis
again
respect
the
idea
that
it
tried
to
send
two
times
the
same
messages,
so
it
won't
send
the
message
again
until
the
developer
will
say
that
I'm
ready,
resend
it
again
I
think
now
I
would
be
able
to
consume
it.
So
going
back
to
the
dead
letter
queue.
Just
you
see
our
fancy,
graphics
and
so
basically,
what
happened
here
is.
We
can
see
like
the
context
of
a
specific
message.
B
B
So
that's
the
main
idea
of
the
dead
letter,
q
and
I.
Think
that
really
emphasized
the
idea
behind
Memphis
of
resiliency
to
never
lose
a
message,
and
that's
really
we're
still
in
in
in
building
the
entire
building
blocks
of
Memphis.
And
so
more
features
are
about
to
come
in
the
coming
in
the
coming
months.
Actually,
but
really
that
demonstrates
the
amount
of
work
that
developers
don't
really
developers
don't
need
to
do
when
using
Memphis
and
not
the
other.
A
One
I
did
like
the
the
poison
message
feature
where
you
can
like
or
the
dead
letters
and
where
you
can
like
resend
stuff,
but
I
also
liked
how
you
you
visualized
it
to
see
like
where
the
producer
is
and
how
the
the
data
is
coming
in
through
there
and
then,
where
the
consumer
is
I'm.
Not
sure
I've
seen
that
from
other
message,
Brokers.
But
it's
not
like
I've,
explored
a
whole
bunch
and
then
I
also
liked
where,
when
it
found
the
poison
message,
instead
of
like
it
being
like
green
little
data
for.
B
A
B
B
Yeah,
so
I
will
quickly
read
it
once
again,
so
is
the
produce
data
held
in
memory
if
not
immediately
consumed
exactly
exactly.
It
doesn't
mean
that
it
doesn't
that
it's
not
persistent,
so
it's
still
persistent
and
and
as
a
redundant
storage
behind
it,
but
it
does
stay
in
the
memory
until
it
gets
consumed.
Yeah.
A
B
A
B
About
why
we
built
Memphis
in
the
first
place,
yeah,
definitely
we've
I
mean
I
will
not
speak
about
myself,
but
I
will
speak
about
my
friends
and
and
the
rest
of
my
co-founders,
which
are
basically
the
most
talented
people.
I've
ever
I've
ever
met,
I
mean
they're,
really
gifted
developers
and
and
gifted
data
engineers,
and
they
really
had
struggles
with
using
Kafka.
Now
Kafka
is
a
is
a
huge
product
with
a
huge
community
and
and
is
great
and
robust
solution.
B
Definitely
but
I
think
that
these
days
developers
as
the
hardest
job
in
the
world
right,
you,
you
you
getting
you
get
in
the
office
in
the
morning.
You
get
some
tasks
from
your
CTO.
You
know
what
you
need
to
do
in
general,
but
you
really
can't
draw
the
entire
steps
that
you
need
to
perform
until
you
get
there.
So
developers
are
really
really
busy
with
many
staff.
Many
Technologies
many
tax
tags
and
I
think
that
such
a
crucial
component,
like
your
messaging
bus
or
messaging
queue
message
broker
shouldn't
be
something
to
worry
about.
B
It
should
be
something
that
supports
your
efforts.
It
should
be
something
that
supports
your
your
code.
Your
I
mean
the
I
mean
at
the
end
of
the
day,
it's
all
about
extracting
value
from
your
data
right,
not
moving
data
around
and
not
and
not
handling
your
message
pocket.
So
that's
I
think
that's
the
main
idea
behind
Memphis
doing
a
lot
more
with
data
with
a
lot
less
time
and
a
lot
less
code
and
efforts
and
resources
and
headaches
really
yeah.
A
B
A
B
So,
first
of
all,
I
mean
use
Memphis
and
just
share
your
thoughts
with
us
and
that's
like
the
number
one
I
think
the
number
one
contribution
and
help
that
any
any
project
can
have
from
from
the
community.
So
if
you
can
share
your
feedback
with
us
and
and
thoughts
and
ideas
and
features
that
you
would
love
to
see,
that
would
be
amazing
and
the
second
thing
is
a
contribution.
It
doesn't
necessarily
mean
that
you
have
to
push
code,
create
pull,
requests
and
stuff
like
that.
You
can
just
I
mean
we
have
contributors.
B
A
Yeah,
let
me
put
the
website
for
y'all
on
here,
even
though
it's
a
pretty
simple
website,
if
you
all
are
interested,
you
can
find
the
map
memphis.dev
it'll,
have
links
to
the
rainbow.
It'll
have
links
to
the
Discord
and
their
Twitter
and
other
things
that
you
can
check
out
and
also
it
should
have
the
contributing
guidelines.
So
you
can
figure
out
how
to
make
a
contribution
which
we.
B
Always
yeah,
we
always
try
to
do
it
this.
We
always
try
to
modify
it
and
and
build
it
as
simple
as
possible.
If,
if
you
want
to
contribute
and
and
the
guide
is
still
not
very
straightforward,
please
reach
out
in
any
way
like
I
mean
through
Discord
through
my
email,
I
I
would
love
to
to
chat
with
everyone
and
and
really
doesn't
have
to
be
necessarily
directly
to
Memphis.
If
you
have
any
question
regarding
data
engineering,
developing
Korean
questions,
either
even
I
I
always
I
always
welcome
and
happy
to
meet
new
people.
Yes,.
A
A
A
B
B
A
Nerve-Wracking,
but
thank
you
so
much.
This
was
a
great
stream
for
me
and
I
hope
it
was
for
you
as
well.
I'll,
probably
upload
this
to
YouTube
in
the
in
the
near
future,
but
thanks
for
showing
us
Memphis
thanks
for
working
to
make
things
like
message
Brokers,
even
though
it
may
sound
like
a
little
bit
more
boring
like
a
message
broker,
may
sound
bro
and
thanks
for
trying
to
make
it
more
exciting
and
easier
to
navigate,
and
also
thanks
for
whoever
tuned
in
thanks
for
listening.
B
Thank
you.
Thank
you.
Everyone
thank
you
for
joining
us
today.
Thank
you.
Thank
you,
Rizal
and,
and
again
it's
basically
at
the
moment.
You
can
find
a
really
really
fancy
strong,
powerful
message
broker,
but
we
are
heading
towards
a
very
interesting,
fascinating
places
and
that
will
take
data
engineering
and
data
pipelines
to
a
whole
different
places.
So
I
I
really
encourage
everyone
to
stay
tuned
and
and
follow
us
yeah.