►
Description
What is CDC? Why is it getting so popular? Get your espresso ready for the OpenShift.TV Coffee Break, as our special guests Hans-Peter Grahsl and Chris Cranford will introduce us Debezium, an open-source project to implement Change Data Capture pattern with your favorite database and data streaming solutions.
A
A
A
Coffee
break
this
morning
at
10
A.M,
European
time
it
looks
like
it's
3
A.M
in
the
night
before
some
of
our
guests,
so
big
shout
out
to
Chris
we're
gonna
introduce
all
the
guests
in
a
few
moments.
Let
me
please
introduce
myself.
My
name
is
natala
vinta
I'm
product
marketing
manager.
Here
at
openshiftv
and
I'm,
together
with
my
fantastic
co-host,
Fabio
Fabio.
A
All
right
all
right
for
the
people
that
doesn't
know
what
is
openshift
TV
coffee
break
is
a
weekly
show
where
we
talk
about
openshift,
kubernetes,
Cloud
native
in
a
web
TV.
This
is
a
kind
of
a
emea
time
zone
like
show
so
I'm
very
happy
to
have
today
our
super
special
guest.
Let
me
introduce
first,
our
special
guest
from
another
time
zone,
which
is
Chris,
hey
Chris.
How
are
you
I'm
doing
fine?
How
are
you
guys?
Oh
final,
fine,
thank
you
for
joining
us
so
early
in
the
morning.
A
D
A
A
Fantastic,
thank
you.
Chris
can
thank
you
once
Peter
for
the
introduction,
because
today's
topic
is
really
the
video
and
change
data
capture,
which
is
a
pattern
right.
We
noticed
in
the
I.T
community
in
the
market
that
this
pattern
is
getting
a
momentum.
That's
why
we
invited
you
to.
Please
explain
us
why
this
momentum
and
what
this
really
this
pattern,
but
before
we
go
into
the
technical,
is
that
Fabio
I
like
to
say
hello
to
our
YouTube
and
twitch
attendees
people?
We
were
watching
us.
Thank
you.
A
Can
you
please
write
in
the
chat?
Where
are
you
from
where
you
are
attending
because
we
see
you
know
the
this
is
a
a
show
that
arises
to
many
countries.
Let
us
know
in
the
chat,
if
you,
if
you
want
and
please
if
you
have
any
question
about
the
pigeon
and
change
data
capture
and
everything,
is
there
send
the
question
in
the
chat?
We're
gonna
bring
the
question
to
the
to
the
speakers
today.
C
Yeah
sure,
let's
start
into
that,
so
we
we
we
can
Dive
Right
In,
and
we
brought
a
couple
of
slides,
of
course,
that
that
should
help
you
also
on
the
stream
a
little
bit.
So
the
BCM
I
I
think
is
best
described
as
an
open
source
change
data
capture
platform
and
it
it
is
basically
a
supporting
a
family
of
different
databases.
We
come
to
that
in
in
the
next
couple
of
slides.
It
is
used
to
actually
read
from
a
databases,
transaction
log.
C
It
has
a
couple
of
nice
features.
It
allows
to
do,
for
instance,
snapshotting,
meaning
that
you
can,
when
you
start
initially
to
to
work
with
your
data,
and
you
want
to
expose
it
to
other
services,
you
can
take
a
so-called
snapshot,
and
this
brings
all
the
existing
data
in
your
database
into
a
different
systems.
We
come
to
that
a
bit
later.
How
we
can
do
that,
because
there
are
different
ways
to
do
it
most
often
it's
done
propagating
these
changes
through
Kafka.
You
can
filter
it.
C
It
has
also,
since
not
too
long
ago,
a
really
nice
web-based
UI
that
you
can
use
to
configure
different
types
of
source.
Connectors
that
you
want
to
run
and
it's
fully
open
source,
it
has
a
very
active
community
and
I.
Think
Chris
can
in
particular
add
a
couple
of
more
notes
here
regarding
also
different
types
of
of
really
large
production
deployments
that
they
see
and
and
add
here
a
couple
of
more
input.
D
D
So
you
know
we
do
have
that
Vibrant
Community.
A
lot
of
people
are
constantly
engaging
with
one
another
through
the
mailing
list,
through
our
chat
and
and
other
avenues.
So
really,
you
know
huge
shout
out
to
our
community
for
really
helping
support
us
through
this
project
and
really
helping
us.
You
know,
add
new
features
and-
and
you
know,
regression
testing
and
so
much
of
that
stuff.
So
that's
really
really
great
in
terms
of
large
production
deployments,.
B
D
They
have
been
using
this
now
for
about
I
believe
six
to
eight
months
when
it
was
going
through
its
incubation
period
and
things
like
this.
We
also
have
other
other
big
large
deployments.
I
know
the
there's.
There's
a
discussion
going
on
with
some
customers
in
Italy.
We
have
some
customers
in
India
that
are
also
looking
to
use
the
museum.
D
Yeah
and
then
I
guess
and
then
I
guess
I
can
talk
about
this
and
then
so
in
terms
of
connectors
with
the
museum.
The
the
biggest
thing
here
is
that
each
of
these
databases
they
they
have
their
own
format.
They
have
their
own
way
in
which
we
have
to
kind
of
interact
with
the
databases
to
get
these
changes,
and
so
you
know
it
kind
of
means
that
from
a
business
perspective,
we
have
to
integrate
with
each
of
these
in
their
own
way.
D
Here,
as
we
are
working
with
these
different
databases
with
whether
it
be
Oracle,
whether
it
be
Cassandra,
whether
it
be
MySQL
really
this.
This
connector
framework
that
division
brings
to
the
table
allows
you
to
be
able
to
kind
of
focus
specifically
on
ingesting
those
change
events
using
the
apis
that
that
data
that
that
database
requires
us
to
integrate
with,
and
so
we'll
kind
of
talk
about
and
touch
on
this
a
little
bit
later,
but
not
only
just
using
the
connectors
themselves.
Does
this
make
this
really
important?
C
There
Hans
Peter,
no,
that's
fine
and
I
think
what
we
should
definitely
touch
upon
here
is
the
the
two
fundamentally
different
ways
to
do:
change,
data
capture
and
integrate
with
databases
you
or
data
stores
in
general
and
and
so
here
on
on
that
slide,
we
tried
to
summarize
that
a
bit
because
at
one
point
remember
you
you
could
think
well.
Why
do
I
need
that?
Why
can
I
not
just
do
and
leverage
a
query
based
approach?
C
One
is
think
about
that.
Maybe
something
changed
and
changed
again
in
within
your
appalling
interval.
Then
you
might
lose
one
of
those
changes
because
you
haven't
seen
that
why?
Because
it
happened
during
the
poll
interval,
another
thing
that's
usually
hard
to
do
is
to
capture
deletions,
because
how
would
you
easily
query
for
something
that
has
been
there
once,
but
isn't
there
any
longer?
Also,
very
often
you
want
to
understand
okay,
not
only.
D
To
point
out
the
the
one
point
here
about
the
transparency:
what's
really
really
important,
at
least
for
me,
is
that
by
using
the
BCM
and
actually
using
the
transaction
logs
to
being
able
to
do
this
log
base
CDC
approach.
It
really
prevents
you
from
needing
to
have
to
like
manipulate
your
data
model.
You
don't
want
those
two
things
to
necessarily
be
integrated,
because
going
back
to
what
you
said.
D
If
you're
wanting
to
do
this
polling
interval
you're
going
to
have
to
have
something
in
your
data
model,
that's
going
to
that
you're
going
to
use
to
be
able
to
know
whether
or
not
from
this
time
stamp
to
this
time
stamp
what
records
have
changed
and
when
you
start
kind
of
mixing
those
concerns.
It
creates
this
on
really
kind
of
like
unnecessary
coupling
that,
at
least
for
me,
I,
really
dislike
and
I.
D
Think
most
people
would
would
agree
in
this
regard,
so
being
able
to
use
a
log
based
approach,
really
kind
of
gives
you
the
ability
to
being
able
to
not
only
capture
all
of
those
individual
changes,
as
you
spoke
about,
but
just
really
being
able
to
allow
your
data
model
to
focus
on
what
it's
meant
to
support
in
your
business
application,
without
really
integrating
concerns
that
you
need
inside
that
business
model
for
change.
Data
capture.
B
Yeah
Chris,
what
you
are
telling
is
I
think
he's
a
great
Concept
in
in
the
world
of
microservices,
where
the
loser
coupled
integration
are
key.
So
keeping
the
communication
between
microservices
without
any
need
to
couple
with
a
specific
data
model
is,
is
great
because
it
allows
to
encourage
me
if
I
want
to
apply
all
that
patterns
to
related
to
the
deploy
as
soon
as
possible
stuff,
without
with
the
lowest
impact
on
the
other
surrounding
microservices.
That
that
specific
one
is
going
to
to
be
integrated.
B
C
Right,
yeah,
and
so
remember,
we
talked
about
that.
We
have
different
ways
to
to
deploy
or
to
use.
In
fact,
dbzoom
and
I
think
still
correct
me
if
I'm
wrong
Chris,
but
still
the
most
common
way
of
doing
it
in
in
the
wild,
is
still
by
using
it
in
in
the
original
way
that
was
supported
from
the
very
early
days
of
the
PCM,
namely
that
you
use
it
within
Kafka
connect.
C
And
so
you
have
these
Source
connectors
for
Kafka
connect
that
are
again
specifically
written
for
these
different
types
of
data
stores,
and
you
run
them
within
a
Kafka
connect
environment.
And
then
those
changes
are
propagated
into
Kafka
topics.
And
what
you
see
there
is
that
you
have
essentially
a
one-to-one
relation
between
a
table
in
a
database
and
a
topic
in
Kafka,
where
you
would
find
all
those
changes
that
happened
to
a
particular
a
table
and
then
from
Kafka
once
it's
there.
C
You
can
work
with
different
types
of
sync
connectors
and
feed
any
of
these
change
data
events
into
your
target
system
of
choice,
and
here
we've
just
a
couple
of
those
that
are
are
pretty
popular
listed
here,
so
think
about
bringing
certain
changes
into
elasticsearch,
where
you
might
want
to
build
some
kind
of
full
text,
indexing
solution
or
think
about
that.
You
use
Infinity
span
as
a
cache,
and
you
can
then
employ
those
change.
C
Events
to
to
update
your
cache
or
to
to
initially
also
warm
up
your
cache
in
that
regard
or
in
general
use
any
kind
of
jdbc
sync
connector
to
propagate
that
further
in
in
in
in
data
warehousing
infrastructure,
and
things
like
that
and
again
you
capture
inserts
updates
and
deletes
there
and
they
are
propagated
in
exactly
that
way.
Using
a
Kafka
topics
in
this
case.
But
that's,
of
course,
not
the
only
way
to
use
division
later
on.
C
D
Correct
so
you
know
like,
like
you
said,
you
know,
we
we've
talked
about
Kafka
connect
but
and
you're
correct.
Ninety
percent
of
most
of
our
users
are
using
division
within
Kafka
connect,
but
with
the
embedded
engine.
This
is
pretty
much
taking
the
BCM
as
a
library,
and
it
allows
you
to
be
able
to
kind
of
embed
the
division
engine
itself
directly.
Inside
of
your
application,
you
may
have
specific
requirements
as
to
why
you
might
want
to
do
this.
D
D
Events
manipulate
those
and
emit
those
events
using
their
their
framework
programmatically,
but
then
there's
also
another
component
called
the
BCM
server
and
this
is
using
the
embedded
engine,
but
in
a
ready-made
runtime
environment,
it's
kind
of
very
similar
to
Kafka
connect,
it's
meant
to
kind
of
fulfill
the
same
role,
but
with
a
slightly
different
Focus.
It's
really
kind
of
designed
to
send
these
change
events
out
to
other
messaging
infrastructures.
So,
for
example,
we
have
adapters
for
Apache
Pulsar
Amazon
Kinesis,
redis,
Google,
Pub
sub.
C
Right,
yeah,
and
so
there
is
just
as
a
hint
there
was
not
too
long
ago,
Chris
showcased,
essentially
this
division
embedded
engine
in
one
of
the
recent
caucus
inside
episodes.
So
there's
the
link
to
that.
If
you
want
to
look
into
that,
he
does
essentially
that
the
whole
session
is
basically
a
live
coding
session
where
he
nicely
shows
what
it
would
take
to
use
the
typesium
embedded
engine
in
in
a
quacos
applications.
It's
a
really
nice
way
of
seeing
how
how
this
actually
works.
C
So
it's
about
simple
cases
are
replicating
data
between
two
systems,
but
again
we
we
mentioned
that.
Okay,
why
do
you?
Why
do
you
do
that?
Maybe
you
feed
search
in
the
indexes.
You
want
to
update,
search
indexes,
a
full
text,
search
engine,
maybe
in
solar
or
or
in
elasticsearch
or
wherever.
Maybe
you
have
a
cache?
You
need
to
warm
up
the
cache.
C
You
need
to
invalidate
the
cache
due
to
all
of
these
changes
that
are
happening
elsewhere
in
your
data
store
and,
and
that's
all
very
valid
use
cases
that
become
possible
pretty
pretty
in
pretty
straightforward
way.
I
would
say
thanks
to
this
really
nice
idea
of
of
change
data
capture
in
that
regard,
also,
micro,
Services
architectures.
Very
often,
you
need
to
somehow
communicate
changes
happening
in
one
particular
service
to
other
services,
and-
and
here
we
also
see
later,
Chris
also
mentioned
it.
C
One
particular
so-called
CDC
pattern
that
that
helps
in
that
regard
a
lot,
namely
the
outbox
pattern,
and
then
there
are
a
couple
of
others.
So
basically
what
you
can
see
CDC
also
as
as
this
giant
enabler
as
it
says
here,
and
it
sets
free
your
data
that
was
originally
as
if
it
was
locked
away
in
a
database
and
you
could
not
easily
get
it
out
and
had
probably
a
tough
time
because
think
about
all
those
Legacy
systems
that
you
might
not
be
able
to
touch.
Maybe
you
don't
have
the
source
code.
C
Maybe
you
are
not
allowed
to
touch
the
source
code
for
that
system
and
it's
just
running,
but
still
you
want
to
build
new
functionality,
new
services
around
it
and
and
that
this
is
how
you
can
then
tap
into
all
the
changes
that
are
happening
in
the
database
of
such
a
legacy
application
and
then
propagate
it
to
any
Downstream
consumer
that
you
want
to
use
yeah.
Maybe
a
few
insights
here.
Maybe
Chris
wants
to
explain
a
little
bit
how
how
you
know
the
the
actual
payloads
that
the
museum
generates.
Basically
look.
D
Like
sure,
absolutely
so,
each
of
our
division,
events
and
kind
of,
if
you
think
about
this
from
a
relational
table
perspective,
there
is
a
key
and
then
there's
some
payload.
That
comes
along
with
each
of
these
events.
Generally,
we
map
the
key,
that's
associated
to
these
events,
to
the
table's
primary
key,
but
of
course
this
is
configurable
for
the
user.
D
The
first
here
is
the
source
block.
This
is
really
really
important
because
it
provides
you
a
lot
of
metadata
information
about
where
we
sourced
this
particular
event
from
it
could
contain
things
such
as
the
database,
the
scheme
of
the
table,
maybe
even
the
transaction
reference
in
the
case
of
things
like
Oracle,
it's
going
to
provide
you
even
more
deeper
metadata
information
about
which
particular
node
in
the
Oracle
rack
cluster
that
that
event
took
place
on.
D
So
you
could
really
kind
of
use
this
information
as
you're
going
through
and
scraping
the
events
that
are
inside
your
Kafka
topic.
You
could
really
cross-reference
every
one
of
those
events
back
to
the
transaction
log
on
the
database.
If
you
wanted
to
using
this
information,
and
then
we
also
capture
things
like
what
is
the
operation
type,
is
it
an
insert,
is
an
update?
Is
it
a
delete
or
is
it
something
that
represents
a
read
event
where
we
did
the
initial
snapshot
and
then,
of
course,
things
like?
D
When
did
we
process
this
particular
event,
and
all
of
these
payloads
can
be
serialized
to
Apache
Kafka
or
to
any
other
messaging
platform,
using
a
number
of
different
serialization
techniques,
whether
it
be
Json,
whether
it
be
Avro?
That's
really
entirely
up
to
you,
because
this
entire
process
is
completely
configurable
and
that's
really
kind
of
the
key.
At
the
end
of
the
day,
a
lot
of
dupism
users
are
not
programmers.
They
are
maybe
data,
analysts
or
maybe
they're
a
little
bit
tech
savvy
and
by
really
allowing
the
beesium
to
be
completely
configurable
by
configuration.
D
We
also
support
being
able
to
emit
our
change
events
using
Cloud
events
if
you're
not
familiar
Cloud
events
is
a
somewhat
standard
technique
for
emitting
events
and
a
particular
type
of
format,
and
so
all
of
our
division
events
can
be
used
with
the
cloud
events
converter
and
configured
as
such
to
being
able
to
emit
those
events
using
this
specific
format,
and
if
you
notice
it,
it
follows
a
very
similar
technique.
There
is
an
identifier
at
the
top
of
the
event.
D
There
are
a
number
of
different
metadata
key
value
pairs
that
are
going
to
be
paired
as
well.
In
this
event,
they're
going
to
all
be
prefixed
with
IOD
beesium,
followed
by
some
additional
descriptor
afterward
to
describe
what
does
that
value
represent
such
as
whether
or
not
it's
the
schema
or
the
table
a
lot
of
these
attributes?
If
you,
if
you
go
back
and
you
look
at
the
previous
slide-
you'll
see
they're
they're
all
mapping
back
to
like
the
source
info
block.
D
B
D
It
variable
could
be
yes,
okay
and
I
mean
again
going
back
to
what
we
were
talking
about
earlier,
with
division
server
and
having
k-native
Eventing
as
a
sync
option,
with
division.
Server
really
kind
of
enables
this
pattern
very
fully,
because
at
that
point
you
can
run
the
museum
inside
of
a
a
pod
capture.
Those
events
convert
those
events
to
Cloud
events
and
then
emit
those
directly
into
kubernetes,
using
k-native
Eventing
as
well.
Okay,.
B
D
And
then,
if
you,
if
coming
back
to
the
cloud
event,
we
have
a
data
section,
and
this
section
very
much
mirrors
what
you
would
see
if
you
were
looking
at
this
inside
of
a
Kafka
a
Kafka
event.
We
have
a
section
for
with
a
schema
that
describes
what
our
payload
structure
looks
like
what
are
all
the
fields,
their
respective
data
types
and
then,
of
course,
in
the
payload
field.
D
D
You
know,
we've
been
quite
at
work
over
the
last
number
of
major
iterations
of
really
trying
to
bring
a
ton
of
new
features
to
the
CDC
platform
and
one
of
the
biggest
new
features
that
we
added
is
a
feature
called
incremental
snapshots,
and
we
introduced
this
into
BSM
1.6,
and
the
idea
here
with
incremental
snapshots
is
that
this
allows
you
to
be
able
to
take
a
snapshot
of
your
existing
data
concurrently,
while
the
connector
is
streaming
changes
from
the
transaction
logs.
This
also
has
the
benefit
as
well
that
this
snapshot
is
resumable.
D
So,
let's
say,
for
example,
you're
running
your
connector,
your
snapshot
at
your
incrementally
snapshotting,
your
data
you're
capturing
changes
from
the
transaction
log,
and
you
need
to
shut
your
database
down
for
a
moment
to
make
some
changes
and
then
bring
it
back
up.
That
snapshot
is
completely
unaffected.
D
The
traditional
snapshots
in
this
particular
scenario
would
require
you
to
rerun
the
snapshot
from
the
beginning,
so
that
is
one
of
the
really
huge
advantages
of
using
incremental
snapshots
with
division,
but
we've
also
improved
upon
this
process
in
recent
versions:
adding
the
ability
to
being
able
to
stop
or
running
incremental
snapshot
being
able
to
partially
stop
parts
of
the
incremental
snapshot.
Let's
say:
you're
you're
wanting
to
snapshot
two
tables,
and
you
want
to
stop
snapshotting
one
of
those,
but
you
want
the
other
to
continue.
D
You
have
this
option
now
of
being
able
to
do
this,
you
could
additionally
choose
to
pause
and
resume
those
incremental
snapshots
and
there's
also
a
new
feature
that
was
added
I,
believe
in
dpsm
1.9.
That
allows
you
to
be
able
to
even
apply
a
predicate
condition
to
the
incremental
snapshot
process.
So
if
you
want
to
only
focus
on
capturing
a
subset
of
the
data
from
a
table,
instead
of
all
the
data
again,
you
you
have
that
option
and
that
flexibility
of
being
able
to
do
so.
D
Some
additional
changes
that
we've
also
added
in
recent
versions
include
adding
change
stream
support
to
the
mongodb
connector
mongodb
was
originally
written
and
implemented
to
read
and
to
tell
the
oplog
of
the
database
or
the
transaction
log,
and
we
have
moved
away
from
this
and
recent
versions
to
using
mongodb's
native
chain
stream
support.
This
provides
so
many
more
features
than
what
the
traditional
oplog
approach
provided
previously
and
some
recent
versions
that
I
believe
in
division,
one
1.9
or
2.0.
D
When
we
introduced
to
this
there
is
the
ability
to
be
able
to
capture
the
full
documents
of
that
change.
Now,
so
there's
there's
tons
of
new
features
that
came
along
with
adding
this
chain
stream
support
for
the
mongodb
connector
and
then
just
to
kind
of
jump
forward
a
little
bit
in
1.9.
We
added
support
for
multi-database
support
using
SQL
Server.
This
is
really
really
huge
from
the
perspective
of
any
environment
where
you
have
a
single
installation
and
the
single
database
installation
supports
multiple
databases,
so
you
have
this
in
SQL
Server.
D
2.0
Java
11
is
now
required
and
we
also
have
added
a
new
feature
for
mongodb
6
and
later,
which
is
being
able
to
provide
you.
What
was
the
state
of
the
document
before
the
change
occurred,
and
this
is
really
starting
to
take
mongodb
from
this
perspective,
of
just
being
able
to
tell
you
what
has
changed
or
what
does
the
document
look
like
after
the
changes
occurred
to
really
kind
of
moving
to
be
more
to
what
that,
before
and
after
state
looks
like
and
those
change
events
from
the
relational
connectors
approach.
D
C
D
D
We're
also
looking
to
bring
Maria
DB
as
a
first
class
connector,
and
then
we're
also
looking
at
implementing
things
like
exactly
one
semantics,
which
Kafka
connect
and
Kafka
nail
support.
So
we're
looking
to
do
this
in
a
multi-phase
approach
next
year,
starting
with
a
couple
of
connectors
initially
and
then
rolling
that
out
to
other
connectors
throughout
the
year
and
then,
of
course,
there's
some
UI
improvements
to
our
UI
division.
D
Ui
component,
we
want
to
be
able
to
allow
you
to
be
able
to
do
things
like
editing,
a
connector
being
able
to
see
some
key
critical
metrics
inside
the
UI
and
so
much
and
so
much
more
so
there's
a
lot
of
advancements
and
features
coming
for
the
UI
next
year
so
be
sure
to
check
that
out
and
keep
an
eye
out
on
the
release.
Notes.
C
Sure
so,
with
that,
let's
talk
about
now
two
selected
CDC
patterns
to
see
how
you
can
like
make
use
of
CDC.
You
do
for
doing
something
called
the
outbox
pattern
or
then
afterwards.
We
look
into
another
pattern,
called
the
strangle
effect
pattern,
but
I
think
Chris
briefly
starts
to
explain
the
outbox
pattern.
A
bit.
D
The
idea
is
that
you
want
to
be
able
to
kind
of
keep
these
micro
services
and
their
own
silos.
We
want
them
to
have
their
own
database.
We
want
them
to
be
able
to
be
able
to
morph
and
evolve
independently
of
other
microservices
that
might
be
participating
in
the
topology
and
in
order
to
be
able
to
do
this,
you're
still
going
to
need
to
have
some
kind
of
mechanism
to
being
able
to
share
data
between
those
microservices
and
one
reliable
way
of
being
able
to
do.
That
is
with
the
outbox
pattern
itself.
D
Maybe
when
we
save
the
order,
we
need
to
be
able
to
tell
the
shipment
service
hey
now,
go
ahead
and
ship
this
order
for
the
customer,
but
we
want
to
do
this
in
a
decoupled
way,
and
so
in
this
particular
case,
even
if
Kafka
could
participate
in
the
distributed
transaction,
which
you
know
some
people
may
come
to
this
approach
and
go
well.
You
know
I'll
just
do
this
distributed
transaction
and
and
facilitate
this
kind
of
this
kind
of
orchestrated
transaction
across
these
Services.
D
D
Out
of
the
box.
I'm
sorry,
a
number
of
different
columns
out
of
the
box.
The
outbox
adapter
expects
these
particular
types
of
columns
to
be
present.
But
this
is
somewhat
configurable,
but
it
really
allows
you
to
be
able
to
kind
of
identify
and
produce,
what's
called
an
Aggregate
and
being
able
to
provide
that
information
to
those
other
consumers,
whether
it
be
a
shipping
service,
whether
it
be
customer
service
or
whatnot,
and
a
real
decoupled
way.
D
First
of
all,
I
just
want
to
point
out
for
those
for
those
watching.
We
do
have
an
example
repository
where
this
particular
outbox
example
is
sourced
from,
if
you're
ever
interested
in
understanding
how
to
use
the
beesium
in
a
variety
of
different
ways,
there's
tons
of
different
implementations
here,
whether
it
be
you're
wanting
to
integrate
with
camel
you're
wanting
to
integrate
with
a
cache
or
you're
wanting
to
do
some
kind
of
case
streams
or
k-sql
functionality
inside
of
Apache
Kafka.
D
All
of
that
stuff
is
here
so
feel
free
to
check
out
our
examples
repository,
but
we're
going
to
be
taking
a
look
at
this
outbots
example
here
in
particular,
and
what
I
have
done
is
I
have
gone
ahead
and
started
up
my
Docker
compose
file,
and
we
can
take
a
look
at
this
here.
Let
me
it's
a
little
bigger,
so
what
we
have
in
our
Docker
compose
here
is
we
have
a
couple
of
services,
we're
running
zookeeper
and
Apache
Kafka.
We
have
a
couple
of
databases
here.
D
We
have
one
for
our
order
database
and
one
for
our
shipment
database.
This
again
is
mirroring
that
Silo
for
the
order,
service
and
The
Silo
for
our
shipment
service,
and
then
we
have
Kafka
connect
running
and
here
we're
actually
leveraging
the
use
of
streamzy
in
conjunction
with
division
of
being
able
to
run
this
together
inside
that
container
and
then
finally,
I
have
our
order
service
and
our
shipment
service.
D
D
So
the
way
I'm
going
to
do
this
I'm
just
going
to
register
this
Json
configuration
with
Kafka
connect
and
if
I
scroll
up
here
I
can
take
a
look
and
what
I'm
going
to
see
is
I'm
connecting
to
my
order.
Database
and
I
am
going
to
be
monitoring
or
capturing
changes
from
this
outbox
event
table
and
I'm
going
to
be
routing
those
events
using
the
outbox
transformation
or
single
message,
transformation
through
into
Apache
Kafka.
D
So
if
I
were
to
pull
up
I,
think
I
have
it
here
somewhere,
there's
this
image
that
we
were
just
showing
on
the
previous
slide.
What
I
have
basically
done
at
this
point
is
I
have
set
up
my
CDC
with
divisium
to
being
able
to
monitor
this
outbox
table
this
component
here
and
being
able
to
send
those
events
into
Apache
Kafka
here,
the
metal
so
at
this
particular
Point.
There's
no
data
in
my
orders
table
so
I
need
to
generate
an
order,
so
I'm
going
to
do
that
by
using
the
order.
D
Services
restful,
API
I'm
just
going
to
create
an
order
here
and
if
I
scroll
up
here,
you
can
see
that
I
have
created
an
order
with
id1.
With
a
couple
of
lines
here,
I've
decided
to
order
two
of
the
division
in
action
and
I've
decided
to
order
one
of
the
bees
in
for
dummies
books
and
then,
let's
say
I
made
a
mistake.
I
want
to
actually
cancel
one
of
those
so
I'm
going
to
cancel
my
order
for
the
abusing
for
dummies,
but
I'm
going
to
keep
my
order
here
for
the
museum
in
action
now.
D
D
So,
let's
jump
over
into
our
order
database
and
just
to
be
able
to
confirm
that
all
of
our
order
information
was
written
to
the
database,
which
we
could
see
here
in
the
log
above
us,
but
you
know
just
to
be
sure
we
can
see
here.
We
have
this
order
and
we
have
two
line
items
and
again
we
can
see
that
the
division
for
dummies
was
canceled.
D
We
can
see
rdb's
in
for
Action
is
still
in
entered
status
and,
if
I
take
a
look
at
my
outbox
table,
we're
going
to
see
there's
no
data
in
this
particular
table
and
that's
expected
and
the
reason
this
is.
The
reason
this
is
expected
is
because,
when
you're
working
with
the
outbox
pattern,
we're
really
only
ever
interested
in
the
insert
events,
we're
not
interested
in
updating
the
events
that
or
updating
rows
that
are
in
this
table.
We're
not
interested
in
deleting
rows
in
this
table.
D
So,
typically,
when
you're,
using
this
pattern,
you're
going
to
write
a
record
to
this
table
and
you're
going
to
immediately
remove
the
record
from
the
table
just
so
that
you
don't
necessarily
need
any
kind
of
housekeeping
on
the
outbox
table
over
the
duration
or
life
cycle
of
your
service.
Now
some
users
may
decide
well
I
want
to
keep
this
data
for
debugging
purposes.
You
certainly
can
do
this
and
then
purchase
this
data
at
some
future
interval
or
on
some
interval,
but
that's
completely
up
to
you.
You
can
do
it
as
you
see
fit.
D
It
will
have
no
impact
on
the
actual
delivery
of
those
events.
Using
the
outbox
pattern
with
divisium
and
then,
if
I
jump
over
into
my
shipment
database,
let's
take
a
look
to
see
what
we
have
over
here.
I
can
see
now
that
I
have
notified
the
shipment
service.
The
shipment
service
sees
that
my
order
has
been
received
here
for
customer
one.
D
D
D
It's
all
driven
by
configuration.
It
really
keeps
the
the
business
concerns
of
your
order
service
completely
separate
from
your
shipment
service.
It
allows
you
to
be
able
to
evolve
your
data
model
in
each
of
those
Services
completely.
However,
you
need
to
evolve
those.
It
also
allows
you
to
be
able
to
define
a
message
structure
that
you
want
to
be
able
to
use
in
your
outbox
table
to
be
able
to
communicate
between
these
services,
and
that
structure
could
be
very,
very
different
and
you
can
even
evolve
that
structure
over
time.
B
Foreign
cool,
if
I,
because
when
we
speak
about
microservice
and
and
data
management,
usually
there's
a
lot
of
it
could
be
dangerous
Integrations
because
you
most
of
the
time
end
up
doing
strange
stuff
to
keep
everything's
up
to
date.
In
this
way,
this
is
a
great
way
to
keep
accomplished
things
simple:
make
it
really
simple
but
effective,
sometimes
so,.
D
Exactly
and
and
the
beauty
about
this
is
that
the
payload,
like
I,
said
the
payload
that
you're
sending
or
communicating
between
these
microservices
is
really
entirely
user
driven.
So
whatever
you
want
this
payload
column
to
contain,
you
really
can
kind
of
you
know,
even
whether
you
want
to
share
the
data
between
these
microservices
as
an
aggregate,
whether
you
want
to
send
individual
events
to
represent
rows
or
whatever
it's
really
entirely
up
to
you
and
your
business
requirements,
but
it
really
yeah.
D
A
D
You
know.
People
have
also
used
things
like
camel
to
do,
polling
to
being
able
to.
You
know,
capture
those
events
and
things
like
this.
You
know,
but
then
you
look
at
other.
You
look
at
other
architectures
of
other
applications.
For
example,
in
past
lives,
I've
worked
with
Oracle,
Erp
and
and
their
particular
example.
In
their
scenario,
they
do
try
to
approach
this.
From
a
perspective
of
where
your
order,
entry
environment
is
separate
from
your
accounts,
receivable
and
accounts
payables
right,
so
they
try
to
have.
D
Even
though
all
the
data
is
within
the
same
database,
they
do
try
to
Silo
things
as
much
as
possible,
and
then
they
use
things
like
interface
tables,
which
is
a
very
similar
way
of.
Like
the
outbox
pattern
of
being
able
to
put
data
into
a
table.
You
then
have
another
process:
that's
responsible
for
reading
this
table
and
ingesting
it
and
manipulating
it
and
importing
it
into
another
subsystem
inside
of
the
database
that
we're
inside
of
that
application,
and
so
really
this
this
pattern
we've
been
using
this
for
quite
a
long
time.
D
D
D
C
A
C
So
then,
let
me
explain
it
briefly
on
a
couple
of
slides
and
then
I
try
to
show
you
the
a
demo
for
that
as
well.
Okay,
yep!
Thank
you
great.
So
the
strangle
effect
pattern.
We
mentioned
that
earlier,
as
well
as
the
other
pattern
that
we
wanted
to
show
you,
and
very
often
so.
Chris
already
touched
upon
also
a
micro
service
architectures
a
couple
of
times,
and
so
sometimes
we
don't
start
with
microservices
architecture.
Sometimes
we
have
Legacy
applications.
C
We
have
monoliths
that
we
want
to
probably
gradually
evolve
into
a
different
architecture
and
let's
say
we
want
to
go
from
a
monolithic
application
towards
microservices,
and
the
idea
is
we
want
to
do
it
step
by
step.
Usually
we
know
from
experiences
in
the
field
that
trying
to
to
do
a
a
big
rewrite
in
one
go
is
very
often
just
like
not
working
out
that
well
so,
and
that
means,
when
you
think
about
it,
you
want
to
extract
different
modules
or
components
of
your
monolith
into
separate
micro
Services.
C
You
need
to
find
a
way
that
the
monolith
and
some
parts
that
have
been
extracted
already
can
temporarily
coexist.
While
it
takes
you
to
migrate
the
whole
application
that
can
be
many
years
in
certain
cases,
so
this
coexistence
idea
is
very,
very
important
here
as
well
and-
and
so
here
we
have
a
very
simple
scenario:
it's
a
monolithic
application,
and
we
assume
that
this
is
somewhat
well
structured
in
the
sense
that
there
are
some
components,
some
modules,
that
in
that
monolith.
So
we
don't
assume
it's
it's
it's.
C
It's
very
bad
messy
big
ball
of
mod
architecture
here
and
then
the
idea
is
that
we
want
to
take
out
some
of
those
modules,
as
they
are
called
here
into
separate
red
services,
and
the
way
this
works
is
that
you
first
introduce
a
proxy
that
just
routes
all
the
read
and
write
related
traffic
still
to
the
monolith.
At
one
point,
then
you
start
to
implement
a
micro
service,
but
that
micro
service
needs
some
data.
C
So
let's
say
we
do
it
for
the
customer
module
here
and
then
you
would
set
up
a
division,
change
data
capture
pipeline
that
captures
everything
that
is
related
to
this
customer
module
and
propagates
it
further
over
Kafka
into
a
separate
database,
and
once
you
have
that
customer
related
data
there,
thanks
to
CDC,
you
can
start
to
implement
that
micro
service.
That
should
work
with
that
data,
and
once
that
is
done-
and
let's
say
we
also
Implement-
that
micro
service
in
multiple
steps
we
just
serve,
read,
requests
to
begin
with.
We
reconfigure
our
proxy.
C
C
C
This
is
engine
X
that
runs
on
localhost
now
and
that
proxy
will
be
used
to
do
that
redirection
of
traffic
between
monolith
and
microservice,
and
here
we
have
chosen
a
deliberately
over
version
of
the
springbed
clinic
as
our
monolithic
application
and
what
it
basically
does
is
it,
among
other
things,
it
you
know,
manages
owner
data
owners
of
different
pets,
and
this
is
backed
by
a
mySQL
database.
So
the
first
thing
is
we
want
to
do.
Is
we
want
to
have
that
owner's
View
and
build
that
in
a
separate
micro
service?
C
We
should
be
able
to
have
one
thing,
namely
the
data
is
in
Kafka,
but
it's
nowhere
else
yet.
So,
in
addition
to
that
think
about
we
mentioned,
usually
we
get
our
changes
from
tables
into
separate
topics,
and
now
we
have
an
owner's
topic,
but
also
a
pets
topic,
and
we
need
to
find
a
way
to
bring
those
together
so
that
we
can
serve
a
read
view
that
contains
owners
and
paths
related
data.
C
So
basically,
we
need
to
join
those
two
topics
together
and
for
that,
in
the
background,
we
have
another
quacos
application,
running
Kafka
streams
and
what
it
does
is.
It
listens
now
to
those
changes
and
will
join
that.
But
first,
let's
see
how
the
owner's
data
looks
in
Kafka.
So
we
have
seen
this
before.
This
is
one
record
one
owner
record
from
the
mySQL
database:
it's
an
initial
snapshot,
so
we
don't
have
a
before
State
here
and
the
pets
data
they
looks
similar.
C
So
we
have
here
another
topic
which
contains
the
different
Paths
of
these
owners.
Okay
and
then
we
have
like
I
said
we
need
to
bring
that
together
in
Flight,
basically,
and
for
that,
in
the
background,
we
have
a
Kafka
streams,
application
running
written
with
caucus
and
it
joins
these
together.
So
what
we
then
get
is
documents
such
as
this.
We
have
owner
related
data
here
with
all
the
owner
day,
owner
fields
from
the
database,
and
we
have
pets
which
is
an
array.
C
Of
course,
an
owner
can
have
multiple
pets
and
now
that
we
have
it
in
Kafka,
what
we
can
do
is
we
can
in
The
Next
Step
register
a
sync
connector,
and
in
our
case
we
say
our
microservice
written
in
Quakers.
That
should
take
that
data
and
serve
read.
Requests
is
running
mongodb,
so
we
configure
a
mongodb
sync
connector,
and
once
we
have
done
that
those
pre-joint
data
documents
make
it
over
to
the
micro
service.
C
So
now,
when
I
refresh
I
have
that
same
view
that
we
have
seen
earlier
in
the
spring
pet
clinic
monolith
in
my
micro
service
here,
together
with
the
pets
names.
Okay,
so
let's
quickly
check
one
thing:
if
we
want
to
edit
an
owner,
we
are
brought
back
again
by
the
proxy.
It's
a
write
request.
We
are
coming
back
to
the
spring
pet
clinic,
okay,
and
let's
make
it
change
here.
C
John
though,
and
write
that
data
we
have
chanto
written
into
the
mySQL
database
and
in
the
background
this
is
propagated
into
the
Kafka
topic.
The
Kafka
streams
chop
will
will
do
that,
a
join
for
us
and
bring
it
to
finally
to
the
to
the
microservice.
So,
let's
just
briefly
check
if
that
change
made
it
over
and
yes,
you
see
it
here.
The
change
made
it
over
into
the
microservice
and
since
we're
about
to
finish,
because
we
are
running
out
of
time.
C
C
We
would
just
reconfigure
the
proxy
once
more
and
set
up
a
CDC
pipeline
into
the
other
direction,
and
then
we
could
change
our
owner's
data
in
the
micro
service
and
we
could
propagate
them
back
into
the
monolith
again,
thanks
to
the
BCM,
it
will
capture
them
the
changes
with
the
mongodb
source
connected
to
Kafka,
and
then
we
have
a
jdbc
sync
connector
that
writes
those
back
into
the
monolith.
Maybe
one
last
thing
that
we
wanted
to
touch
upon.
If
we
have
time,
let
me
know
we
could
do
that.
C
D
B
A
D
C
A
A
Wow
wow,
so
you
can
basically
deploy
stream.
The
operator
you
have,
you
are
able
to
have
Kafka
and
division.
So
you
can
you
could
you
can
test
all
those
example?
The
outboard
outbox
pattern,
Strangler
partner,
a
pattern
you
have
the
in
the
repository.
You
have
some
of
the
examples
and
a
fox
today,
we've
seen
live
demos
and
Chris
big
shout
out
to
you
for
live
demos
early
in
the
morning.
That
is
super
cool.
Thank
you.
B
A
Good
but
and
Peter
your
demo
is
super.
Super
cool
I
mean
that
I,
like
it
also
the
bi-directional
pipeline
for
the
read
and
the
right.
This
is
a
really
cool
real
example
right,
so
I'm
looking
forward
to
see
this
also
in
the
example
directory,
because
it's
super
vertical
example
demo
to
show
so
folks.
Thank
you
all.
Thank
you
both
for
joining
us
today.
A
B
A
A
Please
don't
don't
miss
the
what's
the
news
on
openshift
today
at
openshift
TV,
you,
you
will
find
it
in
the
openshift
TV
official
schedule.
We
will
come
back
Fabio
next
week
with
an
episode
with
dynatrace
with
openshift.
We're
gonna
have
some
full-time
diameters,
so
see
you
next
Wednesday
same
time
and
please
stay
today
connected
to
openshift
TV
for
the
what's
new.
So
what's
new
in
the
next
version
of
openshift.
Thank
you
again.