Jakarta EE Tech Talks, 21 May 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The Trouble with Sharing Data | Jakarta Tech Talk

Description

Decomposing your system into multiple pieces means that you also need to decompose your data storage in similar manner. But it’s working with the whole information that makes the system effective for the user. Whether we decide to nest network calls, or use message brokers to get related pieces of information together, there are lots of dangers hiding in the real world of distributed systems. In this talk we will look at what the dangers are and how to be prepared for them. We will demonstrate one particularly lightweight approach with MicroProfile + JPA application.

///GET SOCIAL!

Subscribe: https://www.youtube.com/user/JakartaEE
Follow us on Twitter: https://twitter.com/JakartaEE
Like us on Facebook: https://www.facebook.com/JakartaEE/
Join us on LinkedIn: https://www.linkedin.com/groups/13597
Tag us: #JakartaEE
Want more great Eclipse Community webinars? Become a member of our Jakarta Tech Talks Meetup: https://www.meetup.com/jakartatechtalks

A

Hi everyone and thank you for joining this Jakarta Tech Talk. Today we have Patrick Judith. So it's going to be sharing. This presentation called the trouble with data sharing, so Patrick I'll. Let you go ahead and take it away.

B

Thank you well welcome everyone.

B

Today, I would like to speak about the trouble with data sharing. My daily job now is with diorama.

C

I work as a core.

D

Server engineer mostly concentrating on para micro and.

B

Cloud other integrations and yeah. You can reach me on pretty much any social network where I are.

C

The enthalpy due dates and.

E

Today, we'll be speaking about.

B

Not really a technical way was a technical issue, more or less an architectural issue, and that one is with this. This is.

E

The business for which you are building your IT.

D

System and that system is driven by some data which live and some data storage.

E

The usual constellation these days is, however, that this business information system is played across multiple domains and each of those domain get their own data store and when a request enters the system to one domain. Very often, it happens that that domain doesn't have enough data to fully answer this request and therefore it makes a request to another domain and.

D

I'll be referring to this.

C

D

C

Consumer producers of consumer being that the one who needs the data, the producer being the one who owns the data and.

D

Is responsible for for its consistency.

E

And even also, what happens is that even that domain might not have enough data.

D

To fully structured the response and then asks.

E

Another domain and as the evolution of the system, progresses and new domains are added and new features are added. We discover that there are lots of large or small data dependencies and that's the trouble with.

C

The data sharing I want to speak about today.

E

Undergone few technical approaches: how how this issue is solved and I, look especially on how to make these things consistent and what are the scaling challenges when that, and also what are the challenges when the implementation of the domain changes and this image in the background is no way complete for any system there will be new domains coming in and how to cope with that.

E

Should you have any questions during the talk? Please use.

D

The button ask a question or put something in the chat and I will regularly check it and respond it as as questions come.

E

Okay to illustrate the problem, I will be using.

B

E

Example so let's say, we've got kind of I do delivery service. So the request for streaming comes in from a user to.

C

Play a video, so he got by the overcut user and.

E

This is a phase platforms of the user about subscription and.

E

Yeah then subscription gives.

D

You access to some collections of ideas and.

C

Those videos are bundled and.

E

Collections and at this point we're asking: can this user play about here and be demonstrating those replication approaches and exactly this? This use.

D

Case, however, as a time progresses, there will come another use case that the user is.

E

Home at some region, and since this is paid flat.

C

Platform- let's assume yours will think the author's.

E

For the video and suddenly we need to collect how many times a.

D

Given by do has been playing and given three key region to calculate fees for parameters and then for distribution and.

E

This is kind of to think about that. The the use cases evolve over time and the data we get for the first request might not be enough to fulfill a future needs.

E

Okay, so what? What are the troublemakers here? What will cause are trouble.

D

When trying to.

C

Manage that data distinct data.

E

Stores first, one are failures, so either you're producing side or your audio consuming side will be unavailable and and kind of progress or producer main give you payload, which.

D

Consumer doesn't get understand and.

E

D

Fails to keep its data in sync or to interpret the response to get further.

B

E

Big troublemaker is when you try to modify the same entity in multiple threats or in multiple processes and we'll see.

D

Some examples how that might happen. What are the usual outcomes.

E

Since the data is not in the same store, there will be some some latency to get it and if.

D

There are multiple networking codes to get data that latency might get a bit unpredictable and.

E

Also, if we want, if one system tries to determine if one change happened before another change, that turns.

D

Out to be also pretty challenging in a distributed system.

E

The next troublemakers for data replication is the evolution of the system. We are adding features. We've got new leaf request for the data part of the system, get refactored removed edit and changed their internal representation, which my cows.

D

I'm incompatibilities on the on the receiving end.

B

E

Let's see which approaches we'll be discussing today, this is a highly subjective chart, based on my experiences before joining PI R I spent several years with system that was managing electric vehicle charging stations. We have lots of integrations. All those charging stations were not reconnected, so in the end we had coordinate.

D

Stage amongst thousands of nodes so.

E

I personally learned a lot about.

D

How things can go wrong when you shuffle data from from place to place over a network, so.

B

We've got three axes here on.

E

The x-axis there's effort that producer needs to make to be sure that the data he pushes are.

D

Consistent and consistently received by.

E

All the consumers! Why accesses consumer effort, which means how much effort it needs to consumer.

D

E

Exactly the same goal, that means that is sure that he got all the data that.

C

Producer had available and.

E

The third axis is represented by color and its infrastructure effort when additional additional brokers, databases or coordination amongst the teams or with.

C

When some infrastructure team is needed, that is.

D

Signaled by the color, so we'll start with the remote call. When the consumer synchronously call producer, let me go on to push replication while.

E

Producer pushes the data over to consumers synchronously.

C

As they happen, messaging.

D

Is kind of middle-ground solution will produce pushes the changes about the data into into a messaging broker with short-lived messages. Then we briefly mention that we can actually do that. The replication of the database level, like we've, been able for for decades.

F

Another solution is to put all those messages.

D

Into persistent storage for some kind of even.

C

Store and finally the example we'll be looking at is using pool replication or consumer regularly ask producer or whether there are any new data.

D

All right, no questions so far.

F

So, let's look closely on the remote call, so that's.

E

F

Solution, there's in space, a request comes in two content: delivery.

D

Where the user ID- and we ask account service, get based on information about this user and I will get back the response response.

C

E

See it usually implemented via HTTP, RPC or.

D

An exact times, REST API.

B

E

Cache usually has quite similar.

C

Characteristics in terms of also failure, modes and.

B

F

It's easy to implement quite straightforward. We will be seeing some remote calls.

C

Today as well, there's.

E

The positive of this solution- this is the only one that is surely consistent.

F

If you ask the producer for the data and.

E

You get back an answer and you understand it, it is surely the.

D

Correct state of the system at the time of the query.

E

But our enemy here.

B

Our failures, so the.

E

Request comes in and we want to ask the account service and it's not there. What do you do.

B

E

Profile gives us nice, nice, API, sand, forms of fault. Tolerance, ap is where you can define three tries where you can give timeouts.

C

And specify how much repeats do you want and what's the upper bound for.

E

A feeler, however, I think the problem here is that the decision of how long a user can wait until as Fido is authorized or whether a user can see of I do if we cannot reach the authorization service. It's not a development question.

D

E

Development, it's not.

D

E

Of infrastructure, it is really a business question, so if you get lots of these remote goals and I believe all of them need to be handled by the.

C

Business side and to decide what what what the broker pheromones are from the business.

D

Point of view that you really produce lots of questions towards your your.

E

Product owners- and that's where you follow me on the consumer side besides, there is lots of overhead of we documenting all of that and.

D

I'm gonna that.

C

E

So the negatives are that, since this is not self-sufficient.

D

We need to handle all those folks, trailers.

E

With each new query that is needed on the data, the producer site, probably.

D

Needs to update their API and at new end point so, for example, for this.

E

For this new summary of played, videos, they're probably need to be another end point.

C

Rather than just get user because with millions.

D

Of videos played per month, there will be quite lots of requests which we learned in Veda.

B

C

E

C

Fact that each piece of data I.

D

Needs to travel- that's worth at least twice in this location in this situation gives some bandwidth considerations.

E

Consumers there for me to handle those retries and fall backs and also take account. The latency of.

D

The remote ball. So if the data lives but remote system at both by definition, this forward, if it both reside in the same database, for example, or are in the same data storage and.

F

Also, if you want to count.

E

Latencies, for example, or some some isolate these get.

D

Really hard to compute once you do lots of downstream calls, because those are independent events.

E

So basically, your probabilities.

D

Or given time multiplied together, meaning.

E

F

E

F

D

Percentile of latency or 100 millisecond, your your 90th percentile latency, is 100 milliseconds, because you've got like 5% chance of hitting above 100 milliseconds in the first case and then disaggregate.

B

Okay on the side of producer, they need to add new and new and.

E

Points with every new query to the data and the most interesting aspect of it is that supposed content delivery is built to handle 1,000 streams per second. What it means that account service whose major responsibility is making sales to users.

D

It also handles 1,000 requests per second to cope with the load. That is there on the content, delivery side.

F

E

The logical next step to take away the scale is to.

D

Change the flow of.

E

The information so.

F

When the subscription changes on the account side, it will speak to the content.

D

E

D

Say again over synchronization TP request as far as those I think of that.

E

Order and when the request comes content, delivery.

F

Can over they respond by using its local data.

D

That has been replicated from from account service.

F

We again usually see that implemented with a rest.

C

Call or a bat hook or a.

D

Self call actually sub-goals are quite common. The other direction and.

E

It's still easy to implement. We solve the bandwidth problem.

D

And the network latency problem, because we are not speaking that much to the account service and therefore it scales better.

B

But there are lots of failures in this in this case.

F

C

I think I spent most of the time speaking about troubles on exactly this case, because this one just keeps giving.

B

E

First, one is what to do when one of the consumer systems.

D

Provided we are replicated to multiple ones if one of them is unavailable, what did you do? We.

E

Stopped the upstream request: you can notify the systems that accepted the change. Sorry, in the end, we managed to roll back and actually implementing distributed transactions within this replication. All we.

B

Do you for or we go for lower.

E

Consistency- and we say: okay, there is not so much data so once a day or once a week we provide a full load of the data to.

D

To each consumer, which.

E

Will work well unless.

D

There are changes during processing of that initial boat and then.

C

Things really get complicated.

B

When we employ refresh strategies.

E

We suddenly get into problem recognizing the ordering, and this gives us this brings us to trouble with time, which is what happens now so.

F

Let's assume: we've got first request from producer to consumer that changes the region of user one to Slovakia and it.

E

Happened at time stem one, but it failed. In the meantime, the region was change to.

B

Czech Republic timestamp.

D

2 and then the original request is retry.

E

So if this will be a really interactive session, I would ask which of following may hope for the timestamps. Yes, the first times time smaller than the second one, equal or greater one.

D

Or greater than the first.

E

One and actually yeah the answer is when enough luck, all of.

D

E

The usual usual way.

B

This happens so.

E

The time stamp this period, that producer was smaller.

D

Then it was on the second message, however,.

E

If this was kind of automated change or a mass change, we.

D

Hit the trouble that computers these days handle more than one transaction per millisecond and therefore the x times will.

E

Be equal and to get the third one, you will need a little bit more luck, because those Maggi caused by synchronizing the clocks, so the clocks and the computers are a lie because.

C

There are approximate and they need to be, they need to be synced.

D

And that sync can happen in both directions or full, where they're not in backward. So it might be that the clock just.

E

Synchronized in between those two messages and suddenly in the system time at.

D

This time point is smaller than it was at this time point or.

E

If you are not using UTC in your database, then your favorite night.

C

Is the weekend of the change from from summer time to enter time or suddenly think that happened one second after time step one has become, has local time step of one hour earlier.

E

And therefore, one.

D

Usually needs to employ some kind of sequence.

C

Numbers just to recognize, which.

D

In which order the operation happened.

E

But even sequence numbers can have their issues. So, let's assume again the same message.

F

Is sent from producer to consumer and.

E

That's an HTTP request, so here's an HTP threat that consumer side.

D

Trying to process the change and.

E

During that proves icing with that threat get suspended.

D

For whatever reason, garbage collection waiting for.

C

Database log waiting for some unrelated counter or something like that and.

E

As this threat is blocked, we get another request with the second change and then threat gets luckier and finishes even before the first threat, the contention all of the first threat is removed. Therefore, this one completes with no issues and then thread.

D

Wants resumed, processing and otherwise the result of the previous of the previous threat.

C

So, that's usually why you are employing your optimist, optimistic locks and JPA.

B

But still you have I'm sorry.

E

You have an issue here.

F

So now this thing fail and you really need to.

E

Recognize that it is a message that is already outdated and no parts.

D

Of the message is necessary to reconstruct the current state.

B

E

Like I showed here, the major problem with synchronously pushing the data from producer to consumer is that it is not consistent. Network failures will result in.

C

Lost messages which tries will result in duplicate messages. The.

E

Fact that it is that.

C

Those are multiple, HTTP requests may result in out of order messages.

F

So consumer needs to be able to cope with all.

E

Of these, remember which messages it seemed.

B

E

Concurrent processing on on the same entity.

D

You still can handle multiple updates.

C

To unrelated John related entities and also to be able to decide what happens if it recognizes that the message came out of order.

F

Producer, on the other hand,.

E

Needs to handle the fact that it's message did not reach the consumer and.

B

E

Who end up building basically a queue.

E

Then still when we've got new system or new domain join in the system and it means the data, it means another means of getting the data in.

F

So we still need some kind of initial load.

C

Even if just not for.

D

Compensating the fact that the data may become inconsistent, but every.

E

New system means that new data and in.

D

That case, as we are pushing out initial load.

C

We should postpone any other changes to the data to preserve the consistency of the.

D

F

Right any questions to these.

B

So the next step right.

F

As a side step, just shortly remind that, there is possibility, then, when subscription is changed, store to.

E

Database, your database on history that may set up magical.

B

E

B

C

D

In your own database and then you can handle them locally, so.

C

It can be done with regal and replicas or you can employ something like Judy's iam, to transfer your ride ahead logs from one database and.

D

Apply to another or good, find all triggers to do that.

F

B

F

Side and really is transactional.

D

Inconsistent because that's what databases are good at and it.

E

Gives you full query possibilities? On the other hand, your data.

C

Model becomes your API to other parts of the system, which might be a bit hard.

E

To to maintain.

B

In the long term and.

C

Still, these replications can fail in that debate, as that usually means that.

D

Your original price will fail because it cannot be completed fully or fail in some.

C

Other way- and it might be quite loud enough- infrastructure, work programs for DB admins, but for us traveling's, Java developers, and it really seems like lots of stuff to you to manage.

B

F

This solution is hard on the consumer side because there are the ones that need to adapt. If, if producer changes, data.

D

Model, its consumer that will start feeling, because the queries.

C

No longer return data or make me feel insecure because things changed.

B

Our producer needs.

F

To be really careful.

D

When changing their data model, there are solutions for that and approaches for that it could burden on the producer.

B

F

Is possible to use any kind of DTO to keep the internals and then well.

C

We are sharing database here, so the DTO will be will be view in this case.

E

Another common approach and the most common approach in.

D

Sharing data amongst systems subsystems the means whatever he call them its messaging. So we've got content delivery, subscribing to you some topic or a queue to.

C

A message broker and account well when, when subscription changes,.

F

The count system will.

D

Send the message to that topic or queue to that destination: content delivery receives it.

C

Processes, it updates its own internal model with that updated data. All.

F

The art parts of guaranteeing that all participants are available, then.

C

There are outstanding messages that need to be delivered. Many of the talents also the ordering of the messages its what's taking care of by the brokers, whether it's less.

D

Infrastructure code and then your application code yeah when the restreaming derives its again locally.

F

In Jakarta ie world we would most preferably use JMS.

C

For this task to even either up team mq or of the open, mq or we've got different messaging brokers with different standards, RabbitMQ and many cloud messaging services, those usually implement I'm QP 1.01.

F

One of the options is to is to go and use our.

C

Club connectors, which are generic jca connectors, so we can write massive student Dean's that will receive messages from from mine and.

C

Sqs or or from Kafka.

F

So, on the positive side, these are the couple scaleable. Usually consumer also can filter if.

C

There are too many events: it can filter those that the consumer is really interested for.

F

With with all the ford broker is doing, there still.

E

A chance of getting either on further messages or.

F

Duplicate messages, duplicate messages, usually when consumer failed to receive.

D

The message and then fails out.

B

Of order messages can be caused either by the broker, not.

F

Will be come on what is probably surprising message driven bean transmissions by are by default, give.

B

You pool or I think ten.

E

Message, human beings.

C

So you can process, a message is faster but as we just seen, that's.

D

Usually not what you want when you're replicating data that may be related.

F

Failure to process a message can get result in this kind of poison pill. One message is delivered endlessly.

E

Or is then dropped to dead letter Q, and it is very important in that case, to have operations.

F

In place to monitor.

C

And to then letter Q and understand that your data might cat are extinct.

F

In some cases, when your Q is limited, you can also block on humor.

C

Which cannot understand those messages in the fields and weekly.

D

Messages to be Qi.

C

Need to cope with that, so it is important that your Q remains remains working and all the time and that you can.

D

Process anything, that's strom that you that that will require force.

C

For the producer at first, let's speak about the consumer consumer.

F

Should still be able to identify a duplicated message when it already seen.

B

One, it should know so: I need to.

C

Implement this important receiver.

D

Pattern and limit consumption, concurrency.

E

That's the thing with the MVPs I mentioned.

C

If there are no other sources of parallelism in your code, producer needs to maintain transitional consistency, especially if it cannot sense. The message probably needs to fail the transaction so that this information.

D

Is not lost and still.

F

If neither man will join the system, it will need different altercation means, so we.

C

Still need some kind of full load replication to handle the new consumers.

B

E

Of the ways of solving this initial replication trouble is just.

B

To store the messages forever and that's what it's meant with this approach of even storage,.

F

So when the change subscription changes, it is stored in even storage and content delivery at any.

B

Time remembers the last message at scene from the.

E

C

Just reads: new new events as they are produced by the producer and then again.

D

Handles the situation locally.

B

We usually see this pattern where, with Kafka or but.

F

There's nothing preventing you from actually doing making the table in relational database just make sure.

C

It's configured properly and that it partitions on the timestamp or something because it's really hard to delete entries.

D

From from a table that growth will be.

C

B

And similar to messaging, it's a.

F

Couple it scales well and if prisoners history and therefore solves our initial load or it can serve for auditing.

D

Purposes as well.

F

Anything that have side- oh yeah, infinite storage is not free, so you might.

C

Need to consider how much data you will be storing and rolling and employing some ways to compact the data by making snapshots or removing some some outdated achievements.

F

And usually this retention retention rules are handled by the infrastructure, especially in the case of.

C

Kafka, so there is some some overhead in coordinating those changes and.

C

Again, these can be quite.

E

B

Systems so there might be some.

C

Operational overhead in running those multiple machines or paying for for appropriate service.

F

Similar to how what database your database, storage or your database schema becomes your API here, your events become your API. This usually causes trouble for even the.

E

Source system, where.

F

E

Need to be source of entire through to interstate state of the system, as those tend to evolve much.

B

More frequently.

E

B

E

F

Same time, taking care of all your consumers that.

E

F

Are still able to interpret your events.

E

Might be a challenging task and yeah, but there are multiple solutions to.

D

That storing multiple.

C

Multiple variants of the same event, for example in version 1 version, 2 version, 3 and so on.

F

But it's that's. What producer needs to do.

C

F

Guarantee that events that once written are.

B

F

B

F

Same thing and.

C

Consumer needs to be able to recognize multiple versions of events and interpreted properly.

F

Limiting consumption concurrency in this case.

C

It's probably an old towards Kafka, where the topics.

D

Are partitioned into into multiple partitions, so you.

B

E

Select proper business key, meaning that change events related to one entity, always.

D

And land and same partition.

B

All right, so we are getting towards the.

F

End, we are getting towards actually seeing some code and full replication as variation of the yen store.

E

And we basically.

B

Built the Egan's store inside.

C

The producer database- that's that's,.

D

The gist of the solution, so, when subscription changes, we've got this event.

E

F

Restore balancing that same transaction into into accounts data storage, content, delivery remembers the last event it's seen from the account and.

C

Will regularly the.

D

Account service for the new events and process them.

B

Doesn't seem like much of the change, but actually it makes things quite simpler.

B

One thing I didn't find any new library.

D

Do you do to do that, but, as you'll see by.

B

Combining few few standard features of Jakarta.

D

And micro performance, it's pretty easy to do.

F

It's quite simple, its transactional, a consistent, because even this only written the entire transaction succeeds, the sequencing is handled by the consumer and you will ask for the range of events. You know that those.

E

Events that you get over the wire are exactly in that seat.

D

In which they've been written or.

C

Sorted at least and.

F

The world handling of retention remains on on the producer side, although all of the evolution even transformation of casting down casting it all, remains on the consumer side. So.

D

It can guarantee that compatibility better than showing that data outside of its control.

F

We still have a problem to solve what to do with the storage or large amount of events that need to be compacted and compared to other solutions. This probably have quite high light latency, because you.

C

Will not be asking the account service for unions every millisecond, because we are back to the beginning and having 1,000 requests per second. So.

F

It will take some time for the data to appear.

C

On the consumer side.

B

F

Responsibilities here is really to handle all the data.

B

And make it compact and being able to scale when there is.

F

Lots of events or lots of consumers.

C

This is where, in some cases again, the replication of the events might kind of be major. 'ti of the type of producer will spent.

D

With user request.

B

F

Need to be again aware of the unknown payloads when the producer breaks their promise and produces an MMO at event.

B

This is again something that needs to be monitored in the same way like it is.

C

Monitored or it will be monitored as as a dead letter Q and a messaging system, because.

F

When you've got the message you cannot get through, you.

C

Get stuck on on that point and in your data storage.

E

And then there's pretty easy task of handling.

C

Cursor storage.

B

All right: let's see that, like the code for.

F

The data is available over to.

B

Github and I would post that somewhere.

B

Lo me he'll.

D

B

D

F

So let's see a demo here.

B

I've implemented.

B

Similar solutions, what we, what I discussed in the presentation.

F

So in this column, we've got the account system.

D

B

Has a list of users.

B

B

And nothing is happening all right. huh Okay, something is happening, but.

F

Probably the streaming is playing a bit with my grocer here and either refresh in the meantime, so.

F

Okay, actually, we arrived at something I.

F

Created a user accounting system and.

F

We can handle the subscriptions that have, but it doesn't know about.

D

Any subscription shows so, let's create one.

B

And let's add some video to it.

B

So at this point.

F

We will make this query: is this user Patrick able to work with this video trouble with data sharing? The response is forbidden and over here we see the last event. On the producer side, the last.

D

Even read on the consumer side and last time of sync, the poll is now set to 15 seconds. So every 15 seconds it will.

D

F

Content of every will.

D

C

Account service.

D

For the new data so we'll now add the subscription to you to use at Patrick, and we see that even.

B

D

B

Don't ask me why we are seeing that.

F

Four times, but in the meantime another synchronization happened.

B

And suddenly, since this user has access to.

F

Subscription 1 and subscription 1 contains this.

B

Video, probably data-sharing everything is.

D

Good and the Vario can be accessed or could be accessed.

B

If there could be anybody, ok, how did we do that.

F

So the front end is written and view and we will not.

D

Be will not be discussing that at all.

B

If you look at the code, I will write the.

C

Path to to the repository at the end of the talk.

F

So we've got consumer application here. You've got this demo, which hosts the front end and we've got producer ad, which is at this point, there's account management. So, let's start in the account management and see how we are creating.

B

F

Or just use plain JPA code, we create a new user entity, set its attributes, the name and the ID. We.

C

Persisted and then we fire a CDI event user won't render in this nice user created event we're.

B

C

What's your ID, there is.

B

C

Then, like whoa lot of hierarchy for the user event, so we're done user created.

D

User deleted subscription added and removes of those operations that you can see on the screen.

F

So we fired that event at.

B

Scene: yeah, because really don't want this to be bound really.

B

Strongly to the act of creating the user, we might have.

D

Other subsystems, within accounting system, to react on the fact that a new user is created or signed up or whatever we've got one.

F

Observer on that event, which will take that event and store.

B

It in a database in a form, really trivial. We store.

C

D

Kind of of the generated sequence, we store the event type and the JSON payload of the of the event.

B

So and we store that.

F

We require that this happens in the transaction, of course, because we need the transaction to.

D

Store the data and.

B

That's what I would like to note one pitfall when you do serving and c-d-I or this during and in.

F

Progress as the default value, which means do it now as the transaction is running, I thought it will be grayed out here to do it before completion. So I know that the database would commit, however, that one of time is too late to do any changes to the database that I've been doing so he.

C

Fold is quite alright.

F

Now, as we created those those.

B

Events, they remain and the database and they.

F

Are available with this simple replication, endpoint sitting on slash replication, so you basically get to the replication and pass the last ID that you've seen as a consumer and how many events you want to get.

B

And all just yeah, you do trivial name, query and replication event handles it's serialization back so.

C

This payload is a JSON string, so we need.

D

Reinforces that create.

C

Jason even and make it chase him again.

B

C

We do in this form that you've seen here, yeah user created that was okay. This is the ID. This is the time- and this is the payload, so user ID and t ID and subscription added.

D

And a subscription just let just doing the subscription.

C

Something moving here, I.

C

Really want your: how I made those three for subscription.

D

The request on the front end.

B

Yeah, since we remove is good to go back to forget.

D

B

So back to code.

F

What's happening on the side of the content, of the very how it gets is event.

C

Or yeah how it gets the answer. The answer is simple, because it's again a name query where we are joining from user to subscriptions.

D

To the contents, where we are looking for the content, whether it belongs to subscription that user has.

F

However, how do we get the data this.

B

Is handled by simple.

F

Aj being with the schedule, every 15 seconds.

D

Paul for new data and the.

C

B

Has this single cursor class, which is really minimal entity for.

F

Just three fields, even not public, then doesn't have any.

D

Any methods it's only used for disrupt occasion, so I didn't include like lots of ceremony about around.

C

F

Guarantee that the cursor for the user is replication.

D

Exists and then we go and fetch it. It's also quite straightforward. We update the last time to fetch something and.

B

F

Account service and for that we are using a micro profile, REST API, RESP.

B

F

So this is description of that of the HTTP endpoint. We do get with query parameters ID in size. If you.

C

Compare this because I clean the interface, the.

B

Fetch method has so get not that way by the in size.

F

And the runtime will generate rest plans for us and handle.

D

Transforming from this goal to actual usage of chucks eros plan and getting the data back.

F

The replication, even we are getting here, it's.

B

A bit different than.

F

D

On the account side, because that's natural different domain as different these different representations.

F

So, on the income side, we had actual hierarchy of objects on the receiving side. We've got an e now that basically corresponds to those various types that exist there and we've got generic payload because we happen to know that we only care about user.

D

Ids and subscription and we don't need to build any.

B

Any complicated object hierarchy for.

C

That but that can be done in every use case, so.

F

Once we fetch the data, which has note how many events we see in last time and.

B

B

E

Animal the event that this kind of strategy on.

D

How to interpret that payload? So if it was the user created, we know we should create.

C

F

C

Remote, we know we should remove the user and so on.

D

So this is how we do that they distinct renunciation, so it needs to be explicit gold for every change about, depending on how you define you have your own, even in protocols. Ok,.

F

Miroslav ask the story even method start even to get cut or bigger, with very long retention period and clean Mozgov, Kaka and.

B

Ireland those approaches almost the same. Yes, they are, they are almost the same. The difference is that.

F

Once you send something to get the client, you cannot change, it varies here. You can evolve your internal events or you can provide end points for.

E

Down casting the events to previous versions.

F

Even if you already store the new ones, so the only difference is that the owner of the data is.

E

In control of access to the data, I didn't put it.

D

Away to to your messaging broker or to Kafka yeah, more control of history, more control of access.

F

B

B

F

E

Our application events and with those methods, the data is.

C

There isn't that a bit of content delivery pack, this.

B

C

B

B

All right, let me just quickly point you to the code.

D

This is the URL you're interested.

F

Okay, infrastructure wise: this has been in the running on kubernetes cluster imager.

B

D

After actually deployed as separate applications.

B

And yeah, if you want to put some that data in its replication demo, that's player and adapt.

F

B

F

B

Back to the talk.

F

I think code wise. There is.

B

Not much to tell.

F

The code should compile itself without problems runs, but latest para micro.

B

Yeah, should you have any questions? Can contact me afterwards or via any.

C

Social network.

B

F

This is pretty basic, so still for production level, there will be like a little bit more stuff. You would like.

B

To do, which is.

F

Things concerning Kevin, even the evolution.

D

So if you want to change the representation of your event,.

C

You might be rebuilding.

F

C

B

F

B

F

That could still deliver them in the.

C

Older version, for example, you could.

B

You might have more control on how things are serialized or how.

D

To store those events in the compact form.

F

It doesn't necessarily need to be changed events. Those can also be snapshots, but if you.

B

Order, those snapshots it will be.

C

It kind serve as a really effective way of transferring the changes so for each.

D

Entity that will be there will be one snapshot you can get, but still that last scene.

E

Evan ID could serve as a point in time concerning which changes you've.

D

Seen already in which you yet not seen.

F

And also polling improvements can be improved, so they can be additional. That's like a channel from which the.

B

F

Can inform the consumers there now it's a good time to find to look for some new events.

F

Alright, so in conclusion think very carefully whatever of those approaches or combinations of those approaches, you are using what can happen so you would lose the consistency of your data because yeah those things always happen on Friday afternoon, if possible, keep your replication called past the single-threaded, not only within your process, within your entire domain.

F

Or very carefully charted according to the business P, so that you are not changing the.

D

Same entity into threads.

F

And I think this approach of using pull replication is really underrated, as it is simple.

D

And one of the most consistent ways to to get data between.

B

Between two systems, so thank you, I hope.

E

That was interested interesting.

F

D

You've got any more questions, please ask now and.

F

Try interest entire I go and download our software and, if.

E

You're interested in our other activities with with.

B

Open source community, please have the.

C

Join fire arethe, which might help you growing.

A

Thanks awesome, thank you so much Patrick and thank you to everybody for joining us. This is going to be posted on YouTube tomorrow, so keep an eye out for that on socials and we'll see you at the next Tech Talk take care. Thank you.

B