Delta Lake Delta Lake Tech Talks, 18 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Tech Talk Series Part One: Engagement Activity Delta Lake

Description

In part one, we’ll talk about how they built the engagement activity Delta Lake to support Einstein Analytics for creating powerful reports and dashboards and Sales Cloud Einstein for training machine learning models.

Abstract:
At Salesforce, their customers are using High Velocity Sales (https://www.salesforce.com/products/sales-cloud/tools/high-velocity-sales/) to intelligently convert leads and create new opportunities. To support it, we built the engagement activity platform to automatically capture and store user engagement activities using delta lake, which is one of the key components supporting Einstein Analytics (https://www.salesforce.com/products/einstein-analytics/features/) for creating powerful reports and dashboards and Sales Cloud Einstein (https://www.salesforce.com/products/sales-cloud/features/sales-cloud-einstein/) for training machine learning models.

We will include: 1. Ingest the data. 2. Incremental Read. 3. Support exact once write across tables. 4. Handle mutation with cascading changes. 5. Normalize tables in data lake.

-----------------
Speakers
-----------------

Zhidong Ke, Software Engineer PMTS, Salesforce
Zhidong is passionate in designing distributed systems, real-time/batch data processing and building applications.

Yifeng Liu, Software Engineer LMTS, Salesforce
Yifeng is a software engineer who has extensive experience in big data processing and distributed system, and interested in high volume, high complexity, low latency data pipeline and framework building.

Aaron Zhang
Title: Software Engineering PMTS, Salesforce
Aaron is an experienced software engineering leader with interests and areas of focus in engineering secure, fault-tolerant, high volume systems built on micro services.

Heng Zhang, Software Engineering PMTS, Salesforce
Heng is a software engineer who is interested and specialized in micro services, distributed systems and big data. Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner

A

Pass it over to craig.

B

Awesome thanks karen, so my name is craig: I'm a solutions architect at databricks and I've had the pleasure of working with the activity platform team at salesforce for the past year as they built out their engagement delta lake or engagement activity delta lake. uh So I'd like to give them before briefly, so that they can each introduce themselves uh aaron. Do you want to go first.

C

Sure yeah, I'm erin, I'm a principal engineer on the activity platform team. I've been with uh salesforce for slightly over six years and uh yeah, I'm gonna be uh the panelists. So I'll answer most of your questions.

B

D

Hi, my name is ham and I work in salesforce for four years and I'm in activity platform team for one and a half years, and I'm interested in microservice dispute system and big data and I'm also one of the panelists. um In the talk series. Yeah.

B

Awesome thanks leo next.

E

Hey guys, I'm leo I've been working with salesforce for about two years. Almost three years I would say, and my passion is about working with a big data at huge scale, with like high complexity, data and high velocities as well, and so I will be the parents to answer questions without today. Yeah.

B

Awesome and uh so here we go.

F

Hi, I'm zidong, I'm I've been working in salesforce for almost for more than four years. I'm a principal engineer here and passionate on building big data pipeline, distribute system and micro service.

B

Awesome thanks so one note before we get started so denny one of our developer advocates uh chris, who is another one of our solutions, architects and I will be monitoring the q, a on zoom, linkedin and youtube. So please ask any adult questions that you have all right. So now please go ahead.

F

Cool, so can you see my screen.

B

F

Cannot how about now.

G

Perfect awesome: this looks great.

F

Cool hi, uh so today we are going to talk about our journey of the building engagement activity, data lake in salesforce to support einstein analytics here is the agenda we are going through. So first we will cover what is engagement, activity data lake in salesforce and why we are building it and then we will go through the high-level architecture and the system design we have and then how we support our customers requirement for incremental, read and how our jobs achieve exactly what's right across tables wing, multiple rights.

F

After this I will talk about the challenge of using the handling mutations in deota lake, including especially for gdpr data policy and the user mutation requires and the optician optimization we have for the cascading mutations and using graph detection and the normalized table in the delta lake. So last we will also have a brief, introduce the performance challenge we have and the solution.

F

We will also keep some time at the end of this section for the q and a's yeah. Let's get started so at salesforce. Our customers are using high velocity sales, also known as hvs to intelligently. Convert leads and create new opportunities. We build engagement, activity platform to automatically capture and store user engagement activities.

F

The engagement data lake is one of the key components that supporting einstein, analystic, creating powerful reports and the dashboards and sales einstein for changing for machine learning model training, and this amount of data can only be scaled uh using engagement activity delta lake, which built on top of the delta lake.

F

So here is uh some key use case of engagement, delta lake, so say sales person um so like a salesperson can like use his engagement, metrics and the rates to uh identify which cadence or template is more effective and, for example, in engagement. Dashboard we easily can get like how many emails sent opened or delivered easily found.

F

This dashboard on the left side and also sales person can also get like how many phone calls um like a meaningful or connect on, like just customers, say not in chest or just left voicemail, so for any date, any sales rep. So we leverage engagement, dashboard to drive intelligence into sales, productivity.

F

uh With all these use case from our product manager, we convert lan into our technical requirements. We break down. um First, our downstream consumer needs incremental reach right, as well as the batch rate from the delta lake, our customers, poor data, one customer pulled data from our delta lake periodically and could generate the snapshots for layer downstream.

F

So one of the like a special case use case for our data lake is our data. Lake is mutable. Unlike the activity data lake we built before everything is immutable so which bring in a lot challenging for us. We will talk about this later and also the most key point for users. Trust is our data lake need to respect to the gdpr and the data policy that the users have.

F

So, after a long discussion and with pm with our engineers with our architects, we come up with a high level architecture. So we basically break down this uh pipeline into two components.

F

uh One is uh like uh the injection on the left side and there is a mutation pipeline mutation related pipeline on the right side, like a user mutation, gdpr or data policy related for deletions.

F

So we capture we capture the user engagement data and the push lane into our internal kafka queue which is heavily used in salesforce, for the data transform transfer and our spark job will read from the kafka queue or in the injection pipeline and start injecting the data and the batch lane in chile, delta data table recorded data table at least point, and to start this, the data table is uh just partitioned by organization id and the date of the engagement.

F

So on the right side is the mutation. So we build another very similar pipeline as injection pipeline, so to support the mutation requires from the customer and once we get the request, we apply lisa deletion or updates to the data table.

F

So the first question: when we design the spark job is: which mode should we use in spark job? Should we use a structure streaming mode? Should we use a batch mode so right now our job is mostly for batch processing, but to keep it flexible and extendable. In future we choose the structure streaming with the micro batch mode.

F

There are few benefits we have one. Is it supposed auto scaling very well with data breaks cluster, which means the cost saving and second, as most of our jobs, are consuming from kafka? We need offset for and metadata information for checkpointing.

F

We also find our job sometimes finished earlier. I mean in our poc stage so to from the course the saving perspective we also uh use we choose to use the trigger once to that means your job.

F

Once your job finished, your class will be shut down and for a period of time- and here your next schedule was up and your job will be ramped up again and is killed the next batch to save some calls.

F

So the delta api is a quite powerful. It comes with the native support for batch reach, such as time range quarry from t1 to t2. Give me all the data or like a organization, label, check organization level. uh Query like returning all the data for this organization.

F

Yeah, it's just a support list. It also supports incremental reads, as the name entails delta. So return me all the data, all the delta data since the last checkpoint, but it has a condition that it only um it only supports for append only mode uh data. But, as I just mentioned, uh our data table is mutable which doesn't qualify for this condition. So we have to come up with our own design for the incremental reading.

F

So we create, so how can we support this incremental? Read? We create a separate table called a notification table which is also partitioned by organization id and the injection time stamp.

F

So when new data come in new engagement data insert into our data table or when existing engagement data is updated or deleted, we will notify our downstream consumer of all leases or changes, and this is a new data and by inserting a new ng into our engagement metadata into our notification table, the downstream consumer can also either you use a streaming mode, listen to our notification table or you.

F

You can use the batch mode, periodically query over notification table to get the what is new once you get, what is new base of for all the data you just get the key, then you use the key to fetch from our data table for a real delta. You may have so, let's kind of solve our problem for incremental reach, and we found the list design pattern feed into a lot of our jobs, because we have a lot of jobs that require similar requirements.

F

So we we create a generic notification table here. Is the class let off for our notification table, as you can see it has organization id engagement date is when the engagement happens, the table name which which table has updates, deletes on uh new data. The modified time is when the time when our job receives the data- and we also keep some uh counters like update, count, insert count, delete count just for the auditing perspective and as well as the metadata.

F

Now you don't have to query the data table. If you just want some, like account for like how many data was deleted or how many data was generated, how many data was updated easily from this table very efficient way? Also, we keep the app name, because this table is kind of shared across multiple jobs, so lisa we need to keep what uh job. What is the app name? Let's write into this notification.

F

So, as you know, like we use delta lake, and then we install the data in data lake is a like a high level database that builds can build on top of some data storage. So we choose use uh s3 as our underneath data file system.

F

So it's all about like uh we're, curious, just curious like how these files hierarchy is uh underneath behind the scene of the table, because the phone table you just get to know all this query, and this is the uh the data you return.

F

So we took a look at of our data table in the notification table and we um here is an example of our data data file hierarchy. As you can see, the prefix is defined by us and it has a partition. Key like organization here is organization, input, 2 and engagement date.

F

It's a whole partition key and underneath uh implementation for the file is parquet and each parquet file may contain many many files, many many records, and then they would also give a magic called the delta law, which stores all the commits, as well as the metadata and your job status in this uh underlisted delta log folder, so which support asks for asic transactions with one write and, as you can see, there's uh be a lot of many many a lot of json files under this folder.

F

So one key point highlight here is you see is uh two days here. One is a modified time.

F

One is engagement date, so for notification table we use modified time, so our downstream consumer can use that time to checkpoint say you want like last time when from any time point you want the data, the delta data from that point, and because uh the use case is more, uh the use case for data table is more like customer wants the engagement when the data, when the engagement happened in the data table, so we use engagement date for the partition, key cool, so.

F

So here is uh um the common question common problems. uh We, when we designed a distributed system, what, if your job failed right so but always uh need to consider. We always need to consider like how we improve our fall, tolerance for the distribute system and then spark does doesn't provide atomic apis and or acid translations natively, but fortunately the delta api supports acid translation. As I just mentioned, it use the delta law to make this magic, so it is uh like a like a delta lake.

F

It's as just mentioned it's like a ordered record of every transaction that has been performed on the delta lake since its uh inception and the delta break this operation into atomic actions, and it will only commit all transactions all actions uh when all actions are successfully complete.

F

So when our right failed, the job will just need to retry. Let's not worry about any uh data data loss, but is it enough for our use case, probably not enough? Well, because while we are introducing the notification table solving our incremental read problem, it also creates another challenger for us. We found, like our jobs, tries to process like uh um writes into two tables, so delta api, suppose the essay transaction within each table so there's no batch transaction uh support across multiple tables, because now we have writes two rights to two tables.

F

So let's go through this uh example see what is the issue we are facing so, for example, in this case let's say our spark job first writes to data table then write to notification table. So if we write successfully write into a data table but when, while writing to notification, table the downstream uh the fail, I mean failed to write to notification table the downstream consumer will completely miss the badge seems it doesn't get a notification, it may get catch up if they quarry directly to the data table, but they will miss the real-time notification.

F

So what about? Let's say the other way: the spark job first to write. Your notification table then write it to data table leach. So if the job successfully write to notification table but fails right to data table, what will happen? That means our downstream will get the notification and acquire the data data table. It couldn't find anything and since they write to data table failed, they may we mainly lost the data forever.

F

So we need a design, a new design to provide. We call it the exact ones right across multiple tables, which requires like any right to table operation in a transaction. If we failed, we will not commit if there is any failure or the right. Our job will only reprocess it and we definitely need to uh so to support this.

F

We definitely need to save the metadata very similar as our data log and includes the state of each right and include the last six check point so and since, as you can see, uh our source is mostly from kafka and we need to store the chat pointing for the kafka.

F

So how are we? How do we resolve this problem?

F

We first create a checkpoint store because we use a kafka checkpoint store to store the star and offset and the cut some metadata of the kafka like broker list and some topics names and also the last job state for given checkpoint, which is a very key point when we need to resume from the last checkpoint.

F

So the first type of our spark job is to read: restore read from the checkpoint, install and get the last checkpoint metadata and the last job state.

F

So since our job is killed in a micro batch mode, each batch will include multiple process like process. Like the write data to table, one write it to table two three and so on.

F

We also need to store the batch stage like uh what is the job? uh Is the future failed or succeeded for this job at this process, or we call it right and the process now? So we each job, we name it each process. We will have a name in timestamp last modified time. So after reading from the chat point, store, spark will get the last job state.

F

If the last batch was completely successfully success for execute, the job will increase the last batch id. Otherwise it will use keep the last batch id. Then our spark job will fetch and our spot job will fetch the metadata based on the job name.

F

It seems complicated, but let's do a quick view of our metadata and we will dig into this with one example, so these are like a example of our batch metadata store table. So, as you can see, we have a job name in this case. It's injection job and the batch id which provided by spark api. In this case, we have a nine and a ten for this uh two process and the process name. One is an injection job. One is a job process process for right to notification table and the last to modify the timestamp.

F

These are when your job is queued.

F

So let's go through uh uh one like flow for happy boss. So first our spa job will get. The current batch id uh in this case is like a 10, 10 phone I mean formless spark checkpoint stall. Then you also fetch our batch metadata store to get the batch id of the last process. In our case, it's a temtem. Let's say these are uh matched all these are like. A spark may provide you a new badge right, so these are matched.

F

That means uh your job. Your last run. Job was like, uh like a complete 100 success for both process, so you will just uh so that means we can resume our next to our next batch. So you read from the structure point store to get the net uh the latest uh offset uh from on the kafka and uh process dispatch as a new badge. It's a happy pass. So once you process complete depends on your job status. You save the batch metadata back to the structure, point install and the checkpointing tool methods are stored.

F

So what about the unhappy parts like what? If there is any failure in between right? So again, it's a very similar that the spark will get this first fetch from the point install and the batch metadata store.

F

So now, in this case you see the injection job has batch id 10 and the injection data notification has a batch id9. That means that data notification rights may failed in last round or it is failed, so it didn't consume. It didn't resume to the next new batch. Instead, it will read the last batch data from the kafka and since the data injection job completely succeeds, it will skip the data injection process and just directly jump into the data notification process.

F

So right into data notification and after job complete, it will do a similar thing as the happy pass. Save the batch metadata and the check pointing back to the splash point install and the batch metadata store.

F

So what do we have achieved with this design? We are able to perform like exact ones right and if there's any failure, we would like to just retry and only retry the specific process, no duplicates right and no data lost.

F

Yeah, so here here we have pretty much covered our injection pipeline part, but, as I mentioned earlier, our engagement data is mutable. As we know, mutation operation is a way more expensive operation than injection in batch jobs. So our data, also our data, has to respect you. The data policy gdpr like ttl organization, delete user delete or data subject delete and the mutation requires is all based on id field, which may or may not be the primary key of our data.

F

For example, you change unless it requires coming in, say: oh change, all records from with id1 to id2 and the list rotation could also be cascaded. Like uh each user say. Oh, I want changelist id, I found user perspective.

F

He changed the cascading, for example, a cascade attendance id one and changing kindness again changing again, so it will be a cascade mutation to our data data end like uh it'll, be translating to like it from request uh id1 to id2, then id2 to id3 id3 to id4 it just it is a corner case, but we have to consider these in the coverless cases.

F

So, as our design is all based on distributed system, so injection and the mutation requires may just come and synchronizely. So there are two scenarios um when we applying updates or deletes into our data table the happy pairs.

F

Everyone is happy the data, the data engaging the data itself comes uh same time before the mutation requires coming in, but it will be an unhappy path that data is still engaging. Data is still in flight, so it's not yet to reach the data table, but the required mutation requires coming. What are we going to do? There's no data here. Are we going to wait or are we going to drop the mutation request, so these are like one challenging for us, so the solution is to add another table.

F

We called it the mutation mutation retry table, as you can see here so every time when our mutation job wake up periodically it not only read from the mutation requires table, but also read from the mutation retry table now.

F

First, instead of a directory applied mutation updates or deletes directly to the data table first, it will check whether my data exists. That means data could be in-flight or could already are landed in your data table. So the two cases one if the data exists based on a key check like a id1 or id2, exists. Okay, I will apply the mutation and if the key doesn't exist, that means your data could still in your chain.

F

So you you just save this request into retry table for for the next row, so uh let's kind of solve our problem for the unsynchronized problem issue, but this list, uh but as we just mentioned mutation could also be cascaded and we have to maintain kind of this kind of uh ordering of a mutation request.

F

As you know like when we apply the batch, there isn't a guarantee for the ordering. So we have to come up with something and the one solution is okay. I will just break these down into multiple uh operations. I will just execute them one by one, but as you can, as you all know, each operation in spark jobs. It will produce a new fire, or at least it will increase the number of ios that will increase, reduce your performance, and we also run to this performance issue, and we will talk about that later.

F

So the goal is uh to reduce mutation operations as little as possible. For example, uh we get a required three requests. One is for id one change. The workflow id one to id2. Second, is change id2 to id3, then change id3 to id4 and in our data table we have a one record with a workflow id one.

F

So the eventual goal is just like changelist id one to id4. How are we going to achieve that? We can do these one by one or we can have some optimizations yeah. So we come up with a um algorithm.

F

We use a graph to detect the cascading changes so first in order to execute these again at least first, we were just a group plan by organization, because our scope is always by organization, and we sort these by mutation execution time.

F

Then, for each group we use a spark graph api to detect the graph. We will go through this diagram on the next slide and then we found the node and ages so h is required and we find all kind of a connected components.

F

Then we see those ruler can all connected components and find the final state of each node. Then, after this we kind of drop. All intermediate stage of learning cascading requires then convert the connect component back to a list of requests. That spark can understand and apply. These in a batch here is one example that to iterate this. So first we group by organization id. So in this case there are two organizations.

F

One have three requests id one, two, three change it to two three four and one requires for id one to id two and we use lisa to build a graph. I also can call it a sub graph for each organization, and in this case we see lisa id 1 id 3 id 3 to id4 and so on and for organization 2. The start graph just have id1 to id2.

F

So after convert after the computation, we figure out the ionization one just need to change id 1 to id m and oid 2 to idm and all id3 to oid m and for organization two. We found that our id1 just need to change to id2. So this is the list of requests after the computation and can be understand by our spark api and that can be applied in a batch to the data table.

F

So, but this kind of solved all kind of optimize our badge right and reduce a huge amount of files, um because we reduce the amount of the request and mutation requires intimidating the mutation required.

F

But still, as we mentioned, mutation requires volume is very, very high, which in fact is almost like a 20 to 30 percent of injection volume, which impact our performance a lot. So we took a close uh look into our data shape and we found there's a lot of records need to mutate, for only one request.

F

For example, uh request is like a change. The name and key table uh like workflow name, one to workflow name to you can touch like thousands or millions of records in our data table so also bring in the sra concern.

F

So we have lisa solution called normalize. Just like our relation database. We also can normalize this uh table in our data lake by split splitting or extract some fields out of the data table. In this case, we split out the s id tid.

F

We have sid tid snam and the t name we break out into sid and a tid in data table, and we will just keep id mapping in the mapping. We call it mapping table and the user will. When you use a query hour say we want to mutate from the id sid 1 to sid 2. We only need to touch one record in the data table uh in a data table and if we want to change the name like change the name, s name to s name one.

F

So you only need to touch the mapping table and you don't have to touch uh millions of records in the data table which saves a lot in our performance. Packs perspective.

F

Cool so finally, we we have our first version of pipeline running in staging end to end with injection job mutation, job with a high fall tolerance and the downstream consumers can either read in a batch mode or incremental incremental mode, and most of the corner cases has been covered.

F

So it's time to run some stress test and due to time limit, I will just briefly discuss that.

F

So, basically, the common problem, we, when we're dealing with a smart job, is the injection and the mutation produce a lot of small files for small orgs and the mutation job isn't fast for us to meet our sra from the product perspective and also the mutation job create a lot of tombstone files, which is also costly and performance concern.

F

Our quick view of the solution is first, we can tune the partition and from org and date, just in into just only and leverage the data skipping and the ordering feature and set up another job to run the delta optimize and enable auto optimize for injection and mutation jobs to clean up those tombstone files. We also scheduled a vacuum job as a nightly job which will clean up roads to save our money.

F

Last but not least, we can also tune the spark, the default of paralysis to have a better read and the right pair reasons. We will have a deep dive uh in our next section for the boost delta lake performance with state housekeeping and the ordering by hand. We are very welcome to come back. Thank you.

G

Thank you very much. Very much is done. This is really cool, uh so karen yeah, let's switch back to speaker or gallery mode. We've got a lot of really great questions from everybody here, we'll try our best to answer them, but we only have about I'd, say five to eight minutes left, so I just want to be very clear to call that out. So if we don't answer your question, uh we'll do our best to actually go ahead and follow up with a do a follow-up blog actually to help with that as well.

G

Anyways, so saying this, uh I want to ask the first set of questions and then between craig and myself, uh we'll toss back and forth questions. So the first question I'll ask and this one I'm going to direct to aaron- um is the downstream data pulled from the engagement activity delta lake in real time. So that's to you, aaron.

C

Yeah, so uh the short answer is no: it's not real time. um So the as sudan had mentioned the delta. The data lake that is built on delta is actually updated once every hour and the downstream uh consumer requirement, which is basically the the einstein analytics dashboard that you see, um it's actually not meant to be a real-time analytics, reporting kind of thing. It's supposed to be like an educated view that you know sales people go in and check.

C

I don't know once or twice a day, so we actually update it uh once every four hours. So the downstream uh consumers pull the data only once every four hours.

B

Awesome thanks erin um cool, so uh I think for our next question: uh we're gonna direct this toward hang. um So the question is: why create the table for notifications instead of consuming from kafka directly.

D

The notification table is to track all the petitions that has been changed since our last successful batch in terms of the org id the date and the operation applied like update, delete and insert so with the with those information our downstream can implement the incremental read instead of read the whole conditions and but only read those petitions that has been.

G

Changed. Thank you very much. uh Okay, I'm gonna go for. The next question is uh this one is uh for you ethan. um What is the advantage of using kafka while spark streaming could be used to directly to create those uh those delta tables, and you did already answer the question in the q a, but I thought it would be still appropriate to let everybody who did not see the qna, especially those folks on linkedin and youtube that they also know this question. Hence the the call of the the re-asking of that question.

E

Sure so so we use capcom activity platform as our messaging system so, um and so it's used widely across the hope, um but it's just like infrastructure and then um it's also a position layer which makes the data replayable and I think, there's no kind of kind of conflict between the the kafka and the streaming job. uh Instead, the kafka is more like a repairable source file for our classroom jobs and then for those like capcom topics. There are other, like downstream use case. That's going to subscribe to that topic.

B

Awesome, okay, so now we have a uh more of a delta specific question, um so uh chris, if you would mind answering this, um is there some kind of absolute maximum for table retention uh with regard to time, travel.

H

That's a good question, um so what I would say is that time travel is more designed for short-term repair than it is for maybe like snapshotting. So I think you know, if you only wrote to your table like once a day, you can definitely have hundreds of versions in the table. You can have thousands I've seen.

H

People do time, travel um and go like a hundred thousand versions into the past, um but it's not equivalent to a snapshotting tool, and so I kind of look at it as like you know, if you compare to a database, um you have transactions and you can roll back individual transactions, but that is separate from the concept of a database snapshot right and the the idea of a snapshot is that you can go back to a very specific point in time for the database.

H

So time travel is not meant for that. Snapshotting use case is what I would say, but you can definitely use it to go back in time um for thousands or hundreds of thousands of versions, and it will work.

G

Yeah, I I definitely would love to add to cr uh chris's uh callout that, basically for send anything that's closer to snapshots. You probably want to do something like cloning, like the database, like you would normally with the database. Excuse me so you would clone your delta table. So that way, it's exactly what you asked for for that for that frozen in time, piece versus oops, I screwed something up or anything like that. That's exactly what time travel is perfect for all right. So uh don't actually I'm going to switch the question to you now.

G

Actually, because uh there's you, you had a really good a cool presentation, especially on this part, about exactly once right across the tables right and um you already presented this, but I think it deserves, especially from the questions we've seen like uh to answer the skin. Could you clarify? Why did you need to do this exactly once right across multiple tables?

G

F

Sure so, uh as I mentioned, um the job can fail. I mean one job failed, all like one job failed partially. So if one job has to write to multiple tables, uh if one of lisa process failed, a spark will just uh if you don't have at least a mechanism. Spa will reach five from scratch and either create a duplicate right or it will just miss the existing badge. So that's why we need to create the mechanism for the exact ones right either. One to save cost one is to make sure our data never lost.

G

Perfect, thank you very much, hey craig! Why don't you toss in the next question about the performance cost with hank.

B

Sure so um so, hang this question for you. So uh can you give us a sense of what your uh daily performance costs are to ensure that you never miss any data or you never miss any rights.

D

So from the design perspective, when we, when we apply the mutations, including a delete, update and insert, we do not directly apply those requests that are coming from our upstream kafka. We first stage those requests into our request tables. So then we have another job to read from the request tables in micro batches to apply those requests.

D

So if the mutation job that failed, then we are able to replay um from our request tables that is, for that is the design from the design perspective from the uh we also uh from the um performance perspective, the cost is the um the the the file layouts under the file layouts and the in the file systems, because uh the um the delta lag and the hood use actually uh use the parquet to um store all those files and we use s3.

D

So if you have too many small files, then it will definitely has a negative impact in terms of read and write. So we will have a detailed discussions in our second talk on to how to solve those problems using delta lake and um yeah.

G

Perfect, this is an excellent segue, uh since we're only got two minutes left uh definitely come to our next session about how to boost your delta lake performance uh hank will actually be the one presenting so hank. Thank you very much. I did want to say thank you very much to zadong, hang aaron leo uh for this awesome session and answering all the questions and presenting for the first of our four-part series on the sales source engineering with delta lake uh craig and fish chris. Thank you very much for answering all the questions as well.

G

um I think we're done for today's session, um so please be on the lookout. uh Karen anything else, you'd like to add before we wrap up today.

A

No, I think you covered everything thanks so much jenny um I'll just go back thanks. Everyone for this awesome presentation for craig and fish for moderating. I dropped the youtube, recording, link and youtube in uh in the linkedin and zoom chat spaces so feel free to check that out and then join. Please join the data plus ai online meetup group uh to stay up to date and check out the upcoming session.

A

So we have three more so I hope you'll check those out so with that uh thanks everyone for attending- and I hope you have a great rest of your day- take care.