Project Harbor Community, 15 Dec 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CNCF Harbor's Community Zoom Meeting - Dec 15, 2021

Description

CNCF Harbor's Community Zoom Meeting

A

Hello, everyone, my name, is william vasilev and I'm the community manager for harbor uh today is december 15th and that's official cncf meeting. So please follow the conduct and just be a nice person. With that said, I can see we have one uh proposal to discuss. So can you please share that or do you want me to share uh yeah.

B

A

Okay, that would be nice, okay,.

B

Okay, can you see my screen yeah yeah, okay, uh hi, everyone, I'm tony and today I want to share the proposal of a synchronized update of artifacts protem and pro count.

B

The background is we when we we find, we found some db connections back in our environment. So we do some performance testing in our environment and find this find this issue. So the protein and pro-count you can, you can see the protein is artifact protein at the artifact page in in the hardware ui, and the pro count will choose at the the repository page. It's the post represents the poor count.

B

Okay, um so so uh currently, if the user pulls an artifact from the hardware and hardware call component will raise two guarantees to update the artifacts protem and pro count in the background and for every proquest.

B

So and since it is a concurrent operation or a single recorder of the same table, we met some problems like routines, peeling up and the database locating in the case of high concrete poor. So this will also take up number of connections to the database and those reduce the performance of harbor.

B

So my my proposal uh is is to synchronize, update and merge the update operation of artifacts protem and pro count to reduce the duplication of update work to the database and also provide, and we also provide the configuration to keep the original version updates, which means the sigma.

B

This is the my proposal is a sync, a sync mode.

B

Okay, our goal our goals. I list the three points. The first one is reduce the number of database logs and the second one is fuel operations on the database output update and the third one is improve the throughput of the harbor. In the case of high concrete pole and our non goals is we we? We cannot ensure the fully accurate protein and four kinds of artifacts and another is.

B

We cannot uh injure the immediate synchronized protein and poor count of artifact after you pulling the artifact, which means if you pull pull an artifact from harbor and you go to the harbor ui page to see the protein or procons. Maybe it's not synchronized.

B

So the scenic update operations will is not perfect. It will bring up some side effects like tag rotation and gc, which depends on which dependence on the artifacts for them to filter candidates, but the probability of issues is dependent on the ascending flash time duration.

B

So, if your interval is relatively small, then the probability of problems will be lower.

B

So the implementation is code related. The code path is under the src controller event, handler internet artifact, dot go and the basic code implementation is. It is very easy. I just separated the two three points. uh The first one is catch the operation of pour into the cat storage in the previous implementation.

B

uh If you, uh if you're the poor artifact, we will update the db and after we will keep, we will not update the db directly but catch the operation into the cat storage and the second is merge the operations and only catch the final state.

B

For example, in the previous, uh if user, post post an artifact for 100 times, we will update the database for 100 times, but uh after we improve, uh we just uh need to update the database for one time and just set the uh just set the four count or four counts to to 100. In previous. We need uh we need to update the pro counts, to plus one every time and and uh for 100 attempts.

B

But after we only need one one time and this the last one is uh we, we will uh pre-nuclear flash data from catch to the database.

B

So we have a configuration time interval to flash the data from cache to the database so about the cache selection. We consider radis and memory, and we do some compilation on these two two two selections, the advantages of radius is raised. Service can be deployed in high available mode and the data of protem and pro count will not lost if captioning not.

B

The available network, I o operations between radius and core, maybe also includes the hardware component hardware performance and the and the business side uh needs to implement additional distributed logs to ensure data consistent when mutable core instance exists exist, because uh if, if you have mutable core instance for high available uh these mutable, this instance may be uh update the radius key at one time, and so you need to ensure the consistence to of the key and the value.

B

So uh the memory, the advantages of memory is uh memory. Operation is very fast, although radishes also fast, but memory is more fast and there's no network io needed, and uh if we use memory uh the we als. uh We also uh don't need too much code to maintenance, the the the heavy business business, business logic and mutable car handlers, their self-update operation.

B

So so no additional logic needed to keep the operation sequence. uh But the disadvantages uh is: is our data? uh What are stored in memory? So, if call crashed or restart the data will be lost and the similar update work when mutable coincidence exists, which means if, if you have multiple coincidence, the these instances uh do the similar update work.

B

So, after all the considerations, we intend to use memory as a catching solution which which has less maintenance costs, although it may bring the problem of data loss, but the the but the protein and protons may be not very important, not very critical data.

B

And but we can reduce the probability of the problem by adjusting the flash cycle. Smaller.

B

Okay below is the code model for this future and and and it's it's just a demo, so you can review the pro request for more code details here. I just I just expanded fast, so we will maintain two uh store.

B

One is pro constor and another is the pro count store a protein and product store and two logs to to to keep the uh sync and and uh if, if user, pull an effect, uh we only update the data in the cache memory and and we we will have two goal routines as consumer or worker to consume the the two stores data.

B

And the default the default flash duration is 10 10 seconds. So if user not provide or configure the flash duration, we will use 10 seconds.

B

If user want to configure this, they can pass the environment variable to override this field.

B

Okay, so the code is not very a very, very fresh. You can reveal the podcast for more newer, your implementation, it's just a design model of demo.

B

So finally, we do some comparation of optimization.

B

uh Our scenarios is, however, deployed by local compose in the host, which is cpu cores and 16 g memory, and we use k6 to simulate 5000 concurrency to pause the same image manifest, and I have linked the test tool. It's under the goal, harbor organization.

B

So let's see the summary data of this before.

B

Before we, we run the test and we closed about 16 minutes and about 1000 requests has failed.

B

The success rate is not 100 and after we fix, we only cost about eight eight minutes and the and all requests has uh has successful, and uh the average uh requests is about five 500 requests and after it's about uh 900.

B

And okay: let's see more runtime metrics.

B

uh First, let's check the pg state activity uh before the the pg connection is very high. It's reached which the limit of car set we set. The max open connections, 900 and the active connection of pg is, is 900 reached the limits until tested down and after we fix, we can see, the yellow line is active connection, it's very low and the green. The green line is idle connection which can be ignored because uh it's it will be released after the test done. It will not influence the the normal user. Normal use.

B

Okay, next, let's check the pt logs count and before the pt logs count number is very high. It's about 3000 and and it's it's, we are not it's, it's not reduced and after we fix the logs number is very small.

B

It's only a little spike.

B

And finally, is the harbourcon guru things before we check the ground things of harvard core. It looks like it looks like.

B

About uh about between uh twelve thousand and fourteen forty fourteen thousand, uh like which legs hung out at a fixed number, but after we fix the ground, things has choose dynamic change and the max guarantees numbers has reached. 20 000, which means uh golan run time, can schedule a mark, can schedule more guarantees to to to handle some business logic uh which can approve, uh uh improves the through output of the harbor.

B

B

Let's go back, that's all about the the proposal and you can check the harbor.

B

Posterior pull request to see more code details and you can leave leave your comments comments if you have any question yeah.

B

Okay. Thank you. That's all! Thank you.

A

Anyone wants to discuss that any questions.

C

um I have a question regarding the inconsistency of the pull time and pull count, and I was thinking about how, because you you mentioned that um it's not consistent, it's clear, because the two systems not synchronized, um but you could turn it. You could turn it into eventually consistent state. So if you update only the latest timestamp and always update the highest number of count, you will end up in an eventually consistent state, meaning that after the pool rates settle down, you will have a correct state of the pool time.

C

The last pool time and the state of the correct pool count.

C

Right, you could, you could have this eventually consistent state, and this would be even even even better if you just you, know, update the latest timestamp like the newer, timestamp and also uh add the count on top, so you always count up, but only overwrite, the timestamp. If it's newer, so this this you will. You will then up end up in an eventually consistent state which would be, I think then yeah. It's fine for everyone.

C

um And then another question I have uh is: did you did you had a chance to look at uh pulling mechanisms of postgres uh not of postgres but the pulling pulling uh pool libraries so because I think it it would also help if we, if harbor, would use a proper pooling library like this, which is not the one that's built in, which is yeah kind of a really low or not not fun, not written functionality.

C

So I think if, if we would use a a proper pooling library like a pgx, this would also be um would have different numbers on than than shown here.

B

Yeah, we, I think we have another easier to trick trick, replace the the library to pg pgx.

C

C

Okay, that's it from from my side, no other questions.

D

Yeah yeah ready, thank you, and would you please comment on that? Pr uh for the uh you mentally consistent.

A

Are you connected.

D

Yeah, thank you anything you have. I also have a question. uh You mentioned the data. We are flush to the db uh periodically right. The default uh default is 10 seconds. I I also can understand if the.

D

If the process is crashed, so we will. We lost the last uh data right but yeah. Yes, if the instance is stopped, it stopped grease free so where that data will be flushed to the db before the process exists,.

B

uh No because the uh the our our our the the artifact uh event handler is is is count can't be stateful, so so the order you stopped your harbor uh state for uh the the data we were also lost.

D

A

Thanks a lot for this one, uh please everyone uh use your vote and comment for this proposal, so we can drive it through.

A

um Okay. What we have next on the agenda, few topics from my site: can you everyone? uh I didn't? Okay, let me share.

A

um There's a a pr to update the governance dock, it's kind of working progress, but please everyone can you take a look. I can all assign you as reviewers, because we need a super majority of this one to accept the new governance dock.

A

So we want to introduce a few new six uh six community and, for example, the upcoming january me and abigail we're going to kick off a new effort to start the docs working group, which will be hosted under a different sick just for purpose of managing it properly.

A

So uh my ask is, if you can go over this pull request in the community uh repository uh 108.80, um if you can read through the document and mark something if you're not happy with it or add new stuff.

A

The next one falls them again, it's about to close in two weeks. I think and kubecon cfps are to close in two days. So please reach out to me if you need help to create your cfp.

A

If you're interested in any of these conferences, I've already submitted two cfps for fosdem uh one for cover and one for another project that I'm working on, and my idea is to do like a harbor, 101 gonna talk very brief.

A

Introduction status, update on the project, deployment and few demos around few stuff that I want to show the audience and primarily is to try to get some some folks joining us from that community, which is huge community of developers, and my last topic for today is, as I said uh me and abigail you're gonna kick off a dog's working group, maybe second half of january, so any recommendations from your side or any inputs would be great, um so we're gonna use already existing process to start it up and announce it properly to the proper channels.

A

So if you have something in mind or if you have friends that want to join open source community or want to join us, the docs working group will will be a great start for everyone yeah. So that's from my side, if I was everyone else, and anyone else has a question or want to discuss something.

C

um I had the question to the cfp: um I'm planning to to make a cfp for harbor for for a coupon um and I'm currently writing down. I was looking at the cfp process and it says uh I've posted youtube on slack as well.

C

There are two two processes: one is for the normal process and one is like the maintainer process, which should be a different one, but I could not find it where and how it is.

A

Are you trying to submit uh for the maintainers track or you as a maintainer, to submit a talk.

C

um Or more likely, the second one, because it will it will be about harbor, but it will be also about other tools like open policy agents, it's more about like integrating harbor into kubernetes and automate, workflows and and tools there. You know presenting the tools, how they're working and bringing it all together for for the kubernetes operator. So it's it's not about the showing new features functionality of hardware, it's more about how to work with hardware and the kubernetes environment.

C

So the question is: is it still kind of the maintainers track, or is it independent of that.

A

I think uh not sure if we already submitted the maintenance track talk. Do you know that.

D

Yeah, I think we we kind of do not submit and we do not get a notification to start me today: maintenance tracking, I don't know- and the redeem just mentioned. I think it is more like to submit a safety.

C

Okay, so it's it's, it's not, then then um it's not the main chain attracting.

D

Yeah, I mean usually.

B

D

The past uh in nintendo chuckle, we will uh introduce the new features and the road map and.

A

D

uh Brief introduction for the new new users to the community and also will include some guidance to tell a community how to participate in the hub community.

C

Okay, okay, then, then this will be different. Yeah then I'll create a normal cfp. Then.

D

Okay yeah, but that's really want to remind the looks like the cfp. I mean I'm sorry, I mean the deadline for uh some militia p. Is this friday.

C

D

Yeah, so only two days uh left.

A

Yeah and and- and I think last time uh steven do uh fill in the deform for uh the maintainers track. I hope we don't. We don't miss this one, this time um yep as we did for china.

A

Sorry, uh okay, yeah, um but I can get into three with cncf if you want, uh if you you're, not able you to find the link to submit a talk as a maintainer, because I think there are two processes, as you said, like a uh normal speaker and one from maintainer point of view, so I'll try to help you out with this one.

C

Right, that's very cool. Okay! Thank you.

A

And are you gonna submit something for fosdem as well, because I think you said that you, maybe you you want to do it.

C

I can okay yeah. If I have it for keep call, I can also submit it for if you post them as well.

A

Yep all right, uh so in that case, I think we can coordinate that that's a bit. If you want, if any, if any of us is accepted, I think we can we can join. If you want and do something together, it would be cool. I think right, okay, um another topic, uh since the christmas holidays are approaching in the vast past of the world since starting next week.

A

I think we can skip all meetings until january the beginning of january, but I just want to have everyone's agreement on this call. uh If everyone is happy with this one.

A

I can see only with him checking his, so I can get that. That's a okay uh and yeah all the other quiet people I'll get your your silence as a yes as well. Okay! So what I'm gonna? Do I'm gonna um write a mail to the manic list and pin a message into the slack channel that we're gonna skip until january. Whatever comes the first meeting.

D

Yeah sunscreen sounds good yeah I have just mentioned.

D

I have just noticed that av and put a question in the chat and the regarding update on 2.5, so I can provide a briefing update for that and, uh like we talked in previous community meeting, when we plan in the 2.05, so in 2.5 we majorly have our focus on support the cosen.

D

We have shared the proposal involved with the community community meeting and another focus is on you: don't improve the performance.

D

About the I mean about the database label, like a can you just present to introduce some catch layer.

D

Also, another area is improve the dc enhancement. I remember our team have collected some gc reality issues in the discussion, so we have included some issues from that to in 2.5.

D

So currently we are still in progress to working with these kind of new features or improvement. So that's the current status. I think.

A

Darling, do we have the plan for the upcoming releases and like overall dates.

D

Yeah we have, uh I think we have mentioned it previously. uh The the uh currently the rough target date for 2005 is by end of february next year.

A

So we, if it's end of february, that means uh that would be like end of may or or june, will be 2.6.

D

Oh, we maintain four months. Is the candidates right.

A

Okay, so, okay, so that will be june july. Something.

D

A

A

This uh trying to figure out uh what will be the update for for kubecon uh if it's go, if it's going to be 2.6, but we won't be there yet. Thank you for that and we have another question about the lock for j2 vulnerability.

A

Anyone can talk about it.

C

Yeah there was, there was one already on the on the slack chat and a similar question, and it does not affect harbor, because harbor is not using any jvm.

C

So there is no search capability built in uh in humber, which is based on elasticsearch or in on solaris, or uh that that like so there is, I mean harbor is not affected by this vulnerability, because there is nowhere in the whole stack uh log4j used.

C

So this we can, we can answer it. It's not affected by my dispensability.

A

Can you write a mail, a very short one, to the to the main list, so we can answer that uh yeah for the whole community, not to uh like I mean, and we can point to that mail in tweet and everything so, okay, we can. We can inform the whole community that we're clear of this one yeah thanks for that. I I'm I'm I I was gonna write it, but I'm not sure if I can wear this properly in the right terms. As you already.

C

Did so I can you can do this now.

A

Yeah yeah, that would be fantastic.

D

Yeah, that's cool.

A

A

A

Nope, okay, then thank you for your attendance today very useful discussions. I think and see you after the new year's, if on like was that 50? No, that should be 6th of january. I suppose, if not the first time right after the christmas break, happy holidays, everyone and talk to you, bye, bye, bye, bye,.