Project Harbor Community, 22 Apr 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Harbor Community Meeting - Americas Time Zone - April 22, 2020

Description

4/22/2020

Agenda
https://github.com/goharbor/community/wiki/Harbor-Community-Meetings

Harbor 2.0 status update
RC released! https://github.com/goharbor/harbor/releases/tag/v2.0.0-rc1
Harbor 2.1 plan introduction
2.1 feature suggestions - https://github.com/orgs/goharbor/projects/1
non-blocking GC's proposal https://github.com/goharbor/community/pull/133
Proxy cache proposal https://github.com/goharbor/harbor/issues/8082
Support for p2p https://github.com/goharbor/harbor/issues/6890
Harbor (core) Operator
V0.5.0 is available now
Deploy harbor stack (required + optional (Clair, ChartMuseum and Notary)
Delete stack
Update(Provision optional components)
Follow the installation guide to have a try
Feedback is welcomed
Great thanks to OVH team(Pierre, Jérémie and Maxime) and ZiMing from VMware Harbor team
Next:
CR per component (separate reconcile process)
Provide chart for deploying operator
Harbor Cluster Operator
V0.5 is planned. You can check the plan items here.
Coding has been started
PRs are opened to the whole community
Welcome reviews!
You can find the related design documents in the docs folder
Feedback is welcomed
We have weekly DEV status sync meeting, you can find the minutes here.
Replication adapter compatibility verification testing framework - Replication WG
Harbor has many replication adapters to adapt kinds of container registries

harbor(base)/docker registry: Wenkai Yin
Docker Hub: De Chen
Ali-ACR: Frank Kong
AWS-ECR/GCR: Ziming Zhang
helm hub: Mingming Pei
hua wei SWR: QINGDOU
Jfrog/quay.(io): Chengyu Zhang
gitlab: @lxShaDoWxl
Other adapters are on the way (pending)

TCR (Tencent container registry)
DTR
We need a way to make sure the adapters can work/still do work well with the given supporting Harbor versions

Harbor is upgrading
External registries (services) are upgrading
Draft ideas

A separate repository under goharbor
Setup periodical(24h) running the pipeline on top of GitHub Actions to run verification testing cases
Testing cases will reuse the current cases running in the nightly pipeline against GCR, Dockerhub and ECR
Publish daily verification testing report
Harbor GCR ECR Dockhub ...
1.10 N Push:Y Pull:N Y ...
2.0 Y Y Y ...
Discussion details

A

Hello, everybody and welcome to the harvard community meeting. This is the european uh well america's friendly time zone, it's the 22nd of april and, as always, is a recorded meeting. So please adhere to the cncf code of conduct, so a few folks today. So we have myself chris jonas tianon john and gary welcome all um I want to give you a couple of quick updates and we have a new location for our harbour community meetings agenda. uh It's actually documented also in our github, so you go to github, go harbor community.

A

And you look at the meeting schedule. It basically talks about where you can find our agenda uh and then it's also in our website as well. um It's just a little bit more friendly to have this versus doing powerpoint presentations so feel free. If you want to add something to the agenda for a future presentation added, we keep the same agenda as the one with the earlier meeting that happens at the day in pacifi, sorry at 6am pacific. um That basically is more friendly to european in asia.

A

uh I want to talk a little about harvard to the dough. We have a release candidate, so if you are, if you're interested in checking out oci compliance, ssl certificates, slack integration for uh for uh for web hooks uh the ability to individually target which webhook you want to get notification on versus all uh all of those are features of to the dough. They are in the release candidate and we're probably uh about maybe a week away from release. So and when I say release, I don't mean release to everybody.

A

Is I'm talking about as a hardware, we're ready to have all our binaries ready and then we're gonna. Take the next two weeks after that to go through documentation, uh finish up our blog post and do all the necessary things to basically get the release tied up, so the code should be done in about a week and and and tied up, but then will take a few more weeks to uh to finish up the activities around.

A

uh You know, field, enablement and customer enablement, but for all intents and purposes check it out, get the bill put it through spin and then, uh if you have any feedback or suggestions come back to us, we have a few links here around 2.1 and going through the two to one feature: set: uh there's pull requests here. We encourage you guys to go, provide feedback. uh We've walked through p2p last week, so I'm not gonna. Do it again.

A

um We've talked about proxy cache, maybe the meeting before even that, so the proposals are here uh go provide if the feature is important to you at least tell us that if you have feedback in how we plan to implement this also give us feedback and then same thing for uh for p2p and if you're interested in helping us deliver.

A

Any of these features tell us that too, okay, um the hardware core operator that we're working on the version 0.5 is available now and we're progressing well towards a vision of actually making this a stable release at some point in the future. um We don't have a date for that, but it's it's not years out. It's a few months out. um The the important thing there is that this is mainly community driven and so, as a community has different needs in terms of what to do around life cycle management um that you know.

A

This is basically uh a work in progress right, so small folks and get involved into the project they bring their own needs and their own requirements, and this is evolving over time. So it started from uh ovh as the company that was the main contributor and now that we have a lot more contributors are trying this out. There's an installation guide, uh give it a try.

A

It's linked here from the documents so um highly encouraged to to give this a shot, and I want to mention one thing that daniel one of our maintainers mentioned he's from aqua security. He asked a very good question on the morning. Call asking hey. So what's going to happen, this uh are you going to integrate with the operator hub for lifecycle management of the operator itself?

A

um The answer is a very good question. We haven't thought of that and- and you know we post it to the folks working on the operator, so they can actually figure it out. uh You know what are the benefits and drawbacks of that, um and that question came up from the fact that there is a home charge now for the operator to make it easier for for consuming the operator. So the idea was h. Should you also post it to the operator hub, so something for us to think about.

A

The harbour cluster operator, which is the operator, that's basically a superset of the core operator that basically would enable postgres and radius and deploy some of the ancillary components of hardware.

A

The coding has started after doing some planning, so you can find some of the design documents here. So, if you're interested in learning more.

A

And then have a quick update here on the some of the replication. Adapters are happening in in hardware, so we're working on a verification testing framework so that it makes it easier for us to validate all these adapters from release a to release b to release c and uh the the different folks are kind of working on that is is outlined here on the document, and we also have a couple more adapters that are on the way kind of identifying who's going to help us with some of the ci around.

A

This and that's kind of the the last of the items that we had here. uh You know the main thing is, you know progressing well on to the dough. We are we're already planning, 2.1 and actually we're gonna have a maintainers uh meeting uh next week. I believe to kind of uh finalize the deliverables for this release and if you guys haven't noticed, our maintenance team right now is like has six companies involved.

A

So uh all of those folks are gonna come in and and kind of agree on uh the modus operandi for the next release of hardware and the kind of the themes that you wanna influence for this release, so so good things um in conjunction with harbor to the dough there's also some exciting things happening in the community. So, as you guys are well aware, harbor is up for graduation.

A

uh We kind of pretty much are the 90 part of finishing up our graduation review with uh sig security, which is our last sig. That's revealing harbor. uh We had a meeting both last night and today, overall, good feedback, good, good, good progress and right now, they're writing down their assessment. So we are in good hands to potentially put harper up for a vote. I will come to every single person. That's used.

A

Hardware, that's been on these meetings and as you guys, when the toc basically asks for the public feedback time, so public can either endorse or say something against harbor in their bid for graduation can ask for everyone for their support. So that means getting your github account going up to the pr and voicing your support for hardware.

A

Hopefully support right, nobody says anything bad. I know where you live just kidding all right, um so um I wanted to kind of bring up another item after this. You know I want to get some feedback in terms of how these meetings are going. um This is useful for you. Do you want to see something different? Is there a different structure that you can be using? That would be more valuable to you.

A

I want to open it up. You know we really want to learn if there's a better way for us to do this.

B

Would it um yeah, would it be more um useful if we did demos and things like that during this meeting as well uh of new features, uh so you could see what's coming in in the next release and things like that.

C

I mean that's my main use case for attending. These meetings is just to learn about what's coming up in harbor and keep kind of my finger on the pulse of like when we need to upgrade or when we need to plan our next upgrade.

A

Anybody else anything else they want to see.

C

Hey this is tienen. um I I think these meetings are awesome. I really appreciate you guys putting them together and giving us basically a synopsis of what the heck has happened in harbor land in the last two weeks I mean it's, I think it's a good format and I think you cover the bases really well.

C

Demos would be fun, but having given demos and prepared for demos, I I know that that's sometimes too much to ask. So I'm not going to ask for demos but.

A

I'm actually going to do one thing: I'm going to be like kelsey hightower right now, and I'm going to do a demo on the fly on an environment that have never opened in a long time. But I'm going to see. If actually I can do this, but.

A

Let's just hope is the first part is that the folks kept the same password.

A

And they have okay, this is hardware 2. This should be hardwood to the dough.

C

I love that dark mode.

A

Yeah, so this is yeah. This is hardware 2.0. This is probably a very very recent build.

A

It was built today, all right, so so, let's take a look at some things. Can we so interrogation services, two built-in interrogation, services, 3v and clear? So now you have 3v out of the box. It's installed whether you use docker compose or the home chart. So you, the trivia, will be the default for new projects, but for upgraded projects you get to have whatever you had before. So if you have no scanner enabled, no scanner will be enabled.

A

If you have clear in the past, claire will maintain, but we really really urge people to start using 3d over time. We're going to phase out clear and you're going to see that disappear. Okay and obviously the some of the other settings remain the same.

A

You get to see a new option in garbage collection called delete, untagged artifacts, when you do garbage collection. If there's an artifact, that's not tagged, and this is important. I'm going to show you guys in a second. You can delete it in the past. We always deleted untagged artifacts, but that was because antarctic untagged artifacts never showed up in hardware now untucked our device has to show up. So if I go to project now- and I look at this, guy seems to have a couple of repositories.

A

So let's look at the repos. So if I look at this now, I have this artifact, which has it which is identified by sha and look at this. This has multiple art.

A

Well, actually has only has one architecture and it has one tag, but I could right click on this and I can say, remove tag and I'm not gonna do it because someone might be using it so now that artifact, when I remove it, it will be won't be orphaned because it will still show up in harbor and you can tag it over and over again. So I can actually come over here and add a second tag and I'm going to call it chris d and now I added the second tag for this.

A

So now you have two tags. You never could do that with hardware before and I can go and say hey you know, I don't need chris d anymore, so I'm gonna go ahead and delete it.

A

Oh my god did he delete both tags?

A

Okay, I don't. I didn't even check what the option was. I just said: okay, okay, someone's test case. It's gonna break.

A

So reading is very important. uh I didn't read what the warning was, so I just said: go ahead and do it.

D

You'll have to watch the recording and you can get the tag back. Yeah.

A

Jonas, do you remember what the tag was?

A

A

All right, so I deleted your tag, so I'm gonna add this here, so someone can actually come. Hopefully, they'll see that uh I delete your title. Maybe I should rename it I deleted your tiger. My name is jonas, so if you have multi-architecture tags, each tag will be. Each bundle will identify that as well. Let me see if they actually have any repos with multi-architectures here.

A

Okay, this one has three, maybe has some multi-architecture stuff.

A

Okay, this one has no artifacts. You have seen that from earlier.

A

This is also one single one. Only okay, so there's nothing in here, but that's basically one thing that you can do right so now, every because we're oci compliant, you have a thing called an oci index and the oci index becomes the first class citizen in hardware and under that oci index you can have multiple architecture, uh images or or if you have a signa bundle. You could have a collection of images, for example.

A

So now you can enforce policy and all the other things based on the bundle uh home charts, slowly we're going to start facing them away because home charts are oci compliant. So we don't want to use the chart museums anymore. It's also something that's antiquated so over time we want to remove it. Members that doesn't change much same thing. uh One of the things we added not too long ago was uh basically in 1.10.

A

I believe we added the support for groups, you know ydc so before you only could do ldap groups now you can do oidc groups, so that's great benefit uh labels, uh same stuff scanners, kind of walk through that. Let's talk a little bit about policy. Both tag, retention and target mutability have been updated to support the new capabilities around oci and what it means to be able to retain an image right. So you're going to see some changes here.

A

So untagged artifacts also become a key part of this here and whether you want to include them or exclude them to target retentions and and tag immutability uh also uh same thing. It has updated them to be compatible with our oci plan.

A

Robot accounts. One of the things that came up today is that today you can only have robot accounts that to push and pull that's great, but a lot of our scanners today, like, for example, if you're sneak or aqua or ankle, you want to pull all the images out of hardware um if they're, using hardboard, not through the cable scanner framework, without actually being succumbed to the policy that says well, don't pull an image that has vulnerabilities.

A

So how do you do that? So we're basically looking into capability for 2.1 about creating an overarching account almost like a robot account with pull only credentials that can be enabled at the pluggable scanner level to enable them a robot like experience, but across all projects. That's that's something they're! Looking up for 2.1 webhooks, one of the ideas that improved here is that you have an option to decide which web books you want to enable. So if you only want artifact push, you can uncheck everything else and you only get artifact push.

A

The second one is that you are going to have slag integration. So if you're interested in getting your notifications in slack, you can give us your endpoint url uh and you want to basically verify remote certificate. You can do that as well, give it a name and and add it into hardware um and then from config. Not not much has changed here. Like you know, code released ago, we added the white list. So that's that's nothing new here, uh but you're gonna see some new things around the replication.

A

So if I actually want to add a new endpoint for application here, you're gonna notice that the providers has exploded, alibaba, aws, azure, docker, hub docker registry- that's the docker distribution, gitlab, google, gcr hardware home hub huawei jet frequency factory quay.

A

So we are adding more and more. We want harvard to be the good ecosystem ecosystem citizen and you want to push and pull into harbor for pretty much anything else out there. You can do that.

A

That's it and if you don't like the dark mode, you click on light and go back to traditional hardware, so bright yeah. So there you go. You got a quick demo of 2.0.

C

um Because I'm I'm really excited this is tiana and I'm really excited about the the multi-architecture functionality available here. But I wanted to point out that one thing that I was extra excited about is that replications also support that so.

B

C

Example, you can replicate images from docker hub into your local harbor instance, and they keep whatever architecture list they had on docker hub as well. That's right.

C

Can I ask you a couple of questions.

A

Yeah yeah, absolutely we might charge you for it, but to ask.

C

Okay, so um on the 2.0 release candidate, I noticed there's an offline installer option, but there's not the online installer bundle. Was that intentional or was that an oversight.

A

Yeah, so the offline installer is a docker compose and that's what we build right away. It usually takes us a few days to build the home charts.

C

Right um on the previous releases, there was also an online installer option that doesn't include all of the saved images.

A

Yeah, I don't know, I don't know why they didn't create that yet jonas, can you take a quick note and ask the team why they maybe maybe they're working on it? I don't know why they didn't produce it.

B

For which specific thing for.

A

Harvard 2.0 the rc yeah, the online installer is not there.

B

Oh okay, I'll make a note.

A

Yeah, I don't know why I mean I haven't looked at the release. Bundle but yeah, that's a good point. You know we can take a look at why.

C

And my other question was around the the oci support, so in the distribution spec of the oci they've started working on some some real conformance tests that can be run against a registry.

C

um Have you played with those at all or has has anyone in the harbor team played with those and hooked them up to harbor and see where harbor stands today?.

A

uh We we have not, but we've talked about this before, and maybe this is a good conversation for us to have one more time. I don't have any data on you. I actually looked at uh some conformance tests both for uh for oci, as well as for ingress for contour, so so this is. This is good uh to to actually bring it up and see if someone has banded to go through this.

C

Yeah, I think they're they're, looking for um registry vendors like harbor, to submit the results of their conformance testing.

A

Can you send me an email? Can you send me an email if you have, if you have a link of where they are expecting these results to be submitted? Do you know.

C

Absolutely yeah I'll send you an email with details about it.

A

Yeah that'd be great, I'm alex is uh who's. Probably asleep. Right now is looking into oci. He might be aware of it, but just in case I'll ask him. I know they've talked about conformance in the past, but it's been a while. So it's definitely something that it's not top of mind so I'll bubble it up again.

C

Awesome and I was curious on the the replication providers. What what makes each of those replication providers different enough to warrant having their own entry like the as far as I understand, most of them have a a standard registry interface and that's kind of the point right. So why? Why are they all listed separately like what makes docker hub, inherently different than replicating from a harbor instance.

A

Yeah so so part of it, I think, is vanity, and since, since each of those like, for example, we had to get alibaba to come in and test it.

A

So it's not just supporting it right, there's also the testing component of it, and that's one of the things I mentioned earlier and we had to get one way to test it uh once they do that they actually want their name to be in there and they they it just bought to vanity, is the engine behind it that supports I almost identical for most of them, yes, but because you have to test independently for every one of them both for the bi-directional replication either in or out.

A

I mean it's really a sync in some ways, then, and and the fact that some of these vendors are doing work. They want to see their name in there and that's why it's in there right. Okay, that's fair! Otherwise, we have had to abstract it into a couple of things right, for example, we had to here. Let me share my screen because not all of them are equal, so let me go back so so not all of them are equal. So let me go to the registries, so I believe quay is the first one.

A

That's different right. I think way. No, it's an artifactory. No one of these is accepts a json.

A

There it is okay, google gcr so from the moment that you have at least one or two that are not the same as others, and you have to call it out specifically, then everybody wants their own name in there yeah. Otherwise it would say google gcr every other registry, but then you know who do you? Support like different versions of harbor might support different ones like quay and artifactory are new, so it makes sense for us to hey, let's give everybody their their their logo and their stuff in there and.

D

A

You, if you look at our decks as well, we actually call it out independently for each one of them, like we have our big hardware architecture document, if I actually search for hardware architecture wiki. Hopefully this search works well,.

A

I think this is it.

A

Yeah, so we start calling them all out here. Obviously this hasn't been updated for 2., but everybody gets their own slot here on the right.

C

Got it? Thank you.

A

You're welcome.

A

Any any other questions anything else.

D

A question for you about the the new functionality for untagged images and the garbage collection there. So it was really cool to see that you can add tags and remove tags, I'm assuming once all of the tags are removed or or even if they're, not all removed, there's a different option to explicitly delete an image, and is that tracked separately so that it is garbage collected? If you don't use the check box of garbage, collect, untagged images.

A

Yeah, so so, essentially, if the layers are are basically all gone, um then obviously you know uh like there's.

A

If there's no reference layers to you and you don't have a tag, then the image will be gone anyway and they're untagged and we're gonna document this fairly well in in our docs, in terms of where does untag come into play for garbage collection like when do you want like and and the reason why I bring it up, is because, if you upgrade from version 1.10 of fireball into 2.0- and you have images are not used, it might show up as untagged you know.

A

Did you want to see them like it used to be before or not right? Your experience with cardboard before was that once an image can have a tag, it would be legible for gc.

A

That's why untagged is an option there, because you may want to enforce that experience like hey. If an image doesn't have a tag, go ahead and kill it.

D

Yeah, that makes sense thanks.

A

All right, everybody well we're out of time. Thank you for the conversation was engaging but yeah like I'm totally. Okay. Every time we have a meeting here to either do a a a quick demo as well. We can you know you can do it. Maybe I won't delete the tag that was there next time, I'll be a little more careful, but we'll uh we'll do that, and, and whatever is valuable to you, feel free to add items to the agenda. I really encourage you like.

A

uh I want to start seeing you guys at items and say hey. I would like to know about x or y, who could also give us some time to prepare all right. Everybody well have a good day. Thank you all for attending appreciate it.

B

Thanks everyone.