GitLab Create Editor Group, 3 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Create:Editor Product/UX Weekly - 2021-03-03

Description

Weekly Editor group sync between Product, Design, and UX Research

A

Hi everyone: this is the uh editor group, product design, ux research, sync for march 3rd 2021 and I usually start off the meeting with a bunch of droning on and on. So I mixed up the agenda so we'll start with catherine and ux research.

B

Okay hi, so I just have some updates from the last time, so I ran an experiment on a proposed navigation structure and now that I think about it. That link might actually be to the issue, not the results, but they are lower. In the point- um and I I found I can't I came across another challenge where the overall success rate was still pretty low, but it was also pointing towards settings again. um So what I'm?

B

What I'm gathering is that settings is, is kind of the hot spot of some issues as well as analytics as we saw in the previous study, um but so there were some proposals that I went against as a result of this study, but I just listed some of the outcomes that I think would be good to explore.

B

One, that's interesting because I created an issue about it, but I also saw that there was one previously is around deploy tokens. This one is slightly mystifying me because, in both tests, less than 10 of people clicked on settings repository, which is where they currently are located, the number one choice was access tokens in the first test. So I was like okay. Well, maybe that is a result of priming.

B

You know tokens and tokens, but it happened again, and so it's something that I think we should consider, but I don't have a direct answer for it, because I'm not sure if I'm not really sure where it would be better for them, so that maybe you might have thoughts on that.

B

I did find an issue where I think we had recently. We had previously considered moving them to cicd settings, but then an issue came up around the permissions, so if pipelines were turned off, then you wouldn't be able to access these deploy tokens or something like that. So that's a challenging spot would appreciate some feedback there. If you have any thoughts on it, but otherwise feel free to take a look at the results of the studies, and let me just comment the link to the issue as well about moving it to cicd settings.

C

uh This is really good work, catherine um I'll.

B

C

A look at this over the next few days, like there's a lot of interesting things here, and it like dovetails really nicely into some of the left. Nav interaction, changes that, like are kind of in a good spot in my world, so yeah. I.

B

C

I'm probably going to make some like like help push some of this stuff forward. Anything that's kind of obvious the token one is kind of interesting I'll dig in deeper, because I know um some other designers are looking at like consolidating some tokens in like the user settings world, so that right.

B

C

But yeah this one's a separate token, but yeah cool.

B

It's definitely an interesting one and it kind of showed up a little bit in even a test that I did on the docs site, which was asking where they would go to find guidance on that, and there was a good percentage of people clicking on ci cd, and it's just it's on the it seems like it's one of those things that's on the cusp, so maybe that's another idea around multiple placements for a setting or considering the context out of all these tests.

B

I would definitely say that that concept of contextual access to settings was definitely strong as a result from that.

B

But yeah okay, but that is the the first point, and so I actually just had a talk with marcel our product design manager for create earlier today, and he talked about some potentially a job to be done: oriented approach to labels, which is basically like build release, analyze things like that, and we figure we'll just experiment with it and see how it goes. So. My plan is to run a hybrid card sort, and what that is is that you give people the categories that you're proposing.

B

So in this case we're going to try four and then you also let users create new categories if none of those apply to the item.

B

So the idea there is to start with a small amount and see kind of, what's the simplest you could go and if people kind of do much more from there, then you know okay, that's too simple, it's too broad, but if it actually kind of works, it might be something to move on to a tree test where we would give them tasks to complete and see how that compares to the existing structure.

B

So that's the test that will be coming up. So if you have thoughts on that feel free to check out check out the issue, leave any suggestions or concerns, and things like that. I also wrote some considerations in the issue description because there's a lot that would come with this sort of change, but it's all experimental so no need for alert or anything like that. It's it's an experiment, um yes and then. Lastly, I am currently working on some materials to communicate.

B

Findings from these various studies, but I that have just piled up so the first thing I'm working on is a slide deck and then also work on a video and probably we'll think of some other ways to communicate out the findings, considering that it touches on a lot of different stage groups and whatnot.

B

But that is where I currently am in terms of that. Oh and the last thing is, I forgot to add this. I completed a walkthrough of different products where I looked at their navigation, design and ia. So if you're curious to see that I will leave a link in the description- and we talked about this, uh but I forgot to share it as well and yeah. If there's any questions feel free to ask. Otherwise, I can move on to michael.

C

um Your um yeah, please link that um issue on your exploration of the competitors. um I left nav ia. I think that's really useful for everyone here on the team to take a look at just because a lot of this uh applications that you looked into are are competitors but like also like things that I never used before. So it's good to see the.

B

C

Of like how they structure and how they think about organizing things- and I think by looking at that- that kind of hints at like why um you're exploring these category labels for the left, nav and things like that. So it's a good visual kind of like oh.

B

C

B

C

Interesting ideas here: let's see, if that might work for us, so um really good work there. Catherine, thank you.

B

C

um Yeah, so on my side, um a pretty short update. um I made a bit of progress on the left, nav interaction.

C

So one of the challenges with our left nav was how do we solve one of the issues about removing like clicking on the titles uh when the popover happens and it's a little bit tricky, because our left nav allows you to click on the item as well as hovering opens a pop-up menu and then trying to solve that in the collapsed state and expanded state, as well as being keyboard accessible, was an interesting challenge.

C

So I think we're in a good state there that we can move ahead with some of the recommendations, and this frees me up to start looking at solution validations that um are probably like a few weeks behind. So I'm going to try to make some moves on setting search um in the editor like how whether we want to use blocks or some other pattern, and then some web id um research as well so over the next week or so, I'm going to start tackling these and these items.

C

Over to you, eric.

A

Thanks yeah, I just left a note under the left, nav stuff, which I think by the way, looks fantastic. The the direction you're headed with those it's looking really good.

A

So um there was a one of the issues: the jobs to be done, oriented category labels um issue that catherine just mentioned referred to this issue related to moving members out of a top level nav and putting it under project, which was something that we had just kind of tossed around over the past couple months, and I know it has been kind of tested and validated already.

A

I it seems like uh um yeah one more single item category that we could take out and move it under the project or group level, uh and I was just curious if you think we should just go ahead and fold that into the work that we're planning for the left, nav or, if you think that needs any further validation before we do. It.

C

A

C

At the issue, gut feel feels like if there's enough cross team like agreement on this, there's a rationale that we can go ahead and do that. But uh I'll answer that later today.

A

Yeah, I just pinged you on the issue so feel free to keep that async. But catherine did you looks like you.

B

Yeah we're about to type a comment yeah. So basically I I just shared this with manage access, so the access group, because they have an open issue around solution, validation and in the last comment loop.

B

They basically asked me um should we continue with this solution, validation issue, or do you think that we could kind of move towards the direction that we're expo that we mentioned, um and I was saying that actually, as a result of what I found in the research, we might consider two two locations, of course, which would potentially be the project overview as like, where you could view the members of a project and then the settings as a place where you can manage them, manage the people manage access things like that, because there was kind of a couple comments around two purposes like if I was doing this kind of looking through this as a project member or a developer or someone who doesn't have maybe higher higher level permissions to it, then project overview might make sense.

B

But if I wanted to manage them, settings would kind of be my more natural click, so I suggested that they explore moving it to both like to two different locations and see how that goes. So, I'm thinking in either case it would not. It would no longer be a top level item, but there still might be some research going around where it ends up exactly that.

A

Makes sense well, it seems like uh then, the the two options are either we move it into under the project, heading project members or we move it into project members and admin settings or something like that and and split the functionality.

A

So if we just move forward with the assumption that it's going to be under project, then we can wait on the outcomes of that research and decide whether we need to coordinate with them on splitting the functionality of that view up or what, um because I I they're still going to probably be the dri for that the functionality of that view. I would imagine we're only moving the link, so we would want to coordinate efforts, but not necessarily adding scope to the left. Nav work.

A

But thanks for letting us know about the ongoing discussion, um I guess on that note. I have um some notes here that I I typed in, um but the summary is basically, we have two main tracks that are forming over the coming, let's just say: quarter three milestones or so um leading up to uh 14-0 and 14-1, and I was talking with roman about this and in general I think we're um we're aligned that there's two main priorities for the group. We'll do other work.

A

You know on the side, I'm sure, but um the first priority is getting the top nav consolidation effort and the left nav ui pattern, ux uh redesign kind of work.

A

All you know full steam ahead, like I think that works pretty much ready to to start executing and in some cases we're already building the scaffolding for that for the top nav, so continuing to put a lot of resources against that over the coming three milestones, however long it takes, and then the second path is working towards the mvc of our rich text, editor component in the wiki and making sure that we have that as soon as possible. So we can start validating it and extending it and testing it.

A

So uh I say this with hesitancy because I don't want to set arbitrary deadlines, but I think it would be wonderful for the 14-0 release to have both of these in them, because 14o is a big number. It's a big milestone. Big changes can come with big milestones and be maybe more widely accepted, uh whereas, like 14 2, we change the top nav people might be a little more surprised.

A

um I say that, but also in in the notes and want to say out loud um they'll ship when they're ready, and so I'm not drawing a line or setting a marketing driven deadline or something like that. But uh roman and I talked about it. We felt like it was a realistic goal. We should move forward, assuming maybe both of those are landing. Roughly 14-0 and I'd like to start thinking about communication plans for those both um if they were both to to land um or when they both land.

A

I should say uh I think, they're big enough, that um some kind of blog post, or maybe even like a youtube unfiltered video where we discuss the motivations, the you know, the future plans and uh outline you know why we did it and what we hope to get out of it would be great for the settings in nav. I feel like that should probably be driven mostly from ux research and design, because those are primarily the the motivations behind the changes for the um rich text, editor for wysiwyg editing in general.

A

It's it's a design problem, but I think it'd be really interesting to write it from the maybe from the standpoint of engineering and how it's building a component, that's reusable across gitlab, and what it might, what doors it might open up down the road for extensibility and things like that. That's just my take right now.

A

I guess I'll probably want to create some issues and start having that discussion async with everybody, um but I think there's probably going to be a lot of documentation to update with the navigation changes too. So I want to make sure we get ahead of that and.

A

Yeah, I I don't think there's any like any other formal processes like we're, not deprecating anything or removing anything any functionality. So it really just be about proactively messaging and getting ahead of things and then having somewhere. We can point in the release post to a blog, post or or more information, whether that's on youtube or our blog.

A

I don't know if you have any thoughts on on all that.

B

That is a really good point about kind of walking through the thought process. Behind some of the changes, I could create something that's around the issues that I've created as a result of the research and then maybe michael. You could do something around the interactions and maybe changes in the the visual appearance and things like that.

C

Yeah, um so I think this is actually a good ground, because last week, catherine- and I were talking about some of the stuff that we're doing with settings in nav and how what we do affects a lot of teams and there's a lot of moving parts and as a group, like you know, from the engineering to research design, settings analysis has been interesting in trying to figure out what to do next or like what move is good. And so this actually primed the whole like proposal for a talk and everything.

C

So yeah, I'm happy to contribute my like angle and yeah.

B

C

Can make a lot of moves like three years worth of uh backlogging issues like how do you make a decision? So I think that's a. I think that was a really good story to tell there.

A

Yeah, I think that'd be a great perspective to write it from and also an audience that we would need to take in account too. It's like people are going to see this inside kid lab and most likely be the first ones to see it and say like wait.

A

Why are you changing this and like what about my thing that you moved, and you didn't tell me about or whatever you know like, we might have forgotten to let somebody know or something um so being able to proactively communicate that in a way that can be consumed, asynchronously would be great, and uh we had talked previously about like a group conversation, but I think since we're executing regardless like it might just be something we can do in a um a recorded video or a blog post or combination of both um anyway early thoughts, but I feel, like you know, we're two and a half ish months away from when that would be if we make 14.

A

Oh so um start thinking about it now, obviously, you know I'll help write whatever needs to be written. If you want me to- and I I know we have, you know, counterparts in um technical writing and and marketing that will review and help with the blog posts and stuff so get all those people brought in as well.

A

um My next topic is really just letting you know I didn't get around to defining those success criteria, but it is related to this. Like blog post, we would probably want to have clear success criteria um written out um so I'll be focusing on that in the coming days, hopefully, by by our next meeting.

B

Yes- and I just wanted to put there that- I think there are probably some different paths for success, considering the different types of changes that we're making, we have a overall success metric that we're starting to gather. We ran the first survey last quarter. I actually still need to report on that and like bring it all to life pending um and then in terms of comparing success for the individual areas. I think it's useful to come up with some hypotheses for them and that could help you determine what you need to measure.

B

So, if you're trying to say hey, this change made it easier to go from groups to projects or something like that. You could compare the existing experience versus the current the proposal or the new experience and just kind of back it up with a study around that. So I would just say if you have some high level hypotheses or just things that you want to test out, we could just list them out and then plan how we can gather the data to support it.

A

Yeah thank you for for those examples and I think that's a good approach.

A

I don't know michael you're, you're typing, I don't know if you want a voice over that or if those are just notes.

C

um Yeah um this week I just created a tracking issue around our left nav.

C

I wouldn't say this is a success metric, but I really wanted to understand when people actually use the sub menu versus just directly accessing an item. So this interaction doesn't really change with our proposed changes, but because it's been it's like something that's in gitlab but like I don't know how to suss it out from like. Just you know, one-on-one usability session that I feel like it needs to be something that's kind of scaled out, because yeah testing this people will be like yeah. I do this, sometimes I I don't do this all the time.

C

So I think is, is a bit too fuzzy to get user testing on, but I love to understand whether people use the sub menu when the menus collapse, because there's a lot of complexity in the way we do it and looking at like um competitors.

C

The menus are a lot simpler in interaction as in like there's no hover. um You know you click on something, then you should see the menu, but I personally I find it's kind of useful to just hover and see things from time to time, because I don't need to click.

C

I can just like hover and see what I need to get to so I always thought it was a um benefit and or like a feature but yeah there can be scenarios where it can be annoying as chad raised um in certain cases when the ui is a little bit close together that the hot spots are close by and thus triggering the.

B

C

Menu unnecessarily.

A

That's a good point also realize how little I use it in collapse state because, like I have a wide monitor right, I don't, I don't think to collapse. It all the time um really good point and I was curious in the code base. Are they the same length presented differently or do we have separate actual like hrefs basic, for whether it's collapsed or expanded? Would we be able to track this easily because I believe the navigation? I don't know if they got to the left navigation but there's a lot of.

A

I was just looking at the dashboard, the dashboard that you created a long time ago, chat about the actions in the top nav. I don't know if they got to the left, nav items.

D

Did we ever even finish that.

A

Well, I was just looking at the raw data that you put into that dashboard. It's not a complete like report or anything, but I was just looking at the raw data.

A

But I don't know that, like I'm just curious, maybe that we don't need to answer right now, but I'm curious, if, like under um I'm, trying to find one. That's not that's ci cd pipelines um when you collapse it and hover and choose hover over ci cd and choose pipelines. Is that literally the same link and we're just changing it with css and javascript?

A

Or are we swapping out divs? Because you know it's easier to keep those separate?

A

I'm not sure it'd be easy to track, but it would be an interesting pattern to to.

C

C

A

C

Have to drop off to drop off my daughter and.

A

C

um I'll I'll check up on your last point a little bit later so bye, everyone.

A

Yep bye thanks uh the last point's.

D

A

Extremely critical anyway, I'm just letting everybody know um the product is um drafting group level okrs for the first time. So I'm working on q2.

B

A

And uh making some suggestions- I don't have any written in the doc yet, but some high level thoughts in there.

B

Do you have any high-level thoughts on what you might be interested in.

A

Yeah, so I mean I'll share what I put in the doc, which was really just meant as a brainstorm between michael roman and I, but obviously we'll want input from everybody. This isn't something we do in isolation, but the guiding themes that we were being asked to to ladder up to are driving a meaningful impact on usability, either sass or security bugs or s1 s2 bugs.

A

Deliver on the monetization of our product by increasing paid email or you.

D

A

Measures other performance indicators related to uh monetization and paid conversion, uh and then capturing new product usage and increasing adoption, mostly through um driving more stages used per user or or per project. So our organization, um my thoughts are- and I've shared this before with you, catherine. I don't think we can actually uh impact and measure change in sus in a single quarter with any kind of confidence like a causal relationship. So I'm not going to try and you know- create a kr around like moving sus by two percent or something like that.

A

I don't think we have a area of the product that would be able to be proven to do that, and maybe there are others who can. But I I that's not something I would like to hold ourselves to.

A

um I do think that dog fooding, which is an example of of something that could drive an impact on usability, is an opportunity for us, especially when it comes to these underlying editing components that we're building around consolidating our code and non-code editing experiences around two editors, and if we can dog food them ourselves then drive consistency in the experience that improves usability and potentially satisfaction. And overall you know sus eventually, maybe so. That is something I'm thinking of. uh We don't have any paid features except group wiki, and I don't think, there's any reason.

A

We should prioritize driving adoption of group wiki beyond organic and and what we're doing already to just clean out the back end. So I don't think we'll ladder up to that kr and um I think, if anything we might find. This is probably something that michael would want to weigh in on, but just find like a bucket of um usability, related issues, s1 s2 kind of things and create an epic that we just burned through.

A

Just like a paper cut, ux debt kind of effort, try and get a dozen or 15 of them done in the quarter and just polish it up. So um I think between dog fooding, and I mean we don't need to have a huge amount of product cars, but I think between dog fooding and um polishing, the ux we could.

A

We could have our answer, but I'll think about this, some more um in regards to what we have planned for the wiki and just thinking that this is like. You know this is q2. So what could we potentially be accomplishing by the end of q1 that we're building on for q2.

B

Yeah, I think these sound good so with the usability issues, it could be anything in editor, so it could be one of the editors wiki, even settings and nav.

A

Yeah, I mean really any of our categories that we might end up picking specific categories to focus on for the quarter I mean, I would say we are not inclined to build a backlog like that on live preview or.

C

Snippets because they're fairly.

A

I mean snippets is, is maybe more strategically important, but um our overall investment as a group is just not focused on that, and um but I I could see us doing like you know, let's do 10 web ide, 10, wiki and 10 settings in nav and then try and fit those all into a quarter um or try and fit a percentage of those in a quarter. 30 issues already that's like.

B

A

B

Yeah yeah, I think I think wiki could be a really good candidate, because because of the fact that investment in it has kind of shifted over time, there are definitely some issues. I remember about wiki in terms of improving usability.

C

B

Settings to a lot of opportunity for settings even in the categorization of them. I think that might be one of the one of the biggest points, alongside search and contextual access.

A

Yeah yeah, I agree uh wiki wiki. I I need to think through this some more but by nature of introducing the rich text editor and a a more visual way to edit markdown files. We are tackling like a large chunk of the backlogs that I've seen. I mean, there's also plenty of other work to do but um of work. That is achievable that we are like equipped to handle over the next couple quarters.

A

I think that's, probably our primary opportunity so finding ways to polish the editing, experience even further and bring those elements into the rich text editor as extensions or something like that that could all I could all be considered for um for kr's on on.

C

A

B

A

We're supposed to have the draft by friday, I'm woefully behind on this. It's not a hard deadline, but we want to draft so that we can share with leadership and then they can start forming their okay. Ours, like we're just we're a little early on this process.

B

Is there anything that would help you in terms of maybe the usability one.

A

uh I'll ping, you uh just your thoughts, already have been helpful, I'll pay. If there's anything specifically, I need, but um I think that at this point we're we're looking, uh I think what is being asked is, for you know, higher level draft, and then we would get more specific if you know as they get refined. So probably you know in a few weeks we might be looking for some more specific details.

B

A

Great uh chad, as our seoul west coast of the us engineering, member and frequent visitor, do you have anything to bring up from the engineering side or any thoughts on this.

D

No, I just have one question: jtbd. Is that job to be done.

A

Job to be done, yes, sorry for the acronym. It's a very productive, speak um yeah. What we're hired to do.

B

A

The user is hiring us to do.

B

D

B

In simple terms, task task, orientator, that sort of thing.

C

D

Yeah, it seems like a lot of stuff is going on. I know that the one discussion that came up in relation to uh like paul and I being assigned the task to sort of create the scaffolding for the beginning of some of the top nav rewards. We do. There was a discussion around using future flags versus using the the get lab experiment gym. That.

D

Jeremy has been working on and pinged you and michael in the thread, and it seems like y'all, are very metrics driven and are likely to want metrics and to do more than one variant and all the sorts of things that the gitlab experiment, jim supports and future flag doesn't. So it seems kind of like a no-brainer to go down that approach.

D

Thoughts on that.

A

Yeah, I um I'd be more than happy to use that framework because it helps develop it as well. So I think um as a it would be a learning experience for me. um I've never had a feature through them, but.

D

And uh yeah like when paul- and I started talking about that- the thing I kept saying over and over was like you know what we want here is a real feature flagging framework, something like launch darkly. That's got all of the features that you'd want in a future flagging framework and if nothing else, gitlab experiment is a lot closer to that than the built-in feature flag. So why not try it yeah, for example, like.

A

D

We're risk-averse we don't know how many people will get mad if we change this okay. The answer to that is you do an incremental percentage-based rollout, and you know you can't do that with the built-in. You know the future flights like we use but get lab experiment which sort of is like scientist under the covers the ruby library. We have a lot more options to do stuff like that or implement it ourselves. If we don't want to yeah so.

A

B

D

That direction and unless there's any opposition.

A

I don't think there's I don't have any reservations, and I think that one thing, if I could just clarify it, sounded like it wasn't then tied to releases either like we'd, be able to to tweak the experiment on the fly after it's after the codes made it into production.

D

I'm not sure, maybe I'm not sure if it's data driven, I need to find that out and the other one it question that came up was like letting people opt out versus opt-in, and I think the point paul made was letting people opt out is like kind of a slippery slope.

D

You know it would be better.

C

D

Do the incremental rollout and like if the people that are hit with hit with the new one get upset about it, then that's, you know we need to roll back and revisit the decision of rather than adding yet more configuration options for people to opt in or out.

A

Yeah- and I went back and forth to this with roman and probably should have taken a more firm stance, but I think the um the complexity that it adds to to surface another configuration is um probably not necessary for this feature for this effort. um We don't necessarily want people to persist this setting like if we do it right we're just going to roll this out to everybody anyway, so like making them make a decision.

A

That's the thing that gives me the most reservation on on letting people opt out explicitly through the ui is like at some point. We would just say like well, sorry, you know, sorry, you don't care for it, but it is proven to be more usable and we're going this direction right. So we would then be removing something they explicitly like wanted, and that feels a little worse than just saying. Like yeah, you know we hear you there's some bugs we're iterating fast and we're get lab and we're we.

A

You know we're capable of changing things and and we're not afraid to roll things back. um I do think putting it behind a feature flag or uh an experiment where we could say like. Oh actually, you know huge misstep, let's, let's roll this back quickly and make sure that if there's a self-managed install that got added to the experiment, there's a path for them to like, like the idea of a feature flag where we could default to on, and they could just go into the console and be like I'll turn.

A

This thing off for now, because it's completely broken and we missed a showstopper bug that was desirable, like that makes sense to me.

D

Right, which you can still do with the experiment framework right.

A

So as long as we have that safety net, I think I think we're good.

D

And to the point of adding you know, configuration values, there's there's a lot of subtle downsides to that, like not just the complexity of yet more settings that we're already trying to deal with the ones we have, but there's also the inconsistency of usage like it behaves differently.

B

D

Person and another and the documentation is harder and one person tries to help another person or, like oh yours, looks and works completely different than mine, like consistency, yeah.

A

I mean yeah, that's a problem with experiments in general, but um but yeah, especially for user configuration. That's.

D

Temporary, like yeah permanently, you have these different cohorts of users that the app behaves differently and they get confused. Yeah try to help each other out.

A

I will say that the conclusion that I came to with roman was was maybe too risk-averse and over engineering for the purpose of a safety net which was like. We would give it a an opt-out but expose that only to um get lab team members.

A

While we were doing a slow rollout so that we didn't disrupt a workflow or slow people down as they're using gitlab to get their work done and we're still kind of tweaking things. And then I had said I'm not committing to, and we probably shouldn't expose that to anybody else externally. Because of that exact reason. But I think it's probably more more engineering than it's worth to expose that opt out in the ui and instead using the experiment framework to to manage that and then being proactive about communication.

D

Right so there's something to do: let's just dig in and learn more about how the experiment works. I just haven't gotten to it yet cool.

D

I think that's it.

B

Yeah that sounds super interesting. I think any way to test things and be able to opt out or be able to roll back on things would be really really helpful for anything. Navigation related.

D

Yeah and like to the point of you know, we paired on the the rich text editor and testing framework today and we're doing that all in get lab ui anything that it's we can iterate on. You know, without going through the full release cycle and more formal review process and stuff uh without incurring undue risk and but still move faster. That's a win.

A

Absolutely yeah thanks for bringing that up. That's a that's a good point and um I think if it works well and you get that framework set up pretty well, we could consider doing that for the left. Nav changes as well, uh depending on how, if that structure works,.

D

Yeah, if we learn it and become confident with it like that's, why jeremy added it it's like, it was just a hole like you. You want to have a feature-rich feature, flagging an experimentation framework, and it's good that we do now. We should try to use it.

A

Definitely well um yeah my full support. I've got a um drop for an interview, but uh thanks for your.

A

um I'll drop something in the agenda just to say that we talked about feature flags and experiments.

D

A

All right thanks, everyone.

B

A