GitLab Release Stage, 5 Nov 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Feature Flags Office Hours 5th Nov 2020

Description

Feature Flags Office Hours 5th Nov 2020

Discussion about Protected Strategies - https://gitlab.com/gitlab-org/gitlab/-/issues/230614
and a demo of A/B testing POC

A

A

Hello how's it going.

B

I'm all right: how about you.

A

A

C

I'm there or it finally found it.

A

You got a haircut.

C

Thank you for noticing. Yes, it is a bit shorter again. Last.

A

Meeting but it seemed inappropriate to say something.

C

What you said it was necessary last meeting.

A

No, I I noticed it last week meeting, but it seemed inappropriate to mention it in that.

C

Other one no worries that that's a good thing like I'm always like. Should I go to the hairdresser I'll, do it next time and then you know you keep extending it, and that was like all right. I'm gonna go.

A

Okay, well, I see we have people that joined. um So, let's start uh we're talking about feature flag office hours. um We have some notes in the agenda uh shiny. You have the first point so.

D

Okay, um please wait a bit, uh I'm just.

E

A

A

Okay, so let's you want to skip to the next point until you get.

D

Oh yeah, please.

A

Okay, so um we've been talking uh asynchronously a lot about this next issue about protected um feature flags, slash protected strategies, and I thought it was worthwhile to discuss us this synchronously.

A

um So, let's open up the issue and I will share my screen.

A

um Okay, so again, let's start from the basic. The idea is what what are we trying to solve? And the basic use case that we're trying to solve is that we want developers to have full control of future flags during development and and any testing staging slash, dev environment. They can do whatever they want, but once we hit production, we're not we don't want. The developers is not the right word, but people without the right permissions to uh turn on flags in a projected environment.

A

So the way that we have protected environments today in gitlab is that you have the option to to specify specific users that can deploy to these environments, and I want to use that same permission model in order to allow people to toggle feature flags on and off, in production, protected environments. Sorry, I may interchange production and protected environments, but I'm I mean the same thing: um okay, so the idea here that we started talking and then there was a lot of uh um spanning of and spinning off of ideas.

A

um The there's two concepts that we need to talk about. One of them is protective strategies and the other one is protected. Flags feature flags. um So the way that- uh and I do want us to think about the final solution and then start thinking about iteration, and what can we accomplish uh in small increments?

A

So um I'm going to start with with solutions unless dimitri you, you stop me and you want to ask something before we solutionize.

C

um Yeah, so um this is a good question or like personally, I have one main question is.

C

It's it's mostly about feasibility like there are two things on the table right there's: either we go for a subsection of a future flag being protected, meaning the strategy, but that would still mean that the feature flag itself can be turned on and off for those environments if they are protected or not, or the entire feature flag is going to be considered protected, meaning that if any of the environments is protected, it means the complete flag, will protect it and nobody can turn it unless they have enough permissions for all included environments.

C

My main question here would be: would there be a difference in feasibility engineering-wise going for feature flag as a level of protection from the get-go or going for the uh protected strategy? Because from a ux perspective speaking, I think it makes a lot more sense to go for the feature flag level rather than the protected strategy level.

A

Before any engineer, answers that what I would like to just mention the ideal user experience would be that if I have permissions for my environments, if I toggle them, that would only take effect to the environments that I um that I can. This will make the observability really really confusing, because what does it? What is it was it? What does it mean if a feature flag is on for a specific environment? Yes or no, it's going to be a little bit more complicated in terms of display, but the best user experience would be.

A

If I have permissions, let me do whatever I want and then, if I don't have permissions just don't, let me touch those specific environments that that would be the ideal experience. But, um as you mentioned immediately, I think the easiest way to start and the more the most iterative is just to say for this first iteration.

A

If there is any environment that is protected, the entire flag is protected, but now I'm gonna, let that and I'm gonna stop talking about others.

D

uh I'm almost thinking that, um instead of locking entire future flag, it's better to just lock strategy level like uh angelo made a really nice comment. um Bottom there, like yeah, um it's it'll be much simpler and easier and potentially doesn't cause some.

D

You know, locking out people who actually have our permission to change the future flag state for their review up or something, but uh by locking into a future vlog there's a possibility that uh kicked them out so uh yeah. I basically think that uh our strategy level is a good first iteration.

C

So, um just to go into that, I'm, like I hear easier and simpler um and no locking out, I think, locking out is- is super valid. um On the other hand, how do you look at the concern that this enables people with less permissions, so people who have no permission for certain protected environments to still be able to control the feature flag? Essentially, because that is the main thing we're omitting here.

D

As long as the people doesn't create a strategy which includes protected environment, it doesn't affect the protected environment, so uh should be fine. So the point is that we should also you know, create delete, update. uh These three actions should be prohibited according to the protected environment,.

C

um Sorry shana, I I'm trying to follow you, but, like your answer to my question, was not clear to me and I'm finding it hard to follow up after you're coming after could you could you go into detail on on my question on how, like my concern, raised around a future flex, still being able to con to be controlled by people with not enough permissions for the protected environments included into the environment? Scope for that like configured for that feature, flag.

D

D

The people who don't have a permission to access protected environment I'll go into future flight page and then create a new strategy.

D

So basically, the the action required is the separate strategy power environment like currently multiple environments, can belong to that. The same strategy right, um but this model wouldn't really work uh with this feature. So it's more like a uh define, different strategy. They create, even if it's duplicate, just create a two strategy and then uh attach the you know: um unprotected environments and the unprotected environments.

C

So you mean like, if there's an existing strategy that involves a protected environment- and you know a certain um strategy is involved, say: user, ids or something like that. Then, if the user doesn't have enough permissions to edit that, then they will create or duplicate that strategy within the same feature flag and then apply it to like the environment. They care about.

D

Yeah so, for example, like an operator who has a permission, everything uh created a strategy with review, app, slash, asterisk and then production environment which is protected and then um developers came after and they realized that they don't have permission to change the review app. Then probably, they need to ask the operator to first remove the environment asterisk from the protected strategy and then create a new entry, create the duplicate strategy and then assign review review environments.

D

uh I could be wrong and there are many ways to maybe I'm missing something.

C

I I have one remaining question, because I think what you're saying makes sense to to how I'm thinking I wasn't thinking about. You know having the exact same strategy twice on a single feature flag, um which apparently can be done. So that is something I was unaware of, um though, with different environment scope there. Still. The main question still stays is that if this is the case, so a future flag includes a strategy for unprotected environment and a strategy for a protected environment.

C

Then the people who can basically only access the unprotected environment strategy can still turn on and off the flag right, which.

D

Will happen top level top level flag? You mean the top level shouldn't be allowed as long as it touches.

D

One of the protected environments.

C

Exactly that's what is where I was aligned on as well.

A

I would like to just stay on this point at the moment. The only place that you have this toggle is on the top of the there's two places where you can toggle all feature flags. You can toggle them on on the edit page and you can tackle them on the feature flag list.

A

Both of them today um actually will enable the flag for whatever you said. So that includes protected and unprotected. Does this mean that we are going to change the toggle to be on a strategy level and not on the top level.

B

Originally, as part of the redesign uh we were going to have toggles on both where the global toggle would be like a kill, switch and shut off the whole flag, and then each strategy would have its own toggle as well, and it got d scoped at some point, and I don't think we ever came back around to rescheduling that. So if that makes this easier, perhaps that's something we should consider.

C

B

C

Would actually solve the thing that I was thinking about if we don't have an ultimate kill switch, but we have a kill, switch per environment scope.

C

To be honest, like that sounds something that could solve this thing and make protected strategies. uh You know valid a valid solution, in my opinion,.

A

uh I agree, I think that um so for the first step, we would need to create this concept uh of protected um strategy.

A

Then we would need to add this option, at least for the ui, because I think in the api it's easier, but uh for the ui to have a toggle per strategy.

A

The top menu needs to turn into a kill switch now each one of this is another iteration um and the, and then we need to figure out on the feature flag list view um how we're going to present this, because now, potentially you don't have an on or off it's on or off per strategy. So we would at the moment it kind of displays the strategy in this badge-like view, but now we would probably need to add the information for each one if it's on or off, in addition to what it displays today,.

B

Originally, the original feature: flag design had a blue badge if the scope was on and a gray badge of the scope was off. um Are we looking to make it editable from that page as well? Or do we just need to display its status.

A

From day one definitely not like, we could force people to go in to edit it specifically for strategy, but I do think, um like I don't know, if you've seen the metrics that you implemented not long ago. The majority of the people are toggling off and on from the list view, and now from the edit.

B

That makes sense it's a lot easier.

C

It's easier and the same at the moment right, but we're creating a difference. If we're gonna go this protected strategy direction and it's going to have different functionality.

A

Well, people are still lazy and don't want to click, so in that sense,.

C

We can still surface it eventually somewhere on the amulets page. If we don't make it to complicate it, and I don't know what the usage of feature like strategies per feature flag is going to be. You know if there's going to be future flags with 20 different strategies applied to one of them. um I think that's going to make things more difficult, but it's two or three.

B

We could make the badges somehow interactable so that clicking on them could toggle a strategy. It was kind of a first iteration.

A

Yeah, or at least, if you click on it, and it asks you you want to turn it on instead of just another page.

B

A

B

So does this mean the idea is to get rid of the the top level switch completely.

A

I think if we go to the protected route, there's it's inevitable, um I think also, regardless to whatever category we're working on now we're talking. Specifically, you need to be as granular as possible and let the users select everything as granular as possible so for environment strategy for, like, I think, the smaller, the better.

C

Yeah, the top level only makes sense from a permissions perspective if it like it can only be toggled on and off if the user has permissions to all environments and all strategies involved in that feature, flag right.

C

This is this was way simpler than in the issue. I saw so many comments as like this is way better. I think this is a sound direction.

A

Well, it also is because we talked about it so much in the issue that we came more prepared with uh what we need to do so this was definitely very productive.

A

um I think we have a plan a thumbs up forever from everyone.

A

Cool um shinya: do you want to talk about your issue.

D

Yeah can I share my screen.

E

D

So I've been thinking about uh a b testing a lot lately like I already posted a lot of videos uh thanks for that, it's really interesting, and then um I was talling around uh unleashed clients and our gitlab the future flight feature and came up with this demo. So I just want to show off this thing.

D

So um what I'm going to show you? The demonstrator, is a b testing, what I believe so the definition of a b testing- maybe you guys already know so I don't want to dig into that. So I'll. Just start off this demo.

E

D

Then first thing is clean up thing and then so uh we are seeing two windows here, the right one. um It's a bit confusing that we are seeing to get love here. uh It's same application technically, but uh just pretend as the right this project. My awesome, ruby app is a project on gitlab.com and then left. One is a customized application, client's application. This is not gitlab. This is also my osrb app okay and then um so our ux engineer one day saw that this application and then.

D

Like I visited this page and then wondering that what if this new page new project button is orange instead of green, so this is where experiments happens, and then here I'm opening the code of this. My awesome application and then the first thing is to do is future variant.

D

This is already supported in english and then so in this at first I'm going to change this button.

D

Currently, the success color is fixed, but we're going to change to success and warning color success is a green color, which is this and the warning is orange. Color you're gonna see it later, and this is done and I'm gonna use this experiment. Experiment, button, color, feature block and the next. uh What we want to know is that, let's say um this user currently and login as cathy and then cathy collected this new project button and then this project, new project creation form, is shown. This is a like a you know. Goal like ux engineer.

D

Wants this happen more, so uh we want to record the event in this page that if a user landed on this page, we record this event as user visits new project page and basically that's it. The setup for this new, my awesome application is done and up next we going to create a feature flag with variants.

D

We're going to visit this page. This is a project on github.com and then the future flag page and something wrong.

D

Because I don't have.

D

D

Page for clarification, I'm going to leave this first.

D

And I'm going to create a new future flag and then future flag is called experiment. Bottom color and the strategy is whatever is fine and then click create a feature flag. And here we have a feature flag and then we also add a valiant.

D

um But since uh I didn't have the time to extend the form, so uh just pretend there's a like there's a variant form uh in this demo. I'm gonna directly inject the variants, definitions.

D

So here's a variety of definition: there are two variants, one is success. As I said, success is a green color. The warning the second one is warning: warning is the orange color and each variant has a weight. uh Success is 30 percent of users. uh The warning 70 percent users gonna see it, and then uh you cannot see the variant setup, but it's already injected internally and at the next. Let's check that the variant is working uh here. I'm logging in as cassie.

D

Oops, oh no! ah This must be incognito.

D

Signing as kelsey secret, password and login, I don't know see that this thing new project button is orange, color and then, let's uh you know pretend as a cassie see so this button. It's very interesting, it's catching attention, so she clicks this button, which is good thing from ux perspective.

D

Add a next sign out: let's login as john.

D

John also oops oops.

D

Yeah see this orange button, uh this is a warning color and then john also uh thought that this button is interesting, so he also landed on this page and and at the next login as a root.

D

D

Everybody is seeing this new project. This is this green button, because uh john and the kathy is belong to this weight 70 because their user id is much later and whereas root id is very re. So it's he's included in the weight circuit, so he's saying sergi and the roots thought that ah rainbow dan I hit that. So he didn't click.

D

And actually um like, I acted as three users: three different users. Everything is recorded, everything uh the like which user so, which bottom color, and also which user landed on the uh new project creation form everything is recorded and sent to the uh server side, this project on github.com and then we're going to see it from there from now.

D

Is it going to visit a b testing page currently, nothing see because no experiment is defined, so we're going to define experiments. uh Here's our experiment definition at first name. This is our experiment. Name, project creation, bottom color, whatever name is fine, and here is a base. Event basement means that uh the like the weird branch happening, uh which is this bottom like where kathy saw orange burden root, saw uh green button thing is a uh that's a called a base event and then next uh conversion event.

D

Commercial event means that um their goal of this experiment, which means landing on the x, uh create new project form right. So uh uh this actually defines that uh this event, name which we injected here, uh is a goal and we're gonna inject. This experiment definition.

D

And now we are seeing that the result of the experiment at first we see a project explainer title and what's interesting, that there are two valiants happening. One is success uh right, which is green bottom color and the other one is warning orange bottom color and success. uh Unique visitors is shown as one because only root visited there.

D

The roots saw that green button and the convergence he didn't land on the new project uh creation form so coma join zelo, whereas warning variant, the orange botan uh there there were two unique visitors, cassiano john saw it and then actually landed on the uh you know, creation form. So a combined change is two, which means that um you know warning body and the one in color is more effective uh on this experiment, and it is the conversion rate or changes thing.

D

It says calculated based on the these values um yeah, so I don't want to dig into that. uh Basically, that's it. This is a simple demo about a b testing yeah. Is there any questions.

A

Really cool shania! Thank you. Thank you for putting this together, so how everything you did right now you did through um through the unleash code, which we already have supported. So how far are we to just present this specific demo that you showed so adding this a b testing a new category in the left pane like how difficult would that be.

D

The creating these things- actually, I finished a few hours, so uh maybe one mile strange it requires one milestone, uh but the the pro it most challenging thing is that uh this is not pure unleash. Ruby client.

D

uh We need to add a one api new api to the uh unleash off show so uh unleash our shell already has a matrix api endpoints, but this is not, if uh sufficient, to achieve this av experiment thing, but um we need a more granular advanced thing, which is called event api, so yeah, that's uh that's the most challenging part. Maybe.

A

Very, very cool thanks!

A

Well we're at time. There was another question on the agenda um that I don't think we will have time to go into. um I'm not sure who wrote it. How do we support enterprise mission, critical, app support via gitlab.com for sas customer? Is there anyone that can take that.

E

That sounds like a question that cherry posted in the future flags channel a while ago.

E

Okay, so maybe it's old.

A

No, I think it's new, I don't I'm not sure so, uh jerry, I'm sorry. We didn't get around to your uh question. Let's take this asynchronously.

A

Bye. Thank you all.