GitLab UX Team, 18 Aug 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: UX collaboration on Growth experiments at GitLab

Description

Emily Bauman, the Product Designer on the Growth team met with Nadia Sotnikova, the Product Designer at Verify:Pipeline Authoring to talk about their collaboration on a growth experiment design.

Learn about the process for designing growth experiments, tips for scoping down an experiment and resources to get you started if you want to try growth experiments on your team.

Experiment design resources: https://gitlab.com/gitlab-org/growth/ui-ux/-/issues/102#note_640989577

How the Growth section works: https://about.gitlab.com/handbook/product/growth/#collaboration-process-with-other-product-teams

Example of a UX Growth experiment issue: https://gitlab.com/gitlab-org/growth/product/-/issues/1662/

Growth experiment Figma template: https://www.figma.com/file/e8tj8XvblnhDlix83FUSij/TEMPLATE-Growth-experiment-design?node-id=0%3A1

A

So hi everyone, um my name is emily and I'm a designer on the growth team and it's nadia.

B

I'm nadia uh I'm the product designer at verify, pipeline authoring and um yeah yeah yeah we're gonna chat about uh growth, uh experimentation, so yeah emily feel free to uh jump in and cover what we're going to talk about.

C

Yeah so recently, we kind of worked together on an experiment and thought it would be useful to just have a quick conversation about what it was like. Collaborating on growth, experimentation together. um What the issue was that we actually collaborated on and then other like tips and tricks for designers who want to get into experimentation. What works! What doesn't as well as after this we'll kind of drop in resources um to read up onto if you're still interested.

B

Yeah sounds awesome, so um I'm gonna share my screen and share some issues just to illustrate kind of what we've been working on um so about a year ago. There was this issue to suggest uh a pipeline uh in namar with no pipelines.

B

So when you uh have an mr and you don't have a pipeline configured, we decided that we're going to show a widget that suggests that you set up a pipeline uh and the idea for this particular issue was to have a guided onboarding. So once you click on the cta in that widget, you would be taken to the single file editor, where you set up your pipeline configuration using templates, and there are little popovers that guide you through the process and tell you what to do pointing towards the template.

B

Picker telling you to commit, and things like that, uh so that was prior to us. Releasing the pipeline editor. Currently we have this dedicated pipeline editor. That is um the easiest way for you to get started with cicd, because we provide you a basic template that you can just run out of the box and get an understanding for how gitlab cicd works. You can also visualize your pipeline. You can see what stages you have. What are the jobs that you have?

B

You can also lint it so ensure that all your syntax is correct um and we also provide some basic information around onboarding in the pipeline. Editor sidebar. So the first thing that we did was we redirected the cda to the pipeline editor. So the current experience when you have nmr without a pipeline, you get this widget. You click to try out github pipelines and you're taken to the pipeline editor where the sidebar is open by default, where you can see some basic onboarding steps.

B

We also tell you that you can just commit the file uh where you're gonna see the pipeline status and then give you some tips and some links to documentation some helpful resources to kind of move on from there. um But given, uh given that this experiment around guided onboarding was successful, we thought that hey.

B

Maybe we can run an experiment where, instead of providing onboarding information in the sidebar, we could use stopovers to guide you through the ui to just get you to quickly commit the template, which would then um allow you to see how the pipeline runs. So, that's kind of when you start realizing the value of ci cd, because just looking at the template, it can be a bit overwhelming.

B

It's not really helpful and it's difficult to see from just looking at code when you're entirely new to ci cd, what it does so um we decided to collaborate with the growth team on this and it was first time for me working on uh a growth experiment so closely so emily, and I collaborated before on some issues where she took the lead on design.

B

But this this was the first time for me actually figuring out the process figuring out what steps we need to take to actually define the experiment and design for it in a way that it's like an effective small experiment. So um the things that um I so I learned a ton of new things generally about the process. I knew nothing about experimentation before that. I haven't really had an opportunity to run any a b tests or anything like that in the past.

B

So that was very interesting and for me I started with basically just setting a hypothesis for the experiment. I knew that I wanted to um to use popovers in the pipeline editor to guide the user towards committing the file, because it's the only thing that you need to do to get started. Really. You only need to commit the template that is already filled in the pipeline editor.

B

So I took a stab at it and defined a hypothesis here, and the next step was to actually decide how we're going to measure success.

B

So then we had a chat around what metrics we're going to use um to to define whether the experiment is successful or not, and initially I proposed that we use the same metric, that we've been tracking at pipeline authoring and it's the percentage of first green pipelines.

B

But then we decided that uh we need to focus more on long-term ci cd adoption, and uh I think uh it was kevin initially who proposed that. But uh we agreed that we want to track the percentage of people who click on the cta and create at least two pipelines, so that would give us a much better understanding of long-term safety adoption so, instead of just tracking who runs this basic template that doesn't really do anything. So it's not really a functional pipeline right because it only runs some uh hello world type of commands.

B

So you can see how the pipeline works, but it doesn't really give value to your project yet, uh but if we can get someone to create two pipelines, it means they actually came back and made some changes to their pipeline and perhaps started adding some tests or build jobs and whatnot whatever they need for their project.

B

So I actually used um a figma template in the growth. Folder is called a folder growth project on sigma and yeah. It was really easy to use it, um so I created a simple flow for the experiment where um we're just going to show one paper, so we decided that okay, it needs to be very simple. It has to be a very small experiment.

B

We need to reduce the amount of changes that we're introducing into the current design. So what we're going to do when you click on the ct in the widget we're going to keep the sidebar closed.

B

So we don't want the sidebar instructions competing with the power and we're going to show only one prompt explaining that this file creates a simple test pipeline and you can commit it to see how it works and clicking on the cta in the popover would auto-scroll to the commit form where you can commit and run your pipeline, and you would be able to see your pipeline running.

B

So that seems like a fairly small change. I had some concerns that it's like very, very minimalist. You know we're not really providing as much guidance as we do in the in the sidebar, but at the same time I kind of feel like that's the point we want to see. If um guiding the user to take the action right away will be more effective to show them the value, rather than trying to provide them all of the resources that they might need at like a further step in their onboarding journey um yeah.

B

So right now we have this proposal and um I don't know emily. Maybe you can talk more about what the next steps would be and generally around some tips and tricks around experimentation.

C

Yeah for sure, so I want to preface this by saying this is like a great kind of like first collaboration, I like with experimentation um outside the growth team and like some of the tips and tricks I've learned because I just joined, get lab six months ago, so I onboarded into growth. I kind of learned really quick. What worked? What didn't work with these experiments- and I think the first thing I realized is: you can always cut down your first idea.

C

So a good example of this is I'll actually share my screen to one of my very first growth experiments that I decided to do.

C

So this was a sassed walkthrough. It was kind of an experiment to walk users through sas, and my first idea was just building out this entire sas to walk through lots of popovers as they went through the entire flow.

C

um When I went to kind of sync this up with the team they're like hey, this is pretty big for an experiment. Is there any way we can cut it down um and I think it was phil. The engineering manager on the team actually mentioned, like hey, you're kind of looking at different ways of entering into the sas workflow.

C

What? If, for a first experiment, we just test out entry points, so we put like a banner entry point and a pop-over entry point and then just see how many people just click this to get in how many people are actually interested in sassed in general and then, if it works out. If it's a kind of works, people are clicking on this more than, if we don't show it, um we can go with whatever entry point wins and then move into like an iterative version of walking them through sas in general.

C

So my main tip was pretty much. The first thing that you come up with. You can always start cutting it down into smaller little experiments, iterate on it as you get more information back and just build it up to be something greater than you would if you tried to build it all out at the beginning, um and what I also learned is like not every experiment will be a success too. There'll be some great ideas that are surprising and don't work out.

C

The way you want them to work out and um you can just go from there potentially make up some user interviews to figure out what happened or take it as a learning for the next experiment. You do.

A

Let's just stop sharing my screen.

C

And yeah there's always um this other thing with growth. Experiments too, is when is it okay to like kind of cut corners when is it okay to uh kind of mix with get lab guidelines or implementation quality? And this is a really hard one for me as well? I think it's really dependent on the experiment.

C

You're doing like are some experiments going to make the overall user experience worse or is it going to affect something throughout the flow further on, and I think it's just something that designers have to discuss with their product managers with the development team figure out what makes sense for that experiment, and if it is something you think is going to impact other user experience just make sure to monitor that get the information and if it is having a bad or um users, are dropping off from something because of the experiment, change it up, turn it off and see what you can do to rework those so yeah, I'm I don't know nadia.

C

Did you encounter any of this when doing your first experiment.

B

So from the thing that stood out for me was I I felt like there was way more freedom in terms of what I could test out compared to like a feature design. I feel like there was way less pressure to get things perfectly right, uh the first time, and especially, if you're able to really cut it down and have a very small change.

B

Then it's it's relatively low commitment right, because if it, if uh implementing that, will not cost a lot of engineering time, then um it's quite easy to just release it and see if it works out or not. And then when you get some data, then you can. It gives you information for like what to test next, because the way I understand it, it's not like the end of the story where you compare uh the test variant to the current design and you just make it the default.

B

Maybe the next step will be for you to run another experiment, comparing a new version with the successful successful version from the first experiment.

C

Yeah and it's funny because it is, you have a lot more flexibility and you can kind of go off. I think there's one experiment we're doing where we're testing a new component of popovers and it doesn't exist in pajamas yet um and we're using it to test that component as well as the experiment.

C

So there's like so many things you can do um with them, which I think it's a lot of fun um and I think it's something a lot designers should try at least once um because it is a different way of working in a.

B

Sense, yeah yeah. So um if, if a product designer from one of the stage groups wants to try it out and wants to propose an idea to their product manager like how would they go about it, how do you plan collaboration uh with other stage groups, because I imagine you know you, you have your hands full, probably so, there's probably some process for like fitting it into your schedule.

C

Yeah- and I think, we've just recently started working more across stage and with more designers, and I think the best way to kind of see if it's a good like if we have time is just start. The conversation in an issue like write up an experimentation idea write it in an issue, tag someone from the growth team and we can kind of go from there to see like. Is it something any of the growth designers have capacity to collaborate on? Do?

C

um Is this something that the development teams can kind of collaborate on and see who takes it over? I think that's like the best first step and in terms of like coming up with an experiment: the growth designers, so jackie, our manager, mate, kevin and I are all putting together like a growth resource.

C

um It's not done yet we're hoping it's going to be like a handbook page for learning and development and I'll be sure to like tag that somewhere um if anyone's interested but yeah until that's kind of out just sync up with growth designers even have like a coffee chat with one of us. If you're interested in I'm always open to kind of chatting about this stuff, and as I learn more about gitlab just learning more about this, it helps me as well so yeah.

C

That's that's, probably my best uh advice and, as you experiment, more it'll come more naturally um so yeah the best thing is to kind of jump in and do it so yeah.

B

Yeah sounds great yeah and we can drop some links to some existing resources and handbook pages under that video as well, but yeah. I don't really have any any other questions. I'm just excited to see how that experiment turns out, because I think it's gonna um provide some more learnings around the experience as well, because designing it is only half of the journey yeah for sure.

C

Yeah, I think I I don't have anything else to add either just it was a fantastic like crosstag collaboration, effort on it and I'm looking forward to like more of these popping up in the future. So.

B

Awesome all right, thank you so much emily yeah, okay,.