GitLab Growth, 13 May 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Merge Request With No Pipeline Experiment Update

Description

Tim & Jensen review the current status of the MR with no pipeline experiment and share ideas on how to make it better. Feel free to contribute here -- https://gitlab.com/gitlab-org/growth/product/-/issues/1559

A

Cool all right, so everyone I'm Tim growth, PM, wanted to walk Jenson through the mr with no pipeline experiment just give a quick status update of where we are what we've been up to Justin. Let me share my screen. Let me know when you can see it.

A

Yeah looks good good awesome, alright, so a couple of things the mr with no pipeline experiment has been live now since march 30th and we've been slowly kind of ramping it up right. Now, it's it's sitting at about thirty percent of users uncom only so that's who we're showing it to, but to give you a a higher level overview of what it actually is.

A

Is we wanted to identify users on the website who have created an mr that have not created a pipeline and see if we could actually get them to move through a workflow and set up that pipeline and really what we wanted to do is a few things really see if we could get them to a not only set up a pipeline which would lead to see I'm minutes usage and potentially an add-on sale. But the second thing really would be to expand or increase stage adoption from create to verify.

A

So, as of today really were reset, is we've got about five and a half percent of users who are actually shown the first nudge that are engaging and then, after they engage about almost 10 percent nine and a half percent of them are actually completing the workflow and creating a funnel creating a pipeline. The success criteria that we had for this was 15 percent of users that actually engaged were to complete the flow, so we're going to continue to kind of iterate on the on the copy and the designs.

A

Look at that set up right here. Really what I wanted to do was maybe just get some I some feedback from you at a high level. We've got the dashboard, we've got a clickable prototype and then I've laid out this issue to to just gather feedback around. You know how this work flow is actually going.

A

You want to dive into the dashboard first sure.

A

Cool, so what we're looking at here is basically all of the instrumentation and the events that we had to to create in order to collect information on the actual and experiment. So it's one thing to to build a workflow. It's another thing to actually capture the information on it at the top level. Here, what we're doing is just capturing all the rock outs and then down here. What we've done is we've split it into two different views. The one on the left here are the actual total events.

A

So what we're looking at is the event counts and then, on the right hand, side we're looking at the unique users. One thing to just be really clear about is the events. Multiple events can be shown to a user and when we say unique user, we're talking about unique user counts, we're not talking about user ID, so it really is just account.

B

And lets users not name spaces.

A

That's correct, it is it's actual users, so so.

B

Only you count when you count the like: they have a merge request with no pipeline, that's at the user level or the namespace.

A

Level, it's at the user level.

B

A

Like a user hasn't.

B

Created a pipeline well.

A

There's no pipeline available right got it. Okay,.

A

So, just to walk you through kind of the funnel here we've got the total user accounts on unique users. I think the event count is interesting, but I'm more curious about more interested in the unique users, and this is how we're measuring our success. But we've shown this to about 35,000 users. Almost 2,000 of them have engaged so about five and a half percent, and then once they actually engage we're displaying the second nudge displaying the third nudge and then really the the completion point for us is when they engage nudge number three.

A

This is the actual commit so they're creating this CIE mo file and when you walk it all the way over after they've been displayed or they've engaged with the first nudge, almost 10% of them are converting.

A

So a first glance. It's you know. Five and a half percent of the users are actually you know clicking in which I think is a pretty good success and then of that about ten percent of them are are completing the funnel.

A

The one of the cool things that I feel like is really important to call out here is Derek from our data team was able to build this out. So we could see what happens at a daily level. We've got the counts here and then we've actually got. We have the conversion funnel broken down by day. So once we start to iterate on this, we're going to be able to see you know what happens quickly. So if any new change is updated, we'll be able to see how it impacts the actual funnel.

B

A

Then I think the the other thing that I wanted to show. You was just actually the clickable prototype, so this is what we're showing users today by looking at the data. We know that only 5% of people are actually saying. Okay. This is interesting. Show me how so there might be some some opportunities here once we show them how we pop another message that says: hey choose a template. I had a request and actually collect this, but it was a little too heavy for our team to pay instrument and then B.

A

If we were to create a table that just collected this information, it was only going to be used by me, so we decided to not build it if the experiments a success and we want to keep it around. It's probably something we'll take a look at, but this is what the second nudge looks like once they click this and they actually go into the the next step. This is the third nudge.

A

So this is what we're considering success is when they click on this commit jizz and then, after this we're popping the the celebratory kind of message that says, congratulations and we're offering up some other content for them to go and take a look at so it's kind of what we got.

A

I'd love your feedback on the designs, the workflow. You know copy edits, design edits whatever. Whatever ideas you have because I know, you've you've built these things before in the past, so it'd be great to to get some feedback for you doesn't have to be right now, but yeah.

B

So the one thing that sort of comes to mind that I think would be a good iteration on this can I see the dashboard one more time. I want to see the drop off between one and two. Yes, so there's a pretty big drop off between one and two mm-hmm and my theory would be my hypothesis would be that's.

B

Developers a lot of times don't want to write code directly in that IDE.

B

So you might want to give them a choice of writing the code directly in their IDE or copying it from from, like a template and pasting it into their own code, editor on their local machine and then committing that code committing that file and count that, as like a as a success, interesting.

A

So basically, right here.

B

Somewhere right there, like maybe but before this or something where you would say like you know, write it directly or or.

A

Or just that copy clipboard kind of icon right that they could call. You know, that's actually a really good idea. I would.

B

Make it a little bit more active like have them choose whether or not like to write it right now or you know copy it for your. Your own code editor copy, a template for your code editor or something like that. I don't know, but that's like there's like a lot of sort of ownership over what code editors they use and how that code is.

B

You know how pretty it is. Just you know. Where are you write code is a big deal, yeah.

A

I think that speaks kind of to the mindset of the developer right and I hadn't really thought about it. That way. Yeah.

B

A

Tools, I think that this the.

B

Ide is great it's great, mostly for like quick changes to to your code, like if you, if you've committed something and you like, haven't :, that's causing failures or something you can just change it real, quick and then merge it right, and it provides like a really quick way to to make fixes. Rather than doing it. You know in your own code editor and then and then get pushing it, but that I would just say, like that's probably why you're getting that big drop off between 1 and 2 as people don't want to.

B

They don't want to commit a full file from the IDE.

A

That's good feedback.

A

The other thing that stood out to me on this was that you know five and a half percent, I I think I think we could do better here and if we obviously, if we increase the conversion at the top of the funnel, it's gonna be easier for us to convert at the bottom of the funnel.

A

There's anything stand out to you when we look at this at the beginning,.

A

B

Know the one thing that stands out to me is like when I was not super familiar with gitlab I, don't really know what a pipeline is like, though, that nomenclature doesn't introduce me to like it doesn't fully explain what that feature is so I think there it's likely that a lot of people are like I, don't know what this is.

B

I got got time to learn about this right now, so maybe like a little bit more visual explanation of like what a pipeline is and what they would be adding would be useful because you know like if you go to learn about CI, it's pretty complicated right, like you have to there's pipelines, there's jobs, yeah, there's triggers there's runners, there's like there's all this different stuff that you have to sort of grok before you know how to use the feature and I think a lot of times. People are like. Oh, this is too complicated.

B

A

An interesting point.

A

Let me think about that. So.

B

Maybe speak a little bit more to the value, also like, as opposed to the feature like you know, deploy your code with a click of a button or or whatever you know whatever. That might be speak a little bit more to the value and visualize that a little bit more here, I.

A

Think if there was any other areas that kind of jumped out at me, it feels like it they're pretty good. Preliminary results just need to know.

A

If we could get this to seven I bet you this would come up almost naturally to 15%. We hit our success criteria. I want to go through a couple of different iterations with the team before we just continue to increase the exposure so trying to be mindful of that.

A