GitLab Key Review, 23 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: [REC] Key Meeting - Product (Public Stream)

Description

First Customer win (Feb 2021) With GitLab Ultimate for IBM Cloud Paks

A

Hello and welcome everyone to the product. Key meeting for march um started out with his family on a well-deserved spring break so hila and I are hosting today's team meeting with the help and rest of the team uh brian. You.

B

Have the first question thanks for jose noop uh and thanks for the presentation, uh very detailed and well done so uh if you can pass that on the to the uh the team. That'd be great, I knew that the team had a number of key hires in budget that they wanted to make. I was just wondering if you could give an update on the hiring status to fill these roles.

A

Yeah, so we have uh three, yes one, so maybe david can you cover the ones in your team and then I'll go over there and he like can cover the ones in growth and then I could go over the rest of them.

C

Sure so, on the director of product for sec, we have a very, very good healthy pipeline now via recruiting, and those interviews are kicking off this week.

C

um I don't have an ets when I think we'll be to a point where we close yet, but um everyone in the team helped by doing a source-a-thon we've gotten some really good candidates that are now engaged on the uh group manager positions that are highlighted. I I think, we'll be extending an offer for create this week. I'm hoping we'll be able to do the same uh for manage next week and then that just kind of leads plan to be filled um outside of that one.

C

I don't think we've called out, but I think, is also critical to our success- is our compliance pm um that we also just searched on last week and now have I think it's a half dozen uh candidates in the pipeline who look very promising.

B

B

Any any changes um in the recruiting process uh with just the pandemic and more companies. uh You know hiring remote people that you've seen.

C

I I don't think I've seen much of a change other than I think we're gonna run into a problem. We haven't before some of the companies who are first time, remote, aren't factoring in vocation factor.

C

So we I had this happen once since I've been here where a candidate was being offered, they are selling remote minnesota, you know, and so some of the companies who are addressing that are still kind of taking that approach of like I'll just hire you at what I would pay you if you were here and that could lead us into some awkward situations like candidates we think are well suited.

C

I think we can work around it by maybe not matching, but our stock offering is exactly the equity is more usually more competitive, and I think you can try to leverage that to kind of balance it.

B

Great thank you for the the input and the context craig. You have number three.

D

Yeah awesome, so, first of all I I have to say I love the experiment readouts in the deck it it took me more time than I had had planned to go through them, but there's just so much information there. So thank you for that. That's it's really really helpful um and please continue that. I actually think this is a great forum to discuss, wins and also learnings right. If experiments don't work out right, I think that's it's a really great forum to do that.

D

um Two things I just wanted to double click on uh one's the invite funnel and the second is the uh the kind of the the lift and trials and but let's start with the first one, which is the invite funnel. um It looks like a really interesting opportunity.

D

um I'd love to- and I looked through the charts that you had I'm just trying to get a clean visualization of how you think about that funnel, and then I saw there's a lot, there's a lot of opportunity to try to optimize it. So I just love to see like what are the the different drop-off points and how you think about the invite funnel and do we have a clean data. Visualization of that anywhere.

E

Yeah, I I can take this and then I can add my comment afterwards. um So the the clean database uh visualization is a little bit challenging in the sense that we don't have user id tracking on the front end. So what we can look at is essentially um we can have some charts that can show us the invite sent to accepted rate for an individual invite, and then we can create some separate reporting for front-end events. Looking at what does that?

E

The funnel look like from viewing particular pages without knowing who the user is, if that makes sense, um so we're actively- and we see this as a huge area of opportunity- and we have a- I can share the epic here in a second, but we have an epic related to hard email confirmation where multiple teams on growth and manage access are working on. Improving that overall experience, which I'm pretty confident will have a nice impact in terms of the user accepted rate.

F

Yeah correct: to give you an example: currently uh anything we don't have something called the magic link, meaning if you, if you try to sign up, you confirm email, you come back. You need to fill in all the information all over again, and that also applies to the invitation flow. Imagine that so like as part of the effort we will be implementing looking to implement that that will help both the overall sign up success, as well as the imitation uh sign of success.

D

And the team feels like you, have good data on those drop-off points, so you can, as you run these things you you know like. Okay, we had you know one percent conversion and now two percent because of these changes like you feel good, that you can measure those drop-off points, and I mean.

E

Yes, so for for those individual experiments we can- and this kind of gets into your next question. um We can understand our lift for like front end experiments, but we can't track it to conversion for experiments that are more back. End focused like changing a overall experience for a name space. Let's say it's onboarding versus no on-boarding, then. Yes, we can track that all the way to revenue and we do okay.

D

That's good yeah awesome. All right. That's that's good! That's good! To see! I you know it's! It's um yeah! You know! I like the fact that when you run it, you run a test, so example in slide 15 the experiment. You know. Obviously the statistical significance, two percent lift awesome. We know those trials are real and if you stick with that, you're gonna get them forever. The next question is: okay. Is the quality of the trials go down or is it the same? Meaning it's a creative right?

D

So, like that's yeah, that's the that's kind of involved question so.

E

Yeah, I totally agree with you on that one I think in an ideal world we would have user id tracking, and that would be the natural thing we would look at is do these users that have see this higher level change on the front end early in the sign up experience. Do they go on to adopt the product at the same rate, and we can't answer that question today, because we don't have user id tracking um it for experiments where we do uh it's more of an experience. Change then.

E

um Yes, that is something that um we do. Look at.

G

D

That makes sense. I I think, if there's a way that we can, you know, follow as many tests through to as many stages of that funnel as we can, so we can figure out.

D

Okay, we test one drop-off point but then see how it all the way it goes through, and I know it may increase uh experiment times, because you know, as you go down the funnel you're, you need more uh more data, more n right, more more more data points in your test of control to kind of get the significance, but I think that's, I think it'd be.

E

A good thing um yeah- and uh this is me thinking off the top of my head and mike feel free to jump in here, uh but I think, with a new experiment framework that the engineering team has been putting a lot of effort into um in theory. I believe we could create a purchase event and not necessarily know who the person is, but at least know if that purchase event occurred, and that would allow us to understand the impact. Just on like absolute conversions between a control and experiment.

E

We wouldn't know the dollars, but at least we would would have a pulse on the conversion rate.

D

Makes sense, yeah and there's like it's actually more important to know that the conversion happened versus the dollar value, especially in these type of experiments, because you get into bias when you, when you start to say oh look at that one, because you get one outlier who closed for fifty thousand dollars like. Oh, this is awesome, but you know. I think that the number of uh conversions is more important, so that sounds promising and um if we can do it with, you know, obviously respecting user privacy in a way that makes the makes sense right.

D

I think that's the right, the right thing to do and there's ways to do that. So.

E

A

A

Thank you, uh cydia. The next one.

H

Yeah, I love the product driver data. The graphs are great, but also there's a lot of context about the data and the parts that are missing the parts that we're adding. It's really good, thanks for that. Thank you. Sid david.

C

Oh, I was just telling that we we skipped over yellowski higher, so I wanted to get the chance to outline those for the team as well.

F

Yeah, thank you david, so yeah really quick added under the the second question as well. So we have a we're hiring a manager of product data analysis.

F

We have two pretty strong candidates, both at last stage, so I'm hopeful we can close this one, and then we have a product data analyst and a senior growth pm both are. We are interviewing.

A

Make sense, thank you. Hilar sid, you invited me.

H

Yeah in the okrs kpis um or in the okrs there's still a link between gitlab private and prospect one. I think the situation changed because github launched github ae and it's now more of a competitive situation where we need to probably want to match that and it's no longer dependent on prospect. One so consider removing that from the okay.

A

Yeah, it's a good point. Thank you. We'll do.

F

H

The next one, as well uh great increase in the paid net promoter score to 33.. That's awesome. First time we're um we're well we're we're almost the 1995 percentile we're almost within within our goal of 40., so we might be there. We just don't know yet, but it's uh it's a great improvement from 25.

A

Thank you sid and yeah. uh As a reminder, that's as pnps and I know foreign team are working on self-managed pnps.

A

I was surprised not to see that question here, so maybe for english. We should just answer it anyway.

G

Yeah, actually I um it might be a bit subtle on slide 33. I highlight that we have an unofficial score of 31 um from our doc site pilot for self-managed pnps. um We found that the doc site is not an ideal channel um for self-managed pnps, but we are going to continue down the path of.

G

Upgrading our broadcast message feature capabilities to be able to collect our self-managed uh pnps that way, so that is my current update and hopefully more progress next time.

A

D

You knew I had one tactical question: did we rename team ow to see now.

A

Yes, we did, I think uh it's based on the feedback from the last meeting or group conversation. I don't remember, we he changed female to cumulative monthly, active. You say: okay, duplicate, the summation of stages.

C

A

Also heard not to add any confusion.

C

But we kind of had a seamount, which was category now with an organization. So um you might see that too, because it's some of the stages for reporting category now as part of the data to everyone.

D

Okay, cool yeah- I just saw that it was. It was a new new term, so I had to look it up and just wanted to confirm. Okay, awesome, we're, not measuring cats. Just to be clear. You.

A

A

Do you want to verbalize your comment.

H

Yeah, um based on the mps survey and uh 13 10 release post discussion on hacker news. Both are full of complaints about our the ux of our merge requests like that merge request, review dealing with large diffs interface is like not doing what you expect it to do. It's slow, it's sluggish.

H

It's I've seen a ton of it in both like there's there's, I think four or five comments to that kind of effect in both items and since, like that's the core, if it had to point in one feature of gitlab, that's core: it's it's our merge request and that's the core of our product.

H

uh I know we already made improvement to the state there. It was very buggy. I think we fixed a lot of that, but it seems that so some of this might be trailing, but I'm worried that there's more needed.

H

What do people think.

A

So I I tend to agree and if christie is on the call she can verbalize that uh yeah. Can you talk about what we're thinking of doing there.

I

Yeah, you bet um so yes, we know that mr usability is a problem that comes up as one of the key negative things in sus. uh What I've been discussing with both anup and uh christopher so far and starting to have a broader discussion, is a series of okrs, uber, q2 and q3, to address some of this, I'm not going to tell you to address all of it.

I

um The idea that we're thinking about now is during q2, there's already a proposal from the development team to rework uh back-end things to make sure that business logic isn't leaking into the front end, which is one of the state problems that we're having right now in parallel, what the design team would do during q2 is: rework our mr widgets, based on a new framework that pedro has been working on and that would address usability concerns with the mr widgets and then that would feed into q3 where we'd actually implement those usability improvements.

I

So q2 would be about performance, stability and teeing up usability improvements for q3 that doesn't get into diffs and things like that. But the mr widgets are a key usability problem with mrs.

H

Cool yeah and then diffs would be q3.

I

Possibly, um is that something that we could talk about having then designers go deep on diffs in q3 and we could just kind of keep working in parallel if we wanted.

H

Yeah, I think, uh that's great, to hear glad you're on it. I think a lot of this is interaction kind of like it's half a performance problem and it's half a ux problem.

H

So it seems that this interaction between ux and performance is is: is our achilles heel where we're no longer able to solve these things in one department, but we need to collaborate um so glad you're focusing on it.

H

It seems important feel free to kind of try to load a lot of this in q2 and then let's talk about how we have to maybe change the organization or maybe work different for one quarter to address this because it seems uh it seems something both important and urgent, and maybe we should accept some inefficiency in solving this inefficiency as moving people temporarily to that or maybe ramping up the amount of calls about this or um I'm open to I'm open to doing something inefficient. Since it's such a dire problem thanks but glad you're.

H

It's already on your radar.

I

Yeah, absolutely I want to just confirm you're right. It is both performance and usability. You got it exactly right. One thing we are doing is moving a designer temporarily from geo and distribute over to code review, to support code review temporarily for a few months and then anub- and I are also talking about how we might get more design support on code review longer term. What we haven't talked about is developer resources, so that's something: maybe we could discuss separately.

H

Awesome. Thank you. That's perfect,.

A

Thanks christy, I think craig you have the next one.

D

Awesome yeah, I was looking at the stage per organization, deep dive and your reflections in slide 29, um pretty interesting chart just shows kind of the curve of adoption. So I guess my conclusion was that if they're not if they haven't adopted a stage by stage 15 by day 15, it's unlikely that they're going to do so. Is that a fair assessment? And how is the team thinking about that?.

F

I can speak to it and I think dave is here. He can speak a little bit more in terms of the data insights high level. I tend to agree like the the that's why we choose to focus on um the stage early on stage adoption early on. uh However, for some of the stages there are a longer tail as well, so the absolute percentage that will adopt a stage past the initial 30 day or 90 days is small, but we do see, for example, the cia adoption it can.

F

It has a long tail, it's just really small, but it it can happen afterwards. Majority focusing on the early on in activity is pretty critical. That's why gross choose to focus there dave anything you want to add there.

J

Yeah, thank you, gila. In addition to what you had to say, there's a lot of by stage there's certain stages where we're seeing not just adoption of the first event, but the time to the second event, we're seeing is very important happening within even the first three five days.

J

An upcoming analysis, we'll be doing here is at also looking at the order of these stages adopted to see if there are certain user paths that lead to higher conversion rates, to paid plan.

A

Yeah and I'll add a couple of other things with respect to customers so smaller, but more uh but an important group as well. We are starting to push data into gain side and that can now help fans encourage stage adoption and they can also see how the stage adoption is increasing or decreasing, which they didn't have the capability to do before, so that active sort of engagement is also gonna change. Hopefully, change disco.

E

And add one more thing to this um over, ideally in the next month or so we're going to launch a new experiment where we give users the ability, a new space for continuous onboarding. So we can recommend additional stages and features for them to adopt.

E

So, ideally the get people to adopt one stage and then help them understand how they can become better at their job. So their organizations can become more efficient by further adoption kit lab.

D

Sam is that going to be uh heuristic based or more ai, based based on behaviors for the first.

E

Version, it's you know it's starting with our gut opinions um and the idea would be successful that then this becomes far more intelligent over time and we um devote more and more resources to you know showing people the right things at the right moment. Awesome so starting simple. I love it. Yeah yeah.

F

It will become stage. You also want to use like longer term.

D

Sounds real exciting. It's uh like I said, the insights in this deck are are awesome, so thank you for that.

K

So one more thing to add to that just to keep the party going. uh We also want to collect the their self-designated jobs to be done in the account creation flow and then tie that into that onboarding so that if they say they're here for ci, we can change the the order of the things we're recommending or something different. Depending on what they've said.

A

Yeah and I'm sure there are a lot more levers to pull here now that we are tracking this, including are things set up by default. So now there is a dashboard for that setup by default and working by default for each category, and uh we are doing conducting more research on which cross stage moments are key cross stage moments to encourage adoption from one stage into the other. So that's going to help us feed into product changes as well.

E

D

As you measure these things and publish, you know like charts that show like funnel and stuff like that, if you need, if you need any guinea pig to say, does this make sense, like I'm happy to be that guinea pig, so I'm happy to help.

F

Craig we'll run them through you.

A

Awesome any other questions we have about a minute left.

A

Great thank you for all the questions and interaction have a great day ahead, great job. Thank you.