GitLab Digital Experience, 15 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: DEX - Launch Darkly data collection sync

Description

Meeting to discuss what the problem is for collecting data on our A/B test and a solve.

https://gitlab.com/gitlab-com/marketing/digital-experience/buyer-experience/-/issues/1244

A

Emit an event hello, um so we've inverted it. So we are getting all the data we need in Google analytics right, Google analytics that sort of Google analytics says you know: did they take an action and we have in the data layer, zero or one and the name of the launch, Barclay experiment. So you could extract that data, but launch Darkly in its experimentation.

A

Feature has its own like data pipeline right where you say, like you, tell launch Darkly to listen to particular events, which we are not firing because we have to like write, custom events to do that, and so we have an A B test so like launch. Darkly right now will not give us any data in the launch Darkly interface, but we could extract the data where we're we're curious about from Google Analytics.

B

Yeah, so the reason why we need both is because launch Darkly is going to be more accurate, whereas GA is going to have some cookie loss, um we're still tracking in GA, because we need to find out down the funnel if the user, you know, purchased it submittedly. Whatever the case is that's um more of a business kpi, whereas launch struggling, is going to be that on page conversion, did they click it? Yes or no? If that makes sense,.

C

Tyler just to Circle back, what's there's been a custom event and just the normal click events that we've done the past well,.

A

Because the click event well, because the click events we do like ga.track, it's like specifically like you, click a button and it like talks to window.ga right launch Darkly wants they can't listen to that, because we're not talking to you. So we have to do something to tell on Sharkey what happened. Oh.

C

Sorry I thought we already had events working in GA or like basic events. We.

A

Yes, we do, but these are two different things right.

C

A

Have events for Google analytics? We have no events for launch Darkly, so launch. Darkly is getting zero information directly to its interface gotcha.

C

Okay and so I'm, assuming that's the issue- that's assigned to me- is the um I'm gonna, be honest. I have no idea. What's going on with those launch character issues I just picked them up because they're in the in the backlog.

B

Yeah for number two one I think I ordered it correctly right. It's like. First, we need to figure out the custom events for launch Darkly, which is that issue after that's done and I. Think Miracle or Javi can then um create the conversion for that experiment.

D

D

Is that code in ventory that we need to set up or launch Darkly.

B

I think in launch Darkly they'll be able to configure it I'm, not really sure. To be honest, just.

E

From my experience, it'll probably be both like we'll be we'll have to configure like a custom event in launch Darkly and then have a script run in our code as well, and then connect the two.

D

All right once we do that, can we then move on to two three and the the ga data layer push from my understanding. That's like the key issue. If we get that solved, we'll resolve this issue.

B

So, on Google, analytics separately from launch dark B right. Currently we have click tracking with launch Darkly IDs, so we're good there. This one is mainly for page views um where the data layer sometimes will be after the Dom load, and then the Dom load is what's triggering the page view. If that makes sense, so sometimes the pageable will have launch directly IDs, and sometimes it won't. So we just need to make sure that it goes before Dom ready.

B

I think this one should be an easy fix right. It's like.

F

G

Sorry Dennis, what's Dom ready.

B

uh Dom ready is document load, so all the content on the page is loaded so push the data layer before that happens. Okay.

D

Number uh 2B that will.

D

And get data and launched directly yeah.

B

Two-Way to be our lunch directly, three is okay, but don't worry about GA too much um just focus on lunch directly. First, for today, all.

D

Right, so that's one, and so actually the priorities are yeah.

B

D

Are one or two that's really good.

F

D

F

Figuring that out, yeah, okay.

D

And this is data for Google D prioritized today.

D

um So then three we can just we can do this one tomorrow or Monday.

F

E

um If we isn't 3A the same thing as 2A.

F

Yes, uh oh yeah, yeah.

B

I think Javi added.

F

That it's not here.

F

I'm gonna just.

A

Oh I'm, looking at the wrong numbers because there's like there's like the the leftmost numbers and then there's a two, a one: two B two I.

D

Just deleted that one, so we we can be clear like.

F

Okay, sorry, there.

D

You go, we need to do two, a and two and two we'll do two c after two a and two B.

C

Okay, does it okay for 2A, it's currently assigned to me I have no context. I'm gonna, be honest. It's gonna take me a bit to get going on this. um If anyone has more context, I would gladly appreciate if someone picked it up, if not, I can I'm just saying it's going to take me probably longer than.

E

Triages have taught me since.

C

I do know Harvey's busy, I guess I could I could help him out on his other um he's working on.

E

He said that he was uh like: we talked about his tickets yesterday. He said he was uh waiting for Content. Oh.

C

E

G

Was Javi working on? Was he doing team-ups.

C

No coupon yeah coupon, but if they're waiting for Content, then.

G

um No, they should have content by now, but you can work on. This is more important. Keep Calm, there's a bit of time so.

D

um Just want to clarify for 2C that will push data to Google, but we we have some of the data already.

B

Yeah so um under the CTA click header, free trial events here um for today, uh we can see that there's a zero which is control. We have this many and then one which is this many, and we know that in launch directly, there's a 50, 50 split so so far one is winning. We just don't have it on other um page views which is fine for now. You know this is what we're looking for anyway, but that's why I put it as three as lower priority yeah.

G

B

G

Dennis, do you think we're okay to just continue then, or do we need to stop and like restart until we're.

B

uh We can continue because we have the data in GA, at least so we can use that for the time being,.

A

And you're you're Google analytics data was always going to be different than monster clay, so this will just be different in a different way. Anyways.

G

Which one would we use as a source of truth, though, if we're gonna have two different launch.

A

Darkly will be more accurate, okay.

B

Yeah lunch directly, but it's only going to be a day right that we don't have data so or two days. However,.

A

Long, it takes right, it was like unplanned work that, like we haven't like totally like tried yet right. It's like you know like it. It shouldn't take too long right, but, like maybe it does. Who knows this isn't just turning on a switch right, it's like writing code and making sure that it works. Yeah, I.

G

Think if it just goes beyond today or tomorrow, then maybe we need to restart it. I, don't know how much data, because they think this test. We only wanted to run for two weeks. So it's like if we're missing like I, don't know four or five days worth of data, then I don't know how accurate it'll be.

F

um I'm just saying.

D

Writing that down, we don't resolve by the end of the week. We will restart experiment.

F

B

Maybe uh Nathan you can let us know by tomorrow, if like, if it's an Ethics or or.

G

Just later today, yeah.

F

G

You like have an Inkling.

C

Sorry, my Wi-Fi is terrible. Can you repeat that oh.

B

F

B

Happy enough today, if you can figure out, if it's going to take a long time or yeah, if it's a quick fix.

D

I'm putting a to-do down there, for when can we deliver a solution, Nathan's working on the initial issue of custom events and launch Darkly you'll? Let us know by end of day how we're looking and then you'll work with Javi and miracle on the solve.

F

D

Do you guys feel about fixing this today? Is it we're still still unknowns like yeah.

A

It's totally unknown yeah, you know, I, don't think we can like yeah.

F

A

Think we just like missed a whole step right. So it's hard to say all.

F

C

No, there is another thing there was just here. Let me post this issue um about the nav URL params or something is that something separate Miracle. Do you know more about this.

E

Url sorry, this is my first time hearing about it.

C

Launched Darkly a b testing for new nav um that the pricing page one or only share it in the doc.

C

Yes, I see, if there's a comment at the very. Let me link that there's a column at the very bottom uh about the url url param is buggy. Oh.

E

A

Yeah, oh that's good to know because it's not working for me either. So is it just not working no.

E

It's just not working, it looks like Megan is on an issue to fix that.

D

Megan's out for the rest of the week, yeah.

F

E

F

E

Okay, yeah, so that override link would um allow you to completely stop the experiment like you would just look if you just use that.

F

Link and right now.

E

It's buggy um I, where, where is this issue, it's.

A

Not changing I I've been playing around with this morning because I've been trying to use it for like testing locally, and it's like the um it's supposed to set the loading to be false right and then it's like and that's happening, but that's not updating the class um or like something isn't happening so like. Basically, it keeps the hidden class on the experiment. Even after loading is done. I think that's the problem, so just stays hidden.

A

But if you go in and you if you remove the hidden class from it um that usually like like you, can you can get it to work.

B

Okay or just load the page, a bunch of times, see if there's a change, I.

F

E

uh Yeah I think I think you need this Dennis right to set up something in ga yeah.

B

But it's fine for me: I think Nathan might need it more if you're gonna test out the events.

C

Yeah I have I, have no a b tests like at least the product. Marketing team isn't doing anything right now. So.

F

C

Really not sure, what's going on with the lasab test, so I'm gonna, it's gonna, take me a bit to jump into some context here to understand. What's going on, but I can hopefully get back to us later today, but whatever I find okay.

E

Maybe maybe you me and hobby should just jump in like a a zoom call and then just work through it.

E

C

That'd be great.

D

That sounds awesome um yeah. How about y'all get to it and um use our slack Channel. We got a dedicated Power, Ranger slack Channel yeah.

F

A

Have another item there um uh just since uh this is unrelated to like the data stuff but like so we have an A B test around the nav and then we're trying to a B test like witch nav in the pricing page I, don't think we can wrap a b tests. I think it's like I, don't think it works, so I need to not use the A B test navigation in the A B test, like does this make sense like pricing pages and a B test, either minimal navigation or full navigation.

A

Full navigation has its own a B test and, like I, don't think we can wrap, launch Darkly components so, like I have to just use like one navigation right, so the pricing page can't run those two things simultaneously. Is that cool yep.

G

That's fine so.

A

I'm gonna have to re I'm gonna have to write because we're doing the A B test in the navigation component. I'm just gonna have to write a separate navigation component just like just for this.

B

I think uh this one will go by pretty quickly because it's going to be on every page and then once it's done, we'll just push the pricing page nav yeah.

G

Yeah I kind of agree with that too Dennis, because.

B

G

A

I thought that was I was told that this was like top priority. Michael told me that, like yesterday and I think Phil's the U.S yesterday too yeah so am I not doing this like today.

B

No I just mean that um this test that we're running now is gonna finish quickly, because it's on every single page, which means it's getting more traffic to it. Whereas on the pricing page, it's going to take a longer time to run sure yeah, so Tyler's.

G

Question is: do we wait or do you just do it now? How much extra work is it to do it now? Tyler.

A

It was not actually it's just like I I guess, I'm, just confused sorry. My dog is freaking out I'm, just I'm just confused about my priorities because, like I was told, this was top priority and then I and then y'all said like oh we'll do this after the other one. So like I, I guess like I, can I'm.

F

A

um I can do I can do whatever um if we run that, like I can't I can never sorry he's freaking out because uh Sarah's gone I can never there's no way for me to do them concurrently right so like I.

G

A

To know which one one of the navigations, either A or B I need to choose one of the navigations, either A or B right yeah. You can put it in there and then, when you're done with the experiment, I'll have to go back and just like put it put whatever one in in there right. Yeah.

G

um I my logical thing is like we should wait until this one's done and then choose the winner and Implement that but I I don't know. Let me just check with Michael to see if he is also agreed with that sure.

A

Cool, that's fine um yeah. It would be less work to choose one because we cannot wrap them, um but I mean it's not like a ton of words. It's like once I get it working. It's fine, I gotta figure this out today, anyways because it's just like it isn't working at all, for whatever reason.

G

And then just on this topic, then do we have confidence that we'll be able to.

F

A

F

D

I think she was gonna ask do we have conflict.

F

Oh okay, yeah all right.

G

Yeah, sorry, do we have confidence in uh that we're able to measure what we need. uh uh Can you hear me yeah, yeah? Okay, um do we have confidence in the metrics we're going to be able to measure in launch directly for the pricing page experiments.

A

No, not yet we haven't done it yet.

G

Okay, so we won't know that anyways, okay,.

A

Yes, it's the same thing we were talking about before, like we have to like hook up the launch Darkly like data right.

G

So we technically shouldn't launch the pricing page months today or tomorrow. Yeah.

A

Yeah yeah, like I, could launch it today, but I wouldn't have anything to give to you right. Okay,.

G

That's why I'm just leaning towards like we should just wait. Let the this register one run and then choose yeah, but I don't know. Let me confirm. Okay,.

D

The first time we've built this and we shipped it quickly and it's best, but there's a little Gap, so we're gonna, iterate and fix it. Yep.

G

And then Lauren, can you send me an invite to launch directly it's not working for some reason.

C

G

Yes, I do remember that.

E

Javi and I talked because I think he and Tyler are like he he's the only person that can get in contact with yesterday that had access to lunch directly.

E

um Is there a way we could just make like a DEX engineer or whatever email and then assign, like all of our meetings, all our recurring meetings and then, like things like launch directly to that email and then shared amongst the engineers, and then we don't have to worry about yeah.

A

I think that would violate I think we're not a lot like that violates like the terms of services for launch like, like they have seat based pricing right, like that's, probably.

E

F

A

A

D

B

Right, yes, all.

D

F

D

That we're gonna end the.

F

D

And get to work all right.