Magento Architectural Discussions, 28 Aug 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Magento Architectural Discussion -- August, 28, 2019

Description

* GraphQL - Generic Filtering Proposal
* Enormous number of files in var/report

Meeting minutes: https://github.com/magento/architecture/issues/235

A

That's our key textual meeting so as I mentioned before, I have couple topics: I see that Prabhu has already joined us. So maybe probably you can start and then I will switch to another topic that I have.

A

Just share the screen music document and.

B

Are you guys able to see my screen?

B

B

Okay, so, as you well know like we are redefining the filtering or the thing is, we came up with the problem, wherein, whenever a customer filterable attribute is being added to the product it, it has to be reflected on the schema so that the clients can use it attribute to filter it. The problem is, we need to refresh to clean the cache and the fresher schema every time and attribute is being added, so it is quite an expensive operation.

B

So I came up with the proposal where in its kind of a generic filtering, so you give the the possible the the attributes which can be possibly filtered. You know in a generic format like so there are like three three different types of filtering 1 is equal filter where you can see the where you can do the equals operation, and there is a like filter where you can do any like operation, and that is arrange, filter bar, which is for the price changes.

B

So these are the only three possible types of filters and if we can take the attributes for this filters in form of an array and then so the on the back end, we can consume this different attribute. So then we can do the possible filtering. So instead of refreshing, the schema every time whenever we add an attribute, this can be used as a generic. Filtering I only share this with Alex and Olga, and they already have comments. So not sure if you have seen this already I have any comments on this Olga.

A

Yes, like just my general comment, is it's not so you mostly describe how obligations will work on for aggregation? It looks fine like you just want to get different like aggregation by different fields and probably that's fine, while I'm not sure how it's supposed to work for filtering. So you usually want to filter by specific field, and if you don't have ski Mahad, how can you do it if you don't exist.

B

So the filtering is driven based on the output, so whenever the client goes to the front and he searches, though the the aggregations are being written so which is in the new model as well, so the clients will know what are the possible fields that can be filtered on. So it need not be essentially present on the schema, but rather it is driven from the output of the previous query. So, let's.

A

Look at the scenario so I want to filter products, somehow most likely I know how I want them, so the research probably and then there is layered navigation. So this is in context of learn a vacation right. We are discussing yes, and also there is sorting as well, which is not part of layered navigation. Yes, you just want to sort by something, and then somebody can implement just custom search like as we have advanced search, for example, where you would need to specify fields by which you want to filter.

A

So because I understand your proposal, you, your flow, would look like. First, you, let's say on PW you just you draw products listing and with that you get all the aggregations right and now you know what filters exist.

A

Ask user to filter by whatever is possible, like you can just do it. Yes, filter yeah.

C

A

Cannot, because you don't know the field, you need this additional requests for aggregations right. Yes,.

B

So, let's see another case right so when we are dynamically updating the possible attributes of the schema, it might even like go beyond like hundred different attributes right, but still for the clients. You know what are the possible stuff. They can even show on that, you wife, so how will they even know like what else have they can possibly show under you it? It has to be driven by the aggregations right.

A

Guests of so in one one, schema changes, schema changes. If some somebody goes to admin, for example, and creates another AV attribute right or somebody goes to the cold or installs extension, and it adds some custom attributes, probably right what other use cases. So it's either code changes with or it's an AV attribute added on the admin. We probably are not very concerned about the first case, because in case you update the code most likely. Yes, you need to check your PW a as well. Maybe you can add more features.

A

Maybe something is removed, but we are most concerned about admin. I, think attributes correct.

A

B

And the clients need to know that an attribute has to be app has been added and they have to clear the cache. And so the thing is I am not aware if there is any cache tags associated with a graph given schema invalidation and whenever a graph itself is invalidated, the schemas stitching and the schema parsing takes a lot of time compared to like any other invalidation. It takes at least like more than a couple of seconds, which is going to be a very expensive operation compared to other in validations. Yes,.

A

But again, even if schema is involuntary I think couple seconds it doesn't matter more like the bigger question is how a client like for client to know that it can filter by this field. It should be implemented in specific way.

D

I'm sorry I thought you were done. My mistake.

A

You have something to add: roids I wanted.

D

To agree with you, Olga is right and I can say specifically that the PWA project is already querying the schema at Build time. So any information about the capabilities of the platform or the capabilities of the instance we expect to find in the schema.

D

The document here mentions that schema ties, income attributes doesn't provide customer value or very much and I understand that there are some downsides to it, because queries are not always portable, but the upside to it is that when we're compiling an app, we can query the schema to see what types are possible and if we have say component for the front-end that only handles a particular custom attribute, then it could detect that custom attribute is not in use and remove it from the compiled application, which is an optimization technique.

D

That's very important to us because building extensible apps. So that's one reason for us who want custom attributes to be available in the schema without querying on a particular product.

A

D

Idea is basically to to have two possible interfaces. One is querying the schema and detecting custom attributes presence, maybe identifying them with a directive for Build time, and the other would be a an interface much like this proposal. It looks good because, like you said, we need to iterate at runtime as well. We were just hoping to also have this information present in the schema.

A

Okay, so in general, I think interesting.

A

Proposal is to describe use cases and expected so, for example, one workflow is adding a custom attribute which happens in the code and just let's describe who is involved. Who should be doing what like? How? How PW a developer will be affected like how you should interact and then what happens if an admin creates an attribute right now we need to clear the cache and then PW application somehow also needs to know that schema is updated. So I'm not sure if this happens but another, so it would be good to have those flows described.

A

It would help to understand us what we want to achieve and what will be the most optimal solution and another note that I have is that schema is just one part. The graphical is just one part which is just presentation layer, but then there is also indexing of the data itself. So when an attribute is added, it can potentially say like I believe it can have some default value or whatever or if, when we move to separate data storage for storefronts api's, which will be used by graph GL.

A

Actually, it means that this data and schema should be updated in this data storage as well so like that will be more. There is more work anyways if an attribute is added an attribute which should be visible on the front-end on the storefront.

E

A

That's just another command that, yes, we might solve graph GL part, but then we will hit the wall because of data.

A

It would be good to cover that part, but my or maybe we can just discuss it later, but ideal. If you at least can add workflows here in the document. Yeah sure.

F

I'll yeah I'm, sorry, one more thing: you propose to remove cache, but we still need to get some data that was cached before and it actually means that. Yes, we are saving two seconds on schema regeneration, but now we need to extract the same data for each request, but probably it will impact have some impact on performance over storefront API or something like that.

B

Extract the data from the aggregations you mean.

F

So I think you propose to add the attribute type to obligation where it is, it already agreed. I am sure yeah I.

B

Propose to add the attribute type of decorations, yeah.

F

So we need to extract it for each request: Thanks.

B

Yeah I see your point, so you must think that it might take. It might be a little expensive to do that as well.

F

So we can extract our aggregation type and this should be cheap. For example, we can say that the provided value is in range pocket or something like that, but having the attribute type is relatively expensive.

B

Even I think aggregation type should essentially solve the purpose, at least what I'm trying to achieve, but yeah I'll I was like little underprepared for this meeting, because I didn't know that I'll update the workflows, what olga suggested and what you're interested so and then I can get back. So we can discuss in much more in detail and.

G

Also can be a comparison between current implementation and post implementation to see what benefits and drawbacks implementation has.

G

A

And thank you for joining yeah I just go, and thanks.

A

Thank you anything else that we want to add to this topic.

A

If not I'd like to discuss another one totally different now, I don't have a proposal and a proposal will be prepared a little bit later by dev team, but I just didn't want to like wait another week. So I wanted to start the discussion. So the problem that we have and I can share the screen for people who like to read more than listen. There is some description. It will be more organized, probably in the proposal, but in general.

A

The problem that we have is that there are customers or merchants who have to face an issue of a lot of Magento logs production. Those are but caused by exceptions happening in magenta. There can be different reasons why it happens.

A

A couple of reasons that we discussed is it can be that something happened just happened in the application and we have repeating error like the same error message locked in envelopes and if it's a rush time so like I, don't know if it's Black Friday, for example, it it will build up a huge box eventually pretty quickly. So that's one issue, that's one reason.

A

Another reason is that the logs can be just ignored, like some exception loads, so I would usually expect that those shouldn't be ignored, but probably you have to analyze what kind of logs are being ignored, like what kind of exceptions might be? There is something in magenta that causes people saying that this is not an important issue like it's not really an issue, but it's been logged as an error. Maybe there are real errors and people just don't get to fixing them so anyways.

A

There is a situation where a server just like just has a lot of logs. It feels the file system.

A

Application cannot, the server cannot really work normally, so the solution that was that is proposed right now is to add, to add nesting level, to the reports that we have and it's about water reports so like when the files which are created for every exception, like for every exception, we have one new file.

A

So one proposal is to have at least some nesting level because one of the issues- it's not even disk space, but inability to even list the files in there, because there are millions of them and the second part of the proposal is to log on the unique unique exceptions in the report. Report includes backed rice, so it's like big amount of data.

A

If we, if we take into account that there are a lot of such files, so the proposal is that, in the exception, would be just an error message, but we create, on the one exception, a report log for each kind of not each kind of error, but for each error that we have so we don't duplicate them. So it's based like there is some prototype implementation. It's by based on calculation.

A

You can calculate in hash from the like from I believe from the request to based on the request and probably like exception, so based on some unique values. We calculate hash and we create file with the name of that fashion, and only and eventually we have only one such file. It should also help with repeating errors so that I just wanted to start the discussion discussion. Maybe somebody has like totally different opinion might be.

A

We shouldn't fix this at all and this responsibility of a site to rotate logs and make sure that they don't build up there. Maybe there are some other ideas. What we should do, or some big concerns about such implementation. If there.

F

A

E

Did you consider the Adobe standards for their logging.

E

We should take them into account because the login is about one of parts of PCI compliance audit and we, like our, follow these PCI compliance rules. So they are not so clear as we look documentation, but adobe has multiple documents which describe different login mechanism.

H

I think is what a different that just Logan so accession still login in log, but additional it a magenta health possibility to create report, so I believe internally have discussion and proposal to remove this mechanism. Next minor version of.

I

H

So, like I said two difference in what the right and world looks and there are different type of role, login possibility and slightly different, that its wrapper chunk that, with a bill somewhere, I see my nizzles that come from magenta one and.

H

Where we put, on each exception, information about succession and stack traces exception to.

A

Be proposed to not put stack trace.

H

It's hard question: I, don't have.

F

Issues with multiple files and that directory couldn't be rotated or something like that, and regarding the possibility to remove duplicate, it seems not very long negative mechanism. So usually, if you have this similar entries in the four files, you still keep them with times time with everything and just compress them. So usually, if you have the similar message, it could be compressed pretty well.

H

Only for look if you fight message, but if you try to ver it stack trace for magenta. It's the huge amount of data.

F

So if you put the same logs, they will be compressed very well, so they will not consume tons of space, and this is, if you want to resolve.

A

I

Said if you have longer equation that should address a lot of the issue, if you lock it down in into one individual lock, and then you get a lot rotate.

I

Right so, if you're talking about which we should probably implement loud rotation inside of our cloud, offering what? If you're talking about si implementation, yeah I think will probably be the.

I

Implementation side of the device environment.

H

If I call, it is already implemented so.

A

H

It's so different, because we've currently put X locks a exception. In the exception work addition. We also put stack traces and reports. My weave. We need just remove, which in stack traces in report and write, owens exception law.

F

Recommendation on tap talks about magenta installation and there is post installation steps that say that we need to the top of rotation links and also we can try to use the virtualization techniques. For example, we may, if we decide to create a docker image for Magento. We can include of rotation like an ism inside.

J

It can be also look into.

J

Visualization of the box instance, there are tools like Splunk that allows you to visualize walks. Maybe we can turn in for for services relation we go have to build some log aggregation and visualization, but we can probably add part of this model it. It would be like much easier to see what's going on, for instance, on the cloud it can be.

J

A dashboard where, like size, would be able to see this logs and monitor what's going on and if probably, if we stream, if we stream this works somewhere, and these tools would could also provide the functionality of pujan old old data.

A

The question whether we need to add something to imagine whether we need to change something in magenta location itself. Or do we say that this should be sold on the like on the infrastructure level by a sigh or.

J

A

J

Application as I see instead of like for iTunes, is two files. We would put this logs into like some some systems that is responsible for logs.

A

H

First reporting because yeah.

A

H

We agree: maybe they agreed to remove this functionality at all right. Well,.

I

Stack traces I think it's pretty important, maybe it's more than documentation thing it. Maybe it's not even necessarily an implementation. I mean I, absolutely recognized the issue of having many many files, and that's it. That's an issue. I think we probably should address but having the the ability to pull stack traces for observability I. Think it's very important. Maybe it is just simply about communication situation. Maybe we start at the base level.

I

How do we integrate AWS logs to capture data in the cloud watch for AWS, for example, as a documentation feature for a size to implement and then further follow on beyond that into? How do we instruct people how to best aggregate logging I think there might be a.

A

So I wanted to get back to what Andrew connoisseur told about duplicating messages. I just want to clarify the solution. The solution is that, in the exception, log V's or maybe it's system, log I think it's pretty confusing right now, so let's say it's an exception. What we still have all these repeating messages, but we don't have stack trace. So if it's repeating we will have multiple masts and multiple records. What we don't want to repeat is this report. Log, which includes stack, trace and I, mean we can probably move it.

A

Move stack trace back to back to the exception log, but then it will probably be harder to like read: I, don't know to not duplicate stack, trace, I, don't know if it's a big problem or not. But if we are talking about millions of such files, we will have millions of Records unless log rotation is not set up properly like if there is some kind of spike yeah.

H

And I won stream it to external service can be also will be preferable if it's megabytes in seconds.

A

H

Decided to put straight race there at pretty big amount of that parent walks, it can be problem Tara.

A

No, for me, it looks pretty convenient as it is right now. We have exceptional obvious messages in general, we can see when it happened and how many times it happened, and then you can actually go to the report log and see back trace if you really need it, what we can probably reckon see there it in the fish. So let's- let's summarize, let's summarize so first thing is considered removing reporting and under removing reporting, good daemon and removing all those files at all and just putting everything.

H

H

My view it's additional questions, put stack, traces is file or not, but.

A

H

Bits we can configure, is user, ok is putting or not.

I

Being able to actually be able to aggregate that information, if you have the statue, is you can structure that data properly, insofar as that, whatever external log and analytics system you decide to stream this to can properly understand it, which would be the ultimate goal of really any SI or our customer.

B

So so the current.

C

B

Our environments are interesting, the locks to Samad logic right, and that has some quite interesting aggregation set. Is that model that's something we can leverage.

H

Their buses still different problems that look a greyish at all. It's not about work at all, so our current looks is work. Well, we aggregated and it's dream latent for work. Well, it's about representations. Is that not is store data in separate file, and it includes much more information that just looked in killed.

H

It's from Logan point of view are: these are cloud roses work well.

J

Seems, like exception log, and this report, their functionality in purpose overlap a little, so it might make.

F

J

To have exceptional organ not to have this report, but good I also notice that most of these reports, the stacked rings, is not very helpful there, because you would just see object, manager trying to initialize some classes, but you even don't know what those classes are, because the pass is truncated after Magento namespace like it would be Magento, slash, ends and truncated, maybe name of the part of the name of the module, but that's it. Maybe we can also look into improving set.

A

Yeah, we can look at that, but, okay, let's go back to the problem. So what conclusions I have? First, look at the don't be logging. I can look at that then consider removing those reports at all, and it means that we will remove this page. That shows that you can go and look in that file.

A

Look at the error in that file, so that would need some.

A

So like we, the idea is that we want to consolidate everything. In the exception log. We need to figure out what we do. This stack trace- maybe we just put it in there and we need some good guidelines on how to set up your system. So you have log rotation or whatever, and some monitoring to not forget about, like to not miss all those exceptions and just build up.

A

I

Document some tips and tricks to.

A

A

Regarding this digital nesting level that we want to introduce so I didn't hear any opinion about that. So, in my opinion, in won't really solve the problem might be try other opinions. Should we introduce this nesting level, so I just would have several folders on then pulled files in there. So you will have not millions, but maybe I, don't know 100 thousands in each folder or whatever. There is also an approach to have nesting level starting from a year and up to a minute.

A

Should we look in that direction or we probably just don't, need those reports at all and, let's just I, think.

E

All you have some implementation for the nesting level.

H

E

They actually really.

H

Discuss this implementation.

A

You have one level of nesting right, one code or this is what is being implemented, I'm, not sure, because it's part of Magento so I'm not sure what means what it means that you have it on cloud.

H

We have some patches, okay, so.

A

You have only one level right.

H

A

So this is one approach, and this is what we want to implement in in magenta, instead of just having watchful cloud. Another approach that I heard from support team is having, as I mentioned, like year-month-day, probably I, don't know R&B, not like it's bigger nesting yeah. My question is whether we even should look into nesting, or we should just focus on- maybe removing this reporting at all the.

I

Purpose of having all of these individual files, as opposed to one one file for exceptions in one file for errors, names like.

A

E

Some artifacts actually, for example, if we have like payments concept like payments request log and the merchant facing the problem, is like some transaction which made like months ago. It's actually hard to find this transaction because give you, for example, you know you, you know the date, but you don't know actually how to find in your custom logo data. If we have some defined mechanism like agreement, some lobe rotator will be done divided by day, so, for example, by months, etc. It's easier for even for debugging purposes, to fight something. Oh.

I

Okay, so I think a lot.

E

Of it actually.

F

I

Addressed by analytic systems and log aggregation systems honestly try and solve that for people, or should we try and advise them on other products that can help them in that regard,.

A

So, are you asking about current reporting mechanism where we have one file for exception, or are you asking about which different files I.

I

Think both obviously just trying to wrap my brain around. Why why you would have individual files, one.

A

I

Yeah, it seems to me, like a better practice, would be to use logon Eurasian and push that off to a analytic system. I.

A

Guess I was not present when it was introduced, but one of the features that we have in magenta is when you are in production mode, so you're not supposed to see an error with back-trace on the page. You are shown a file name like report name, it's just some digits, I, don't remember it's maybe timestamp and then like. There is a message that go look into this report. So if you have access to the file system, you can go and find that report. It's like an identifier, yeah.

H

Additional because it's sort of client file can provide this number merchants say. Is it some.

C

H

Is shown this number this check.

I

You can have it in a single file with that same identifier and strain that out to analytic system that you can do search upon as a possibility.

J

Single files can become too big to talk, and sometimes so I know that Apache it creates files of like certain size. You can pick this file size, it will just create a start right into a different file and it will name, it will have a sequence number in the name of Sam said.

J

A

Kind of archive old current flow and create new one. So what you still can.

I

Yeah, so the basic idea, blog rotation, is exactly that. Yes, you just kind of purge out and rotate the logs that exist, but part of law patient as a practice is really pushing that data that log data out to a some kind of analytic system that you can then store long term. So you can run either short-term or long-term analytics across the the data set over a long periods of time. If you so choose so yeah well, part of it is purging that data, it's also storing it as well.

I

So log rotation, when you say log rotation, really the actual active library, patient, yes and this purging data, but the the concept around it is capturing the data before you project.

A

Okay, so let's go back to that I guess there are many things that we can improve in our logging system.

A

So for now action items, look at Adobe logging, look at removing this reporting feature at all and just having one exception: log figure out in the follow-up discussions what we do with stack trace. Maybe it's fine just to put.

A

A

A

I

Just for specifically your fire hose that you can then generate those logs and push that into various different analytic systems. It's actually fairly easy to set up as I think. If we explained it to them, would be very pretty easy for them to actually set up for their customers. I think would go a long way towards getting visibility into this type of information.

A

Mm-Hmm, okay, maybe we can sync with you later and we can come up with some good recommendations.

I

A

Okay, then I have some action items, I, think and okay back to this nesting level. I didn't really understand opinion on that, so I would probably even not implement it. I think it's just hiding the problem. Just postpones it.

I

F

We're going to change something in that area. It's I think it's better to remove that functionality completely. Okay,.

A

Okay, good, then, thanks all interesting discussion, anything else, any other questions might be topics so.

J

I updated addressed comments on proposal about CI CD.

A

They want to discuss it now.

A

Walking by him.

A

Can anybody hear me.

A

Okay, I'm sure what happened, but we'll probably just finish this meeting.