Magento Community Engineering Hangouts, 10 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Community Engineering Hangouts. Feb 10, 2021

Description

Meeting agenda:

0:00 - Introduction
0:43 - Composer 2 Support by Max Lesechko
22:42 - PHP8 Support by Sergii Ivashchenko
44:42 - Performance Measurements and Optimizations by Vitalii Zabaznov

Slides:
- Composer 2 Support: https://www.slideshare.net/StanislavIdolov/magento-community-hangouts-2-feb-2021-composer-2-support/StanislavIdolov/magento-community-hangouts-2-feb-2021-composer-2-support
- PHP8: https://www.slideshare.net/StanislavIdolov/magento-community-hangouts-2-feb-2021-php-8-support/StanislavIdolov/magento-community-hangouts-2-feb-2021-php-8-support
- Performance Improvements: https://www.slideshare.net/StanislavIdolov/magento-community-hangouts-2-feb-2021-performance-improvements/StanislavIdolov/magento-community-hangouts-2-feb-2021-performance-improvements

A

Come on the first community can go out this year, so it's the first event in 2021. uh We have pretty much good agenda today. We will start with composer two updates uh for the magento 2.4.

A

Let's, after that, we will have a topic on php, 8 migration and also like. In the end, we will have a huge topic about performance improvements in support of 200 million in schools in your catalogs, so max go ahead.

B

Hi everyone could someone confirm that you can see slides.

A

B

Okay, great uh hi, everyone, one more time and uh to someone who does not know who I am I'm maxwell sage curse and your software engineer at adobe, and today we are going to talk about a bit uh about uh upgrade uh like adding support of composer 2 to magento 2.42 version. It is great that we have uh this session today day after g88 for 2.42.

B

I think someone of you already have this version install it. Maybe someone of you even tried to install the composer too.

B

I think you should not have. I believe you don't have any problem with it. But if you have please let me know, I need to update this slide and change to the next patch version of magento.

B

Okay and let's move forward what we have today so first, I would like to start from brief overview. What is composer is not in deep details but uh about some funny stuff about composer. First of all composer, as you know, this is a dependency manager for php written in php. Why? I want to mention it which is important sometime?

B

Maybe some of you already even tried to debug composer for some specific cases, and this is funny thing to do, and it is great that it's written in php and we can not only guess how it works, but sometimes fix it, allows us to debug it and understand uh all the internal processes and adjust our plugins, which works with a composer and initially this tool was developed by niels, adermann and georgie buggiano in 2012 for information from wikipedi from wikipedia nothing new, but- and these guys uh were inspired by uh not npm and ruby's bundler when they created this tool.

B

uh And many of you remember how this tool become more famous and right now, this tool is, I would say, the most important tool in the php world.

B

Are you familiar with the packages which is the main open source repository?

B

Why it's say open source, because this is for open source projects and there is a counter part of it which names as packages.com, which is private packages, and I think uh some of you familiar with this tool as well. There is also software which allow you to have your local composer repository, install it uh configure it so technically, and we as magento developers, we're familiar that we have dedicated uh composer, a repository which is named repo magento com and many of us.

B

Maybe all of us are using it as well and so computer one was initially developed created about nine years ago and just few months ago, the next my renovation of composer 2 was released was on october 24 at 2020 and at that date our journey began. We just after the release of composer tool.

B

We realized that we wanna to have support uh for magento of composer two, and we did not have a lot of time to implement it because of we expected to have uh magento 242 release uh pretty soon in uh one and a half months after that. uh Like two months is if you're talking about uh ga release, but we have some other stuff which we need to do, and we have code freeze a feature complete date early for it.

B

So, let's, let's review what we have new in composer 2.. So in composer 2 there is a few uh cool scenes which was added in there as well as breaking changes. They all came together all the time, the most important thing in composite- okay, not maybe the most important, but one of them is performance, improvement.

B

Composer team: they did changes to the architecture, they did changes to protocol communication with composer repositories. They also did significant changes in the processes of composer process installations.

B

A lot of changes in there, which resulted in at least 50 percent performance improvement for this tool, which is significant previous uh and one thing which they added in composer 2, is downloading. In parallel, many of you already familiar with extension prestissimo, which we use for composer one. They added similar behavior for composer tool. I think it is even better.

B

Maybe it feels better, maybe because of all other improvements, but this is it doing similar scene and officer of prestissimo has uh on this exten on this plugin of composer knows that we don't need it in use anymore with composer 2, because it's already was done there.

B

Next, one scene, which is a new scene about composer 2, is a deterministic update process is a different and we will discuss.

B

I will explain it later, but I wanted to give some inside right now, so the most like the the most important part of it composer, one was kind of from point one to point two, just uh one single process, so it resolved dependencies per each uh package and uh try to update this package result dependencies in there and after that, a final step was updating the the log file with composer 2.

B

We have two stages: one is update, update stage and another one install stage so at update stage, composer analyze, all the dependencies and the right composer look before even update starting, but it checks all the inconsistency which can be in there. So technically previously, we may had some unexpected errors because of uh because of the code which we have in composer local. We have in vendor directory, and you uh pretty worry about this general recommendation like remove composer local, remove vendor directory and try to install from the very beginning.

B

So composer 2 should not have this issue anymore because of the process and, as I mentioned on update process, we resolve all the dependencies and update composer log and after that, it dispatch. Few more actions and start processing packages and install them, but this process is more reliable. You can find more technical details in the description.

B

uh There are few articles about changes to composer 2.. You can find technical details there, but this is the summary of what they did in there uh run time features. uh This is like pros and cons at the same moment. So my right hand fishes is added validation of the php version and extensions of php, which is optional each time when you run composer updates. So previously this values can be ignored and this may cause issues later after composer was run and there was additional flux which needs to be used for it.

B

But right now this is enabled by default and installation will be filed. If you don't have proper version of php or all the required php extensions in here error reporting uh was improved and guys from composer teams. They say that right now, error messages should be shorter, clear and less repetitive, and you can see error reports now, if like. If you will break anything, you will see them and yeah we'll see what will be feedback from community on it. On this change, but today looks nicer and partial updates.

B

This is cool feature as for me, you know when we do development, uh we all the time wanna, maybe change some uh version of the library just to check our idea, and previously we need to use, require instruction for composer, which leads in updated, composer.json, changed, composer log and sometime. You know we can forget about it.

B

We can commit it push it so right now we have this feature which is composer update and you can specify model, name and constraint of it, and the dependency will be updated, but this will no effect on composer, json or composer log, which is good for local development, uh recording, compatible changes caused by this. This is uh uh changes to plugins like uh this is new api for plugins. So technically, this will affect plugins uh platform check, feature which I mentioned before as right time feature.

B

This also uh make up will result in errors for some deployment cases, but this can be. This is better to know that you don't have a proper environment before you proceed with your deployment process repository priorities. This is important thing right now. If you have a package present in your mind, repository for example, you install magento from people magento. It means if pre, if package is present in ripplemagen.com.

B

All the versions of this package needs to be present in this main repository, and this will uh this feature will ignore uh available versions from packages or any third party uh like secondary repository, and we need to maintain this consistency this. uh What I heard is this significantly reduced amount of traffic to the packages.

B

That's this benefit for composer team. I completely understand that, but right now we will have consistent state for at least your main repository and psr. 0Ps are 4 configurations.

B

In short, they should be correct if they incorrect, you will see failures, and this is expected uh during upgrade uh during adding support of composer 2 for magento 2.4. We face it with the following, uh like the following backward incompatible changes affected, our code base, plugins repository priority and psr issue issues with psr configuration, so each resvspsr configuration was fixed in the code repository priority was easy to fix. We just uploaded missed packages, and we moved forward with fixing of plugins.

B

Here is a list of magento plugins that we released. A new version of these plugins here is the list of third-party plugins. These plugins were updated by community. We just use a new version of it and libraries. One of these libraries was updated. Also, we have a new version of it and yeah. I I'd like to thank you community who participated in fixing of some issues, I'd like to thank you, peter, who created pull requests which allow us to fix each existing issues with magento center and framework one.

B

I would like to thank you many other few who who did your contributions for magento, composer, installer and magento inventory, composer, install it and sorry development team. uh They did not find your contributions and we was limited in time. That's when we just developed it from the scratch, and we can ignore it, your contributions.

B

Sorry for that, so you see a list of new versions which already supports composer 2 and composer one, so we preserve backward compatibility for magento plugins and we preserve backward compatibility for the framework one and this automatically exists in third party plugins.

B

I wanted to show you also. uh You see, highlighted right now, two x plugins, which are hosted on repo.

C

B

All other plugins are hosted on packages.

B

This is not very important, but maybe you wanted to know it. These two plugins are hosted only on the ripo magenta comp. There is no information. There is no packages on.

B

B

uh Issues which we face it with a common change which we will need to do for all our plugins. This was uh changes to plugin inverse interface, which resulted in missed public methods for plugins implementations across all the plugins. We did this change also, we updated dependencies, adding support, composer composer version, 2 and composer plugin api version, 2, which was easy to do, and also there were functional changes for two plugins for root, update, plugin and magento composer installer, which required development and a lot of testing.

B

Because of this composer root update plugin, we was safe, we kind of saved, but we we also needed to test our solutions with multiple versions of magenta, because then the same for composer installer, because as soon as we release these packages publicly uh magento version, 2.3 2.4, all the versions will uh getting a latest version of this plugin.

B

So we need to test upgradeability, and this was uh this was not easy to do test upgradability, because we need to predict that upgradability will work with composer too, but we did not have any previous version of magento, which will work uh with composer 2, and we was.

B

It tried to do some surrogate testing with it in order to be sure that it works as expected.

B

That's it. I guess I will not provide technical details, how you fixed it. You can find fixes in the court, some of them in public repository, and you can find fixes in code of magento composer of the plugin.

B

You can find information about changes uh related to composer 2 how to upgrade to it. I think, if you wanna, to upgrade your your uh plugins, that you have for your system or project, you can find two articles uh about upgrading of it and features and everything related to composer 2, and you can find a lot of existing performance measurements done by different teams related to different projects.

B

And if you have any questions you can ask, we have few minutes more or you can ask me in chat.

A

uh Yeah we have one question from stephan: how do you handle uh magento update tests db? Compare tests, for example?

A

I believe it's related when we should have like two different magento versions: the previous magenta, with composer one and the new version with composer too.

B

Okay database compare test is not about composer it's about installing magento. So technically, if you do it on your local machine, this is easy to do it just install one version with composer one and otherwise composer two and database compiler just running cla2, which compare two states of the database.

B

If we are talking about changes to the build, this is tricky question yeah. You need to do adjustments, it's not easy to do.

B

uh I hope I answered your question about uh how to compare it so database compare is unrelated uh compared to the installation process like if we are talking about installation process, but if we are talking about automation of it yeah you need to use some ifs and try to cover all this stuff.

B

Any other question: does it does it? I answered your question yeah yeah. Thank you.

B

And we also tried to use surrogate testing when we updated some of the dependencies of the previous magento version, and after that we was able to upgrade to the next version of magento using composer 2.. But this, as I mentioned, this is surrogate testing. We just wanted to test it because we did not have any version.

B

Any previous version which supports composer 2, and this was kind of challenge for us.

A

Ok, thank you max so yeah platform platform, health and platform technology stack. It's really important to magenta, especially like the big steps when we need to update to the next major versions.

A

So it's really a big big chunk of work uh so guys, if you have any questions, just put the questions to the chat and we will answer the question asynchronously. While someone talking on the next topic. Okay and the next topic- is about migration to php, eight, so, sir, go ahead.

D

Hello guys, hello, oh some, some aftermute is going here, hello, everyone. So uh my name is sergey washington, I'm from community engineering team and here so max started. This can go out with composer 2 and think I will continue because composer 2 was actually probably the first step on the way for our compatibility with php8. This is one of the requirements and some of the dependencies were updated as well during the composer to upgrade so yeah. I will. I will touch composer topic as well.

D

So I guess composer is really popular still will try to do really fast with my presentation, because I believe you guys are also excited about 200 million skills story sharing. My screen.

D

Can you see the first slide of my presentation?

D

Yep good, so uh today, I'm going to tell you about magento, 2 and php 8 plans and and our experiences start in the the community project on page paid compatibility, but first of all, let's review the php releases and support approaches.

D

So basically php is releasing new version every year and once it's released, it has two years of active support and on top of that one year of security support. So in general the each php release is supported for the only three years.

D

That means that 7.3 version is rich instead of life in just nine months and 7.4 is reaching end of active support in nine months and then we'll have just one year of security support and next autumn, we'll have php 8.1. In addition to eight that we are working on compatibility right now. So considering that rapid schedule, that's always a good idea to take a look on your environments and update not only the frameworks as we do it, but also the environments the instances.

D

So each php release is bringing a lot of features. A lot of new uh cool functionality to us and php 8, considering that that is a major upgrade, is bringing even more cool stuff.

D

I will not focus on php features in this presentation. However, I would like to show you the results of the surveys that that we have published recently and just collected the interest from magenta community like uh which features from php8.

D

uh Do they think, are interesting and should be used in magento, as you can see, the most the votes are for constructor property promotion, and I really support in that probably saving lots of lines, lots of typing and lots of time.

D

Considering that how how magenta classes usually look like there are always lots of properties, you need to define and constructor property promotion features, so just simplifying that a lot for coding from coding types from coding perspective, union types, probably the second, the second place as of now the survey is still going on, and union types is probably useful uh feature for uh like making strict types more flexible and maybe upgrade from non-strip types to street types can be smoother.

D

Attributes are on the third place right now and attributes are really cool, uh a cool feature that I believe this. This is the feature that will have the most ideas, the most new implementation around it, because that's really something that that is. That is ready for our in your creativity. Let's see what we can come up with, the null safe operator is kind of convenient and yeah. So each php release, uh probably each php releases, bringing slight performance optimization to us anyway, but php.

D

It contains the option of just-in-time compilation that that has a really high potential for performance improvements. Once we will, we will be able to install magento on php8 like some middle point and php compatibility project. Probably we can.

D

We can check it and do some metrics.

D

However, the side effects of all the features that we are getting are always backwards and compatible changes, and there are lots of them in php8.

D

They are as low as string to number comparison was changed and and lots of other stuff, like a lot of changes in exceptions, warnings notices, like lots of notices, became warnings, lots of warnings between exceptions, so we'll have better error reporting. I guess also configuration was changed and so on uh now, let's uh take a look on.

D

How do we actually update to php8, while this presentation is based on uh on the plan that I have prepared for magento 2, but I believe these the approaches described here are applicable for any project and actually for any upgrade to any php version, because basically the approach is the same.

D

First of all, we cannot do everything at once. We have to divide the upgrade process, at least in two stages, because at the beginning, from this crazy project cannot be uh cannot be executed. So we cannot uh have the runtime project on the latest php version. First, something should be done and this something is dependencies update.

D

So all the composer composer dependencies have to be updated to php compatible version because before we can install whole project on php and while we have depending update uh while we are, we are updating dependencies.

D

uh The uh we still cannot check the project and run time and test it with these extensions, but we can start working on the code base using the static analysis, so static analysis, static tests, probably the the migration guide from php, can help us to start preparing the code base, but it should be finalized on the second stage. So the second stage is when all the dependencies are updated.

D

We can install the project in our infrastructure, run tests on the on actually php 8, stabilize the project and as the main project in this case, magenta 2 is earlier compatible with php.

D

We can start working on the extensions that have the main project as dependency for them and complete the work on the extensions and test everything together and finally, finalize the upgrade and and yeah uh so talking about magento 2. At the beginning of of the project, there were 32 dependencies that are already compatible with php 23 dependencies.

D

Had to be updated, some of them are already updated in scope of php compatibility and community project, and there are 12 dependencies that are still working on their php compatible version.

D

So that's that's also something to take a look to pay attention to, because not all the dependencies, not all the projects, are also providing the compatible version for new php version at once.

D

The challenges that we faced during the pro during the first month of the project running is well, first of all backward and compatibility, and there we had to address double backward incompatibility when we are talking about dependencies, because, first of all, php 8 is bringing some backward incompatible changes and, secondly, there are even more backward incompatible changes coming from the new versions of dependency section, because we have to update to the php compatible version of dependency.

D

That sometimes is a major version upgrade and apart from just php compatibility, there are additional changes which are backwards and compatible, and we have to adjust to them.

D

One of the challenges you can face, as well as limited php version compatibility, for example, on our project. We would like so right now. The project is based on the code, from 2.4 to the for develop branch and to that for developers supporting php 7374, and we are adding the compatibility of php8.

D

That means that we have to be compatible with all the three versions and not all the dependencies can be compatible with all these three versions. In this case, we'll need to to support several versions of the same.

D

D

D

And that's basically applies to to all the dependency updates, because if we are working even on the last dependency update, we still cannot test the project in the well, probably on the last one we can, but one before the last one still cannot test it on php. uh So this can be uh so. The this risk can be reduced by testing the project on the previous php version, in combination with some static tests that are covering uh changes of php.

D

One big challenge is also the cross-dependency between the dependencies.

D

That's that and limited compatibility between them and limited compatibility of the dependencies to the php as well. All this in combination makes tasks decomposition for such upgrade much more complicated, so there are dependencies.

D

We have to wait until uh certain packages to be upgraded, updated before we start other work, composer, lock, conflicts so yeah. I do remember that we, we probably had a conversation. I believe that that max was also participating in this like if we need composer lock in the repository in general. However, in in scope of the php 8 upgrade, I think that composer log does help to ensure that there are no side effects from some patch updates of of of additional dependencies.

D

While you are focusing on some specific dependency update, but still if we are working in parallel on several uh dependencies to be updated, there will be composer log conflicts, just something that needs to be uh faced uh during such project execution and finally, the stale projects.

D

So the technology is evolving really fast, new projects are emerging and old projects sometimes can be abundant, so there is always a risk when you're updating to the new php version, for example, that your dependency will not release a php, a new php, compatible version.

D

In this case, uh different approaches can be applied uh like switching to more active project or maybe helping out with this project. If this this is open source and basically that's uh that's what we used to to work with in in community engineering team right or maybe just elimination, elimination of this dependency.

D

uh The tips for updating dependency, first of all, before starting the investigation before starting an update of certain package, take a look if they have a dependency migration guide. uh Usually, usually the packages do provide such documentation.

D

There is very nice migration guide for most of laminas components, especially for major versions, and that can save you a lot of time additionally to analyze that cross-dependency to see to plan for dependencies update on the project. uh The dependency trio dependency graph is very useful and there is a nice feature on github website. That's github dependency graph check it out. It's turned on by default for open source repositories for private repositories, it has to be turned on in settings and so and they are displaying a really nice dependency graph.

D

Navigatable right on the website for sure you can generate the dependency tree locally. Using composer depends on minus minus 3 and that can generate the dependency for dependency, 3 for whole project or just for specific package as well.

D

Now to the code base update the code base for codebase update, I see three stages.

D

The first, as I already described, is just while the dependency is not updated. The code base can still be tweaked. We still can start work on the code base. Using the static analysis, static testing, then, when the project can be executed in the runtime, we can use tests on runtime or just exit your test and run on and stabilizing.

D

The latest version ensure that it is working and, finally, all the extensions can be connected and the whole project with all the extensions can be tested as a whole and finalized the challenges for code base update is uh yeah, so static analysis can be used from the first stages. However, static tests uh are also a project and they have to be also adjusted to the latest php version.

D

In our case, we are using a php compatibility project for for our static tests for php8 compatibility project and they still don't release the stub haven't released a stable version covering the php 8. The tests for php changes.

D

That's why we are using a developed version right now that that is providing still a lot still a lot of valuable information, but yeah.

D

That's that's something so static tests have to be prepared as well to use them and yeah codebase cannot be fully tested and changes to con code based cannot be fully tested on the target php version until the final dependency is updated and as for the tips as for dependencies migration guide in this case, php is providing a very nice migration guide so uh also before going into investigation how to replace, search and functionality, how to refactor it for the compatibility check out the php migration guide there are.

D

There are brilliant examples and lots of useful information that can save time for static tests. Even if static tests are not available. Development version of static tests can give you the information right. That's you don't need to wait until until this time stable release that just a static test it still its output needs to be evaluated, so it can help you a lot and finally, the uh not only our or not only your projects is going to go into update to php.

D

We all are facing this update in the php world and probably we are facing the similar issues so interacting with community. Maybe posting a problem that you faced or looking if somebody has communicated about this problem is helpful and finally, a closer look at the project itself.

D

So project is based on peach. We are working, php, develop branch, the php develop is a name and this branch is available for not just for magenta core repositories but for all extensions. All magento projects, basically- and this branch is based on 2.4, developing a synchronizing student for develop branch for now. So the compatibility on this branch is php, 7.3 7.4 and we are working on 8.0 compatibility.

D

The documentation can be found on magento magento to repository wiki, just go to the github wiki, and so you can easily find the project page in the sidebar on the right side there is, there are links for documentation for all the projects include, including php compatibility and also, if you have, if you are not watching the magento dev blog. Yet I recommend you to do so, because we have already done a couple of updates on the project on magento dev blog.

D

We are also doing and planning to do regular uh updates on all the projects we are working with. So a lot of useful information is coming to magento devblog. uh If you don't know how to find it, just google magento block, I think it will be. It will appear somewhere on top of the of the results.

D

So as all other magento projects, we have a separate uh dashboards track our backlog and track our pull request.

D

If you are interested in uh in our progress or interested in participation in the project, just go to the github magento magento 2 switch to project stop and you will find php compatibility, backlog and php compatibility prs for pull requests boards uh there, where you can see all the uh tasks and all the pull requests in appropriate states and, as you can see on the screenshot here, the the work is actually going on and a lot of things are are in progress.

D

A lot of things are already done right now for the project, and I would like to thank you to to the contributors who have participated at the php compatibility project from the first from the early stages.

D

Well done, thank you guys and also thanks to the contribution partners who are involved.

D

That's very nice when the when the company is engaging the developers uh in in the open source collaboration that I believe is beneficial for everybody and yeah. If you're, if you're working for a solution partner or if you're a solution, partner and you're still not on our contribution program, please contact us and we will. We will enroll you and yeah. Basically, that's it! I I would.

D

I would ask you to please scan this barcode. It will lead you to the survey where you can provide your opinion on php 8 features, and so what php features should are you interested in and what should be used in magento? We will use this later on, because php compatibility is just the first step, and uh sooner or later we will have to go for the support.

D

Thank you for your attention and I am open for your questions.

A

Thank you so yeah. Thank you for sharing the plans for being php8 to the magenta, so guys join the public project share your vision, uh check the dev blog, so we will have more things coming here so yeah as we have not so much time left. So, let's move on to the next topic answer you please check the chat uh regarding the questions and.

D

I will try. I will answer the questions in the chat just post post them there: okay, okay, thank you.

A

So, let's move on to the next hot topic to the last one for today, so it's performance, optimizations and large catalogues, so vitaly go ahead.

C

Hello, yes, okay! So today we're going to talk about how to support 200 million esq's and even more so my presentation is divided into two parts. So the first one is about optimizations that we've done on a magenta core in order to support storefront side, and the other part is about indexation improvements. So, let's move on so yeah performance fixes over you. One of the most annoying problems that we faced recently was an issue with the customer sections.

C

That issue leads to really huge unnecessary web traffic. So if you see uh like the email, the picture right now so it shows like sixty percent of whole traffic is a customer section load. You definitely haven't this issue. I will. I would suggest you to check the latest releases. Everything is fixed regarding this. So recently we had like three or four requests in addition to one uh so yeah. If user come to your website, uh magento generated three additional or even four additional requests to magenta core, which is leads to unnecessary traffic.

C

So that's how it works right now, it's already fixed so right now it's no invalidation without proper trigger and radiation happens. Only based on rules- and it is described on section xml,.

C

We have we made some changes in the core logic. So now, customer sections are independent from the private cookie version and card section card rewards only on on store and website changes. So right now, if you open magento websites, there will be no additional, no additional http request to the magento core. So you will see much more like better performance because of that the next one is we present at level 2 cache.

C

So generally it saves traffic, so we transfer up to 10 times less less less data. We store this cache locally and we prevent we prevent already slow addition. Let's see how it looks like so on the left side, we see the common cache adapter. The yellow. uh The yellow line is a radius on the right side. You can see uh like the same testing approach, but with a level two cache.

C

We see that radius response become much better. Let's see how we prove it even more so yes, this is. This is uh how traffic goes so, with the common cache adapter. We transfer like up to 100 megabytes per second and with level 2 cache only 800 kilobytes, so yeah. It's it's all also leads to slightly better memory usage.

C

So so here is the traffic of live client. We applied level two cache to our white client and you can see that uh you can barely see the radius response, uh so the yeah, the green arrow, shows you the timeline when we applied it and on a throughput site, you can see that throughput winds up really easily and radius is not like the bottleneck anymore, and so how even get a better result.

C

So we introduce a preload feature, so that means so generally speaking, like magenta lots, more than 50 or even 16, sometimes 100 uh independent piece from reddish side, and we made some preload features that allows you to generate a lot of the data in in one request: let's see the difference so, on the left side, common cache on the middle level, two cache and on the right side, you see that radius is not like a thing anymore and we face the php which shows us like. We have a better throughput.

C

We have less radius response time. So please go check the commentation. It will help you to support really each catalog.

C

Also, we we have two more features regarding this. We have style cache where what is doing quite it fixes some common issue with the lock mechanism. So if you struggling with a wizard with actually locks, you can use them, so what style cache do is like. Meanwhile, the new cache is generating its service, a style one, and it's really like it.

C

It will not take a lot of amount of time, but for some client it's really like a business case and the last one feature about this is a parallel generation, so it doesn't mean that blocks is never squared and meanwhile one process is generating.

C

The other one is allowed to generate the cache in certain cache too so yeah we have a documentation about it and it will help you if you have like a large amount of block cash. If you have a large amount of config data.

C

And the next part so what's happening with indexation, how we support it from index site.

C

Is the profile that we used it's in the enterprise edition and you can see that we generated 6 million simple products, 100.

C

000 configurable products, 6 000 categories, so in in total, is 250, a million effective uh skus on our account. So previously the maximum maximum amount that we tested was 10 million.

C

Let's see what how it goes so at the beginning, we had issue with the product eav index and it it runs well. It runs like more than 12 hours, so I stop the processes, check the documentation and guess what it appears that after we we use elastic as uh like. So we have right now we have dependency on elasticsearch and with price with search index. Here on elasticsearch, we can just turn the eav indexer off yeah. We have it in a official documentation.

C

So if you're using magento, 2.4 and 2.4.12.4.2, you can just tune ev indexer off. If but you need yeah, you need to double check that your extension doesn't rely on av indexer because we eliminate all dependencies from magento core site.

C

So what's next, we struggled with category products index which runs around like three hours and a stock index. That's never ends. We wait like a couple days and I want to mention that we made this reindexation in a parallel.

B

C

We also have a documentation about it, so that our search, indexer and product price indexers are. They are split into dimensions, so you can use uh php stress advantage to make radixation in parallel mode.

C

Yeah, so stock index is the next issue and what we fixed yeah. We have small amount of batches like only.

C

200 products per one batch and we fixed some performance issues, so uh we found that stock index tried to re-index product types that even don't exist in the system. It's already fixed in 2.4.2 and also we found some uh require some mysql uh queries that were fixed. We used limit and offset in my sequel request and, as you guessed in a 10 million stable, it leads to performance issues, so we're rewriting.

C

And starting for from next release, 2.4.3 we support the batch sizes for the indexers that you can set in temp.php file. So the documentation is coming. It will be an official website. Also. We will write at devblog article for devblog, so yeah. It definitely will help you if you're struggling with some index here, you can go ahead and use.

B

C

And find the best the best numbers for you that works for you. So after we done this, three improve improvement, like fix the batch size, write, the secret mysql queries and eliminate god errors. We run stock index with this profile within 21 minutes, which is pretty fast. I guess in eight threads, so the next one as I showed you was category product indices. We also wrote some queries, but because, but category product indexer cannot run in a parallel mode.

C

That's why we obtain only 30 minutes up to 30 minutes improvement, but it's still a thing right and the final result is.

B

C

On I want to mention with 250 million esqueues 16 threads and, as you can see, the numbers work well.

C

Full price in the service run only for about 5 hours and search indexer runs only on runs only 18 minutes with 16 threads. We can go even further. So if you use power cloud environments, we test it on a split architecture, so you can use all your virtual cores for the threads. So in case you have like 25 or 40 virtual course. You can use them all for 40 threads.

C

It will do even faster and also you can always play with the batch sizes and just to set the needed options like to use your memory in a proper way, so it's possible to obtain even better results with it.

C

So yeah with all fixes before it's. The indexation takes like infinite amount of time and after it was like our target, was to do the whole index process below nine hours, and we achieved it.

C

Actually, uh that's it my site. Are you happy to answer any question.

A

uh Yeah, we have a few questions. Few questions in the chat, so the first one from jonathan. Is it possible to report customer sections improvement to 2.3 release, line.

C

Yeah, it's already fixed in 2.3.6.

A

Okay, so so it should be available like in the next release, okay and the next few questions from the constant team, the first one, uh okay, the first one, it's like about about the presentation. So, yes, we will. We will post the links to the presentation and they will be available after the meeting and also. The second question is: what is esq, it's virtually skew or what is it.

C

Yeah, it's a virtual sku, so we have 10 million skus. It means that we have 10 million products, 25 million years to use how how we count it. So, basically, basically it's amount of entities that we have in a product price indexer. So it's amount of energies that affects a price. How we count it. So we just multiply the products on a website amount and multiply it on customer group amount. That's how we count it.

A

Okay and the next question just just appears in the chat from safal uh what about performance with shared catalogs if we use it for more than 50 companies.

C

So basically, companies is the same, so shared catalyst is the same right now as a customer groups, and we test that we already tested 130 million esq's on b2b and we have even better performance result even better indexation performance results. So we run index here on that profile, like within five hours.

A

Yep sounds great. Okay, guys, I don't see any other questions in the chat. So maybe maybe you can ask them right now. We are right on time.

C

So the last one question is so that about customer admin grid index here, no, I don't have I'm sorry. I don't have any updates regarding this.

A

Okay, so if there is no question, so thank you guys for attending the meeting so yeah see you soon.

D

Thanks guys, thanks guys, thank you for hanging out with us bye, bye.

B

Thank you have a nice day. Thank you, guys, bye, bye,.

A

See you guys thank you.

D