Kubernetes SIG Node, 8 Mar 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Node CI 20230308

Description

SIG Node CI weekly meeting. Agenda and notes: https://docs.google.com/document/d/1fb-ugvgdSVIkkuJ388_nhp2pBTy_4HEVg5848Xy7n5U/edit#heading=h.2v8vzknys4nk

GMT20230308-180458_Recording_1524x1120.mp4

A

Hello, hello, it's March, 8th 2023, uh it's a CI subgroup meeting called signals. Welcome everybody. We have few items on agenda today. um First, you I think it's like this. One.

A

Yeah I wanted to let everybody know that uh there is this effort uh going on. uh We want to run rm64 tests, I mean test on item 64 machines. Right now we have like we have these machines available in uh in gcp, where we run all the tests, so uh it shouldn't be too complicated to edit and Ike I think will be working on that. So, if anybody interested to help Ike, please join the party.

B

I'm just curious: what is the speed difference you see in the test running on r164 missions that you have over there.

A

B

You mean workout Yeah, well, yeah, the performance of that uh is it significantly better in the gke. I've, never tried it so to.

A

Be fair, I, don't know the numbers I I know that there are workloads that run faster. I know some workloads are not running faster because they just compiled not optimized for rm64., uh but I know that uh you can find workload that runs significantly faster yeah.

B

Kubernetes built significantly faster on my M run, with the I'm running a Ubuntu on uh in a UTM Camu, so I'm expecting really good things here. This is great work.

A

Yeah and right now some vendors already running and including Google like we run uh rm64 kubernetes on arm 64.. It's already supported this effort is just to make sure that the tests are on uh early, so we will get yeah regressions, 464. uh shift effort.

A

Okay, so you can see yourself if you want to see progress and such okay.

A

um This one is another um I think that yeah those APR uh Clayton um Clayton is Agent and implementing some Logic on uh removing ports that are failed, so like removing unknown State and like making sure that what's been eliminated and to make sure that they are calculating adding couple metrics.

A

So now we we will have in our log dump script, um we'll have code that will collect some metrics from nodes when test is completed same way as we collect a set of like all ports running and all events from the supports and all runs from the node. So you can add some information on the end of the test, especially on failed tests, and this one will collect some metrics. It's a very good Improvement. But it reminds me of this thing that happens at some point.

A

We switched from log dump script from KK repository to lockdown script in testing free Repository, it's a good switch because we want to minimize amount of test code and KK repository, but at the same time it had some incompatibility. So there is a flag that you need to apply to your test job to use this new script and I was wondering if anybody here is interested in this task to just apply this flag to whatever signal test. We have I know that set of views that are still using old Scripts.

A

If you're interested in uh some task by getting started. Tasks is a good opportunity to contribute.

C

It sounds sounds like a simple and educational one. uh Why don't you send it to me? Is.

D

C

Urgent because I, it's like we.

A

Live like that like for maybe when I joined it's this ever started so like before my time, uh two years at least so it's not urgent, but more and more tests will rely on this. I mean if we want this metrics. We want us a lockdown script, so yeah.

C

A

I, don't think uh we have.

A

Github but I can create one.

A

Okay, uh Francesca. Do you want to talk about your item? Yes,.

E

So um this is about really to get the feeling about this forum if there are any concerns or comments about uh promoting some of the existing for resources. End-To-End test, node conformance about the process, I had an item, but it's on me I will do the research and talk with cigarch or any other substituted, but if there are concerns from this forum, so why? Why do that at all?

E

Because we need to enable the endpoint on Windows and the easiest way to have tests running on Linux and also on Windows is to have them as not conformance, which is arguably a good thing in general, and a good thing to explore in general, so I think provided a tester. Stable should be no problem but I'm here to get the the feeling. The pills from this form definition.

A

Not conformance is not owned by cigarch it's owned by us, so we can. We can decide and um right. There was a cab that uh um was accepted in one of the previous releases and unfortunately, we didn't follow through um I mean we pulled through partially, so um it changed definition of not conformance a little bit uh clarifying what it means and not conformance means that this functionality is supposed to be working everywhere, so it doesn't need to have special like uh setup or like special components installed in addition to kubernetes.

A

So I think this is this feels pretty well with not conformance I, don't see any issue with that 4ga, it should be. It can be not conformance even in data yeah,.

E

Yeah and then this this thing just popped out because of the ga, the ga conversation so yeah. There is the research manager tests in general, which the related to produce routes, and so, in some cases they produce routes. There are dependencies on extra components. This is why I'm talking a subset of them, but such a subset exists and we can totally move forward so yeah yeah. Thank you for the updates.

A

Heavily run everywhere so make sure that they're green before you yeah.

E

Absolutely absolutely I will update in the coming weeks. No worries thanks.

A

Thank you, yeah I think you. This is probably sources you're, uncovering so many things like uh this promotion and support on Windows, plus uh scalability and DDOS protection. Yes,.

F

A

Pioneer yeah no uh new endpoints yeah.

E

Yeah, nice, okay,.

A

Yeah uh I don't know if Mike is on the call here.

A

G

A

Yeah I just wanted uh to ask Okay, so this is um as discussed last week. uh We don't have like somehow we lost also the test coverage so on all the releases on like 124 and up and Mike said sign up to support it. uh So we have first job already here right. Can you talk more yeah.

G

Solve this job, basically, what it does is it's running.

A

G

Now it's just running conformant with no conformance tests. Eventually we could. We could uh increase the conversion this, but I wanted to make it something that's consistently green I would um I would like to maybe ideally onboard ethereal and some other tests, but until they're, uh unstable I would rather have them outside um this.

F

Is the first one.

G

Of other jobs that I plan to have I plan to test all of the supported GTA, all of the supporting kubernetes versions in in open source, which are 124 125 126.. We already run a test for, for uh whatever, is in master right now. So it doesn't make sense to have a 127 branch.

D

Eventually, once.

G

Once 1270 is released and uh new version is cut, then we can have a 127 branch um but yeah, that's that's about it I'm using uh cause. For now uh we can. We can expand this to Ubuntu as well I'm using a cost version that the costing officially supports that kubernetes version.

G

B

F

G

It's uh if it's 124, maybe this class 97 and each each uh job will have their own independent image. For this.

A

We have separate image file because in the past it was always like I remember what we did like we removed the remote uh was a remote runtime flag from kubernetes and it broke I. Think uh because we shared the image file between versions, right, I. Think it's a good Improvement here. uh Do you think we can move it out of this file into separate files? So you will have like fully separate 124 files, fully separate 125 files.

A

um Maybe that will improve situation further, so we can have a clear way to distinguish. This is 124, they think, and then we will move like remove this files and create new files. This will be 125 for testing right.

G

Yeah we could. Another thing we could do is like.

A

G

All the all these release Branch jobs into one single file, so it's a little bit clearer to find them out, uh but both options work yeah, ideally I added to the container, because I didn't want to add a file at the moment, but I think it makes sense to separate it.

A

Okay, so when you'll be aging, 125 and 126, think about like how how we can make it easier to maintain for future versions, definitely.

G

Yeah I plan to add the next jobs. Thank.

C

A

Renee, do you want to talk about your uh in place, yeah.

B

So um the CI enablement uh job that the pl that I had yesterday now it got merged. Thank you, sir k for merging that uh What uh I was observing to see how it does, and uh the first first runs that it did are looking red when I dug into it. It I found that about 11 or 12 of the 34 tests that run have failed, and uh the reason that shows up is the pods were pending we're expecting.

B

We have a timeout of 300 seconds for scheduling the pods and the timeout exceeded, and the pods were pending uh unschedulable because of not enough CPU. Now looking at the node, which has uh like the pull jobs, which we run with multiple like three or four worker nodes, this one is just a single node and it has two Milli CPUs a two thousand Milli CPUs, sorry, and that uh the tests were when the tests failed. They locked the parts that were running on that on that node and there were several.

B

So my guess is that this is happening because all the spots they when they run they take about 200, 300, Milli, CPUs and six or seven of them would exhaust what's there and if there are in parallel, they'll fail. So the immediate, the short the quick fix for now is to run them in serial and have this potential to fix. Pr I, just I'm, not sure if that is sufficient. I don't have much uh experience with these on the test side of things.

B

I just wanted someone to take a quick look and see if this is what I need to run all these tests in par in serial, as opposed to having those spots being scheduled at the same time, which is, in my opinion, causing the failure.

A

Yeah we had similar problems with probes like whenever you test props. You heavily depend on timing, and if your Port scheduling takes a while and then like uh Pro execution takes a while, and there is a CPU saturation. We always had this flakes that we fixed with timeout updates I I, really don't like the idea to do something serial just because we don't have resources on the nodes like uh it, doesn't feel healthy and it will lead to more expensive infrastructure.

A

B

So one of the tests in there the scheduling, scheduler test, has to be serial because at the end there was a potential flake uh causing issue that long way identified and Wong Chen. She submitted a PR that was merged yesterday as well. uh I am not sure if.

B

Increasing the number of nodes could I mean I could really lower uh bring down the number of uh possibly you know instead of allocating 300 400, which I just picked a number pulled out of the air. I can go with like 40 to 60 Milli CPUs, and that will help, but it still leaves this question mark of it'll. The flakiness is there.

A

um Go ahead. Sorry.

E

um You probably already checked that, but I want to mention just in case, because we we had this issue when doing the CPU manager tests, which want to allocate exclusive CPU. So now the CPU amount becomes a bottleneck, so the CI machines are actually quite weak, meaning they have two c two two entities that kubernetes called CPUs available. So I.

E

Just you probably already checked the amount of resources available and just wondering if that could be an issue, the meaning the machine is not doesn't touch and doesn't have enough actual CPU to actually allocate the Border. It's up about a timing issue no.

B

It's not timing, I I checked so in the uh in the bug. uh Let me just post the link to the bug the issue I created uh I did some initial analysis and I think the uh the? Where did it go? Okay, let me get it from this. One.

B

Oh give me one: second, okay: there we go.

A

B

uh Let me take a look yeah uh yeah. This is the one. So the issue, that's in there uh I think I created one.

B

I thought I had analysis posted on this one yeah.

A

While Renee is looking for that, I want to say how I appreciate the joke from just come with, like whatever kubernetes calls for the CPU yeah I.

E

Know what I meant is about it's not. uh It could be a hyper thread or a physical core, and it doesn't really matter what that entity um I'm. Just thinking a lot about it could be that if the the tests go go well, while in serial could be still related, because when they run serially, all the worker resources are available. So if you have two cores and two cores enough, then you are sure you are sure you have the full two cores for your tests. If they run alongside anything else, resources could be taken.

E

So that could be a good reason to to to to to propose running it in serial. Even though the serial has its drawbacks like, for example, it runs nightly, and you probably want to run them more often if possible, but if possible becomes the issue I'm afraid.

B

No, it is so it it's not in the issue. The analysis is in the pr description itself. uh So if you scroll down to the pr description, it's okay you'll see that I have a comment there, and this is exactly what it is. When I looked at, uh why these are sitting not scheduling the uh I looked at the node when it dumps out it tells you how much node allocutable it is elocitable is there and that is 2000 ml CPU.

B

So, as you said, uh this confirms that we have weak machines running over there and I couldn't I. The choice is, you know, add more add more nodes to the job which I really don't want to do, because that's going to cost more or in this case run it longer. So it's allocated either ways. You're gonna pay a little bit longer when I looked at uh the pull jobs after uh creating this PR, uh both the alpha, the alpha features, uh Alpha enabled and then my own In-Place rig size.

B

I expected those pull job timings to increase uh because I'm running them in serial I was another impression that they were running in parallel. But it looks like those timings have not changed, which means the CI jobs are somehow running in parallel, but the pull jobs are not. Is that the case.

A

Wait a second: what was the question? Is it uh sorry.

B

I'm, so when I uh created this PR, it runs through all a bunch of jobs. Right, uh it runs all the jobs I have added to it. The alpha alpha features job which runs the reporters in place, bodily size tests and, uh if you scroll down further at the bottom, you'll see the test results.

B

So there is that uh Alpha features cos. Alpha features that job uh that one runs all the In-Place pod resource tests as well and uh when I compared the timings it time it took, it took 1 or 29 minutes to run. uh I was expecting this to increase because you know I'm running in serial now. This was not the case before, but the time of this has not changed, so that suggests that this is already running in serial in the pull job, but in the CI job they run in parallel.

B

When you look at the ca job, it finishes in 30 about 30 minutes with those failures, so the cost is going to be that it's going to run three times longer. I guess um I would just want to with this PR I want to see validate, there's no way to validate the CI job. Unless this goes in right is there if I'm missing, maybe I missed. Something is.

A

There maybe serial CI job as well um so I think serial as I think is uh like serial tag is not serial setting, so it doesn't mean the test will run serially. You need to configure it. So that's why we have separate jobs for that.

B

Oh okay, okay, then, please take a look at my PR and see if that uh really maybe I need to do more than just a Serial tag.

A

So I think uh here you wouldn't find anything serial right, yeah! Oh thank you futurism out at all.

A

F

B

So this was not there before and uh it still took an hour and a half. So the timing on the pull job has not changed. The CI job, which is linked in the issue, uh you'll see that it runs in 30 minutes, which is great, but if it fails, then it's no good I'm expecting that. If we, if we do this, then it will take one and a half hours just at the pull job. But then we are also running this. uh The C group even job we're running it uh in.

B

uh If you look at click on the issue, then the jobs are there.

F

This issue yeah.

B

So the yeah, if you look at that, it's running 26 minutes with those failures and they all seem to be running in parallel, which is causing the out of CPU, can schedule issue and if we run them in serial. This is my theory. My hypothesis I just don't know if I'm on the right track with this, so I wanted someone to take a look at the pr, and if this is not, then fine, what density is needed to get the student in Syria.

A

Yeah, let me and try to if I'll find really quickly. I will let you know, but if not I'll just comment on PR sure sure yeah thanks.

B

B

About the the two jobs that we have, the C group V2 in place, uh CI runs every 24 hours and the V1 runs 48 hours, so um at least for now, I'm. Okay, with the we can at least try this out if this works, but if we know it from code review, it doesn't work then I'll, do it what we need to do to fix it, so, as you can see, most of them run about. 70 of them ran fine and the 30 percent, uh which failed are failing, because of uh not enough CPU.

B

Just scheduling failure.

A

Yeah afraid we increase the periodic uh for everything just to save on infrastructure costs. Right now, um it's running very high.

B

A

Yeah I couldn't find 3D quickly. So let me let me find out later: okay, okay,.

A

Okay, thank you. Let's go to something uh to the next item and we're out of agenda item. So let's go to uh triage.

A

Yeah I do some pretty hours today morning to remove everything that is clearly not related to CI group.

A

Okay. This is what we discussed last time and yeah more people looked at it. So.

A

We just need to take it a little more okay. I will double check on that, but I think this will this needs to be just merged? One more adjustment here.

H

I did have one question: I've not been able to find my test log, but when I run locally, it's fine using focus and stuff I can you know see my test run and show that fast, but I haven't been able to find them in the uh um the like the pr logs there where the test is actually on.

A

uh Okay, so I think we're on not conformance uh there.

H

Which jobs did you check? I might check the latest, one that are McKees, told me, which was uh Patrick several. um He said check.

A

H

Dte container d.

A

And I think this one: it's not conformance check.

C

Is this that one where you were getting them to run sequentially and check in the order for that sidecar project? Or is this different yeah cool.

H

You know I just wanted to make sure they were actually running before the pr went in.

C

A

So Focus here is uh yeah, so I think we can be. We can just add not conformance to that, and it will be it resources.

F

A

Just add the stack and to run it.

A

uh Do you want me to add comment right now.

A

Okay, we discussed it with David um and we uh may be a good idea to um add some timestamps as well to the logs, because this way we can test what started at there is a field called started at that this field value at least somehow close to the timestamp that it's supposed to be so that may be another addition, but I don't want to complicate this PR. Maybe it can come later so.

H

There actually is a timestamp on each log. If you scroll up there, I think where the sample yeah. Oh it's already here, yeah, it's just not it's not.

A

Parsed out I'm not easy to recognize the limit timestamps but yeah, okay, uh okay, thank.

H

A

Please yeah: this is what we discussed right.

D

A

Okay, uh is it work in progress um so yeah? We started reverting PR quite aggressively right now, because we have been like come up to release, so one of the PRS were reverted and that's why she, oh.

D

No, this one is.

A

Different, this one is back porting to 126.

D

Yeah this one was the fourth point that you see on the pr description is the one that was back ported, so I think this is in a way blocked as well, uh so I'm currently working on fixing that and then we can think about backboarding.

D

So yeah it doesn't need attention right now.

A

A

This one uh that's on AWS.

H

Yeah so I'm not I'm, not sure of uh status, whether it should like what the the end goal is really um so I know.

H

So there was a GCE runner on that ran a remote test on uh GCE and it was uh in use by some of the node tests, and then there was someone from the Sig infra I think that's working on setting up a product cluster on eks to use some of the AWS credits and try to remove some of that cost from GC over to AWS, um and so this was a PR I created sort of in support of that.

H

But then there's some discussion yesterday on Signal, whether um if that was a good idea or not or I guess maybe the the registry changes. Is it just gonna be enough, so so I'm not I've got no strong feelings about this. What's up uh just curious uh kind of what the what everyone else thinks.

A

um Is there any pieces of this PR that uh definitely just uh good improvements uh that we can take, so we can detangle like some improvements of scripts versus uh decision and moving tasks, yeah.

H

um Yeah I could so the refactoring of the runner, um the remote Runner out into like a GCE Runner, and then the SSH Runner um is probably worthwhile without the AWS change.

H

The like there's a bit of tension between like uh a desire to remove cloud provider code from kubernetes kubernetes and then a desire to also uh use AWS credits which might require keeping some of the code there and whether the route approaches move the test Runner entirely out of kubernetes kubernetes into a different repo, so that you can also remove the vendor dependencies um yeah. Let's so it's kind of where it's at the moment.

A

Okay, are you working with stick testing on that.

H

um Yes, yeah: there's a guy named Mohammed over there, that's working on uh setting up the the prowl cluster I, don't I, don't think, there's been a I guess, I'm kind of I'm a little bit concerned that they're that they're sort of approaching hey, let's build this cks crop cluster and do this and I don't think, there's been a lot of cross communication regarding the actual use of it.

A

Yeah I, don't think uh there is any concern from the I think people are afraid of um having something run on AWS and not any and not being supported at all. So like some infrastructure that start running but then like nobody looking at that, it's uh it's ready um concerning and we had situations like that before similar situations.

A

So I think that the biggest worry that needs to be to go but I think testing is the best place to resolve this for it, and then you can go and signals and like just yeah okay, so let me I mean if you can take some pieces out and like uh Republican, do it earlier?

A

uh I will make it original author for now, just because, because of the discussion, yeah.

H

I can look at just doing the refactoring first.

A

Okay, okay, this uh issue.

A

Charging six storage as well.

A

I wonder if it's a flake or you'll like 100 issue, let me go to test grid and see.

A

Yes, 20 calls a time. It is no flake.

A

Okay, um I can try to Ping six storage on that.

A

And protests um so yeah I think we looked at it last time last week, I'm not sure why it's not triage so yeah. What is happening is uh some um liveness and Readiness probe test started Trading uh with a strange condition, so sometimes liveness fails but uh and we detect that it fails, but then for the container is not being restarted in the same way. uh Sometimes suppose it is true expected number of restarted zero. You see like sometimes it expects not to restart, but it actually restarts, and sometimes it's opposite. It.

A

Opposite what is opposite? Yeah? um Okay?

A

Yes, it was about opposites. Oh.

A

Some reason I only see uh this kind of or number of restarts one, but are the same.

A

Okay I promise you I saw it Opposite as well uh anyway, so it's failing now and I think it may be critical now. So let me market as.

A

If anybody interested to take a look, please do so, otherwise we will catch this bug. When you look at the importance for the release.

C

um Can you see see me so I can look if I have time I don't want to sign up for it.

A

Don't be me if you want some assistance with initial investigation.

A

Okay, so we're done with this uh triage other two items.

A

Oh I need to move I'll move later. We look at all this uh I'll just move them and you can call them and we have 20 minutes for bugs. So we're done with test agenda. I will now switch to bugs. If you interested bugs just please stay and help out.

A

As you remember, last time we looked at uh important soon box, uh bye, Mike um this time, I I want to triage items so 26. um Maybe you can go through a few of them.

A

A

I mean coming much slower, I, don't know why.

F

F

H

Yes, I was curious on this one like so. What's the policy for like reported against the end of life, kubernetes.

A

We try to read it, but then, like uh we typically go uh needs information.

A

Because yeah, it also.

A

To investigate and it's a very old version, so what we'll do is, uh we will say they are needs information.

A

There is a special problem for those uh that we periodically pitch items from.

F

Okay, okay, yeah.

A

It's a rotten, maybe it was fished out of the bottom.

F

A

A

To nodes could lead to Cluster instability.

H

I think this is one where I landed PR, that updated dog saying don't do this.

H

So they were manually, updating, node name on their pods, to assign them to a node in the node. Didn't have enough resources.

A

um Do you think you can close it or it's still being discussed.

H

There was some discussion down below regarding, uh like a possible feature, enhancement to the API server, to allow you to disable um sort of like an admission hook to disable setting the node name on a pod. Unless you wrote a scheduler and I, don't think there's anything for like Cuba to do.

A

Okay, so it will be scheduling.

H

Yeah, perhaps I'll, maybe just transfer.

A

A

A

And I'm gonna recognizations how to debug it.

A

They provided some additional information.

F

I tend to accept it.

F

I think it's authentic vegetable, yeah, okay, there, it may be a good person.

A

um We're gonna need to find owners that.

F

A

Yeah, that's what I'm talking about it's like sourceful and started very recently.

F

Thankful for technical covers.

F

A

Okay, so it's related to cap.

F

A

I wonder how she know it was healthy. Okay,.

A

Let's work from as expected from signal perspective, I will remove signal out of here and move on.

A

So uh yeah it's and it's all related to this uh caps that c caps around command.

A

The containers and they're just even running status.

F

uh Is this related to the static part.

F

B

F

An open issue around static for behaving exactly like this.

A

It doesn't say it's static, I think it's regular ports, okay,.

C

Is there proof that the cube that was healthy.

A

Yeah there is no content. Coupon logs touched.

C

There's the command.

C

C

There was a ticket about this before related to static pods, but there was a lot of talk on the ticket and the conclusion is the source of Truth for the static pods is the file on the disk, not the mirror pod. You can see in the API server, so it should not respond to the delete for the mirror pot. If you want to see, I can try to find that ticket for you.

F

There's a recent comment by Clayton on the issue that I opened uh where he is proposing some solution to it, but yeah I don't have the number and yet I know.

C

That, whatever it is, I'd like to see it please, if.

F

You can send me.

C

The ticket or see me or something thanks: okay,.

A

Okay, so this is in this information.

A

uh Anybody want to be sit here just in case.

F

A

A

A

Stock, full coverage, lock source code I think it has everything.

A

Yeah I think I already opened it from very fast, like it's 2021.

A

So the problem is that uh uh if customers specified the incorrect command uh in 11, so regions probes- and we wouldn't really recognize a failure to run this probe as failure of a profiler. So we'll just say it's a a warning, so I already opened it mostly okay, um I love the chat so I already opened it uh mostly to as I thought that there is no information that this happens. But apparently there is I I thought that we hit in this.

A

uh There is a very interesting catch here, so in this code in exact property check that, uh if exit code present, then we go through this case and if it's not present, then we can uh get it. It's a timeout and we show its timeout but then like in opposite case. It's unknown and unknown will not result in any logs or events, so it will be swoled silently.

A

uh Apparently there is an exit code. When there is no like binary cannot be found. There is exit code, so it this, this branch of code will be used so we'll see. Failure, I think, I. Think this is what happened so um yeah I tried to reoperate and uh I couldn't find, uh couldn't get it to get into this branch of code, but I wonder if the if there are cases on this branch of Court will be executed.

A

So there is no exit code and it's not a timeout like how to get uh get uh CRI to return. This response I still want to investigate that uh I. Don't think.

A

I, don't think it's blocking on anything. It's a long-standing behavior.

F

F

H

Me add me to uh R Phillips.

F

A

Do you see the same behavior, um possibly.

H

uh Maybe not sure it's interesting to look at.

A

Yeah we have couple customers complaining about it. It's uh it's not! It's not natural, like uh that. um Pro like wrong executable will not result in test well in profile.

A

Foreign Ubuntu.

A

Let's see it passed, okay,.

H

Yes, so this is some of the other work that was being done to try to get to the CR test to run on eks. This is Mohammed. It was making changes to use just the base of my two Army.

F

A

H

Yeah I've run into like some some odd things about the test: I wonder if I should just document them or actually try to make changes like um like the node and and the node conformance test will fail if your disk is too large um the fail, if you're just too small. So it took me a minute.

F

Because you guys did this feature.

A

Timer Mission from controller manager.

A

It's a dog bug.

F

A

A

Okay, I think we only have three minutes left I, don't think we will be able to go through many more items. We have 19 left.

D

A

um If you have any other agent items, please pick up now, otherwise I think we can uh finish the meeting.

A

Nothing in the chat. Thank you. Everybody have a good rest of the day, bye, bye. Thank you. Bye.