Ceph Teuthology, 16 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Teuthology Training: Developing Tests

Description

* Notes: https://pad.ceph.com/p/testing-ceph-2021/timeslider#1647

* Ceph Developer Guide: https://docs.ceph.com/en/latest/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-intro/

* Ceph Teuthology Documentation: https://docs.ceph.com/projects/teuthology/en/latest/

* Ceph Teuthology project wiki page: https://tracker.ceph.com/projects/ceph/wiki/Teuthology

A

All right, let's look, we've got a stuff to start um so welcome to the three two thousand training session today, we're going to talk about how to develop tests and uh what kinds of what different kinds of tests we have and stuff like the other sessions, feel free to uh ask questions at any point: it's even better if these are more interactive rather than just just a presentation.

A

So uh if any questions about anything we've already covered or anything we're talking about today feel free to chime in.

A

So I thought we'd start with um little discussion about um what different kinds of tests exist and stuff, and why, when you want to use one or.

A

Another, so we've got a few different ways that you can use integration tests with stuff in terms of um simpler tests or unit tests. I won't be going over those in detail, but um generally, if you can write unit tests, uh it's quite helpful, since there are, of course much faster to execute much faster to get feedback from, but I'll be going over one more detail. But what what different ways that you can use to write integration tests.

A

So in general, there are a bunch of different ways here.

A

One of the most common ones is writing some kind of scripts that runs against the stuff cluster and a common one mistake is uh what we call a work unit. These live in the uh repository under the keyway work units directory.

A

They could be any kind of programming language, but bash and python are pretty common and since they're just running these commands against the cluster.

A

The idea is that if the script exits um zero, then as a success and if it exits non-zero, that's a failure and you can run these scripts against uh your local v-start environment or you can run them within too lg.

A

So, if you're running these intology toothology we'll go ahead and clone this, this repository at the version that you specified and run the scripts from there.

A

So that means it's easy to run the right version of scripts in an upgraded scenario or um just make sure you're using this consistent version of the script and not getting something from from master when you meant to run a nautilus test, for example,.

A

Sometimes these scripts aren't really necessary to run tests, but sometimes we use them to add some extra logic. For example, there is a wrapper. We have a number of um tests and stuff that use the details, framework that are like integration tests, using like different libraries like the radios with little ready, and these end up in in this bunch of the binaries called starting with a self underscore test underscore and then the test name.

A

So one example where we use a work unit to provide some extra functionalities with the libraries tests in this work unit instead of running each individual test, serially we're we're running all the tests with all the test, binaries in parallel and waiting for all their sub processes to exit.

A

So it must make that the test process that much.

A

Faster, as you can see, this is a typical kind of work unit. It's just a bash script which um uses set minus e so that the script exits uh um with narrow if any of the commands fail and minus x, so that you can see what what is actually being executed by the script so that when, if something does fail, you can tell what led up to that point and what may have caused.

A

Failure and when these are run by tothology, all the output from these scripts ends up going to standard out in standard error and is saved in the topology.

A

A

These, uh when you're, when the markings are run, they're always run with us. A fixed set of environment variables, representing this left version, the test directory that the technology has set up and a few other options that are commonly.

A

Used you can also specify your own environment variables to pass into these scripts.

A

For example, with the test the test crash script here, it uses the test directory environment variable that we saw earlier um that relies on topology to set that. So this is an example of one of the work units where it doesn't really make sense to run against v-start since it it's it's relying on on the technology setup of uh humans being running in a certain spot.

A

But most most of these work units are don't really care about the environment that much and can be run in in in v, star within v start originally or against any other cluster. Just fine.

A

For the test test crash uh script that we just looked at, uh this is the place that exists in the radar suite in the singleton uh sub subsweeter subscribers, which means that uh the singleton section um incorporates the kinds of tests that don't need to be run against a wide variety of configurations. So we have all the things that you need in one nail file with no extra combinations of other files at all, so it makes it a nice easy example to look at since it's all self-contained in one place.

A

This is kind of a typical test like this, where we first install the packages run.

A

The cluster in this case we're turning off some warnings that we would normally trigger and mark as errors, but we know we expect we're expecting that this test is going to cause crashes, so we're going to ignore those kinds of those warnings in this test, and then we have the working of task running on client.0, which is the only client where we have in this particular case, and it's specifying that we're running the work unit in their readers, slash testcrash.sh location.

A

These are specified as as relative to the work units directory in the step repository, and you can also give it just a directory name. You could say just videos and it would run all the scripts below that point in the tree.

A

You can find more information about any of these kinds of tasks by looking at their uh documentation in the in the python docs string for where they're implemented. So if you look go back to the um working a task here,.

A

You can see there's a few other options that you have. You can run tests on more than one client at once.

A

You can specify a particular version like a tag or a shot, one of the working scripts to run. You can pass arbitrary environment variables and configure the timeout.

A

Like by default, all these tests are run with us. Some sort of timeout and and um in hours, um looks like it defaults in three hours, just so that the the something gets stuck. You don't hang there forever and we exit and clean up eventually so work units are typically quite useful because they can be running in both the b-start and technology environment, and it gives you a lot of flexibility in terms of how you're running a test against yourself. Cluster.

A

A

The pieces that they're not as good at are things that require a direct interaction with the themes of the cluster like changing their running states or otherwise, manipulating the cluster, if you're, trying to like uh microsd's out or in or remove them from the cluster or add new demons. Those are pieces that are handled within technology itself. So for those kinds of situations you usually want to have something that hooks into a like a an existing pathology task or perhaps write a new task to handle those kinds of uh test cases.

A

Any questions so far.

A

All right I'll keep going so in another um common cases, scripts in the yellow files themselves. These are nice for, like shorter sort, simple tests where you maybe don't need to worry too much about any things between different separation branches, and you just want to execute a few commands.

A

So an example of this is in the sfadm upgrade suite where.

A

We use the sfadm shell task to run these commands inside of a a surf based container.

A

We pass again the sha1 from as an environment variable so that the we can we can use whatever shot one we're running against in these commands, and we have a little script here that goes ahead and pulls the orchestrator upgrade status when it's finished upgrading you go through and look at the processes just so that we have a record of what what the different uh demons are running and what the versions are get the ethology log.

A

Since all this all the standard, output and standard error from these these commands gets it goes into the toothology log, and then we uh go ahead and check that the linked number is exactly one uh version present and that it is the expected sha. One.

A

So if you need, if you don't need to customize too much if you just want to execute some commands- and this is a good way to go in terms of- is adding adding a few extra commands to a.

A

A

If you're, not using zip, adm, um there's an exec task which executes in our uh any commands on the given host, and it follows essentially the same format as this.

B

One question just is it's about the there is any place where we can find the templates for different, for example, in this case, it's just to execute several commands using the theft class that is started with fdm, but well. This is a specific test. Okay, so if we are trying, if we want to to see different examples, well, there is a folder with uh templates or examples of the of the test.

A

There's not a specific um template folder, but um right. What I tend to do is I grab through the suites directory uh for the like a given thing that I'm interested in. But if you look for, like all the tasks that are tested using a separate edm shell or using the exact task, you'll find a number of places where we're doing that.

B

um Okay and the list of different tasks that we can use. uh What the? Where recommendation, the commentation.

A

Yeah, so the documentation is all in the repository itself as a doc string for the each individual module I'm going to cover that a little bit later, but the in general.

A

um The every task is a module in either the cfqa tasks directory or in toothology itself.

B

A

B

A

Yeah, so you can you, can you can look through those um the file, the files and those those directories to see what is available.

B

A

B

A

Sure sure, okay I'll go into more detail on that later. Okay,.

A

A

Yeah there you'll you'll see you're looking through the test, suites that there are like, um like different formats and different styles, that are used in different places. uh So there's no one right way to do things.

A

Another form of testing up tests is the science best test runner form, uh despite his name. This was originally created for uh for running suffice list, but it runs many manager and dashboard tests these days as well.

A

Nice thing about this is that it runs exactly the same tests in toothology or against your research cluster using this v start runner task, which has some documentation at the top of it for how to do that locally.

A

But it's using this, the python unit test framework and with a kind of interface, be able to control talk to the cluster and and control it, whether it's running anthropology or in restart.

A

So this is quite nice, um since it gives you something that's very uh much more similar to a play. The unit test, type environment, but for integration, tests.

A

You'll see this in in the um in the sweets where it's uh represented by the type of fest test, runner task and, and you configure it to run a given module within the tasks directory and that module will have a bunch of these tests in it. You'll look at that format of those tests, real.

A

A

So this one in fact only has a single test, but they're um they'll hear it from some kind of uh test test based class. In this case it looks like we have an xfs tests based class and then this is add some extra.

A

This module quite nice because it does allow you to interact with the cluster more uh through these python tests, um but it doesn't support running like uh other tasks other than what's in your python test, though it wouldn't be so simple to add, for example, osd thrashing, which is a separate task in totology to this at this interface here.

A

But that might be an interesting feature.

A

A

Another kind of time to test is the upgrade tests. These tend to be formed mostly out of different different ways of configuring, um stuff and and running workloads against it via relatively um complicated structures in the suites themselves.

A

Typically, these will be doing things like upgrading um the cluster in parallel, while running some testing workload and they often see things like minus x, the sweet name, which means it's meant to run against whatever um upgrades to whatever version you specify. So, for example, if you run the octopus x suite now, you could run that against master, and I would upgrade from octopus to master. You can run specific and they would have created from octopus to pacific.

A

You'll see with these kinds of suites, we tend to um split them into different steps and number those steps. So since the technology is going to go through and to find the yellow files from each of these directories in in order of the directory name so zero first first, then one then two et cetera.

A

So, just looking at a little more depth, one of these we tend to first just install the cluster um specifying in the first part just what roles the cluster has. This means that there's um two three four five different nodes in this test. One of them has only a clients. uh One of them has only a monitor the rest, have monitors, managers, osps or just a mix of different.

A

Demons in the past we had we had to separate these out a bit uh when we were doing upgrades in a packaged environment so that we could uh upgrade easily upgrade one host, but not another one. For example, we need like we need the client to be on a separate host that we could run the older version of the clients against the upgraded cluster with us fadem. That is no longer an issue. Since result, all the demons are running in containers, so the version on the host doesn't matter.

A

Then we tend to.

A

Basic install of the cluster just setting things up before we upgrade and installing the packages in this case running the cluster with uh certain some extra configurations.

A

And making sure that we're all set up as a as an octopus baseline, panic cluster, it's I'm getting ready for further testing by also adding our energy w deem into the cluster. We can run tests against later.

A

You know, there's a lot of laser, creates, upgrade suites also have um the print task included, which is to add so that just lines the log at that point. So you can more easily see which section of the test you're you're in when you're examining the log.

A

Some more interesting part happens next, in the actual upgrade.

A

So here we're running the install.free task, which upgrades the packages on each of the nodes running monitor a b and c, but it doesn't do anything with the demons. Yet until we go ahead and use the sept.restart task to restart the demons with those new versions of the binaries, we tell it to restart a whole bunch of these demons, but not not all of them. You'll notice that we're missing the third manager and we're missing or the third monitor are missing a bunch of the osds.

A

So the idea is is to and use a situation where we have a part of the cluster upgraded and part and not upgraded and then run against a bunch of workloads against that.

A

Next step is.

A

Running a whole bunch of uh workloads while flashing the cluster, so we have this parallel task, which in this case is only one one thing, but this is a common kind of form that we use in upgrade suites, where we will often run more than one thing in parallel.

A

So that's why I think this is like around and you'll see this. This reference is um an arbitrary label that you would put somewhere else in your ammo files. So in this case it's it happens to be in the same. You know file, but it could. We could be adding additional tasks to the stress tasks list in in later files as well.

A

We started out with uh just um we're using the threshold to use task which runs in the background and makes a whole bunch of changes to the cluster like thinking it was used up and down.

A

Maybe in this case we're disabling objects to our tool test, so we're not running uh running those, but otherwise we would be um running like doing things uh to the osc's like exporting vg's from one and putting them to another or it might be increasing and decreasing, beat the number of pgs in the cluster uh changing the fullness settings so that the cluster thinks it's full when it's 50 full or 10 full. That kind of thing, let's try and just address the cluster in general.

A

You look at the next section, so here we are adding more two more tasks to this stress tasks: label. That means that these are all going to be running against this, um the cluster, when it's in this mixed state of partially upgraded. While this flash usb thrashing osd is going on.

A

So here we're just running a bunch of radius benches. It provides some right load against the cluster.

A

And then we have a bunch of different kinds of workloads, we're adding to adding to tests like correctness.

A

Here we're again adding to the stress test, the stress tasks, label and we're going to run the.

A

Test doc, cls rvd work unit from the athletes branch.

A

Since we're still running a bunch of octopus osds, we want to make sure we're running a version of the test. That's compatible with that those older osds.

A

And you'll see that this is a part of the suite has a plus a filename plus in it. This means that every one of these email files will be added to the test to each test, but when the suites are constructed, all of these will be concatenated together.

A

And look at showing the directories separately if you're listing things in graphical order, you'll see five six and seven four, so we have the after. After all, those uh different kinds of tests are done and when the cluster is in mixed mode, we go ahead and finish the upgrading, the rest of the daemons.

A

So we upgrade the packages on the nodes that weren't upgraded, yet the ones running of d8 and client.0 and then restart the remaining demons that we haven't upgraded.

A

A

Yeah, let's finish off the upgrade with them. This is actually a sim link. It doesn't show us very well, but there's some links to an um like the qa directory. We have some common files that we use across different suites.

A

So if we go to um the violence, referencing qa releases, specific yaml, we'll see this is setting the cluster up. Now that has been fully upgraded to require specific osds and require specific clients.

A

This is the final step in completing the upgrade and then we after this point, we just wait for the cluster to become healthy.

A

For uh this is a bit of a leftover from earlier versions um where we, when we didn't, have messenger v2 enabled by default, so we were um after the upgrade is complete. We can find um we can finally upgrade enable messenger, b2 and turn off the warning about it.

A

And then, finally, after the class was all upgraded, then we've really completed all the changes we want to make to it and we run a few more uh correctness tests at the end to verify that it's still functioning.

A

A

So that's that's a that's one example of an upgrade suite, there's a a few other formats. um This is this that one also called a stress, split upgrade suite, which is one of the common ones where the cluster is previously left in a partially upgraded state.

A

For a long time, while we were running a bunch of workloads, the parallel for um the upgrade suites um it makes it does this less deterministically, where it's upgrading different demons and running workloads at exactly at the same time and not intentionally leaving the cluster stuck in a partially partial upgraded state.

A

But it's focused more on testing things like uh upgrading things, upgrading with demons in different orders to make sure that that doesn't have an impact on correctness.

A

So that's kind of a brief look at the upgrade suites um any questions on that or anything else.

C

Josh uh in the previous directory, I saw that there were some yaml files numbered zero one and one which was not numbered, so uh the number ones are sequentially executed and the not numbered ones are picked up randomly. Is that so.

A

They're all picked up in in uh lexicographic order, so uh in this case I think these ones are doing things um like adding some configuration parameters that don't need to be included at any particular time.

A

So um that's why they're not in a numbered list. These are all siblings, so they're not being shown by gita but they're, setting a general configuration that applies to the at the entire test.

A

C

As a plus sign does concatenation, I read somewhere percentage also does something, but.

A

Yeah the percent, my sense line is uh the default behavior, which means that um we're going to choose a combination of things from everything, but um in in the subject in this directory.

A

So if you, if you go to the there's, a readme in the qa directory, that explains more about exactly what these different kinds of files do.

A

So here is an example here.

A

With us so simplified um directory structure of a suite, so with it with the percent sign here, it means it's going to take uh the former matrix from all using one of the uh uh yellow file from from each subdirectory.

A

So it's got. We've got four different subdirectories, a or four different different files in this directory, a b c and d, and for each each test job is going to take one of each of those different facets.

C

A

D

Josh, I had a question if I am like writing a python script for testing some feature, and I want to test out that script locally or the best way to run. It would be using this fs test journal.

A

It it depends on um what what you're, what you're testing exactly like, what your python script is doing.

D

A

D

If I don't need to make any changes to the cluster like just a single node restart should be okay.

A

In that case, I'd probably make it a work unit, just put it in networking directory. um Okay.

A

It could be any kind of script that runs against the cluster. It could be python, it could be, um it could be go. It could be even b uh like plus plus, oh, maybe like that go, but I think we do actually um run make in those directories too. So if you have something- and that needs some compilation that can be done as well.

E

A

um See there's a question from matt in the chat about that. What makes third runner specific to ffs at this point- and I don't think, there's anything- that's very specific to south africa at this point. um Patrick correct, correct me if I'm wrong, but I think it's fairly general, since it's running manager and dashboard tests.

F

Yeah, I think just the name is there, because when john spray was working on this, it was uh just geared towards ffs, but I believe it's been generalized. um It should just probably be renamed at this point.

A

Any other questions at the.

A

A

Okay, well, uh I want to talk about a couple other kinds of tests um briefly, so we also have this um standalone test framework.

A

This was originally used in uh make check years ago, but we took it out of there, since it takes uh the tests that use this and tend to run for quite a long time they it's a kind of batch framework with a bunch of other methods for setting up clusters in different ways and running some more invasive testing on it things things where you might want to interact with a particular osd or particular ipg in a a certain environment.

A

So this is kind of similar to the work units in the sense that it can be easily run from the v-star environment or within toothology, but it is kind of setting up these clusters in a very unique way. That's not not used by anything else. So at this point uh I wouldn't recommend writing your tests in this with this format.

A

There's a readme file in it in the repository that describes how you can run it and locally and how you can add them to the suites in.

A

A

The final kind of test that I wanted to cover today was a fashion test, so.

E

A

Cover a lot of different ground and are one of the areas that help us find the most bugs and stuff.

A

These are implemented in toothology, with a few different tasks, which tend to have a similar kind of structure where they have a bunch of different ways that they manipulate the cluster.

A

Depending on which game you're talking about it could be mds's these and they let you kind of specify different weights for the different kinds of actions that they take.

A

So if you want, you might want to, for example, really stress pd splitting behavior, for that we have some configurations where you can do a whole bunch of pg splitting operations all the time and not so much other. Not so many other operations like taking overseas in and out or you might want to um run against an erasure, coded pool and really stress for kind of recovery around uh below inside the minimum size.

A

So you might you can change the configuration in terms of like how many usds uh you want to click you want to guarantee are, are still in the cluster. That sort of thing.

A

But that this is a maybe a good time to look at the totally tasks a little bit more detail.

A

So, as I was mentioning earlier for any of these uh tasks, you can find them in the um sf repository for the specific specific pieces, either in the qa tasks directory or it for more common pieces that are more general in the toothology tasks directory into validate repository.

A

So if you look at, for example, the threshosd's task, you can see.

A

A lot lot of the tasks are defined by a task method.

A

Which is uh decorated as a context manager. This simply means that they will have some kind of initial setup and then some kind of tear down at the end and in their dock string they'll describe the various kinds of options they.

A

Have um so for the swatch osd's, you can choose like how many osd's to make sure are in the cluster uh how how much delay there is between doing doing different things to the cluster.

A

A whole bunch of different options here, essentially different ways: you can affect the cluster and try to induce different kinds of behavior.

A

Interesting one is uh the power cycle: option, for example, which ends up using ipmi for the altimeter metal hosts to directly or virtually pull the plug on the system and start it back up again.

A

We have a specific speed that does that that does this, we run it on a little bit or less frequent basis, since uh in the past, when we've run it uh too often, it tends to actually kill the hardware.

A

But generally these these tasks will, in addition to listing out the configuration options. We'll also have a few examples so giving a simple one. Here we have the soft task and then match usds with a a few of these options, changed from the defaults.

G

I had a quick question actually so these tasks, these thrashing tests can be done for all the client uh level uh client or at the client level, as well right for rgw or for rbd, for example,.

A

So I believe, there's a fashion for the mirror demons I think for everyday, mirroring, um I'm not sure about rgw. If there's a similar sort of test for, for instance, demons.

A

Maybe if matt's here he would know.

E

What's the precise question.

A

Is there some kind of thrashing for rgw demons? I don't.

E

Think, though, I think it might be a gap.

D

The messenger failure injection work with rgw.

C

That's something that works elsewhere, at least.

E

I'm not aware I mean if so we haven't noticed, I haven't noticed it a bit, but we don't have a particular a thrash scenario, specific character that I know of this has been crafted too, are due to rgb demons. Are they you know they assemble things or other things? They don't currently have concepts like that, but, as time goes on, we'll probably want look more at that idea.

A

Yeah, the messenger failure injection and delay injection, something that's pretty generic that, um like, I think I think that could be certainly enabled um without much uh work there. Just adding those configurations to.

A

A

For rbd, there's like that, rvd mirror thrash task that does similar things to the redeemer demons.

A

So this is actually a good example.

A

Of uh like a new style task, um I don't know it's a little though so. There's um you'll see the the original um style of tasks was that they were defined using a task method or other methods that were context. Managers within a.

F

A

Module there's also a newer style where you can inherit from a task class and influence some methods to represent that the kind of setup tear down and active activity stage um in the in the old style. You know all the setup is everything before you get to this yield statement um once you reach the yields, other tasks after this get get run and then and then once they're done and the geology starts cleaning up um it unwinds the context, managers and each task's.

A

Finally, section is running in order to do any cleanup, that's necessary, and in this case next stop the stop the thresher and revert to finish.

A

There's another style with with the object-oriented approach, where.

A

You instead, instead of using a context manager, you can make a subclass of this general task class and override a few of these methods.

A

So it has it um so you have a like. I said, a method, that's called before it executes. You have the begin method for the main functionality functioning of the task. In the fashion case, this would be empty. That's not doing anything in the.

A

During during the begin phase, it is that's effectively the this would be used more for things that run in the foreground, like I say, the radius bench task to actually run the radius bench command and then the end and tear down phases that clean up and set story.

A

Actually it's original, please.

A

Nice thing about the this option-oriented way of defining these is: it does have some helper methods that make things a little bit easier.

A

um There's an apply overrides method, which is a common kind of structure that we use in a lot of tasks where you can specify like these overrides at the top level of your configuration so pretty much from any emo file. You can add some overrides that would.

A

Apply to any kind of to a lot of tasks um configuration so, for example, we use this a lot to apply um different kinds of stuff configuration settings to the stuff task or to change the version that's being installed for the install task, but in general it allows you to kind of very easily use the same kind of functionality.

A

A

An example of this, the new style task is the console log task into the intuology, which sets up logging from the consoles of the machines and saving that log. While the test is.

A

A

So it sets up by just setting up the data structures and then, when it begins, it starts collecting the.

A

A

A

I should explain the concept of remotes here so once you sell, these is working with these test nodes.

A

It's usually uh accessing them via this um interface and geology called the the orchestra library, which is kind of a wrapper around um a python ssh library. So you can run all kinds of commands over ssh and.

A

Presents is these: each machine is, is a remote um so we're looking at looping through each of the machines here and for each one.

A

We're um got some console methods implemented to connect to the console and get the log from it and whenever you're running something on the remote it'll return, that process objects that you can. You can save and you can kill or wait for it to exit later, but by default that um near anything, it'll be asynchronous.

A

Using this kind of method,.

A

Looking at another kind of example,.

A

I mean the exact task is a simple one. This is this. Is the task list lets you. I kind of execute arbitrary commands on a given.

A

Machine so note that um these all have the same kinds of two parameters: the context and the config configuration context is kind of like a global context. Object that's passed through each task, as it executes and uh tasks can kind of store extra. Their own kind of extra configuration there uh when, like the sf or selfie m task, runs they'll, add their information about the um running demons and demons from different clusters. Since you can set up more than one cluster to this context object.

A

So the later tasks can refer to that and understand like what's the configuration for this demon and for this fbi conf or they can um go and inspect that a particular cluster and operate on it on it. The other cluster, if they want to, but you can use this context, object to kind of uh store state that you want to have accessible to later tasks as well.

A

I think the rgw task is a good example of that, where it's it stashes away like things like the uh ports, the addresses of the rgw's that it's setting up, but the later later, tasks that run tests against large w um nowhere to find it and the configuration that each task gets. That's the um this section of the yaml, so everything below the task piece is going into that configuration variable. So in this case this would be a dictionary where the key client a and the value which is a list of two.

A

A

Let's look at the implementation of this task, we're just logging what we're going to do, making sure that our configuration is dictionary as expected.

A

If we don't have a particular.

A

Role specified to run this, we use if we can say that we can say all to run this command on every role.

A

Then we loop through the the roles in our configuration and the list of commands to execute on them on them, and this is a pretty common pattern. You'll see a lot where we look at the context.cluster object, which represents the the nodes in the cluster and the demons that are running on them, which is generated from the roles in your test configuration.

A

Let me transfer, um we can try to execute these commands just on a particular role like say, say: uh monitor, modded, a or osd at zero, um we're going to filter that down to just monitor that a for example.

A

And get their remote machines that represent just that that one monitor.

A

And then run that command run the command on that single remote.

A

Passing in the test test directory as an environment variable in case, we need to use that within the commands and then just bashing passing the full command along to bash directly.

A

And since this task doesn't do anything that needs any kind of teardown or anything, it's just executing some commands. This one is actually not marked as a context manager, so it doesn't have any kind of teardown.

A

A

So that's a brief look at um that's a kind of structure of tasks within toothology. There are a lot of different ones here. um I think there's a a brief guide in the totality documentation.

A

Linked here, which kind of describes a few of the common tasks that we have and some of the common weight common ways that we that they can be used. But I think some of the best best way to learn about this is to look at what's being used in actual suites and.

A

Examining what what kind of tasks being used and looking at their documentation in their in that in the tasks implementation file?

A

I guess one other thing worth noting is that um we saw like this f dot, restart task used earlier in the upgrade suites things like that um that when you see like a dot in the task name, that means it's a sub. That's another method within the um the subtask.

A

So the sf task would have a restart method. That also has is a context manager and implements that self.restart functionality.

A

Taking a brief look at.

F

A

You can see it's psychotic manager as well, so that means that this this function can be referred to in yaml files, for example, saying step.restart and then specifying a list of demons.

A

Now we're coming up on the end of the time pretty soon, so I want to leave some more time for questions um any more questions right now. Okay, it looks like um we probably don't have a lot of time to cover um the rest of this. So we'll continue.

A

This topic a little bit next week, um looking at how to actually run technology while you're developing a test, and perhaps a little bit more going into a little bit more depth about what kinds of uh what kind of code you need to write or what kinds of options you have um when you, when you are writing a new test and to ethology in a task in particular.

A

So thanks everybody for joining and so I'll see you next week feel free to uh send any questions. You have a that, you think of later, I'm always happy to help.