Ceph Teuthology, 16 Mar 2021

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Teuthology Internals: Overview and Scheduling

Description

* Ceph Developer Guide: https://docs.ceph.com/en/latest/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-intro/

* Ceph Teuthology Documentation: https://docs.ceph.com/projects/teuthology/en/latest/

* Ceph Teuthology project wiki page: https://tracker.ceph.com/projects/ceph/wiki/Teuthology

A

All right looks like we've got a good group already. Let's kick this off so today we're going to be talking about um the the totology code more I'm going to. We can start by going over the general structure and then dive a bit more deeply into how running a suite works internally from the the scheduling of it and the to how it actually runs a bit.

A

So we're going to start off looking at the overall picture of the topology code, so in general, technology is composed of a number of different sub commands. You can see them listed in the technology, documentation.

A

Under docsistaff.com, you can see there's a whole bunch of different with commands here, the the main one, the topology command itself is what actually runs tests and tasks and the rest help with um a number of different um ancillary uh procedures, but the for all of these generally, um you can find underneath the scripts directory.

A

So if you want to know what kind of options something has you can look in here, for example, the domain topology.

A

Command line tool itself is in run.pi and we use a a nice python module called uh docopt to implement argument pricing and what this mitral does is. uh Instead of writing in code. What the options are?

A

You actually write them in this documentation string and it infers all the options it needs from the the usage help and help desk.

A

So this is actually defining the usage information that the user sees in addition to codifying what the options will be so for each one of these options, we'll end up with a an argument in python that we can access to see its value with uh things like dashes, replace with underscores.

A

And then, for each of these script files you'll see a very simple main function that calls doc ops on these options and then goes and runs the main function, usually with from another module within toolology.

A

So in this case we're calling into theology.run that mean.

A

So that the technology subdirectory is where most of the main code is here and you'll see a number of um files in the in in this directory itself, which are often related to those um cli commands. So the run.5, for example, that we were just looking at corresponds to the main topology command and if we look for the main function here, see the entry point to it.

A

So doc. Ops gives us an args dictionary, which contains contains um all the arguments that were passed in as well as the default values. If then, were passed and we're just accessing those here and instead of getting everything set up to actually start writing a job.

A

That's just a brief look at um where to find the code for any of these scripts in general, you can look at the uh scripts directory to see uh the particular command like, for example, lock um see the usage there. This one's actually using an old style of argument, parsing um using arc bars but it'll end up having a similar kind of thing where it calls into the a different modules main function to actually run the the bulk of the execution.

A

But another um and a commonly used elements in topology is the concept of accessing remote machines and uh interacting with them. This is all done through the orchestra, sub module or not not sub module, but.

A

Module within technology.

A

So orchestra has a few general concepts as a concept of a connection which it represents an ssh connection to a particular machine.

A

These are kind of set up when, when the job initially starts running and kept open, the entire time.

A

For certain kinds of tests, we need to reconnect like, if we're doing a power, failure, testing or power cycling, the machine or someone that when we after we need to reconnect. But this is just providing a kind of simple api around managing those ssh.

A

Connections um to represent a given node, we have this concept of a remote.

A

So then an orchestra remote, is again a kind of wrapping. The ssh client this case from um using the underlying ssh library from parameco, but um essentially it's letting you connect and reconnect to the same host and run commands on that host and get some information about thing.

A

It like its ip address, post, name, sort of.

A

A

There's also an aggregation of representation of multiple nodes called the cluster.

A

This is this. Is this can be used to execute the same command across multiple machines at once, and it's often doesn't often see it used in in the technology via the um context slide cluster attribute within the.

A

If you remember from last time, the context uh variable in pathology and um it serves as kind of a global object that aggregates state that's accessible throughout the the the entire program and there's a context like cluster object. That's part of that which is one of these classes, which represents the all the nodes that technology is talking to, and you can do things like filter to a cluster to a particular subset of roles.

A

For example, if you wanted to only address the osds or the nodes that are running clients or with nodes that are running mds or archw, you could do that so sometimes in the code, you'll see like context that cluster that that only and you say might say monday to only talk to the the talk to the node, that's running monitor.a and run some command there.

A

And, conversely, you can exclude a particular role when you um to create a new cluster object, that acts is able to run commands on everything other than a particular demon plant.

A

The other really helpful part of the orchestra is it's a abstraction around a running demon.

A

This is um in the demon subdirectory, we'll see. We have this concept of a demon group.

A

Which is a a collection of all the all the demons that are running and this this kind of object is created by the except for stuff adm tasks when they're setting up a cluster, so there's um it'll be well generally one daemon group for each stuff- cluster- that's that's ed, is set up and it'll happen a yet a demon object for each of those running or non-running demons for each osd for each monitor for each mds, for each rgw, et cetera,.

A

So these are essentially added into this dictionary internally, based on the uh generic role like the osv monitor mds, and that particular id like osg, like a 0 for osd 0 or a for monday.

A

And internally, the the uh demonize of um understanding, how how to um start, restart and stop the demon and have different kinds of modes.

A

You can kind of see how this is implemented more directly in the uh demonstrate object, which represents the individual daemon.

A

So typically, this will have like the commit the arguments that are used with this, this demon, the particular remote machine that it's it's running on the cluster, it's part of since you can have more than one such cluster.

A

It's my individual id, uh the python log and the general uh id id of the cluster. This is not generated when the cluster is is not boots up and finally, we get the process object which search out empty until they start the process later and extra uh keyword. Arguments for that are used when running the particular demon.

A

So this this demonstrate base class is used for experimental deployments for the seth task. Today, where we're manually running the the daemons in the foreground,.

A

Those wave methods to kind of restart um we start with new arguments, figure out whether the process is running or not, and to start and stop the demon.

A

Social studies assembly wrapper around uh restarted the checks, whether we're running or not, and issues a warning. If we are trying to start a already running demon and a lot of these have timeouts so that we don't end up waiting forever for something that that, uh if that, if the demon fails to come up or fail to stop for some.

A

A

So that, in this kind of bare metal case, we're simply taking the command arguments that we store for this demon, adding them to the command line and then using the remote.run method to run that command on a particular particular node, where it's supposed to be run.

A

This works a little bit differently in a separate m setup where we're actually we're running the demons via systemd, just like a regular uh stuff user, would we're not doing anything, especially in technology, so for his um the fading commands, rather than running the beams directly we're uh using system d commands to control them.

A

But you'll see we'll use that system cuddle to do while starting and restarting stopping, etc.

A

And then using the things like the delicate command to kill them and uh journal cuddles, so I get the actual logs from that from the the services.

A

So this is um a slow class of demon state. So it's in it's a it's overriding, pretty much all the methods there. It has the same basic structure.

A

Again, relying on the remote abstraction to access the machines over ssh and but in this case, running systemd commands to control the demons. Instead of running the commands.

A

A

One thing you'll see commonly used with them in any of these kinds of places. We're running, commands you'll see a few different.

A

Utilities used before running a lot of commands, so the these would be.

A

A

E adjust use limits batch groups which make sure that we are setting the um uh correct the the uh unit for core finals, for example, to unlimited, and so that we can get core dumps.

A

If you look at the, um for example, the dsf task you can see where the way those are used, this is uh back going back to the sec.

A

A

You'll see this is pretty typical of any commemorating technology. um We usually have these uh adjusted limits and stuff coverage helpers. The coverage helper is needed for uh code coverage for t plus, plus programs and stuff. um These aren't this isn't enabled currently, but the kind of building blocks are still there. If code coverage was something we wanted to resurrect, it could be done pretty without too much trouble and for the pyramidal deployments today, we're still using this demon helper command to um keep connected to the the cluster um and terminate the demons automatically.

A

If our ssh connection fails.

A

This makes it a little bit simpler to interact with the demons right directly, um but it's it's not it's not very helpful um in sfdm world, where we're doing everything through system uh systemd anyway,.

A

But it's that it's doing things like determining whether we're going to kill the process with a sig term or sig kill. That's why we have this extra demon signal argument here.

A

So those are some of the kind of basic building blocks that are used in teethology.

A

Any questions on these so.

A

A

All right, let's dive in a little bit more to how the scheduling mechanism works, then, and what that what what that looks like overall.

A

So this was um improved recently over the summer by a google summer code student who landed a new.

A

Way to process the the queue of jobs and and run them run them directly, so she's included a nice diagram here which shows the process overall.

A

So generally, when you're running it running a test suite, you will have um pushed your branch to a ci repository to kick off package builds in shaman, then you'll run the technology suite command to go ahead and schedule jobs based on using those packages.

A

A

Individual job json uh blobs, are stuck into a beanstalk d work hue, which is a priority queue that runs strictly in order.

A

And then, eventually, on another side in the central on a central um machine, the topology dispatcher command is running listening to the iq one for each machine type and taking jobs out of the queue and running the next highest. One and reporting its status to the um paddles api, which is a wrapper around postgres database, um which isn't also the back end for propito the web interface for viewing the the results of technology.

A

So once the dispatcher chooses to run a job, it locks the machines and then spawns a subprocess which is a dispatcher in in uh with that in supervisor mode which goes ahead and does the um time consuming work of re-imaging the machine, the newly locked machines and I'm actually running the tests.

A

In a in the background, so this is a bit of a departure from the earlier operation where, before, prior to this this summer, we had a uh a technology worker process instead of a dispatcher which was pulling things off the queue. But there are a whole bunch of them pulling things off the queue in parallel and they were all competing to acquire, lock some machines and run jobs at the same time.

A

So that meant that the priorities in the queue weren't strictly followed, and if you had a job that was trying to block more machines than others like say five or ten machines, it would often have a hard time um finding enough three machines and there would be jobs competing with it. That would only be waiting for two or three machines, for example.

A

That would get those machines locked um much sooner, but with this uh dispatcher since we're only trying to lock machines for one job at a time, that's no longer an issue, so we can run jobs easily now that require more machines and we're strictly following the order uh uh prior to priority order in the queue. So there's no more kind of priority. Inversions happening.

A

This is a float describing the um general uh process here, so the dispatcher is running continuously waiting for that for jobs to come in from the queue then locking machines and adding those machines into the configuration of the job under the target section and then invoking the topology dispatcher in supervisor mode, which then runs the technology commands actually run the test.

A

So the the supervisor is doing a few things here.

A

It's um it's starting out by re-imaging the machines, and I know that it's it's has. It's got it's uh bringing its own log file called supervisor.jobid.log in the jobs results directory. So if you ever have some kind of error occur before the job actually gets run, you may be able to debug it by looking at the supervisor log see if something went wrong, for example, during the re-imaging process,.

A

In the end of um constructing the tool, energy commands um based on the jobs, configuration and running the job and waiting for it to finish, and if it ends up timing out um we'll go ahead and the supervisor process itself will connect to the machines that the job was using copy the logs over to um and compress them and then clean everything up.

A

If the job runs correctly or or fails early without timing out, then all that kind of log handling and unlocking and cleaning up machines will be handled by the toothology command itself.

A

Jobs timing out in this way um are what make the job show up as uh dead in torpedo.

A

So another big improvement with the dispatcher is that we actually get logs when jobs go dead, whereas with the the old technology worker mode, it didn't have a separate process watching and and uh transferring logs, and we hit this timeout. So we weren't getting uh any any kind of vlogging uh beyond the technology log itself.

A

uh Prior to this.

A

So, let's uh take a quick look through how this works um internally.

A

So we'll start it with the.

A

That's that that first step here, where we're actually kicking off a job and scheduling it.

A

This is all handled by the suite command technology suite.

A

Good shot generally we'll take a suite that you've provided it and generate all the different combinations of email, fragments from that suite. Combine them together and add them to the to the queue.

A

So we're when we're scheduling we're always specifying a particular machine type. Currently we have one queue for machine type, so this might be smithy or diva or mira in the cpu lab.

A

We're kind of collecting um information about where we're gonna, what we're gonna do at the end of the job like we're, gonna email, the results somewhere.

A

And we're going to put the archives somewhere particular.

A

There's a bunch of handling for what, if we need to filter um to get information from our previous job, for example, if we're doing, if we're trying to rerun only the failed jobs from another run, we'd look at the private um previous runs descriptions to generate some filters automatically to get those same exact jobs again, and we end up. If we're doing a rerun again, um we'll be bringing out the the randomized seed from the archive version. There, too.

A

Then we'll be constructing. This run object to represent this um particular instance of a suite being scheduled and all the jobs inside of it and we'll go ahead and prepare and schedule all the jobs in that run.

A

So notice that everyone has a number of different pieces of metadata associated with it whenever we're taking off a suite like this, where we're specifying up a sf branch to run against and ends.

A

And as part of the scheduling process that the totality suite command is is figuring out which, um where it's going to get the tasks from the sex repo as well, this is called the the sweet repro and the sweet. Repro um can be also can be edited the same as this and the sf repo or it can be a different branch. For example, if you're trying to test how to change to uh one of these fidelity tasks, you might create a new branch and and run that against the master version of stuff. For.

A

Example um yeah another useful piece to look at here is um when we're creating a run and creating the initialize initial configuration um for every single job. We end up.

A

Generating the final version of configuration based on what's in the qa suite, but also including some basic template that is used for every single job and that is being scheduled and that's generated from this um placeholder dictionary here.

A

So every single job we'll have a branch and and show one specified for seth for pathology. um It's gonna have where to upload things um the machine type. This is running on for any kind of schedule, jobs we're always going to um you know after ourselves, and if we encounter an error, we're not going to leave the machines sitting around and we're going to have a particular os type inversion for each each job.

A

This might be like centro, s8 or ubuntu 2004. sort of thing and finally, we'll have a whole bunch of overrides.

A

Which provide extra configuration like turning on debug levels for a bunch of different subsystems, just uh in general, and so these are, these are being applied to every single job, you're scheduling.

A

So that's why every job um you'll see we'll have all of these overrides in its configuration file, in addition to any that are explicitly added by the suites.

A

A

Only two, um the seven geology branches are specified above we're, also going to be specifying the repository where we're looking for the um the sfqa tasks directory, which is the sweet repo and then the corresponding branch in sha. One that it's part of.

A

This is this all gets ends up getting turned into adjacent blob, along with the actual combination of emo files from the job and put into the beanstalk.

A

Queue uh the actual interaction with uh with beanstalk ends up happening in a second sub command call.

A

Called technology schedule which is implemented in the schedule.py file here.

A

This is a kind of a simple wrapper to interact with the queue directly and create the final configuration.

A

There's a couple of um kind of housekeeping things here, where we have a a special option for with a to represent. um Actually, we have a couple extra jobs in the free as part of each run. That represents whether we're just starting off the run, which is the this is the first in suite option um which lets the um the server side kind of do some initialization in terms of checking out the right versions of the the suite that we're running in particular for this.

A

For this this uh this run, I think at the specified shot one and the correspondingly. We have a extra job added at the end uh called the last and sweet job which pathology uses to and to know, when a run is over um when this one, when, when this job is finished, it'll kick off, um we can put any kind of post processing that we do, which would include um running the scrape type py to analyze the results and and see what the different kinds of failures were, um as well as sending in an email.

A

If, if that was configured with the summarize summarize results there.

A

But essentially this this command is taking in the configuration that the in the suite adding a few extra pieces of metadata to it like the person's speed last and sweet the email. The description of the job, which is generated from the file names in the suite.

A

The tube and the priority and then adding that going ahead and adding that to beanstalk.

A

The bean select is a very simple abstraction. It doesn't really provide too many methods. Essentially, you can add things to the queue and get things out and that's almost the entirety of it.

A

So this this job is the uh yml file like describing the configuration of this particular test, we're just adding it to the to the suite um at the jobs given priority level, and then we're reporting talking to paddles, to tell it that we've just queued a job.

A

So at that point the job is in the bean select queue and it's up to the dispatcher to take it out and actually start running. It.

A

If you do want to inspect these kinds of yaml files before you're running them, that the duology suite command has a dry run option, and if you use the verbose mode, it will spit out the full gamut blob that it would be scheduling. So you can see exactly what's going to be scheduled before you actually run it.

A

So, let's look at the server side and the dispatcher.

A

So where this has the two modes, so in the the main mode, we're running in a single loop and reading things in the queue and in supervisor mode, we're actually running the individual job.

A

So we'll look at the main mode. First.

A

So in this mode we're going to be listening to a single queue which is called a tube and beanstalk.

A

Terminology um we're going to have a particular archive directory where we're storing all the results of all the jobs that are running and we'll set up the login levels and and log file for this dispatcher particular.

A

We'll connect beanstalk and watch our particular tube, which means we're listening on for the um for new jobs coming out of a particular queue.

A

That's based on a machine type, so we'll have one dispatcher running for smithy's and one one for giveaways or um other kind of machine. We have and we'll make sure we have a fresh copy of the technology repository and the qa suite.

A

These are um internally just cloning, their repositories from github or within cpr, a mirror of them. Let's make sure we've got the latest code up today up to date, and we just got a um like an infinite loop that where we try to.

A

First check: if we have any outstanding results processes to to look at.

A

You've got a couple control options where, if you touch a certain file, you can make the dispatchers uh restart themselves or stop themselves. This is helpful if you want to.

A

If you want to make sure that the they're getting an updated version of toothology code, um you can touch a file and they'll since that they're, not in the middle of doing anything here that once they've finished processing, whatever they're doing at the moment, they'll go ahead and restart themselves check out the new version of pathology and continue continue running.

A

Now this is reading from the the beginnings.q um trying to get the next job insult will just give it to us directly in priority order. So we this would be the um next job in priority order from from beanstalk.

A

And if there is a 60 second timeout, so if the key is empty, then we wait for 60 seconds and continue um instead going through our usual text, see if we need to restart or anything like that before checking the cue again, but usually the queue is not empty, so we have a job to run.

A

um So you use the beanstalk api to what's called variate, which means that it's being processed.

A

um Then we'll start loading the job configuration.

A

uh From the yellow, parser and start setting it up, this is all handled in the prep job function, which is right. I still shared with the topology worker code.

A

This will go ahead and.

A

Take a look at that in a little more detail, it's going to prepare.

A

All the kind of things we need to run the job and make sure that it's valid at a very high level in terms of the branches being there as expected.

A

So we'll end up creating a a new path for this archive directory for this particular job and setting that as the archive path in the drive config and we'll figure out which tools you branch, and this is supposed to run on check out that version of technology.

A

If these branches aren't found for any reason, we'll raise the skipjob exception and continue on along our way, marking the job as dead in paddles.

A

This is um again cloning, the technology repository from git. If we need to and then doing we'll do the same thing for the cef branch and the suite repository.

A

We won't actually be fetching them quite yet. I think we will do that later down here, um but we're just we're verifying that everything um we have a kind of valid shot ones here and here we're actually going ahead and fetching the qa suite. So this will include all the.

A

The the tasks in from the stuff repository that we might need during this or this this job.

A

And since we're fetching it at this point, we're actually setting the the sweet path in the job, the jobs configuration so rather than referring to a branch anymore. We're going to be referring to the exact path. On the host that, where we've checked out the git repository.

A

If you were running topology manually, you could do this too, but you could be I'm setting a sweet path to refer to some version of stuff. You've had checked out already without needing to push your branch anywhere or or make any commits to it. You could just use that your local copy.

A

Directly and for any for any reason, we can't find the branch or the commit that was required.

A

Then we'll report the job as dead to battles or similarly, if we for some reason, we couldn't um finish cloning or bootstrapping to ethology in time again. Mark's job is dead and raised, skip jobs to the higher level process of the dispatcher.

A

Finally, returning the modified job configuration and the version of technology that we've we may have had to check out so that the dispatcher knows what to run uh where, where to run totally next.

A

So going back to the caller here, um so if, for any reason we uh had this, this job configuration was invalid. Like the branch wasn't there, we just raised the skipjob exception. We just continue on go and go, try the next job.

A

Otherwise, we'll start the process of locking machines.

A

This ends up talking to the um to paddles, um which which, in addition to storing the results, also stores the database of machines and which ones are locked and unlocked, and this call will will block until we. um If I have enough machines- and I have locked them.

A

Then we end, we finally have enough machines to run on, so we go ahead and launch a slight process of our technology. Dispatcher in supervisor mode serve as a watchdog for this individual test. Job um always running it for most would say, get some extra logging and using the specified version of topology.

A

This is the tooth technology path to the totality binaries and that archive directory that we've already created for this job.

A

We'll make sure that we have created this archive directory in fact write the job configuration there. This is the orange config.yml that you'll see in there in the results.

A

Now dump the ammo file, you know for the configuration there and then finally run this supervisor as a sub process. You know that this is uh asynchronous, so we're kicking off the job as a sub process, um but typically we won't actually see that um wait, wait for it to finish. Let's go ahead and continue with the next job, and the only blocking call here would be when we're actually going and trying to lock machines.

A

If we did have a problem spawning the supervisor itself, that's when we get an exception here and try to clean up and nuke the machines. If we locked any.

A

And if you successfully spawned the process to actually run the job, we'll go ahead and delete it from the queue and continue on with the next job.

A

um Any questions at this point before we dive deeper into the supervisor and the actual toothology process.

A

A

Okay, let's take a look at the supervisor process, then. So, if you recall that this is uh just basically taking the job configuration and setting up, we already have the machines. We need to run it now, so we we need to do the time-consuming steps of actually re-emitting those machines. These machines are in the targets in the drive configuration so we'll go ahead and uh re-image them this could this is a kind of abstraction uh over different kinds of machines. So if these machines were virtual machines, they would just be started with new image.

A

There wouldn't be any actual being being done, but if they're bare metal, then we'd be talking to a fog server in the cpu lab, so ask it's a image with a particular os and and uh version, and that process could take um say 10 to 15 minutes, sometimes at most.

A

Then we actually go and try to run the.

A

A

So if we do have these, if this job is one of the special ones where it's a marker, it's it's only representing the um the sweet beginning or the sweet ending. It's marked as the first and sweet or last in sweden.

A

And in that case, we um use the topology results of command to do some pre-processing at if it's first one, we write down the uh random seed, we that was used to schedule this run and the particular subset that was used in scheduling this friend so that we can recreate it later if we want to, and if it was the last and last of the suite represent the the end.

A

The end of the jobs will um use the results of the command to wait for everything all the tests prior to it, to finish, and then send out a email of the results and run the scrape and process to analyze. It.

A

That's just that the.

A

The kind of so that this is all just the handling for the um beginning and end markers of this week of the overrun.

A

For the normal jobs, if you're actually running a test.

A

We'll start constructing the technology command that we'll use to actually run the test.

A

This is a little bit of a compatibility hack. It said not not really used in um some backwards compatibility code. We could probably remove at this point.

A

um We have some options for whether to make it reverse or not, then we're adding in the um the owner for the lock machines the archive path that we set up the job name.

A

The description is that again, the concatenation of all the yaml fragments file names.

A

And the original config.yml file, which is the final configuration that technology will use to actually run the j the test, then we'll actually run that run, that toothology sub process.

A

This is again an asynchronous call for with the open and then we'll um watch that sub process to make sure that it's continuing to run and if it, if it does time out, we can kill it and clean up after it and save the logs.

A

So this is taking in the process, objects we're representing that toothology command we're just looping. While that process is still alive and checking whether we've hit the timeout or not.

A

If we have, then we go ahead and kill the process and save it. Save its logs.

A

Copy this logs to the archive directory and finally clean up any of the machines.

A

If the job is successful, then all all the reporting to volpido and cleaning up of machines will be handled within the job itself.

A

Any uh any more questions at this point, I think we've gone over how to how the jobs start out and from being scheduled to going through the queue to actually being run.

A

I think the next phase is how they actually get run by the topology command.

A

Itself don't check here, hey, I'm, not asking anything just checking if it so yep.

A

um I think we only have like five minutes left, so I don't think we're gonna have time to look at how the technology command itself works in too much detail today.

A

um Maybe it's worth noting just uh some general other other about developmentally itself um like, for example, we have tests for theology as well um you'll see we have several subdirectories in different areas called tests which contain a bunch of uh unit tests. For the most part, um these can all be run with talks.

A

This is a common way to run python tests.

A

You'll see we have a few different environments and talks. um Pi 3 is the one for running the unit tests, there's also flakehead for checking style um and making sure that they're doing some kind of a basic static analysis.

A

So these are very helpful if you're doing any kind of development technology, you can run talks, minus e, pi 3 to run the unit tests or this talks to run all the tests directly without regard without regard to choosing particular ones. I think we may have a few more uh sessions like this in the future. uh Let's figure out exactly what we want, what else we want to cover, but I think this is uh concludes what we've got scheduled for now, and at least we'll all be up on youtube in the future as well.

A

Thanks for attending and see you next time take care.

A