Rust Programming Language Rust Linz, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: AMD Hypervisor with Rust - Matthias Heiden - Rust Linz, October 2022

Description

Hypervisors are an incredibly powerful tool that can be used to run your FaaS or just virtual machines. Windows also secured their OS by utilizing a hypervisor. But hypervisors can also be used for bad things. Malware and game hacks use it to get around detections. This talk aims to show you the process of how I wrote an AMD hypervisor for Rust.

Matthias is a passionate Software Engineer with an interest in Software Security. He first heard about Rust in school and decided to give it a try. A few months after reading the Rust book, he finally had the time to rewrite some personal projects and started to understand why Rust is awesome.

Resources:
Matthias Blog: https://not-matthias.github.io/
Hypervisor List: https://github.com/stars/not-matthias/lists/hypervisor
Custom Allocators in Rust: https://not-matthias.github.io/posts/rust-kernel-adventures/

Rust Linz: https://rust-linz.at
Speak at Rust Linz: https://sessionize.com/rust-linz
Twitter: https://twitter.com/rustlinz

A

Yes, my talk is about writing a hypervisor with rust, and so a little about me and I'm, currently studying computer science at the jqu and I've been using rust since, like 2020 and I really fell into love with it and I'm also really interested in reverse engineering and game hacking and that's how this project got started.

A

So, let's take a look. What exactly a hypervisor is and Wikipedia defines it as a software that runs virtual machines and that's actually a distinction with between two types, A type, one that runs directly on the hardware, which means it's a lot more effort to implement, because you need to do everything on your own and then there's also a type 2 hypervisor which runs on the operating system. So there you don't have to do everything on your own.

A

You can use the operating system to allocate virtual memory or physical memory and it's a lot easier, but still a little complicated.

A

So where are hypervisors used? What can you use them for and one of the most common use cases is just virtual machines. They run another operating system on your already running operating system and examples for this are VMware virtualbox, KVM, I'm sure all of you have used it before another really powerful example is serverless Computing, so, for example, running functions, so functions as a service on a server and one really popular example is Amazon firecracker, which uses rust and is built on top of KVM.

A

This is a Linux kernel, module and handles all the hypervisor stuff and, for example, they don't have virtual machines, they call them micro, VMS and with that so with that they can restrict the resources they have access to. So, for example, you can say this function should only access, 120, megabytes of memory or should only use up to two virtual CPUs, and this is really powerful because you don't just want some random guy that uploaded a function to access the entire memory or entire system.

A

So it's really good that there's some security built in and that's exactly what hypervisors are used for so another example is antivirus so with a hypervisor. So it's a hardware feature supported by the CPU, and because of that, you have a lot more control over the system and the antivirus can effectively use it to find. If a program is doing something weird, but also they can protect against it, they can protect some resources.

A

Some memory or yeah do some other stuff and one example for an antivirus is the windows virtualization based security and there they have code. Integrity checks, also protection against direct memory attacks and it's very hard to recommend because they are using a hypervisor and another really cool example is fuzzing so in fasting. The goal is to find uh states of the program that may make it crash and there you need to trace over all the instructions and also pause at certain instructions and with the hypervisor.

A

You can do that, and on top of that you can even pause at the certain point and resume at another point. This is really powerful and I really recommend checking out the gamotsu labs GitHub repository. He has a ton of really interesting projects, also written in Rust and then there's all the bad sides. You can also use it for malware or cheats, just like the antivirus can use it to have more power for the goods.

A

Bad people can use it to gain more power and to hide from the antivirus or anti-cheat, because everything has to rely on something. Usually it's about the operating system, but you also have to rely on the hardware. So how would they know that they're running inside a virtual machine there's some ways?

A

Some timing attacks I'll come back to that later, but you can still do it and make it seem like they're running on a real system when they're not so why would you write your own hypervisor and for me it's a lot about just the challenge and it's a really challenging and also rewarding project and all wants to learn more about the details about CPUs and like how everything works and since I'm, coming from a game, hacking and security backgrounds. I was also interested in figuring out if I could actually bypass the entity and yeah.

A

So what exactly can you do with a hypervisor? And the answer is pretty much everything you have control over the entire system. You can intercept instructions on memory accesses, so you can do whatever you want.

A

So, let's take a look at some parts that are involved when you are writing a hypervisor. So one of the most powerful Parts is a nested page table, and this this case is just a regular page table. You have a virtual address and it goes through all the page tables and at the end you have the physical address and with a hypervisor you can have two of those and that's called enlisted page table on AMD.

A

So if you're inside a virtual machine, you think you have, let's say four gigabytes of memory, and this virtual memory is resolved to physical memory, but because the virtual machine thinks it only has four gigabytes but in fact, on the actual Host, this four gigabytes is resolved again to the real machine memory and there you can do some really interesting things like, but only setting certain pages to read so no right access and that's also what virtualization based security. Does it's really powerful in other parts of a hypervisor?

A

Is an MSR bitmap, so MSR stands for model specific register and they basically allow you to read certain data from the CPU or change the behavior, and with this bitmap you can essentially intercept those requests. So you can read an MSR also right to one and with my hypervisor you can intercept those and when intercepted a virtual machine exits, a short Vmax, it will occur, and you can imagine it somewhat like an interrupt which just caused a callback in the hypervisor.

A

Then the vmx that will be handled and returns to the original location where it occurs and another part of a hypervisor is the virtual machine control plug and it basically consists of two things: the safe area and the control area and the safe area contains all the information about a virtual machine.

A

So the registers, the segments and also some debug Fields. So if you pause the virtual machine, all the information is written to this product. So you can pause. It do some other stuff and come back later and resume it at this point and the other part is the control area where you can, as the name says, control certain Behavior. So you can specify what you want to intercept, for example, the rdtsc instruction, which reads the timestamp counter.

A

Also the CPU ID also read, or write to the CR, the control registers and a ton of other Productions.

A

You can also enable the nested paging, uh which we saw before with the nested page table, also set the next instruction pointer and other things, and because modern CPUs have multiple cores.

A

You need to read this structure: vcpu data for each core and each core has a guest vmcb virtual machine control, plug and a host vmcb, and then also we have a pointer to the shared data. So we don't want an MSR bitmap unless it's page table for each core. We could but does not really a use case for that. So we just have a shared data for that.

A

So, let's take a look at the implementation and for that I'm going to take a look at how you can virtualize a single CPU. So let's switch to the code.

A

Okay, so I have a hypervisor struct and it basically just consists of all the CPU cores you have, and this is the vcpu struct and when you call the virtualize method, we just iterate over all the processes you have and then we use the windows API to switch the execution to a specific processor, because we really have to make sure that we're not virtualizing the same CPU core I, just call it processor price, that's gonna, feel horribly and we just call the virtualize method on the process, respect on the vcpu struct.

A

So let's take a look at it. What it does there, so we pass the shared data and then we create the context, and this context is really important- that it's at this place. I'll come back to that later. The context just has all the registers and all the instruction pointer and then we make a check if this DQ core has already been virtualized, because, as I said, we don't want to virtualize it twice and you can do that with an atomic and yeah. It's not really important.

A

It has not been if it has not been neutralized, we virtualize it. So we set the flag, we do some initialization and then we create the vcpo data.

A

We also create uh our own stack and we launch the VM with some manual assembly. I'm, not gonna, go for it that's way out of scope, and then we launch it and the important thing is. We created the context and when we launched the VM it will resume execution at this context.

A

So we launch it and then we're suddenly back here. Then we check again. Is this CPU core? Already virtualized? We already set the flag, so it is so now we are not virtualizing it again.

A

We just returned through and that's done for every CPU core and yeah in the launch script. We just uh use the VM run and that's running until we get on vmx it. So that's how you virtualize a single CPU or multiple. It's effectively the same for multiple, multiple cores, so I mentioned the term VM exit and it's one of the most important things you need when interacting with the guests.

A

So we have the dmaxes Handler, which is called from our custom assembly. You can find it here and that's not important. The important thing is we get the vcpu data in the guest registers, so the Vmax is again called for a specific core and that we need to need to do some initialization, that's not important, and then we can with the vcpu data we can access the guest virtual machine control, clock, bmcb and then the control area, that's the exit code.

A

So if we got a VM exit and our callback has been called, we want to know which vmx it was so, for example, did someone try to read the timestamp counter or the nested page folder cool, or did somebody try to execute the CPU ID instruction, or did someone try to read MSR or write to one?

A

So there's a ton of options and I basically programmed it that you can specify your own Handler and I can take a look yeah.

A

So one example for Handler I mentioned it before you can do a timing attack to protect if your code is running inside a virtual machine or running under a hypervisor, and one of those ways is to execute the rdtsc instruction.

A

So we disabled the interrupts, because we want to make sure it's not uh yeah it's correct and then we executed a thousand times a little more and we execute the rdtsc instruction and remember now a vmx it happens. So we jump to the DMX's Handler. If we maxit Handler execute some code and then it jumps back- and this is the first hexon RTC Returns the CPU and cycles and every time the CPU does some work, it's uh incremented.

A

So when we do the second rdtc, we read the Cycles again and we take the difference. We're gonna notice, the difference, because we have been executing the vmax's Handler code and on the normal system. It's pretty much always between 25 and 500 Cycles.

A

So this is just a heuristic, but in 99.99 of cases it's always this. So how could we get around that and I already showed you we can. Let me switch back. We can check if there's a VM exit for an rdtsc. So let me show you an implementation that gets around that.

A

So this is the handle audio to see and it's cold whenever so, it's called here and there we first of all, we get the real handsome and we also saved the previous timestamp and we take a difference, and so, if we so the first one is just zero, but then we save it and then for the second one. We compare the difference to the first rtrc and if it's below a certain threshold, uh we then check what was the previous VM accent.

A

So if the previous VM exit was also for an rdtsc instruction, we can just return a random number, that's between 25 and 35, how long it usually takes and then just add that to our fake rdtsc and then we just set the correct registers and we can return, and the important thing is with the RDC thresholds.

A

What, if someone as a sleep for 1, 000 milliseconds, so one second in between there and we just increments by 20 or 35, something in between then it would seem like there was no sleep. So that's the important thing about the threshold and you can do the same thing on a similar thing for CPU ID. So you can check if there's a VM exit for CPU, ID and.

A

Then you can basically just execute it normally and, for example, there's a flag that detects if a hypervis is present and it's set automatically by the CPU.

A

But a program can just check that if, if it's running under hypervisor and what we can do, we can execute the normal, the real CPU ID and then just toggle the flag, just remove it and then return it again, and the guests will not know that this was ever set and yeah you're the same. We can just return zero and the same form model specific register. We can also again just get the vcpu and the registers and modify the register there.

A

So it's really powerful and you can do a lot of things with it. So, let's take a look at some other things that I think are really cool about this project.

A

So for some structures you need to allocate contagious so memory. That's in one uh error, not split into multiple chunks, contagious physical memory and the example for that is the MSR bitmap, but most of the times you only need virtual memory. So if you have a global allocator, you need to find a way to now allocate contiguous physical memory. You could do it on your own, but I found a really neat solution. We can just create a new struct which is called physical allocator, and then we can implement the allocator trade for that.

A

And if you want to take a look at the implementation, you can check out my repository. I'll publish it after this talk and we just have to implement the allocate and de-allocate function, and once we did that we can just use the new in functions of all the structs in the log right and we can specify the allocator so now our instance is actually allocated on with physical memory instead of virtual memory, and you can even catch errors because, often if the system is out of memory, you don't want to just crash and stop.

A

You want to handle the memory and really some memory or do some other tricky stuff, so you can actually handle it, which is really cool and with a virtual memory allocator. uh This is just a recap, but I think it's also important to mention. I created the crates, which is called kernel alloc, and you just need those three lines and you have a virtual memory allocator, so you don't need to go through the new elements, creating your own allocator. You can just use this crate, which is also really cool in my opinion.

A

So another thing I found really useful, was bit Fields. So, for example, in this case, we want certain fields of this struct uh called event injections to be a certain size. So the vector Fields should only be go from bit 0 to bit 7 then type from 8 to 10 and the error code. Valid should only be bit 11.

A

and with this bit field creates. We can essentially do that and it will automatically generate those functions which is really useful, no more ends and or bits metric directory and another thing are bit Flags.

A

So this is really important, because the AMD manual says how they have to be defined and I didn't want to just create tons of constants that define those you can actually with this create Define them as a struct, which is also really cool.

A

So in this case, the end now enable lbr is the several bits and the virtualized VM save is the first bit set and and so on, and one thing that really annoyed me: I had a ton of Errors just because of misaligned structs or because they had the wrong size, because I used the wrong data type and unfortunately, rust doesn't have compile time assertions, but I found a really useful, create called Static assertions, and with that you can just use the cons as third equals macro then specify the size of and the type and then the size of the product.

A

It really saved me a ton of time, so how can we actually use my hypervisor and I? Think I already went over that yeah, so I already took that away. So this was the example with the rdtc and yeah implementing the hand, loss yeah.

A

So of course, I prompted some problems because it's just yeah it's really low level and there's not a lot as much information available about it. So it's inevitable. So one issue was alignment, so the MV manual is really strict and, for example, the nested page table has to be four kilobyte aligned.

A

And if you don't follow that rule bad things will happen and yeah it's just gonna, it's not even gonna crash. It's gonna do weird things and you have a really hard time to figure out what went wrong and the solution for that is to just use assertions with like static assertions create I mentioned and also found this Elaine trade, which made it easier to make sure the extracts are aligned.

A

Another thing is the app the invalid structurizes and yeah. It's very easy to get the wrong data type, especially if you copy C, plus plus code, because there's these weird data types and yeah and unfortunately, there's also no automatic generator for all the structs from the MD manual.

A

There is one for C, plus plus that just passes the Intel manual and then updates all the offsets and Fields I would have wished something similar existed for us with MD, but unfortunately doesn't and I tried to do it, implement it on my own, but housing PDF is really no fun. I can't recommend it but yeah the manual solution for that again is assertions and also had one really really weird bug and it so I didn't use.

A

The allocator I showed you before I used my own struct and it's called allocated memory and it had two functions: I look normal, so normal virtual memory and aloe contagious, which allocates which uh physical memory and the weird thing is. If we have a struct flu and we embed another instance of this allocated memory inside through and it goes out of scope, it just leaks memory, and it's really weird I wrote a blog post about it.

A

I have a read and if you know why it's happening feel free to message me: uh yeah I solved it by just using the allocator and coming back to the box and back because, as it turns out, box doesn't have a drop function. So if you take a look at the drop function of box, it just says it's implemented by the compiler and I figured. There has to be a reason why it's com implemented in the compiler, not in the rust code, so I, just stuck with it and hypervisor debugging is also really painful.

A

If you have an error, it slows down the system, there's also, sometimes no and really bad errors. For example, if you have a miscellaneous truck or a wrong value at some fields, you just get a VM exit invalid. It just doesn't tell you where, where the error is, it just tells you vmx is invalid. Have fun figured out yourself, so you have to be prepared to spend a lot of time in windybug. The support is really good there.

A

Also for us, which was really surprising, but you have to be prepared for that and there's gonna be lots of head banging, because if you have a bug in the Vmax Handler, it causes the system to freeze. And so you don't want to develop it on your main system. So you need a virtual machine and then you need to set up nested, paging and also nested virtualization.

A

And ideally you want to automate it because trust me you're going to need to reset the VM a lot and all the performance can be a problem, because if you have lots of VM exits, it's bad for performance and if you have a slow, vmax's Handler, it's even worse for performance, and if you combine those two, if you have lots of vmaxes and the slow video maxit Handler, it's arable and also there's really a lack of resources.

A

There's a few really good blog posts, but most of them on almost all of them were for inflow and there's also mostly just projects on GitHub and I created a list uh on my guitar feel free to check it out, and this helped me the most and for the documentation. I just recommend sticking with the Intel or MD manual, because they're up to date and yeah have all the information you need and also really important tip is maybe just ask someone who already has something done something similar because there's a high chance.

A

They had a similar error or know someone who had a similar error, and it will save you a lot of time and to come to the conclusion. I really enjoyed using rust. Despite all of the problems, I still think the compiler have has saved me from a lot of issues. I would have had if I had, for example, used C plus plus, maybe I wouldn't even have finished it and also found a ton of useful noise that you create.

A

For example, I also implemented some function, hooking and it turns out the ice grid also has no STD support. It's really useful I didn't want to implement my own. This assembler in a yeah Windows kernel, module and yeah, as I mentioned a bit flag and bit field crate were also really nice and yeah, also custom, allocators, they're, so powerful and also being implemented into the analog grades. Take me a ton of time and inline and Global assembly were also really cool, and that's it.

A

Thanks for your attention and I'm happy to answer any questions.