# Windows Internals: Secure Calls – The Bridge Between NT and SK

## Introduction

As many are aware, without the presence of Hyper-V on modern Windows systems – kernel-mode is the “highest privilege boundary” in terms of the OS. Because the kernel is responsible for privileged operations (like memory-management) a special, very-well known interface exists – the system call interface – which allows user-mode to request the services of kernel-mode. However, with the advent of Virtualization-Based Security (VBS), the presence of Virtual Trust Level 1 provides the OS a higher security boundary. Typically whenever a higher-privileged boundary exists, this means that the lower-privileged boundary still either needs to enlighten the higher-privileged boundary with some information and/or it may need to request the higher-privileged boundary to do something on it’s behalf. VBS is no different. Just as user-mode may need to request the services of kernel-mode (system call), the kernel in VTL 0 may need to request the services of the Secure Kernel in VTL 1 (secure call).

_Windows Internals 7th Edition, Part 2_ talks about this interface in some detail. However, I have always been interested in debugging and interacting with the Secure Kernel because I think it is very well designed. For some time it has piqued my interest to write some sort of harness that would allow anyone to issue their own secure calls.

This blog post will be an in-depth analysis of the secure call interface and a release of a tool I wrote, which I am calling `SkBridge`. Specifically this post will be taking a look at the architecture which allows NT, which is in a completely isolated region of memory from the Secure Kernel, to “hand off” execution to the Secure Kernel, as well as showcase some of the common patterns NT and SK use in regards to copying and encapsulating arguments/parameters. Lastly, one of the motivating factors for this tool and post were two older posts by my dear friend and someone who always helps me, Alex Ionescu, about writing a bridge to fuzz hypercalls. This “gist” of this post and tool is effectively a “higher level” abstraction, which instead of being a harness to issue hypercalls, is instead a harness specifically for secure call (which, as we will see, still leverages hypercalls) and is not necessarily meant to be used to “fuzz” the Secure Kernel, but instead to showcase the “secure call interface” (which is not a documented “interface” like hypercalls. I am intentionally using the term “interface” loosely).

## Secure Call Interface

As a primer, there are a few different mechanisms which exist for communication between the Secure Kernel and NT. Namely they are:

1. Secure calls
2. Normal calls
3. Secure system calls

Normal calls allow the Secure Kernel to request the services of NT. The Secure Kernel is a small binary which only implements functionality it needs in order to avoid exposing a large attack surface. Notably, as an example, file I/O is not present in the Secure Kernel and requests to write to a file (like a crash dump for an IUM “trustlet” that is configured to allow a crash dump to occur, also known as a “secure process”) are actually delegated to NT.

Secure system calls provide services specifically to secure process running in VTL 1 (again, like a trustlet) and do not result in a “transition” between SK and NT. Just as user-mode in VTL 0 needs to request the services of kernel-mode, so too “secure processes” running in VTL 1 need to request the services of kernel-mode in VTL 1. This is the interface to allow this.

This blog post will instead focus on the _secure call_ interface, which often is erroneously called the “secure system call” interface (even by myself! The terms are confusing!).

The secure call interface, which allows the NT kernel to request the services of the Secure Kernel, is not the same type of interface we may think of when we think of the “typical” system call interface. There is no “direct pipe” which allows the processor to start executing “in context of the Secure Kernel”, similar to a system call (which simply “transitions” execution into KM, with a few nuances like switching to the thread’s kernel stack, etc.). The secure system call interface is really a “wrapper” for a specifc _hypercall_. A hypercall is a special operation (represented by the `vmcall` instruction) which transitions a processor which was previously executing in context of a _guest_ (e.g., the processor was running code in context of a virtual machine, also known as guest) to what is known as Virtual Machine Monitor, or VMM mode (meaning execution on the processor is now executing in context of the hypervisor). This is similar to a system call, which transitions execution to the kernel – but in this case a hypercall is used to transition execution into the VMM software (hypervisor).

One common misconception is that the Secure Kernel “runs in the hypervisor”. This is actually not true. The Secure Kernel runs in an _isolated physical address space_ (VTL 1), just like any other VM. When a secure call occurs, it is not NT being “directly piped” to SK. When the secure call happens, a hypercall occurs. When the hypercall occurs, Hyper-V’s VM exit handler starts executing. A VM exit is an “event” that occurs when the processor starts running in VMM mode. A VM exit contains information about _why_ the switch occured into VMM mode. The “magic” which makes this all work is that a hypercall, in our case, will _cause_ a VM exit.

So how does the hypercall work then? Or why is it important? The Microsoft Hypervisor Top Level Functional Specification, (also known as the TLFS), contains a list of all of the supported hypercalls. The “secure call” interface is effectively a wrapper for the `HvCallVtlCall` hypercall. In NT, a pointer to the stub dedicated to this hypercall can be found at `nt!HvlpVsmVtlCallVa`.

The “secure call hypercall code” is that of `0x11`, or 17 in decimal. This effectively means a secure call is “just” a hypercall which specifies this code. It is important to note that a `vmcall` instruction is spec’d to only run if the processor (which is currently running in “guest” mode) is at current privilege level (CPL) 0, or kernel-mode. `vmcall` is undefined in user-mode. A `vmcall` (hypercall), therefore, effectively just causes a VM exit (and, thus, specifying the reason for the VM exit).

Once Hyper-V has execution it is then responsible for transitioning execution to the Secure Kernel (this is how execution goes from VTL 0 to VTL 1!). Hyper-V is the bridge between NT and SK, SK does not live “in the hypervisor”! This would mean that if we can locate the VM exit handler for Hyper-V, it would be possible to examine Hyper-V dispatching the Secure Kernel.

## Locating the Hyper-V VM Exit Handler

We now know that a hypercall will result in a VM exit. Because of this, it is prudent to locate where, in Hyper-V ( `hvix64.exe` for Intel or `hvax64.exe` for AMD), VM exits are handled.

There is existing art on locating the VM exit handler for Hyper-V (for both AMD and Intel builds of Hyper-V). The canonical example is searching for a `vmresume` instruction (on Intel), which indicates the current processor will “go back” to executing in context of a particular guest (which indicates that the reason for entry into the hypervisor software was handled). The VM exit handler would then be in-and-around where a `vmresume` occurs. However, I am familiar enough with Hyper-V to know that there are certain debugging print statements located in the VM exit handler with the string `MinimalLoop` present. Searching for this string in IDA yields these print statements.

As we can see, a few strings like “EPT violation” (which can be a reason for a VM exit) and “ `VMX_EXIT_REASON_INIT_INTR`” indicate logging is occuring in the VM exit handler. If we examine where this logging occurs, and if we then convert all integer-style values to appropriate VM exit reasons, we can see the VM exit handler is responsible for determining how to service the VM exit event.

It should be noted, additionally, that the VM exit reason is stored in the VMCS structure. The VMCS, or Virtual Machine Control Structure, is a per-processor structure. A VMCS represents the state of the “guest” running on a particular processor. Remember, with virtualization a processor can either be running in context of a particular guest (VM) or in context of the hypervisor software. We will see, later on, that both VTL 0 and VTL 1 have a VMCS which represents each of these “VMs”. What this means is that there is one VMCS loaded at a time on a processor (the VMCS is “per-processor”) but the _data_ in the VMCS is per-guest. This is because there is a special CPU instruction, `vmptrld`, which allows the CPU to load a target VMCS pointer for a particular guest (thus allowing “multiple guests”). One VMCS “per-processor”, but we can swap out which VMCS that is based on the guest we want run on that processor.

The VM exit reason can be extracted by the hypervisor simply by invoking the `vmread` instruction with a particular _VMCS encoding_ value. Because the VMCS, however, lives in physical memory, Hyper-V has the concept of “enlightenments” where the VMCS is mapped into _virtual_ memory and is simply written to/read from its virtual address. The `gs` segment register is very important when execution occurs in context of Hyper-V, as it maintains many of the constants found in virtual memory (like the “current virtual processor” or the “current VMCS” or the “current partition”). Saar Amar talks about this in his Hyper-V research paper. These offsets (which we will demonstrate how to find in this blog post) often change, and the data may not look the same from version-to-version.

As we can see, `gs:[2C680h]`, on this particular build of Windows (24H2), contains the _virtual_ address of the “current” VMCS. We know this because we can see here either the physical address of the VMCS is used, or the “enlightened” version. Because of this, we can deduce that since the VMCS is tracked via the current CPU’s `gs` segment register it is also very likely also that the rest of the important structures related to the hypervisor’s capabilities (like the “current virtual processor”) are also tracked via the `gs` segment register.

Because the rest of our analysis will require knowledge of where these structures are, we need to find where they reside. A wonderful blog exists on this, from Quarkslab, talking about how to identify much of this data. Unfortunately much of the data has changed between the time that blog was written, and now. In fact, even some of the structures in-memory do not contain the same “layout” as that of the Quarkslab blog. Because of this, its worth examining how to first identify this information. We will do this by first continuing into our VM exit handler, by locating where hypercalls are handled.

## Locating the Hypercall Handler

Coming back to the VM exit handler, we now can see there is a switch/case branch for `VMX_EXIT_REASON_EXECUTE_VMCALL`. This is our hypercall handler!

We still do not know what the arguments to, what we now will call `HandleVmCall`, will be – but we know that this is where hypercalls are handled. Taking a closer look, we can once again see _another_ switch/case going over many of the hypercall values. The hypercall values can be extracted either from the TLFS, or more-easily through Alex Ionescu’s HDK project.

In this case we can see that the `HvCallVtlCall` hypercall is our target. This has a value of `0x11`, and is what VTL 0 issues in order to request the services of VTL 1!

Now, to get a full list of all of the hypercall handlers – an easier method exists. We simply need to locate the “hypercall table”, which is stored in the `.rdata` portion of Hyper-V (it was once in a `.CONST` section). It is important to know where this table is, as Saar mentions in his blog, because most hypercalls first check the _current partition_ for the correct permissions/privileges. Some hypercalls require special privileges, and the “privileges mask” exists in the partition. So, the theory is that if we can locate the handlers for the hypercalls we can then inspect where this privilege check occurs and then determine where the “current partition” exists.

As we can see, the hypercall table has a layout where the hypercall’s number is mapped to a particular hypercall handler routine. This is either an actual function which sets up a proper stack frame/etc., or is an assembly routine which does some necessary manual tasks.

To locate the current partition, let’s take one of the hypercall routines – in this case `HvRetrieveDebugData`, which is hypercall number `0x006a` according to the TLFS.

Here we can now use WinDbg to load Hyper-V as data and examine this assembly stub. Use the command: `windbgx -z C:Windowssystem32hvix64.exe` (for Intel-based Hyper-V).

There is a constant, in this case, located at `gs:[360h]` which is some sort of structure that has a bitmask at offset `0x1b0`. We know that all hypercalls (usually) have this exact check at the beginning of the routine in order to validate privileges. This indicates that `gs:[360h]` must be the “current partition” and that `0x2b` is the privilege mask! Additionally, if we examine the `HV_PARTITION_PRIVILEGE_MASK` enumeration, we can see that `0x2b` is the `Debugging` bit – all but verifying that this is the partition, as the hypercall we are investigating is a debugging-related hypercall.

We now know the locations of the current VMCS and of the current partition. However, because there are some details still missing (especially because we don’t know how the VM exit handler receives its arguments and, thus, we don’t know the arguments for the secure call handler). The next step of the equation is to locate one of the most crucial data structures in Hyper-V, the Virtual Processor (VP). This data structure provides most of the arguments to both the secure call handler and the VM exit handler.

## Locating the Virtual Processor (VP)

With the VM exit handler located, locating the VP is fairly straightforward, but more relies on some trial-and-error. The Quarkslab blog outlines how they were able to find the VP, but on my build of Hyper-V (which is now 4 years newer), some of the semantics and offsets have changed. In our case, to find the VP, we “go back” as far as we can (using cross-reference functionality in IDA) to see how the VM exit handler receives its arguments. The VM exit handler has its arguments “passed on” several times. What I mean by this is that the VM exit handler receives arguments from a function, which were just _directly_ passed on (meaning the “originator” is several functions “up” in the code flow). Eventually we come to the following function in Hyper-V which passes on the arguments.

Hyper-V does not ship with any public symbols. So although this looks abstract, `sub_FFFFF800003321C8` is the function which will eventually invoke the VM exit handler. In this case, a few things can be noticed. Firstly we can see that from `gs:[0h]` a structure, referred to as “ `self`” in this case, is preserved. Saar’s blog gives insight into what is present here. `gs:[0h]`, when execution is occuring in Hyper-V, refers to “itself” – meaning that it is just a pointer “back to itself”. We can then see that what refer to as “the virtual processor” is extracted at offset `0x368` from the self-pointer. This is another way of expressing `gs:[368h]`. This is the current processor’s virtual processor structure! The VP structure has a specific structure member, located at offset `0xFC0`, which is passed to the VM exit handler. The VM exit handler also will preserve the virtual processor as a local variable.

The virtual processor is then passed to the VM call handler which, in turn, will pass it specifically to the secure call handler (which is just a hypercall with a hypercall code of `0x11`).

## The Secure Call Handler

Now that we have our feet under us, we can turn our attention to the actual “secure call handler”, which is just a hypercall handler for hypercall code `0x11` ( `HvCallVtlCall`).

The secure call handler will, first, extract `VirtualProcessor + 0x3c0`, which seems to be a structure, and then will extract from what seems to be _another_ structure at 0ffset `0x14`. One thing we must remember is that, when Virtual Secure Mode (VSM) is enabled, we have (currently) _two_ Virtual Trust Levels (VTLs). We have VTL 0 (normal world) and we have VTL 1 (secure world). The thing to remember here is that a particular processor, when VSM is enabled, executes _in context of a particular VTL_ as well! Hyper-V manages the “current VTL” information via the VP structure. In this version of Hyper-V, the “current VTL” is maintained through the current virtual processor at offset `0x3c0`. Additionally, offset `0x14` into this “VTL structure” contains the VTL associated with the VTL structure (which, in this case, means the VTL of the _current_ processor).

The curious reader may wonder where, what I am calling `VtlInitializedMask` comes from. As part of the “song-and-dance” that Hyper-V and the Secure Kernel perform, to initialize the VTLs, a “mask” (managed by the VP) maintains “state” associated with the VTLs that are initialized. This also brings up, since it is seen in the screenshot below, the VP maintains both the _current_ VTL information _and_ an array of all known VTLs.

The first thing the secure call handler does, if we are eligble to issue the secure call (the target VTL is initialized), is we “fixup” the instruction pointer for the current VTL. One crucial detail to recall here – with the presence of VSM we have _two_ VMCS structures which can be used – the VMCS associated with VTL 0 (which is the current VTL, since this is a secure call and VTL 0 and requesting services of VTL 1) and the VMCS associated with VTL 1. The “typical” specification for handling VM exit (like our secure call) is to then increment the instruction pointer to the _next_ instruction to be executed when the VM enter occurs later (when the hypervisor is done). This is the first thing that is done so that VTL 0 returns to the “next” instruction and does not re-issue the hypercall (in this case “secure call”). This is done be either leveraging the “enlightened” VMCS, or by reading from the VMCS directly using the `vmread` instruction.

Once the instruction pointer for VTL 0 has been fixed up, the transition to the new VTL (VTL 1) begins. This is achieved through, what I am calling, the `BeginVtlTransition` function. For our purposes this function will ensure that the target VTL differs from the current VTL (as this is a VTL _transition_).

When the actual VTL transition occurs, the first thing that happens is the current VTL data for the current virtual processor is updated. In this case, the current VTL is now VTL 1.

After the relevant information is updated, the actual VMCS of the current VP needs to be updated to that of the new VTL (VTL 1). This is done through a function I have named `TransitionToNewVtlViaVmcs`. From the “new VTL Data” comes what I am referring to as _private_ VTL data. This could also be renamed to “VTL state data”. The VTL structure is quite large, and we will see shortly why this “VTL private data” is used, as we need it to update the VMCS.

The transition to VMCS occurs. `vmptrld` will be used if enlightenments are not available, otherwise (if enlightenments are available) the _virtual_ address of the VMCS is used.

The “guest RIP”, “guest RSP”, etc. are now all that of VTL 1. Execution is still in the hypervisor. Now, the “guest RIP” and “guest RSP” will be used, when the “VM resume” occurs to allow the processor to start executing in context of the guest (which is now VTL 1 after the VTL transition). The new guest RIP and guest RSP come from _the last time_ VTL 1 caused a VM exit. So whatever VTL 1 was doing at the time it performed the last action that caused a VM exit is the state of the processor when the VM resume will occur.

This means we now have a primitive (secure call) to transition into VTL 1, requested by VTL 0 and serviced by the hypervisor as we have seen, but the crucial question here is _what_ will be executed when the VM resume occurs? The Secure Kernel is setup in such a way, when handling secure calls, to cleverly leverage code routines and hand-crafted assembly code that exist very close together in memory so that when the hypervisor issues the VM resume and execution occurs in VTL 1, the correct handlers are present in the Secure Kernel to service the secure call.

## VTL 1 State Preservation And VM Exit Back To Hyper-V

Let’s now turn our attention to the Secure Kernel’s “famous” function, `securekernel!IumInvokeSecureService`. Using SourcePoint’s debugger, which I have previously outlined using, we can debug the Secure Kernel to gain insight into how VTL 1 preserve’s it state in such a way, that when a secure call occurs, execution seamlessly results in the secure call being serviced by `securekernel!IumInvokeSecureService`.

When the secure call has been services (via `securekernel!IumInvokeSecureService`), an indirect jump occurs to `securekernel!SkpPrepareForNormalCall`. It is _crucial_ here that this is a jump, not a call, as no return address is pushed onto the stack, as the thread currently executing may not end up being the thread which actually processes the return back into Hyper-V.

Secure calls are handled, usually, in context of a particular thread (more on the actual interface towards the end of this blog post). Because of this two functions are called, `securekernel!SKiDeselectThread` and (potentially, if a specific thread is necessary – we will talk about this later) `securekernel!SkiDetachThread`. This allows us to “stop executing” in context of the particular thread in which the secure system call was handled.

We are now “back” to the thread which was handed execution in VTL 1, when the secure call execution was passed to VTL 1 (via VM resume in Hyper-V). At this point, execution continues linearly. As I mentioned, however, all of functions related to preparing the state of execution and of the processor back into Hyper-V are executed in these assembly “thunks”. `securekernel!SkpPrepareForNormalCall` ends up invoking `securekernel!SkpPrepareForReturnToNormalMode`, and this is where “the magic happens”.

Eventually an indirect call to `securekernel!ShvlpVtlReturn` occurs. This time we issue a _call_ instead of a jump. This is crucial, because a `call`, as you may know, will push the address of the _next_ instruction onto the stack.

In this case the address of the _next_ instruction is `securekernel!SkpReturnFromNormalMode`! This means that when the VM exit from VTL 1 occurs, back into Hyper-V (which is known as a “secure call return”) it will be _this_ address which is pointed to by the top of the stack (RSP). Why does this matter? The current function about-to-be executed ( `securekernel!ShvlpVtlReturn`) simply issues a `vmcall` (hypercall) with the secure call return hypercall code ( `0x12`). When this happens, the VM exit happens back into Hyper-V – and the address on the stack is that of `securekernel!SkpReturnFromNormalMode`.

Hyper-V, on receiving the secure call return hypercall, will _also_ perform a similar fixup to that which we saw earlier, to the current instuction pointer of the VMCS for VTL 1 (which points to the `vmcall` instruction in `securekernel!ShvlpVtlReturn`). In this case the _next_ instruction after the `vmcall` is simply a `ret`! What this allows the Secure Kernel to do is that, upon the next VM entry into VTL 1, this `ret` will execute and, thus _return_ into `securekernel!SkpReturnFromNormalMode`! `securekernel!SkpReturnFromNormalMode` is the Secure Kernel function responsible for dispatching the appropriate logic as to why the VM entry into VTL 1 occured (hypercall, intercept, etc.)! This “packing together” of functions near the `vmcall` instruction allows the Secure Kernel to “always be ready” to handle any VM entry!

Now that we have examined the underlying mechanism which allows for VTL 0 -> Hyper-V -> VTL 1 “secure calls” and returns from VTL 1 -> Hyper-V -> VTL 0, let’s actually examine, from the “NT” side the actual “secure call interface” and the nuances surrounding it.

## Secure Call “Interface”

The secure call interface, as I have mentioned in previous blogs (and this one), all revolves around the NT function `nt!VslpEnterIumSecureMode`, which I have prototyped as such:

“`
NTSTATUS VslpEnterIumSecureMode ( _In_ __int16 OperationType, _In_ ULONG64 SecureCallCode, _In_ ULONG64 OptionalSecureThreadCookie, _Inout_ SECURE_CALL_ARGS *SecureCallArgs );
“`

The `SECURE_CALL_ARGS` structure is undocumented, but is _known_ to be 0x68 (108 bytes) in size from _Windows Internals 7th Edition, Part 2_. To the best of my ability I have reverse engineered this structure to the following layout:

“`
union SECURE_CALL_RESERVED_FIELD { ULONGLONG ReservedFullField; union { struct { UINT8 OperationType; UINT16 SecureCallOrSystemCallCode; ULONG SecureThreadCookie; } FieldData; } u; }; typedef struct _SECURE_CALL_ARGS { SECURE_CALL_RESERVED_FIELD Reserved; ULONGLONG Field1; ULONGLONG Field2; ULONGLONG Field3; ULONGLONG Field4; ULONGLONG Field5; ULONGLONG Field6; ULONGLONG Field7; ULONGLONG Field8; ULONGLONG Field9; ULONGLONG Field10; ULONGLONG Field11; ULONGLONG Field12; } SECURE_CALL_ARGS, *PSECURE_CALL_ARGS;
“`

Other researchers have shown that the first argument passed to `VslpEnterIumSecureMode` is the “operation type”. Almost all of these are set to `2`, but other values do exist. `2` seems to indicate “requesting a secure service” or a “secure system call”, in which a secure process requests a “normal mode” service. Additionally, `OptionalSecureThreadCookie` is unused except for the case of starting a secure thread and calling into an enclave. These are “special” cases where a specific “thread cookie” needs to be passed to the Secure Kernel.

A “secure thread cookie” is created by the Secure Kernel when the NT kernel requests that a _secure thread_ be created. The Secure Kernel is then responsible for setting up the secure thread and will then, on success, return a “secure thread cookie” back to the NT kernel. This cookie is effectively a “handle” of sorts, and lets the Secure Kernel know (who tracks all known secure threads) which thread a particular secure system call needs to be serviced on. Using WinDbg we can identify

“`
lkd> dx -g @$cursession.Processes.Where(p => p.Threads.Any(t => t.KernelObject.Tcb.SecureThreadCookie != 0)).Last().Threads.Where(t => t.KernelObject.Tcb.SecureThreadCookie != 0).Select(t => new {Process = (char*)(((nt!_EPROCESS*)(t.KernelObject.ProcessFastRef.Object & ~0xf))->ImageFileName), TID = t.Id, SecureThreadCookie = t.KernelObject.Tcb.SecureThreadCookie}) ======================================================================================= = = (+) Process = (+) TID = SecureThreadCookie = ======================================================================================= = [0x172c] – 0xffff9e0942b543b8 : “NgcIso.exe” – 0x172c – 0x15 = =======================================================================================
“`

In this case `NgcIso.exe` is associated with “Windows Hello” (another feature of Windows is that the biometric authentication can be implemented in VTL 1!) process. In this case the secure thread cookie, managed by the `KTHREAD` object, is `0x15`. This can optionally be provided to the secure call interface to instruct the Secure Kernel to handle a secure call on a particular thread.

`nt!VslpEnterIumSecureMode` will do a few things, in addition to packaging up the arguments. If the type of operation type is “3” (a request to flush the translation buffers, or TB) an ETW event can be generated for the enter into VTL 1 (you can see my tool Vtl1Mon for more information). In this case a `nt!HvlSwitchToVsmVtl1` is made – which simply issues the hypercall for code `0x11`, which is a secure call.

MappedSystemVa` to process the parameter. VTL 0 is usually responsible for providing the virtual address of the MDL in VTL 0 and the physical page (PFN) backing the MDL.

Additionally a common pattern is the use of “secure handles”. These are typically found in the form of processes and threads, and also images (section objects). These handles usually start with `0x140000000`. They are, just like “normal handles”, indexes into tables which manage the secure objects in VTL 1. An example is the “secure PEB” retrieval secure call. A list of all the valid secure calls can be found through the `nt!_SKSERVICE` enum in the symbols.