CPU_ON on QEMU ARMv8A using PSCI from EL2/EL3 - qemu

Question

I have 4 core ARMv8A (Cortex-A53) emulated on QEMU 6.2.0. The primary code (CPU#0) is running and I am able to debug it using GDB. I wanted to bring up other cores. For that I have used the following GDB commands. From the different experiments conducted, my conclusion is that only CPU#0 is running and all other CPUs never started.
(gdb) thread 3
(gdb) info thread
Id Target Id Frame
1 Thread 1.1 (CPU#0 [running]) 0x0000000040000008 in ?? ()
2 Thread 1.2 (CPU#1 [halted ]) 0x0000000040000000 in ?? ()
* 3 Thread 1.3 (CPU#2 [halted ]) 0x0000000040000000 in ?? ()
4 Thread 1.4 (CPU#3 [halted ]) 0x0000000040000000 in ?? ()
(gdb) where
#0 0x0000000040000000 in ?? ()
Exploting further, came across this thread about turning on CPU using PSCI. Start a second core with PSCI on QEMU.
Also came across SMC calling convention related to this. I have gone through the documentation of SMCCC and PSCI.
I am implementing a minimal hypervisor. The guest is a Linux and it is booting. The Linux boot log shows
[ 0.072225] psci: failed to boot CPU1 (-22)
...
Further debugging the code revealed that Linux is throwing a synchronous exception to the hypervisor with necessary parameters as per the specification using the "HVC" instruction.
If my understanding is correct, PSCI implementation is vendor specific- that is, the code running at EL2/EL3 has to use some vendor provided mechanism to turn on the CPU(core). Is this correct? on on system without EL3, how the code running at EL2 turn on the CPU?
My QEMU command line is given below
$qemu-system-aarch64 -machine virt,gic-version=2,virtualization=on -cpu cortex-a53 -nographic -smp 4 -m 4096 -kernel hypvisor.elf -device loader,file=linux-5.10.155/arch/arm64/boot/Image,addr=0x80200000 -device loader,file=1gb_4core.dtb,addr=0x88000000
Any hint is greatly appreciated.

Answer 1

When the guest is not booting at EL3, the QEMU virt machine implements its own internal PSCI emulation. This is described in the DTB file passed to the guest, and will say that PSCI calls should be done via the SMC instruction (if the guest is starting at EL2) or the HVC instruction (if the guest is starting at EL1). Effectively, QEMU is emulating an EL3 firmware for you.
(If the guest does boot at EL3, then QEMU assumes that the EL3 guest code will be implementing PSCI; in that case it provides some simple emulated hardware that does the power on/off operation and which the EL3 guest's PSCI implementation will manipulate as part of its implementation of the CPU_ON and CPU_OFF calls. But that's not the case you're in.)
If you are running a hypervisor in your guest at EL2, then it is your hypervisor's job to implement PSCI for your EL1 guests (it's unlikely that you want to allow an EL1 guest to be able to directly shut down a CPU under your hypervisor's feet, for instance). So you want to pass your EL1 guest a different DTB that describes the view an EL1 guest has of its emulated hardware, and which says "PSCI via HVC". Then your hypervisor's HVC handling should emulate PSCI. Separately, your hypervisor's bootup code should be using the real PSCI-via-SMC to power up the secondary CPUs as part of its bootup sequence.