QEMU AARCH64 "virt" Machine SMP CPUs Starting in "running" vs. "halted" State - qemu

Question

I'm working on bare-metal. No Linux, libraries, etc. I'm writing processor boot code in ASM and jumping to my compiled C code.
My command line is:
% qemu-system-aarch64 \
-s -S \
-machine virt,secure=on,virtualization=on \
-cpu cortex-a53 \
-d int \
-m 512M \
-smp 4 \
-display none \
-nographic \
-semihosting \
-serial mon:stdio \
-kernel my_file.elf \
-device loader,addr=0x40004000,cpu-num=0 \
-device loader,addr=0x40004000,cpu-num=1 \
-device loader,addr=0x40004000,cpu-num=2 \
-device loader,addr=0x40004000,cpu-num=3 \
;
When I connect gcc at the beginning, I can see:
(gdb) info threads
Id Target Id Frame
* 1 Thread 1.1 (CPU#0 [running]) _start () at .../start.S:20
2 Thread 1.2 (CPU#1 [halted ]) _start () at .../start.S:20
3 Thread 1.3 (CPU#2 [halted ]) _start () at .../start.S:20
4 Thread 1.4 (CPU#3 [halted ]) _start () at .../start.S:20
I want those other three processors to start in the "running" state, not "halted". How?
Note that my DTS contains this section:
psci {
migrate = < 0xc4000005 >;
cpu_on = < 0xc4000003 >;
cpu_off = < 0x84000002 >;
cpu_suspend = < 0xc4000001 >;
method = "smc";
compatible = "arm,psci-0.2\0arm,psci";
};
However, I'm not sure what to do with that. Adding many different lines of this form, don't seem to help:
-device loader,addr=0xc4000003,data=0x80000000,data-len=4
I'm not sure if I'm on the right track with this ARM PSCI thing? ARM's specification seems to define the "interface", not the system "implementation". However, I don't see the PSCI as "real" registers mentioned in the "virt" documentation/source. There is no "SMC" device mentioned in the DTS.
How does QEMU decide whether an SMP processor is "running" or "halted" on start and how can I influence that?
Based on #Peter-Maydell's answer below, I need to do one of two things...
Switch "-kernel" to "-bios". I do this, but my code doesn't load as I expect. My *.elf file has several sections; some in FLASH and some in DDR (above 0x40000000). Maybe that's the problem?
Change my boot code to setup and issue the SMC instruction to make the ARM PSCI "CPU_ON" call that QEMU will recognize and powerup the other processors. Code like this runs but doesn't seem to "do" anything...
ldr w0, =0xc4000003 // CPU_ON code from the DTS file
mov x1, 1 // CPU #1 in cluster zero (format of MPIDR register?)
ldr x2, _boot // Jump address 0x40006000 (FYI)
mov x3, 1 // context ID (meaningful only to caller)
smc #0 // GO!
// result is in x0 -> PSCI_RET_INVALID_PARAMS

Answer 1

Using the response provided by Peter Maydell, I am providing here a Minimal, Reproducible Example for people who may be interested.
Downloading/installing aarch64-elf toolchain:
wget "https://developer.arm.com/-/media/Files/downloads/gnu-a/8.3-2019.03/binrel/gcc-arm-8.3-2019.03-x86_64-aarch64-elf.tar.xz?revision=d678fd94-0ac4-485a-8054-1fbc60622a89&la=en"
mkdir -p /opt/arm
tar Jxf gcc-arm-8.3-2019.03-x86_64-aarch64-elf.tar.xz -C /opt/arm
Example files:
loop.s:
.title "loop.s"
.arch armv8-a
.text
.global Reset_Handler
Reset_Handler: mrs x0, mpidr_el1
and x0,x0, 0b11
cmp x0, #0
b.eq Core0
cmp x0, #1
b.eq Core1
cmp x0, #2
b.eq Core2
cmp x0, #3
b.eq Core3
Error: b .
Core0: b .
Core1: b .
Core2: b .
Core3: b .
.end
build.sh:
#!/bin/bash
set -e
CROSS_COMPILE=/opt/arm/gcc-arm-8.3-2019.03-x86_64-aarch64-elf/bin/aarch64-elf-
AS=${CROSS_COMPILE}as
LD=${CROSS_COMPILE}ld
OBJCOPY=${CROSS_COMPILE}objcopy
OBJDUMP=${CROSS_COMPILE}objdump
${AS} -g -o loop.o loop.s
${LD} -g -gc-sections -g -e Reset_Handler -Ttext-segment=0x40004000 -Map=loop.map -o loop.elf loop.o
${OBJDUMP} -d loop.elf
qemu.sh:
#!/bin/bash
set -e
QEMU_SYSTEM_AARCH64=qemu-system-aarch64
${QEMU_SYSTEM_AARCH64} \
-s -S \
-machine virt,secure=on,virtualization=on \
-cpu cortex-a53 \
-d int \
-m 512M \
-smp 4 \
-display none \
-nographic \
-semihosting \
-serial mon:stdio \
-bios loop.elf \
-device loader,addr=0x40004000,cpu-num=0 \
-device loader,addr=0x40004000,cpu-num=1 \
-device loader,addr=0x40004000,cpu-num=2 \
-device loader,addr=0x40004000,cpu-num=3 \
;
loop.gdb:
target remote localhost:1234
file loop.elf
load loop.elf
disassemble Reset_Handler
info threads
continue
debug.sh:
#!/bin/bash
CROSS_COMPILE=/opt/arm/gcc-arm-8.3-2019.03-x86_64-aarch64-elf/bin/aarch64-elf-
GDB=${CROSS_COMPILE}gdb
${GDB} --command=loop.gdb
Executing the program - two consoles will be needed.
First console:
./build.sh
Output should look like:
/opt/arm/gcc-arm-8.3-2019.03-x86_64-aarch64-elf/bin/aarch64-elf-ld: warning: address of `text-segment' isn't multiple of maximum page size
loop.elf: file format elf64-littleaarch64
Disassembly of section .text:
0000000040004000 <Reset_Handler>:
40004000: d53800a0 mrs x0, mpidr_el1
40004004: 92400400 and x0, x0, #0x3
40004008: f100001f cmp x0, #0x0
4000400c: 54000100 b.eq 4000402c <Core0> // b.none
40004010: f100041f cmp x0, #0x1
40004014: 540000e0 b.eq 40004030 <Core1> // b.none
40004018: f100081f cmp x0, #0x2
4000401c: 540000c0 b.eq 40004034 <Core2> // b.none
40004020: f1000c1f cmp x0, #0x3
40004024: 540000a0 b.eq 40004038 <Core3> // b.none
0000000040004028 <Error>:
40004028: 14000000 b 40004028 <Error>
000000004000402c <Core0>:
4000402c: 14000000 b 4000402c <Core0>
0000000040004030 <Core1>:
40004030: 14000000 b 40004030 <Core1>
0000000040004034 <Core2>:
40004034: 14000000 b 40004034 <Core2>
0000000040004038 <Core3>:
40004038: 14000000 b 40004038 <Core3>
Then:
./qemu.sh
Second console:
./debug.sh
Output should look like:
GNU gdb (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) 8.2.1.20190227-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=aarch64-elf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.linaro.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
warning: No executable has been specified and target does not support
determining executable automatically. Try using the "file" command.
0x0000000040004000 in ?? ()
Loading section .text, size 0x3c lma 0x40004000
Start address 0x40004000, load size 60
Transfer rate: 480 bits in <1 sec, 60 bytes/write.
Dump of assembler code for function Reset_Handler:
=> 0x0000000040004000 <+0>: mrs x0, mpidr_el1
0x0000000040004004 <+4>: and x0, x0, #0x3
0x0000000040004008 <+8>: cmp x0, #0x0
0x000000004000400c <+12>: b.eq 0x4000402c <Core0> // b.none
0x0000000040004010 <+16>: cmp x0, #0x1
0x0000000040004014 <+20>: b.eq 0x40004030 <Core1> // b.none
0x0000000040004018 <+24>: cmp x0, #0x2
0x000000004000401c <+28>: b.eq 0x40004034 <Core2> // b.none
0x0000000040004020 <+32>: cmp x0, #0x3
0x0000000040004024 <+36>: b.eq 0x40004038 <Core3> // b.none
End of assembler dump.
Id Target Id Frame
* 1 Thread 1.1 (CPU#0 [running]) Reset_Handler () at loop.s:5
2 Thread 1.2 (CPU#1 [running]) Reset_Handler () at loop.s:5
3 Thread 1.3 (CPU#2 [running]) Reset_Handler () at loop.s:5
4 Thread 1.4 (CPU#3 [running]) Reset_Handler () at loop.s:5
All four cores are stopped at address 0x40004000/Reset_Handler, and were started by the continue command in loop.gdb.
Press CTRL+C in the second console:
^C
Thread 1 received signal SIGINT, Interrupt.
Core0 () at loop.s:16
16 Core0: b .
(gdb)
Core #0 was executing code at Core0 label.
Enter the following command (still in the second console):
(gdb) info threads
Id Target Id Frame
* 1 Thread 1.1 (CPU#0 [running]) Core0 () at loop.s:16
2 Thread 1.2 (CPU#1 [running]) Core1 () at loop.s:17
3 Thread 1.3 (CPU#2 [running]) Core2 () at loop.s:18
4 Thread 1.4 (CPU#3 [running]) Core3 () at loop.s:19
(gdb)
Cores #1,#2 and #3 were executing the code at the respective Core1, Core2, Core3 labels prior to be stopped by the CTRL+C.
Description of the MPIDR_EL1 register is available here: the two lasts bits of MPIDR_EL1.Aff0 were used by all four cores for determining their respective core numbers.

Answer 2

This depends on the board model -- generally we follow what the hardware does, and some boards start all CPUs from power-on, and some don't. For the 'virt' board (which is specific to QEMU) what we generally do is use PSCI, which is the Arm standard firmware interface for powering SMP CPUs up and down (among other things; you can also use it for 'power down entire machine', for instance). On startup only the primary CPU is running, and it's the job of guest code to use the PSCI API to start the secondaries. That's what that psci node in the DTS is telling the guest -- it tells the guest what specific form of the PSCI ABI QEMU implements, and in particular whether the guest should use the 'hvc' or 'smc' instruction to call PSCI functions. What QEMU is doing here is emulating a "hardware + firmware" combination -- the guest executes an 'smc' instruction and QEMU performs the actions that on real hardware would be performed by a bit of firmware code running at EL3.
The virt board does also have another mode of operation which is intended for when you want to run a guest which is itself EL3 firmware (for instance if you want to run OVMF/UEFI at EL3). If you start QEMU with -machine secure=true to enable EL3 emulation and you also provide a guest firmware blob via either -bios or -drive if=pflash,..., then QEMU will assume your firmware wants to run at EL3 and provide PSCI services itself, so it will start with all CPUs powered on and let the firmware deal with sorting them out.
A simple example of making a PSCI call to turn on another CPU (in this case cpu #4 of 8):
.equ PSCI_0_2_FN64_CPU_ON, 0xc4000003
ldr x0, =PSCI_0_2_FN64_CPU_ON
ldr x1, =4 /* target CPU's MPIDR affinity */
ldr x2, =0x10000 /* entry point */
ldr x3, =0 /* context ID: put into target CPU's x0 */
smc 0