qemu aarch64 gicv3 memory-mapped registers can't be modified - qemu

According to the info in generated dtb, the memory-mapped registers of gicv3-distributor have a base addr, which is 0x0800_0000.
And I have checked the validity by reading gicd_typer, which means the base addr is right.
But when I want to configure the gicv3-distributor (like changing GICD_ICACTIVER<n>), the value is not changed, keeping the default value. The same thing happens on any register of GICD in my machine.
I change them with assembly code as soon as the machine starts, not enable MMU.
ldr x8, =0x8000380
ldr x9, =0xffffffff
str x9, [x8]
dsb sy
ldr x0, [x8]
(gdb) i r x0
x0 0x0 0
Even I write to this memory by gdb set command, the value is also unchangeable.
The addr of GICD_ICACTIVER<n> should be 0x0800_0380(offset=0x380), which should be readable and writable, isn't it?
The UART(pl011)-related address(0x0900_0000) can be written normally.
Host machine: Ubuntu 18.04
qemu version: 7.1.50 (v7.1.0-256-g79dfa177)
Target OS: bare-metal
qemu command:
./qemu-system-aarch64 -machine virt,gic-version=3,secure=on,virtualization=on -cpu cortex-a76 -m 1G

The GICD_ICACTIVER register bits are write-1-to-clear, so if you write all-1s to the register then you should expect it to read back as all-0s.
More generally, not all hardware device registers will read back the same value that you write to them -- it depends entirely on the device.

Related

How to fix "qemu-system-mipsel: The following two regions overlap (in the memory address space)"?

I would like to run a Linux root filesystem for MIPSEL on qemu-system-mipsel.
The root filesystem was extracted from the firmware using "firmware-analysis-toolkit" (firmadyne).
However, After I build a root filesystem as required I encountered an error when I run
The script for run qemu is:
qemu-system-mipsel -M malta -kernel vmlinuz.elf \
-drive if=ide,format=raw,file=squashfs-factory.raw \
-append "root=/dev/sda1 console=ttyS0 nandsim.parts=64,64,64,64,64,64,64,64,64,64 \
rdinit=/firmadyne/preInit.sh rw debug ignore_loglevel print-fatal-signals=1 user_debug=31 firmadyn \
-nographic
If i use the vmlinux.elf provided by firmadyne toolkit (kernel 2.6.39.4+) everything works.
If i want to use a vmlinux.elf (kernel 5.4) provided by openwrt-imagebuilder (or compiled by me) i encountered an error this error:
The following two regions overlap (in the memory address space):
vmlinux-5.4.111.mipsel ELF program header segment 0 (addresses 0x0000000000001000 - 0x000000000084b910)
prom (addresses 0x0000000000002000 - 0x0000000000003040)
I've tried everything. How can it be fixed?
QEMU is complaining that the ELF file you've asked it to load is overlapping with the blob of 'prom' data that contains data to pass to the kernel such as memory size and the kernel command line. That PROM data always starts at address 0x2000. You need to build your kernel so that it doesn't try to put anything at that address.

QEMU "-bios" vs. "-kernel" vs. "-device loader,file=..."

For background, I'm running bare-metal QEMU-4.1.0 on aarch64.
There are several ways to get QEMU to load compiled code into memory. I'd like to understand what the underlying differences are, because I see very different behavior and the documentation doesn't shed any light.
Consider this first command line:
qemu-system-aarch64 \
-s -S \
-machine virt,secure=on,virtualization=on \
-m 512M \
-smp 4 \
-display none \
-nographic \
-semihosting \
-serial mon:stdio \
-kernel my_file.elf \
-device loader,addr=0x40004000,cpu_num=0 \
-device loader,addr=0x40004000,cpu_num=1 \
-device loader,addr=0x40004000,cpu_num=2 \
-device loader,addr=0x40004000,cpu_num=3 \
;
In another shell, if I launch gdb to see what QEMU has loaded into memory, it corresponds exactly to what I expect. In fact, gdb has a built-in command for this...
(gdb) compare-sections
Section .start, range 0x40004000 -- 0x40006164: matched.
Section .vectors, range 0x40006800 -- 0x40006f90: matched.
Section .text, range 0x40006fc0 -- 0x4002ca7c: matched.
...
Section .stacks, range 0x4207c120 -- 0x420bc120: matched.
(gdb) x/10x 0x40004000
0x40004000 <_start>: 0x14000800 0x00000000 0x00000000 0x00000000
...
Perfect! Everything in my ELF is at 0x40004000, and I see it all in memory, just how I expect! My first core boots and runs as I expect.
It is interesting to note that if I dump what is at location zero in memory, then there is stuff loaded down there. I didn't ask for it. I didn't explicitly load it. I don't execute it. It's not in my ELF file. I don't know what it is or where it came from. My GUESS is that QEMU has ASSUMED that I want some BIOS in the flash and has put one there. I don't know for sure. It also places something (small) at 0x40000000. I don't know what that is either... I do want to be careful that if I load something, we won't step on each other...
My first question then becomes: Can I enable some debug messages to understand what QEMU is loading where, and perhaps even WHY?
Continuing...
If I change my command line to REPLACE the "-kernel my_file.elf" switch with "-bios my_file.elf" switch (changing nothing else), and I repeat my run/gdb, then I see two things that are different...
First, I see that all my cores are running. I don't need to use PSCI calls to start them. Okay, but I don't think that's relevant to my issue. Second (and VERY important) is that my memory does NOT contain what I expect!
(gdb) compare-sections
Section .start, range 0x40004000 -- 0x40006164: MIS-MATCHED!
Section .vectors, range 0x40006800 -- 0x40006f90: MIS-MATCHED!
Section .text, range 0x40006fc0 -- 0x4002ca7c: MIS-MATCHED!
...
Section .stacks, range 0x4207c120 -- 0x420bc120: matched.
(gdb) x/8x 0x40000000
0x40004000 <_start>: 0x00000000 0x00000000 0x00000000 0x00000000
0x40004010 <_start+16>: 0x00000000 0x00000000 0x00000000 0x00000000
(gdb) x/8x 0x40006800
0x40006800 <my_vector_name>: 0x00000000 0x00000000 0x00000000 0x00000000
0x40006810 <my_vector_name+16>: 0x00000000 0x00000000 0x00000000 0x00000000
(gdb) x/8x 0x40006fc0
0x40006800 <my_symbol_name>: 0x00000000 0x00000000 0x00000000 0x00000000
0x40006810 <my_symbol_name+16>: 0x00000000 0x00000000 0x00000000 0x00000000
Everything is zero. I don't see MY code anywhere, although the mysterious code is still getting loaded at both 0x0 and 0x4000000. As you might also expect, the cores immediately die with an "Undefined Instruction" exception as soon as I issue "nexti" in my gdb.
Hmmm...
Okay, now I'll change the "-bios my_file.elf" to "-device loader,file=my_file.elf". I get the same result. I cannot find my code in memory.
What happens differently under-the-hood in QEMU between "-bios" and "-kernel"? Where is that documented, or where in the source can I follow it? How can I best debug this?
Thank you kind Sir/Madam!
Edit:
For debugging, all the good/relevant stuff seems to be in "virt.c"...
More Edit (to add info from "-device loader=my_file.elf")
My command line is:
/tools/gnu/qemu-4.1.0/bin/qemu-system-aarch64 \
-s -S \
-machine virt,secure=on,virtualization=on \
-cpu cortex-a53 \
-d int \
-m 512M \
-smp 4 \
-display none \
-nographic \
-semihosting \
-serial mon:stdio \
-device loader,file=NEW_AT_ZERO.elf \
;
Here's some of the relevant section of NEW_AT_ZERO.dis:
NEW_AT_ZERO.elf: file format elf64-littleaarch64
NEW_AT_ZERO.elf
architecture: aarch64, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x0000000000000000
Program Header:
LOAD off 0x0000000000010000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**16
filesz 0x00000000020b8120 memsz 0x00000000020b8120 flags rwx
NOTE off 0x0000000000043484 vaddr 0x0000000000033484 paddr 0x0000000000033484 align 2**2
filesz 0x0000000000000024 memsz 0x0000000000000024 flags r--
private flags = 0:
Sections:
Idx Name Size VMA LMA File off Algn
0 .start 00002164 0000000000000000 0000000000000000 00010000 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
1 .vectors 00000790 0000000000002800 0000000000002800 00012800 2**11
CONTENTS, ALLOC, LOAD, READONLY, CODE
2 .text 00025bbc 0000000000002fc0 0000000000002fc0 00012fc0 2**6
CONTENTS, ALLOC, LOAD, READONLY, CODE
3 .bss 0000a904 0000000000028b80 0000000000028b80 00038b7c 2**3
ALLOC
....
Contents of section .start:
0000 00080014 00000000 00000000 00000000 ................
....
1000 fd170094 a00038d5 01044092 020c7892 ......8...#...x.
1010 261842aa 660000b5 00038052 00005ed4 &.B.f......R..^.
1020 7f2003d5 ffffff17 00000000 00000000 . ..............
....
...but of course...
GNU gdb (Linaro_GDB-2017.05.09) 7.12.1.20170417-git
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=aarch64-none-elf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Remote debugging using localhost:1234
warning: No executable has been specified and target does not support
determining executable automatically. Try using the "file" command.
0x0000000000000000 in ?? ()
Reading symbols from ./NEW_AT_ZERO.elf...done.
(gdb) compare_sections
Undefined command: "compare_sections". Try "help".
(gdb) compare-sections
Section .start, range 0x0 -- 0x2164: MIS-MATCHED!
Section .vectors, range 0x2800 -- 0x2f90: MIS-MATCHED!
Section .text, range 0x2fc0 -- 0x28b7c: MIS-MATCHED!
....
Section .stacks, range 0x2078120 -- 0x20b8120: matched.
warning: One or more sections of the target image does not match
the loaded file
(gdb) x/4x 0
0x0 <_start>: 0x00000000 0x00000000 0x00000000 0x00000000
(gdb) x/12x 0x1000
0x1000 <symbol>: 0x00000000 0x00000000 0x00000000 0x00000000
0x1010 <symbol+16>: 0x00000000 0x00000000 0x00000000 0x00000000
0x1020 <end_symbol>: 0x00000000 0x00000000 0x00000000 0x00000000
QEMU's command line options for loading code into the guest are various, and often have different semantics between architectures or even between machine types for the same architecture. This is unfortunate but is the result of backwards-compatibility with older QEMU versions and a gradual accumulation of "it would be nice to Do The Right Thing for this image file type" special cases.
Broad summary:
-kernel is the "load a Linux kernel" option. It will load and boot the kernel in whatever way seems best for the architecture being used. For instance, for the x86 PC machine it will just provide the file to the guest BIOS and rely on the guest BIOS to do the actual loading of the file into RAM. On Arm, loading a Linux kernel means that we follow the rules the kernel lays down for how to boot it (https://www.kernel.org/doc/Documentation/arm64/booting.txt for 64-bit or https://www.kernel.org/doc/Documentation/arm/Booting for 32-bit), and we achieve that with a little bit of stub bootloader code (this is what you are seeing in low memory). The kernel boot rules also require that we provide it with a device tree blob in RAM, and this is the data at 0x40000000. We also, in accordance with Linux kernel boot expectations, handle secondary CPUs by either keeping them in PSCI powered-off state or via a little bit of secondary-CPU bootloader code which uses a WFI loop so that the primary can wake them up. (Which we do depends on the board model being used, because we do what the real board does, which especially for 32-bit boards varies a lot.)
As an oddball exception, for Arm if you pass an ELF file to -kernel, we'll assume that it is not a Linux kernel, and will boot it by just starting at the ELF entrypoint. We provide the DTB blob at the base of RAM, but only if it wouldn't overlap with the loaded ELF file. (Aside: for 'virt' in particular you want the DTB anyway, because we don't guarantee to keep devices in the same physical addresses between QEMU versions -- the DTB is how we tell guest code where it should look for things. You can rely on flash at 0x0 and RAM starting at 0x4000_0000, but really should pull all other device addresses from the DTB. In practice we have made efforts to avoid rearranging the board memory map, but reading the DTB is the right thing for guest code to do.)
-device loader is the "generic loader", which behaves the same on any architecture. It just loads an ELF image into guest RAM, and doesn't do anything to change the CPU reset behaviour. This is a good choice if you have a completely bare-metal image which includes the exception vector table and want to have it start in the same way the hardware would out of reset.
-bios is the "load a bios image, in whatever way seems good for this machine model" option. Again, this is a "do what I mean" kind of option whose specifics vary from machine model to machine model and from architecture to architecture; some machines don't support it at all. Some machines (eg x86 PC) will always load a bios, using a default binary if the user didn't specify. Some will load a bios if the user asks, but not otherwise (the arm virt board is like this). Generally a bios image is expected to be a "bare metal raw binary" image which will get loaded into some flash or ROM memory which corresponds to wherever the hardware starts execution when it comes out of reset. On at least some machines, including 'virt', you can instead provide the contents of the flash/ROM devices using a command line like "-drive if=pflash,...". This is an example of a common pattern in QEMU where you can either use a short "do what I mean" option that is convenient but has a lot of magic under the hood, or a longer "orthogonal" option which lets you specify lots of sub-options and get exactly the behaviour you want. Note that BIOS images should not be ELF files, they're expected to just be the raw data to put into the ROMs.
A lot of this is undocumented, because "I want to run a bare metal program of my own devising" is a very niche use case and because we don't have a good place in our documentation to make it easy to document the specifics of different board models.

How to build and boot Linux aarch64 with U-Boot with Buildroot on QEMU?

I tried:
git clone git://git.buildroot.net/buildroot
cd buildroot
git checkout 2019.08
make qemu_aarch64_virt_defconfig
make menuconfig
In menuconfig, I set:
Bootloaders
U-Boot configuration (Using an in-tree board defconfig file)
qemu_arm64
Kernel
Install kernel image to /boot in target
and finally:
make BR2_JLEVEL="$nproc"
Now, I can boot fine without U-Boot with the command line mentioned at: How to download the Torvalds Linux Kernel master, (re)compile it, and boot it with QEMU?
./output/host/usr/bin/qemu-system-aarch64 -M virt -cpu cortex-a57 -nographic -smp 1 -kernel output/images/Image -append "root=/dev/vda console=ttyAMA0" -netdev user,id=eth0 -device virtio-net-device,netdev=eth0 -drive file=output/images/rootfs.ext4,if=none,format=raw,id=hd0 -device virtio-blk-device,drive=hd0
but that is not using U-Boot.
When I do:
ls -l output/images/
it contains:
-rw-r--r-- 1 ciro ciro 6.5M 2019-09-20_13:36:23 Image
-rw-r--r-- 1 ciro ciro 60M 2019-09-20_13:39:02 rootfs.ext2
lrwxrwxrwx 1 ciro ciro 11 2019-09-20_13:36:25 rootfs.ext4 -> rootfs.ext2
-rw-r--r-- 1 ciro ciro 583K 2019-09-20_13:34:15 u-boot.bin
so there is a U-Boot binary there: u-boot.bin, but how do I use that with QEMU?
I tried as mentioned at: Can ARM qemu system emulator boot from card image without kernel param? to remove -kernel and -append and add -bios u-boot.bin:
./output/host/usr/bin/qemu-system-aarch64 -M virt -cpu cortex-a57 -nographic -smp 1 -bios output/images/u-boot.bin -netdev user,id=eth0 -device virtio-net-device,netdev=eth0 -drive file=output/images/rootfs.ext4,if=none,format=raw,id=hd0 -device virtio-blk-device,drive=hd0
Now I do get the U-Boot shell, but boot fails and leaves me no the U-Boot prompt:
U-Boot 2019.07 (Sep 20 2019 - 13:34:10 +0100)
DRAM: 128 MiB
Flash: 128 MiB
*** Warning - bad CRC, using default environment
In: pl011#9000000
Out: pl011#9000000
Err: pl011#9000000
Net:
Warning: virtio-net#31 using MAC address from ROM
eth0: virtio-net#31
Hit any key to stop autoboot: 0
starting USB...
No working controllers found
USB is stopped. Please issue 'usb start' first.
scanning bus for devices...
Device 0: unknown device
Device 0: QEMU VirtIO Block Device
Type: Hard Disk
Capacity: 60.0 MB = 0.0 GB (122880 x 512)
... is now current device
** No partition table - virtio 0 **
starting USB...
No working controllers found
BOOTP broadcast 1
DHCP client bound to address 10.0.2.15 (2 ms)
Using virtio-net#31 device
TFTP from server 10.0.2.2; our IP address is 10.0.2.15
Filename 'boot.scr.uimg'.
Load address: 0x40200000
Loading: *
TFTP error: 'Access violation' (2)
Not retrying...
BOOTP broadcast 1
DHCP client bound to address 10.0.2.15 (0 ms)
Using virtio-net#31 device
TFTP from server 10.0.2.2; our IP address is 10.0.2.15
Filename 'boot.scr.uimg'.
Load address: 0x40400000
Loading: *
TFTP error: 'Access violation' (2)
Not retrying...
=>
so it appears that U-Boot cannot handle the VirtIO device? Or according to Peter, I have to create a partition table. I couldn't find that automatically in Buildroot, but I could do it manually, here is one approach: https://unix.stackexchange.com/questions/209566/how-to-format-a-partition-inside-of-an-img-file/527132#527132
Another approach would be to keep -kernel -append and let QEMU put the kernel into memory as done without U-Boot, and then use the booti U-Boot command I've found on help:
booti - boot Linux kernel 'Image' format from memory
so I just need to find out its address. But that is kind of cheating since I want U-boot to do the hard work rather than cheat with QEMU.
My goal is to reach a good setup to develop U-Boot and QEMU's early boot stuff.
Given that u-boot correctly detects the virtio block device, I think it is unlikely that it cannot handle it. The error printed is "** No partition table - virtio 0 **", which is correct, because you've set up the block device to contain just rootfs.ext4, which will be a filesystem image. That suggests that you'll have more luck if you create a disk image with a partition table and write the rootfs to a partition within the disk image.
I followed Peter's advice and put it into a partitioned image with the sfdisk-fs-to-img command from https://unix.stackexchange.com/questions/209566/how-to-format-a-partition-inside-of-an-img-file/527132#527132
I am now able to read the root filesystem with:
ls virtio 0 /boot
and that contains Image file.
Now I think there are only some U-Boot specifics to resolve, which I'm not very familiar with:
load /boot/Image into memory with something like load virtio 0 0x100000 /boot/Image. TODO which address is valid? This arbitrary choice gave ** Reading file would overwrite reserved memory **
find out how to load the DTB and kernel CLI arguments. The DTB would need to be auto-generated with QEMU with qemu-system-aarch64 -machine dumpdtb=dtb.dtb
boot it with something like: booti 0x100000
I was hoping Buildroot would have automated things a bit more for me sadface.

Raspberry Pi 1B Secure Configuration Register

Now I make low level bare-metal tool for RPi.
And I need to get Secure Configuration Register value.
I wrote the following instruction mrc p15, 0, r0, c1, c1, 0 to get it.
But CPU goes into Undefined Exception Mode and CPSR value is 0x600001DB.
Instruction of reading SCR value is the first instruction being executed by CPU.
I'd read ARM1176JZF-S TRM r0p7 several times but I've not found any restriction on using SCR reading instruction except being CPU in the Secure Privileged Mode but according to TRM this CPU starts from Secure Privileged Mode. If to be more concrete the initial mode is Secure Supervisor Mode.
I use the following command to execute code with QEMU
qemu-system-arm -cpu arm1176 -M versatilepb -m 256 -nographic -kernel start.elf -s -S -monitor stdio
I can't understand what I overlooked?
QEMU's versatilepb board does not support TrustZone: it creates a CPU with that feature disabled.
Other QEMU board models do support TZ, if you want to play with it: for instance vexpress-a9, vexpress-a15 and raspi2; also "virt", if you pass -machine secure=on on the QEMU command line.

how to communicate an exit code from the QEMU guest to the host on arm64 / aarch64 (equivalent to isa-debug-exit)

I've found I can have the guest write to the port specified by the command line flag isa-debug-exit, and the value written to the port gets used as the exit code of QEMU (after some predictable transformation).
Is there an equivalent mechanism for aarch64 / arm64?
You can use the semihosting API for that: see
qemu-system-aarch64 exit from within the guest system for details (and also the caveats about needing to trust the code in your guest if you use it).