qemu version is 7.1.0
I use qemu to simulate xlnx-zcu102. my qemu command is below, it can run OK:
./qemu-system-aarch64 -M xlnx-zcu102 -smp 4 -m 4G ......
but, when I add -device virtio-gpu or -device virtio-gpu-device,
(I added only one device: virtio-gpu)
qemu tell me ERROR like this:
qemu-system-aarch64: -device virtio-gpu: No 'PCI' bus found for device 'virtio-gpu-pci'
I don't know how to add a virtio-gpu device to xlnx-zcu102 in qemu command.
You can't add that device to that machine type. The xlnx-zcu102 machine type simulates a piece of real hardware that does not have a PCI bus, and so you cannot plug PCI devices into it. You need to either use a different machine type, or else not use virtio devices.
For background, I'm running bare-metal QEMU-4.1.0 on aarch64.
There are several ways to get QEMU to load compiled code into memory. I'd like to understand what the underlying differences are, because I see very different behavior and the documentation doesn't shed any light.
Consider this first command line:
qemu-system-aarch64 \
-s -S \
-machine virt,secure=on,virtualization=on \
-m 512M \
-smp 4 \
-display none \
-nographic \
-semihosting \
-serial mon:stdio \
-kernel my_file.elf \
-device loader,addr=0x40004000,cpu_num=0 \
-device loader,addr=0x40004000,cpu_num=1 \
-device loader,addr=0x40004000,cpu_num=2 \
-device loader,addr=0x40004000,cpu_num=3 \
;
In another shell, if I launch gdb to see what QEMU has loaded into memory, it corresponds exactly to what I expect. In fact, gdb has a built-in command for this...
(gdb) compare-sections
Section .start, range 0x40004000 -- 0x40006164: matched.
Section .vectors, range 0x40006800 -- 0x40006f90: matched.
Section .text, range 0x40006fc0 -- 0x4002ca7c: matched.
...
Section .stacks, range 0x4207c120 -- 0x420bc120: matched.
(gdb) x/10x 0x40004000
0x40004000 <_start>: 0x14000800 0x00000000 0x00000000 0x00000000
...
Perfect! Everything in my ELF is at 0x40004000, and I see it all in memory, just how I expect! My first core boots and runs as I expect.
It is interesting to note that if I dump what is at location zero in memory, then there is stuff loaded down there. I didn't ask for it. I didn't explicitly load it. I don't execute it. It's not in my ELF file. I don't know what it is or where it came from. My GUESS is that QEMU has ASSUMED that I want some BIOS in the flash and has put one there. I don't know for sure. It also places something (small) at 0x40000000. I don't know what that is either... I do want to be careful that if I load something, we won't step on each other...
My first question then becomes: Can I enable some debug messages to understand what QEMU is loading where, and perhaps even WHY?
Continuing...
If I change my command line to REPLACE the "-kernel my_file.elf" switch with "-bios my_file.elf" switch (changing nothing else), and I repeat my run/gdb, then I see two things that are different...
First, I see that all my cores are running. I don't need to use PSCI calls to start them. Okay, but I don't think that's relevant to my issue. Second (and VERY important) is that my memory does NOT contain what I expect!
(gdb) compare-sections
Section .start, range 0x40004000 -- 0x40006164: MIS-MATCHED!
Section .vectors, range 0x40006800 -- 0x40006f90: MIS-MATCHED!
Section .text, range 0x40006fc0 -- 0x4002ca7c: MIS-MATCHED!
...
Section .stacks, range 0x4207c120 -- 0x420bc120: matched.
(gdb) x/8x 0x40000000
0x40004000 <_start>: 0x00000000 0x00000000 0x00000000 0x00000000
0x40004010 <_start+16>: 0x00000000 0x00000000 0x00000000 0x00000000
(gdb) x/8x 0x40006800
0x40006800 <my_vector_name>: 0x00000000 0x00000000 0x00000000 0x00000000
0x40006810 <my_vector_name+16>: 0x00000000 0x00000000 0x00000000 0x00000000
(gdb) x/8x 0x40006fc0
0x40006800 <my_symbol_name>: 0x00000000 0x00000000 0x00000000 0x00000000
0x40006810 <my_symbol_name+16>: 0x00000000 0x00000000 0x00000000 0x00000000
Everything is zero. I don't see MY code anywhere, although the mysterious code is still getting loaded at both 0x0 and 0x4000000. As you might also expect, the cores immediately die with an "Undefined Instruction" exception as soon as I issue "nexti" in my gdb.
Hmmm...
Okay, now I'll change the "-bios my_file.elf" to "-device loader,file=my_file.elf". I get the same result. I cannot find my code in memory.
What happens differently under-the-hood in QEMU between "-bios" and "-kernel"? Where is that documented, or where in the source can I follow it? How can I best debug this?
Thank you kind Sir/Madam!
Edit:
For debugging, all the good/relevant stuff seems to be in "virt.c"...
More Edit (to add info from "-device loader=my_file.elf")
My command line is:
/tools/gnu/qemu-4.1.0/bin/qemu-system-aarch64 \
-s -S \
-machine virt,secure=on,virtualization=on \
-cpu cortex-a53 \
-d int \
-m 512M \
-smp 4 \
-display none \
-nographic \
-semihosting \
-serial mon:stdio \
-device loader,file=NEW_AT_ZERO.elf \
;
Here's some of the relevant section of NEW_AT_ZERO.dis:
NEW_AT_ZERO.elf: file format elf64-littleaarch64
NEW_AT_ZERO.elf
architecture: aarch64, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x0000000000000000
Program Header:
LOAD off 0x0000000000010000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**16
filesz 0x00000000020b8120 memsz 0x00000000020b8120 flags rwx
NOTE off 0x0000000000043484 vaddr 0x0000000000033484 paddr 0x0000000000033484 align 2**2
filesz 0x0000000000000024 memsz 0x0000000000000024 flags r--
private flags = 0:
Sections:
Idx Name Size VMA LMA File off Algn
0 .start 00002164 0000000000000000 0000000000000000 00010000 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
1 .vectors 00000790 0000000000002800 0000000000002800 00012800 2**11
CONTENTS, ALLOC, LOAD, READONLY, CODE
2 .text 00025bbc 0000000000002fc0 0000000000002fc0 00012fc0 2**6
CONTENTS, ALLOC, LOAD, READONLY, CODE
3 .bss 0000a904 0000000000028b80 0000000000028b80 00038b7c 2**3
ALLOC
....
Contents of section .start:
0000 00080014 00000000 00000000 00000000 ................
....
1000 fd170094 a00038d5 01044092 020c7892 ......8...#...x.
1010 261842aa 660000b5 00038052 00005ed4 &.B.f......R..^.
1020 7f2003d5 ffffff17 00000000 00000000 . ..............
....
...but of course...
GNU gdb (Linaro_GDB-2017.05.09) 7.12.1.20170417-git
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=aarch64-none-elf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Remote debugging using localhost:1234
warning: No executable has been specified and target does not support
determining executable automatically. Try using the "file" command.
0x0000000000000000 in ?? ()
Reading symbols from ./NEW_AT_ZERO.elf...done.
(gdb) compare_sections
Undefined command: "compare_sections". Try "help".
(gdb) compare-sections
Section .start, range 0x0 -- 0x2164: MIS-MATCHED!
Section .vectors, range 0x2800 -- 0x2f90: MIS-MATCHED!
Section .text, range 0x2fc0 -- 0x28b7c: MIS-MATCHED!
....
Section .stacks, range 0x2078120 -- 0x20b8120: matched.
warning: One or more sections of the target image does not match
the loaded file
(gdb) x/4x 0
0x0 <_start>: 0x00000000 0x00000000 0x00000000 0x00000000
(gdb) x/12x 0x1000
0x1000 <symbol>: 0x00000000 0x00000000 0x00000000 0x00000000
0x1010 <symbol+16>: 0x00000000 0x00000000 0x00000000 0x00000000
0x1020 <end_symbol>: 0x00000000 0x00000000 0x00000000 0x00000000
QEMU's command line options for loading code into the guest are various, and often have different semantics between architectures or even between machine types for the same architecture. This is unfortunate but is the result of backwards-compatibility with older QEMU versions and a gradual accumulation of "it would be nice to Do The Right Thing for this image file type" special cases.
Broad summary:
-kernel is the "load a Linux kernel" option. It will load and boot the kernel in whatever way seems best for the architecture being used. For instance, for the x86 PC machine it will just provide the file to the guest BIOS and rely on the guest BIOS to do the actual loading of the file into RAM. On Arm, loading a Linux kernel means that we follow the rules the kernel lays down for how to boot it (https://www.kernel.org/doc/Documentation/arm64/booting.txt for 64-bit or https://www.kernel.org/doc/Documentation/arm/Booting for 32-bit), and we achieve that with a little bit of stub bootloader code (this is what you are seeing in low memory). The kernel boot rules also require that we provide it with a device tree blob in RAM, and this is the data at 0x40000000. We also, in accordance with Linux kernel boot expectations, handle secondary CPUs by either keeping them in PSCI powered-off state or via a little bit of secondary-CPU bootloader code which uses a WFI loop so that the primary can wake them up. (Which we do depends on the board model being used, because we do what the real board does, which especially for 32-bit boards varies a lot.)
As an oddball exception, for Arm if you pass an ELF file to -kernel, we'll assume that it is not a Linux kernel, and will boot it by just starting at the ELF entrypoint. We provide the DTB blob at the base of RAM, but only if it wouldn't overlap with the loaded ELF file. (Aside: for 'virt' in particular you want the DTB anyway, because we don't guarantee to keep devices in the same physical addresses between QEMU versions -- the DTB is how we tell guest code where it should look for things. You can rely on flash at 0x0 and RAM starting at 0x4000_0000, but really should pull all other device addresses from the DTB. In practice we have made efforts to avoid rearranging the board memory map, but reading the DTB is the right thing for guest code to do.)
-device loader is the "generic loader", which behaves the same on any architecture. It just loads an ELF image into guest RAM, and doesn't do anything to change the CPU reset behaviour. This is a good choice if you have a completely bare-metal image which includes the exception vector table and want to have it start in the same way the hardware would out of reset.
-bios is the "load a bios image, in whatever way seems good for this machine model" option. Again, this is a "do what I mean" kind of option whose specifics vary from machine model to machine model and from architecture to architecture; some machines don't support it at all. Some machines (eg x86 PC) will always load a bios, using a default binary if the user didn't specify. Some will load a bios if the user asks, but not otherwise (the arm virt board is like this). Generally a bios image is expected to be a "bare metal raw binary" image which will get loaded into some flash or ROM memory which corresponds to wherever the hardware starts execution when it comes out of reset. On at least some machines, including 'virt', you can instead provide the contents of the flash/ROM devices using a command line like "-drive if=pflash,...". This is an example of a common pattern in QEMU where you can either use a short "do what I mean" option that is convenient but has a lot of magic under the hood, or a longer "orthogonal" option which lets you specify lots of sub-options and get exactly the behaviour you want. Note that BIOS images should not be ELF files, they're expected to just be the raw data to put into the ROMs.
A lot of this is undocumented, because "I want to run a bare metal program of my own devising" is a very niche use case and because we don't have a good place in our documentation to make it easy to document the specifics of different board models.
I tried:
git clone git://git.buildroot.net/buildroot
cd buildroot
git checkout 2019.08
make qemu_aarch64_virt_defconfig
make menuconfig
In menuconfig, I set:
Bootloaders
U-Boot configuration (Using an in-tree board defconfig file)
qemu_arm64
Kernel
Install kernel image to /boot in target
and finally:
make BR2_JLEVEL="$nproc"
Now, I can boot fine without U-Boot with the command line mentioned at: How to download the Torvalds Linux Kernel master, (re)compile it, and boot it with QEMU?
./output/host/usr/bin/qemu-system-aarch64 -M virt -cpu cortex-a57 -nographic -smp 1 -kernel output/images/Image -append "root=/dev/vda console=ttyAMA0" -netdev user,id=eth0 -device virtio-net-device,netdev=eth0 -drive file=output/images/rootfs.ext4,if=none,format=raw,id=hd0 -device virtio-blk-device,drive=hd0
but that is not using U-Boot.
When I do:
ls -l output/images/
it contains:
-rw-r--r-- 1 ciro ciro 6.5M 2019-09-20_13:36:23 Image
-rw-r--r-- 1 ciro ciro 60M 2019-09-20_13:39:02 rootfs.ext2
lrwxrwxrwx 1 ciro ciro 11 2019-09-20_13:36:25 rootfs.ext4 -> rootfs.ext2
-rw-r--r-- 1 ciro ciro 583K 2019-09-20_13:34:15 u-boot.bin
so there is a U-Boot binary there: u-boot.bin, but how do I use that with QEMU?
I tried as mentioned at: Can ARM qemu system emulator boot from card image without kernel param? to remove -kernel and -append and add -bios u-boot.bin:
./output/host/usr/bin/qemu-system-aarch64 -M virt -cpu cortex-a57 -nographic -smp 1 -bios output/images/u-boot.bin -netdev user,id=eth0 -device virtio-net-device,netdev=eth0 -drive file=output/images/rootfs.ext4,if=none,format=raw,id=hd0 -device virtio-blk-device,drive=hd0
Now I do get the U-Boot shell, but boot fails and leaves me no the U-Boot prompt:
U-Boot 2019.07 (Sep 20 2019 - 13:34:10 +0100)
DRAM: 128 MiB
Flash: 128 MiB
*** Warning - bad CRC, using default environment
In: pl011#9000000
Out: pl011#9000000
Err: pl011#9000000
Net:
Warning: virtio-net#31 using MAC address from ROM
eth0: virtio-net#31
Hit any key to stop autoboot: 0
starting USB...
No working controllers found
USB is stopped. Please issue 'usb start' first.
scanning bus for devices...
Device 0: unknown device
Device 0: QEMU VirtIO Block Device
Type: Hard Disk
Capacity: 60.0 MB = 0.0 GB (122880 x 512)
... is now current device
** No partition table - virtio 0 **
starting USB...
No working controllers found
BOOTP broadcast 1
DHCP client bound to address 10.0.2.15 (2 ms)
Using virtio-net#31 device
TFTP from server 10.0.2.2; our IP address is 10.0.2.15
Filename 'boot.scr.uimg'.
Load address: 0x40200000
Loading: *
TFTP error: 'Access violation' (2)
Not retrying...
BOOTP broadcast 1
DHCP client bound to address 10.0.2.15 (0 ms)
Using virtio-net#31 device
TFTP from server 10.0.2.2; our IP address is 10.0.2.15
Filename 'boot.scr.uimg'.
Load address: 0x40400000
Loading: *
TFTP error: 'Access violation' (2)
Not retrying...
=>
so it appears that U-Boot cannot handle the VirtIO device? Or according to Peter, I have to create a partition table. I couldn't find that automatically in Buildroot, but I could do it manually, here is one approach: https://unix.stackexchange.com/questions/209566/how-to-format-a-partition-inside-of-an-img-file/527132#527132
Another approach would be to keep -kernel -append and let QEMU put the kernel into memory as done without U-Boot, and then use the booti U-Boot command I've found on help:
booti - boot Linux kernel 'Image' format from memory
so I just need to find out its address. But that is kind of cheating since I want U-boot to do the hard work rather than cheat with QEMU.
My goal is to reach a good setup to develop U-Boot and QEMU's early boot stuff.
Given that u-boot correctly detects the virtio block device, I think it is unlikely that it cannot handle it. The error printed is "** No partition table - virtio 0 **", which is correct, because you've set up the block device to contain just rootfs.ext4, which will be a filesystem image. That suggests that you'll have more luck if you create a disk image with a partition table and write the rootfs to a partition within the disk image.
I followed Peter's advice and put it into a partitioned image with the sfdisk-fs-to-img command from https://unix.stackexchange.com/questions/209566/how-to-format-a-partition-inside-of-an-img-file/527132#527132
I am now able to read the root filesystem with:
ls virtio 0 /boot
and that contains Image file.
Now I think there are only some U-Boot specifics to resolve, which I'm not very familiar with:
load /boot/Image into memory with something like load virtio 0 0x100000 /boot/Image. TODO which address is valid? This arbitrary choice gave ** Reading file would overwrite reserved memory **
find out how to load the DTB and kernel CLI arguments. The DTB would need to be auto-generated with QEMU with qemu-system-aarch64 -machine dumpdtb=dtb.dtb
boot it with something like: booti 0x100000
I was hoping Buildroot would have automated things a bit more for me sadface.
I've found I can have the guest write to the port specified by the command line flag isa-debug-exit, and the value written to the port gets used as the exit code of QEMU (after some predictable transformation).
Is there an equivalent mechanism for aarch64 / arm64?
You can use the semihosting API for that: see
qemu-system-aarch64 exit from within the guest system for details (and also the caveats about needing to trust the code in your guest if you use it).
I am using qemu emulator for aarch64 and want to create an outside checkpoint (or fast forwarding) to save all I need to restart the system just from the point when I create checkpoint. (In fact, I want to skip the booting step) I only found something on qemu VM snapshot and fast forwarding, but it does not work for the emulator. Is there any checkpoint function for qemu emulator?
A savevm snapshot should do what you want. The short answer is that you need to set up a QCOW2 disk for the snapshots to be saved to, and then in the monitor you can use the 'savevm' command to take the snapshot. Then the command line '-loadvm' option will let you resume from there. This all works fine in emulation of AArch64.
https://translatedcode.wordpress.com/2015/07/06/tricks-for-debugging-qemu-savevm-snapshots/ has a more detailed tutorial.
Minimal example
Peter's answer just worked for me, but let me provide a fully reproducible example.
I have fully automated everything at: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/1e0f0b492855219351b0bfa2eec4d3a6811fcaaa#snapshot
The key step is to convert the image to qcow2 as explained at: https://docs.openstack.org/image-guide/convert-images.html
cd buildroot/output.x86_64~/images
qemu-img convert -f raw -O qcow2 rootfs.ext2 rootfs.ext2.qcow2
And the final QEMU command used was:
./buildroot/output.x86_64~/host/usr/bin/qemu-system-x86_64 -m 128M -monitor telnet::45454,server,nowait -netdev user,hostfwd=tcp::45455-:45455,id=net0 -smp 1 -M pc -append 'root=/dev/vda nopat nokaslr norandmaps printk.devkmsg=on printk.time=y console=ttyS0' -device edu -device virtio-net-pci,netdev=net0 -drive file=./buildroot/output.x86_64~/images/rootfs.ext2.qcow2,if=virtio,format=qcow2 -kernel ./buildroot/output.x86_64~/images/bzImage -nographic
To test it out, login into the VM, and run:
i=0; while true; do echo $i; i=$(($i + 1)); sleep 1; done
Then on another shell, open the monitor:
telnet localhost 45454
savevm my_snap_id
The counting continues. Then, if we load the vm:
loadvm my_snap_id
the counting goes back to where we saved. This shows that CPU and memory states were reverted.
We can also verify that the disk state is also reversed. Guest:
echo 0 >f
Monitor:
savevm my_snap_id
Guest:
echo 1 >f
Monitor:
loadvm my_snap_id
Guest:
cat f
And the output is 0.