For background, I'm running bare-metal QEMU-4.1.0 on aarch64.
There are several ways to get QEMU to load compiled code into memory. I'd like to understand what the underlying differences are, because I see very different behavior and the documentation doesn't shed any light.
Consider this first command line:
qemu-system-aarch64 \
-s -S \
-machine virt,secure=on,virtualization=on \
-m 512M \
-smp 4 \
-display none \
-nographic \
-semihosting \
-serial mon:stdio \
-kernel my_file.elf \
-device loader,addr=0x40004000,cpu_num=0 \
-device loader,addr=0x40004000,cpu_num=1 \
-device loader,addr=0x40004000,cpu_num=2 \
-device loader,addr=0x40004000,cpu_num=3 \
;
In another shell, if I launch gdb to see what QEMU has loaded into memory, it corresponds exactly to what I expect. In fact, gdb has a built-in command for this...
(gdb) compare-sections
Section .start, range 0x40004000 -- 0x40006164: matched.
Section .vectors, range 0x40006800 -- 0x40006f90: matched.
Section .text, range 0x40006fc0 -- 0x4002ca7c: matched.
...
Section .stacks, range 0x4207c120 -- 0x420bc120: matched.
(gdb) x/10x 0x40004000
0x40004000 <_start>: 0x14000800 0x00000000 0x00000000 0x00000000
...
Perfect! Everything in my ELF is at 0x40004000, and I see it all in memory, just how I expect! My first core boots and runs as I expect.
It is interesting to note that if I dump what is at location zero in memory, then there is stuff loaded down there. I didn't ask for it. I didn't explicitly load it. I don't execute it. It's not in my ELF file. I don't know what it is or where it came from. My GUESS is that QEMU has ASSUMED that I want some BIOS in the flash and has put one there. I don't know for sure. It also places something (small) at 0x40000000. I don't know what that is either... I do want to be careful that if I load something, we won't step on each other...
My first question then becomes: Can I enable some debug messages to understand what QEMU is loading where, and perhaps even WHY?
Continuing...
If I change my command line to REPLACE the "-kernel my_file.elf" switch with "-bios my_file.elf" switch (changing nothing else), and I repeat my run/gdb, then I see two things that are different...
First, I see that all my cores are running. I don't need to use PSCI calls to start them. Okay, but I don't think that's relevant to my issue. Second (and VERY important) is that my memory does NOT contain what I expect!
(gdb) compare-sections
Section .start, range 0x40004000 -- 0x40006164: MIS-MATCHED!
Section .vectors, range 0x40006800 -- 0x40006f90: MIS-MATCHED!
Section .text, range 0x40006fc0 -- 0x4002ca7c: MIS-MATCHED!
...
Section .stacks, range 0x4207c120 -- 0x420bc120: matched.
(gdb) x/8x 0x40000000
0x40004000 <_start>: 0x00000000 0x00000000 0x00000000 0x00000000
0x40004010 <_start+16>: 0x00000000 0x00000000 0x00000000 0x00000000
(gdb) x/8x 0x40006800
0x40006800 <my_vector_name>: 0x00000000 0x00000000 0x00000000 0x00000000
0x40006810 <my_vector_name+16>: 0x00000000 0x00000000 0x00000000 0x00000000
(gdb) x/8x 0x40006fc0
0x40006800 <my_symbol_name>: 0x00000000 0x00000000 0x00000000 0x00000000
0x40006810 <my_symbol_name+16>: 0x00000000 0x00000000 0x00000000 0x00000000
Everything is zero. I don't see MY code anywhere, although the mysterious code is still getting loaded at both 0x0 and 0x4000000. As you might also expect, the cores immediately die with an "Undefined Instruction" exception as soon as I issue "nexti" in my gdb.
Hmmm...
Okay, now I'll change the "-bios my_file.elf" to "-device loader,file=my_file.elf". I get the same result. I cannot find my code in memory.
What happens differently under-the-hood in QEMU between "-bios" and "-kernel"? Where is that documented, or where in the source can I follow it? How can I best debug this?
Thank you kind Sir/Madam!
Edit:
For debugging, all the good/relevant stuff seems to be in "virt.c"...
More Edit (to add info from "-device loader=my_file.elf")
My command line is:
/tools/gnu/qemu-4.1.0/bin/qemu-system-aarch64 \
-s -S \
-machine virt,secure=on,virtualization=on \
-cpu cortex-a53 \
-d int \
-m 512M \
-smp 4 \
-display none \
-nographic \
-semihosting \
-serial mon:stdio \
-device loader,file=NEW_AT_ZERO.elf \
;
Here's some of the relevant section of NEW_AT_ZERO.dis:
NEW_AT_ZERO.elf: file format elf64-littleaarch64
NEW_AT_ZERO.elf
architecture: aarch64, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x0000000000000000
Program Header:
LOAD off 0x0000000000010000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**16
filesz 0x00000000020b8120 memsz 0x00000000020b8120 flags rwx
NOTE off 0x0000000000043484 vaddr 0x0000000000033484 paddr 0x0000000000033484 align 2**2
filesz 0x0000000000000024 memsz 0x0000000000000024 flags r--
private flags = 0:
Sections:
Idx Name Size VMA LMA File off Algn
0 .start 00002164 0000000000000000 0000000000000000 00010000 2**2
CONTENTS, ALLOC, LOAD, READONLY, CODE
1 .vectors 00000790 0000000000002800 0000000000002800 00012800 2**11
CONTENTS, ALLOC, LOAD, READONLY, CODE
2 .text 00025bbc 0000000000002fc0 0000000000002fc0 00012fc0 2**6
CONTENTS, ALLOC, LOAD, READONLY, CODE
3 .bss 0000a904 0000000000028b80 0000000000028b80 00038b7c 2**3
ALLOC
....
Contents of section .start:
0000 00080014 00000000 00000000 00000000 ................
....
1000 fd170094 a00038d5 01044092 020c7892 ......8...#...x.
1010 261842aa 660000b5 00038052 00005ed4 &.B.f......R..^.
1020 7f2003d5 ffffff17 00000000 00000000 . ..............
....
...but of course...
GNU gdb (Linaro_GDB-2017.05.09) 7.12.1.20170417-git
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=aarch64-none-elf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Remote debugging using localhost:1234
warning: No executable has been specified and target does not support
determining executable automatically. Try using the "file" command.
0x0000000000000000 in ?? ()
Reading symbols from ./NEW_AT_ZERO.elf...done.
(gdb) compare_sections
Undefined command: "compare_sections". Try "help".
(gdb) compare-sections
Section .start, range 0x0 -- 0x2164: MIS-MATCHED!
Section .vectors, range 0x2800 -- 0x2f90: MIS-MATCHED!
Section .text, range 0x2fc0 -- 0x28b7c: MIS-MATCHED!
....
Section .stacks, range 0x2078120 -- 0x20b8120: matched.
warning: One or more sections of the target image does not match
the loaded file
(gdb) x/4x 0
0x0 <_start>: 0x00000000 0x00000000 0x00000000 0x00000000
(gdb) x/12x 0x1000
0x1000 <symbol>: 0x00000000 0x00000000 0x00000000 0x00000000
0x1010 <symbol+16>: 0x00000000 0x00000000 0x00000000 0x00000000
0x1020 <end_symbol>: 0x00000000 0x00000000 0x00000000 0x00000000
QEMU's command line options for loading code into the guest are various, and often have different semantics between architectures or even between machine types for the same architecture. This is unfortunate but is the result of backwards-compatibility with older QEMU versions and a gradual accumulation of "it would be nice to Do The Right Thing for this image file type" special cases.
Broad summary:
-kernel is the "load a Linux kernel" option. It will load and boot the kernel in whatever way seems best for the architecture being used. For instance, for the x86 PC machine it will just provide the file to the guest BIOS and rely on the guest BIOS to do the actual loading of the file into RAM. On Arm, loading a Linux kernel means that we follow the rules the kernel lays down for how to boot it (https://www.kernel.org/doc/Documentation/arm64/booting.txt for 64-bit or https://www.kernel.org/doc/Documentation/arm/Booting for 32-bit), and we achieve that with a little bit of stub bootloader code (this is what you are seeing in low memory). The kernel boot rules also require that we provide it with a device tree blob in RAM, and this is the data at 0x40000000. We also, in accordance with Linux kernel boot expectations, handle secondary CPUs by either keeping them in PSCI powered-off state or via a little bit of secondary-CPU bootloader code which uses a WFI loop so that the primary can wake them up. (Which we do depends on the board model being used, because we do what the real board does, which especially for 32-bit boards varies a lot.)
As an oddball exception, for Arm if you pass an ELF file to -kernel, we'll assume that it is not a Linux kernel, and will boot it by just starting at the ELF entrypoint. We provide the DTB blob at the base of RAM, but only if it wouldn't overlap with the loaded ELF file. (Aside: for 'virt' in particular you want the DTB anyway, because we don't guarantee to keep devices in the same physical addresses between QEMU versions -- the DTB is how we tell guest code where it should look for things. You can rely on flash at 0x0 and RAM starting at 0x4000_0000, but really should pull all other device addresses from the DTB. In practice we have made efforts to avoid rearranging the board memory map, but reading the DTB is the right thing for guest code to do.)
-device loader is the "generic loader", which behaves the same on any architecture. It just loads an ELF image into guest RAM, and doesn't do anything to change the CPU reset behaviour. This is a good choice if you have a completely bare-metal image which includes the exception vector table and want to have it start in the same way the hardware would out of reset.
-bios is the "load a bios image, in whatever way seems good for this machine model" option. Again, this is a "do what I mean" kind of option whose specifics vary from machine model to machine model and from architecture to architecture; some machines don't support it at all. Some machines (eg x86 PC) will always load a bios, using a default binary if the user didn't specify. Some will load a bios if the user asks, but not otherwise (the arm virt board is like this). Generally a bios image is expected to be a "bare metal raw binary" image which will get loaded into some flash or ROM memory which corresponds to wherever the hardware starts execution when it comes out of reset. On at least some machines, including 'virt', you can instead provide the contents of the flash/ROM devices using a command line like "-drive if=pflash,...". This is an example of a common pattern in QEMU where you can either use a short "do what I mean" option that is convenient but has a lot of magic under the hood, or a longer "orthogonal" option which lets you specify lots of sub-options and get exactly the behaviour you want. Note that BIOS images should not be ELF files, they're expected to just be the raw data to put into the ROMs.
A lot of this is undocumented, because "I want to run a bare metal program of my own devising" is a very niche use case and because we don't have a good place in our documentation to make it easy to document the specifics of different board models.
I tried:
git clone git://git.buildroot.net/buildroot
cd buildroot
git checkout 2019.08
make qemu_aarch64_virt_defconfig
make menuconfig
In menuconfig, I set:
Bootloaders
U-Boot configuration (Using an in-tree board defconfig file)
qemu_arm64
Kernel
Install kernel image to /boot in target
and finally:
make BR2_JLEVEL="$nproc"
Now, I can boot fine without U-Boot with the command line mentioned at: How to download the Torvalds Linux Kernel master, (re)compile it, and boot it with QEMU?
./output/host/usr/bin/qemu-system-aarch64 -M virt -cpu cortex-a57 -nographic -smp 1 -kernel output/images/Image -append "root=/dev/vda console=ttyAMA0" -netdev user,id=eth0 -device virtio-net-device,netdev=eth0 -drive file=output/images/rootfs.ext4,if=none,format=raw,id=hd0 -device virtio-blk-device,drive=hd0
but that is not using U-Boot.
When I do:
ls -l output/images/
it contains:
-rw-r--r-- 1 ciro ciro 6.5M 2019-09-20_13:36:23 Image
-rw-r--r-- 1 ciro ciro 60M 2019-09-20_13:39:02 rootfs.ext2
lrwxrwxrwx 1 ciro ciro 11 2019-09-20_13:36:25 rootfs.ext4 -> rootfs.ext2
-rw-r--r-- 1 ciro ciro 583K 2019-09-20_13:34:15 u-boot.bin
so there is a U-Boot binary there: u-boot.bin, but how do I use that with QEMU?
I tried as mentioned at: Can ARM qemu system emulator boot from card image without kernel param? to remove -kernel and -append and add -bios u-boot.bin:
./output/host/usr/bin/qemu-system-aarch64 -M virt -cpu cortex-a57 -nographic -smp 1 -bios output/images/u-boot.bin -netdev user,id=eth0 -device virtio-net-device,netdev=eth0 -drive file=output/images/rootfs.ext4,if=none,format=raw,id=hd0 -device virtio-blk-device,drive=hd0
Now I do get the U-Boot shell, but boot fails and leaves me no the U-Boot prompt:
U-Boot 2019.07 (Sep 20 2019 - 13:34:10 +0100)
DRAM: 128 MiB
Flash: 128 MiB
*** Warning - bad CRC, using default environment
In: pl011#9000000
Out: pl011#9000000
Err: pl011#9000000
Net:
Warning: virtio-net#31 using MAC address from ROM
eth0: virtio-net#31
Hit any key to stop autoboot: 0
starting USB...
No working controllers found
USB is stopped. Please issue 'usb start' first.
scanning bus for devices...
Device 0: unknown device
Device 0: QEMU VirtIO Block Device
Type: Hard Disk
Capacity: 60.0 MB = 0.0 GB (122880 x 512)
... is now current device
** No partition table - virtio 0 **
starting USB...
No working controllers found
BOOTP broadcast 1
DHCP client bound to address 10.0.2.15 (2 ms)
Using virtio-net#31 device
TFTP from server 10.0.2.2; our IP address is 10.0.2.15
Filename 'boot.scr.uimg'.
Load address: 0x40200000
Loading: *
TFTP error: 'Access violation' (2)
Not retrying...
BOOTP broadcast 1
DHCP client bound to address 10.0.2.15 (0 ms)
Using virtio-net#31 device
TFTP from server 10.0.2.2; our IP address is 10.0.2.15
Filename 'boot.scr.uimg'.
Load address: 0x40400000
Loading: *
TFTP error: 'Access violation' (2)
Not retrying...
=>
so it appears that U-Boot cannot handle the VirtIO device? Or according to Peter, I have to create a partition table. I couldn't find that automatically in Buildroot, but I could do it manually, here is one approach: https://unix.stackexchange.com/questions/209566/how-to-format-a-partition-inside-of-an-img-file/527132#527132
Another approach would be to keep -kernel -append and let QEMU put the kernel into memory as done without U-Boot, and then use the booti U-Boot command I've found on help:
booti - boot Linux kernel 'Image' format from memory
so I just need to find out its address. But that is kind of cheating since I want U-boot to do the hard work rather than cheat with QEMU.
My goal is to reach a good setup to develop U-Boot and QEMU's early boot stuff.
Given that u-boot correctly detects the virtio block device, I think it is unlikely that it cannot handle it. The error printed is "** No partition table - virtio 0 **", which is correct, because you've set up the block device to contain just rootfs.ext4, which will be a filesystem image. That suggests that you'll have more luck if you create a disk image with a partition table and write the rootfs to a partition within the disk image.
I followed Peter's advice and put it into a partitioned image with the sfdisk-fs-to-img command from https://unix.stackexchange.com/questions/209566/how-to-format-a-partition-inside-of-an-img-file/527132#527132
I am now able to read the root filesystem with:
ls virtio 0 /boot
and that contains Image file.
Now I think there are only some U-Boot specifics to resolve, which I'm not very familiar with:
load /boot/Image into memory with something like load virtio 0 0x100000 /boot/Image. TODO which address is valid? This arbitrary choice gave ** Reading file would overwrite reserved memory **
find out how to load the DTB and kernel CLI arguments. The DTB would need to be auto-generated with QEMU with qemu-system-aarch64 -machine dumpdtb=dtb.dtb
boot it with something like: booti 0x100000
I was hoping Buildroot would have automated things a bit more for me sadface.
I have an embedded Linux system that uses ramdisk boot so it has run time no persistent storage available (it does have Flash to store kernel and ramdisk).
The only connectivity is RS-232 serial login console. So I am limited by what is provided by its built in busybox. I want to retrieve the ramdisk, modify it, and rewrite the ramdisk. The kernel does not have Flash filesystem support built-in. The ramdisk partition size is about 10 MBytes. When all files in the user directory are deleted, the free ramdisk size is about 14 MBytes.
The command dd is available so I can copy the ramdisk partition to the ramdisk, and can write to the flash from a ramdisk file. flashcp is also available.
So my problem is now how to receive and send binary files through the RS-232 serial console?
I research the followings and none is useful for me:
Linux command to send binary file to serial port with HW flow control? on stackoverflow
Binary data over serial terminal on stackoverflow
Transferring files using serial console on k.japko.eu
File transfer over a serial line on superuser.com
How to get file to a host when all you have is a serial console? on stackexchange
Mostly because x/y/zmodem are not available in the busybox.
Any idea? Thanks!
Per the request, here's what I should have included in the first place.
Available u-boot commands:
U-Boot >?
? - alias for 'help'
askenv - get environment variables from stdin
base - print or set address offset
bdinfo - print Board Info structure
boot - boot default, i.e., run 'bootcmd'
bootd - boot default, i.e., run 'bootcmd'
bootm - boot application image from memory
cmp - memory compare
coninfo - print console devices and information
cp - memory copy
crc32 - checksum calculation
crc32_chk_uimage- checksum calculation of an image for u-boot
echo - echo args to console
editenv - edit environment variable
env - environment handling commands
exit - exit script
false - do nothing, unsuccessfully
fatinfo - print information about filesystem
fatload - load binary file from a dos filesystem
fatls - list files in a directory (default /)
fatwrite- write file into a dos filesystem
go - start application at address 'addr'
gpio - input/set/clear/toggle gpio pins
help - print command description/usage
i2c - I2C sub-system
iminfo - print header information for application image
imxtract- extract a part of a multi-image
itest - return true/false on integer compare
loadb - load binary file over serial line (kermit mode)
loads - load S-Record file over serial line
loady - load binary file over serial line (ymodem mode)
loop - infinite loop on address range
md - memory display
mdc - memory display cyclic
mm - memory modify (auto-incrementing address)
mw - memory write (fill)
mwc - memory write cyclic
nm - memory modify (constant address)
printenv- print environment variables
reset - Perform RESET of the CPU
run - run commands in an environment variable
saveenv - save environment variables to persistent storage
saves - save S-Record file over serial line
setenv - set environment variables
sf - SPI flash sub-system
showvar - print local hushshell variables
sleep - delay execution for some time
source - run script from memory
sspi - SPI utility command
test - minimal test like /bin/sh
true - do nothing, successfully
usb - USB sub-system
usbboot - boot from USB device
version - print monitor, compiler and linker version
U-Boot >
Available busybox commands:
BusyBox v1.13.2 (2015-03-16 10:50:56 EDT) multi-call binary
Copyright (C) 1998-2008 Erik Andersen, Rob Landley, Denys Vlasenko
and others. Licensed under GPLv2.
See source distribution for full notice.
Usage: busybox [function] [arguments]...
or: function [arguments]...
BusyBox is a multi-call binary that combines many common Unix
utilities into a single executable. Most people will create a
link to busybox for each function they wish to use and BusyBox
will act like whatever it was invoked as!
Currently defined functions:
[, [[, addgroup, adduser, ar, ash, awk, basename, blkid,
bunzip2, bzcat, cat, chattr, chgrp, chmod, chown, chpasswd,
chroot, chvt, clear, cmp, cp, cpio, cryptpw, cut, date,
dc, dd, deallocvt, delgroup, deluser, df, dhcprelay, diff,
dirname, dmesg, du, dumpkmap, dumpleases, echo, egrep, env,
expr, false, fbset, fbsplash, fdisk, fgrep, find, free,
freeramdisk, fsck, fsck.minix, fuser, getopt, getty, grep,
gunzip, gzip, halt, head, hexdump, hostname, httpd, hwclock,
id, ifconfig, ifdown, ifup, inetd, init, insmod, ip, kill,
killall, klogd, last, less, linuxrc, ln, loadfont, loadkmap,
logger, login, logname, logread, losetup, ls, lsmod, makedevs,
md5sum, mdev, microcom, mkdir, mkfifo, mkfs.minix, mknod,
mkswap, mktemp, modprobe, more, mount, mv, nc, netstat,
nice, nohup, nslookup, od, openvt, passwd, patch, pidof,
ping, ping6, pivot_root, poweroff, printf, ps, pwd, rdate,
rdev, readahead, readlink, readprofile, realpath, reboot,
renice, reset, rm, rmdir, rmmod, route, rtcwake, run-parts,
sed, seq, setconsole, setfont, sh, showkey, sleep, sort,
start-stop-daemon, strings, stty, su, sulogin, swapoff,
swapon, switch_root, sync, sysctl, syslogd, tail, tar, tcpsvd,
tee, telnet, telnetd, test, tftp, tftpd, time, top, touch,
tr, traceroute, true, tty, udhcpc, udhcpd, udpsvd, umount,
uname, uniq, unzip, uptime, usleep, vconfig, vi, vlock,
watch, wc, wget, which, who, whoami, xargs, yes, zcat
In uboot you could use loady/loadx to get file from pc via uart.I usually use teraterm to send file.
The process should be this:
run loady in uboot
use teraterm send data
the file is transfer to you device's memory located in 0x01000000.
Independently I found a way to upload binary files through the Linux console and I'll document the steps here in case others find it useful since I had a hard time looking for this information on the net.
Here's the theory: change the console mode to raw so all the binary traffic are't interpretted as console command, e.g. ctrl-C. Turn off echo so it doesn't add extra serial traffic. Run tar to accept input from the stdin. Since ctrl-C won't work, and tar won't know when to terminate, use a background task to kill the login shell so you can login again to do your staff.
Steps:
Create a script to run in the background. Change myvar variable so it kills the login shell after the transfer is complete. Currently 120 corresponds to 1200 seconds, sufficient for a 10 MBytes file. In addition edit the 808 to match your login shell PID:
create bg file:
myvar=120
while [ $myvar -gt 0 ]
do
myvar=$(( $myvar-1 ))
echo -e " $myvar \n"
ls -l
sleep 10
done
kill -9 808
Launch the script in the background:
in console type:
source ./bg &
Use stty to change console to raw mode and do not echo
in console type:
stty raw -echo
Start tar to untar stdin. Note: I have to use ctrl-J since no longer work after the stty command
in console type and ends with ctrl-j, not :
tar zx -f - 1> 1.log 2> 2.log
Start Teraterm to send binary file
Wait for completion and the new login prompt
I forgot I asked this question. I figured out how to make ssh connection which in turn allows many more things to be done more easily. Of course it requires sshd in addition to nc and stty so you are out of luck if these are not available on your embedded Linux. I have tried it several times and it seems to work well, allowing multiple ssh sessions to be established, and mc to transfer files.
You will need two shell sessions on the host computer, one to loop the serial port to socket, and the other for the ssh, and more if you want to establish more ssh sessions.
First you need to setup the serial port. The '--noreset' option for picocom does this:
sudo picocom --noreset -b 115200 -e b /dev/ttyUSB3
Quit picocom once this is done (^B^X to exit).
Next we need to verify that the line endings are not translated or else ssh won't work. In the first shell run:
cat /dev/ttyUSB3 | hexdump -C
In the second shell run:
echo "echo -e \"LFLF\\n\\nCRCR\\r\\rEND\"" > /dev/ttyUSB3
You may see that \n (0x0A) is translated to \r\n (0x0D0x0A)
Use stty to set raw mode without echo and you should see no more translation:
echo "stty raw -echo" > /dev/ttyUSB3
echo "echo -e \"LFLF\\n\\nCRCR\\r\\rEND\"" > /dev/ttyUSB3
Finally in the first shell run nc to funnel local traffic between the serial port and ssh socket:
cat /dev/ttyUSB3 | nc -l -p 2222 > /dev/ttyUSB3
and funnel remote serial traffic to sshd:
echo "while true ; do nc localhost 22 ; done" > /dev/ttyUSB3
and connect ssh with port forwarding:
ssh -vvv root#localhost -p 2222 -L 0.0.0.0:22022:localhost:22
you can make more ssh connections simultaneously:
ssh -vvv root#localhost -p 22022
if you use mc, you can connect to it so you can easily browse the remote file system and copy files:
sh://root#localhost:22022
Last words: nc strips the TCP headers so the ssh packets are no checksumed and are not retried. If there were data error, the connection will break. If you remember your login shell PID, you can kill it and login again, otherwise you have to reboot. The '-vvv' flag for the ssh is for debugging.
I am using qemu emulator for aarch64 and want to create an outside checkpoint (or fast forwarding) to save all I need to restart the system just from the point when I create checkpoint. (In fact, I want to skip the booting step) I only found something on qemu VM snapshot and fast forwarding, but it does not work for the emulator. Is there any checkpoint function for qemu emulator?
A savevm snapshot should do what you want. The short answer is that you need to set up a QCOW2 disk for the snapshots to be saved to, and then in the monitor you can use the 'savevm' command to take the snapshot. Then the command line '-loadvm' option will let you resume from there. This all works fine in emulation of AArch64.
https://translatedcode.wordpress.com/2015/07/06/tricks-for-debugging-qemu-savevm-snapshots/ has a more detailed tutorial.
Minimal example
Peter's answer just worked for me, but let me provide a fully reproducible example.
I have fully automated everything at: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/1e0f0b492855219351b0bfa2eec4d3a6811fcaaa#snapshot
The key step is to convert the image to qcow2 as explained at: https://docs.openstack.org/image-guide/convert-images.html
cd buildroot/output.x86_64~/images
qemu-img convert -f raw -O qcow2 rootfs.ext2 rootfs.ext2.qcow2
And the final QEMU command used was:
./buildroot/output.x86_64~/host/usr/bin/qemu-system-x86_64 -m 128M -monitor telnet::45454,server,nowait -netdev user,hostfwd=tcp::45455-:45455,id=net0 -smp 1 -M pc -append 'root=/dev/vda nopat nokaslr norandmaps printk.devkmsg=on printk.time=y console=ttyS0' -device edu -device virtio-net-pci,netdev=net0 -drive file=./buildroot/output.x86_64~/images/rootfs.ext2.qcow2,if=virtio,format=qcow2 -kernel ./buildroot/output.x86_64~/images/bzImage -nographic
To test it out, login into the VM, and run:
i=0; while true; do echo $i; i=$(($i + 1)); sleep 1; done
Then on another shell, open the monitor:
telnet localhost 45454
savevm my_snap_id
The counting continues. Then, if we load the vm:
loadvm my_snap_id
the counting goes back to where we saved. This shows that CPU and memory states were reverted.
We can also verify that the disk state is also reversed. Guest:
echo 0 >f
Monitor:
savevm my_snap_id
Guest:
echo 1 >f
Monitor:
loadvm my_snap_id
Guest:
cat f
And the output is 0.