can't run '/etc/init.d/rcS': No such file or directory - qemu

I am trying to emulate a firmware image using qemu. During booting, I get the following error
can't run '/etc/init.d/rcS': No such file or directory
can't open /dev/ttyS0: No such file or directory
can't open /dev/ttyS0: No such file or directory
can't open /dev/ttyS0: No such file or directory
.
.
.
This is the content of the inittab file
# Startup the system
null::sysinit:/etc/init.d/rc.sysinit
# now run any rc scripts
::sysinit:/etc/init.d/rcS
# Put a getty on the serial port
ttyS0::respawn:/sbin/getty -L ttyS0 115200 vt100
# Stuff to do before rebooting
null::shutdown:/bin/umount -a -r
It is able to run the rc.sysinit, but not the rcS.
I have checked permissions of the rcS. Also, the filesystem is mounted as read-only cramfs. Could this be causing an issue?
This is the command I am running:
QEMU_AUDIO_DRV=none \qemu-system-arm -m 256M -M versatilepb
-kernel ~/linux-2.6.23/arch/arm/boot/zImage
-append "console=ttyAMA0,115200 root=/dev/ram rdinit=/sbin/init"
-initrd ~/tmpcramfs2
-nographic
These are the boot messages obtained on running the command:
Linux version 2.6.23 (hsailer#SvanteArrhenius) (gcc version 4.0.2) #1 Thu May 27 09:31:10 EDT 2021
CPU: ARM926EJ-S [41069265] revision 5 (ARMv5TEJ), cr=00093177
Machine: ARM-Versatile PB
Memory policy: ECC disabled, Data cache writeback
CPU0: D VIVT write-through cache
CPU0: I cache: 4096 bytes, associativity 4, 32 byte lines, 32 sets
CPU0: D cache: 65536 bytes, associativity 4, 32 byte lines, 512 sets
Built 1 zonelists in Zone order. Total pages: 65024
Kernel command line: console=ttyAMA0,115200 root=/dev/ram rdinit=/sbin/init
PID hash table entries: 1024 (order: 10, 4096 bytes)
Console: colour dummy device 80x30
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 256MB = 256MB total
Memory: 249600KB available (2508K code, 227K data, 100K init)
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
NET: Registered protocol family 16
NET: Registered protocol family 2
Time: timer3 clocksource has been installed.
IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
TCP established hash table entries: 8192 (order: 4, 65536 bytes)
TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
TCP reno registered
checking if image is initramfs...it isn't (bad gzip magic numbers); looks like an initrd
Freeing initrd memory: 7184K
NetWinder Floating Point Emulator V0.97 (double precision)
Installing knfsd (copyright (C) 1996 okir#monad.swb.de).
JFFS2 version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
JFS: nTxBlock = 2007, nTxLock = 16063
io scheduler noop registered
io scheduler anticipatory registered (default)
io scheduler deadline registered
io scheduler cfq registered
CLCD: Versatile hardware, VGA display
Clock CLCDCLK: setting VCO reg params: S=1 R=99 V=98
Console: switching to colour frame buffer device 80x60
Serial: AMBA PL011 UART driver
dev:f1: ttyAMA0 at MMIO 0x101f1000 (irq = 12) is a AMBA/PL011
console [ttyAMA0] enabled
dev:f2: ttyAMA1 at MMIO 0x101f2000 (irq = 13) is a AMBA/PL011
dev:f3: ttyAMA2 at MMIO 0x101f3000 (irq = 14) is a AMBA/PL011
fpga:09: ttyAMA3 at MMIO 0x10009000 (irq = 38) is a AMBA/PL011
RAMDISK driver initialized: 16 RAM disks of 8192K size 1024 blocksize
smc91x.c: v1.1, sep 22 2004 by Nicolas Pitre <nico#cam.org>
eth0: SMC91C11xFD (rev 1) at d098e000 IRQ 25 [nowait]
eth0: Ethernet addr: 52:54:00:12:34:56
armflash.0: Found 1 x32 devices at 0x0 in 32-bit bank
Intel/Sharp Extended Query Table at 0x0031
Using buffer write method
RedBoot partition parsing not available
afs partition parsing not available
armflash: probe of armflash.0 failed with error -22
mice: PS/2 mouse device common for all mice
input: AT Raw Set 2 keyboard as /class/input/input0
TCP cubic registered
NET: Registered protocol family 1
NET: Registered protocol family 17
VFP support v0.3: implementor 41 architecture 1 part 10 variant 9 rev 0
input: ImExPS/2 Generic Explorer Mouse as /class/input/input1
RAMDISK: cramfs filesystem found at block 0
RAMDISK: Loading 7184KiB [1 disk] into ram disk... done.
VFS: Mounted root (cramfs filesystem) readonly.
Freeing init memory: 100K
can't run '/etc/init.d/rcS': No such file or directory
can't open /dev/ttyS0: No such file or directory
can't open /dev/ttyS0: No such file or directory
can't open /dev/ttyS0: No such file or directory
.
.
.

The errors about /dev/ttyS0 are because your inittab is specifying the wrong device name for the serial port for the (emulated) hardware you're running on. Your QEMU command specifies the 'versatilepb' board, whose serial devices are PL011s, which appear in /dev/ as /dev/ttyAMA0, /dev/ttyAMA1, etc. (/dev/ttyS0 is what the serial ports on an x86 PC appear as.) You need to fix that line of the inittab to refer to ttyAMA0 instead.
For the rcS error, I would suggest you start by double-checking all the things listed in all the responses to this older question.

Related

How to mass flash STM32 Microcontrollers through JTAG Chain with OpenOCD?

I am a beginner to OpenOCD and I am trying to flash 3 STM32 targets in a daisy chain with an ST-Link v2 debugger or OLIMEX as shown below using OpenOCD.
The code that I use works if only one Target is connected. But if I connect it to more than one Target, OpenOCD throws an error stating that init failed.
"C:\Program Files\GNU ARM Eclipse\OpenOCD\0.10.0-201601101000-dev\bin\openocd" -f "C:\Program Files\GNU ARM Eclipse\OpenOCD\0.10.0-201601101000-dev\scripts\interface\stlink-v2.cfg" -f "C:\Program Files\GNU ARM Eclipse\OpenOCD\0.10.0-201601101000-dev\scripts\target\stm32f3x.cfg" -c init -c targets -c "halt" -c "flash erase_sector 0 0 127" -c "reset halt" -c "flash write_image C:/Users/Buero-1/Desktop/openOCD/init.hex" -c "verify_image C:/Users/Buero-1/Desktop/openOCD/init.hex" -c "reset run" -c shutdown
A successful result I get when this code is executed is shown below.
GNU ARM Eclipse 64-bits Open On-Chip Debugger 0.10.0-dev-00287-g85cec24-dirty (2016-01-10-10:13)
Licensed under GNU GPL v2
For bug reports, read
http://openocd.org/doc/doxygen/bugs.html
Info : auto-selecting first available session transport "hla_swd". To override use 'transport select <transport>'.
adapter speed: 1000 kHz
adapter_nsrst_delay: 100
Info : The selected transport took over low-level target control. The results might differ compared to plain JTAG/SWD
none separate
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : clock speed 950 kHz
Info : STLINK v2 JTAG v29 API v2 SWIM v7 VID 0x0483 PID 0x3748
Info : using stlink api v2
Info : Target voltage: 3.223311
Info : stm32f3x.cpu: hardware has 6 breakpoints, 4 watchpoints
TargetName Type Endian TapName State
-- ------------------ ---------- ------ ------------------ ------------
0* stm32f3x.cpu hla_target little stm32f3x.cpu halted
Info : device id = 0x20006432
Info : flash size = 256kbytes
erased sectors 0 through 127 on flash bank 0 in 0.025984s
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : Unable to match requested speed 1000 kHz, using 950 kHz
adapter speed: 950 kHz
stm32f3x.cpu: target state: halted
target halted due to debug-request, current mode: Thread
xPSR: 0x01000000 pc: 0xfffffffe msp: 0xfffffffc
Info : Padding image section 0 with 31880 bytes
Info : Padding image section 1 with 1 bytes
stm32f3x.cpu: target state: halted
target halted due to breakpoint, current mode: Thread
xPSR: 0x61000000 pc: 0x2000003a msp: 0xfffffffc
wrote 47676 bytes from file Z:/Elektronik/GSV13/Fertigung_GSV-13iu/Init/GSV13init_Ver1_6.hex in 1.635796s (28.462 KiB/s)
stm32f3x.cpu: target state: halted
target halted due to breakpoint, current mode: Thread
xPSR: 0x61000000 pc: 0x2000002e msp: 0xfffffffc
stm32f3x.cpu: target state: halted
target halted due to breakpoint, current mode: Thread
xPSR: 0x61000000 pc: 0x2000002e msp: 0xfffffffc
verified 15795 bytes in 0.483325s (31.914 KiB/s)
Info : Unable to match requested speed 1000 kHz, using 950 kHz
Info : Unable to match requested speed 1000 kHz, using 950 kHz
adapter speed: 950 kHz
shutdown command invoked
But as mentioned, if I connect multiple targets in a JTAG chain, the process stops at init and the program ends.
The config files are target/stm32f3x.cfg, interface/ftdi/olimex-arm-usb-ocd-h.cfg and interface/stlink-v2.cfg.
Please excuse me if my question was very basic. It would be of great help if I am provided an update on how to proceed with my problem.
Thank you.
As far as I know STLink v2 does not support daisy chain.
https://community.st.com/s/question/0D50X00009XkZTdSAN/does-stlink-utility-support-multiple-devices-on-jtag-chain

How to build and boot Linux aarch64 with U-Boot with Buildroot on QEMU?

I tried:
git clone git://git.buildroot.net/buildroot
cd buildroot
git checkout 2019.08
make qemu_aarch64_virt_defconfig
make menuconfig
In menuconfig, I set:
Bootloaders
U-Boot configuration (Using an in-tree board defconfig file)
qemu_arm64
Kernel
Install kernel image to /boot in target
and finally:
make BR2_JLEVEL="$nproc"
Now, I can boot fine without U-Boot with the command line mentioned at: How to download the Torvalds Linux Kernel master, (re)compile it, and boot it with QEMU?
./output/host/usr/bin/qemu-system-aarch64 -M virt -cpu cortex-a57 -nographic -smp 1 -kernel output/images/Image -append "root=/dev/vda console=ttyAMA0" -netdev user,id=eth0 -device virtio-net-device,netdev=eth0 -drive file=output/images/rootfs.ext4,if=none,format=raw,id=hd0 -device virtio-blk-device,drive=hd0
but that is not using U-Boot.
When I do:
ls -l output/images/
it contains:
-rw-r--r-- 1 ciro ciro 6.5M 2019-09-20_13:36:23 Image
-rw-r--r-- 1 ciro ciro 60M 2019-09-20_13:39:02 rootfs.ext2
lrwxrwxrwx 1 ciro ciro 11 2019-09-20_13:36:25 rootfs.ext4 -> rootfs.ext2
-rw-r--r-- 1 ciro ciro 583K 2019-09-20_13:34:15 u-boot.bin
so there is a U-Boot binary there: u-boot.bin, but how do I use that with QEMU?
I tried as mentioned at: Can ARM qemu system emulator boot from card image without kernel param? to remove -kernel and -append and add -bios u-boot.bin:
./output/host/usr/bin/qemu-system-aarch64 -M virt -cpu cortex-a57 -nographic -smp 1 -bios output/images/u-boot.bin -netdev user,id=eth0 -device virtio-net-device,netdev=eth0 -drive file=output/images/rootfs.ext4,if=none,format=raw,id=hd0 -device virtio-blk-device,drive=hd0
Now I do get the U-Boot shell, but boot fails and leaves me no the U-Boot prompt:
U-Boot 2019.07 (Sep 20 2019 - 13:34:10 +0100)
DRAM: 128 MiB
Flash: 128 MiB
*** Warning - bad CRC, using default environment
In: pl011#9000000
Out: pl011#9000000
Err: pl011#9000000
Net:
Warning: virtio-net#31 using MAC address from ROM
eth0: virtio-net#31
Hit any key to stop autoboot: 0
starting USB...
No working controllers found
USB is stopped. Please issue 'usb start' first.
scanning bus for devices...
Device 0: unknown device
Device 0: QEMU VirtIO Block Device
Type: Hard Disk
Capacity: 60.0 MB = 0.0 GB (122880 x 512)
... is now current device
** No partition table - virtio 0 **
starting USB...
No working controllers found
BOOTP broadcast 1
DHCP client bound to address 10.0.2.15 (2 ms)
Using virtio-net#31 device
TFTP from server 10.0.2.2; our IP address is 10.0.2.15
Filename 'boot.scr.uimg'.
Load address: 0x40200000
Loading: *
TFTP error: 'Access violation' (2)
Not retrying...
BOOTP broadcast 1
DHCP client bound to address 10.0.2.15 (0 ms)
Using virtio-net#31 device
TFTP from server 10.0.2.2; our IP address is 10.0.2.15
Filename 'boot.scr.uimg'.
Load address: 0x40400000
Loading: *
TFTP error: 'Access violation' (2)
Not retrying...
=>
so it appears that U-Boot cannot handle the VirtIO device? Or according to Peter, I have to create a partition table. I couldn't find that automatically in Buildroot, but I could do it manually, here is one approach: https://unix.stackexchange.com/questions/209566/how-to-format-a-partition-inside-of-an-img-file/527132#527132
Another approach would be to keep -kernel -append and let QEMU put the kernel into memory as done without U-Boot, and then use the booti U-Boot command I've found on help:
booti - boot Linux kernel 'Image' format from memory
so I just need to find out its address. But that is kind of cheating since I want U-boot to do the hard work rather than cheat with QEMU.
My goal is to reach a good setup to develop U-Boot and QEMU's early boot stuff.
Given that u-boot correctly detects the virtio block device, I think it is unlikely that it cannot handle it. The error printed is "** No partition table - virtio 0 **", which is correct, because you've set up the block device to contain just rootfs.ext4, which will be a filesystem image. That suggests that you'll have more luck if you create a disk image with a partition table and write the rootfs to a partition within the disk image.
I followed Peter's advice and put it into a partitioned image with the sfdisk-fs-to-img command from https://unix.stackexchange.com/questions/209566/how-to-format-a-partition-inside-of-an-img-file/527132#527132
I am now able to read the root filesystem with:
ls virtio 0 /boot
and that contains Image file.
Now I think there are only some U-Boot specifics to resolve, which I'm not very familiar with:
load /boot/Image into memory with something like load virtio 0 0x100000 /boot/Image. TODO which address is valid? This arbitrary choice gave ** Reading file would overwrite reserved memory **
find out how to load the DTB and kernel CLI arguments. The DTB would need to be auto-generated with QEMU with qemu-system-aarch64 -machine dumpdtb=dtb.dtb
boot it with something like: booti 0x100000
I was hoping Buildroot would have automated things a bit more for me sadface.

502 Proxy Error from OpenShift DIY project

On my Openshift account I have setup Tomcat 8 and JDK 8 on a DIY application with the MySql and PHPAdmin cartridges installed.
My war file points to everything correctly and there are no errors on startup in any of the logs. However, when I try to go to my OpenShift URL I receive this 502 Proxy Error in the browser. I'm using Chrome.
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /.
What could be causing this problem?
#Graham Where's the fun in that? So I'm going to share my experience, in case anybody else gets here. I think in my instance I was hitting the upper limit of authorized CPU / memory usage for my 'free' gear. Nothing really jumps out and yells "You hit the limit" but It was pretty clear something was wrong. I'm pretty happy with the results, glad I stuck it out. I've learned a whole lot about deployment to an online server with meager $$ resources.
General troubleshooting instructions start here.
First, I shut down the server hard with a $rhc app-force-stop <app_name> After that I was able to start up the system again and it would work fine. In my case I was trying to do too much with the size of server I was paying for (free!) The free server includes 512Mb Ram and 1 Gig storage. I was trying to run Node, a MongoDB and a Cron cartridge in there. Additionally I had a whole lot of asynchronous Input/Output with quite a large stack built up. In hind sight, not clever.
Error detection wasn't real easy. I didn't learn anything at all from the log files. Generally when something went wrong they just stopped recording anything at all.
There are 11 tests to do. First login to the server via SSH, and your command line tool. Note, there is no magic "you screwed up here message" You've got to look at your usage, and compare it to your authorized usage levels. So yeah, this took me awhile, but I documented this for my own notes. Here's a good place to share with others. I've learned a whole lot with this exercise. Good luck. (oh and in my case, I deleted the cron cartridge and the mongodb cartridge. I'm hosting the DB at mlab.com where its accessible from my other projects. Success for me .)
1) Memory Fail Counts: (results should be zero...)
oo-cgroup-read memory.failcnt // my results --> 160031
oo-cgroup-read memory.memsw.failcnt // my resluts --> 8572
2) Check disk Quotas
[xyz-abc.rhcloud.com 5xxx3]\> quota -s
Disk quotas for user 5xxx3 (uid 3488):
Filesystem blocks quota limit grace files quota limit grace
/dev/mapper/EBSStore01-user_home01
608M 0 1024M 12664 0 80000
3) Check for your actual disk usage. (du = Disk Usage
Sum of directories (-s) in human-readable format (-h : Byte, Kilobyte, Megabyte, Gigabyte, Terabyte and Petabyte): )
du -sh ~
du: cannot read directory `/var/lib/openshift/5xxx3/.tmp': Permission denied
du: cannot read directory `/var/lib/openshift/5xxx3/.sandbox': Permission denied
du: cannot read directory `/var/lib/openshift/5xxx3/.ssh': Permission denied
du: cannot read directory `/var/lib/openshift/5xxx3/.gearstats': Permission denied
607M /var/lib/openshift/5xxx3/
4) List open files (lsof is a command meaning "list open files", which is used in many Unix-like systems to report a list of all open
files and the processes that opened them. -n Do not resolve hostnames (no DNS). -P Do not resolve port
names (list port number instead of its name). )
lsof -n -P
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
mongod 471639 3488 11u IPv4 423798423 0t0 TCP 127.x.y.z:27017 (LISTEN)
node 475151 3488 10u IPv4 423815802 0t0 TCP 127.x.y.z:8080 (LISTEN)
5) Display top CPU intensive processes (top Provide information (frequently refreshed) about the most CPU-intensive processes currently running. You do not
need to include a - before options. -b Run in batch mode; don't accept command-line input. Useful for sending
output to another command or to a file. -n num Update display num times, then exit.)
top -b -n 1
top - 00:48:37 up 13 days, 23:52, 0 users, load average: 2.91, 2.27, 2.09
Tasks: 13 total, 1 running, 12 sleeping, 0 stopped, 0 zombie
Cpu(s): 11.6%us, 10.0%sy, 0.1%ni, 77.5%id, 0.5%wa, 0.0%hi, 0.2%si, 0.1%st
Mem: 15297608k total, 14537912k used, 759696k free, 36456k buffers
Swap: 52428792k total, 16372136k used, 36056656k free, 2720680k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
60898 3488 20 0 12800 968 744 R 1.9 0.0 0:00.02 top
55776 3488 20 0 106m 2740 808 S 0.0 0.0 0:00.00 sshd
55779 3488 20 0 104m 2260 1432 S 0.0 0.0 0:00.09 bash
432471 3488 20 0 106m 888 884 S 0.0 0.0 0:00.00 sshd
432475 3488 20 0 55144 1540 1536 S 0.0 0.0 0:00.11 sftp-server
471611 3488 20 0 9508 412 404 S 0.0 0.0 0:00.00 control
471612 3488 20 0 181m 2152 1720 S 0.0 0.0 0:00.01 logshifter
471624 3488 20 0 4072 456 448 S 0.0 0.0 0:00.00 scl
471625 3488 20 0 9236 812 808 S 0.0 0.0 0:00.00 bash
471639 3488 20 0 373m 14m 13m S 0.0 0.1 0:03.53 mongod
475123 3488 20 0 778m 5264 5172 S 0.0 0.0 0:00.08 node
475124 3488 20 0 117m 2148 1708 S 0.0 0.0 0:00.00 logshifter
475151 3488 20 0 863m 114m 6776 S 0.0 0.8 0:04.10 node
6) Review memory usage. (free -- Display statistics about memory usage: total free, used, physical, swap, shared, and buffers used by the kernel.
Options: -b Calculate memory in bytes. -k Default. Calculate memory in kilobytes. -m Calculate memory in megabytes.)
free
total used free shared buffers cached
Mem: 15297608 14767896 529712 766468 36484 2746820
-/+ buffers/cache: 11984592 3313016
Swap: 52428792 16334312 36094480
This is where I've gone astray. There is still a tiny bit of free space, but it doesn't take me much to figure out when I'm doing an intensive I/O that I'm going to go south fast here. When that happened I didn't see any error log / messages at all. Things just stop working.
7) Check your sockets. (ss - socket statistics. The output will contain all tcp, udp and unix socket connection details. )
ss
State Recv-Q Send-Q Local Address:Port Peer Address:Port
(in this case there are no open sockets.. the line above is just the column headers..)
8) Check VMstat. (vmstat – Summary information of Memory, Processes, Paging etc. Free – Amount of free/idle memory spaces.
si – Swapped in every second from disk in Kilo Bytes. so – Swapped out every second to disk in Kilo Bytes. )
vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 16248996 425248 33476 2946912 88 90 321 247 4 3 12 10 78 0 0
9) Check I/O stats. (iostat – Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions.)
iostat
Linux 2.6.32-573.12.1.el6.x86_64 (ex-std-node842.prod.rhcloud.com) 03/14/2016 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
11.60 0.12 10.21 0.49 0.06 77.52
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
xvda 6.56 197.23 147.83 238703267 178916836
xvdf 15.08 337.29 347.44 408209376 420504392
xvdg 15.13 337.45 347.44 408413143 420502512
xvdp 65.18 1603.17 1060.59 1940282568 1283607613
dm-0 7.97 108.87 33.25 131768290 40238544
dm-1 70.00 1574.18 1060.36 1905191416 1283329611
dm-2 3.48 87.89 114.58 106366791 138678084
10) (mpstat - Report processors related statistics. )
mpstat
Linux 2.6.32-573.12.1.el6.x86_64 (ex-std-node842.prod.rhcloud.com) 03/14/2016 _x86_64_ (4 CPU)
01:10:59 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
01:10:59 AM all 11.60 0.12 10.01 0.49 0.00 0.21 0.06 0.00 77.52
11) User Limits (ulimit User limits - limit the use of system-wide resources. -a All current limits are reported. )
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 59663
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 350
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

NTFS/GPT Mount exited with Exit Code 13

This is a duplicated post since I didn't get any help on askubuntu.com.
I have a 1TB external hard drive that I recently formatted to NTFS. It was mounting on my Ubuntu 11.10 fine until just now. I didn't make any changes to affect my OS or my exhdd.
The error that I get is:
Error mounting: mount exited with exit code 13: $MFTMirr does not match $MFT (record 0).
Failed to mount '/dev/sdb2': Input/output error
NTFS is either inconsistent, or there is a hardware fault, or it's a
SoftRAID/FakeRAID hardware. In the first case run chkdsk /f on Windows
then reboot into Windows twice. The usage of the /f parameter is very
important! If the device is a SoftRAID/FakeRAID then first activate
it and mount a different device under the /dev/mapper/ directory, (e.g.
/dev/mapper/nvidia_eahaabcc1). Please see the 'dmraid' documentation
for more details.
I did read this and this. But neither helped.
I tried installing ntfsfix but no such package exists anymore.
I have never used this HDD on a windows machine. If I need to use an other machine to do stuff to fix this, I have access to a mac.
Any advice?
This is my sudo fdisk -l output:
What in the world is GPT? I didn't do that. It used to be NTFS.
Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000586fb
Device Boot Start End Blocks Id System
/dev/sda1 * 2148 961320312 480659082+ 83 Linux
/dev/sda2 961320313 976773167 7726427+ 5 Extended
/dev/sda5 961320314 976773167 7726427 83 Linux
WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xcfd88605
Device Boot Start End Blocks Id System
/dev/sdb1 1 1953525167 976762583+ ee GPT
This is the thing that worked:
I first needed to get ntfs-3g (sudo apt-get install ntfs-3g)
Run sudo fdisk -l to figure out where the mount point is. Mine was /dev/sdb1
I ran ntfsfix -b /dev/sdb1 and that fixed the problem.
Error mounting: mount exited with exit code 13: $MFTMirr does not match $MFT (record 0). Failed to mount '/dev/sda1': Input/output error
NTFS is either inconsistent, or there is a hardware fault, or it's a SoftRAID/FakeRAID hardware. In the first case run chkdsk /f on Windows then reboot into Windows twice. The usage of the /f parameter is very important! If the device is a SoftRAID/FakeRAID then first activate it and mount a different device under the /dev/mapper/ directory, (e.g. /dev/mapper/nvidia_eahaabcc1).
Please see the 'dmraid' documentation for more details.
Solution :-
sudo fdisk -l
sudo ntfsfix /dev/select_disk_name
To find Disk name:
Go dashboard -> Disk utility -> Click disk -> then show Device /Dev/***

Erlang and its consumption of Heap Memory

I have been running a highly concurrent application on my HP Proliant Servers. The application is a file system indexer i coded in erlang. It spawns a process per Folder it finds on the file system and records all file paths in a fragmented Mnesia Database. (Database consists of disc_only_copies type of tables and a screen shot of its file system can be viewed here.)
The Snippet of code that does the high intensive job of going through the file system is shown below:
%%% -------- COPYRIGHT NOTICE --------------------------------------------------------------------
%% #author Muzaaya Joshua, <joshmuza#gmail.com> [http://joshanderlang.blogspot.com]
%% #version 1.0 free software, but modification prohibited
%% #copyright Muzaaya Joshua (file_scavenger-1.0) 2011 - 2012 . All rights reserved
%% #reference OpenSource Erlang WebSite
%%
%%% ---------------- EDOC INTRODUCTION TO THE MODULE ----------------------------------------------
%% #doc This module provides the low level APIs for reading, writing,
%% searching, joining and moving within directories.The module implementation
%% took place on #date at #time.
%% #end
-module(file_scavenger_utilities).
%%% ------- EXPORTS -------------------------------------------------------------------------------
-compile(export_all).
%%% ------- INCLUDES -----------------------------------------------------------------------------
%%% -------- MACROS ------------------------------------------------------------------------------
-define(IS_FOLDER(X),filelib:is_dir(X)).
-define(IS_FILE(X),filelib:is_file(X)).
-define(FAILED_TO_LIST_DIR(X),error_logger:error_report(["*** File Scavenger Utilities Error ***** ",{error,"Failed to List Directory"},{directory,X}])).
-define(NOT_DIR(X),error_logger:error_report(["*** File Scavenger Utilities Error ***** ",{error,"Not a Directory"},{alleged,X}])).
-define(NOT_FILE(X),error_logger:error_report(["*** File Scavenger Utilities Error ***** ",{error,"Not a File"},{alleged,X}])).
%%%--------- TYPES -------------------------------------------------------------------------------
%% #type dir() = string().
%% Must be containing forward slashes, not back slashes. Must not end with a slash
%% after the exact directory.e.g this is wrong: "C:/Program Files/SomeDirectory/"
%% but this is right: "C:/Program Files/SomeDirectory"
%% #type file_path() = string().
%% Must be containing forward slashes, not back slashes.
%% Should include the file extension as well e.g "C:/Program Files/SomeFile.pdf"
%% -----------------------------------------------------------------------------------------------
%% #doc Enters a directory and executes the fun ForEachFileFound/2 for each file it finds
%% If it finds a directory, it executes the fun %% ForEachDirFound/2.
%% Both funs above take the parent Dir as the first Argument. Then, it will spawn an
%% erlang process that will spread the found Directory too in the same way as the parent directory
%% was spread. The process of spreading goes on and on until every File (wether its in a nested
%% Directory) is registered by its full path.
%% #end
%%
%% #spec spread_directory(dir(),dir(),funtion(),function())-> ok.
spread_directory(Dir,Top_Directory,ForEachFileFound,ForEachDirFound) when is_function(ForEachFileFound),is_function(ForEachDirFound) ->
case ?IS_FOLDER(Dir) of
false -> ?NOT_DIR(Dir);
true ->
F = fun(X)->
FileOrDir = filename:absname_join(Dir,X),
case ?IS_FOLDER(FileOrDir) of
true ->
(catch ForEachDirFound(Top_Directory,FileOrDir)),
spawn(fun() -> ?MODULE:spread_directory(FileOrDir,Top_Directory,ForEachFileFound,ForEachDirFound) end);
false ->
case ?IS_FILE(FileOrDir) of
false -> {error,not_a_file,FileOrDir};
true -> (catch ForEachFileFound(Top_Directory,FileOrDir))
end
end
end,
case file:list_dir(Dir) of
{error,_} -> ?FAILED_TO_LIST_DIR(Dir);
{ok,List} -> lists:foreach(F,List)
end
end.
The function spread_directory/4 is generic in a way that it takes two funs. One fun: ForEachFileFound/2 takes along with the Top Most Directory, the found file and does anything with it and the other fun: ForEachDirFound/2 takes along with the Top Most Directory, the folder it finds and uses it in any way it wants.
The start script i use for this application makes sure that erlang will be able to spawn as many processes as possible. Once a process finishes indexing a folder it exits.
#!/usr/bin/env sh
echo "Starting File Scavenger System. Layer 1 on the P2P File Sharing System....."
erl \
-name file_scavenger#127.0.0.1 \
+P 13421779 \
-pa ./ebin ./lib/*/ebin ./include \
-mnesia dir '"./database"' \
-mnesia dump_log_write_threshold 10000 \
-eval "application:load(file_scavenger)" \
-eval "application:start(file_scavenger)"
There is a gen_server which interfaces the intensive module with the database in which i record all paths. A snippet of where it starts the spread_directory work is shown here below:
handle_cast(index_dirs,#scavenger{directory_paths = Dirs} = State)->
{File,Folder} = case {State#scavenger.verbose,State#scavenger.verbose_to} of
{true,tty} ->
{
fun(TopDir,Fl)->
io:format(" File: ~p~n",[Fl]),
file_scavenger_database:insert_file(filename:basename(Fl),file,Fl,TopDir,filename:extension(Fl))
end,
fun(TopDir,Fd) ->
io:format(" Folder: ~p~n",[Fd]),
file_scavenger_database:insert_file(Fd,folder,Fd,TopDir,undefined)
end
};
{true,SomeFile}->
{
fun(TopDir,Fl)->
os:cmd("echo File: " ++ Fl ++ " >> " ++ SomeFile),
file_scavenger_database:insert_file(filename:basename(Fl),file,Fl,TopDir,filename:extension(Fl))
end,
fun(TopDir,Fd)->
os:cmd("echo Folder: " ++ Fd ++ " >> " ++ SomeFile),
file_scavenger_database:insert_file(Fd,folder,Fd,TopDir,undefined)
end
}
end,
Main = fun(Dir) ->
error_logger:info_msg("*** File scavenger Server indexing directory: ~p~n",[Dir]),
spawn(fun() -> file_scavenger_utilities:spread_directory(Dir,Dir,File,Folder) end)
end,
lists:foreach(Main,Dirs),
{noreply,State};
handle_cast(stop, State) -> {stop, normal, State}.
More Source details can be found in the whole application.
The application entire Source and build can be found here: File_scavenger-1.0.zip.
Now, i start the application on the Server (HP Proliant G6, containing Intel processors (2 processors, each 4 cores, 2.4 GHz speed each core, 8 MB Cache size), 20 GB RAM size, 1.5 Terabytes disk space. Now, 2 of these high power machines are in our disposal. System Database should be replicated across the two. Each server runs Solaris 10, 64 bit), whose terminal now looks like this below:
bash-3.00# sh file_scavenger.sh
Starting File Scavenger System. Layer 1 on the P2P File Sharing System.....
Erlang R14B03 (erts-5.8.4) [source] [smp:8:8] [rq:8] [async-threads:0] [hipe] [kernel-poll:false]
Eshell V5.8.4 (abort with ^G)
(file_scavenger#127.0.0.1)1>
=INFO REPORT==== 18-Aug-2011::09:36:04 ===
Starting File Scavenger Database......
=INFO REPORT==== 18-Aug-2011::09:36:04 ===
Database Successfully Started....
=INFO REPORT==== 18-Aug-2011::09:36:04 ===
Starting File Scavenger Database......
=INFO REPORT==== 18-Aug-2011::09:36:04 ===
Database Successfully Started....
=INFO REPORT==== 18-Aug-2011::09:36:04 ===
File Scavenger Server starting with default verbose settings....
(file_scavenger#127.0.0.1)1> file_scavenger_server:index_dirs().
The server starts to run and verboses to the terminal all files and folders it finds. The server is equipped with too much RAM (20 GB), and Swap space (Swap is 16 GB). However, it ran for about 18 hours and finally, the erlang Virtual machine reported this:
File: "/proc/4324/root/opt/csw/gcc4/share/locale/ja/LC_MESSAGES/gcc.mo"
Folder: "/proc/4324/root/opt/csw/gcc4/share/locale/da"
Folder: "/proc/4324/root/opt/csw/gcc4/share/locale/es/LC_MESSAGES"
File: "/proc/4324/root/proc/4984/root/.thumbnails/normal/dc259e3897e8af4b379c6d956b6c1393.png"
File: "/proc/4324/root/proc/4984/root/.thumbnails/fail/gnome-thumbnail-factory/223c19786421b7101d14075bdec46f61.png"
File: "/proc/4324/root/opt/csw/gcc4/libexec/gcc/i386-pc-solaris2.10/4.5.1/install-tools/mkheaders"
File: "/proc/4324/root/opt/csw/gcc4/libexec/gcc/i386-pc-solaris2.10/4.5.1/cc1plus"
File: "/proc/4324/root/opt/csw/gcc4/lib/libsupc++.la"
Crash dump was written to: erl_crash.dump
eheap_alloc: Cannot allocate 153052320 bytes of memory (of type "heap").
Abort - core dumped
bash-3.00#
Question 1. With such a powerful server, why would the operating system fail to provide such memory to the application (it was the only application running)?
Question 2. The Erlang Emulator i start is instructed to be able to spawn as many processes as it may need. the value +P 13421779. Is Erlang VM failing to access this memory or failing to allocate it to its processes ?
Question 3. To Solaris, it sees one process: epmd, perhaps containing and starting thousands of micro threads. What configurations can i make to Solaris to be able to never stop my application however much "memory hungry" it may be? Swap space available is 16 GB, RAM 20 GB, honestly, there must be something wrong.
Question 4. Which configurations can i make to the Erlang Emulator, to avoid these heap memory crash dumps especially when all the memory it may need is available on the server? How will i run more memory consuming apps on this server if Erlang still fails to allocate such memory to a simple file system indexer (well its heavily concurrent)?
finally, all other tweaks i could do to avoid heap memory problems on such capable hardware are welcome. Thanks in advance
I haven't had time to look at the source, but here are some comments:
Question 1. With such a powerful server, why would the operating
system fail to provide such memory to the application (it was the only
application running)?
Because the Erlang VM tried to consume more than the available free memory.
Question 2. The Erlang Emulator i start is instructed to be able to
spawn as many processes as it may need. the value +P 13421779. Is
Erlang VM failing to access this memory or failing to allocate it to
its processes ?
No. If you would have run out of processess, the Erlang VM would have said so (and the VM would still be up and running):
=ERROR REPORT==== 18-Aug-2011::10:04:04 ===
Error in process <0.31775.138> with exit value: {system_limit,[{erlang,spawn_link, [erlang,apply,[#Fun<shell.3.130303173>,[]]]},{erlang,spawn_link,1},{shell,get_command,5}, {shell,server_loop,7}]}
Question 3. To Solaris, it sees one process: epmd, perhaps containing
and starting thousands of micro threads. What configurations can i
make to Solaris to be able to never stop my application however much
"memory hungry" it may be? Swap space available is 16 GB, RAM 20 GB,
honestly, there must be something wrong.
epmd is the Erlang port mapping deamon. It's responsible for managing distributet Erlang and has nothing to with your individual Erlang application. The processes you should look for will be name beam.smp most likely. These will show the OS memory consumption of the Erlang VM etc.
Question 4. Which configurations can i make to the Erlang Emulator, to
avoid these heap memory crash dumps especially when all the memory it
may need is available on the server? How will i run more memory
consuming apps on this server if Erlang still fails to allocate such
memory to a simple file system indexer (well its heavily concurrent)?
The Erlang VM should be able to use all of the available memory in your machine. However, it depends on how your application is written. There can be many reasons for memory leaks:
Atom table filling up (you create too many unique atoms)
ETS or Mnesia tables are not garbage collected (you do not delete old unused elements)
Not enough memory for processes (you spawn too many processess)
Too many binaries are created (you might keep unused references to old binaries)