root#kali:~/ROUTER# binwalk new-firmware.bin
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
84 0x54 uImage header, header size: 64 bytes, header CRC: 0xE52A7F50, created: 2012-02-10 07:27:12, image size: 819799 bytes, Data Address: 0x80002000, Entry Point: 0x801AC9F0, data CRC: 0x6A10D412, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: lzma, image name: "Linux Kernel Image"
148 0x94 LZMA compressed data, properties: 0x5D, dictionary size: 8388608 bytes, uncompressed size: 2386252 bytes
917588 0xE0054 Squashfs filesystem, little endian, version 4.0, compression:lzma, size: 2588426 bytes, 375 inodes, blocksize: 16384 bytes, created: 2016-02-05 02:05:56
root#kali:~/ROUTER# dd if=new-firmware.bin of=uboot.lzma skip=148 bs=1
3735488+0 records in
3735488+0 records out
3735488 bytes (3.7 MB, 3.6 MiB) copied, 4.16712 s, 896 kB/s
root#kali:~/ROUTER# dd if=new-firmware.bin of=kernel.squash skip=917588 bs=1
2818048+0 records in
2818048+0 records out
2818048 bytes (2.8 MB, 2.7 MiB) copied, 3.46517 s, 813 kB/s
All looks to have worked so far but then I try the following:
root#kali:~/ROUTER# lzma -d uboot.lzma
lzma: uboot.lzma: Compressed data is corrupt
root#kali:~/ROUTER# unsquashfs kernel.squash
Parallel unsquashfs: Using 4 processors
lzma uncompress failed with error code 0
read_block: failed to read block #0x277af0
read_fragment_table: failed to read fragment table index
FATAL ERROR:failed to read fragment table
Any idea why it may be happening? As the prompt says, I am on Kali Linux. ALso get this
jj#ubuntu:~/Firmware$ binwalk -e new-firmware.bin
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
84 0x54 uImage header, header size: 64 bytes, header CRC: 0xE52A7F50, created: 2012-02-10 07:27:12, image size: 819799 bytes, Data Address: 0x80002000, Entry Point: 0x801AC9F0, data CRC: 0x6A10D412, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: lzma, image name: "Linux Kernel Image"
148 0x94 LZMA compressed data, properties: 0x5D, dictionary size: 8388608 bytes, uncompressed size: 2386252 bytes
WARNING: Extractor.execute failed to run external extractor 'sasquatch -p 1 -le -d '%%squashfs-root%%' '%e'': [Errno 2] No such file or directory
WARNING: Extractor.execute failed to run external extractor 'sasquatch -p 1 -be -d '%%squashfs-root%%' '%e'': [Errno 2] No such file or directory
917588 0xE0054 Squashfs filesystem, little endian, version 4.0, compression:lzma, size: 2588426 bytes, 375 inodes, blocksize: 16384 bytes, created: 2016-02-05 02:05:56
Try
lzma -d < uboot.lzma > uboot
The error you found is due to LZMA Utils having been superseded by XV Utils, which don't ignore the trailing garbage. If you use it as a stream (like I gave you), you'll still have the corruption error message in the end, but you'll get your file alright.
Source (search for "corrupt" and you'll find other related comments)
Try
binwalk -e new-firmware.bin
you will end up with 2 file one being extracted
In the extracted one you will get 2 one unzippped and a zipped one
you can use dd at this point and unlzma should work at least it did work out for me
this is what i mean
root#ROOT20:~/Desktop/project/dlink/dlink2# binwalk dcs932l_v1.14.04.bin
DECIMAL HEXADECIMAL DESCRIPTION
106352 0x19F70 U-Boot version string, "U-Boot 1.1.3"
106816 0x1A140 CRC32 polynomial table, little endian
124544 0x1E680 HTML document header
124890 0x1E7DA HTML document footer
124900 0x1E7E4 HTML document header
125092 0x1E8A4 HTML document footer
125260 0x1E94C HTML document header
125953 0x1EC01 HTML document footer
327680 0x50000 uImage header, header size: 64 bytes, header CRC: 0x88345E96, created: 2016-09-09 13:52:27, image size: 3804958 bytes, Data Address: 0x80000000, Entry Point: 0x803B8000, data CRC: 0x531E94DE, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: lzma, image name: "Linux Kernel Image"
327744 0x50040 LZMA compressed data, properties: 0x5D, dictionary size: 33554432 bytes, uncompressed size: 6558763 bytes
root#ROOT20:~/Desktop/project/dlink/dlink2# dd if=dcs932l_v1.14.04.bin skip=327744 bs=1 of=kernel.lzma
3866560+0 records in
3866560+0 records out
3866560 bytes (3.9 MB, 3.7 MiB) copied, 9.61104 s, 402 kB/s
root#ROOT20:~/Desktop/project/dlink/dlink2# unlzma kernel.lzma
unlzma: kernel.lzma: Compressed data is corrupt
root#ROOT20:~/Desktop/project/dlink/dlink2# binwalk -e dcs932l_v1.14.04.bin
DECIMAL HEXADECIMAL DESCRIPTION
106352 0x19F70 U-Boot version string, "U-Boot 1.1.3"
106816 0x1A140 CRC32 polynomial table, little endian
124544 0x1E680 HTML document header
124890 0x1E7DA HTML document footer
124900 0x1E7E4 HTML document header
125092 0x1E8A4 HTML document footer
125260 0x1E94C HTML document header
125953 0x1EC01 HTML document footer
327680 0x50000 uImage header, header size: 64 bytes, header CRC: 0x88345E96, created: 2016-09-09 13:52:27, image size: 3804958 bytes, Data Address: 0x80000000, Entry Point: 0x803B8000, data CRC: 0x531E94DE, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: lzma, image name: "Linux Kernel Image"
327744 0x50040 LZMA compressed data, properties: 0x5D, dictionary size: 33554432 bytes, uncompressed size: 6558763 bytes
root#ROOT20:~/Desktop/project/dlink/dlink2# ls
dcs932l_v1.14.04.bin _dcs932l_v1.14.04.bin.extracted kernel.lzma
root#ROOT20:~/Desktop/project/dlink/dlink2# cd _dcs932l_v1.14.04.bin.extracted/
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted# ls
50040 50040.7z
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted# binwalk 50040
DECIMAL HEXADECIMAL DESCRIPTION
3145804 0x30004C Linux kernel version 2.6.21
3175792 0x307570 SHA256 hash constants, little endian
3389960 0x33BA08 Unix path: /usr/gnemul/irix/
3392244 0x33C2F4 Unix path: /usr/lib/libc.so.1
3393940 0x33C994 Unix path: /dev/vc/0
3408260 0x340184 Copyright string: "Copyright (c) 2010 Alpha Networks Inc."
3491536 0x3546D0 Unix path: /etc/Wireless/RT2860STA/RT2860STA.dat
3573187 0x3685C3 Neighborly text, "neighbor %.2x%.2x.%.2x:%.2x:%.2x:%.2x:%.2x:%.2x lost on port %d(%s)(%s)"
3807776 0x3A1A20 CRC32 polynomial table, little endian
4038656 0x3DA000 LZMA compressed data, properties: 0x5D, dictionary size: 1048576 bytes, uncompressed size: 8072704 bytes
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted# dd if=50040 skip=4038656 bs=1 of=kernel2.lzma
2520107+0 records in
2520107+0 records out
2520107 bytes (2.5 MB, 2.4 MiB) copied, 8.23342 s, 306 kB/s
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted# unlzma kernel2.lzma
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted#
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted# mkdir cpio; cd cpio
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted/cpio# cpio -idm --no-absolute-filenames < ../kernel2
cpio: Removing leading `/' from member names
15767 blocks
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted/cpio# ls
bin dev etc etc_ro home init lib media mnt mydlink proc sbin sys tmp usr var
Related
From the snippet below, I can see that the relocation entry is to be applied at offset 0x000000201018 of some section, but how can I determine what section?
Also all of my sections are smaller than this offset. What are the implications of this?
Relocations:
Relocation section '.rela.plt' at offset 0x4f8 contains 1 entries:
Offset Info Type Sym. Value Sym. Name + Addend
000000201018 000300000007 R_X86_64_JUMP_SLO 0000000000000000 printk + 0
Section Headers:
Section Headers:
[Nr] Name Type Address Offset
Size EntSize Flags Link Info Align
...
[ 6] .rela.plt RELA 00000000000004f8 000004f8
0000000000000018 0000000000000018 AI 3 21 8
...
[21] .got.plt PROGBITS 0000000000201000 00001000
0000000000000020 0000000000000008 WA 0 0 8
The section to which the relocation is applied is specified in the info field of the section headers. From the example elf in the question, the relocation section .rela.plt applies to section number 21, the .got.plt section.
The offset field has different meaning for object file and shared object file. "For a relocatable file, the value is the byte offset from the beginning of the section to the storage unit affected by the relocation. For an executable file or a shared object, the value is the virtual address of the storage unit affected by the relocation."
Source/More Info: http://www.skyfree.org/linux/references/ELF_Format.pdf
I try to convert an heavy .txt which is the string of a dicionary like (a part):
"{'A45171': {'Gen_n': 'Putative uncharacterized protein', 'Srce': 'UniProtKB', 'Ref': 'GO_REF:0000033', 'Tax': 'NCBITaxon:684364', 'Gen_bl': 'BATDEDRAFT_15336', 'Gen_id': 'UniProtKB:F4NTD6', 'Ev_n': 'IBA', 'GO_n': 'ergosterol biosynthetic process', 'GO': 'GO:0006696', 'Org': 'Batrachochytrium dendrobatidis JAM81', 'Type': 'protein', 'Ev_e': 'ECO:0000318', 'Con': 'GO_Central'}, 'A43886': {'Gen_n': 'Uncharacterized protein', 'Srce': 'UniProtKB', 'Ref': 'GO_REF:0000002', 'Tax': 'NCBITaxon:9823', 'Gen_bl': 'RDH8', 'Gen_id': 'UniProtKB:F1S3H8', 'Ev_n': 'IEA', 'GO_n': 'estrogen biosynthetic process', 'GO': 'GO:0006703', 'Org': 'Sus scrofa', 'Type': 'protein', 'Ev_e': 'ECO:0000501', 'Con': 'InterPro'}}"
I've tryed ast module:
import ast
dic_gene_definitions = open("Gene_Ontology/output_data/dic_gene_definitions.txt", "r")
dic_gene_definitions = dic_gene_definitions.read()
dic_gene_definitions = ast.literal_eval(dic_gene_definitions)
Which weight 22Mb and when don't crush, it runs so slow.
I really wants to open an 500 Mb files...
I've look json module which can open so faster, but in heavy dictionary string it crash also (not with short examples).
Any solution...?
Thank you so much.
I've look some non-RAM memory consuming method.
By use in Ubuntu's terminal:
sudo swapon -s
we can appreciate the RAM memory consumed by different opperations:
Filename Type Size Used Priority
/swapfile file 2097148 19876 -1
By to operate with this example file (500Mb) by make an dictionary, for example, best way is to open it from normal text tabulated data format, and operate with minour RAM consumption:
with open("Gene_Ontology/output_data/GO_annotations_dictionary.txt", "r") as handle:
for record in handle.read().splitlines():
anote = record.split("\t")
ast module is fine but not by large files.
Is there a way to analyze the contents of a specific index (fdb file)? I know I can see the index creation statement and try to guess from there but it would be nice if there is a way to see the contents/records inside an fdb file.
two tools cbindex and forestdb_dump can help. These are available in the bin folder along with other couchbase binaries. Note that, these tools are not supported, as documented at http://developer.couchbase.com/documentation/server/4.5/release-notes/relnotes-40-ga.html
given bucket/indexname, tool cbindex gets index level details:
couchbases-MacBook-Pro:bin varakurprasad$ pwd
/Users/varakurprasad/Downloads/couchbase-server-enterprise_451_GA/Couchbase Server.app/Contents/Resources/couchbase-core/bin
couchbases-MacBook-Pro:bin varakurprasad$ ./cbindex -server 127.0.0.1:8091 -type scanAll -bucket travel-sample -limit 4 -index def_type -auth Administrator:couch1
ScanAll index:
[airline] ... airline_10
[airline] ... airline_10123
[airline] ... airline_10226
[airline] ... airline_10642
Total number of entries: 4
Given a forestdb file, the tool forestdb_dump gets more low level details:
couchbases-MacBook-Pro:varakurprasad$ pwd
/Users/varakurprasad/Library/Application Support/Couchbase/var/lib/couchbase/data/#2i/travel-sample_def_type_1018858748122363634_0.index
couchbases-MacBook-Pro:varakurprasad$ forestdb_dump data.fdb.53 | more
[FDB INFO] Forestdb opened database file data.fdb.53
DB header info:
BID: 1568 (0x620, byte offset: 6422528)
DB header length: 237 bytes
DB header revision number: 3
...
Doc ID: airline_10
KV store name: back
Sequence number: 14637
Byte offset: 2063122
Indexed by the main index
Length: 10 (key), 0 (metadata), 24 (body)
Status: normal
Metadata: (null)
Body:^Fairline
...
From ELF Documentation:
SHT_STRTAB
The section holds a string table. An object file may have multiple string table sections. See ‘‘String Table’’ below for details.
(Note: I didn't notice any information regarding multiple string table sections in ‘‘String Table’’ paragraph)
Does multiple string table sections mean string table for the section headers and string table for the object file itself?
There is no mention in the documentation regarding how to read a string if there are multiple string table for the object itself (.strtab).
Any clarification about the subject appreciated.
The manpage only has a overview/summary of (some parts of) the ELF file format, you might want to look at the System V ABI spec.
Explanation
Linking view
An ELF file has multiple string tables. Usually you have 3-4 string tables:
One string table (usually called .shstrtab) is used for section names. All section names (in the section header table) are taken from a single string table. This string table is identified by its index in the section header table: the index of the section name string table is indicated in the ELF header (e_shstrndx).
Another string table (usually called .strtab) is used for the full symbol table (.symtab). The same string table is used by the .dynamic section.
Another string table (usually called .dynstr) is used for the minirmal symbol table (.dynsym).
Another string table is used for
For a given symbol table section, the section used as string table is indicated in the sh_link field of the section header table (see Fig 4-12 of the system V ABI spec).
Execution view
For the execution view (the program header table), the address of the string table used for the symbol table (DT_SYMTAB) is given in the DT_STRTAB entry of the dynamic section.
Example
Linking view
This is a hello world program (shown with readelf -a).
.shtrtab
The ELF header:
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: EXEC (Executable file)
Machine: Advanced Micro Devices X86-64
Version: 0x1
Entry point address: 0x4003c0
Start of program headers: 64 (bytes into file)
Start of section headers: 4624 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 56 (bytes)
Number of program headers: 8
Size of section headers: 64 (bytes)
Number of section headers: 30
Section header string table index: 27
tells us that the section names are in section 27. Conveniently enough this is .shtrtab:
Section Headers:
[Nr] Name Type Address Offset
Size EntSize Flags Link Info Align
[...]
[27] .shstrtab STRTAB 0000000000000000 000008e0
0000000000000108 0000000000000000 0 0 1
.dynstr
For .dynsym we have:
[ 5] .dynsym DYNSYM 0000000000400280 00000280
0000000000000048 0000000000000018 A 6 1 8
^
HERE
Its names are taken from section 6 which is .dynstr:
[ 6] .dynstr STRTAB 00000000004002c8 000002c8
0000000000000038 0000000000000000 A 0 0 1
This string table is used by other sections as well:
[ 8] .gnu.version_r VERNEED 0000000000400308 00000308
0000000000000020 0000000000000000 A 6 1 8
[21] .dynamic DYNAMIC 0000000000600698 00000698
00000000000001d0 0000000000000010 WA 6 0 8
.strtab
For .symtab:
[28] .symtab SYMTAB 0000000000000000 000009e8
0000000000000600 0000000000000018 29 45 8
^
HERE
the names are taken from section 29 which happens to be .strtab:
[29] .strtab STRTAB 0000000000000000 00000fe8
0000000000000224 0000000000000000 0 0 1
Execution view
Dynamic section at offset 0x698 contains 24 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x000000000000000c (INIT) 0x400370
0x000000000000000d (FINI) 0x400544
0x0000000000000019 (INIT_ARRAY) 0x600680
0x000000000000001b (INIT_ARRAYSZ) 8 (bytes)
0x000000000000001a (FINI_ARRAY) 0x600688
0x000000000000001c (FINI_ARRAYSZ) 8 (bytes)
0x000000006ffffef5 (GNU_HASH) 0x400260
0x0000000000000005 (STRTAB) 0x4002c8 <= HERE
0x0000000000000006 (SYMTAB) 0x400280
0x000000000000000a (STRSZ) 56 (bytes)
0x000000000000000b (SYMENT) 24 (bytes)
0x0000000000000015 (DEBUG) 0x0
0x0000000000000003 (PLTGOT) 0x600870
0x0000000000000002 (PLTRELSZ) 48 (bytes)
0x0000000000000014 (PLTREL) RELA
0x0000000000000017 (JMPREL) 0x400340
0x0000000000000007 (RELA) 0x400328
0x0000000000000008 (RELASZ) 24 (bytes)
0x0000000000000009 (RELAENT) 24 (bytes)
0x000000006ffffffe (VERNEED) 0x400308
0x000000006fffffff (VERNEEDNUM) 1
0x000000006ffffff0 (VERSYM) 0x400300
0x0000000000000000 (NULL) 0x0
The string table for dynamic linking is located at 0x4002c8 in the program memory.
Note: this is .dynstr.
I am trying to stream a WebM format video being generated using gstreamer and individual frame being sent over websockets. A typical byte arrangement of webm file is like this (you may be already familiar with this).
EBML (head size: 12 bytes, data: 16 bytes, pos: 0, '0x0')
DocType (head size: 3 bytes, data: 5 bytes, pos: 12L, '0xcL') : 'webm\x00'
DocTypeVersion (head size: 3 bytes, data: 1 bytes, pos: 20L, '0x14L') : 2
DocTypeReadVersion (head size: 3 bytes, data: 1 bytes, pos: 24L, '0x18L') : 2
...
...
SegmentInfo (head size: 12 bytes, data: 91 bytes, pos: 192L, '0xc0L')
TimecodeScale (head size: 4 bytes, data: 3 bytes, pos: 204L, '0xccL') : 1000000
Duration (head size: 3 bytes, data: 8 bytes, pos: 211L, '0xd3L') : 0.0
MuxingApp (head size: 3 bytes, data: 31 bytes, pos: 222L, '0xdeL') : 'GStreamer plugin version 1.2.4\x00'
WritingApp (head size: 3 bytes, data: 25 bytes, pos: 256L, '0x100L') : 'GStreamer Matroska muxer\x00'
DateUTC (head size: 3 bytes, data: 8 bytes, pos: 284L, '0x11cL') : 447902803000000000L
Video (head size: 9 bytes, data: 8 bytes, pos: 295L, '0x127L')
Pixel Width (head size: 2 bytes, data: 2 bytes, pos: 351L, '0x15fL') : 640
Pixel Height (head size: 2 bytes, data: 2 bytes, pos: 355L, '0x163L') : 480
Codec Id (head size: 2 bytes, data: 6 bytes, pos: 359L, '0x167L') : 'V_VP8\x00'
Cluster (head size: 12 bytes, data: 72057594037927935L bytes, pos: 367L, '0x16fL')
TimeCode (head size: 2 bytes, data: 2 bytes, pos: 379L, '0x17bL') : 1514
SimpleBlock (head size: 4 bytes, data: 44618 bytes, pos: 383L, '0x17fL') : 'binary'
track number : 1, keyframe : True, invisible : 'no', discardable : 'no'
lace : 'no lacing', time code : 0, time code(absolute) : 1514
SimpleBlock (head size: 3 bytes, data: 793 bytes, pos: 45005L, '0xafcdL') : 'binary'
track number : 1, keyframe : False, invisible : 'no', discardable : 'no'
lace : 'no lacing', time code : 27, time code(absolute) : 1541
<<conitnued....>>
What I see, the absolute time code and relative time code is being written correctly when I redirect the gstreamer output to filesink. The same gstreamer pipeline is used to extract the byte sequences (samples). These samples are then transmitted over websocket and received on client side using MediaSource API.
My implementation of client javascript is described here.
When I run the client in Firefox, the video runs smoothly without any glitches. But on Chrome, the video freezes after some time or at the beginning.
I tried modifying sourceBuffer.mode = "sequence" or "segments", none of the options work on Chrome, whereas the video feed on Firefox is totally unaffected by any value of "sourceBuffer.mode". The description of these modes is here. (I am assuming that the MediaSource API works the same way on IE and Firefox, as no documentation available on Mozilla website).
Also, mediaSource.duration is Infinity/NaN in Chrome and Firefox both.
No matter which way I try, the live feed on chrome is not at all working, whereas Firefox displays smooth video. Any suggestions, why this could be happening?
UPDATES:
I upgraded to Chrome version 41 which gives more details on chrome://media-internals. The message that is shown is:
render_id: 23
player_id: 1
pipeline_state: kStopped
EVENT: WEBMEDIAPLAYER_DESTROYED
url: blob:http%3A//localhost%3A8080/172f68c8-9ff3-4983-9dcb- 396b3f843752
found_video_stream: true
video_codec_name: vp8
duration: unknown
video_dds: false
video_decoder: FFmpegVideoDecoder
error: Append: stream parsing failed. Data size=2283 append_window_start=0 append_window_end=inf
pipeline_error: pipeline: decode error
Timestamp Property Value
00:00:00 00 pipeline_state kCreated
00:00:00 00 EVENT PIPELINE_CREATED
00:00:00 00 EVENT WEBMEDIAPLAYER_CREATED
00:00:00 00 url blob:http%3A//localhost%3A8080/172f68c8-9ff3-4983-9dcb-396b3f843752
00:00:00 00 pipeline_state kInitDemuxer
00:00:01 687 found_video_stream true
00:00:01 692 video_codec_name vp8
00:00:01 692 duration unknown
00:00:01 692 pipeline_state kInitRenderer
00:00:01 694 video_dds false
00:00:01 694 video_decoder FFmpegVideoDecoder
00:00:01 695 pipeline_state kPlaying
00:00:10 989 EVENT PLAY
00:00:11 276 error Got a block with a timecode before the previous block.
00:00:11 276 error Append: stream parsing failed. Data size=2283 append_window_start=0 append_window_end=inf
00:00:11 276 pipeline_error pipeline: decode error
00:00:11 276 pipeline_state kStopping
00:00:11 277 pipeline_state kStopped
00:01:14 239 EVENT WEBMEDIAPLAYER_DESTROYED
How to fix or calculate the "append_window_end"???