Reading index content, possible? - couchbase

Is there a way to analyze the contents of a specific index (fdb file)? I know I can see the index creation statement and try to guess from there but it would be nice if there is a way to see the contents/records inside an fdb file.

two tools cbindex and forestdb_dump can help. These are available in the bin folder along with other couchbase binaries. Note that, these tools are not supported, as documented at http://developer.couchbase.com/documentation/server/4.5/release-notes/relnotes-40-ga.html
given bucket/indexname, tool cbindex gets index level details:
couchbases-MacBook-Pro:bin varakurprasad$ pwd
/Users/varakurprasad/Downloads/couchbase-server-enterprise_451_GA/Couchbase Server.app/Contents/Resources/couchbase-core/bin
couchbases-MacBook-Pro:bin varakurprasad$ ./cbindex -server 127.0.0.1:8091 -type scanAll -bucket travel-sample -limit 4 -index def_type -auth Administrator:couch1
ScanAll index:
[airline] ... airline_10
[airline] ... airline_10123
[airline] ... airline_10226
[airline] ... airline_10642
Total number of entries: 4
Given a forestdb file, the tool forestdb_dump gets more low level details:
couchbases-MacBook-Pro:varakurprasad$ pwd
/Users/varakurprasad/Library/Application Support/Couchbase/var/lib/couchbase/data/#2i/travel-sample_def_type_1018858748122363634_0.index
couchbases-MacBook-Pro:varakurprasad$ forestdb_dump data.fdb.53 | more
[FDB INFO] Forestdb opened database file data.fdb.53
DB header info:
BID: 1568 (0x620, byte offset: 6422528)
DB header length: 237 bytes
DB header revision number: 3
...
Doc ID: airline_10
KV store name: back
Sequence number: 14637
Byte offset: 2063122
Indexed by the main index
Length: 10 (key), 0 (metadata), 24 (body)
Status: normal
Metadata: (null)
Body:^Fairline
...

Related

Octave buffering console output

I'm new to Octave.
I'm working with a huge matrix with thousands of rows and just 2 columns.
When I print it the terminal seems to be cutting part of it, but I want the whole thing to be shown.
Any way to do that or maybe to redirect the output in a file or something?
I had the same in problem in Eclipse but it was much easier to solve the same issue back then.
to redirect a Matrix to a file use save
A=rand(5,5)
A =
0.552789 0.570720 0.896715 0.982821 0.622758
0.704791 0.514302 0.380680 0.344768 0.459588
0.230478 0.744468 0.668264 0.659316 0.414710
0.819015 0.572315 0.404128 0.074092 0.813156
0.099461 0.031186 0.499781 0.758091 0.243281
save("-ascii","file-ascii.txt","A")
save("file-text.txt","A")
pure text
$ cat file-ascii.txt
5.52789005e-01 5.70719716e-01 8.96714571e-01 9.82821412e-01 6.22757836e-01
7.04791031e-01 5.14301972e-01 3.80679512e-01 3.44768074e-01 4.59587827e-01
2.30478499e-01 7.44467506e-01 6.68264066e-01 6.59315817e-01 4.14709954e-01
8.19014881e-01 5.72314594e-01 4.04128230e-01 7.40916214e-02 8.13155903e-01
9.94607828e-02 3.11857103e-02 4.99780576e-01 7.58091470e-01 2.43280663e-01
octave default style
$ cat file-text.txt
# Created by Octave 5.2.0, Wed May 27 21:43:22 2020 CEST <Marco#LAPTOP-82F08ILC>
# name: A
# type: matrix
# rows: 5
# columns: 5
0.55278900467056769 0.57071971558810353 0.89671457115004172 0.98282141203898321 0.622757835845834
0.70479103140202992 0.51430197227101204 0.3806795123567624 0.34476807355856237 0.45958782700965189
0.2304784985452572 0.74446750644064241 0.66826406647042169 0.65931581744685042 0.41470995365627888
0.81901488130574696 0.57231459443213173 0.40412823039438278 0.074091621362711677 0.81315590327055964
0.099460782832668723 0.031185710267227097 0.49978057628118927 0.75809147040922908 0.24328066285306477

TCL Expect. How to get specific symbols from output?

Has very little experience in programming and allmost no expirience in TCL Expect, but forced to use it.
I has output like that:
SMG2016-[CONFIG]-SIP-USERS> add user 1200011 adm.voip.partner.ru 0
Creating new Sip-User.
'SIP USER' [00] ID [1]:
name: Subscriber#000
IPaddr: 0.0.0.0
SIP domain: adm.voip.partner.ru
dynamic registration: off
number: 1200011
Numplan: 0
number list:
00) ---
01) none
02) none
03) none
04) none
AON number:
AON type number: subscriber
profile: 0
category: 1
access cat: 0
auth: none
cliro: off
pbxprofile: 0
access mode: on
lines: 1
No src-port control: off
BLF usage: off
BLF subscribers: 10
Intercom mode: sendonly
Intercom priority: 3
So i need to put in variable 00 from the 'SIP USER' [00] string, and the number in bracers could be up to four digits in row.
How shoud i do this? Any help please?
UPD:
Done this, works for me, even while having trouble with first zero.
expect -indices -re "'SIP USER' .{1}(\[0-9]{2,4}).{1}"
set userid [string trimleft $expect_out(1,string) "0"]

neo4j: How to load CSV using conditional correctly?

What I am trying to do is to import a dataset with a tree data structure inside from CSV to neo4j. Nodes are stored along with their parent node and depth level (max 6) in the tree. So I try to check depth level using CASE and then add a node to its parent like this (creating a node just for 1st level so far for testing purpose):
export FILEPATH=file:///Example.csv
CREATE CONSTRAINT ON (n:Node) ASSERT n.id IS UNIQUE;
USING PERIODIC COMMIT 500
LOAD CSV WITH HEADERS
FROM {FILEPATH} AS line
WITH DISTINCT line,
line.`Level` AS level,
line.`ParentCodeID_Cal` AS parentCode,
line.`CodeSet` AS codeSet,
line.`Category` AS nodeCategory,
line.`Type` AS nodeType,
line.`L1code` AS l1Code, line.`L1Description` AS l1Description, line.`L1Name` AS l1Name, line.`L1NameAb` AS l1NameAb,
line.`L2code` AS l2Code, line.`L2Description` AS l2Description, line.`L2Name` AS l2Name, line.`L2NameAb` AS l2NameAb,
line.`L3code` AS l3Code, line.`L3Description` AS l3Description, line.`L3Name` AS l3Name, line.`L3NameAb` AS l3NameAb,
line.`L1code` AS l4Code, line.`L4Description` AS l4Description, line.`L4Name` AS l4Name, line.`L4NameAb` AS l4NameAb,
line.`L1code` AS l5Code, line.`L5Description` AS l5Description, line.`L5Name` AS l5Name, line.`L5NameAb` AS l5NameAb,
line.`L1code` AS l6Code, line.`L6Description` AS l6Description, line.`L6Name` AS l6Name, line.`L6NameAb` AS l6NameAb,
codeSet + parentCode AS nodeId
CASE line.`Level`
WHEN '1' THEN CREATE (n0:Node{id:nodeId, description:l1Description, name:l1Name, nameAb:l1NameAb, category:nodeCategory, type:nodeType})
ELSE
END;
But I get this result:
WARNING: Invalid input 'S': expected 'l/L' (line 17, column 3 (offset:
982)) "CASE level " ^
I'm aware there is a mistake at syntax.
I'm using neo4j 3.0.4 & Windows 10 (using neo4j shell running it with D:\Program Files\Neo4j CE 3.0.4\bin>java -classpath neo4j-desktop-3.0.4.jar org.neo4j.shell.StartClient).
You have several syntax errors. For example, a CASE clause cannot contain a CREATE clause.
In any case, you should be able to greatly simplify your Cypher. For example, this might suit your needs:
USING PERIODIC COMMIT 500
LOAD CSV WITH HEADERS
FROM {FILEPATH} AS line
WITH DISTINCT line, ('l' + line.Level) AS prefix
CREATE (:Node{
id: line.CodeSet + line.ParentCodeID_Cal,
description: line[prefix + 'Description'],
name: line[prefix + 'Name'],
nameAb: line[prefix + 'NameAb'],
category: line.Category,
type: line.Type})

Binwalk - Compressed data is corrupt

root#kali:~/ROUTER# binwalk new-firmware.bin
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
84 0x54 uImage header, header size: 64 bytes, header CRC: 0xE52A7F50, created: 2012-02-10 07:27:12, image size: 819799 bytes, Data Address: 0x80002000, Entry Point: 0x801AC9F0, data CRC: 0x6A10D412, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: lzma, image name: "Linux Kernel Image"
148 0x94 LZMA compressed data, properties: 0x5D, dictionary size: 8388608 bytes, uncompressed size: 2386252 bytes
917588 0xE0054 Squashfs filesystem, little endian, version 4.0, compression:lzma, size: 2588426 bytes, 375 inodes, blocksize: 16384 bytes, created: 2016-02-05 02:05:56
root#kali:~/ROUTER# dd if=new-firmware.bin of=uboot.lzma skip=148 bs=1
3735488+0 records in
3735488+0 records out
3735488 bytes (3.7 MB, 3.6 MiB) copied, 4.16712 s, 896 kB/s
root#kali:~/ROUTER# dd if=new-firmware.bin of=kernel.squash skip=917588 bs=1
2818048+0 records in
2818048+0 records out
2818048 bytes (2.8 MB, 2.7 MiB) copied, 3.46517 s, 813 kB/s
All looks to have worked so far but then I try the following:
root#kali:~/ROUTER# lzma -d uboot.lzma
lzma: uboot.lzma: Compressed data is corrupt
root#kali:~/ROUTER# unsquashfs kernel.squash
Parallel unsquashfs: Using 4 processors
lzma uncompress failed with error code 0
read_block: failed to read block #0x277af0
read_fragment_table: failed to read fragment table index
FATAL ERROR:failed to read fragment table
Any idea why it may be happening? As the prompt says, I am on Kali Linux. ALso get this
jj#ubuntu:~/Firmware$ binwalk -e new-firmware.bin
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
84 0x54 uImage header, header size: 64 bytes, header CRC: 0xE52A7F50, created: 2012-02-10 07:27:12, image size: 819799 bytes, Data Address: 0x80002000, Entry Point: 0x801AC9F0, data CRC: 0x6A10D412, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: lzma, image name: "Linux Kernel Image"
148 0x94 LZMA compressed data, properties: 0x5D, dictionary size: 8388608 bytes, uncompressed size: 2386252 bytes
WARNING: Extractor.execute failed to run external extractor 'sasquatch -p 1 -le -d '%%squashfs-root%%' '%e'': [Errno 2] No such file or directory
WARNING: Extractor.execute failed to run external extractor 'sasquatch -p 1 -be -d '%%squashfs-root%%' '%e'': [Errno 2] No such file or directory
917588 0xE0054 Squashfs filesystem, little endian, version 4.0, compression:lzma, size: 2588426 bytes, 375 inodes, blocksize: 16384 bytes, created: 2016-02-05 02:05:56
Try
lzma -d < uboot.lzma > uboot
The error you found is due to LZMA Utils having been superseded by XV Utils, which don't ignore the trailing garbage. If you use it as a stream (like I gave you), you'll still have the corruption error message in the end, but you'll get your file alright.
Source (search for "corrupt" and you'll find other related comments)
Try
binwalk -e new-firmware.bin
you will end up with 2 file one being extracted
In the extracted one you will get 2 one unzippped and a zipped one
you can use dd at this point and unlzma should work at least it did work out for me
this is what i mean
root#ROOT20:~/Desktop/project/dlink/dlink2# binwalk dcs932l_v1.14.04.bin
DECIMAL HEXADECIMAL DESCRIPTION
106352 0x19F70 U-Boot version string, "U-Boot 1.1.3"
106816 0x1A140 CRC32 polynomial table, little endian
124544 0x1E680 HTML document header
124890 0x1E7DA HTML document footer
124900 0x1E7E4 HTML document header
125092 0x1E8A4 HTML document footer
125260 0x1E94C HTML document header
125953 0x1EC01 HTML document footer
327680 0x50000 uImage header, header size: 64 bytes, header CRC: 0x88345E96, created: 2016-09-09 13:52:27, image size: 3804958 bytes, Data Address: 0x80000000, Entry Point: 0x803B8000, data CRC: 0x531E94DE, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: lzma, image name: "Linux Kernel Image"
327744 0x50040 LZMA compressed data, properties: 0x5D, dictionary size: 33554432 bytes, uncompressed size: 6558763 bytes
root#ROOT20:~/Desktop/project/dlink/dlink2# dd if=dcs932l_v1.14.04.bin skip=327744 bs=1 of=kernel.lzma
3866560+0 records in
3866560+0 records out
3866560 bytes (3.9 MB, 3.7 MiB) copied, 9.61104 s, 402 kB/s
root#ROOT20:~/Desktop/project/dlink/dlink2# unlzma kernel.lzma
unlzma: kernel.lzma: Compressed data is corrupt
root#ROOT20:~/Desktop/project/dlink/dlink2# binwalk -e dcs932l_v1.14.04.bin
DECIMAL HEXADECIMAL DESCRIPTION
106352 0x19F70 U-Boot version string, "U-Boot 1.1.3"
106816 0x1A140 CRC32 polynomial table, little endian
124544 0x1E680 HTML document header
124890 0x1E7DA HTML document footer
124900 0x1E7E4 HTML document header
125092 0x1E8A4 HTML document footer
125260 0x1E94C HTML document header
125953 0x1EC01 HTML document footer
327680 0x50000 uImage header, header size: 64 bytes, header CRC: 0x88345E96, created: 2016-09-09 13:52:27, image size: 3804958 bytes, Data Address: 0x80000000, Entry Point: 0x803B8000, data CRC: 0x531E94DE, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: lzma, image name: "Linux Kernel Image"
327744 0x50040 LZMA compressed data, properties: 0x5D, dictionary size: 33554432 bytes, uncompressed size: 6558763 bytes
root#ROOT20:~/Desktop/project/dlink/dlink2# ls
dcs932l_v1.14.04.bin _dcs932l_v1.14.04.bin.extracted kernel.lzma
root#ROOT20:~/Desktop/project/dlink/dlink2# cd _dcs932l_v1.14.04.bin.extracted/
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted# ls
50040 50040.7z
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted# binwalk 50040
DECIMAL HEXADECIMAL DESCRIPTION
3145804 0x30004C Linux kernel version 2.6.21
3175792 0x307570 SHA256 hash constants, little endian
3389960 0x33BA08 Unix path: /usr/gnemul/irix/
3392244 0x33C2F4 Unix path: /usr/lib/libc.so.1
3393940 0x33C994 Unix path: /dev/vc/0
3408260 0x340184 Copyright string: "Copyright (c) 2010 Alpha Networks Inc."
3491536 0x3546D0 Unix path: /etc/Wireless/RT2860STA/RT2860STA.dat
3573187 0x3685C3 Neighborly text, "neighbor %.2x%.2x.%.2x:%.2x:%.2x:%.2x:%.2x:%.2x lost on port %d(%s)(%s)"
3807776 0x3A1A20 CRC32 polynomial table, little endian
4038656 0x3DA000 LZMA compressed data, properties: 0x5D, dictionary size: 1048576 bytes, uncompressed size: 8072704 bytes
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted# dd if=50040 skip=4038656 bs=1 of=kernel2.lzma
2520107+0 records in
2520107+0 records out
2520107 bytes (2.5 MB, 2.4 MiB) copied, 8.23342 s, 306 kB/s
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted# unlzma kernel2.lzma
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted#
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted# mkdir cpio; cd cpio
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted/cpio# cpio -idm --no-absolute-filenames < ../kernel2
cpio: Removing leading `/' from member names
15767 blocks
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted/cpio# ls
bin dev etc etc_ro home init lib media mnt mydlink proc sbin sys tmp usr var

Multiple string tables in ELF object

From ELF Documentation:
SHT_STRTAB
The section holds a string table. An object file may have multiple string table sections. See ‘‘String Table’’ below for details.
(Note: I didn't notice any information regarding multiple string table sections in ‘‘String Table’’ paragraph)
Does multiple string table sections mean string table for the section headers and string table for the object file itself?
There is no mention in the documentation regarding how to read a string if there are multiple string table for the object itself (.strtab).
Any clarification about the subject appreciated.
The manpage only has a overview/summary of (some parts of) the ELF file format, you might want to look at the System V ABI spec.
Explanation
Linking view
An ELF file has multiple string tables. Usually you have 3-4 string tables:
One string table (usually called .shstrtab) is used for section names. All section names (in the section header table) are taken from a single string table. This string table is identified by its index in the section header table: the index of the section name string table is indicated in the ELF header (e_shstrndx).
Another string table (usually called .strtab) is used for the full symbol table (.symtab). The same string table is used by the .dynamic section.
Another string table (usually called .dynstr) is used for the minirmal symbol table (.dynsym).
Another string table is used for
For a given symbol table section, the section used as string table is indicated in the sh_link field of the section header table (see Fig 4-12 of the system V ABI spec).
Execution view
For the execution view (the program header table), the address of the string table used for the symbol table (DT_SYMTAB) is given in the DT_STRTAB entry of the dynamic section.
Example
Linking view
This is a hello world program (shown with readelf -a).
.shtrtab
The ELF header:
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: EXEC (Executable file)
Machine: Advanced Micro Devices X86-64
Version: 0x1
Entry point address: 0x4003c0
Start of program headers: 64 (bytes into file)
Start of section headers: 4624 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 56 (bytes)
Number of program headers: 8
Size of section headers: 64 (bytes)
Number of section headers: 30
Section header string table index: 27
tells us that the section names are in section 27. Conveniently enough this is .shtrtab:
Section Headers:
[Nr] Name Type Address Offset
Size EntSize Flags Link Info Align
[...]
[27] .shstrtab STRTAB 0000000000000000 000008e0
0000000000000108 0000000000000000 0 0 1
.dynstr
For .dynsym we have:
[ 5] .dynsym DYNSYM 0000000000400280 00000280
0000000000000048 0000000000000018 A 6 1 8
^
HERE
Its names are taken from section 6 which is .dynstr:
[ 6] .dynstr STRTAB 00000000004002c8 000002c8
0000000000000038 0000000000000000 A 0 0 1
This string table is used by other sections as well:
[ 8] .gnu.version_r VERNEED 0000000000400308 00000308
0000000000000020 0000000000000000 A 6 1 8
[21] .dynamic DYNAMIC 0000000000600698 00000698
00000000000001d0 0000000000000010 WA 6 0 8
.strtab
For .symtab:
[28] .symtab SYMTAB 0000000000000000 000009e8
0000000000000600 0000000000000018 29 45 8
^
HERE
the names are taken from section 29 which happens to be .strtab:
[29] .strtab STRTAB 0000000000000000 00000fe8
0000000000000224 0000000000000000 0 0 1
Execution view
Dynamic section at offset 0x698 contains 24 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x000000000000000c (INIT) 0x400370
0x000000000000000d (FINI) 0x400544
0x0000000000000019 (INIT_ARRAY) 0x600680
0x000000000000001b (INIT_ARRAYSZ) 8 (bytes)
0x000000000000001a (FINI_ARRAY) 0x600688
0x000000000000001c (FINI_ARRAYSZ) 8 (bytes)
0x000000006ffffef5 (GNU_HASH) 0x400260
0x0000000000000005 (STRTAB) 0x4002c8 <= HERE
0x0000000000000006 (SYMTAB) 0x400280
0x000000000000000a (STRSZ) 56 (bytes)
0x000000000000000b (SYMENT) 24 (bytes)
0x0000000000000015 (DEBUG) 0x0
0x0000000000000003 (PLTGOT) 0x600870
0x0000000000000002 (PLTRELSZ) 48 (bytes)
0x0000000000000014 (PLTREL) RELA
0x0000000000000017 (JMPREL) 0x400340
0x0000000000000007 (RELA) 0x400328
0x0000000000000008 (RELASZ) 24 (bytes)
0x0000000000000009 (RELAENT) 24 (bytes)
0x000000006ffffffe (VERNEED) 0x400308
0x000000006fffffff (VERNEEDNUM) 1
0x000000006ffffff0 (VERSYM) 0x400300
0x0000000000000000 (NULL) 0x0
The string table for dynamic linking is located at 0x4002c8 in the program memory.
Note: this is .dynstr.