Task : Importing a json to a local couchbase DB. This is from the tutorial CB110 on learn.couchbase.com.
Issue : Only the last row, from thousands of other json rows, gets imported.
Command :
$cbimport json -c couchbase://127.0.0.1 -u Administrator -p abcd -b couchmusic2 -f lines -d file://C:/Users/Deep_Kulshreshtha/Downloads/CB110-Data/couchmusic2-userprofiles.json -t 1 -g %type%::%username%
Result :
Json file://C:/Users/Deep_Kulshreshtha/Downloads/CB110-Data/couchmusic2-userprofiles.json imported to http://127.0.0.1:8091 successfully
Couchbase Admin Screen :
You can see only a single row is imported. This is the last row from the import document, which contains 50,000 records.
Any pointers or help is much appreciated ! Thanks.
Deep
Got the solution.
Windows users : Please use ^ as escape character in your key generation parameters.
Following worked for me:
-g %type%::^%username^%
Thanks.
Related
Just a quick question to solve an issue I've been facing for days now: how to get an wget json response in a shell variable?
I have so far a wget command like this:
wget "http://IP:PORT/webapi/auth.cgi?account=USER&passwd=PASSWD"
The server reponse is normally something like:
{"data":{"sid":"9O4leaoASc0wgB3J4N01003"},"success":true}
What I'd like to do is to grep the sid value in a variable (as it is used as login ticket), but also the success value in order to ensure that the command has been executed correctly...
I think it is a very easy command to build, but I've never practised wget/http reponse in shell command...
Thanks a lot for your help!
EDIT: Thanks for your help. I did gave a try to both answers, but I am having the same error message (whatever I do):
--2022-07-16 14:21:38-- http://xxxxxxxx:port/webapi/auth.cgi?api=SYNO.API.Auth&method=Login&version=3&account=USER&passwd=PWD&session=SurveillanceStation&format=sid
Connecting to 192.168.1.100:5000... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
auth.cgi?api=SYNO.API.Auth&method=Login&version=3&account=USER&passwd=PASSWD&session=SurveillanceStation&format=sid: Permission denied
Cannot write to `auth.cgi?api=SYNO.API.Auth&method=Login&version=3&account=USER&passwd=PASSWD&session=SurveillanceStation&format=sid' (Permission denied).
The annoying thing: execution the URL from a web browser works just fine... :/
You can first store the result of wget command in variable and then use it:
VAR=$(wget "http://IP:PORT/webapi/auth.cgi?account=USER&passwd=PASSWD")
and then using jq extract from JSON file:
sid=$(echo $VAR|jq .data.sid)
success=$(echo $VAR|jq .success)
If you have problem with execution of wget you can try something like:
wget -O output_file 'http://xxxxxxxx:port/webapi/auth.cgi?api=SYNO.API.Auth&method=Login&version=3&account=USER&passwd=PWD&session=SurveillanceStation&format=sid'
and then set variables:
sid=$(jq .data.sid output_file )
success=$(jq .success output_file )
I do not know why I am facing this Permission Denied error. Thus I gave a try to save cookie on a dedicated folder... And it works just fine :)
The final command lloks like:
VAR=$(wget -q --keep-session-cookies --save-cookies "/var/tmp/cookie_tmp" -O- "http://IP:PORT/webapi/auth.cgi?api=SYNO.API.Auth&method=login&version=1&account=USER&passwd=PWD&session=SurveillanceStation");
Thanks for your help (I learned a lot about sed ;) )
So this can be done using the stream editor or "sed". There is a lot to learn but for this post here is an idea of a code:
sid=$(wget <your url> | sed 's/.*sid":"\(.*\)"},.*/\1/')
success=$(wget <your url> | sed 's/.*success":\(.*\)}/\1/')
This will create 2 variables $sid and $success.
you can learn more about sed in depth here.
Hope this helped!
When I run the following command:
mongoimport -v -d ntsb -c data xml_results.json --jsonArray
I get this error:
2020-07-15T22:51:41.267-0400 using write concern: &{majority false 0}
2020-07-15T22:51:41.270-0400 filesize: 68564556 bytes
2020-07-15T22:51:41.270-0400 using fields:
2020-07-15T22:51:41.270-0400 connected to: mongodb://localhost/
2020-07-15T22:51:41.270-0400 ns: ntsb.data
2020-07-15T22:51:41.271-0400 connected to node type: standalone
2020-07-15T22:51:41.271-0400 Failed: error processing document #1: invalid character '}' looking for beginning of object key string
2020-07-15T22:51:41.271-0400 0 document(s) imported successfully. 0 document(s) failed to import.
I have tried all the solutions in this file and nothing worked. My JSON file is 60ish MB in size so it would be really hard to go through it and find the bracket issue. I believe that it is a problem with the UTF-8 formatting maybe? I take an XML file I downloaded on the internet and convert it into JSON with a Python script. When I try the --jsonArray flag, it gives the same error. Any ideas? Thanks!
It turns out within this massive file there were a few unnecessary commas. I was able to use Pythons built in JSON parsing to jump to lines with errors and remove them manually. As far as I can tell, the invalid character had nothing to do with the } but with the comma that caused it to expect another value before the closing bracket.
After solving this, I was still unable to import successfully because now the file was too large. The trick around this was to surround all the JSON objects with array brackets [] and use the following command: mongoimport -v -d ntsb -c data xml_results.json --batchSize 1 --jsonArray
After a few seconds the data imported successfully into Mongo.
I have a cypher script file and I would like to run it directly.
All answers I could find on SO to the best of my knowledge use the command neo4j-shell which in my version (Neo4j server 3.5.5) seems to be deprecated and substituted with the command cyphershell.
Using the command sudo ./neo4j-community-3.5.5/bin/cypher-shell --help I got the following instructions.
usage: cypher-shell [-h] [-a ADDRESS] [-u USERNAME] [-p PASSWORD]
[--encryption {true,false}]
[--format {auto,verbose,plain}] [--debug] [--non-interactive] [--sample-rows SAMPLE-ROWS]
[--wrap {true,false}] [-v] [--driver-version] [--fail-fast | --fail-at-end] [cypher]
A command line shell where you can execute Cypher against an
instance of Neo4j. By default the shell is interactive but you can
use it for scripting by passing cypher directly on the command
line or by piping a file with cypher statements (requires Powershell
on Windows).
My file is the following which tries to create a graph from csv files and it comes from the book "Graph Algorithms".
WITH "https://github.com/neo4j-graph-analytics/book/raw/master/data" AS base
WITH base + "transport-nodes.csv" AS uri
LOAD CSV WITH HEADERS FROM uri AS row
MERGE (place:Place {id:row.id})
SET place.latitude = toFloat(row.latitude),
place.longitude = toFloat(row.latitude),
place.population = toInteger(row.population)
WITH "https://github.com/neo4j-graph-analytics/book/raw/master/data/" AS base
WITH base + "transport-relationships.csv" AS uri
LOAD CSV WITH HEADERS FROM uri AS row
MATCH (origin:Place {id: row.src})
MATCH (destination:Place {id: row.dst})
MERGE (origin)-[:EROAD {distance: toInteger(row.cost)}]->(destination)
When I try to pass the file directly with the command:
sudo ./neo4j-community-3.5.5/bin/cypher-shell neo_4.cypher
first it asks for username and password but after typing the correct password (the wrong password results in the error The client is unauthorized due to authentication failure.) I get the error:
Invalid input 'n': expected <init> (line 1, column 1 (offset: 0))
"neo_4.cypher"
^
When I try piping with the command:
sudo cat neo_4.cypher| sudo ./neo4j-community-3.5.5/bin/cypher-shell -u usr -p 'pwd'
no output is generated and no graph either.
How to run a cypher script file with the neo4j command cypher-shell?
Use cypher-shell -f yourscriptname. Check with --help for more description.
I think the key is here:
cypher-shell -- help
... Stuff deleted
positional arguments:
cypher an optional string of cypher to execute and then exit
This means that the paremeter is actual cypher code, not a file name. Thus, this works:
GMc#linux-ihon:~> cypher-shell "match(n) return n;"
username: neo4j
password: ****
+-----------------------------+
| n |
+-----------------------------+
| (:Job {jobName: "Job01"}) |
| (:Job {jobName: "Job02"}) |
But this doesn't (because the text "neo_4.cypher" isn't a valid cypher query)
cypher-shell neo_4.cypher
The help also says:
example of piping a file:
cat some-cypher.txt | cypher-shell
So:
cat neo_4.cypher | cypher-shell
should work. Possibly your problem is all of the sudo's. Specifically the cat ... | sudo cypher-shell. It is possible that sudo is protecting cypher-shell from some arbitrary input (although it doesn't seem to do so on my system).
If you really need to use sudo to run cypher, try using the following:
sudo cypher-shell arguments_as_needed < neo_4.cypher
Oh, also, your script doesn't have a return, so it probably won't display any data, but you should still see the summary reports of records loaded.
Perhaps try something simpler first such as a simple match ... return ... query in your script.
Oh, and don't forget to terminate the cypher query with a semi-colon!
The problem is in the cypher file: each line should end with a semicolon: ;. I still need sudo to run the program.
The file taken from the book seems to contain other errors as well actually.
I'm trying to import some GeoJson data into MongoDB. The entire file is about 24MB, so in theory the per-document limit of 16MB shouldn't be exceeded. But it looks like it's complaining about the size. I have tried solutions offered here, but none seems to work. I type the command:
mongoimport -d userdata -c countries < countries.geojson
and I get
2017-11-17T01:09:29.561+0400 connected to: localhost
2017-11-17T01:09:31.055+0400 num failures: 1
2017-11-17T01:09:31.055+0400 Failed: lost connection to server
2017-11-17T01:09:31.055+0400 imported 0 documents
and the mongod logs show (after backtrace):
2017-11-17T01:09:31.055+0400 I - [conn153] AssertionException handling request, closing client connection: 10334 BSONObj size: 17756597 (0x10EF1B5) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "countries"
2017-11-17T01:09:31.055+0400 I - [conn153] end connection 127.0.0.1:61806 (2 connections now open)
I have tried
mongoimport -d userdata -c countries < countries.geojson --batchSize 1
and
mongoimport -d userdata -c countries -j 4 < countries.geojson
based on other similar answers but got the same result, with the same response and logs.
Anyone have clues as to what's going on here? Should I break the GeoJson into two and give that a shot? I thought the 16MB limit was on individual documents, not collections or collection imports.
I am using orabbix to monitor my db. The data from the queries executed on this db using orabbix are sent to zabbix server. However, I am not able to see the data reaching zabbix.
On my zabbix web console, I see this message on the triggers added - "Trigger expression updated. No status update so far."
Any ideas?
My update interval for the trigger is set to 30 sec.
Based on the screenshots you posted, your host is named "wfc1dev1" and you have items with keys "WFC_WFS_SYS_001" and "WFC_WFS_SYS_002". However, based on the Orabbix XML that it sends to Zabbix, the hostname and item keys are different. Here is the XML:
<req><host>V0ZDMURFVg==</host><key>V0ZDX0xFQUZfU1lTXzAwMg==</key><data>MA==</data></req>
From this, we can deduce the host:
$ echo V0ZDMURFVg== | base64 -d
WFC1DEV
The key:
$ echo V0ZDX0xFQUZfU1lTXzAwMg== | base64 -d
WFC_LEAF_SYS_002
The data:
$ echo MA== | base64 -d
0
It can be seen that neither the host name, nor item key match those configured on Zabbix server. Once you fix that, it should work.