What is the right format of query argument of mongoexport utility?
While running the following command in the command line:
mongoexport -h localhost:27000 -d dbName -c collName -q "{'time': { $gt: new Date('2014-01-28T12:00:00Z')}}" -o output.js
I'm getting the following error:
connected to: localhost:27000 assertion: 16619 code FailedToParse:
FailedToParse: Expecting '}' or ',': offset:37
Reading Mongo Export query arg and JSONDocument docs haven't helped me to understand the expected format of query argument.
Running the same query in mongo shell succeeds.
If:
>new Date ("2014-01-28T12:00:00Z").getTime()
1390910400000
You will have to construct your query as follows:
-q "{sendToServerTime: {\$gt: {\$date : 1390910400000}}}"
The problem is your new Date() command. This no valid json. Try this:
mongoexport -h localhost:27000 -d DeploymentJan01 -c sensorsData -q '{sendToServerTime: { $gt: "2014-01-28T12:00:00Z"}}' -o output.js
Related
I'm trying to produce Kafka message with kcat tool. I have a message that follows topic schema:
{
"meta": {
"correlationId": "244c7ed3-0472-496e-bed9-7071f8ddb921",
"payload": {
"id": "3ae843e9-0c96-4ee8-b1f0-fbdfef30e9dd",
"timestamp": 1658846522
}
}
}
store it to the file k-msg-test.json and run kcat:
kcat \
-b localhost:9092 \
-r localhost:8081 \
-t test-topic \
-T \
-P \
/tmp/k-msg-test.json
and I get an error:
% Failed to open : No such file or directory
/tmp/k-msg-test.json: line 2: meta:: command not found
/tmp/k-msg-test.json: line 3: correlationId:: command not found
/tmp/k-msg-test.json: line 4: payload:: command not found
/tmp/k-msg-test.json: line 5: id:: command not found
/tmp/k-msg-test.json: line 6: timestamp:: command not found
/tmp/k-msg-test.json: line 8: syntax error near unexpected token `}'
/tmp/k-msg-test.json: line 8: ` }'
So the question is – how can I produce a message?
Your file is apparently being interpreted as literal text to your shell rather than read by kcat.
You need to use shell redirection as kcat accepts stdin, rather than file arguments.
You'll also need to remove the new lines and produce as a single line. You can do that with jq
kcat \
-b localhost:9092 \
-r localhost:8081 \
-t test-topic \
-T \
-P < $(jq -rc '.' /tmp/k-msg-test.json)
I tried to extract data from below cbq command which was successful.
cbq -u Administrator -p Administrator -e "http://localhost:8093" --script= SELECT * FROM `sample` where customer.id=="12345'" -q | jq '.results' > temp.json;
However when I am trying to import the same data in json format to target cluster using below command I am getting error.
cbimport json -c http://{target-cluster}:8091 -u Administrator -p Administrator -b sample -d file://C:\Users\{myusername}\Desktop\temp.json -f list -g %docId%
JSON import failed: 0 documents were imported, 0 documents failed to be imported
JSON import failed: input json is invalid: ReadArray: expect [ or , or ] or n, but found {, error found in #1 byte of ...|{
"requ|..., bigger context ...|{
"requestID": "2fc34542-4387-4643-8ae3-914e316|...],```
```{
"requestID": "6ef38b8a-8e70-4c3d-b3b4-b73518a09c62",
"signature": {
"*": "*"
},
"results": [
{
"{Bucket-name}":{my-data}
"status": "success",
"metrics": {
"elapsedTime": "4.517031ms",
"executionTime": "4.365976ms",
"resultCount": 1,
"resultSize": 24926
}
It looks like the file which was extracted from cbq command has control fields details like RequestID, metrics, status etc. Also json in pretty format. If I manually remove it(remove all fields except {my-data}) then put in a json file and make json unpretty then it works. But I want to automate it in a single run. Is there a way to do it in cbq command.
I don't find any other utility or way to use where condition on cbexport to do that on Couchbase, because the document which are exported using cbexport can be imported using cbimport easily.
For the cbq command, you can use the --quiet option to disable the startup connection messages and the --pretty=false to disable pretty-print. Then, to extract just the documents in cbimport json lines format, I used jq.
This worked for me -- selecting documents from travel-sample._default._default (for the jq filter, where I have _default, you would put the Bucket-name, based on your example):
cbq --quiet --pretty=false -u Administrator -p password --script='select * from `travel-sample`._default._default' | jq --compact-output '.results|.[]|._default' > docs.json
Then, importing into test-bucket1:
cbimport json -c localhost -u Administrator -p password -b test-bucket1 -d file://./docs.json -f lines -g %type%_%id%
cbq documentation: https://docs.couchbase.com/server/current/tools/cbq-shell.html
cbimport documentation: https://docs.couchbase.com/server/current/tools/cbimport-json.html
jq documentation:
https://stedolan.github.io/jq/manual/#Basicfilters
I'm trying to post a trace to a remote machine by giving the trace path and name as parameters in the curl command like this:
sshpass -p password ssh username#ip_adress "curl -X POST -d '{"parameters":{"name":"trace","uri":"/usagers4/username/Documents/Bench_1/kernel"}}' http://0.0.0.0:8080/tsp/api/traces -H Accept:application/json -H Content-Type:application/json"
But I'm getting the following error:
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('p' (code 112)): was expecting double-quote to start field name| at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 3]
Does anyone have an idea about what I'm doing wrong ?
Thanks
I Solved!
i'm need to get out for python use os.system(Curl ... )
it command return code 112 double quote to start field name
i'm use os.system("curl -d '{\"agr\" : value}' -H 'value' -H 'value' ... etc.")
you should use the following code
"curl -X POST -d '{\"parameters\":{\"name\":\"trace\",\"uri\":\"/usagers4/username/Documents/Bench_1/kernel\"}}' http://0.0.0.0:8080/tsp/api/traces -H 'Accept:application/json' -H 'Content-Type:application/json'"
this is the query:
mongoexport --host our.dbhost.com --port 27017 --username peter -p clark --collection sent_mails --db dbname --query '{trigger_id:ObjectId( "50c62e97b9fe6a000200000c"), updated_at: {$lt : ISODate("2013-02-28"), $gte : ISODate("2013-02-01") }}'
when I run this command I get:
assertion: 10340 Failure parsing JSON string near: , updated_
any ideas? (i want all records that match the trigger_id that were updated in February.)
as explained in this issue: Mongoexport using $gt and $lt constraints on a date range, you have to use Unix time stamps for date queries in mongoexport
The time stamps have to be in milliseconds
Invoking this in the bash shell would look like this:
let "date_from=`date --utc --date "2013-02-01" +%s` * 1000"
let "date_to=`date --utc --date "2013-03-01" +%s` * 1000"
mongoexport -d test -c xx --query "{updated_at:{\$gte:new Date($date_from),\$lt:new Date($date_to)}}"> xx.json
> connected to: 127.0.0.1
> exported 1 records
The xx colletion contains:
> db.xx.find().pretty()
{
"_id" : ObjectId("5158f670c2293fc7aadd811e"),
"trigger_id" : ObjectId("50c62e97b9fe6a000200000c"),
"updated_at" : ISODate("2013-02-11T00:00:00Z")
}
I want to use JSON to batch upsert to a mongo collection.
$ mongoexport -d myDB -c myCollection
connected to: 127.0.0.1
{ "_id" : "john", "age" : 27 }
But using the syntax I would in the mongo shell yields:
0$ echo '{_id:"john", {$set:{gender:"male"}}' | mongoimport --upsert --upsertFields _id -d myDB -c myCollection
connected to: 127.0.0.1
Fri Jul 27 15:01:32 Assertion: 10340:Failure parsing JSON string near: , {$set:{g
0x581a52 0x528554 0xa9f2e3 0xaa1593 0xa980cd 0xa9c062 0x3e7ca1ec5d 0x4fe239
...
/lib64/libc.so.6(__libc_start_main+0xfd) [0x3e7ca1ec5d] mongoimport(__gxx_personality_v0+0x3c9) [0x4fe239]
exception:Failure parsing JSON string near: , {$set:{g
imported 0 objects
encountered 1 error
When I try it without the curly brackets, it yields no error but doesn't change the table:
0$ echo '{_id:"john", $set:{gender:"male"}}' | mongoimport --upsert --upsertFields _id -d myDB -c myCollection
connected to: 127.0.0.1
imported 1 objects
0$ mongoexport -d myDB -c myCollection
connected to: 127.0.0.1
{ "_id" : "john", "age" : 27 }
exported 1 records
I've searched everywhere but can't find an example using JSON. Please help!
To the best of my knowledge, MongoImport doesn't evaluate commands.
Just to add to Andre's answer.
Mongoimport takes a single file that contains 1 JSON/CSV/TSV string per line and inserts it. You can import from standard out but not as a command as above. You can use mongoimport to perform an upsert as per here.
You can run mongoimport with the stoponError option, which will force mongoimport to stop when it encounters an error.
Here's the complete manual for mongoimport and, as a FYI, mongoimport doesn't reliably preserve all rich BSON data types.
Mongoimport does not take modifiers such as your $set. You will need to use the mongo --eval command to update.
mongo myDB --eval 'db.myCollection.update({_id: "john"}, {$set:{gender:"male"}}, upsert=true)'
Hope this helps.