I want to use JSON to batch upsert to a mongo collection.
$ mongoexport -d myDB -c myCollection
connected to: 127.0.0.1
{ "_id" : "john", "age" : 27 }
But using the syntax I would in the mongo shell yields:
0$ echo '{_id:"john", {$set:{gender:"male"}}' | mongoimport --upsert --upsertFields _id -d myDB -c myCollection
connected to: 127.0.0.1
Fri Jul 27 15:01:32 Assertion: 10340:Failure parsing JSON string near: , {$set:{g
0x581a52 0x528554 0xa9f2e3 0xaa1593 0xa980cd 0xa9c062 0x3e7ca1ec5d 0x4fe239
...
/lib64/libc.so.6(__libc_start_main+0xfd) [0x3e7ca1ec5d] mongoimport(__gxx_personality_v0+0x3c9) [0x4fe239]
exception:Failure parsing JSON string near: , {$set:{g
imported 0 objects
encountered 1 error
When I try it without the curly brackets, it yields no error but doesn't change the table:
0$ echo '{_id:"john", $set:{gender:"male"}}' | mongoimport --upsert --upsertFields _id -d myDB -c myCollection
connected to: 127.0.0.1
imported 1 objects
0$ mongoexport -d myDB -c myCollection
connected to: 127.0.0.1
{ "_id" : "john", "age" : 27 }
exported 1 records
I've searched everywhere but can't find an example using JSON. Please help!
To the best of my knowledge, MongoImport doesn't evaluate commands.
Just to add to Andre's answer.
Mongoimport takes a single file that contains 1 JSON/CSV/TSV string per line and inserts it. You can import from standard out but not as a command as above. You can use mongoimport to perform an upsert as per here.
You can run mongoimport with the stoponError option, which will force mongoimport to stop when it encounters an error.
Here's the complete manual for mongoimport and, as a FYI, mongoimport doesn't reliably preserve all rich BSON data types.
Mongoimport does not take modifiers such as your $set. You will need to use the mongo --eval command to update.
mongo myDB --eval 'db.myCollection.update({_id: "john"}, {$set:{gender:"male"}}, upsert=true)'
Hope this helps.
Related
I tried to extract data from below cbq command which was successful.
cbq -u Administrator -p Administrator -e "http://localhost:8093" --script= SELECT * FROM `sample` where customer.id=="12345'" -q | jq '.results' > temp.json;
However when I am trying to import the same data in json format to target cluster using below command I am getting error.
cbimport json -c http://{target-cluster}:8091 -u Administrator -p Administrator -b sample -d file://C:\Users\{myusername}\Desktop\temp.json -f list -g %docId%
JSON import failed: 0 documents were imported, 0 documents failed to be imported
JSON import failed: input json is invalid: ReadArray: expect [ or , or ] or n, but found {, error found in #1 byte of ...|{
"requ|..., bigger context ...|{
"requestID": "2fc34542-4387-4643-8ae3-914e316|...],```
```{
"requestID": "6ef38b8a-8e70-4c3d-b3b4-b73518a09c62",
"signature": {
"*": "*"
},
"results": [
{
"{Bucket-name}":{my-data}
"status": "success",
"metrics": {
"elapsedTime": "4.517031ms",
"executionTime": "4.365976ms",
"resultCount": 1,
"resultSize": 24926
}
It looks like the file which was extracted from cbq command has control fields details like RequestID, metrics, status etc. Also json in pretty format. If I manually remove it(remove all fields except {my-data}) then put in a json file and make json unpretty then it works. But I want to automate it in a single run. Is there a way to do it in cbq command.
I don't find any other utility or way to use where condition on cbexport to do that on Couchbase, because the document which are exported using cbexport can be imported using cbimport easily.
For the cbq command, you can use the --quiet option to disable the startup connection messages and the --pretty=false to disable pretty-print. Then, to extract just the documents in cbimport json lines format, I used jq.
This worked for me -- selecting documents from travel-sample._default._default (for the jq filter, where I have _default, you would put the Bucket-name, based on your example):
cbq --quiet --pretty=false -u Administrator -p password --script='select * from `travel-sample`._default._default' | jq --compact-output '.results|.[]|._default' > docs.json
Then, importing into test-bucket1:
cbimport json -c localhost -u Administrator -p password -b test-bucket1 -d file://./docs.json -f lines -g %type%_%id%
cbq documentation: https://docs.couchbase.com/server/current/tools/cbq-shell.html
cbimport documentation: https://docs.couchbase.com/server/current/tools/cbimport-json.html
jq documentation:
https://stedolan.github.io/jq/manual/#Basicfilters
I want to manually add some data into a new mongodb. For this I write the json files and run mongoimport for importing.
This is an example of an original file:
{
"val": {}
}
Yet, in the new database, the value {} is turned into null:
{
"_id": ObjectId("6023b9a532d713e97f5dc70c"),
"val": null
}
I don't understand why this is happening. Is there a way to prevent this?
Due to some restrictions I have to use the --legacy flag on mongoimport.
My versions are:
$ mongoimport --version
mongoimport version: 100.2.0
$ mongod --version
db version v4.4.1
I've been given an odd requirement to store an Excel spreadsheet in one JSON document within Couchbase. cbimport is saying that my document is not valid JSON, when it is, so I believe something else is wrong.
My document goes along the style of this:
[{ "sets": [
{
"cluster" : "M1M",
"type" : "SET",
"shortName" : "MARTIN MARIETTA MATERIALS",
"clusterName" : "MARTIN MARIETTA",
"setNum" : "10000163"
},
{
"shortName" : "STERLING INC",
"type" : "SET",
"cluster" : "SJW",
"setNum" : "10001427",
"clusterName" : "STERLING JEWELERS"
},
...
]}]
And my cbimport command looks like this:
cbimport json --cluster localhost --bucket documentBucket \
--dataset file://set_numbers.json --username Administrator \
--password password --format lines -e errors.log -l debug.log \
--generate-key 1
I've tried to format as lines as well as list. Both fail. What am I doing wrong?
I wrote your sample to a json file called set_numbers.json and tried it locally with list.
cbimport json --cluster localhost --bucket documentBucket --dataset
file://set_numbers.json --username Administrator --password password
--format list --generate-key 1
It imported successfully into a single document.
use cbimport to upload json data
cbimport json -c couchbase://127.0.0.1 -b data -d file://data.json -u Administrator -p password -f list -g "%id%" -t 4
What is the right format of query argument of mongoexport utility?
While running the following command in the command line:
mongoexport -h localhost:27000 -d dbName -c collName -q "{'time': { $gt: new Date('2014-01-28T12:00:00Z')}}" -o output.js
I'm getting the following error:
connected to: localhost:27000 assertion: 16619 code FailedToParse:
FailedToParse: Expecting '}' or ',': offset:37
Reading Mongo Export query arg and JSONDocument docs haven't helped me to understand the expected format of query argument.
Running the same query in mongo shell succeeds.
If:
>new Date ("2014-01-28T12:00:00Z").getTime()
1390910400000
You will have to construct your query as follows:
-q "{sendToServerTime: {\$gt: {\$date : 1390910400000}}}"
The problem is your new Date() command. This no valid json. Try this:
mongoexport -h localhost:27000 -d DeploymentJan01 -c sensorsData -q '{sendToServerTime: { $gt: "2014-01-28T12:00:00Z"}}' -o output.js
this is the query:
mongoexport --host our.dbhost.com --port 27017 --username peter -p clark --collection sent_mails --db dbname --query '{trigger_id:ObjectId( "50c62e97b9fe6a000200000c"), updated_at: {$lt : ISODate("2013-02-28"), $gte : ISODate("2013-02-01") }}'
when I run this command I get:
assertion: 10340 Failure parsing JSON string near: , updated_
any ideas? (i want all records that match the trigger_id that were updated in February.)
as explained in this issue: Mongoexport using $gt and $lt constraints on a date range, you have to use Unix time stamps for date queries in mongoexport
The time stamps have to be in milliseconds
Invoking this in the bash shell would look like this:
let "date_from=`date --utc --date "2013-02-01" +%s` * 1000"
let "date_to=`date --utc --date "2013-03-01" +%s` * 1000"
mongoexport -d test -c xx --query "{updated_at:{\$gte:new Date($date_from),\$lt:new Date($date_to)}}"> xx.json
> connected to: 127.0.0.1
> exported 1 records
The xx colletion contains:
> db.xx.find().pretty()
{
"_id" : ObjectId("5158f670c2293fc7aadd811e"),
"trigger_id" : ObjectId("50c62e97b9fe6a000200000c"),
"updated_at" : ISODate("2013-02-11T00:00:00Z")
}