mongoexport JSON assertion: 10340 Failure parsing JSON string - json

I'm trying to export CSV file list from mongoDB and save the output file to my directory, which is /home/asaj/. The output file should have the following columns: name, file_name, d_start and d_end.
The query should filter data with status equal to "FU" or "FD", and d_end > Dec. 10, 2012.
In mongoDB, the query is working properly. The query below is limited to 1 data output. See query below:
> db.Samples.find({ $or : [ { status : 'FU' }, { status : 'FD'} ], d_end : { $gte : ISODate("2012-12-10T00:00:00.000Z") } }, {_id: 0, name: 1, file_name: 1, d_start: 1, d_end: 1}).limit(1).toArray();
[
{
"name" : "sample"
"file_name" : "sample.jpg",
"d_end" : ISODate("2012-12-10T05:1:57.879Z"),
"d_start" : ISODate("2012-12-10T02:31:34.560Z"),
}
]
>
In CLI, mongoexport command looks like this:
mongoexport -d maindb -c Samples -f "name, file_name, d_start, d_end" -q "{'\$or' : [ { 'status' : 'FU' }, { 'status' : 'FD'} ] , 'd_end' : { '\$gte' : ISODate("2012-12-10T00:00:00.000Z") } }" --csv -o "/home/asaj/currentlist.csv"
But i always ended up with this error:
connected to: 127.0.0.1
Wed Dec 19 16:58:17 Assertion: 10340:Failure parsing JSON string near: , 'd_end
0x5858b2 0x528cb4 0x52902e 0xa9a631 0xa93e4d 0xa97de2 0x31b441ecdd 0x4fd289
mongoexport(_ZN5mongo11msgassertedEiPKc+0x112) [0x5858b2]
mongoexport(_ZN5mongo8fromjsonEPKcPi+0x444) [0x528cb4]
mongoexport(_ZN5mongo8fromjsonERKSs+0xe) [0x52902e]
mongoexport(_ZN6Export3runEv+0x7b1) [0xa9a631]
mongoexport(_ZN5mongo4Tool4mainEiPPc+0x169d) [0xa93e4d]
mongoexport(main+0x32) [0xa97de2]
/lib64/libc.so.6(__libc_start_main+0xfd) [0x31b441ecdd]
mongoexport(__gxx_personality_v0+0x3c9) [0x4fd289]
assertion: 10340 Failure parsing JSON string near: , 'd_end
I'm having error in ", 'd_end' " in mongoexport CLI. I'm not so sure if it is a JSON syntax error because query works on MongoDB.
Please help.

After asking someone knows MongoDB better than me, we found out that the problem is the
ISODate("2012-12-10T00:00:00.000Z")
We found the answer on this question: mongoexport JSON parsing error
To resolve this error, first, we convert it to strtotime:
php > echo strtotime("12/10/2012");
1355126400
Next, multiple strtotime result by 1000. This date will looks like this:
1355126400000
Lastly, change ISODate("2012-12-10T00:00:00.000Z") to new Date(1355126400000) in the mongoexport command.
Now, the CLI mongoexport looks like this and it works:
mongoexport -d maindb -c Samples -f "id,file_name,d_start,d_end" -q "{'\$or' : [ { 'status' : 'FU' }, { 'status' : 'FD'} ] , 'd_end' : { '\$gte' : new Date(1355126400000) } }" --csv -o "/home/asaj/listupdate.csv"
Note: remove space between each field names in -f or --fields option.

I know it has little to do with this question, but the title of this post brought it up in Google so since I was getting the exact same error I'll add an answer. Hopefully it helps someone.
My issue was adding a MongoId query for _id to a mongoexport console command on Windows. Here's the error:
Assertion: 10340:Failure parsing JSON string near: _id
The problem ended up being that I needed to wrap the JSON query in double quotes, and the ObjectId had to be in double quotes (not single!), so I had to escape those quotes. Here's the final query that worked, for future reference:
mongoexport -u USERNAME -pPASSWORD -d DATABASE -c COLLECTION
--query "{_id : ObjectId(\"5148894d98981be01e000011\")}"

Related

mongoexport - issue with JSON query (extended JSON - Invalid JSON input)

I have started learning MongoDB recently. Today the instructor taught us the mongoexport command. While practicing the same, I face a typical issue which none of the other batchmates including the instructor faced. I use MongoDB version 4.2.0 on my Windows 10 machine.
If I use mongoexport for my collection without any -q parameter to specify any filtering condition, it works fine.
mongoexport -d trainingdb -c employee -f empId,name,designation -o \mongoexport\all-employees.json
2019-09-17T18:00:30.300+0530 connected to: mongodb://localhost/
2019-09-17T18:00:30.314+0530 exported 3 records
However, whenever I specify the JSON query as -q (or --query) it gives an error as follows.
mongoexport -d trainingdb -c employee -f empId,name,designation -q {'designation':'Developer'} -o \mongoexport\developers.json
2019-09-17T18:01:45.381+0530 connected to: mongodb://localhost/
2019-09-17T18:01:45.390+0530 Failed: error parsing query as Extended JSON: invalid JSON input
The same error persists in all the different flavors I had attempted with for the query.
-q {'designation':'Developer'}
--query {'designation':'Developer'}
-q "{'designation':'Developer'}"
I had even attempted with a different query condition on the 'empId' as -q {'empId':'1001'} But no luck. I keep getting the same error.
As per one of the suggestions given in the StackOverflow website, I tried with the following option but getting a different error.
-q '{"designation":"Developer"}'
The error is : 'query '[39 123 101 109 112 73 100 58 49 48 48 49 125 39]' is not valid JSON: json: cannot unmarshal string into Go value of type map[string]interface {}'.
2019-09-17T20:24:58.878+0530 query '[39 123 101 109 112 73 100 58 49 48 48 49 125 39]' is not valid JSON: json: cannot unmarshal string into Go value of type map[string]interface {}
2019-09-17T20:24:58.882+0530 try 'mongoexport --help' for more information
I am really not sure what is missing here ? Tried with a bit of Googling and also gone through the official MongoDB documentation of the mongoexport - but no luck.
The employee collection in my system looks like the follows with 3 documents.
> db.employee.find().pretty()
{
"_id" : ObjectId("5d80d1ae0d4d526a42fd95ad"),
"empId" : 1001,
"name" : "Raghavan",
"designation" : "Developer"
}
{
"_id" : ObjectId("5d80d1b20d4d526a42fd95ae"),
"empId" : 1002,
"name" : "Kannan",
"designation" : "Architect"
}
{
"_id" : ObjectId("5d80d1b40d4d526a42fd95af"),
"empId" : 1003,
"name" : "Sathish",
"designation" : "Developer"
}
>
Update
As suggested by #NikosM, I have saved the query in a .json file (query.json) and tried the same mongoexport command with the new approach. Still, no luck. Same Marshal error.
cat query.json
{"designation":"Developer"}
mongoexport -d trainingdb -c employee -f empId,name,designation -q 'query.json' -o \mongoexport\developers.json
2019-09-17T21:16:32.849+0530 query '[39 113 117 101 114 121 46 106 115 111 110 39]' is not valid JSON: json: cannot unmarshal string into Go value of type map[string]interface {}
2019-09-17T21:16:32.852+0530 try 'mongoexport --help' for more information
Any help on this will be highly appreciated.
The following different approach made it work at last - where I had specified the JSON query with the double quotes escaped with the backslash : -q "{\"designation\":\"Developer\"}".
mongoexport -d trainingdb -c employee -f empId,name,designation -q "{\"designation\":\"Developer\"}" -o \mongoexport\developers.json
2019-09-17T21:33:01.642+0530 connected to: mongodb://localhost/
2019-09-17T21:33:01.658+0530 exported 2 records
cat developers.json
{"_id":{"$oid":"5d80d1ae0d4d526a42fd95ad"},"empId":1001.0,"name":"Raghavan","designation":"Developer"}
{"_id":{"$oid":"5d80d1b40d4d526a42fd95af"},"empId":1003.0,"name":"Sathish","designation":"Developer"}
Thank you very much #Caconde. Your suggestion helped.
But I am really not sure why this does not work in my machine alone and the reason for this tweak in the format of the query.
There is another approaches that I found out to work which were using the triple double-quote (""") for outside encasing.
mongoexport -d trainingdb -c employee -f empId,name,designation -q """ {"designation":"Developer"} """ -o \mongoexport\developers.json
The following different approach made it work at last - where I had specified the JSON query with the double quotes escaped with the backslash : -q "{"designation":"Developer"}".
for me it was
"{\"sensor_name\":\"Heat Recovery System Header Mass Flow\"}"
THIS ANSWER SOLVED MY ISSUE TYSM

How to read and parse the json file and add it into the shell script variable?

I am having the file named loaded.json which contains the below json data.
{
"name" : "xat",
"code" : "QpiAc"
}
{
"name" : "gbd",
"code" : "gDSo3"
}
{
"name" : "mbB",
"code" : "mg33y"
}
{
"name" : "sbd",
"code" : "2Vl1w"
}
Form the shell script i need to read and parse the json and add the result to the variable and print it like this.
#!/bin/sh
databasename = cat loaded.json | json select '.name'
echo $databasename
When i run the above script i am getting error like
databasename command not found
json command not found
I am new to shell script please help me to solve this problem
Replace this,
databasename=`cat loaded.json | json select '.name'`
or try jq command,
databasename=`jq '.name' loaded.json`
For more information read this article.
I can able to get the result using jq command like below
databasename=`cat loaded.json | jq '.name'`

Recognising timestamps in Kibana and ElasticSearch

I'm new to ElasticSearch and Kibana and am having trouble getting Kibana to recognise my timestamps.
I have a JSON file with lots of data that I wish to insert into Elasticsearch using Curl. Here is an example of one of the JSON entries.
{"index":{"_id":"63"}}
{"account_number":63,"firstname":"Hughes","lastname":"Owens", "email":"hughesowens#valpreal.com", "_timestamp":"2013-07-05T08:49:30.123"}
I have tried to create an index in Elasticsearch using the command:
curl -XPUT 'http://localhost:9200/test/'
I have then tried to set up an appropriate mapping for the timestamp:
curl -XPUT 'http://localhost:9200/test/container/_mapping' -d'
{
"container" : {
"_timestamp" : {
"_timestamp" : {"enabled: true, "type":"date", "format": "date_hour_minute_second_fraction", "store":true}
}
}
}'
// format of timestamp from http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-date-format.html
I then have tried to bulk insert my data:
curl -XPOST 'localhost:9200/test/container/_bulk?pretty' --data-binary #myfile.json
All of these commands run without fault however when the data is viewed in Kibana the _timestamp field is not being recognised. Sorting via the timestamp does not work and trying to filter the data using different periods does not work. Any ideas on why this problem is occuring is appricieated.
Managed to solve the problem. So for anyone else having this problem:
The format we had our date saved in was incorrect, needed to be :
"_timestamp":"2013-07-05 08:49:30.123"
then our mapping needed to be:
curl -XPUT 'http://localhost:9200/test/container/_mapping' -d'
{
"container" : {
"_timestamp" : {"enabled": true, "type":"date", "format": "yyyy-MM-dd HH:mm:ss.SSS", "store":true, "path" : "_timestamp"}
}
}'
Hope this helps someone.
There is no need to make and ISO8601 date in case you have an epoch timestamp. To make Kibana recognize the field as date is has to be a date field though.
Please note that you have to set the field as date type BEFORE you input any data into the /index/type. Otherwise it will be stored as long and unchangeable.
Simple example that can be pasted into the marvel/sense plugin:
# Make sure the index isn't there
DELETE /logger
# Create the index
PUT /logger
# Add the mapping of properties to the document type `mem`
PUT /logger/_mapping/mem
{
"mem": {
"properties": {
"timestamp": {
"type": "date"
},
"free": {
"type": "long"
}
}
}
}
# Inspect the newly created mapping
GET /logger/_mapping/mem
Run each of these commands in serie.
Generate free mem logs
Here is a simple script that echo to your terminal and logs to your local elasticsearch:
while (( 1==1 )); do memfree=`free -b|tail -n 1|tr -s ' ' ' '|cut -d ' ' -f4`; echo $load; curl -XPOST "localhost:9200/logger/mem" -d "{ \"timestamp\": `date +%s%3N`, \"free\": $memfree }"; sleep 1; done
Inspect data in elastic search
Paste this in your marvel/sense
GET /logger/mem/_search
Now you can move to Kibana and do some graphs. Kibana will autodetect your date field.
This solution works for older ES <2.4
For the newer version of ES you can either use the "date" field along with the following parameters:
https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html

How to convert BSON to JSON with human-readable date format

I would like to transform a BSON dump of MongoDB to JSON.
To do that, I'm using the bsondump tool provided with Mongo, but I get an output like :
{ "_id" : ObjectId( "5316d194b34f6a0c8776e187" ), "begin_date" : Date( 1394004372038 ), "foo" : "bar" }
{ "_id" : ObjectId( "5316d198b34f6a0c8776e188" ), "begin_date" : Date( 1394004407696 ), "foo" : "bar" }
Can anyone tell me how to get the dates appear in a human readable format (e.g. hh:mm:ss dd/mm/yyyy) ?
Edit
It looks like that a more recent version of mongodump outputs dates as:
{ "_id" : ObjectId( "5316d194b34f6a0c8776e187" ), "begin_date" : {"$date":"2015-11-11T08:45:03.974Z"}}, "foo" : "bar" }
So this question is not relevant anymore.
Thanks everybody for your help here.
bsondump converts BSON files into human-readable formats,
including JSON. For example, bsondump is useful for reading the output
files generated by mongodump.
Source: https://docs.mongodb.com/manual/reference/program/bsondump
Examples
bsondump --outFile collection.json collection.bson
The --pretty option outputs documents in a pretty-printed format JSON, eg:
bsondump --pretty --outFile collection.json collection.bson
To create a JSON file directly from the database, use mongoexport
mongoexport --db myDatabase --collection myCollection --jsonArray --out myJsonFile.json
db.players.find({"name" : "John"}).forEach(printjson);

Why does MongoDB produce invalid JSON? The ObjectId is not quoted, breaks jq parser

I've searched the forum and seen many folks with a similar problem, but not this exact one.
I think my question is the simplest form, and there must be something I'm missing because no one is asking it.
I have a shell script that calls a MongoDB script and gets the results in a file. I then want to parse that file with jq.
jq is breaking because the output from the query is not valid JSON. The offender is the ObjectId. I'm at a total loss as to how something that's "ALL JSON ALL THE TIME" produces invalid JSON.
I'm pretty sure there's something fundamental that I'm missing.
I have a file called MyMongoScript.js. Its contents look like this:
db.WorkflowJobs.find().sort({"STATUS":1}).forEach(printjson)
I call MyMongScript.js with the following command:
mongo -u $MONGO_U -p $MONGO_P $MONGO_DB -quiet --eval "var DATE_TO_RUN=\"$DATE_TO_RUN\"" MyMongoScript.js
Here's the results to STDOUT:
{
"_id" : ObjectId("52816fd50bc9efc3e6d8e33f"),
"WORK_TYPE" : "HIVE",
"Script_Name" : "upload_metrics_LANDING_to_HIST.sql",
"Stop_On_Fail" : true,
"STATUS" : "READY",
"START_TS" : "NULL",
"END_TS" : "NULL",
"DURATION" : "NULL",
"INS_TS" : "Mon Nov 11 2013 16:01:25 GMT-0800 (PST)"
}
Here's what jsonlint.com says about it:
Parse error on line 2:
{ "_id": ObjectId("52816fd50b
------------^
Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '['
Any help much appreciated.
Try this for your MyMongoScript.js:
db.WorkflowJobs.find().sort({"STATUS":1}).forEach(function(myDoc){myDoc._id=myDoc._id.valueOf();print(tojson(myDoc))});
The key is valueOf() which will set your ObjectId to a String.
EDITED Left out a paren.