Unexpected end of JSON input in MongoDB Compass - json

I want to import data of type JSON in MongoDB compass,
the import function gives this error
" unexpected end of JSON input "
there is a some of my JSON file
[
{
"id":4,
"user":"test#example.com",
"date1":"2019-03-01",
"date2":"2019-04-01",
"statut":"Good",
"guest_number":4
}
]

the solution is to write all JSON in one line, but if we have a big doc !!
I just found a solution that I can import data with this command in terminal :
mongoimport --jsonArray --db YourDatabase --collection YourCollection --file Yourfile.json

I had this issue 6 month ago, the solution is write all JSON in one line.
[{"id":4,"user":"test#example.com","date1":"2019-03-01","date2":"2019-04-01","statut":"Good","guest_number":4}]
MongoDB Compass will told you:
Import success!
But definitely the document will not appear in your collection, so better use Robo3T if you gonna insert json. Then you can use again Compass like I do.
It is weird, yes, but I didnt found other solution yet.
[UPDATE]
I achieve import data with Compass, but I achieve exporting first a document from Compass to see how it write the json.
{"_id":{"$oid":"5e4cf105c9ba1a21143d04a2"},"tPreguntas":["Pregunta 1","Pregunta 2","Pregunta 3","Pregunta 4","Pregunta 5"],"tCategorias":[],"tPublico":true,"tFechaCreacion":{"$date":{"$numberLong":"1582100741716"}},"tCodigo":"test1","tTitulo":"Test 1","tDescripcion":"Test de muestreo nĂºmero uno para comprobar.","tCreadoPor":"eoeo#eoeo.com"}
It look to different to the json online I have post in my first post. (look that objectId "$oid" for example). So if you follow that pattern Compass will import you fine.

This parsing error can be solved using minification. So, minify json like this. Although, it is quite a hectic process to do this for each object.
And this kind of minification like this worked for me.
{
"_id" : ObjectId("5b9ecf9a64f634289ca895bb"),
"name" : "Mark"
}
{
"_id" : ObjectId("5b9edd9064f634289ca895e4"),
"name" : "David"
}
To :
{"_id":"ObjectId(\"5b9ecf9a64f634289ca895bb\")","name":"Mark"}
{"_id":"ObjectId(\"5b9edd9064f634289ca895e4\")","name":"David"}

Just copy the contents of your json file then in Mongodb Compass select your database then click on Add Data which will drop down then click on insert document a dialog pops up then paste it in there and click insert.

This parsing error can be solved using minification. So, minify json like this. Although, it is quite a hectic process to do this for each object.
{
"_id" : "123456",
"name" : "stackoverflow"
}
change to :
{"_id":"123456","name":"stackoverflow"}

This answer here Solution solved the issue for me. It seems to be a formatting issue.

It's an issue with the end-of-line characters (EOL).
In a Windows environment line terminations are normally CR NL (\r\n), while MongoDB Compass seems to only support CR (\r).
You can open the file in Notepad++, enable the "Show all characters" toggle in the toolbar and inspect your current end-of-line character.
To fix the issue, select Edit > EOL Conversion > Macintosh (CR).

The structure of your JSON is incorrect, you might want to read info regarding JSON standards
A value can be a string in double quotes, or a number, or true or false or null, or an object or an array. These structures can be nested.
try using double quotes instead of single ones:
JSON validators could help you aswell
[
{
"id" : 4,
"user" : "test#example.com",
"date1" : "2019-03-01",
"date2" : "2019-04-01",
"statut" : "Good",
"guest_number" : 4
}
]

I had a similar issue but it turned out to be additional line feeds at the end of the file. Removing these fixed the issue. I suggest opening your file in an editor that shows line feeds e.g. Notepad++

Add --jsonFormat=canonical to your mongoexport script:
mongoexport --db=quotes --collection=quotes --jsonFormat=canonical --out=data/quotes.json
JSON can only directly represent a subset of the types supported by BSON. To preserve type information, MongoDB adds the following extensions to the JSON format.
Source

You can also use the command line of mongodb like this :
db.user.insert(
[
{
"id" : 4,
"user" : "test#example.com",
"date1" : "2019-03-01",
"date2" : "2019-04-01",
"statut" : "Good",
"guest_number" : 4
},
{
"id" : 5,
"user" : "test2#example.com",
"date1" : "2019-03-01",
"date2" : "2019-04-01",
"statut" : "Good",
"guest_number" : 4
}
]

Run this command in cmd and the cmd path should be in the same folder where the JSON file occurs.
mongoimport --jsonArray --db YourDatabase --collection YourCollection --file Yourfile.json

Related

Restoring a MongoDB collection from a text file of json documents

I have been given a text file, containing thousands of json documents (not ideal I know).
I need to put said documents into a mongodb collection.
So far, I have saved the text file as JSON and tried to mongoimport, added commas between each document and attempted mongorestore with a bson equivalent - all to no success
Here is an example of what is in the text file:
{
"_id" : ObjectId("78ahgodjaodj90231"),
"date" : ISODate("1970-01-01T00:00:00+0000),
"comment" : "Hello"
}
{
"_id" : ObjectId("99151gdsgag5464ah"),
"date" : ISODate("1970-01-02T00:00:00+0000),
"comment" : "World"
}
and so on...
Using mongoimport I get this error message:
Failed: invalid JSON input. Position: 16. Character: O
After saving as a BSON file, using mongorestore I also get this error:
Failed: db.collection: error restoring from file.bson: reading bson input: invalid BSONSize: 537534587 bytes
Any help would be greatly appreciated!
Let's say we have the following data in the file:
{
"_id" : ObjectId("78ahgodjaodj90231"),
"date" : ISODate("1970-01-01T00:00:00+0000),
"comment" : "Hello"
}
{
"_id" : ObjectId("99151gdsgag5464ah"),
"date" : ISODate("1970-01-02T00:00:00+0000),
"comment" : "World"
}
We need to refactor it to a code like below and save it with .js extension, say insert_data.js
db.collection.insertMany([
{
"_id" : ObjectId("78ahgodjaodj90231"),
"date" : ISODate("1970-01-01T00:00:00+0000),
"comment" : "Hello"
},
{
"_id" : ObjectId("99151gdsgag5464ah"),
"date" : ISODate("1970-01-02T00:00:00+0000),
"comment" : "World"
}
])
Finally run the following command:
mongo HOST:PORT/DB insert_data.js
I managed to import the documents successfully using Studio3T's import feature.
After renaming the text file to a JSON file, and letting Studio 3T validate the JSON before import, it worked perfectly.
Not the best solution, but it seemed to work for me.

PutElasticsearchHttpRecord: Invalid char between encapsulated token and delimiter

I am trying to fetch a specific record from Database using QueryDatabaseTable -> UpdateAttribute-> PutElasticSearchHttpRecord
The ES processor is throwing error as Java.IO.Exception Invalid char between encapsulated token and delimiter.
Please find attached my config. How to fix this?
I am getting the correct result in the queue after 'UpdateAttribute' but not able to push it into ES. I have added the schema.name property to appropriate schema.
The following is the correct result i am getting in the queue after UpdateAttribute processor. How to fix the error of Invalid character between token and delimiter?
[ {
"TimeOfDay" : "2018-09-20T18:10:36.941",
"BMU_Debug_Pack_BlkVolt_Max2" : 4114.0,
"BMU_Debug_Pack_BlkVolt_Max1" : 4114.0,
"BMU_Debug_Pack_BlkVolt_Max3" : 4114.0,
"BMU_Debug_Pack_BlkVolt_Max0" : 4116.0,
"BMU_Debug_Pack_CTemp_Min" : 21.0,
"BMU_Debug_Pack_CurrVolt_Curr" : 2.0,
"BMU_Debug_Pack_Blk_Volt_Delta" : 6.0,
"total_Difference" : 15.0
} ]
Thank you! Please help what should I change?
You need to configure Avro Reader instead of CSV Reader in PutElasticSearchHttpRecord as QuerydatabaseTable processor outputs flowfile in Avro format.
Use embedded avro schema in Avro Reader controller service.

How to export large dataset collection of mongodb to CSV file on button click via node express

I have db contain large dataset - json objects - (array) around ~10k i have for now. I want to to fetch all from db and generate csv and download via route..
Here's sample json object:
{
"_id" : ObjectId("56bc3a7da30befd952349542"),
"asin" : "B00T2Q1S18",
"searchRank" : 113,
"name" : "FREEing Racing Miku 2014 (EV Mirai Version) Figma Action Figure",
"createdAt" : ISODate("2016-02-11T07:38:37.774Z"),
"updatedAt" : ISODate("2016-02-11T07:44:07.667Z"),
"linkIds" : [
"25b1071a9e908806338c4106"
],
"price" : {
"amazon" : 50.49
},
"ranks" : [
{
"number" : 43619,
"category" : "Baby Toys"
}
],
"upc" : ""
}
Is there any better npm (node) library which can converts my json collection to csv..
Though I have tried those but on large dataset they aren't working.
papaparse / babyparse
json2csv
Is there any other libs that you know better or any other better approach?
Thanks.
I have done this before using an npm library called csv-builder. Based on my experience I can say that it gives good performance and It is quite easy to implement.
I have made a CSV of about 2 LAC rows and around 8-10 columns,with manipulation in between using this library.
I tried with many libs and at last I found one - a great npm module which handles large dataset problem nicely....
https://www.npmjs.com/package/csvwriter
exported upto 5 lacs + json objects (for now)..
Here is my small demo large dataset json to csv exporter app via node, express, mongodb
Hope this helps others as well, when they come over here.
Cheers,
Thanks.

Cannot import a json in MongoDB

I am using RockMongo in Openshift to import a json file in MongoDB database. I exported directly the json from another MongoDB and I haven't changed anything. Here is a part of the json:
{ "_id" : "10352",
"author" : "8988607",
"country" : "...",
"views" : 1716,
"title" : "...",
"comments" : 1,
"likes" : 28,
"text" : "...",
"date" : { "$date" : 1278070740000 },
"approved" : "8480596" }
And I have this error message:
exception: field names cannot start with $ [$date] at src/mongo/shell/collection.js:147
As I said, I exported the json directly from another MongoDB. How can I solve this problem now?
I came up against this problem and my dba replaced the dollar sign with \uFF04 and that did the trick for us.
MongoDB uses its Extended JSON. Rockmongo likely uses a standard JSON parser, thus the mismatches.
Can you use the provided mongoimport application? You will need to use v2.4.0 or greater to include all the extended types see: SERVER-5675

How to indent JSON data inside of TextMate, Emacs, BBEdit, or Sublime Text 2?

[Update: 8 hours after this question was posted, the author of JSON bundle was notified of the issue and he fixed it.]
I have the following JSON data in a file application.json, shown at the end of this post, and I have used TextMate with the JSON bundle, Emacs, BBEdit, and Sublime Text 2 to properly indent it, but all seemed like they couldn't.
Both TextMate and Sublime Text 2 insisted that the first { should not be indented, and the first major issue was for the closing brace for "child": {. Both TextMate and Sublime Text 2 refused to align the } under the left side of "child": {. Emacs kept on indenting further and further for each line, and BBEdit didn't seem to have an re-indent function at all (could this be?).
Is there a way to properly indent the file, or are TextMate and Sublime Text 2 both doing the right thing for the JSON data?
[
{
"settings": [ "master" ],
"appPort": "8666",
"specs": {
"frame" : {
"type" : "HTMLFrameMojit",
"config": {
"deploy": true,
"child": {
"type" : "HelloWorldMojit"
},
"assets": {
"top": {
"css": [
"/static/HelloWorldMojit/assets/index.css"
]
}
}
}
}
}
},
{
"settings": [ "environment:development" ],
"staticHandling": {
"forceUpdate": true
}
}
]
EDIT: For BBEdit use siegel's suggestion of Text > Reformat Document
Original Reply:
I found a solution for BBEdit that is easy and works well.
Put the following script in
~/Library/Containers/BBEdit/Data/Library/Application Support/BBEdit/Text Filters/FormatJSON.sh (on MacOS 11 Big Sur, or above)
For MacOS 10.15 Catalina and below, use this location: ~/Library/Application Support/BBEdit/Text Filters/FormatJSON.sh
#!/bin/bash
python -m json.tool
Open a JSON file in BBEdit. There is no need to restart BBEdit because BBEdit rocks!
Select Text > Apply Text Filter > FormatJSON
I tested this with a JSON file that had 3,612,683 characters on a single line. BBEdit opened this file and reformatted without showing a "Spinning Beachball of Death" busy-wait mouse cursor.
Solution 1: Using Python
This answer is similar to this answer, except I am using python file to do the JSON format.
Exit bbedit application if it is open,
put following script pretty-json.py in ~/Library/Application\ Support/BBEdit/Text\ Filters/ path
#!/usr/bin/env python
# You can change above she-bang line depending on your Mac configuration
import sys
import json
def main():
input = sys.stdin.read()
try:
obj = json.loads(input)
except Exception as e:
print input + "\n\nERROR: " + str(e)
return 1
print(json.dumps(obj, indent=2))
return 0
if __name__ == '__main__':
sys.exit(main())
To Test, open a JSON file in BBEdit.
Select Text --> Apply Text Filter --> pretty-json.py
If you face any issue like formatting error, then the above script will add error in New file and will not change the original JSON.
which is not the case with this answer
Ref: https://gist.github.com/brokaw/95ade1358954cd97d0f2c8e992e14b08
For more info: Refer this
The above filter works fine for smaller JSON files, but if the JSON file is large(~ 40MB) then formatting will be slow.
To solve this, use the following solution
Solution 2: Using jq
For faster json formatting,
Install jq brew install jq
Check if you are able to execute jq in terminal, or need a full path, add whichever works in following file in place of jq
Add fast-json-pretty.sh file in ~/Library/Application\ Support/BBEdit/Text\ Filters/ location
Restart bbedit.
#!/bin/bash
jq
In BBEdit 14.0 and later, "Reformat Document" on the text menu will reflow JSON. You can also use the built-in Language Server Support with a JSON language server that supports reformatting.
I just corrected this issue in the bundle, for 2.0 users the bundle should update within 24 hours with the correction.
According to http://jsonprettyprint.com/ Textmate and Sublime aren't doing the right thing.
What version of Emacs did you use?
With 24.2.1, your JSON blob indented perfectly without issues in js-mode (Emac's default javascript major-mode).
If you do any significant Javascript development I recommend checkint out js2-mode https://github.com/mooz/js2-mode, which turns Emacs into a great JS IDE.
Sublime Pretty JSON
Sublime Pretty JSON indents the first { well.
This is what I get:
[
{
"settings": [
"master"
],
"appPort": "8666",
"specs": {
"frame": {
"type": "HTMLFrameMojit",
"config": {
"deploy": true,
"child": {
"type": "HelloWorldMojit"
},
"assets": {
"top": {
"css": [
"/static/HelloWorldMojit/assets/index.css"
]
}
}
}
}
}
},
{
"settings": [
"environment:development"
],
"staticHandling": {
"forceUpdate": true
}
}
]
Installation
Within Sublime Text 2: Preference => Package Control => Install Package => "Pretty Json" => Restart Sublime => Select JSON Text => Press:
Linux: ctrl+alt+j
Windows: ctrl+alt+j
OS X: cmd+ctrl+j
This is a solution simply using Google Chrome or NodeJS itself, and here is how:
Just open up the Google Chrome dev tool or NodeJS prompt, and type in
obj =
and copy and paste the object into it, so it will be like
obj = [
{
// etc
Now, in Google Chrome, just type
JSON.stringify(obj, null, 4)
and you will have the formatted JSON. In NodeJS, since it prints out the \n verbatim, you can use
console.log(JSON.stringify(obj, null, 4))
If you want the indentation to be just 2 spaces, just use 2 instead of 4.
If you want the indentation to be tabs instead of spaces, simply use
JSON.stringify(obj, null, "\t")
Create a script file fast-json-pretty.sh in ~/Library/Application\ Support/BBEdit/Text\ Filters/
Insert the following script:
#!/usr/bin/env bash
/usr/local/bin/jq .
Apply the text filter as follows: Menu bar --> Text --> Apply Text Filter --> fast-json-pretty
This is a reworked version of DKB's version of his JQ script as it, at least for me, required the full path and a dot (.) at the end to make this work.
Just check to see which version of python is installed:
using json_pretty will only work if the python interpreter is whatever is installed in the shell being called. For example, my json_pretty works and I have modded it as follows on Ventura:
#!/bin/zsh
python3 -m json.tool