I am storing my data from external file to mongodb in localhost. it's quite huge dataset of volume 1.70GB with ~10 million tweets. While importing from file to mongodb it shows me the error "JSON reader was expecting a name but found ':'"
I dint have any error on previous files. But this I cant figure it out. The data is just a real time collection of tweets from streaming API in a json format.
BufferedReader br = new BufferedReader(new FileReader(file));
int counter = 0;
String line;
while ((line = br.readLine()) != null) {
System.out.println(line);
Document old_status = Document.parse(line);
// If it is a deleted tweet, then continue
if (old_status.containsKey("delete")) {
continue;
}
//populate original tweets
Document original_status = new Document();
if(line.contains("retweeted_status")){
Document retweets = (Document)old_status.get("retweeted_status");
original_status.append("status",retweets.get("text"));
original_status.append("Likes",retweets.getInteger("favorite_count"));}}
its a sample code for importing data from file to mongo collection.
Help me to solve this. I really stuck in this place and it takes my time.
Thanks in advance.
Late to the party, and also a little unrelated to this specific question, but google brought me here so going to leave an answer.
I got this error when I was trying to parse a mongodb document that was converted to JSON in Scala. Make sure that the variable's you're trying to parse are actually in the JSON you're parsing if it is a strict parsing.
Related
I am using a service builder that is retrieving the form data fine from mysql db. I have a field that has the json data and I tried to map it using object mapper and using com.fasterxml.jackson.databind.ObjectMapper to display the json content. However, the Blob Data is shown as: com.mysql.cj.jdbc.Blob#4ca74f7f
How do I actually get/extract the data from the storing link above? Here is my code snippet:
for (ddmcontent MyTemp : myList) {
System.out.println("Content ID : "+myList.getContentId());
System.out.println("User Blob Data : "+myList.getData());
Blob responseBody =myList.getData();
ObjectMapper objectMapper = new ObjectMapper();
List<ModelData> myDat = objectMapper.readValue((DataInput)
responseBody,objectMapper.getTypeFactory().constructCollectionType
(List.class,ModelData.class));
for (ModelData dt : myDat) {
System.out.println("User Name : "+dt.Name);
System.out.println("Users Email : "+dt.Email);
}
}
Please note, I have defined my ModelData elements as all String.
Any suggestion? What am I missing?
Thanks in advance!
The toString() representation hints at the object's type com.mysql.cj.jdbc.Blob. If you look up its javadoc (or the interface it implements) you'll see the options that you have to decode the contents of the Blob, namely getting an InputStream or a byte[] representation, which you'd have to subject to the correct character set decoding to turn it into a String.
Make sure you nail the character set by testing it with all kinds of Unicode content, so that you don't have to fix badly encoded database content later when your table contains a lot of data in unknown encodings.
As you're using Liferay's Service Builder, you might want to share the relevant parts of your service.xml (optional model-hints.xml) to check for an easier implementation.
I finally got this working by changing the field in question to String and addining a max length to certain number of Char
Thanks!
I hate json files. They are unwieldy and hard to handle :( Please tell me why the following doesn't work:
with open('data.json', 'r+') as file_object:
data = json.load(file_object)[user_1]['balance']
am_nt = 5
data += int(am_nt['amount'])
print(data)
file_object[user_1]['balance'] = data
Through trial and error (and many print statements), I have discovered that it opens the file, goes to the correct place, and then actually adds the am_nt, but I can't make the original json file update. Please help me :( :( . I get:
2000
TypeError: '_io.TextIOWrapper' object is not subscriptable
json is fun to work with as it is similar to python data structures.
The error is: object is not subscriptable
This error is for this line:
file_object[user_1]['balance'] = data
file_object is not json/dictionary data that can be updated like above. Hence the error.
Try to read the json data:
data=json.load(file_object)
Then manipulate the data as python dictionary. And save the file.
I'm new in creating jsons and having bais knowledge of java.
I'm trying to convert database table data to json.
Having option to store table data in any format of file than convert that into json.
Here is my table data.
Table: PKGS
Price, pd, Id, Level
1 , 266 , 59098 , 5
2 , 247 , 59098 , 5
I want my table data in this json format. Its just an example...to show level in JSON
"Id":59098
"pd":266
"Level":5
"price":1
"Id":59098
"pd":247
"Level":5
"price":2
In this json there is two loops are going If am not wrong. I was able to do it for one loop in ETL..but couldnt do it for two loops.
Not getting values for reimbursementId and packageId
Have goggled alot but couldn't find any code to understand properly and approach for the same.
Code tried little bit
FileInputStream inp = new FileInputStream("D:/json.xlsx" );
Workbook workbook = WorkbookFactory.create( inp );
Sheet sheet = workbook.getSheetAt( 0 );
JSONObject json = new JSONObject();
JSONArray rows = new JSONArray();
but dont know what to next !!
can anyone tell me how to do this ?
I advice you to use a specific tool. This is not a whole new case.
Try Talend Open Studio. It is not so complicate to use if you want to convert a file (CSV, JSON, Database directly, etc) to another. Please see TalendForge for basics.
In your case, you can connect to your database, and send all data in JSON.
Edit:
Your representation is not following the same logic than JSON. Here how I see it (and this is probably wrong because I can't understand)
If you just want Excel to JSON without any changes:
{
"rows":[
{
"Price":"1",
"pd":"266",
"Id":"59098",
"Level":"5"
},
{
"Price":"1",
"pd":"266",
"Id":"59098",
"Level":"5"
},
//and again and again
{
"Price":"2",
"pd":"247",
"Id":"59098",
"Level":"5"
}
]
}
If you want to reorganize, then define what you want. Try to imagine a sample of your data in a Java Object using ArrayList, int, String and subclass or even better in a JavaScript Object.
For the last example it will give you:
public class myJson{
ArrayList<myObject> rows;
with
public class myObject{
String Price;
String pd;
String Id;
String Level; //Or int, or date, or whatever
If you want to reorganize your data model, please give us this model.
Converting Excel file data to Json format is a a bit complex process, it depends on the structure of data, we do not have exact online tool as such so far...
custom code is required, there are various technologies available to be used, but best suited should be VBA, because VBA fits within Excel and can generate Json file quickly and flexible to edit code compared to any other technology that requires to automate to excel and import data and then process.
We can find there are various websites provide code to generate json from excel data, here is one such site looks expertise in this area. http://www.xlvba.net/tools/excel-automation-to-convert-excel-data-to-json-format.html
Ragavendra
Good day,
I need to get some values from a JSON file, but I can not deserialize the file.
It is available here:http://files.minecraftforge.net/maven/net/minecraftforge/forge/json
What I would like to get are the values of "mcversion" and "version" found in the "number" section, the remaining data not interest me.
I generated classes but do not work due to JSON format, this special is different.
I hope you can help me, thank you very much.
To Read from a File you can use
Dim json As String = File.ReadAllText("JSON File Path")
Dim o As JObject = JObject.Parse(json)
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
Does anyone know if there is application that will let me convert preferably XLS to JSON?
I'll also settle for a converter from CSV since that's what I'll probably end up having to write myself if there is nothing around.
You can try this tool I made:
Mr. Data Converter
It converts to JSON, XML and others.
It's all client side, too, so your data never leaves your computer.
This worked perfectly for me and does NOT require a file upload:
https://github.com/cparker15/csv-to-json?files=1
Since Powershell 3.0 (shipped with Windows 8, available for Windows 7 and windows Server 2008 but not Windows Vista ) you can use the built-in convertto-json commandlet:
PS E:> $topicsjson = import-csv .\itinerary-all.csv | ConvertTo-Json
PS E:\> $topicsjson.Length
11909
PS E:\> $topicsjson.getType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Object[] System.Array
Online Help Page on Technet
If you can't find an existing solution it's pretty easy to build a basic one in Java. I just wrote one for a client and it took only a couple hours including researching tools.
Apache POI will read the Excel binary.
http://poi.apache.org/
JSONObject will build the JSON
After that it's just a matter of iterating through the rows in the Excel data and building a JSON structure. Here's some pseudo code for the basic usage.
FileInputStream inp = new FileInputStream( file );
Workbook workbook = WorkbookFactory.create( inp );
// Get the first Sheet.
Sheet sheet = workbook.getSheetAt( 0 );
// Start constructing JSON.
JSONObject json = new JSONObject();
// Iterate through the rows.
JSONArray rows = new JSONArray();
for ( Iterator<Row> rowsIT = sheet.rowIterator(); rowsIT.hasNext(); )
{
Row row = rowsIT.next();
JSONObject jRow = new JSONObject();
// Iterate through the cells.
JSONArray cells = new JSONArray();
for ( Iterator<Cell> cellsIT = row.cellIterator(); cellsIT.hasNext(); )
{
Cell cell = cellsIT.next();
cells.put( cell.getStringCellValue() );
}
jRow.put( "cell", cells );
rows.put( jRow );
}
// Create the JSON.
json.put( "rows", rows );
// Get the JSON text.
return json.toString();
This works for me and runs client-side:
http://www.convertcsv.com/csv-to-json.htm
I just found this:
http://tamlyn.org/tools/csv2json/
( Note: you have to have your csv file available via a web address )
Take a try on the tiny free tool:
http://keyangxiang.com/csvtojson/
It utilises node.js csvtojson module
None of the existing solutions worked, so I quickly hacked together a script that would do the job. Also converts empty strings into nulls and and separates the header row for JSON. May need to be tuned depending on the CSV dialect and charset you have.
#!/usr/bin/python
import csv, json
csvreader = csv.reader(open('data.csv', 'rb'), delimiter='\t', quotechar='"')
data = []
for row in csvreader:
r = []
for field in row:
if field == '': field = None
else: field = unicode(field, 'ISO-8859-1')
r.append(field)
data.append(r)
jsonStruct = {
'header': data[0],
'data': data[1:]
}
open('data.json', 'wb').write(json.dumps(jsonStruct))
Instead of hard-coded converters, how about CSV support for Jackson (JSON processor): https://github.com/FasterXML/jackson-dataformat-csv. So core Jackson can read JSON in as POJOs, Maps, JsonNode, almost anything. And CSV support can do the same with CSV. Combine the two and it's very powerful but simple converter between multiple formats (there are backends for XML, YAML already, and more being added).
An article that shows how to do this can be found here.
See if this helps: Back to CSV - Convert CSV text to Objects; via JSON
This is a blog post published in November 2008 that includes C# code to provide a solution.
From the intro on the blog post:
As Json is easier to read and write then Xml. It follows that CSV (comma seperated values) is easier to read and write then Json. CSV also has tools such as Excel and others that make it easy to work with and create. So if you ever want to create a config or data file for your next app, here is some code to convert CSV to JSON to POCO objects