How to auto format wrong json with code in flutter - json

I as trying to parse a json response. However, there is a trailing comma at the end of the reponse which throws an error while decoding it. How do i automatically format if from code such that it removes the comma and validates it?
[{"id" : "9991","last_message" : "How about tomorrow then?","members" : ["John", "Daniel", "Rachel"],"topic" : "pizza night", "modified_at" : 1599814026153}, {"id" : "9992","last_message" : "I will send them to you asap","members" : ["Raphael"],"topic" : "slides", "modified_at" : 1599000026153}, {"id" : "9993","last_message" : "Can you please?","members" : ["Mum", "Dad", "Bro"],"topic" : "pictures", "modified_at" : 1512814026153},]
Error
D/EGL_emulation( 7121): app_time_stats: avg=32.94ms min=4.95ms max=83.26ms count=30
E/flutter ( 7121): [ERROR:flutter/shell/common/shell.cc(93)] Dart Unhandled Exception: FormatException: Unexpected character (at character 435)
E/flutter ( 7121): ..."Mum", "Dad", "Bro"],"topic" : "pictures", "modified_at" : 1512814026153},]

So, you should give some code from source, but I think you're using a http.Response object so you can do something like this:
String bodyRes = response.body;
bodyRes = bodyRes.endsWith(',]') ? bodyRes.replaceFirst(',]', ']', bodyRes.length - 2) : bodyRes;

Related

UniqueDecodeError from urllib2 output from webpage with no non-unicode characters

I am trying to read data off an api webpage using urllib2 in Python2.7. I am using the following lines to read the page:
url = 'https://api.edamam.com/api/nutrition-data?app_id=<my_app_id>&app_key=<my_app_key>&ingr=1cheeseburger'
json_obj = urllib2.urlopen(url)
data = json.load(json_obj)
These lines give me this error (the error is on the last line in the above code):
UnicodeDecodeError: 'utf8' codec can't decode byte 0xb5 in position 0: invalid start byte
I understand that this error means that there are non-unicode characters in json_obj but I am not sure why this is the case, because the same url opens in a browser and the first few lines on the webpage looks like the following:
{
"uri" : "http://www.edamam.com/ontologies/edamam.owl#recipe_2a58ff3e1fec41d79da72f0be446baaa
"calories" : 312,
"totalWeight" : 119.0,
"dietLabels" : [ "BALANCED" ],
"healthLabels" : [ "PEANUT_FREE", "TREE_NUT_FREE", "ALCOHOL_FREE" ],
"cautions" : [ ],
"totalNutrients" : {
"ENERC_KCAL" : {
"label" : "Energy",
"quantity" : 312.96999999999997,
"unit" : "kcal"
},
As you can see, there are no non-unicode characters on this webpage, so I don't really follow what is going on.

AWS Athena Nested JSON error

I have a nested JSON file that looks like below:
\n \"total\" : 510,\n \"start\" : 0,\n \"count\" : 500,\n \"data\" : [ {\n \"id\" : 294,\n \"candidate\" : {\n \"id\" : 5275,\n \"firstName\" : \"bob\",\n \"lastName\" : \"bob\"\n },\n \"sendingUser\" : {\n \"id\" : 5,\n \"firstName\" : \"tom\",\n \"lastName\" : \"tom\"\n },\n \"dateAdded\" : 1487865908960,\n \"jobOrder\" : {\n \"id\" : 71,\n \"title\" : \"Job\"\n },\n \"status\" : \"1st Interview\",\n \"_score\" : 1.0\n }
I have this stored in S3 and am trying to create a table in AWS Athena, the editor I have done is below:
CREATE EXTERNAL TABLE IF NOT EXISTS cvtest (
data struct < candidate struct <id string, firstName string, lastName string>,
sendingUser struct <id string, firstName string, lastName string>,
dateAdded string,
jobOrder string,
score string
>
)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
LOCATION 's3://es-cvsent';
But the query runs into an error below;
FAILED: ParseException line 2:26 missing : at 'struct' near '<EOF>' line 2:37 missing : at 'string' near '<EOF>' line 2:55 missing : at 'string' near '<EOF>' line 2:72 missing : at 'string' near '<EOF>' line 3:28 missing : at 'struct' near '<EOF>' line 3:39 missing : at 'string' near '<EOF>' line 3:57 missing : at 'string' near '<EOF>' line 3:74 missing : at 'string' near '<EOF>' line 4:26 missing : at 'string' near '<EOF>' line 5:25 missing : at 'string' near '<EOF>' line 6:22 missing : at 'string' near '<EOF>'
This query ran against the "test" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: 84e876e8-b947-490e-b2b6-7bf9c376266e.
Can anyone see what I am doing wrong?
The data itself doesn't look like a valid JSON, but regardless of that you should be able to create a table because the underlying data is not validated at this point in time. (Just querying the table afterwards wouldn't work).
The problem you are having is that your syntax is wrong, see this documentation.":" is used between column and datatype within the struct definition.
This should work
data struct<candidate:struct<id:string, firstName:string, lastName:string>,
sendingUser:struct<id:string, firstName:string, lastName:string>,
dateAdded:string,
jobOrder:string,
score:string
>

Elasticsearch bulk data insertion

In my node app i am using Elasticsearch as my backend process. I am trying to insert data from a json file but I got an error.
My json:
{"index":{"_index":"mfissample", "_type":"place_mfi", "_id": "1"}}
{"PAR" : 42.31,"Center":"xx","District":"yy","Country" : "vv","GLP" : 13073826.63,"State" : "zz","SSScore" :null, "location":"80.102134,12.897401"}
{"index":{"_index":"mfissample", "_type":"place_mfi", "_id": "2"}}
{"PAR" : 42.31,"Center" : "xx","District" : "yy","Country" : "zz","GLP" : 13073826.63,"State" : "vv","SSScore" :null,
"location":"80.102134,12.897401"}
My command:
curl -XPOST 'http://localhost:9200/_bulk' --data-binary #jsonbulk.json
The error:
{"error":"JsonParseException[Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')\n at [Source: [B#792c4b55; line: 1, column: 12]]","status":500}
Remove the \n after "SSScore" :null, and before the "location":"80.102134,12.897401".

Parsing complex json in pig?

I have json file in follwoing format:
{ "_id" : "foo.com", "categories" : [], "h1" : { "bar==" : { "first" : 1281916800, "last" : 1316995200 }, "foo==" : { "first" : 1281916800, "last" : 1316995200 } }, "name2" : [ "foobarl.com", "foobar2.com" ], "rep" : null }
So, how do i parse this json in pig..
also, the categories and rep can have some char in it..and might not be always empty.
I made the following attempt.
a = load 'sample_json.json' using JsonLoader('id:chararray,categories:[chararray], hostt:{ (variable_a: {(first:int,last:int)})}, ns:[chararray],rep:chararray ');
But i get this error:
org.codehaus.jackson.JsonParseException: Unexpected character ('D' (code 68)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
at [Source: java.io.ByteArrayInputStream#4795b8e9; line: 1, column: 50]
at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1291)
at org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
at org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:306)
at org.codehaus.jackson.impl.Utf8StreamParser._handleUnexpectedValue(Utf8StreamParser.java:1582)
at org.codehaus.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:386)
at org.apache.pig.builtin.JsonLoader.readField(JsonLoader.java:173)
at org.apache.pig.builtin.JsonLoader.getNext(JsonLoader.java:157)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:211)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
You can use elephant bird pig jar for parsing json. It can parse all sort of json data.
Here are certain examples for parsing json via elephant bird pig using this jar.
https://github.com/twitter/elephant-bird/tree/master/examples/src/main/pig
It doesn't break even if an expected json tag isn't present.

Aptana gives error with JSON format

Format:
{
"lastUpdate" : "20/9/2012-12:12",
"data":[{
"user" : "_name_",
"username" : "_fullname_",
"photoURL" : "_url_"
}, {
"user" : "_name_",
"username" : "_fullname_",
"photoURL" : "_url_"
}, {
"user" : "_name_",
"username" : "_fullname_",
"photoURL" : "_url_"
}]
}
Aptana gives errors at the :
Screenshot Aptana JSON format
Why is that? It seems I'm not having any problems receiving and processing the data.
[EDIT 1] Error given: Syntax Error: unexpected token ":"
In Aptana json is parsed "as json" only when you create/open a file with extension .json.
When have a json object inside a .js file works only the javascript parser, for that you see the error, is not a valid token for JS.