Excel - We found extra characters at the end of JSON input - json

I am trying to import my json file to excel via the get data function. When doing this i get an error saying that "We found extra characters at the end of JSON input"
i ran the json file in jsonformatter and got this additional piece of information:
Parse error on line 1:
...s":"1555615338756"}
{"created_at":"Thu A
-----------------------^
Expecting 'EOF', '}', ',', ']', got '{'
Edit: line 1
{"created_at":"Thu Apr 18 19:22:18 +0000 2019","id":1118957948263206913,"id_str":"1118957948263206913","text":"Arsenal jersey looks weird. #NapoliArsenal","source":"\u003ca href=\"https://mobile.twitter.com\" rel=\"nofollow\"\u003eTwitter Web App\u003c/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":955479430377373696,"id_str":"955479430377373696","name":"Yash Iyer","screen_name":"MesutOziI28","location":"Bengaluru South, India","url":null,"description":"RM,Nerazzurri,BFC,RCB,bcci,rafa nadal and so on! Lately into B99,superstore! Sympathetic liker of tweets!","translator_type":"none","protected":false,"verified":false,"followers_count":258,"friends_count":454,"listed_count":0,"favourites_count":47788,"statuses_count":5318,"created_at":"Mon Jan 22 16:37:02 +0000 2018","utc_offset":null,"time_zone":null,"geo_enabled":false,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"F5F8FA","profile_background_image_url":"","profile_background_image_url_https":"","profile_background_tile":false,"profile_link_color":"1DA1F2","profile_sidebar_border_color":"C0DEED","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":true,"profile_image_url":"http://pbs.twimg.com/profile_images/1109886916609007616/9rAavtGh_normal.jpg","profile_image_url_https":"https://pbs.twimg.com/profile_images/1109886916609007616/9rAavtGh_normal.jpg","profile_banner_url":"https://pbs.twimg.com/profile_banners/955479430377373696/1544903252","default_profile":true,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"is_quote_status":false,"quote_count":0,"reply_count":0,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[{"text":"NapoliArsenal","indices":[28,42]}],"urls":[],"user_mentions":[],"symbols":[]},"favorited":false,"retweeted":false,"filter_level":"low","lang":"en","timestamp_ms":"1555615338756"}

Forever old, but I was trying to ingest an AWS CloudCheckr JSON into Excel via Power Query and getting the "We found extra characters at the end of JSON input."
Finally figured out, with the help of https://jsonformatter.org/ that some data was provided as True -- without quotes, which Excel PQ needed. Simple find/replacing :True, with :"True", did the trick.
Seriously, Microsoft, you did not recognize Boolean when you found it? Excel would have had no problem.

The answer is in the error message:
Expecting 'EOF', '}', ',', ']', got '{'
Looking at where { appears notice that directly before that is }. The JSON has no separator after the closing curly } and thus cannot process it because it's looking for one of the following:
EOF
}
,
]
In this case, it most likely needs a ,. It could also need ],, if it's an array of items. If neither of those fix it, you will need to post the entire line 1 of your JSON.

I had to do two changes to make this work: changed json body to Pascal case notation even thought it was in Camel case in chrome payload
, and secondly I was returning a simple string to verify the service is working - once I changed that to an object (as a json formatted response) this resolved the error I was getting.

Related

Not able to read json file in JMETER

I am trying to read json's from a text file using below command:
{__FileToString((${JSON_FILE},,)).replaceAll(' ','')}
File not readable.
Error: {"timestamp":1586945558777,"status":400,"error":"Bad Request","exception":"org.springframework.http.converter.HttpMessageNotReadableException","message":"Could not read document: Unexpected character ('' (code 95)): was expecting double-quote to start field name\n at [Source: java.io.PushbackInputStream#6df97f39; line: 1, column: 3]; nested exception is com.fasterxml.jackson.core.JsonParseException: Unexpected character ('' (code 95)): was expecting double-quote to start field name\n at [Source: java.io.PushbackInputStream#6df97f39; line: 1, column: 3]","path":"/service"}
I have gone through all related posts too but still not able to find the solution to it. Please can anyone help.
Refereneces:
JMeter - How to read JSON file?
https://devqa.io/perf/jmeter-send-json-file-as-request-in-body
https://www.360logica.com/blog/how-to-use-http-request-to-send-multiple-json-files
Thanks,
Mrinalini
The correct syntax would be:
${__strReplace(${__FileToString(${JSON_FILE})}, ,,)}
You cannot append normal Java functions like String.replaceAll() to JMeter Functions, if you want to get rid of whitespace characters you need to invoke __strReplace() function like shown above (this function is a Custom JMeter Function, it can be installed using JMeter Plugins Manager)
If you cannot use JMeter Plugins for some reason you can achieve the same using __groovy() function like:
${__groovy(new File(vars.get('JSON_FILE')).text.replaceAll(' '\, ''),)}
Demo:

What does this error code mean? JSONDecodeError: Expecting Value

I'm using discord.py to create a bot for my server. I'm following a tutorial made by Spyros, and I'm getting this error:
JSONDecodeError: Expecting value: line 1 column 1 (char 0)]
It's just an empty JSON file, just curly brackets with no data.
Anyone knows what I can do? I'll post more code if needed, thanks!
JSONDecode is expecting data and as you said, you are passing an empty JSON "file"

How to decode an HTTP request with utf-8 and treat the surrogate keys (Emojis)

I'm having a hard time dealing with some parsing issues related to Emojis.
I have a json requested through the brandwatch site using urllib.(1) Then, I must decode it in utf-8, however, when I do so, I'm getting surrogate keys and the json.loader cannot deal with them. (2)
I've tried to use BeautifulSoup4, which works great, however, when there's a &quot on the site result, it is transformed to ", and then, the json.loader cannot deal with it for it says that a , is missing. After tons of search, I gave up trying to escape the " which would be the ideal.(3)
So now, I'm stuck with both "solutions/problems". Any ideas on how to proceed?
Obs: This is a program that fetchs data from the brandwatch and put it inside an MySQL database. So performance is an issue here.
Obs2: PyJQ is a JQ for Python with does the request and I can change the opener.
(1) - Dealing with the first approach using urllib, these are the relevants parts of the code used for it:
def downloader(url):
return json.loads(urllib.request.urlopen(url).read().decode('utf8'))
...
parsed = pyjq.all(jqparser,url=url, vars={"today" : start_date}, opener=downloader)
Error Casted:
Exception ignored in: '_pyjq.pyobj_to_jv'
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' in position 339: surrogates not allowed
*** Error in `python': munmap_chunk(): invalid pointer: 0x00007f5f806303f0 ***
If I print the result of urllib.request.urlopen(url).read().decode('utf8') instead of sending it to json.loader, that's what appears. These keys seems to be Emojis.
"fullname":"Botinhas\uD83D\uDC62"
(2) Dealing with the second approach using BeautifulSoup4, here's the relevant part of the code. (Same as above, just changed the downloader function)
def downloader(url):
return json.loads(BeautifulSoup(urllib.request.urlopen(url), 'lxml').get_text())
...
parsed = pyjq.all(jqparser,url=url, vars={"today" : start_date}, opener=downloader)
And this is the error casted:
Expecting ',' delimiter: line 1 column 4814765 (char 4814764)
Doing the print, the " before Diretas Já should be escaped.
"title":"Por "Diretas Já", manifestações pelo país ocorrem em preparação ao "Ocupa Brasília" - Sindicato dos Engenheiros no Estado do Rio de Janeiro"
I've thought of running a regex, however, I'm not sure whether this would be the most appropriate solution to this case as performance is an issue.
(3) - Part of Brandwatch result with the &quot problem mentioned above
UPDATE:
As Martin stated in the comments, I ran a replace swapping &quot for nothing. Then, it raised the former problem, of the emoji.
Exception ignored in: '_pyjq.pyobj_to_jv'
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' in position 339: surrogates not allowed
*** Error in `python': munmap_chunk(): invalid pointer: 0x00007f5f806303f0 ***
UPDATE2:
I've added this to the downloader function:
re.sub(r'\\u(d|D)([a-z|A-Z|0-9]{3})', "", urllib.request.urlopen(url).read().decode('utf-8','ignore'))
It solved the issue, however, I don't think it's the best way to solve it. If anybody knows a better option.

How to import Google Maps API into PostgreSQL?

I am trying to transfer data from a JSON file produced by the Google Maps API onto my PostgreSQL database. This is done through cURL and I made sure that the permissions have been correctly set.
The url:
https://maps.googleapis.com/maps/api/distancematrix/json?units=imperial&origins=London&destinations=Paris&key=AIza-[key-redacted]-3z6ho-o
The query:
copy bookings.import(info) from program 'C:/temp/mycurl/curl "https://maps.googleapis.com/maps/api/distancematrix/json?units=imperial&origins=London&destinations=Paris&key=AIzaSyBIhOMI68hTIFarH4jrb_eKUmvY3z6ho-o" --insecure'
However, when I try to do this on my table with column 'info' of type 'json', I get the following error:
ERROR: invalid input syntax for type json DETAIL: The input string
ended unexpectedly. CONTEXT: JSON data, line 1: { COPY import, line
1, column info: "{"
********** Error **********
ERROR: invalid input syntax for type json SQL state: 22P02 Detail: The
input string ended unexpectedly. Context: JSON data, line 1: { COPY
import, line 1, column info: "{"
I am trying to not include things such as PHP or any other tool currently, yet if the only option is that I would certainly consider it.
What exactly do you guys think I am doing wrong? Is it the syntax, the format or am I missing something?
Thanks!
COPY assumes that each newline indicates a new record. Unfortunately, the Google Maps DistanceMatrix API is pretty-printing your response which means that it comes through as 23 rows, none of which are valid JSON.
You can get around this by piping the curl response through something like jq.
copy imports(info) from program 'curl "https://maps.googleapis.com/maps/api/distancematrix/json?units=imperial&origins=London&destinations=Paris&key=<my_key>" --insecure | /usr/local/bin/jq "." -c'
jq has lots of useful features if you want to massage the response a bit more before stashing it in the database.

How to load OSM (GeoJSON) data to ArangoDB?

How I can load OSM data to ArangoDB?
I loaded data sed named luxembourg-latest.osm.pbf from OSM, than converted it to JSON with OSMTOGEOJSON, after I tried to load result geojson to ArangoDB with next command: arangoimp --file out.json --collection lux1 --server.database geodb and got hude list of errors:
...
2017-03-17T12:44:28Z [7712] WARNING at position 719386: invalid JSON type (expecting object, probably parse error), offending context: ],
2017-03-17T12:44:28Z [7712] WARNING at position 719387: invalid JSON type (expecting object, probably parse error), offending context: [
2017-03-17T12:44:28Z [7712] WARNING at position 719388: invalid JSON type (expecting object, probably parse error), offending context: 5.867441,
...
What I am doing wrong?
upd: it's seems that converter osm2json converter should be run with option osmtogeojson --ndjson that produce items not as single Json, but in line by line mode.
As #dmitry-bubnenkov already found out, --ndjson is required to produce the right input for ArangoImp.
One has to know here, that ArangoImp expects a JSON-Subset (since it doesn't parse the json on its own) dubbed as JSONL.
Thus, Each line of the JSON-File is expected to become one json document in the collection after the import. To maximize performance and simplify the implementation, The json is not completely parsed before sending it to the server.
It tries to chop the JSON into chunks with the maximum request size that the server permits. It leans on the JSONL-line endings to isolate possible chunks.
However, the server expects valid JSON for sure. Sending the chopped part to the server with possibly incomplete JSON documents will lead to parse errors on the server, which is the error message you saw in your output.