JSON Formatting error - json

I am getting this error while trying to import this JSON into google bigquery table
file-00000000: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. (error code: invalid)
JSON parsing error in row starting at position 0 at file: file-00000000. Start of array encountered without start of object. (error code: invalid)
This is the JSON
[{'instrument_token': 11192834, 'average_price': 8463.45, 'last_price': 8471.1, 'last_quantity': 75, 'buy_quantity': 1065150, 'volume': 5545950, 'depth': {'buy': [{'price': 8471.1, 'quantity': 300, 'orders': 131072}, {'price': 8471.0, 'quantity': 300, 'orders': 65536}, {'price': 8470.95, 'quantity': 150, 'orders': 65536}, {'price': 8470.85, 'quantity': 75, 'orders': 65536}, {'price': 8470.7, 'quantity': 225, 'orders': 65536}], 'sell': [{'price': 8471.5, 'quantity': 150, 'orders': 131072}, {'price': 8471.55, 'quantity': 375, 'orders': 327680}, {'price': 8471.8, 'quantity': 1050, 'orders': 65536}, {'price': 8472.0, 'quantity': 1050, 'orders': 327680}, {'price': 8472.1, 'quantity': 150, 'orders': 65536}]}, 'ohlc': {'high': 8484.1, 'close': 8336.45, 'low': 8422.35, 'open': 8432.75}, 'mode': 'quote', 'sell_quantity': 998475, 'tradeable': True, 'change': 1.6151959167271395}]
http://jsonformatter.org/ also gives parse error for this JSON block. Need help understanding where the formatting is wrong - this is the JSON from a rest API

This is not valid JSON. JSON uses double quotes, not single quotes. Also, True should be true.
If I had to guess, I would guess that this is Python code being passed off as JSON. :-)
I suspect that even once this is made into correct JSON, it's not the format Google BigQuery is expecting. From https://cloud.google.com/bigquery/data-formats#json_format, it looks like you should have a text file with one JSON object per line. Try just this:
{"mode": "quote", "tradeable": true, "last_quantity": 75, "buy_quantity": 1065150, "depth": {"buy": [{"quantity": 300, "orders": 131072, "price": 8471.1}, {"quantity": 300, "orders": 65536, "price": 8471.0}, {"quantity": 150, "orders": 65536, "price": 8470.95}, {"quantity": 75, "orders": 65536, "price": 8470.85}, {"quantity": 225, "orders": 65536, "price": 8470.7}], "sell": [{"quantity": 150, "orders": 131072, "price": 8471.5}, {"quantity": 375, "orders": 327680, "price": 8471.55}, {"quantity": 1050, "orders": 65536, "price": 8471.8}, {"quantity": 1050, "orders": 327680, "price": 8472.0}, {"quantity": 150, "orders": 65536, "price": 8472.1}]}, "change": 1.6151959167271395, "average_price": 8463.45, "ohlc": {"close": 8336.45, "high": 8484.1, "open": 8432.75, "low": 8422.35}, "instrument_token": 11192834, "last_price": 8471.1, "sell_quantity": 998475, "volume": 5545950}

OP has a valid JSON record but that wouldn't work with Biq Query, and here's why:
Google Big Query supports, JSON objects {}, one object per line. Check this out.
This basically means that you cannot supply list [] as json records and expect Big Query to detect it. You must always have one json object per line.
Here's a quick reference to what I am saying.
and there are more.
at last,
I highly recommend you read up the below and check out the link for more information on different forms of JSON structures, read this from the json.org

Related

JSON_SEARCH return null for json array containing only integer values [duplicate]

I have a problem with a query. In my DB I have a field that contains a JSON with this format:
[[0, 16, 22, 37, 0, 0, 0, 71, 82],
[0, 18, 0, 36, 43, 0, 60, 0, 88],
[9, 10, 0, 0, 0, 58, 69, 77, 0]]
With this query
SELECT JSON_SEARCH(NumeriJSON, 'all', 77,null, '$[*]') AS Indice FROM Cartella WHERE JSON_CONTAINS(NumeriJSON->'$[*]', '77')
I want to get the position of the number in the JSON, but it returns null. Why? The structure of JSON is valid, because JSON_CONTAINS works well.
Thanks a lot.
https://dev.mysql.com/doc/refman/8.0/en/json-search-functions.html#function_json-search says:
JSON_SEARCH(json_doc, one_or_all, search_str[, escape_char[, path] ...])
Returns the path to the given string within a JSON document.
I emboldened the word string. JSON_SEARCH() doesn't work if the values in your JSON are integers.
Your example works if you make them strings:
insert into Cartella (NumeriJson) values (
'[["0", "16", "22", "37", "0", "0", "0", "71", "82"],
["0", "18", "0", "36", "43", "0", "60", "0", "88"],
["9", "10", "0", "0", "0", "58", "69", "77", "0"]]');
mysql> select json_search(numerijson, 'all', '77', null, '$[*]') as `index`
from cartella where json_contains(numerijson->'$[*]', '"77"');
+-----------+
| index |
+-----------+
| "$[2][7]" |
+-----------+
This issue was reported as a bug in 2015, but it was closed by simply documenting that it only supports string searches.
Supporting non-strings was then reported as a new feature request in 2018. So far, there is no solution.
In the meantime, you must store integers as strings if you want to use JSON_SEARCH().
I would also recommend to stop using JSON, and instead store integers in normal rows and columns. Then searches for integers work just fine.
P.S. Please don't use the back-formation "indice." There is no such word.

Azure Stream Analytics - JSON

I am trying to pass the below json through Azure Stream analytics to an Azure SQL server. The data is coming from the Azure IOT HUB and data is coming through happily.
nodes": {
"SN0013A20041E23697": {
"firmware_version": 5,
"transmission_count": 42,
"reserve_byte": 0,
"battery_level": 3.29406,
"type": 32,
"node_id": 0,
"rssi": 9,
"mass_concentration_pm_1_0": 0.88,
"mass_concentration_pm_2_5": 1.04,
"mass_concentration_pm_4_0": 1.13,
"mass_concentration_pm_10_0": 1.17,
"number_concentration_pm_0_5": 5.73,
"number_concentration_pm_1_0": 6.92,
"number_concentration_pm_2_5": 7.07,
"number_concentration_pm_4_0": 7.09,
"number_concentration_pm_10_0": 7.09,
"typical_particle_size": 0.48,
"humidity": 45.35,
"temperature": 20.84
},
"SN0013A20041E2367B": {
"firmware_version": 5,
"transmission_count": 43,
"reserve_byte": 0,
"battery_level": 2.99782,
"type": 32,
"node_id": 0,
"rssi": 16,
"mass_concentration_pm_1_0": 1.35,
"mass_concentration_pm_2_5": 1.43,
"mass_concentration_pm_4_0": 1.43,
"mass_concentration_pm_10_0": 1.43,
"number_concentration_pm_0_5": 9.13,
"number_concentration_pm_1_0": 10.77,
"number_concentration_pm_2_5": 10.83,
"number_concentration_pm_4_0": 10.83,
"number_concentration_pm_10_0": 10.83,
"typical_particle_size": 0.41,
"humidity": 45.72,
"temperature": 20.2
I can use a query like this and it will pass through one of the devices but not the other.
SELECT
"nodes"."SN0013A20041E23697"."temperature" as Temperature
, "nodes"."SN0013A20041E23697"."humidity" as Humidity
From input
Is there a way to pass through both devices in the same query?

Getting a syntax error how do I remove quotes from json.dumps()

I'm getting information from an external api. I'm getting an error when I try to update the database as the string formatting is incorrect.
There is a mixture of Double quotes and single quotes caused by the single quote O' in the first line which must tell python to use Double quotes.
To get over this I tried json.dumps() and that gives me the result below and causes a syntax error when inserting to the Database, because of the single quotes around every object.
[
'{"author_name": "Michael P O\'Shaughnessy", "rating": 4, "text": "Stayed for a midweek"}',
'{"author_name": "camille williams", "rating": 5, "text": "We looked around and found"}',
'{"author_name": "natasha sevrugina", "rating": 5, "text": "Stayed at the "}',
'{"author_name": "niamh kelly", "rating": 5, "text": "Great hotel in a central location"}',
'{"author_name": "Janette Wade", "rating": 5, "text": "Excellent staff
\\ud83d\\udc4f\\n\\nJanette\\nSpiritual Ceremonies"}'
]
It should look like this to achieve, valid Json
[
{"author_name": "Michael P O\' Shaughnessy", "rating": 4, "text": "Stayed for a midweek night"},
{"author_name": "camille williams", "rating": 5, "text": "We looked around and found"}
]
With out jsondumps the return from the api is, notice only the first line is changed
[
{'author_name': "Michael P O'Shaughnessy", 'rating': 4, 'text': "Stayed for a midweek night"}
{'author_name': 'camille williams', 'rating': 5, 'text': 'We looked around and found that was great."},
{'author_name': 'natasha sevrugina', 'rating': 5, 'text': 'Stayed at the Hotel '},
{'author_name': 'niamh kelly', 'rating': 5, 'text': 'Great hotel in a central location. '},
{'author_name': 'Janette Wade', 'rating': 5, 'text': 'Modern rooms. Great room service. '}
]
Which also gives a syntax error because of the mixture of single, double and single quotes.
Here are my calls to the api:
review_author = dictionary['result']['reviews'][i]['author_name']
review_rating = dictionary['result']['reviews'][i]['rating']
review_text = dictionary['result']['reviews'][i]['text']
dict_keys = ["author_name", "rating", "text"]
res_dict = {dict_keys[0]: review_author, dict_keys[1]: review_rating, dict_keys[2]: review_text}
bus_reviews.append(json.dumps(res_dict))
How can I remove the single quotes around json.dump()
I changed this
bus_reviews.append(json.dumps(res_dict))
to
bus_reviews.append(res_dict)
and I put the return statement to include the json dumps
return json.dumps(bus_reviews)

Python open() as write giving wrong encoding in json

I have a python dictionary that looks like this:
{'id': 5677240, 'name': 'Conjunto de Panelas antiaderentes com 05 Peças Paris', 'quantity': 21, 'price': '192.84', 'category': 'Panelas'}
But when I try to write it to a JSON file, there's a little mess on the encoding:
{"id": 5677240, "name": "Conjunto de Panelas antiaderentes com 05 Pe\u00e7as Paris", "quantity": 21, "price": 192.84, "category": "Panelas"}
I've already tried putting encoding='utf-8 and using locale as false, but neither helped me.
You will have to disable ensure_ascii while writing to a file.
import json
data = {'id': 5677240, 'name': 'Conjunto de Panelas antiaderentes com 05 Peças Paris', 'quantity': 21, 'price': '192.84', 'category': 'Panelas'}
with open("a.json", "w") as f:
json.dump(data, f, ensure_ascii=False)
Output in file:
{"id": 5677240, "name": "Conjunto de Panelas antiaderentes com 05 Peças Paris", "quantity": 21, "price": "192.84", "category": "Panelas"}

reading and editing json file

I am new to Python and trying to edit a .json file that looks like the dictionary below:
{
"name": "MB_NDE_AX667_ECU[500-2000]",
"physical_quantity": "acceleration",
"unit": "m/s2",
"okrangelow": 0,
"okrangehigh": 10,
"input": "ch2_ECU1.rms",
"history": {
"ds": 30,
"timer": 24,
"files": 30
},
"reg": 44
},
My goal is to get another json file that will have the dictionary restructured to single line dictionary, e.g:
{"name":"MB_NDE_AX667_ECU[500-2000]","physical_quantity":"acceleration",......}
How could I do this?
Thanks