I use requests to download json files. However, there are square brackets ([]) enclosing the json elements. And these brackets prevent me from importing the json file into Postgresql (jsonb field). How can I remove the square brackets?
My codes to download the json file:
with open ("C:\DATA\outputfile.json", "a") as f:
json.dump(response,f)
Thank you in advance for your help!
PostgreSQL can't accept JSON array structures, to import you must first convert the file to New Line Delimited JSON (NDJSON).
If you are a developer or have access to scripting tools, then you can do this by deserializing the JSON and manually serializing it into a flat NDJSON file.
If the post is updated with a language or scripting tag then I can provide a specific example if you need it.
There are online tools you can use like this one that can help: https://konbert.com/convert/json/to/ndjson
Related
I have a really frustrating error trying to parse basic Json read from Blob Storage using a data set within ADF
My Json is below
[{"Bid":0.197514880839,"BaseCurrency":"AED"}
,{"Bid":0.535403560434,"BaseCurrency":"AUD"}
,{"Bid":0.351998712241,"BaseCurrency":"BBD"}
,{"Bid":0.573128306234,"BaseCurrency":"CAD"}
,{"Bid":0.787556605631,"BaseCurrency":"CHF"}
,{"Bid":0.0009212964,"BaseCurrency":"CLP"}
,{"Bid":0.115389497248,"BaseCurrency":"DKK"}
]
I have tried all 3 Json source settings and every one of them gives the error
Malformed records are detected in schema inference. Parse Mode: FAILFAST
The 3 settings as in
Single Document
Array Of Documents
Document Per Line
Can anyone help? I just simply need this to be a list of objects thats it!
Paul
It should work for the JSON setting - Array of documents.
We face this issue when we have the json file with UTF-8 BOM encoding, the ADF DataFlow is unable to parse such files. You can specify the encoding as UTF-8 without encoding while creating the file, it will work.
In my case, I am using copy activity to merge and create the json file and have specified encoding as UTF-8 without BOM, and it resolved my issue.
Note: For some reason, we cant use the dataset which has "UTF-8 without BOM" encoding in DataFlow, in that case, you can create two datasets one with default UTF-8 encoding (which will be used in DataFlow) and one with UTF-8 without BOM(which will be used in copy activity sink/while creating a file).
Thank you.
Hi I'm trying to parse any of the files from the link underneath. I've tried reaching out to the owner of the data dumps, but nothing works in trying to parse the files as proper JSON files. No program we use (Power BI, Jupyter, Excel) anything really, wants to recognise the files as JSON and we can't figure out why this might be. I was wondering if anyone could help figuring out what the issue is here as this dataset is very interesting to me and my co-students. I hope I'm using the word 'parsing' correctly.
The link to the data dumps is linked underneath:
https://files.pushshift.io/reddit/comments/
The file I downloaded (I just tried one at random) was handled just fine by jq, my preferred command-line tool for processing JSON files.
jq accepts an input consisting of a sequence of JSON objects, which is what I found when I decompressed the test file. This format is commonly known as JSON lines, and many tools can handle it. The Wikipedia article on JSON streaming contains more information and a (possibly outdated) list of tools.
If your tools aren't capable of handling more than one JSON object in an input, you could turn the files into something which you can handle by adding a comma to the end of every line except the last one (since each JSON object is a single line) and then surrounding the whole input inside a pair of brackets to turn the sequence into a JSON list. Since JSON does not actually care about newlines, it would be sufficient to add a line containing [ at the beginning and a line containing ] at the end. I don't know what command-line tools you have available and are comfortable with, but the task shouldn't be too difficult.
I went through previous posts on SO and some of the answers say that a JSON file is used to send data from server to client.
Well that seems to be okay but then we can create package.json, Apidoc.json, manifest.json which do not interact with the client and server
So can someone tell me what actually is a JSON file?
JSON stands for JavaScript Object Notation. It is used to describe a data structure in a simple format. It can be a plain text file, which may be used to pass data from the server to a client, but it could be equally used to hold and consume that data at the same layer e.g. you could have a configuration file at the client side which is read an interpreted by your application.
Note also that JSON does not need to be held in a file; you could create a string variable with JSON data in it and pass this from one method to another without ever storing it in a file.
The tag definition in Stack Overflow can be found here https://stackoverflow.com/tags/json/info and further information can be found here https://www.json.org/.
JSON is a file format, just like CSV. Just because CSV is used with Microsoft Excel, does not mean that is all it is used for (just like with JSON). Just because it is common to get info from a server in JSON format, does not mean that is all JSON is used for. Do some googling before asking a question like this on Stack Overflow.
Here is an intro to JSON. JSON Intro W3Schools
In my application I need to enter multi-line data in one of the field. Using JSON file how can I pass the data in my protractor script?
var dataFile = require('../../../test-data/queues.json');
Inside the spec file you can access the same using dataFile.variableName.
To pass multi-line data you can separate the lines using \n in json file.
When you pass the data to your application via protractor it will be send as multi-line data.
Please let me know if you need any help regarding json structure.
I have a rather huge JSON file which I want to convert to a excel file so that I can use it for feature selection.
Any suggestions on steps to import the JSON file and convert it to an excel file with column names?
THnaks a lot!
I think you need to parse these json in key value pair using some programming language and store these key and value pair in excel file using coding and it will be easy way in my case.