I have a bunch of Pandas data frames. I want to view them in HTML (and also want the json). So, this is what I did:
masterDF = concatenated all dfs (pd.concat([df1, df2, df3, ..])
masterDf.to_json(jsonFile, orient = 'records') => this gives a valid json file, but, in a list format.
htmlStr = json2html.convert(json = jsonString)
htmlFile = write htmlStr to a myFile.html.
The json file looks like this:
[{"A":1458000000000,"B":300,"C":1,"sid":101,"D":323.4775570025,"score":0.0726},{"A":1458604800000,"B":6767,"C":1,"sid":101,"D":321.8098393263,"score":0.9524},{"A":1458345600000,"B":9999,"C":3,"sid":29987,"D":125.6096891766,"score":0.9874},{"A":1457827200000,"B":3110,"C":2,"sid":787623,"D":3010.9544668798,"score":0.0318}]
Problem I am facing:
pd.to_json outputs a jsonfile with [] format. Like a list. I am unable to use this json file to load. Like this:
with open(jsonFile) as json_data:
js = json.load(json_data)
htmlStr = json2html.convert(json = js)
return htmlStr
Is there a way to load a json-file like the above and convert to html?
why not use pandas.DataFrame.to_html? (http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_html.html)
Related
I am making a curl call to rest api visa curl in groovy. Response is coming fine but the response is very large, it is a 17MB of data, following is my script :
def converter = "curl.......'"
def initialSize = 4096
def out = new ByteArrayOutputStream(initialSize)
def err = new ByteArrayOutputStream(initialSize)
def process = [ 'bash', '-c', converter].execute()
process.consumeProcessOutput(out, err)
process.waitFor()
Curl response is coming fine, when I print response on console ,store in variable out, it gives response data where it is not neat json as I see some "/n" characters. When I write this to file then I dont see any new line and neat json, all I see data in one line in key value format.
{"key1":"value1","key2":"value2",} in one huge line only
This is when i view in sublime. Now I want to convert this to pretty json and write neatly into file.I tried following to approaches but both prints empty ({ }) in console and in file.
def json = JsonOutput.toJson(out)
println new JsonBuilder(out).toPrettyString()
What did I miss?
I am trying to use groovy libraries only.
UPDATE:
As i try to debug, i found that it may be because all JSON parsers expect string but my output is ByteArrayOutputStream. But now how can I convert the out to string ? I tried out.toString and out.text, it does not work.
Use StringWriter instead of ByteArrayOutputStream
Then JsonOutput.prettyPrint( stringWriter.toString() )
I am importing .CSV file from an angular app into MVC and i am able to get the files like this
Int32 strLen, strRead;
System.IO.Stream stream = Request.InputStream;
strLen = Convert.ToInt32(stream.Length);
byte[] strArr = new byte[strLen];
strRead = stream.Read(strArr, 0, strLen);
here the files which is being imported is converted into byte[] because i am reading the file using
System.IO.Stream stream = Request.InputStream
Then i convert it into string like this
string a = System.Text.Encoding.UTF8.GetString(strArr);
and try to split the content and retrieve the data but it becomes very complex, i wonder if there is any alternate way for it. In a simple .CSV file like this
I get the result after converting the byte[] to string like this
and once i apply logic for splitting the string and retrieving the data, the logic gets very messy like this
Is there any efficinet way where i can convert the imported .CSV file to JSON
Save stream as text file in to the TEMP folder.
Use any parcer for working with CSV file. (Example FileHelpers)
Use any Json helper to convert it to the output format. (Example: newtonsoft)
You can use Cinchoo ETL - an open source library, to convert CSV to JSON easily.
using (var parser = new ChoCSVReader("IgnoreLineFile1.csv")
.WithField("PolicyNumber", 1)
.WithField("VinNumber", 2)
.Configure(c => c.IgnoreEmptyLine = true)
.Configure(c => c.ColumnCountStrict = true)
)
{
using (var writer = new ChoJSONWriter("ignoreLineFile1.json")
.WithField("PolicyNumber", fieldName: "Policy Number")
.WithField("VinNumber", fieldName: "Vin Number")
)
writer.Write(parser.Skip(1));
}
In above, you can pass stream to the reader and writer as well for your requirement.
Hope this will help.
Disclaimer: I'm the author of this library.
I have a file with UTF-8 encoding.
I write a groovy script to load a file with a JSON structure, modify it and save it:
def originPreviewFilePath = "./xxx.json"
//target the file
def originFile = new File(originPreviewFilePath)
//load the UTF8 data file as a JSON structure
def originPreview = new JsonSlurper().parse(originFile,'UTF-8')
//Here is my own code to modify originPreview
//Convert the structure to JSON Text
def resultPreviewJson = JsonOutput.toJson(originPreview)
//Beautify JSON Text (Indent)
def finalFileData = JsonOutput.prettyPrint(resultPreviewJson)
//save the JSONText
new File(resultPreviewFilePath).write(finalFileData, 'UTF-8')
The problem is that JsonOutput.toJson transforms UTF-8 data to UNICODE. I don't understand why JsonSlurper().parse can use UTF-8 but not JsonOutput.toJson?
How to have JsonOutput.toJson use UTF-8? I need to have the exact inverse of JsonSlurper().parse
In case anyone is still struggling with this, the solution is to disable unicode escaping:
new JsonGenerator.Options()
.disableUnicodeEscaping()
.build()
.toJson(object)
This worked for me in Groovy 3:
StringEscapeUtils.unescapeJavaScript(
JsonOutput.prettyPrint(resultPreviewJson)
)
I believe that the encoding is applied at the incorrect statement while reading itself.
Change below statements from :
def originFile = new File(originPreviewFilePath)
def originPreview = new JsonSlurper().parse(originFile,'UTF-8')
To:
def originFile = new File(originPreviewFilePath).getText('UTF-8')
def originPreview = new JsonSlurper().parseText(originFile)
Is there any way to get the JSON Code from a HTML Website? If I use a code like those:
r = requests.get(url)
if r.status_code == 200:
r.json()
result = json.loads(r)
I will always have an error at HTML pages. What modules should I use for getting HTML pages to an Python-dictionary?
You only have one error in your code.
Once you did
r.json()
You didn't assign it to anything. To correct this problem just change your previous line with the line below and you should be good :).
r = r.json()
Not all webpages responds with JSON data. But you can use json.loads to print data in json string. You can also use r.contents or r.text to know the type of data coming from webpage. Most of the time it will be just HTML Content
import requests
import json
r = requests.get('http://www.google.com')
# you can use r.content to print the webpage data
print r.content
# json.loads(data) `json_loads` is to convert data into `json string`
print json.loads(r.content)
json.loads will go into ValueError if the data cannot be decoded into JSON Object
I am trying to: 1. read a json file with groovy; 2. change the object a little bit; 3. output/override as a new json file.
so far, I know we can use import groovy.json.JsonSlurper;, but I searched some samples, all for parse the .txt file (convert txt to json format object) like this:
def inputFile = file('json_input/' + fileName)
def inputJson = new JsonSlurper().parseText(inputFile.text)
print out...
but not read json file directly.
Is there any simple way to read and get the json object from a .json file so that I can make some changes of the data?
Thanks!