Importing CSV file in MVC and converting it in JSON using C# - json

I am importing .CSV file from an angular app into MVC and i am able to get the files like this
Int32 strLen, strRead;
System.IO.Stream stream = Request.InputStream;
strLen = Convert.ToInt32(stream.Length);
byte[] strArr = new byte[strLen];
strRead = stream.Read(strArr, 0, strLen);
here the files which is being imported is converted into byte[] because i am reading the file using
System.IO.Stream stream = Request.InputStream
Then i convert it into string like this
string a = System.Text.Encoding.UTF8.GetString(strArr);
and try to split the content and retrieve the data but it becomes very complex, i wonder if there is any alternate way for it. In a simple .CSV file like this
I get the result after converting the byte[] to string like this
and once i apply logic for splitting the string and retrieving the data, the logic gets very messy like this
Is there any efficinet way where i can convert the imported .CSV file to JSON

Save stream as text file in to the TEMP folder.
Use any parcer for working with CSV file. (Example FileHelpers)
Use any Json helper to convert it to the output format. (Example: newtonsoft)

You can use Cinchoo ETL - an open source library, to convert CSV to JSON easily.
using (var parser = new ChoCSVReader("IgnoreLineFile1.csv")
.WithField("PolicyNumber", 1)
.WithField("VinNumber", 2)
.Configure(c => c.IgnoreEmptyLine = true)
.Configure(c => c.ColumnCountStrict = true)
)
{
using (var writer = new ChoJSONWriter("ignoreLineFile1.json")
.WithField("PolicyNumber", fieldName: "Policy Number")
.WithField("VinNumber", fieldName: "Vin Number")
)
writer.Write(parser.Skip(1));
}
In above, you can pass stream to the reader and writer as well for your requirement.
Hope this will help.
Disclaimer: I'm the author of this library.

Related

Dart CSV Writing

Hey so I haven't really messed around with it too much, but I was wondering if there was actually a way (before I go down a neverending rabbit hole) to read and write to CSV files in Dart/Flutter? I need to write to the files, not necessarily read them, and I'm willing to go to quite extreme lengths to do so. Any library works, built-in functions are even better. Any and all help is appreciated!
Use package csv from https://pub.dartlang.org/packages/csv
If you have a List<List<dynamic>> items that needs to convert into csv,
String csv = const ListToCsvConverter().convert(yourListOfLists);
If you want to write the csv to a file,
/// Write to a file
final directory = await getApplicationDocumentsDirectory();
final pathOfTheFileToWrite = directory.path + "/myCsvFile.csv";
File file = await File(pathOfTheFileToWrite);
file.writeAsString(csv);
Also, if you want to read a csv file directly into list<list<dynamic>>
final input = new File('a/csv/file.txt').openRead();
final fields = await input.transform(UTF8.decoder).transform(csvCodec.decoder).toList();
According to new packages and guidelines(2019) use following code
package csvfrom https://pub.dartlang.org/packages/csv.
If you have a List<List<dynamic>> items that needs to convert into csv,
String csv = const ListToCsvConverter().convert(yourListOfLists);
Then you can write the string to a file using file operations.
file.writeAsString('$csv');
Also, if you want to read a csv file directly into list<list<dynamic>>
final input = new File('a/csv/file.txt').openRead();
final fields = await input.transform(utf8.decoder).transform(new CsvToListConverter()).toList();

How can I continuously read a CSV file in Flink and remove the header

I am working with Flink streaming API and I want to continuously read CSV files from a folder, ignore the header and convert each row in the CSV file into a Java class (POJO). After all this processing, I should obtain a stream of Java objects(POJOs).
So far, I do the following to partially achieve the behavior(code below):
read the CSV files as regular text files, continuously
get a stream of strings from the CSV files
convert the stream of strings to a stream of Java objects
String path = "/home/cosmin/Projects/flink_projects/flink-java-project/data/";
TextInputFormat format = new TextInputFormat(
new org.apache.flink.core.fs.Path(path));
DataStream<String> inputStream = streamEnv.readFile(format, path, FileProcessingMode.PROCESS_CONTINUOUSLY, 100);
DataStream<MyEvent> parsedStream = inputStream
.map((line) -> {
String[] cells = line.split(",");
MyEvent event = new MyEvent(cells[1], cells[2], cells[3]);
return event;
});
However, with this I don't manage to remove the header line in each CSV file.
I have read that I can build a custom connector for reading CSV files by using createInput() or addSource () methods on the StreamExecutionEnvironment class.
Can you help with some guidance on how to achieve this, as I haven't found any examples beyond the Javadoc?
You could chain a filter function before your map function to filter out header lines
inputStream.filter(new FilterFunction<String>() {
public boolean filter(String line) {
if (line.contains("some header identifier")) return false;
else return true;
}
}).map(...) <Your map function as before>

Python convert json file to html

I have a bunch of Pandas data frames. I want to view them in HTML (and also want the json). So, this is what I did:
masterDF = concatenated all dfs (pd.concat([df1, df2, df3, ..])
masterDf.to_json(jsonFile, orient = 'records') => this gives a valid json file, but, in a list format.
htmlStr = json2html.convert(json = jsonString)
htmlFile = write htmlStr to a myFile.html.
The json file looks like this:
[{"A":1458000000000,"B":300,"C":1,"sid":101,"D":323.4775570025,"score":0.0726},{"A":1458604800000,"B":6767,"C":1,"sid":101,"D":321.8098393263,"score":0.9524},{"A":1458345600000,"B":9999,"C":3,"sid":29987,"D":125.6096891766,"score":0.9874},{"A":1457827200000,"B":3110,"C":2,"sid":787623,"D":3010.9544668798,"score":0.0318}]
Problem I am facing:
pd.to_json outputs a jsonfile with [] format. Like a list. I am unable to use this json file to load. Like this:
with open(jsonFile) as json_data:
js = json.load(json_data)
htmlStr = json2html.convert(json = js)
return htmlStr
Is there a way to load a json-file like the above and convert to html?
why not use pandas.DataFrame.to_html? (http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_html.html)

How to read and ouput Json file with groovy

I am trying to: 1. read a json file with groovy; 2. change the object a little bit; 3. output/override as a new json file.
so far, I know we can use import groovy.json.JsonSlurper;, but I searched some samples, all for parse the .txt file (convert txt to json format object) like this:
def inputFile = file('json_input/' + fileName)
def inputJson = new JsonSlurper().parseText(inputFile.text)
print out...
but not read json file directly.
Is there any simple way to read and get the json object from a .json file so that I can make some changes of the data?
Thanks!

Append data to existing file in Windows Store 8 using JSON

I have created an application in which I am inserting data to the file. It is working fine. Following is my code:
private async void btnSearch_Click(object sender, RoutedEventArgs e)
{
UserDetails details = new UserDetails
{
Name= TxtName.Text,
Course= TxtCouse.Text,
City=TxtCity.Text
};
string jsonContents = JsonConvert.SerializeObject(details);
StorageFolder localFolder = await ApplicationData.Current.LocalFolder.CreateFolderAsync("Storage", CreationCollisionOption.ReplaceExisting); ;
StorageFile textFile = await localFolder.CreateFileAsync("UserDetails.txt", CreationCollisionOption.ReplaceExisting);
using (IRandomAccessStream textStream = await textFile.OpenAsync(FileAccessMode.ReadWrite))
{
// write the JSON string!
using (DataWriter textWriter = new DataWriter(textStream))
{
textWriter.WriteString(jsonContents);
await textWriter.StoreAsync();
}
}
this.Frame.Navigate(typeof(BlankPage1));
}
Now I want that, when a user enter new data the data will append to the same existing file.
Appending data to a JSON text file would mean doing some parsing of the file to find the correct location to insert the text. That is, because JSON is structured with {} delimiters, it's not a simple matter of just appending text to the end of the file.
Given that your data doesn't look that large, the easiest thing to do is to deserialize the JSON from the existing file into memory, add your additional properties to that data structure, and then serialize back to JSON. In that case you probably just want to maintain the structure in memory during the app session, and just overwrite the file with new data whenever you need to. But of course you could also reopen the file, read/parse the JSON into memory, and then rewrite the contents.