I want to create .csv files with the Report Generation Toolkit in Labview.
They must actually be .csv files which can be opened with Notepad or something similar.
Creating a .csv is not that hard, it's just a matter of adding the extension to the file name that's going to be created.
If I create a .csv file this way it opens nicely in excel just the way it should, but if I open it in Notepad it shows all kind of characters and it doesn't even come close to the data I wrote to the file.
I create the files with the Labview code below:
Link to image (can't post image yet because I've got to few points)
I know .csv files can be created with the Write to Spreadsheet VI but I would like to use the Report Generation Toolkit because it's pretty easy to add columns and rows to the file and that is something I really need.
you can use the Robust CSV package on the lavag.org forum to read and write 2D arrays to CSV files.
http://lavag.org/files/file/239-robust-csv/
Calling a file "csv" does not make it a CSV file. I never used the toolkit to generate an Excel file, but I'm assuming it creates an XLS or XLSX file, regardless of what extension you give it, which is why you're seeing gibberish (probably XLS, since it's been around for a while and I believe XLSX is XML, not binary).
I'm not sure what your problem is with the write spreadsheet VI. It has an append input, so I assume you can use that to at least add rows directly to a file, although I can't say I ever tried it. I would prefer handling all the data in memory explicitly, where you can easily use the array functions to add rows or columns to the array and then overwrite the entire file.
Related
I am trying to do some optimization in ADF. Setup is a third-party tool copies one JSON file per object to a BLOB storage container. These feed to a Mapping Data Flow. The individual files written by the third party tool work great. If I copy these files to a different BLOB folder using an Azure Copy Data activity, the MDF can no longer parse the files and gives an error: "JSON parsing error, unsupported encoding or multiline." I started this with a Merge Files, but outcome is same regardless of copy behavior I choose.
2ND EDIT: After another day's work, I have found that the Copy Activity Merge File from JSON to JSON definitely adds an EOL character to each single JSON object as it gets imported to the Merge file. I have also found that the MDF fails definitely with those EOL characters in the Merge file. If I remove all EOL characters from the Merge file, the same MDF will work. For me, this is a bug. The copy activity is adding a character that breaks the MDF. There seems to be a second issue in some of my data that doesn't fail as an individual file but does when concatenated that breaks the MDF when I try to pull all the files together, but I have tested the basic behavior on 1-5000 files and been able to repeat the fail/success tests.
I took the original file, and the copied file, ran them through all of sorts of test, what I eventually found when I dump into Notepad++:
Copied file:
{"CustomerMasterData":{"Customer":[{"ID":"123456","name":"Customer Name",}]}}\r\n
Original file:
{"CustomerMasterData":{"Customer":[{"ID":"123456","name":"Customer Name",}]}}\n
If I change the copied file from ending with \r\n to \n, the MDF can read the file again. What is going on here? And how do I change the file write behavior or the MDF settings so that I can concatenate or copy files without the CRLF?
EDIT: NEW INFORMATION -- It seems on further review like maybe the minification/whitespace removal is the culprit. If I download the file created by the ADF copy and format it using a JSON formatter, it works. Maybe the CRLF -> LF masked something else. I'm not sure what to do at this point, but its super frustrating.
Other possibly relevant information:
Both the source and sink JSON datasets are set to use UTF-8 (not default(UTF-8), although I tried that). Would a different encoding fix this?
I have tried remapping schemas, creating new data sets, creating new Mapping Data Flows, still get the same error.
EDITED for clarity based on comments:
In the case of a single JSON element in a file, I can get this to work -- data preview returns same success or failure as pipeline when run
In the case of multiple documents merged by ADF I get the below instead. It seems on further review like maybe the minification/whitespace removal is the culprit. If I download the file created by the ADF copy and format it using a JSON formatter, it works. Maybe the CRLF -> LF masked something else. I'm not sure what to do at this point, but its super frustrating.
Repro: Create any valid JSON as a single file, put it in blob storage, use it as a source in a mapping data flow, to do any sink operation. Create a second file with same schema, get them both to run in same flow using wildcard paths. Use a Copy Activity with Merge Files as the Sink Copy Activity and Array of Objects as the File pattern. Try to make your MDF use this new file. If it fails, download the file created by ADF, run it through a formatter (I have used both VS Code -> "Format Document" from standard VS Code JSON extension, and VS 2019 "Unminify" command) and reupload... It should work now.
don't know if you already solved the problem: I came across the exact same problem 3 days ago and after several tries I found a solution:
in the copy data activity under sink settings, use "set of objects" (instead of "array of objects") under File Pattern, so that the merged big JSON has the value of the original small JSON files written per line
in the MDF after setting up the wildcard paths with the *.json pattern, under JSON Settings select: Document per line as the Document form.
After that you should be good to go, as least it solved my problem. The automatic written CRLF in "array of objects" setting in the copy data activity should be a default setting and MSFT should provide the option to omit it in the settings in the future.
According to my test:
1.copy data activity can't change unix(LF) to windows(CRLF).
2.MDF can also parse unix(LF) file and windows(CRLF) file.
Maybe there is something else wrong.
By the way,I see there is a comma after "name":"Customer Name" in your Original file,I delete it before my test.
Using exiftool, I have exported image file data to a CSV file. I have manipulated that data and now I want to import the new values to the original files. Everything works as I would expect, except that I want also to rename images to new file names contained in the CSV file (those new names were generated manually as well as by programs; they could not be generated by a rule). I know how to rename files using data that is in the source image file (e.g. I've found advice on how to incorporate the camera model name into the file name), but I don't know how to rename the source image file to the name that I have specified in the CSV file.
It would be simple enough to do this renaming separately from exiftool, but I'm curious to know whether exiftool can do it. It seems to be able to do pretty much anything else.
According to this thread, Phil Harvey (creator of exiftool) says it isn't possible to rename files from a CSV file. It's "a feature to prevent people from messing up their files unintentionally."
I am able to find some information on how to read a csv file on a computer but is there any way I can modify one? In my chrome extension I need to add data to each row one at a time after scraping some websites. Is there any better way then read csv, store data as variable and rewrite is everytime? This becomes problematic when the file gets large. I am looking for a way to “append ” to a existing file or a work around. Any suggestions appreciated.
Update: From comment I see it is not possible to read from file system. But is there anyway to read from within the extension directory? How should I do so if the csv file is included with in the zip file of the extension? Can I access them somehow? Code snippets would be helpful.
I'm in the middle of creating something which might help you. Right now you can upload the CSV file and append a "modifier". You can adjust the code according to your requirement. Here's the repo https://github.com/amanrOnly/CSV_Modifier
I have used the "extract" command, but it never was able to find as much information as FOCA found on these excel spreadsheets I am dealing with.
For example, I am using the FOCA application to harvest and download files from the web. Afterwards, it is extracting metadata from all of the files.
With regards to excel files, it appears that these files are containing more metadata than the average pdf file. That being said, FOCA is able to detect printer names, email addresses, and a few other things that are stored within this spreadsheet file. However, I cannot find any way to get this same information in Linux using the "extract" command.
Anyone know a way to extract files within Linux and grab ALL of its metadata? Seems like the extract command may be limited from what I understand.
Thanks,
Excel files store a lot of meta data within the file, so you would have to parse the file itself to get at it. Since you're on Linux and can't use the Excel interop, you could try to use an Excel library like ExcelWriter or something similar. ExcelWriter is written for .Net, so you'd have to use mono.
lately I've been into combining multiple OpenXML speadsheets via PHPExcel which
showed me that this framework has certain issues which makes it pretty much unusable
for what I want to do (my related SO question).
To make it short: it's hard to guarantee that all formatting features of Excel 2007 will
persist a file merge performed with that particular framework.
Anways, now I'm thinking of a more general approach. I want to open a template XLSX
which contains various formatting and add some plain alphanumeric data worksheets 'at the end' of the workbook.
Is sensefully possible to do the following:
unzip template XLSX
parse XML files
add worksheets
save xml files
rezip files to get valuid XLSX
Any hints or experiences would be highly appreciated.
thanks in advance
K
I haven't worked with .xlsx too much, but I've altered .docx files by manually adding and editing the XML.
The biggest concern with adding new parts to a document is to make sure you update the .rels files. The best way to figure out what needs to be updated is to create a new .xlsx document in Excel, add a worksheet, save the file and then unzip it to see what has changed. You can also use the DocumentReflector tool that comes with the OpenXML SDK if you want to see the internals of the file without having to unzip it.
I found the OpenXML reference manual very helpful when hand editing files because it tells you what elements you have to keep and what elements are optional to make a valid document. It makes it easier to work with when you can remove some of the extraneous elements that Excel adds automatically.