Context:
For a data pipeline we need to ingest excel spreadsheets directly into foundry (arriving via email). In order to avoid any manual handling error, we'd like to build a small slate app that basically just uploads an excel sheet and automatically appends it to an existing dataset (given schema, headers, etc.).
Unfortunately, there is very little documentation on the "File Import" widget or the API that gets called when drag and dropping a file into a folder.
Idea: Is there a way of uploading a file with slate? Could this file then be added to a dataset, similarly as with the prompt that opens when dropping it into a folder?
You actually don't have to build a Slate app to do this! Datasets that are made up of underlying .csv files support new additions of files directly.
Note: All of the following screenshots are from the dataset preview page.
For example, the following dataset I created from 4 .csv files:
And I can click on the Import button in the top right to add in more files (with the same schema, or not. Depends on if you want to strictly adhere to your applied schema.
If you have already applied a schema, you can also simply Import new files on top of the dataset, but the schemas of the files must exactly match those already present, otherwise your dataset will fail when attempted to be read.
Related
I'm trying to save several of my queries as XML files in order to re-assemble my database in another location where the only viable transfer method is text or XML files via email (long story).
When I use the built-in export function, Access allows me to select a save location and nest the schema inside of the XML file, and then says that the export was completed successfully. The file is not in the destination folder, and no error was thrown.
This only happens when exporting bound queries. Other Access elements (tables and forms, for example) export just fine.
If I watch the folder during the export process, I see a file appear very briefly, and then dissapear. Has anyone else experienced this?
Within Jupyter, there's two options to export notebooks to other formats, using nbconvert either on the command-line or as a library. Both require you to pass the original notebook as input (either as a file name or JSON data).
What I would like to do is essentially automatic, self-contained report generation, i.e. use the nbconvert library within a notebook, to export that same notebook to HTML/PDF on execution. I envisage the export code in the final cell of the document, with instructions to ignore that last cell on export.
My problem is in getting the data. Within the notebook I can access the code cells through In[1], Out[2] etc, but AFAIK there is no such functionality for markdown cells. There's also no (official) method to grab the name of the notebook (no __file__ attribute) to pass information that way.
There's plenty of examples to grab all the cell data from other notebooks - but is there a way to access all the cell data from the current notebook?
You can do this with the notebook filename and nbconvert, as you surmise. Doing some stuff client side might require extensions, one I can envision is a server extension to do the convert, and a bit of JS you write in the last code cell which renders a button, for example, and posts the notebook back to the convert server extension. Hiding certain cells (say, the Js cell) you can do in the convert extension, or with cell metadata and nbconvert templates. Given the flexibility of the architecture, there are quite a few ways to do this
Using Microsoft Visual Studio Community 2015.
Goal of project
-create "*\temp\email" directory
-start program to extract all emails that include xls attachments to the previously created folder
-use for each loop to cycle through each file in the folder, process, and shift to sql table.
The problem I am running into is caused by either a blank excel document (which is occasionally sent from a remote location) or some of the original xls reports only contain 5 columns instead of 6 that I have mapped now. Is there any way to separate files that include the correct columns from those that do not match?
** as Long as these two problems do not exist I can run the ssis package and everything runs without issue.
Control flow;
File System Task (creates directory --->Execute Process Task (xls extraction)-->ForEach Loop(Data flow Task "email2Sql")
Data Flow;
Excel Source (uses expression ExcelFilePath,#user:filepath) delay validation ==true
(columns are initially set to f1-f6 and are mapped to for ex. a,b,c,d,e,f. The Older files that get mixed in only include a,b,c,d,e.) This is where I want to be able to separate the xls files
Conditional Transformation split (column names are not in row 1, this helps remove "null" values)
Ole Db destination (sql table)
Sorry for the amount of reading, but for the first post I tried to include anything that I thought may be relevant.
There are some tools out there which would allow you to open the excel doc and read it. However, I think the simplest thing to do would be to use SSIS out of the box:
1 - add a file system task after the data flow which reads the file.
2 - Make the precedence constraint from the data flow to the file system task "failure." This will cause that to only fire when the data flow task fails.
3 - set the file task to move the "bad" files to another folder
This will allow you to loop through all the files and move the failed ones. Ultimately, the package will end in failure. If you don't want that behavior you can change the ForceExecutionResult property to be success. However, it might be good to know that there were problems with some files so that they can be addressed.
m
I want to create .csv files with the Report Generation Toolkit in Labview.
They must actually be .csv files which can be opened with Notepad or something similar.
Creating a .csv is not that hard, it's just a matter of adding the extension to the file name that's going to be created.
If I create a .csv file this way it opens nicely in excel just the way it should, but if I open it in Notepad it shows all kind of characters and it doesn't even come close to the data I wrote to the file.
I create the files with the Labview code below:
Link to image (can't post image yet because I've got to few points)
I know .csv files can be created with the Write to Spreadsheet VI but I would like to use the Report Generation Toolkit because it's pretty easy to add columns and rows to the file and that is something I really need.
you can use the Robust CSV package on the lavag.org forum to read and write 2D arrays to CSV files.
http://lavag.org/files/file/239-robust-csv/
Calling a file "csv" does not make it a CSV file. I never used the toolkit to generate an Excel file, but I'm assuming it creates an XLS or XLSX file, regardless of what extension you give it, which is why you're seeing gibberish (probably XLS, since it's been around for a while and I believe XLSX is XML, not binary).
I'm not sure what your problem is with the write spreadsheet VI. It has an append input, so I assume you can use that to at least add rows directly to a file, although I can't say I ever tried it. I would prefer handling all the data in memory explicitly, where you can easily use the array functions to add rows or columns to the array and then overwrite the entire file.
I have a csv imported into my Hyperion v8.3 bqy file. I have some custom columns and a pivot already created. I just want to refresh the data. In the past, I would hit Process Current and it would direct me to my computer and I could select the csv file to update from. Now it will not do that. It doesn't go to my computer at all.
Any ideas?
Eric,
I'm not a power user but I accomplish the same thing by ensuring that there is a file with the same name in the same location from which the original csv was imported and "Processing All". This allows me to update the data and import it into my bqy and automatically update any reporting based on the csv.
Don't know if this will help or not.
Dennis
dennis.van.camp#vanderlande.com
When you import a file as a Section into Hyperion, it maintains a link to the existing file for the exact path and file name. When that link is broken and that section gets a Process or Refresh command, is the only time it will prompt you for a new file. Otherwise, it will refresh the data from the existing.
So, if you want to force it to prompt you for a new file, you have to move or rename the old file.
But, you're looking for the Pivot and Computed columns to refresh. Two things on that ...
Computed Columns: You don't have to re-import the file, if the rest of your data is current. Each column can be refreshed individually by right-click, Modify, then clicking Ok. You don't have to change any of the code; just hit okay instead of Cancel.
Pivot: In the menu, you have to set your pivot options appropriately to update when you Process (when the underlying data is Processed, really), or Manually (when you Process that section).