In my Grails app, I would like admin users to be able to upload a CSV file that contains data such as:
List of users to be added to system
List of groups to be added to system
Assignment of users to groups
I have no idea how the user will generate these CSV files - most likely from Excel, Access or similar, and therefore I've no way of knowing which column will contain which data. So I'm planning to allow the user to specify which column contains users, groups, etc.
I'm wondering if there's a JavaScript component that could help with this. Ideally I'd like to implement the following:
User uploads file
In browser, user is shown first N lines of uploaded file and prompted to select the column that contains the users, groups, etc.
Column information is uploaded to server
Is there a client/server side component that could help with this, or an entirely different approach which would be superior to that outlined above?
I should emphasise that the users of this system will not be technically gifted, so expecting them to provide an XML/JSON file instead is out of the question (and you can definitely forget about asking them to call a Web Service instead of uploading a file).
Thanks,
Don
I like your solution so far, given that the users are non-technical, and that you want to be able to accept this data as a file upload, rather than have the users enter it directly into your application.
I would simply suggest that when the user uploads the file, the server returns the first five (or so) lines back to the client as an HTML table. Then you can have <select> drop-downs as the headers for each column, with the pre-set options you're looking for. You can validate that the user has assigned all available options to each column (use JS to remove options from the select as they use them, but be sure to provide a method to undo and change selections), and allow some columns not to be labeled (which the server will just ignore when parsing the file.
If possible, also illustrate (perhaps in a graph format or just an example sentence, if applicable) how their label choices will apply to the relationships. For example, "New user ABC will be a member of new group XYZ." If ABC and XYZ are unexpectedly backwards, the user will recognize they made a mistake.
Also, some users will inevitably upload a file where they used rows as columns and columns as rows. Either provide a GUI function to reverse this ("rotate" the table), or let them choose which axis to label.
I would also suggest providing your users with a collection of example files in various formats (Excel, Access, etc), and give them explicit instructions for how to enter the data they want, and step by step instructions to export as CSV and upload.
I have no idea how the user will generate these CSV files - most likely from Excel, Access or similar, and therefore I've no way of knowing which column will contain which data.
I should emphasize that the users of this system will not be technically gifted
With these two things in mind, are you sure that CSV import is the best way to handle bulk user creation? It's a great technical solution, but the question is, will your users be able to take advantage of it?
It may be worth implementing an alternative bulk create option for those who don't get CSV or are scared off by Excel. Perhaps a JS grid that has the required fields where they could manually enter the data for each field and enter as many as they need at once, with a link to upload a CSV file as an option for those who would use it.
For the CSV option, since your users are not technically-minded, it would be better to give them instructions on how to create the csv files that specify the order fields should be in. Along with a screen shot and a sample file.
Another option is to require the field names be the first row of the document, and require that they use specific labels for the fields. If you do that, you could figure out from the first row what order the data is in. You could also put in a check that looks for the titles in the first row and if they're not found, tell the user they need to add the field names to the CSV and re-upload.
Related
Im running Filemaker Server 19 and FileMaker 19 client
I have an app that manages Document Templates, ie Word documents
Each Document Template master record contains information about that template, plus connects to a Document Samples table that contains one Word document (the template) and one PDF document (an output of the template file when used , a bit like mail merge).
We've recently had an update to the system I support (not a Filemaker system) and many hundreds of the Word templates have been updated. If I have to do one or two in a session it's pretty simple to drag the latest version of each template to the Container field; but recently I've been having to do 10s/100s of Document updates very quickly to get them into our system. I've put copies of all the Word documents into a folder on my Mac. They all have a unique name and a unique template id in the format of 1234doctemplate01. So, the 1234 equates to my Document template ID in my database and the filename is also mirrored in a 'Template Name' field in the database.
So, my question is how to build a Script to go through the database one record at a time and, for each, to check the Mac folder and see if there's matching document name and, if there is, to paste/copy it into the appropriate Container field
Can anyone advise, please? Further detail available if this isn't too clear
I would do something slightly different:
Import all the files in the folder into a new table (see https://help.claris.com/en/pro-help/content/importing-folder.html) and create a relationship between this table and your existing table, based on matching template ID or name.
Then you can loop over the records in either table and if a match is found, replace the data in the existing container. Or perhaps you could make this a permanent structure, where the file is stored in a related record.
Doing it the way you have proposed would be awkward using only Filemaker's built-in tools (and outright impossible if the folder is not somewhere within the Documents folder). It might be somewhat easier if you use a plugin (e.g. BaseElements) to read the folder's contents.
You can use base element plugin to select multiple files in a folder with
BE_FileSelectDialog
or check if a file exist with
BE_FileExists
and then looping over the list to insert every file into a container via
set field ["your_container" ; BE_FileImport ( $file )]
With Google Financial, you can specify which columns of data you want to download. Can that be done with Quandl data? If so, I can't find an example that illustrates how.
I want to download Open and Close data only, not the entire table which is quite large. Quandl does supply a URL parameter, column_index that allows specification of a single column, so I can query for Close data only, or Open data only, but not both.
Perhaps, this can't be done. Could someone please confirm?
Here's some background information to help understand my dilemma. I've got a system written in PHP where users can fill out a form, upload a few images and everything gets archived in a .zip, I then have an "Account Page" that echo's a simple table containing a list of all of that specific users uploaded .zip's, the reason I'd need separate filepaths for each and every .zip is because no matter the user, they all need to be in the same single folder on the server for the broader application to work (these .zips are used for a mobile based Package Manager; see Cydia).
Would the smarter choice be to have every name of each .zip stored in a table on the database with things like the username and user_id tying to each entry, which would result in a few entries being for one user, and a few more being for a different user (depending on upload time), and everything will be unorganized. If I need to enter the database and manually search for a specific user's uploaded .zip's, I'll have to sort them out by user_id.
OR
Would it be more cost effective (in terms of passing data and hardware limitations) to have an XML file or even a plain .txt file on the server with a list of the users uploaded .zip's, and an exact file path pointing to them, and in the database simply have one column tied to the rest of the users information that points to that file containing multiple lines of filepaths?
It's a mouth full I know, and I have no problem achieving either of them as it's well within my skillset and knowledge of PHP & MySQL, I'm just not too sure which is the most secure and the best practice in the industry for this type of system.
I have an external file that I don't create which I need to import on a rolling basis, most of the column headers/field names have spaces in them. Is there a query I can write to change all of them at once? I'd rather not write a long query to get rid of spaces for each individual field name. The field names are always the same and in the same order in the file, the spaces are in the middle of the field names (ex: "Employee Number").
First of all, "query" refers to an SQL statement (including those viewed in Design View) that retrieve or act on data already in the database. Importing data from an external file is a separate action, not generally called a query. So strictly speaking the answer is "no".
However, Access does have built-in import functionality in Access. I suppose you can call these import "functions" or "actions" or "processes", just not queries. And I'm not being a smart aleck, since much of getting help with applications and code is learning and using the correct terms.
Go to the External Data ribbon (a.k.a. toolbar) along the top of Access.
Click the Import Text File icon (careful not to click the Export Text File icon, since they look similar. Hover the mouse cursor over each button to see the text description of it).
Choose the filename, and pick which import option
As Gustav instructs in his answer, choosing "Link to data source by creating a linked table" is the most efficient solution for external files that don't change format. The linked table (hence the external file) can be re-queried without repeating numerous steps.
Walk through the Import Wizard steps. Play with the options if you need to figure it all out.
In particular, make sure to check "First Row Contains Field Names"
On one of the wizard steps, you can edit the field names to remove the spaces.
On the last step, click the "Save Import Steps" checkbox, specify a name, then click the "Save Import" button
To re-use the previously-saved import steps:
Go to the External Data ribbon (a.k.a. toolbar) along the top of Access.
Click "Saved Imports" button
Choose your saved import settings
Click Run
OR if you created a Linked table
There is no need to "re-import". Instead, a normal Access query can be used to get the data and update one of your normal data tables.
If the path of the external files changes, this can also be updated by right-clicking the linked table and choosing Linked Table Manager (also available on the External Data ribbon). Select the table in the list and also check "Always prompt for new location" before clicking OK. A standard file selection dialogue will be shown for selecting a new filepath.
(Just to be complete, it is also possible to write VBA code in Access to open a file, read and analyze the headers and then import the data according to your custom behavior, but this isn't for you if you'd "rather not write a long..." something to do this.)
I'd rather not write a long query to get rid of spaces for each individual field name.
Maybe not, but there is no smart way to overcome this.
However, don't import the file but link it. Then use the linked file as source in your query. In this, alias the field names as you prefer, and do basic filtering and conversion of data. Then use this query for your further processing.
I'm working on a feature that imports users into a MySQL database.
The initial goal of this feature was to add new users from CSV files.
Gradually my client wants more of this tool. Indeed, if the CSV file contains a row with an already existed user, the data will be updated (the feature erases the old data with the new). So we implement it.
After that, he wants to update users (i.e. remove data, add data et cetera) but he already has a man/machine interface to do that.
I feel we're going in the wrong way.
What do you think about this ? Is it a good idea to manage database with csv files ?
I can see no problem using CSV to do that. You need to define a clear file format which specify object type and action, like for example :
<object type>;<action>;<value1>;<value2>;etc…
So you can have
user;add;Bob;Stone;fr
then
user;update;Bob;Stone;uk
then
user;del;Bob;Stone
etc…