We are trying to import CSV file into Access Database via Powershell. My input file size is 1GB and it is getting difficult to iterate through each row and use insert command. Any quick suggestions here are highly appreciated
Thanks!!
as expresssed by #AlbertD.Kallal - what is the reason to use powershell at all? ... I simply made an assumption that you sought something that would run automatically, daily, unattended - - - as that is a typical reason.
if that is the case then it really breaks down to 2 parts:
make the import work manually in Access - - and then set up that import to fire automatically upon start/open of the Access file (auto exec).
just use powershell to start/open the Access file daily (or whenever...).
Access is not designed to be open full time and run unattended. So this is the typical approach to use it in that mode.
Ok, now having stated no need for power-shell, there are cases in which the IT folks and people are using power-shell to automate processes. So it not "bad" to consider power-shell - especially if it is being used.
I only wanted to point out that PowerShell will not help performance wise - and probably will be slower.
If you have (had) to say schedule a import to occur every 15 min or whatever?
Then I suggest setting up a VBA routine in a standard code module in Access to do the import. You then in the power shell, or windows script launch access, and then call that import routine. So, first step is to setup that routine in Access - even if using some kind of batch system for scheduling that import routine to run.
So, you use the windows scheduler.
It would: launch access, run the VBA sub, shutdown Access.
And using the windows scheduler is quite robust. So, we don't need (or want) to keep access running, but only launch it, run the import, and then shutdown access.
Next up, if the import process is "huge" or rather large, then on startup, a temp accDB file can be created, and then we import into that. We then can take the import table and send it into the production data table. (often column names are different etc. It also of course much safer to import into that temp table, and better yet, we can delete that temp file after - and thus we never suffer bloating or file size problems (no need to compact + repair).
So, the first thing to do is manually import the csv file using the Access UI. This ALSO allows you to create + setup a import spec. That import spec can thus remember the data types (currency, or often date time columns).
Once we have the import working and the import spec created?
Then we now can write code to do the above same steps, and THEN take the imported table and put that data into the production data table.
It it not clear if you "stage" the imported csv into that temp table, and then process that table into the real production data table, but I do suggest doing this.
(too dangerous to try and import directly into the production data).
you also don't share right now what kind of pre-processing, or what additonal code is required after you do the import of that csv (but, still, we assume now that such imports will be into a new temp table).
So, I would assume the steps are:
we import the csv file using built in import abiity of Acces
we then send this data table to the production table, perhaps with some code processing of each row before we send that temp table to production table.
Once we done the import, then we dump + delete the temp accDB file we used for the import, and thus we eliminate the huge data bloat issue.
Thus for the next time, then we create that temp file for a fresh import, and thus each time we start out with a nice empty database file.
So the first question, and you can create a blank new database for this test. Do you or can you import the csv file using Access. You want to do this, since such imports are VERY fast and VERY high speed. Even if the imported format is not 100% as you want, you do need to confirm and try if using the access UI you can import the csv file. if you can, then we can adopt VBA commands to do the same thing, but no use writing code if a simple csv import via Access UI can't be used.
Related
The git repo for my Django app includes several .tsv files which contain the initial entries to populate my app's database. During app setup, these items are imported into the app's SQLite database. The SQLite database is not stored in the app's git repo.
During normal app usage, I plan to add more items to the database by using the admin panel. However I also want to get these entries saved as fixtures in the app repo. I was thinking that a JSON file might be ideal for this purpose, since it is text-based and so will work with the git version control. These files would then become more fixtures for the app, which would be imported upon initial configuration.
How can I configure my app so that any time I add new entries to the Admin panel, a copy of that entry is saved in a JSON file as well?
I know that you can use the manage.py dumpdata command to dump the entire database to JSON, but I do not want the entire database, I just want JSON for new entries of specific database tables/models.
I was thinking that I could try to hack the save method on the model to try and write a JSON representation of the item to file, but I am not sure if this is ideal.
Is there a better way to do this?
Overriding save method for something that can go wrong or that can take more than it should is not recommended. You usually override save when changes are simple and important.
You can use signals but in your case it's too much work. You can instead write a function to do this for you but still not exactly after you saved the data to database. You can do it right away but it's too much process unless it's so important for your file to be updated.
I recommend using something like celery to run a function in the background separated from all of your django functions. You can call it on every data update or each hour for example and edit your backup file. You can even create a table to monitor the update process.
Which solution is the best is highly depended you and how important the data is. And keep in mind that editing a file can be a heavy process too so creating a backup like everyday might be a better idea anyway.
How do I import a database just like in phpmyadmin at DataGrip?
I have the .sql exported from phpmyadmin... but those are lots of lines so that the IDE stops working when trying to run the whole .sql
In DataGrip go to File > Open and select your mysql dump file. Then right click the tab for the file to get the context menu, and select "Run [your filename...]" option. It may ask you to select your schema to apply the run to. But this is how I accomplished importing a dump from phpMyadmin using DataGrip.
Jetbrains documentation on running SQL scripts does not provide a ton of information on processing large insert statements. There is a discussion in the Datagrip community forums and apparently upcoming features to make working with large scripts easier.
Quote from thread:
Huge SQL files can be executed from Files view (use a context menu action).
I assume you are attempting to import a database export which is a series of SQL statements saved to a file. There could be a memory issue if you are attempting to run a large SQL file in memory. Try the following.
Insert commit statements in your SQL file in a text editor. This can even be done from within datagrip. Every couple of hundred statements you can place the line
commit;
which should purge the previous statements from memory. I strongly recommend saving the file which you edit separately from the export script. This method is not applicable if you need an all or nothing import, meaning if even one statement or block fails you want all of the statement to be rolled back.
1 - Going to View->Tool Windows->Files
2 - Going to schema folder and open it in windows explorer after that past your dump file in my example i will past MyDump.dmp .
3 - Right click on the MyDump.dmp and run it .
To import data from a script file, run the file as it is described in Run database code. In addition to script files, you can import a CSV, TSV, or any other text file that contains delimiter-separated values.
https://www.jetbrains.com/help/datagrip/import-data.html
I've got a 90MB .csv extract I need to push into an Access 2007 DB.
AS A TEST - If I do a straight import of the file with default settings into a new table I end up with a DB of 134MB.
WHAT I NEED TO DO - The .csv extract contains a couple of columns I need to process before pushing them to into the Access DB. To accomplish this I am using SSIS (from SQL Server 2008 install) and using a couple of derived columns to contain the processed columns which then all get pushed into an existing Access table (which has no rows at the start of the process) via an ADO.NET Connection using the following connection string "Data Source=C:\Import\InTheGarden.accdb;Provider=Microsoft.ACE.OLEDB.12.0;". (Connection String from Connection in Connection Manager in SSIS)
When I process the data using SSIS I end up with a file of 1.16GB which when compacted comes down to a size of approximately 180MB!!!
Two things: -
Firstly I don't understand what is causing the bloat and how I might get around to it. I've read I don't know how many articles today on "Access 2007 Bloat" and can't seem to find anything that exlpains what exactly is going on:(
Secondly, whilst I can import the .csv file above I have another text file of 154MB which needs importing and given that the smaller file caused the DB to bloat to 1.16GB on import I'm not holding out much hope of success with this bigger file not exceeding the 2GB limit!
I would normally use SQL Server as a back end but there are reasons beyond my control as to why this can't happen:(
Happy to provide further information, Access is not something I use loads so there may be key information which I've missed!! :(
Thx!
Why not use a staging table in Access to do the import, process the data, and then insert to the real table? You'd use a separate temp database for that so as to avoid the bloat of importing twice. I've done dozens of imports into Access and this has always been the way I do it when I need to process the data before the final append.
Try breaking the CSV into several smaller files and running Access' Compact Database command after importing each one.
How can I import initial table data to a .mwb file? I know that there is an inserts tab for each table, but I would like to import around 200 records and I don't want to do this by hand.
It is not possible with the modern version of MySQL Workbench. There is no way, essentially, to model data - you can only upload it to the server (not the model). The only way currently is to edit one by one which isn't practical. Even if you reverse engineer a table filed with data, the inserts table of the EER model will be blank. You'll note that right-clicking on the row of the inserts tab gives a number of greyed out options including "load from file". I suspect the team didn't have time to implement them or something. Anyway, there is a simple work around if you know phpMyAdmin, which seems to handle CSV files well, or MySQL Workbench, which I have not gotten to work at all with CSV files.
Solution:
Draw your DB model in MySQL Workbench, structure only. Place all your data in associated CSV files - I use Excel and save as CSV - very easy. Once you have your data modeled in Excel and the structure modeled in Workbench, forward engineer the DB, then use some other tool or technique to upload your Excel modeled data.
Not the greatest solution, but bug them to provide data modeling and maybe we'll be lucky in the next version.
Currently this seems not to be possible. I too was hoping to be able to Reverse engineer from the insert statements in a script file, but 1. it didn't work :P and 2. actually the documentation explicitly states that these will be ignored:
http://download.oracle.com/docs/cd/E19078-01/mysql/mysql-workbench/wb-data-modeling.html#wb-reverse-engineering
7.7.9.1. Reverse Engineering Using a Create Script
Reverse engineering using a create script is done by using the File, Import, Reverse Engineer MySQL Create Script ... menu options. Doing this opens a file open dialog box with the default file type set to an SQL script file, a file with the extension sql.
You can create a data definition (DDL) script by executing the mysqldump db_name --no-data > script_file.sql command. Using the --no-data option ensures that the script contains DDL statements only. However, if you are working with a script that also contains DML statements you need not remove them; they will be ignored.
It seems that the lesson is that we ought to handle such resources (that are too large to be manually inserted) through some other medium, such as a versioned sql file. :(
I am building my first database driven website with Drupal and I have a few questions.
I am currently populating a google docs excel spreadsheet with all of the data I want to eventually be able to query from the website (after it's imported). Is this the best way to start?
If this is not the best way to start what would you recommend?
My plan is to populate the spreadsheet then import it as a csv into the mysql db via the CCK Node.
I've seen two ways to do this.
http://drupal.org/node/133705 (importing data into CCK nodes)
http://drupal.org/node/237574 (Inserting data using spreadsheet/csv instead of SQL insert statements)
Basically my question(s) is what is the best way to gather, then import data into drupal?
Thanks in advance for any help, suggestions.
There's a comparison of the available modules at http://groups.drupal.org/node/21338
In the past when I've done this I simply write code to do it on cron runs (see http://drupal.org/project/phorum for an example framework that you could strip down and build back up to do what you need).
If I were to do this now I would probably use the http://drupal.org/project/migrate module where the philosophy is "get it into MySQL, View the data, Import via GUI."
There is a very good module for this, node import. It allows you to take your GoogleDocs spreadsheet and import it as a .csv file.
It's really easy to use, the module allows you to map your .csv columns to the node fields you want them to go to, so you don't have to worry about setting your columns in a particular order. Also, if there is an error on some records, it will spit out a .csv with the error files and what caused the error, but will import all good records.
I have imported up to 3000 nodes with this method.