I'm working on integrating my data between two different database systems - our site itself is run on Drupal 6.2.8, but our customer database is run by a third-party on a different system. It is capable of importing data from a csv.
I'm capturing data with a Drupal webform - that's working great. I know that I can manually download a csv from the webform - but I want to run imports frequently - at least daily. I found this thread: http://drupal.org/node/1276098 that talks about a drush command for exporting webforms, but it doesn't seem to be complete.
I know that I can use views to create a csv, but I don't seem to have access to the submissions themselves from views. Likewise I know that the data module can somehow be tied into the answer for this, but I am not at all sure how to get started with it.
If there were just a simple way to schedule downloads of the data, I could set up an rsync or something like that to handle the rest - any suggestions?
The following code may work for the case:
drush webform-export <Specify-WebFormID> --completion-type=finished --delimiter="|" --file=<Specify-Filename> --format=delimited
You may also use --range-start and --range-type options to be more precise in which result you'd like to get.
Related
The git repo for my Django app includes several .tsv files which contain the initial entries to populate my app's database. During app setup, these items are imported into the app's SQLite database. The SQLite database is not stored in the app's git repo.
During normal app usage, I plan to add more items to the database by using the admin panel. However I also want to get these entries saved as fixtures in the app repo. I was thinking that a JSON file might be ideal for this purpose, since it is text-based and so will work with the git version control. These files would then become more fixtures for the app, which would be imported upon initial configuration.
How can I configure my app so that any time I add new entries to the Admin panel, a copy of that entry is saved in a JSON file as well?
I know that you can use the manage.py dumpdata command to dump the entire database to JSON, but I do not want the entire database, I just want JSON for new entries of specific database tables/models.
I was thinking that I could try to hack the save method on the model to try and write a JSON representation of the item to file, but I am not sure if this is ideal.
Is there a better way to do this?
Overriding save method for something that can go wrong or that can take more than it should is not recommended. You usually override save when changes are simple and important.
You can use signals but in your case it's too much work. You can instead write a function to do this for you but still not exactly after you saved the data to database. You can do it right away but it's too much process unless it's so important for your file to be updated.
I recommend using something like celery to run a function in the background separated from all of your django functions. You can call it on every data update or each hour for example and edit your backup file. You can even create a table to monitor the update process.
Which solution is the best is highly depended you and how important the data is. And keep in mind that editing a file can be a heavy process too so creating a backup like everyday might be a better idea anyway.
I have an MS Access 2016 application that a few people use in one department. I know this whole thing has web dev written all over it but this access database has been their process for a while and there is no time right now to switch over.
Recently, a different department wants to use this application, but having their own copy. Currently, if I need to make changes, I'll make the changes in a copy of the app, they send me a current version when I'm ready to import their data, I import it and send them back a new one. However, currently I copy the data table by table and past it into the new database. This is inefficient and tedious, and now with 2 sets of data I'd be doing this for, that's crazy. There's over 20 tables so I don't want to have to manually copy over 40+ tables across the 2 apps for even the smallest change like altering a message to the user.
I know I can copy the code so I can avoid importing the data, but sometimes for big changes I'll change over 15-20 vba files.
So, a couple questions:
1.Is there a way to generate insert statements for the entire database that I could run in a script? So when I create the new copy I just upload 1 file and it populates all the data?
2.Are there any kind of dev tools that will help this process? Right now I'm thinking that it's just a downfall of creating an MS Access app, but there must be some way that people have made the "new release" process easier. My current system seems flawed and I'm looking to have a more stable process.
EDIT:
Currently I have all my data stored locally, attached to the same access file as the front end. Since I will have 2 different departments using the same functionality, how do I manage the data/the front-end? These 2 departments should have their own access file to enter data using the forms, so having 1 front end between the 2 departments won't work.
Also, should I create 2 separate back-ends? Currently I would have nothing to distinguish what is being inserted/changed/deleted from one department from the other. If I were to attach a field specifying who entered the record, that would require a complete overall of all my queries which I don't have the time for as there are deadlines I need to meet.
First thing is to split the database. There is a wizard for this.
Then you can maintain the frontend without touching the real data.
Next, consider using a script to distribute revised versions of the frontend. I once wrote an article on one proven method to handle this:
Deploy and update a Microsoft Access application in a Citrix environment
In our rails app we sometimes have db entries created by users that we'd like to make part of our dev environment, without exporting the whole table. So, we'd like to be able to have a special 'dev and testing' dump.
Any recommended best practices? mysqldump seems pretty cumbersome, and we'd like to pull in rails associations as well, so maybe a rake task would make more sense.
Ideas?
You could use an ETL tool like Pentaho Kettle. Once you have initial transformation setup that you want you could easily run it with different parameters in the future. This way you could also keep all your associations. I wrote a little blurb about Pentaho for another question here.
If you provide a rough schema I could probably help you get started on what your transformation would look like.
I had a similar need and I ended up creating a plugin for that. It was developed for Rails 2.x and worked fine for me, but I didn't have much use for it lately.
The documentation is lacking, but it's pretty simple. You basically install the plugin and then have a method to_sql available on all your models. Options are explained in README.
You can try it out and let me know if you have any issues, I'll try to help.
I'd go after it using a Rails runner script. That will allow your code to access the same things your Rails app would, including the database initializations. ActiveRecord will be able to take advantage of the model relationships you've defined.
Create some "transfer" tables in your production database and copy the desired data into those using the "runner" script. From there you could serialize the data, or use a dump tool, since you'll be dealing with a reduced amount of records. Reverse the process in the development environment to move the data into the database.
I had a need to populate the database in one of my apps from remote web logs and wrote a runner script that fired off periodically via cron, ftps the data from my site and inserts the data.
This is a doozy of a question...
I am developing a new wordpress website to replace my client's massively sized blog (tens of thousands of pages). Their existing blog is built on a custom PERL blogging platform from 1992...so as you can imagine, it needs updating.
My job is to find the BEST possible way to migrate all of their existing data into a format that wordpress can understand.
The new theme is very advanced, and this job is very advanced as well. After searching for specialists, I can't seem to find anyone who specialized in this field. What would you do in this situation? Thanks!
I've done this before, it's not that hard. I approached it as a wordpress plugin. First get the plugin to connect to the old database and get it to pull the information you need. Then you can use native wordpress function to insert new users, posts, comments, etc.
I did it line by line, which isn't the most efficient approach, but it is the easiest. I used an AJAX front-end to display the conversion status and repeatedly call the converter back-end, as the actual migration took up to several hours.
If the site can't be shut down for that long while you migrate the database, you could either look at doing a proper bulk export/import, or lock old topics and migrate those over first.
My converter was for an old PHP nuke site, and due to the fact that we were using WP-United, I didn't have to worry too much about user credentials and comments. However, the code might help you get started: http://www.wp-united.com/releases/php-nuke-converter
I have done a couple of these Wordpress migrations. The theme you're using in Wordpress is really not that big a deal, most likely. Themes in Wordpress don't ordinarily impact the database structure.
The Wordpress side is easy, it's in MySQL in most cases. The place to start for you would be to determine how the data in the PERL blog is stored. If it's a custom blog solution, there probably isn't a script you can find to do the migration. Hopefully, it's in a data form that will allow you to do a data dump in a format MySQL will allow you to import using something like phpMyAdmin (a popular GUI for MySQL). At that point, you can create a MySQL statement to match up the relevant fields in your old data with those in MySQL. If you're not comfortable doing it and want an expert, the thing to do is find out how the PERL blog stores it's data and find someone who is familiar with both that format and MySQL.
For some security issues I'm in an envorinment where third party apps can't access my DB. For this reason I should have some service/tool/script (dunno what yet... i'm open to the best option, still reading to see what I'm gonna do...)
which enables me to generate on a regular basis(daily, weekly, monthly) some csv file with all new/modified records for a certain application.
I should be able to automate this process and also export at any time a new file.
So it should keep track for each application which records he still needs.
Each application will need some data in some other format (csv/xls/sql), also some fields will be needed for some application and some aren't... It should be fairly flexible...
What is the best option for me? Creating some custom tables for each application? Based on that extracting modified data?
I think you best thing here, assuming you have access to the server to let you set this up is to make a small command line program that can do the relativley simple task you need. Languages like pearl are good for this sort of thing I do believe.
once you have that 'tool' made you can schedule it through the OS of the server to run ever set amount of time. Either schedule task for a windows server or a cronjob for a linux server.
You can also (with out having to set up the scheduled task if you don't / can't want to) enable this small command line application to be called via 'CGI' this is a special way of letting applications on the server be executed at will by a web user. If you do enable this though, I suggest you add some sort of locking system so that it can only be run every so often and to stop it being run five times at once.
EDIT
You might also want to just look into database replication or adding read only users. This saves a hole lot of arseing around. Try to find a solution that dose not split or duplicate data. You can set up users to only be able to access certain parts of the database system in certain ways, such as SELECT data