I am using SSIS to move data between a local MSSQL server table to a remote MYSQL table (Data flow, OLEdb source and ODBC Destination). this works fine if im only moving 2 lines of data, but is very slow when using the table I want which has 5000 rows that fits into a csv of about 3mb, this currently takes about 3 minutes using ssis's options, however performing the steps below can be done in 5 seconds max).
I can export the data to a csv file copy it to the remote server then run a script to import straight to the DB, but this requires a lot more steps that I would like as I have multiple tables I wish to perform the steps on.
I have tried row by row and batch processing but both are very slow in comparison.
I know I can use the above steps but I like using the SSIS GUI and would have thought there was a better way of tackling this.
I have googled away multiple times but have not found anything that fits the bill so am calling on external opinions.
I understand SSIS has its limitations but I would hope there is is a better and faster way of achieving what I am trying to do. If SSIS is so bad I may as well just rewrite everything into a script and be done with it, But I like the look and feel of the Gui and would like to move my data in this nice friendly way of seeing things happen.
any suggestions or opinions would be appreciated.
thank you for your time.
As above have tried ssis options including a 3rd party option cozyroc but that sent some data with errors (delimiting on columns seemed off) now and again, different amount of rows being copied and enough problems to make me not trust the data.
Related
Good Day,
I am trying to learn on how to save my data in MySQL database using C in real-time.
I am using a Raspberry Pi MCU, and an external web server where the data will be saved. I am also using C to get the data from the sensors and would like to save it to my external database, but I do not know how to proceed with this problem as I am not that familiar with using C and MySQL together. though my main concern here is how do i make sure that my data is real-time, or when my sensors get the data it will then be saved to the database.
I'm thinking of doing an infinite loop inside my main and will place an if statement that will serve as a trigger whenever there is a data from the sensors and will save it to the mysql server.
but i am not sure if that is the most efficient way of doing this, that is why if you have any better ideas of how to retrieve my data in real-time using C and saving it to MySQL then it would be greatly appreciated.
in PHP i would have simply made a cron job for this but since i will be doing this in C, I am at lost on how to proceed or if my idea is correct.
You are looking at two independent problem:
Retrieve the data at a fixed interval
Save the data to a database.
For the first, there are two known methods, the first is polling which means stay in a while loop and constantly check if updates are available. The second method is using interrupts, you should choose the most appropriate for your problem but for beginning you can use the first method and when the program works (maybe) move it to interrupts.
For the second, just install MySQL and mysql C connector, just go to their site and download and install it. Its connection is pretty simple and there are a lot of examples online, both for combining and syntax.
An efficient way to do such things is called 'the hardware interrupt'. You should read the docs to check if the hardware supports it.
I have inherited a legacy Access app that has a TON of code running before the main form comes up. This code imports data, deletes data, and changes data a million ways.
Is there a way where (after the startup stuff is finished) I can list the tables and when each of them had data affected most recently?
Thanks!
Sorry, but I'm afraid the answer to your question is simply: No.
Here's an idea though:
Make a backup of the database file.
Open the app so it runs the code you are concerned about.
Compare the DB to the backup using a tool like Red Gate SQL Data compare.
BTW: I'm not sure if the RedGate tool works against access databases, but FMSInc. has one that claims to.
We are currently having an OLTP sql server 2005 database for our project. We are planning to build a separate reporting database(de-normalized) so that we can take the load off from our OLTP DB. I'm not quite sure which is the best approach to sync these databases. We are not looking for a real-time system though. Is SSIS a good option? I'm completely new to SSIS, so not sure about the feasibility. Kindly provide your inputs.
Everyone has there own opinion of SSIS. But I have used it for years for datamarts and my current environment which is a full BI installation. I personally love its capabilities to move data and it still is holding the world record for moving 1.13 terabytes in under 30 minutes.
As for setup we use log shipping from our transactional DB to populate a 2nd box. Then use SSIS to de-normalize and warehouse the data. The community for SSIS is also very large and there are tons of free training and helpful resources online.
We build our data warehouse using SSIS from which we run reports. Its a big learning curve and the errors it throws aren't particularly useful, and it helps to be good at SQL, rather than treating it as a 'row by row transfer' - what I mean is you should be creating set based queries in sql command tasks rather than using lots of SSIS component and dataflow tasks.
Understand that every warehouse is difference and you need to decide how to do it best. This link may give you some good idea's.
How we implement ours (we have a postgres backend and use PGNP provider, and making use of linked servers could make your life easier ):
First of all you need to have a time-stamp column in each table so you can when it was last changed.
Then write a query that selects the data that has changed since you last ran the package (using an audit table would help) and get that data into a staging table. We run this as a dataflow task as (using postgres) we don't have any other choice, although you may be able to make use of a normal reference to another database (dbname.schemaname.tablename or somthing like that) or use a linked server query. Either way the idea is the same. You end up with data that has change since your query.
We then update (based on id) the data that already exists then insert the new data (by left joining the table to find out what doesn't already exist in the current warehouse).
So now we have one denormalised table that show in this case jobs per day. From this we calculate other tables based on aggregated values from this one.
Hope that helps, here are some good links that I found useful:
Choosing .Net or SSIS
SSIS Talk
Package Configurations
Improving the Performance of the Data Flow
Trnsformations
Custom Logging / Good Blog
I've perused the threads here on migration from SQL 2000 to SQL 2008 but haven't really run into my question, so here we go with another one.
I'm building a strategy to move specific SQL 2000 databases to a new SQL 2008 R2 instance. My question comes with regards to the best method for transferring the schema and data. One way I know of is to do the quick 'n' dirty detach - copy - attach method, which should work so long as I've done my homework wrt compatibility and code and such.
What if, though, I wrote the schema and logins via script and then copied the data via SSIS? I'm thinking of trying that so I can more easily integrate some of my test cases into the package (error handling and whatnot). What would I be setting myself up for if I did this?
Since you are moving the data between servers or instances, I would recommend moving the data via data flows. If you don't expect to run the code more than once, then you can let the wizard generate your code for this move. However, when I did this once 2+ years ago, the wizard code generated combined execute sql tasks that combined many "create table" commands into one task and created a few data flow tasks that had multiple source and destinations in them to insert data in the destination. This was good to get up and running, but it was inadequate when I wanted to refresh the tables one more time after I modified the schema of the new target tables. If you expect to run the refresh more than once, then you may want to take the time to create the target schema first and then manually create the data flows.
Once you have moved the data, then you can enable full-text search on the new server. I don't believe you will need to have this enabled on your first load.
One reason I recommend against the detach-attach method for migration is that you bring all the dirty laundry from the 2000 database to the 2008 R2 database. If you had too lax security on the 2000 server or many ancient users that shouldn't exist, it could be easier to clean this up by starting from scratch. If you use the detach-attach method, then you have to worry about users.
I need a local copy of our production database, and I need to refresh it every few days so testing and development is not working with terribly stale data. A few days old is just fine. Here is the pseudo plan:
Write a script on the Production server that mysqldump's + gzip the database.
Add a cron process to run the script every other day during non-peak hours.
Write a script on the workstation that rsync's that gzipped dump and loads it up.
Is there any better, cleaner, or safer way of doing this?
EDIT: Just to add clarity. We still have in place Test Data that is known, along with our test library (test driven development). Once THOSE tests pass, its on to the (more) real stuff.
You may wish to consider MySQL replication. It isn't a thing to be trifled with but may be what you are looking for. More information here... http://dev.mysql.com/doc/refman/5.0/en/replication-features.html (I don't personally know anything about it other than that it can be done).
Testing should be working with "known" data; not production data. You should have scripts to load "Test" data into the system to achieve this. Test/Dev shouldn't have to deal with a moving target of data. Besides, if you have any sensitive data in production (doesn't everyone"); your dev/test teams shouldn't have access to it.
Some suggestions for creating test data:
1) Excel spreadsheets with VBA behind them to create sql to run against the DB
2) Raw sql scripts
3) Data creation programs that generate data in a known pattern.