I am using a data scraper app which scrapes the data from single webpage and generates a CSV file.
I want to bulk import the data in SQL Express 2008 from this csv file at FREQUENT INTERVALS.
Please suggest a solution to schedule data import by first truncating the entire table than importing the fresh data.
In SQL Server I know it can be done via SQL Server Agent but I have no idea how to do it in SQL Server Express.
Or It would be great if you can suggest a single solution to scrap the tabular data (at fixed intervals) from a webpage and import it directly in the DB and produce some log.
Can it be done via some ETL tool?
I am looking for some open source light-weight solution to achieve this but if there is none than please suggest a commercial solution.
Thanks!
Related
I am struggling a bit with the following. Our forecasting tool generates some 100k of records of forecast information in Access each month. The tool is an Excel - Access combination. I want to upload the data from Access to SQL Server after the forecast is done (for multiple purposes). I tried to look up the most easy way but not sure what to use. ADO seems handy where I could integrate that in an existing macro that cleans up the database. Anybody done this before and has some snippet of code to use?
We use SQL Server Management Studio 2008, but not sure whether that's relevant here.
Thanks in advance,
Michiel
In Access, link via ODBC the tables in SQL Server you wish to upload to.
Then create and run append queries to insert the data in the linked tables.
I suggest using SSIS package which is started by job on sql server side (by shedule or manual). You can create package with the help of sql server import/export wizard. I think this is the best way to import data from access to sql server.
Thanks for the answers. I will try to do the linked table way, but I already had a look to that and it seems that due to the fact that I use citrix server I was not able to link it directly to SQL server via ODBC. So I have to check what's different there. Not familiar with the SSIS package yet but I will definitely look into that if the first option does not seem to work out.
Thanks again guys
I am new to SSIS is there is any component to load data from MYSQL to SQL server using SSIS. Currently am loading data using ODBC connection it is really slow and it around the speed of 30000 rows/Minute. Is there any way to make the load run fast.
Thanks in Advance...
You can install the .NET Connector for MySQL: http://dev.mysql.com/downloads/connector/net/
Then you can create a script task to act as a data source, import MySql.Data.MySqlClient, and query MySQL directly in C#. The data will then enter your Data Flow and you can map it to a SQL Server destination the same as normal.
I find that when using the SSIS connection manager with .Net Providers I get malformed SQL errors, but this way you write all the SQL yourself.
To improve the performance, we can add Conditional Split Transformation build some parallelism into the common data flow that directly load data from ODBC Source to OLE DB Destination.
For more information about speeding up SSIS Bulk Inserts into SQL Server, please see the following blog:
http://henkvandervalk.com/speeding-up-ssis-bulk-inserts-into-sql-server
In DataFlowTask property, Increase buffer size and no of row commit
Problem Definition: I would like to export all the data from all the tables from QuickBooks into SQL Server 2008 database. I tried to create a SSIS and using RssBus .NET Data Provider for QuickBooks - http://www.rssbus.com/ado/quickbooks/. I am able to export data from QuickBooks Customers table into SQL Server 2008 database table by using ADO.NET (source data reader) and OLE DB Connection (for destination SQL Server).
Question: I would like to loop through all the tables on source side and then export one by one. Is it possible through SSIS package? I will really appreciate if someone can point good resource links or example or detailed instruction will be much appreciated!
Then I would like to do import from SQL Server to QuickBooks. I am expecting SSIS can be used to accomplish the import process.
Thanks in anticipation!
You can't loop over the tables, you have to define each source and target table in advance in the package.
Of course, you always have the option of creating packages dynamically from .NET code, so you could read the list of tables in QuickBooks then generate a package. If the table structures change a lot or if there is a very large number of tables it could be worth it (I know nothing about QuickBooks).
I am trying to migrate DBF files to MS SQL Server 2008. While importing files from SQL Server Import And Export Wizard. I am unable to import the files. I have created the connection. But when I try to Edit Mappings/ Preview the table It shows error
"External table is not in expected format"
Anyhelp would be appreciated.
If you can't get the wizard to work, you'd have to write your own code, using linked servers or OPENROWSET. If you are new to SQL Server, linked servers and OPENROWSET are about the last thing you want to have to figure out.
Once upon a time long long ago, I wrote drivers in Foxpro that read data from and wrote data to SQL Server tables. This would have been SQL Server 7.0, and I was probably using OLE DB. If SQL can't talk to Foxpro (which wouldn't surprise me), you may need to get Foxpro to talk to SQL.
Failing that, dumping from Foxpro to some form of text delimited file and then importing them will work, though it'd take more effort.
I would recommend using SQL Bulk XML Load to move data from DBF tables to MS SQL Server. Doug Hennig wrote a blog post here with sample code.
I'm trying to create a completely new database from an existing MySQL database, bringing over both data and schema, but so far the only way I've been able to do this is to first import the MySQL database into MS Access, and then into SQL Server 2005? Crazy right? Surely, there is a way that doesn't involve a tedious, custom time-consuming programming, right (perhaps using SSIS)?
A few additions to my original description above:
Its a pretty good size database (easily a few gigs).
I'm working in an MS environment (asp.net, C#)
I'm under a tight deadline so I'm looking for an automated process that requires little to no effort in the conversion process.
SSIS would be the preferred way via BIDS (VS 2005)
Thanks for all the great input!
I believe that using the phpMyAdmin tool you can script the MySQL database structure and data into a sql script. Then you simply run those two scripts on your SQL Server 2005 database and it should, in most cases, create the database and fill it with data. It's been a couple years since I had to do it myself, but as I recall that was the process I used to transfer a MySQL database to SQL Server in the past. You will probably have to alter the structure script to change some of the data types to their SQL Server equivalents, but the data should load just fine once you've got the data types all sorted.
I think you can use SQLYog to generate some fairly standard SQL which will dump out and recreate your db, with data. You may have to massage its output for SQL Server's dialect of SQL a bit, though...
The responses I received were certainly helpful, but the solution it would seem is to do a mysqldump and then run that script from SSIS, massaging the output as needed; however, AFAIK it is not possible to use VS 2005 BIDS to create an SSIS package that completely transfers a MySQL database to a SQL Server 2005 database (data and schema) using Windows Vista 64. I said AFAIK, but who knows the interwebs have much to reveal :)