I got an excel file for pulling data to SQL Server DB. At Excel Source, I wanted to create ROW_NUMBER() like in T-SQL, as an additional column. Is it possible in EXcel Source query ? How ?
you can do it using a script component: Generating Surrogate Keys
or you can download a custom component that does that for you. There are a few, but the downsize of this option is that you have to deal with the deployment of this component (which is simple, just copy the dll, but is one more thing to worry about)
Related
I have a Data Flow with OLE DB Source, Script Component (Transformation), and Flat File Destination:
The OLE DB Source task has 100+ columns. The script component is going to cleanup data in each column and then output it to the Flat File Destination.
Adding output columns by hand in Script Component is unthinkable to me.
What options do I have to mirror the output columns with the input columns in the Script Component? While the output column name will be the same, I plan to change the datatype from DT_STR to DT_WSTR.
Thank you.
You are short of luck here. Possible scenarios:
Either you use Script Component and have to key in all columns and its properties manually. In your case, you have to set proper datatype.
Or you can create your own Custom Component which can be programmed to create output columns based on input columns. It is not easy and I cannot recommend a simple guideline, but it could be done.
This might have sense if you have to repeat similar operations in many places so it is not a one-time task.
You can create a BIML script that creates a package based on metadata. However, the metadata (list of columns and its datatypes) has to be prepared before running BIML script or do some tricks to get it during script execution. Again, some proficiency with BIML is essential.
So, for one-time job and little experience with BIML I would go for a pure manual approach.
I have the following problem:
We have a case where we need to import several dozen, if not hundreds, of uniquely structured source files with SSIS. Perfect opportunity to use BIML for this. We do not have metadata information for the files as is, so I need to get them - not manually, file by file, though.
So I thought, easy, build a table with filepaths, use BIML to create a package with the source connection, SSIS identifies metadata (maybe not 100% correctly) and I can use this metadata (column name, datatype, length, etc.) to initially persist in a metadata table for further use.
But: there seems to be no way to achieve this. While I can view the metadata and even paste it to the clipboard in SSDT, I cannot get it to the pipeline. Tried a script component as well (not my best skill), can get everything BUT the column name in ProcessInput, but can't create an output.
So: is there any known way of achieving this? Googled for several hours already to no avail.
I am able to migrate data between two SQL Server tables easily using a SSIS data flow task. Can I use format files to specify the columns to choose from the source and destination? If so, can you give me an example?
In our current system, our Source and Destination tables are always not the same. We were using SQL-DMO with format files so far and are now upgrading to SSIS.
Thanks in advance for your suggestions.
So I think that you can look up info on how to create a format file here: http://msdn.microsoft.com/en-us/library/ms191516.aspx
Google SSIS Bulk Insert Task to find more on that.
I would recommend using a data flow if you can because this can eliminate columns from the source that do not exist in the destination and it can out perform bulk inserts. It's worth consideration.
Mark
Here is a post to which I just finished answering my own question and thought will link the two posts together.
SSIS - Export multiple SQL Server tables to multiple text files
I'm using Firebird database and I need to load Excel file into a database table. I need a tool that does this well. I tried some I found on Google, but all of them have some bugs.
Since Excel data is not created by me, it would be good if it could scan the file and discover what kind of data is inside and suggest a table to be created in the database.
Also, it would be nice if I could compare the file against the data that is already in the database table, and I can pick which data to load and which not.
Tools that load CSV files are also fine, I can "Save as" CSV from Excel before loading.
Well, if you can use CSV, the I guess XMLWizard is the right tool for you. It can load a CSV file and compare with database data. And you can select the changes you wish to make to the table.
Don't let the name fool you, it does work with XML, but it also works very well with CSV files. And it can also estimate the column datatypes and offer CREATE TABLE statement for your file.
Have you tried FSQL?
It's a freeware very similar to Firebird's standard ISQL, but with some extra features, like import data from CSV files.
I've used it with DBF files and it worked fine.
There is also EMS Data import tool for Firebird and Interbase
http://www.sqlmanager.net/en/products/ibfb/dataimport
Not free, though, but it accepts a big variety of formats, including CSV and Excel.
EDIT
Another similar payware tool is Firebird Data Wizard http://www.sqlmaestro.com/products/firebird/datawizard/
There are some online tools which can help you to generate DDL/DML scripts from csv header/sample dump file, check out: http://www.convertcsv.com/csv-to-sql.htm
You can then use sql-workbench's Data Pumper or WbImport Tool from command line.
Orbada has GUI which support for importing csv file also.
DBeaver Free edition also support importing csv out of the box.
BULK INSERT
Other way is on Excell you build formula in new cells with data you want to export. The formula consists to format in strings and lenght to your field according lenght your field in firebird. So you can copy all this cells from excell and past on txt editor, so is possible to use the strategy of BULK INSERT in Firebird.
See more details in http://www.firebirdfaq.org/faq209/
The problem is if you have blob or null data to import, so see if you have this kind of values and if this way is to you.
If you have formated data in txt file, BULK INSERT will be quick way.
Hint: You can too to disable trigger and index associated with your table to accelerate BULK INSERT, and after enable them.
Roberto Novakosky
I load the excel file to lazarus spreadsheet and then export to firebird db. Everythong is fine and the only problem is fpspreadsheet will consider string field with numbers only as a number field. I can check the titles in the first row to see whether the excel file is valid or not.
As far as I can see all replies so far focus on tools that essentially read the Excel (or CSV) file and uses SQL inserts to insert the records into the Firebird database. While this works, I have always found this approach painstakingly slow.
That's why I created a tool that reads an Excel file and writes one file that has a (text) format suitable for Firebird external table (including support for UTF8 char columns) and one DDL file to create the external table in Firebird.
I then use regular SQL to select from the external table, cast as needed, and insert into whatever normal Firebird table I want. The performance with this approach is orders of magnitude faster than SQL inserts from a client app in my experience.
I would be willing to publish the tool. It's written in C#. Let me know if there's any interest.
I have a .NET webforms front end that allows admin users to upload two .xls files for offline processing. As these files will be used for validation (and aggregation) I store these in an image field in a table.
My ultimate goal is to create an SSIS package that will process these files offline. Does anyone know how to use SSIS to read a blob from a table into its native (in this case .xls) format for use in a Data Flow task?
In my (admittedly limited) experience with SSIS, it is quite good at rapidly getting something up and running, but frusteratingly limited in getting something that "feels" like the most elegant, efficient solution to a programmer.
Since the Excel Source Editor seems to take only files as input, you need to give it a file or reimplement its functionality in code that can take a blob. I understand that this is unsatisfying, but in the end, this is a time saving tool.