I have created an SSIS package in Visual Studio 2008 that take's a SQL select statement and populates a excel sheet, the excel sheet is duplicated from a template file with all the formatting and cells set up.
The issue I am having is that no matter what I do I can not change the excel destination formatting to anything other than general, it overwrites the source destination and puts decimal numbers a '1.50 always adding the ' to fields.
i have tried inserting a row as per some suggestions as people think this is where SSIS scans for formatting types. However the field always comes up as Unicode string [DT_WSTR] in the advance editor and always defaults back if i change them.
Please can someone help! Happy to provide any additional info if I've missed anything, I've seen some posts with the same issue, but none of the solutions seem to be working or i'm missing something else.
****Update****
Figured out the reason behind none of the recommended fixes working, this was due to using a select statement in the excel destination instead of selecting the table.
This essentially wipes out any change if changing formatting.
So what I decided in the end was to create a data only sheet(which is hidden) using the basic table data access mode, then reference that in a front end sheet with all the formatting all ready in and using a =value(C1) formula to return just the value. Protected the cells to hide the formula's.
I have found that, when I change a Data Flow Task in SSIS, that exports to (or imports from) Excel, I often have to "start over", or SSIS will somehow retain the some of the properties of the old Data Flow Task: data types, column positions... For me, that often means:
1) Deleting the Source and Destination objects within the Data Flow Task, AND ALSO deleting/recreating the Connection Object for the Excel spreadsheet. I've done this enough times that I now save myself time by copy/pasting my Source and Destination names to-and-from a Notepad window, and I choose names that remind me of the objects they referred to (the table and file, respectively).
2) Remembering to rebuild the ARROW's metadata, too: after you change and/or recreate the Source object, you have to remember to DOUBLE-CLICK THE ARROW NEXT, before re-creating the Destination. That shows the arrow's metadata, but it also creates/updates the arrow's metadata.
3) When recreating the destination, DELETE THE SPREADSHEET from prior runs (or rename or move, etc.), and have SSIS recreate it. (In your new destination object, there's a button to create that spreadsheet, using the metadata.)
If you still have problems after the above, take a look at your data types... make sure you've picked SQL datatypes that SSIS supports.
At the link below, about 2/3rds of the way down the page, you'll find a table "Mapping of Integration Services Data Types to Database Data Types", with SSIS data types in the 1st column ("Data Type"), and your T-SQL equivalent data types in the 3rd column ("SQL Server (SqlClient)"):
Integration Services Data Types
Hope that helps...
Related
I have an excel spreadsheet with multiple sheets. So In For Each loop container have a script which is reading the sheets and saving them to the variable. In dataflow still inside of the For Each loop container is the process which leads to recordset destination which is saving all the columns to another variable. Then outside of the for each loop container is another dataflow which has to read from the variable all rows check for duplicity (in second and 3 sheet is duplicit product id), remove duplicity and upload data into database. I have been searching everywhere and cannot find how to setup the recordset destination to not replace the variable but append it with the values, because end up only with last sheet of data.
Cannot be doing changes in the foreach loop container in settings because of the looping through the sheets.
Thank you in advance for any advise.
While hopefully someone wiser in SSIS will chime in here, but I don't think your current approach will work.
Generally speaking, you can use Expressions within SSIS to get dynamic behaviour. However, the VariableName of the Recordset Destination does not support that.
You might be able to have a Script Task after the Data Flow that copies from rsCurrent into rsIteration1, rsIteration2, etc based on the current loop but at that point, you're double copying data for no real value.
Since you're doing a duplicate check, perhaps a read of sheet 1 goes into a Cache Connection Manager
And then the read from subsequent pages will use the CCM as the lookup. For rows that have matches, then you know you have duplicates (or maybe you only import what doesn't match, I don't quite get your logic)
Debugging some of this is going to be quite challenging. If at all possible, I would stage the data to tables. There you could load all the data + the tab name and then you can test your deduplication and refer back to your inputs and outputs.
The tooling for SSIS variables of type Object is pretty limited, which is a pity.
I am looking to refresh a data set in quicksight, this is in Spice. The data set comes from a csv file that has been updated and now has more data than the original file I uploaded.
I can't seem to find a way to simply repoint to the same file with same format. I know how to replace the file but whenever i do this it states that it can't create some of my calculated fields and so drops multiple rows of data!
I assume I'm missing something obvious but I can't seem to find the right method or any help on the issue.
Thanks
Unfortunately, QuickSight doesn't support refreshing file data-sets to my knowledge. One solution, however, is to put your CSV in S3 and refresh from there.
The one gotcha with this approach is that you'll need to create a manifest file pointing to your CSV. This isn't too difficult and the QuickSight documentation is pretty helpful.
You can replace the datasource by going into the Analysis and clicking on the pencil icon as highlighted in Step 1. By replacing dataset, you will not lose any new calculated fields that might have been calculated already on the old dataset.
If you try to replace the data source by going into the Datasets as highlighted below, you'll lose all calculated fields and modifications etc
I don't know when this was introduced but you can now do this exact thing through the "Edit Dataset", starting either from the Dataset page or from the 'pencil' -> Edit dataset inside an Analysis. It is called "update file" and will result in an updated dataset (additional or different data) without losing anything from your analysis including calculated fields, filters etc.
The normal caveat applies in that the newer uploaded file MUST contain the same column names and datatypes as the original - although it can also contain additional columns if needed.
Screenshots:
I'd like to automate a procedure some. Basically, what I do is import a few spreadsheets from Excel, delete the old spreadsheets that I previously imported, and then change a few queries to reflect the title of the new imports. And then I change the name of the queries to reflect that I've changed them.
I suppose I could make this a bit easier by keeping the imported documents the same name as the old ones, so I'm open to doing that, but that still leaves changing the queries. That's not too difficult, either. The name stays pretty much the same, except the reports I'm working with are dated. I wish I could just do a "find and replace" in the SQL editor, but I don't think there's anything like that.
I'm open to forms, macros, or visual basic. Just about anything.
I've just been doing everything manually.
Assuming I have correctly understood the setup, there are a few ways in which this could be automated, without the need to continually modify the SQL of the queries which operate on the imported spreadsheet.
Following the import, you could either execute an append query to transfer the data into a known pre-existing table (after deleting any existing data from the table), avoiding the need to modify any of your other queries. Alternatively, you could rename the name of the imported table.
The task is then reduced to identifying the name of the imported table, given that it will vary for each import.
If the name of the spreasheet follows logical rules (you mention that the sheets are dated), then perhaps you could calculate the anticipated name based on the date on which the import occurs.
Alternatively, you could store a list of the tables present in your database and then query this list for additions following the import to identify the name of the imported table.
I have a SQL Server that is ODBC into my Access which is being utilized as my front end. I created a column in SQL with a BIT data type and created a checkbox with the yes/no data type that uses my SQL Column as its control source. When I go into my form and try to change anything in there and update my tables, I get a writing error and it wont let me append the updates I'm trying to make. As soon as I delete the bit data type, and the check box, then I am able to again append information to my form. Does anyone know a solution or a way I could utilize the Boolean check boxes to report to my SQL Database so that I could track progress of an order, but still be able to append my records? Also if anyone knows why this is happening I would appreciate the information just for my own notes and understanding.
this image shows the column created in SQL Server
This image shows what it looks like in the ODBC table in Access
This image shows the control source I picked which is from a query that utilizes the names of the form comboboxes, text boxes, and hopefully check boxes
This is image shows what happens after I try to update and and close
Thanks
haven't done this for a long time, but the problem could be related to the numbers that are stored. yes/no in access = -1/0, in sqlsrvr = 1/0 (or similar). you could use an INT field in sqlsrvr which will then store whatever access sends (-1 or 0), or keep the BIT but don't use yes/no in access - create a custom yes/no table where yes=1 and no=0 (or whatever BIT needs) and use the custom table as a lookup table for that field which, when updated will send the appropriate value. hope that's all relevant and makes sense.
So I have been messing around with this and after a few other problems running through SQL, Cabinet Vision (our drafting software with a ancient access database) and crystal reports I came to realize that Access cannot accept null values.
So in my SQL Bit data type column I set the default value as 0 and didn't allow nulls. now it records all my check boxes as true or false. and I no longer have a write conflict.
This most commonly arises when a db is developed in Access and at some later date the tables are exported via ODBC to SQLServer. The Checkbox controls on the Access forms were probably placed there using the 'Available Fields' button on the Access 'Design' ribbon. Before getting into complicated solutions with T-SQL modifications to metadata etc, try simply deleting the checkbox control from the problem Access form. Insert an unbound checkbox to replace it, then use the the Properties dialog to set the relevant SQLServer field as the control source for that checkbox. Usually works.
I have several packages that are almost identical. Differ only by columns added/removed in different database versions. When I copy a package and modify the data flow of the copy, I delete the OLE DB Data Source and add a new one. Once the new one is defined, its preview shows exactly what I expect. Columns, however, are from the OLE DB source that was deleted. It's like it is being cached somewhere.
Seems like I need to close the package and re-open it after removing the data source. Is there some other way to clear this cached state? What's going on internally that causes this to happen?
More... it looks like it's the parametrized connection manager that is holding on to previous parameters until the package is closed and re-opened.
If I understand your work flow, you are copy and pasting packages and then tweaking the source definition in the data flow. The challenge is that the CustomerID in one system is varchar(7) and defined as varchar(12) in another. The "trick" becomes having the design engine recognize the metadata change and behave accordingly.
My usual hack is radically change the source. I find using the query SELECT 1 as foo does the trick. After doing that, the metadata for the OLE DB Source component drops all references to existing columns which percolates to the downstream components. I then switch back to the proper source and double click the first red X to have it map the IDs from old to new.
If you want a more brain surgical route than civil war surgery, change the column name in your source for anything that should have registered a metadata change. Thus SELECT T.MyColumn, T.IsFine FROM dbo.MyTable AS T becomes SELECT T.MyColumnX, T.IsFine FROM dbo.MyTable AS T Now only the first column gets kiboshed throughout the dataflow. Reset it back to the "right" column name and all is well.
Internally, I don't know but that never stops me from guessing. The Validation fires off, the SSIS Engine recognizes that the data types are still compatible so it doesn't change the existing metadata. A column no longer existing is enough to make it sit up and take notice and so the cached sizing goes away.
Some folks like to try and use the Advanced properties to change the sizes but I find I have better success just using the above approach than changing the size only to have the Designer slap my hand and disallow my proposed changes.