Maybe somebody can provide assistance for the following question:
I have a SSIS package with Target SQL Server Version 2014. I am not 100% sure which version my target server is running, but it is at least 2014, but i assume its 2016.
I have developed the package with SQL Server Data Tools 2015. It contains of:
Flat File source + Connection Manager
OLEDB Destination
Conversion Step
The source file is encoded in UTF-8, the target database uses ANSI 1252. The file is located on a network drive (essentially the same windows server the SQL Server with the target database is running on).
Contained in the file is a decimal field with precision, scale 18,2. The actual data in the field is always 0.00.
I have specified the input field in the connection manager as decimal.
Now my question: when i execute the package directly in Visual Studio (Data Tools), it works flawlessly, all rows are imported into the target table.
When I call the dtsx package from an SQL Server Agent Job (it is the only step) it fails with a conversion error regarding the decimal field:
Source: Data Flow Task Flat File Source [88] Description: Data conversion failed. The data conversion for column "xxx" returned status value 2 and status text "The value could not be converted because of a potential loss of data.".
The error also arises when i switch between target server versions 2014 / 2016.
A list of things i tried without being successful:
remove column headers from the input file
test it with only one data row... no success.
created a new solution with a new dtsx package and a slightly different input path
deleted the SQL Server agent job and created a new one
changed the encoding of the input file to UTF-8
I didnt find a solution for this exact error phenomenon on the net.
Can anyone provide help?
Many thanks in advance!
Got it. The reason lies in the different decimal formats being awaited when calling the package from the SQL Server Agent Job versus the execution directly in Visual Studio / SQL Server Data Tools.
The Locale Id of the SSIS / dtsx package was by default set to "German" (which is the default language of my OS) whereas the language in effect on the target SQL Server is "English (US)". The input file contains the decimal values in the US format (dotted).
As soon as i changed the locale ID of the package to "English (United States)" it worked. (it would be also possible to change the decimal separator from a dot to a comma in the input file and leave the SSIS locale ID on German).
Obviously the execution via Visual Studio ignores the language setting on the server (otherwise it would crash in that mode too...)
this seems to be a rather similar question (it lead me to the correct solution):
https://dba.stackexchange.com/questions/88895/difference-in-number-format-comma-and-dot-on-ssms-and-ssis
Im getting this error when running an SSIS project that moves all of our files from FoxPro into SSMS Dbs. The corresponding DBF file opens fine on FoxPro and the memo field which is the FPT file to knowledge works fine too, So I don't what the solution is, I tried making a little changes to the memos to see if that would update the file but that didnt work. I tired re indexing the DBF but no luck there either. How can i generate a new FPT file so I can run this project. How could it be invalid. There is no support online for this.
You are using a third party .NET tool (RatSql.DbfReader). Judging from the exception message this tool was written in a .NET language. It's not a Microsoft product and not part of Visual FoxPro.
DBF files are a family of formats that are similar but have subtle differences. Many tools that claim to support DBF files, will only support DBF files created by certain products, such as VFP 6, FP 2.6, dBase or Clipper.
The options I see for you:
Get in contact with the vendor and make sure that Visual FoxPro tables are supported by their product. They might have an update or can suggest alternative solutions.
If this is an in-house developed product you need to get into touch with the original developers.
Since you have Visual FoxPro, you can convert the file to an older version using the following commands:
USE table
COPY TO newTable TYPE FOX2X
replace table with the path and name of the DBF file, and newTable with the path and name of a new file. Then use the new file for your import. There's no guarantee this works, but the old FoxPro 2.x format is more likely to be supported by third party libraries.
If this is a repeated process, you can compile these commands into an EXE file and incorporate calling the EXE into your import process.
Replace the .NET component with the Visual FoxPro ODBC driver or OLEDB provider. The ODBC driver does not support features introduced after VFP 6, but should work in most cases. Both are only available as 32-bit drivers which might require you to use 32-bit tools to connect to the DBF file.
It might actually be a problem with one ore more records where the pointer to the memo field is wrong. You can find this with Visual FoxPro. Open the DBF file in Visual FoxPro and then open a Browse window. You can do this interactively in the menu or by executing USE and BROWSE commands.
Then double click the column saying "memo" or "Memo". If there are multiple columns, you need to repeat the following step for every column. Now click on the BROWSE window's title bar and hold down the down arrow key. Visual FoxPro should begin scrolling through the table and displaying every record. If there is an error in the memo field, you will get an error message.
Alternatively you can write a FoxPro program that uses a SCAN...ENDSCAN loop and accesses every memo field with =memofieldname if there are too many rows in the table to do this interactively.
I'm trying to integrate a few .dbf and .fpt files into an MS SQL Server using Visual Studio (SSIS) and the OLEDB Visual Fox Pro Driver. For most of the files the integration works fine. But 5 of them are not recognized correctly by the Connection Manager. I tried to figure out the problem (using a HEX Editor) and find out that there is a datatype "M" for a few columns:
STPNAME....M◄... <-ASCII
STPNAME....M.... <-ANSI
This seems not to be a known datatype for .dbf files.
I also tried to open the files by using the App "DBF open File Tool". This Tool could open the database but when i try to fix the file through the recovery function the file gets saved without any changes.
Do you have any ideas how i could fix that issue?
M is the Memo type, which is basically an unbounded Char.
I'm surprised the OLEDB driver doesn't recognize it -- unless the associated .fpt files for those five are missing?
As for working around, you could remove the Memo fields (and lose the data -- unless those .fpt files are gone, in which case the data is already lost).
I'm in the process of creating a suite of packages to import data from our ERP system running on Informix IDS 11.7 into SQL server (2012).
Using SSIS to import this data I've come across an issue. I can access the data in 2 ways, using an ODBC connection and an ADO.NET data source, or using the OLEDB connection and provider.
Using ODBC is about 3 times slower (conservatively!), so naturally I'm keen to move away from that.
However the problem is, when I use OLEDB the data source is reporting the wrong data types.
NVARCHAR data types on the server are being reported as VARCHAR (DT_STR) to SSIS. This causes problem when importing data as any unicode data that come in causes the package to fail. There's no opportunity to do data conversions here, the package is failing when the data hits the data source component. I can set the component to ignore these errors and it will run fine, but with missing data, which isn't acceptable at all.
I've tried setting the DB_LOCALE and CLIENT_LOCAL in setnet32, doesn't have any effect.
When you create the OLEDB data source it complains about the default code page not being able to be found, so you have to set the property to "AlwyasUseDefaultCodePage" to true for that warning to go away, however setting the default code page doesn't seem to actually change it's behavior, it's still trying to bring this data through as VARCHAR even if I set the code page to something like 65001.
So, my question is, how can I either get the OLEDB driver from Informix working correctly, or set/force SSIS to think this data is actually DT_WSTR, not DT_STR?
Thanks!
So, to summarise the steps required to get this working (for posterities sake at least).
When setting up your OLEDB connection you need to add the 2 parameters -
RSASWS=TRUE;
UNICODE=TRUE;
These 2 parameters are NOT shown in the GUI for the connection, at least with the Informix 4.1 driver.
To add these in you need to modify the "ConnectionString" property of the connection, adding these 2 properties to the end of the connection string. Be aware that this property gets overwritten each time you open the connection GUI. You will need to make sure you modify the connection string manually after each time you enter this interface.
The other option for setting the connection string is using variables (or parameters in SSIS on SQL 2012), so any changes that get automatically made to the connection string will be corrected at run time (they actually get corrected at design time when using parameters).
One final caveat I've found with this data source, is appears to be padding out nvarchar's with empty spaces, as if they were nchar's. This could be an issue with my source data though, but something to check if you setting this up, you may need to add a trim step in.
When I open up the solution that contains SSIS packages created by a colleague, I get this awkward error that tells me nothing about what I'm supposed to do to fix it.
He left instructions to take all the "variables" out of the connection string in the dtsx file manually before opening up the solution. I have done that, now when try to view the package in the designer I just get an image of a red x and this message.
EDIT: You cannot see any design elements, no tabs across the top to switch to errors or data flows. Just a gray center area on the screen with a red x, and the message, its like VisualStudio dies in the process of reading the dtsx file.
The question is rather unspecific so it’s of course difficult to get on the right track here. All of the given answers focus different issues. I would say that PeterX had the best guess. The reason for the error could be as simple as a modified data source.
I came across with a bug "error output has no corresponding output" quite often when adding a new column to a table that needs to be processed by an existing SSIS package. This bug came along with an error message saying that a "Value does not fall within the expected range".
A newly added column needed to be processed by an existing SSIS Package. The expected behavior is that SSIS will recognize that there is a new column and select this column on the columns page of the OLEDB Source Task SSIS to be processed. However, when opening the OLEDB Source Task for the first time after having modified the table I got twice the following error message: "Value does not fall within the expected range." The error message showed up when opening the editor and when opening the Columns page of the editor. Within the Advanced Editor of the OLEDB Source Task the new column showed up in the OLEDB Source Output Columns Tree, but not in the OLEDB Source Error Output Columns Tree. This is the actual underlying problem of the error message. Unfortunately, there seems to be no way to add the missing column manually.
To solve the problem, remove and re-add the newly added column on the Columns Page of the normal Editor as mentioned by Jeff.
It is worth to be mentioned that the data source of the OLEDB Source task was a modified MDS View. Microsoft CRM Dynamics – as mentioned in the related thread – is using views, too. That leads me to the conclusion, that using views as a data source may produce either of the above mentioned errors, when modifying datatypes or adding/removing columns.
Related Thread: Error" ...The OLE DB Source.Outputs[OLE DB Source Output].Columns[XXXXXXXX] on the non-error output has no corresponding output
The described workaround refers to Visual Studio 2008 Version 9.0.30729.4462 QFE with Mircorsoft.NET Framework 3.5 SP1. The database is SQL Server 2008 R2 (SP2).
I had to delete and recreate the OLE DB Data source in my Data Flow - this is where I got the error. I also noted I had to "re-select" the "OLE DB connection manager" in the drop-down-list to force it to recognise the new connection.
This was probably a combination of getting the solution from TFS (where I noticed the data-sources didn't come-across properly and it complaining about a missing connection GUID) and/or copying and pasting the elements from another package.
(For BIDS 2008).
I had this issue for my OLE DB Source component with an SQL command after adding new columns to the database, and it wouldn't let me select columns or anything else to add the new columns.
I'm working with an Oracle database, and the only way I could get it to update was to change the SQL query to select 1 from dual, and preview it. Then revert it back to my old query.
You get a similar message if someone uses EncryptAllWithUserKey as the ProtectionLevel. However, I believe the message is slightly different (even though you get a grey design surface with a red X).
Have you tried viewing the file in Notepad? Is it just a series of GUIDs or is there anything in it that is humanly readable? If it doesn't have any readable code, then it was probably encyrpted with the user key.
If the employee deployed the packages to a server and used SQL Server as the deployment destination (not File System or SSIS Pacakge Store) then you can download the packages to your machine. Just connect to the SQL Server Integration Services engine, expand Stored Packages, expand MSDB, expand the relevant folder, right-click on the package, and click Export Package. Save the file on your local machine and open it. The package will probably lose annotations and pretty formatting, but otherwise it should be identical to what the employee deployed.
I just struck the same issue. After flailing about for a bit, I found the solution was to edit the Solution Configuration.
The Solution Configuration appeared to have a matching Project configuration, as shown:
However clicking the drop-down arrow for that Project (SSIS-Advance in this example) revealed that there was no Project Configuration for that project called Production - Sub Reports. I'm not sure how that came about - this Solution has a 7-year history and many developers.
Anyway once I created a New Project configuration (using that same drop-down menu), it is all happy now.
If it has Oracle data sources, you may need to install the Microsoft Connectors v4.0 for Oracle by Attunity:
https://www.microsoft.com/en-us/download/details.aspx?id=52950
I also had to use VS 2015 - the version originally used to create the project and package.
I had this exact problem and installing these connectors and using VS 2015 fixed the issue.
I had this occur as well when I tried to call a stored procedure with OUTPUT parameters with OLE DB.
I found this: http://sqlsolutions.blogspot.com/2013/04/ssis-value-does-not-fall-within.html, which resolved my issue. The relevant action was to rename the SSIS parameter mappings to '0', '1', etc.
So for example, when calling dbo.StoredProc #variable0 = ?, #variable1 = ? OUTPUT, #variable2 = ?;, in the parameter mapping dialog, you would name the parameters '0', '1', 2' to correspond to those. Ah, SSIS <3
I get this when I do not follow the convention for parameter naming, e.g. not name parameters 0,1,2,... in the right order for OLE DB connections.
The details are documented here.
In your connection manager, convert your connections to package level instead of project level
Delete connection manager and re-create and setup ssis package solve the problem.
I got this issue after I Add Existing Connection Manager in a SSIS project. I was just importing a Project Connection Manager from a different project (.conmgr) to my project. My solution to fix the issue was:
Deleting the imported .conmgr
Recreating it from scratch